FetchContent stuck in gitlab-runner pipeline
I posted the issue also here: https://gitlab.com/gitlab-org/gitlab-runner/-/issues/29024 But when it is a cmake issue, it does not belong there. Please help me out here, to find possible cmake related fixes.
CMake Version: 3.23.1-0kitware1ubuntu20.04.1
Summary
I have a docker-gitlab-runner creating docker containers to test our software.
In there, we use cmake and cmake-CPM to fetch all dependencies, which are not part of the package repository.
Directly after a CMake FetchContent, the pipeline is stuck, until it times out. I really don't know what's the problem here, and if this is really a cmake bug, but the only thing I know is, that the pipeline succeeds, when I open the container with sudo docker exec -it runner-<< contairer_id >> /bin/bash
and execute the pipeline script myself.
So it has somehow to do with the fact, that cmake is running in the container.
Steps to reproduce
- create docker based pipeline runner
- have a cmake based project, using dependency management over cpm with that gitlab-ci.yml:
# Ubuntu release-build
license-client-docker-release:
variables:
DOCKER_DRIVER: overlay2
DOCKER_TLS_CERTDIR: "/config/certs/"
image: ubuntu:rolling
stage: build
rules:
- when: always
tags:
- docker
script:
- export DEBIAN_FRONTEND=noninteractive
- ls /config/certs || true
- ls /config ||true
- ls /usr/local/share/ca-certificates/ || true
- ls /usr/local/share/ca-certificates/custom
- apt update -y && apt upgrade -y
- apt install -y wget git ca-certificates gpg
- apt install -y libssl-dev sqlite3 argon2 gcc g++ wget ninja-build git libboost-system-dev libboost-date-time-dev libboost-serialization-dev libboost-program-options-dev
- apt install -y software-properties-common lsb-release && apt clean all
- wget -V
- wget https://apt.kitware.com/keys/kitware-archive-latest.asc
- cat kitware-archive-latest.asc | gpg --dearmor - | tee /etc/apt/trusted.gpg.d/kitware.gpg >/dev/null
- apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 6AF7F09730B3F0A4
- apt-add-repository "deb https://apt.kitware.com/ubuntu/ focal main"
- apt update
- apt install -y kitware-archive-keyring
- rm /etc/apt/trusted.gpg.d/kitware.gpg
- apt update
- apt install -y cmake
- mkdir build-release
- cd build-release
- cmake -GNinja -DCMAKE_BUILD_TYPE=RelWithDebInfo -DFETCHCONTENT_QUIET=Off ..
- cmake --build .
- cmake --build . --target license_client license_server Client
Actual behavior
stuck at FetchContent:
[1/9] Creating directories for 'common-populate'
[1/9] Performing download step (git clone) for 'common-populate'
Cloning into 'common-src'...
HEAD is now at ced8b31 Fixup precompiled header cased missing header.
Expected behavior
the pipeline is not stuck and continues to build
Environment description
- self managed gitlab instance
- self managed gitlab-runner in a docker container using docker executor.
Client:
Context: default
Debug Mode: false
Plugins:
app: Docker App (Docker Inc., v0.9.1-beta3)
buildx: Docker Buildx (Docker Inc., v0.8.0-docker)
scan: Docker Scan (Docker Inc., v0.17.0)
Server:
Containers: 23
Running: 2
Paused: 0
Stopped: 21
Images: 8
Server Version: 20.10.13
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 2a1d4dbdb2a1030dc5b01e96fb110a9d9f150ecc
runc version: v1.0.3-0-gf46b6ba
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: default
cgroupns
Kernel Version: 5.13.0-35-generic
Operating System: Ubuntu 21.10
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 1.145GiB
Name: srvdoc01
ID: WM5W:2ZW6:KSY7:3EJP:ZYWI:TS64:XE4P:TOLT:Z34Y:2NF4:WPED:XOXH
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
config.toml contents
concurrent = 20
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "docker-runner"
url = "https://gitlab.my-gitlab.de"
token = "sec"
executor = "docker"
pre_build_script = """
apt-get update -y > /dev/null
apt-get install -y ca-certificates git > /dev/null
mkdir -p /usr/local/share/ca-certificates/custom
cp -R -a /config/certs/. /usr/local/share/ca-certificates/custom/
update-ca-certificates --fresh > /dev/null
git config --global credential.helper "cache --timeout=3600"
git config --global url."https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.my-gitlab.de/".insteadOf "ssh://git@gitlab.optomet.de/"
git config --global url."https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.my-gitlab.de/".insteadOf "git://git@gitlab.my-gitlab.de:"
git config --global url."https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.my-gitlab.de/".insteadOf "https://gitlab.my-gitlab.de/"
"""
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[runners.docker]
tls_verify = false
image = "ubuntu:latest"
privileged = false
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache", "gitlab-runner-config:/config:ro"]
shm_size = 0
Used GitLab Runner version
sudo docker exec -it gitlab-runner gitlab-runner --version
Version: 14.9.1
Git revision: bd40e3da
Git branch: 14-9-stable
GO version: go1.17.7
Built: 2022-03-22T21:26:27+0000
OS/Arch: linux/amd64