Build 17.6.0 timed out

[lekoder]

CI build for 17.6.0 seems to have timed out and 17.6.0 is not available on dockerhub. Please run the task again.

[kkimurak]

duplicate of #3036 and meaningless suggestion.

The build time of gitlab itself is getting longer and longer, and will almost certainly reach 1 hour (the upper limit in the free plan). If we don't review the build process itself, we will just waste time running CI.

[lekoder]

Apologies, search didn't pick that up, since the specific version is not mentioned in the issue.

Might I suggest a workaround then of building it in another environment and pushing to the dockerhub?

[kkimurak]

Might I suggest a workaround then of building it in another environment and pushing to the dockerhub?

I'm running a locally built version of 17.6.0, but if someone publishes it on dockerhub it will help people who can't build it themselves.
On the other hand, I'm just a contributor. You should ask the maintainer who has the final words (@sachilles has been the only active one in recent years)

[lekoder]

Fair enough. In the meanwhile, if anybody needs this very image built to 17.6.0 up to 17.6.2 I forked it, built them and published on the dockerhub here.

[th-2021]

I'm working on a multistage Dockerfile. Using BuildKit (docker buildx build or DOCKER_BUILDKIT=1 docker build), which allows parallel execution of stages I could save around 5 minutes.

[sachilles]

@kkimurak Unfortunately, I don't have the option of giving you the role of maintainer.

@sameersbn Do you see a possibility to either change the plan for the CI or take up kkimurak's offer. He has also maintained the project remarkably so far.

[tDo]

@sachilles @kkimurak given that @sameersbn does not seem to be very active on github anymore (At least there have been no contributions in the last year) would moving the ci/cd + docker-image to a new namespace controlled by either of you be a feasible option?

[Thomas-Ganter]

Just for curiosity’s sake: What are the requirements to build the image?
I got errors 137 after 2 hours and after increasing memory to 6GB error 134 during the build after 3.5 hours — I found no info in any "building yourself" docu (or was merely looking in the wrong places).

Any insights?

[kkimurak]

@tDo If necessary, we should do so. I would like to get permission from all maintainers (with respect) if possible, but if we can't contact them, there is nothing we can do.
Stalled project are one of my greatest fears, and the confusion that would result from moving namespaces is a minor pain compared to that.

However, I am still just a contributor (the most active - at least in terms of commits - except for the maintainer) and do not have (nor will I have) any decision-making authority.

So what I'm trying to say is that I'm asking for more human resources with administrative privileges that can respond to emergencies.

[kkimurak]

@Thomas-Ganter I don't know what the actual requirements are, but I think it's less than 6GB.

Hints for 6GB memory requirement:

Hint 1 : rake task gitlab:assets:compile is executed during build

docker-gitlab/assets/build/install.sh

Line 215 in 76dad78

exec_as_git bundle exec rake gitlab:assets:compile USE_DB=false SKIP_STORAGE_VALIDATION=true NODE_OPTIONS="--max-old-space-size=4096"

Hint 2 : It invokes system command yarn webpack (with some options based on environment variable - to detect CI)
https://gitlab.com/gitlab-org/gitlab/-/blob/f42b800081f2c4c751e8fc59df70857704c1ddee/lib/tasks/gitlab/assets.rake#L120-133

Hint 3 : node command webpack is defined in package.json and it sets --max-old-space-size to 5120
https://gitlab.com/gitlab-org/gitlab/-/blob/f42b800081f2c4c751e8fc59df70857704c1ddee/package.json#L45

Anyway, I usually do local builds on two different virtual machines (Ubuntu) with 8GB / 16GB RAM. I have run out of memory once before on an 8GB machine, but since then I have disabled most services, including the GUI, and do no other work during the build, and have not run out of memory.

[Thomas-Ganter]

@Thomas-Ganter I don't know what the actual requirements are, but I think it's less than 6GB.

Hints for 6GB memory requirement:
[…]

Thanks @kkimurak — Running the build on a dedicated 8 core / 60GB machine worked for the single-architecture build, the multi-arch build timed out after 2 hrs and 11GB consumed memory in the builder container. Bumped the pipeline timeout to 4hrs and trying again ...

[th-2021]

Running a remote build (buildx with kubernetes driver) with 4 CPU, 4 GB memory worked (in 40 minutes) Am 31.12.2024 um 11:02 schrieb Thomas Ganter:

…

@Thomas-Ganter <https://github.com/Thomas-Ganter> I don't know what the actual requirements are, but I think it's less than 6GB. Hints for 6GB memory requirement: […] Thanks @kkimurak <https://github.com/kkimurak> — Running the build on a dedicated 8 core / 60GB machine worked for the single-architecture build, the multi-arch build timed out after 2 hrs and 11GB consumed memory in the builder container. Bumped the pipeline timeout to 4hrs and trying again ... — Reply to this email directly, view it on GitHub <#3042 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AVVFCFYVCKD7I4S3Y2WNHC32IJTT5AVCNFSM6AAAAABTK5PYLWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNRWGMYDKNRQGU>. You are receiving this because you commented.Message ID: ***@***.***>

[Thomas-Ganter]

Running a remote build (buildx with kubernetes driver) with 4 CPU, 4 GB memory worked (in 40 minutes)

Hmmm … for me a build with a 4hrs timeout just aborted.
I now have bumped the timeout to 8hrs, let's see whether this helps.

Then, I also do not know how to properly parallelize multi-architecture builds.
Maybe this is my incompetence being the main culprit in my build drama ... any hints welcome.

[th-2021]

I've no experience with multi-platform builds, but I'm working on multistage (parallelized build steps), which might help. Am 01.01.2025 um 02:09 schrieb Thomas Ganter:

…

Running a remote build (buildx with kubernetes driver) with 4 CPU, 4 GB memory worked (in 40 minutes) Hmmm … for me a build with a 4hrs timeout just aborted. I now have bumped the timeout to 8hrs, let's see whether this helps. Then, I also do not know how to properly parallelize multi-architecture builds. Maybe this is my incompetence being the main culprit in my build drama ... any hints welcome. — Reply to this email directly, view it on GitHub <#3042 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AVVFCF56TOJUJLSRVUHQUBT2IM523AVCNFSM6AAAAABTK5PYLWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNRWG44DIOJVGU>. You are receiving this because you commented.Message ID: ***@***.***>

[Thomas-Ganter]

I've no experience with multi-platform builds, but I'm working on multistage (parallelized build steps), which might help.

Sounds interesting.
As the bulk of the time is spent inside the "install.sh", I was also wondering whether this could be split up into separate steps (which would yield in cacheable intermediate layers) which would make it easier to restart builds …

[th-2021]

Please see https://github.com/th-2021/docker-gitlab/ multistage branch.
Dockerfile.multistage / install2.sh are the new files (kept separate for now)

docker buildx build -o type=registry -f Dockerfile.multistage --build-arg MAX_OLD_SPACE=${MAX_OLD_SPACE} . -t gitlab:17.6.0.1'
MAX_OLD_SPACE=8192 works fine

gitlab is coming up, but more testing is needed.

[Thomas-Ganter]

My multi-arch-build also just completed a few minutes ago for the first time, and the memory requirement was hillarious:

And those 2:36 hours were on a 12 core 32 GB VM on an 14th gen Core i9 … and yes, also here, more testing is needed, but it looks good so far.

[th-2021]

Are there additional patches needed for arm? I can try it on my setup.

[kkimurak]

@th-2021 See #2803 - you may be need to edit around golang installation : https://github.com/th-2021/docker-gitlab/blob/616e07f1dc73851c13ebd678c94984b234288fc8/Dockerfile.multistage#L103

[Thomas-Ganter]

Are there additional patches needed for arm? I can try it on my setup.

These are the changes I had to make

• in the .gitlab-ci.yml file:
  bump Docker version, introduce buildx, remove the separate push
• in the Dockerfile:
  included some scripting to properly set the architecture for later use
• in the install.sh script:
  included the arch in the golang download URL, bumped the Node Memory, added lots of debugging output (becuase it was super difficult to understand where when what and why things failed).

My repo should be publicly reviewable — if not please ping me and I will remediate.

Once I am sufficiently confident I will update my homelab install, and then let's see … 8^)

But beware … resource demand for a build is higher than a single-platform build:

Update 2025-01-03

in the Dockerfile, additional packages are required:

      libmagic1 \
      libpixman-1-dev libcairo2-dev libpango1.0-dev libjpeg8-dev libgif-dev \    <---- required on ARM (but doesn't hurt on x86)
 && update-locale LANG=C.UTF-8 LC_MESSAGES=POSIX \

Background: The install.sh script downloads https://github.com/Automattic/node-canvas in binary form, but there only is an x86 release, hence it tries to build it from source, which fails due to the missing pixman-1 dependency. The above libs are listed as build prerequisites ...