Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cilium-envoy failing on RPi4 + RaspbianOS #1027

Open
dioguerra opened this issue Dec 7, 2024 · 7 comments
Open

cilium-envoy failing on RPi4 + RaspbianOS #1027

dioguerra opened this issue Dec 7, 2024 · 7 comments

Comments

@dioguerra
Copy link

While trying to deploy cilium in my RPi4, cilium-envoy crashes wih:

external/com_github_google_tcmalloc/tcmalloc/system-alloc.cc:625] MmapAligned() failed - unable to allocate with tag (hint, size, alignment) - is something limiting address placement? 0x15acc0000000 1073741824 1073741824 @ 0x555ab304c4 0x555ab2c848 0x555ab2c1d4 0x555ab0ba40 0x555ab29ec8 0x555ab29c9c 0x555ab01208 0x555aa1a8e0 0x555aa19614 0x7fb577754c
external/com_github_google_tcmalloc/tcmalloc/arena.cc:58] FATAL ERROR: Out of memory trying to allocate internal tcmalloc data (bytes, object-size); is something preventing mmap from succeeding (sandbox, VSS limitations)? 131072 632 @ 0x555ab30834 0x555ab0bad0 0x555ab29ec8 0x555ab29c9c 0x555ab01208 0x555aa1a8e0 0x555aa19614 0x7fb577754c

I still have available RAM on the system. Further debugging might indicate that i am missing some Kernel enabled configs:

cat config-6.6.62+rpt-rpi-* | grep -E 'CONFIG_BPF_JIT|CONFIG_NET_SCH_INGRESS|CONFIG_CRYPTO_USER_API_HASH'
CONFIG_BPF_JIT=y
# CONFIG_BPF_JIT_ALWAYS_ON is not set
CONFIG_BPF_JIT_DEFAULT_ON=y
CONFIG_NET_SCH_INGRESS=m
CONFIG_CRYPTO_USER_API_HASH=m
CONFIG_BPF_JIT=y
# CONFIG_BPF_JIT_ALWAYS_ON is not set
CONFIG_BPF_JIT_DEFAULT_ON=y
CONFIG_NET_SCH_INGRESS=m
CONFIG_CRYPTO_USER_API_HASH=m

From page https://docs.cilium.io/en/latest/operations/system_requirements/#base-requirements. Cant really install linux-modules-extra-raspi from raspios unless i download and install the packages from the ubuntu which might be compatible.
Otherwise would need to rebuild the kernel?(which i never did) and might not persist changes between future updates./

Not exactly sure on the best course of action here:

cilium was deployed with cilium-cli downloaded directly on the RPi node with:

cilium install   --helm-set ipam.operator.clusterPoolIPv6MaskSize=120   --helm-set ipv6.enabled=true   --helm-set ipv4.enabled=false   --helm-set routingMode=native   --helm-set autoDirectNodeRoutes=true   --helm-set ipv6NativeRoutingCIDR=bipbop::/64   --helm-set kubeProxyReplacement=true   --helm-set ipam.operator.clusterPoolIPv6PodCIDRList="fd00:10:244::/104"   --helm-set nodeinit.enabled=true   --helm-set k8s.requireIPv6PodCIDR=true   --helm-set k8sServiceHost=bipbop   --helm-set k8sServicePort=6443   --helm-set hubble.enabled=true   --helm-set hubble.relay.enabled=true
@sayboras
Copy link
Member

👋

We have one workaround with gperftools, can you give it a try ?

https://github.com/cilium/proxy?tab=readme-ov-file#building-for-the-raspberry-pi-kernelc

@dioguerra
Copy link
Author

👋

We have one workaround with gperftools, can you give it a try ?

https://github.com/cilium/proxy?tab=readme-ov-file#building-for-the-raspberry-pi-kernelc

Should be able to cross compile yeah? Having some errors thrown out, need to check with more attention

@dioguerra
Copy link
Author

Very cryptic failure, not sure what. Seems like I can build fine without using the extra build argument
BAZEL_BUILD_OPTS="--define tcmalloc=gperftools"

#17 8.241 [1 / 10] [Prepa] BazelWorkspaceStatusAction stable-status.txt
#17 9.682 [3,622 / 9,640] Foreign Cc - Configure: Building gperftools_build; 0s remote-cache, processwrapper-sandbox ... (4 actions, 2 running)
#17 10.73 [6,852 / 9,646] Foreign Cc - Configure: Building gperftools_build; 1s remote-cache, processwrapper-sandbox ... (29 actions, 28 running)
#17 11.74 [6,852 / 9,646] Foreign Cc - Configure: Building gperftools_build; 2s remote-cache, processwrapper-sandbox ... (30 actions running)
#17 11.79 ERROR: /cilium/proxy/.cache/bazel/_bazel_cilium/5f5d3cf254342807836215c78e76aef5/external/envoy/bazel/foreign_cc/BUILD:32:15: Foreign Cc - Configure: Building gperftools_build failed: (Exit 77): bash failed: error executing command (from target @envoy//bazel/foreign_cc:gperftools_build) 
#17 11.79   (cd /cilium/proxy/.cache/bazel/_bazel_cilium/5f5d3cf254342807836215c78e76aef5/sandbox/processwrapper-sandbox/1/execroot/cilium && \
#17 11.79   exec env - \
#17 11.79     BAZEL_COMPILER=clang \
#17 11.79     BAZEL_LINKLIBS=-l%:libstdc++.a \
#17 11.79     BAZEL_LINKOPTS=-lm \
#17 11.79     CC=/usr/lib/llvm-17/bin/clang \
#17 11.79     CXX=/usr/lib/llvm-17/bin/clang++ \
#17 11.79     LLVM_CONFIG=/usr/lib/llvm-17/bin/llvm-config \
#17 11.79     PATH=/usr/lib/llvm-17/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
#17 11.79   /bin/bash -c bazel-out/aarch64-opt/bin/external/envoy/bazel/foreign_cc/gperftools_build_foreign_cc/wrapper_build_script.sh)
#17 11.79 # Configuration: d13fb2b1db2396dd0d63c26115f21fb2fb530ca15c43b13b95e8be6f91e06e6a
#17 11.79 # Execution platform: @local_config_platform//:host
#17 11.79 
#17 11.79 Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
#17 11.79 rules_foreign_cc: Build failed!
#17 11.79 rules_foreign_cc: Keeping temp build directory and dependencies directory for debug.
#17 11.79 rules_foreign_cc: Please note that the directories inside a sandbox are still cleaned unless you specify --sandbox_debug Bazel command line flag.
#17 11.79 rules_foreign_cc: Printing build logs:
#17 11.79 _____ BEGIN BUILD LOGS _____
#17 11.79 
#17 11.79 Bazel external C/C++ Rules. Building library gperftools_build
#17 11.79 
#17 11.79 Environment:______________
#17 11.79 BUILD_SCRIPT=bazel-out/aarch64-opt/bin/external/envoy/bazel/foreign_cc/gperftools_build_foreign_cc/build_script.sh
#17 11.79 EXT_BUILD_ROOT=/cilium/proxy/.cache/bazel/_bazel_cilium/5f5d3cf254342807836215c78e76aef5/sandbox/processwrapper-sandbox/1/execroot/cilium
#17 11.79 BUILD_LOG=bazel-out/aarch64-opt/bin/external/envoy/bazel/foreign_cc/gperftools_build_foreign_cc/Configure.log
#17 11.79 PWD=/cilium/proxy/.cache/bazel/_bazel_cilium/5f5d3cf254342807836215c78e76aef5/sandbox/processwrapper-sandbox/1/execroot/cilium
#17 11.79 LLVM_CONFIG=/usr/lib/llvm-17/bin/llvm-config
#17 11.79 CXX=/usr/lib/llvm-17/bin/clang++
#17 11.79 BUILD_WRAPPER_SCRIPT=bazel-out/aarch64-opt/bin/external/envoy/bazel/foreign_cc/gperftools_build_foreign_cc/wrapper_build_script.sh
#17 11.79 BAZEL_LINKOPTS=-lm
#17 11.79 TMPDIR=/tmp
#17 11.79 EXT_BUILD_DEPS=/cilium/proxy/.cache/bazel/_bazel_cilium/5f5d3cf254342807836215c78e76aef5/sandbox/processwrapper-sandbox/1/execroot/cilium/bazel-out/aarch64-opt/bin/external/envoy/bazel/foreign_cc/gperftools_build.ext_build_deps
#17 11.79 BAZEL_LINKLIBS=-l%:libstdc++.a
#17 11.79 BUILD_TMPDIR=/cilium/proxy/.cache/bazel/_bazel_cilium/5f5d3cf254342807836215c78e76aef5/sandbox/processwrapper-sandbox/1/execroot/cilium/bazel-out/aarch64-opt/bin/external/envoy/bazel/foreign_cc/gperftools_build.build_tmpdir
#17 11.79 SHLVL=2
#17 11.79 INSTALLDIR=/cilium/proxy/.cache/bazel/_bazel_cilium/5f5d3cf254342807836215c78e76aef5/sandbox/processwrapper-sandbox/1/execroot/cilium/bazel-out/aarch64-opt/bin/external/envoy/bazel/foreign_cc/gperftools_build
#17 11.79 PATH=/cilium/proxy/.cache/bazel/_bazel_cilium/5f5d3cf254342807836215c78e76aef5/sandbox/processwrapper-sandbox/1/execroot/cilium:/usr/lib/llvm-17/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
#17 11.79 BAZEL_COMPILER=clang
#17 11.79 CC=/usr/lib/llvm-17/bin/clang
#17 11.79 _=/usr/bin/env
#17 11.79 __________________________
#17 11.79 + mkdir -p /cilium/proxy/.cache/bazel/_bazel_cilium/5f5d3cf254342807836215c78e76aef5/sandbox/processwrapper-sandbox/1/execroot/cilium/bazel-out/aarch64-opt/bin/external/envoy/bazel/foreign_cc/gperftools_build.build_tmpdir/gperftools_build
#17 11.79 + ARFLAGS=rcsD
#17 11.79 + AR_FLAGS=rcsD
#17 11.79 + ASFLAGS='--target=aarch64-unknown-linux-gnu -fuse-ld=lld -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -Wno-unused-command-line-argument -Wno-deprecated-builtins -g0 -O2 -D_FORTIFY_SOURCE=1 -DNDEBUG -ffunction-sections -fdata-sections -Wno-builtin-macro-redefined -D__DATE__=redacted -D__TIMESTAMP__=redacted -D__TIME__=redacted -DABSL_MIN_LOG_LEVEL=4 -fdebug-types-section -fPIC -Wno-deprecated-declarations -fexceptions'
#17 11.79 + CFLAGS='--target=aarch64-unknown-linux-gnu -fuse-ld=lld -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -Wno-unused-command-line-argument -Wno-deprecated-builtins -g0 -O2 -D_FORTIFY_SOURCE=1 -DNDEBUG -ffunction-sections -fdata-sections -Wno-builtin-macro-redefined -D__DATE__=redacted -D__TIMESTAMP__=redacted -D__TIME__=redacted -DABSL_MIN_LOG_LEVEL=4 -fdebug-types-section -fPIC -Wno-deprecated-declarations -fexceptions'
#17 11.79 + CXXFLAGS='--target=aarch64-unknown-linux-gnu -fuse-ld=lld -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -Wno-unused-command-line-argument -Wno-deprecated-builtins -g0 -O2 -D_FORTIFY_SOURCE=1 -DNDEBUG -ffunction-sections -fdata-sections -std=c++0x -Wno-builtin-macro-redefined -D__DATE__=redacted -D__TIMESTAMP__=redacted -D__TIME__=redacted -DABSL_MIN_LOG_LEVEL=4 -fdebug-types-section -fPIC -Wno-deprecated-declarations -std=c++17'
#17 11.79 + LDFLAGS='--target=aarch64-unknown-linux-gnu -fuse-ld=lld -Wl,-no-as-needed -Wl,-z,relro,-z,now -lm -Wl,--gc-sections -l:libstdc++.a -L/usr/lib/llvm-17/lib -Wl,-rpath,/usr/lib/llvm-17/lib -Wl,--strip-all -lpthread'
#17 11.79 + AR=/usr/bin/llvm-ar-17
#17 11.79 + CC=/usr/bin/clang-17
#17 11.79 + CXX=/usr/bin/clang-17
#17 11.79 + RANLIB=:
#17 11.79 + CPPFLAGS=
#17 11.79 + /cilium/proxy/.cache/bazel/_bazel_cilium/5f5d3cf254342807836215c78e76aef5/sandbox/processwrapper-sandbox/1/execroot/cilium/external/com_github_gperftools_gperftools/configure --prefix=/cilium/proxy/.cache/bazel/_bazel_cilium/5f5d3cf254342807836215c78e76aef5/sandbox/processwrapper-sandbox/1/execroot/cilium/bazel-out/aarch64-opt/bin/external/envoy/bazel/foreign_cc/gperftools_build.build_tmpdir/gperftools_build --enable-shared=no --enable-frame-pointers --disable-libunwind
#17 11.79 checking build system type... x86_64-pc-linux-gnu
#17 11.79 checking host system type... x86_64-pc-linux-gnu
#17 11.79 checking for a BSD-compatible install... /usr/bin/install -c
#17 11.79 checking whether build environment is sane... yes
#17 11.79 checking for a race-free mkdir -p... /usr/bin/mkdir -p
#17 11.79 checking for gawk... no
#17 11.79 checking for mawk... mawk
#17 11.79 checking whether /cilium/proxy/.cache/bazel/_bazel_cilium/5f5d3cf254342807836215c78e76aef5/sandbox/processwrapper-sandbox/1/execroot/cilium/bazel-out/k8-opt-exec-2B5CBBC6/bin/external/rules_foreign_cc/toolchains/make/bin/make sets $(MAKE)... yes
#17 11.79 checking whether /cilium/proxy/.cache/bazel/_bazel_cilium/5f5d3cf254342807836215c78e76aef5/sandbox/processwrapper-sandbox/1/execroot/cilium/bazel-out/k8-opt-exec-2B5CBBC6/bin/external/rules_foreign_cc/toolchains/make/bin/make supports nested variables... yes
#17 11.79 checking whether to enable maintainer-specific portions of Makefiles... no
#17 11.79 checking for git... /usr/bin/git
#17 11.79 checking whether /cilium/proxy/.cache/bazel/_bazel_cilium/5f5d3cf254342807836215c78e76aef5/sandbox/processwrapper-sandbox/1/execroot/cilium/bazel-out/k8-opt-exec-2B5CBBC6/bin/external/rules_foreign_cc/toolchains/make/bin/make supports the include directive... yes (GNU style)
#17 11.79 checking for gcc... /usr/bin/clang-17
#17 11.79 checking whether the C compiler works... yes
#17 11.79 checking for C compiler default output file name... a.out
#17 11.79 checking for suffix of executables... 
#17 11.79 checking whether we are cross compiling... configure: error: in `/cilium/proxy/.cache/bazel/_bazel_cilium/5f5d3cf254342807836215c78e76aef5/sandbox/processwrapper-sandbox/1/execroot/cilium/bazel-out/aarch64-opt/bin/external/envoy/bazel/foreign_cc/gperftools_build.build_tmpdir':
#17 11.79 configure: error: cannot run C compiled programs.
#17 11.79 If you meant to cross compile, use `--host'.
#17 11.79 See `config.log' for more details
#17 11.79 _____ END BUILD LOGS _____
#17 11.79 rules_foreign_cc: Build wrapper script location: bazel-out/aarch64-opt/bin/external/envoy/bazel/foreign_cc/gperftools_build_foreign_cc/wrapper_build_script.sh
#17 11.79 rules_foreign_cc: Build script location: bazel-out/aarch64-opt/bin/external/envoy/bazel/foreign_cc/gperftools_build_foreign_cc/build_script.sh
#17 11.79 rules_foreign_cc: Build log location: bazel-out/aarch64-opt/bin/external/envoy/bazel/foreign_cc/gperftools_build_foreign_cc/Configure.log
#17 11.79 
#17 12.04 Target //:cilium-envoy failed to build
#17 12.05 INFO: Elapsed time: 8.387s, Critical Path: 3.38s
#17 12.05 INFO: 16 processes: 16 internal.
#17 12.05 FAILED: Build did NOT complete successfully
#17 12.05 make: *** [Makefile:121: bazel-bin/cilium-envoy] Error 1
#17 ERROR: process "/bin/sh -c if [ -f /tmp/bazel-cache/ENVOY_VERSION ]; then CACHE_ENVOY_VERSION=`cat /tmp/bazel-cache/ENVOY_VERSION`; ENVOY_VERSION=`cat ENVOY_VERSION`; if [ \"${CACHE_ENVOY_VERSION}\" != \"${ENVOY_VERSION}\" ]; then echo \"Building Envoy ${ENVOY_VERSION} with bazel archive from different Envoy version (${CACHE_ENVOY_VERSION})\"; else echo \"Building Envoy ${ENVOY_VERSION} with bazel cache of the same version\"; fi; else echo \"Bazel cache has no ENVOY_VERSION, it may be empty.\"; fi &&     touch /tmp/bazel-cache/permissions-check &&     if [ -n \"${COPY_CACHE_EXT}\" ]; then PKG_BUILD=1 make BUILD_DEP_HASHES; if [ -f /tmp/bazel-cache/BUILD_DEP_HASHES ] && ! diff BUILD_DEP_HASHES /tmp/bazel-cache/BUILD_DEP_HASHES; then echo \"Build dependencies have changed, clearing bazel cache\"; rm -rf /tmp/bazel-cache/*; rm -rf /cilium/proxy/.cache/*; fi ; cp BUILD_DEP_HASHES ENVOY_VERSION /tmp/bazel-cache; fi &&     BAZEL_BUILD_OPTS=\"${BAZEL_BUILD_OPTS} --disk_cache=/tmp/bazel-cache\" PKG_BUILD=1 V=$V DEBUG=$DEBUG DESTDIR=/tmp/install make install &&     if [ -n \"${COPY_CACHE_EXT}\" ]; then cp -ra /tmp/bazel-cache /tmp/bazel-cache${COPY_CACHE_EXT}; ls -la /tmp/bazel-cache${COPY_CACHE_EXT}; fi" did not complete successfully: exit code: 2
------
 > [builder 5/6] RUN --mount=mode=0777,uid=1337,gid=1337,target=/cilium/proxy/.cache,type=cache,id=arm64,sharing=private     --mount=target=/tmp/bazel-cache,source=/tmp/bazel-cache,from=builder-cache,rw     if [ -f /tmp/bazel-cache/ENVOY_VERSION ]; then CACHE_ENVOY_VERSION=`cat /tmp/bazel-cache/ENVOY_VERSION`; ENVOY_VERSION=`cat ENVOY_VERSION`; if [ "${CACHE_ENVOY_VERSION}" != "${ENVOY_VERSION}" ]; then echo "Building Envoy ${ENVOY_VERSION} with bazel archive from different Envoy version (${CACHE_ENVOY_VERSION})"; else echo "Building Envoy ${ENVOY_VERSION} with bazel cache of the same version"; fi; else echo "Bazel cache has no ENVOY_VERSION, it may be empty."; fi &&     touch /tmp/bazel-cache/permissions-check &&     if [ -n "${COPY_CACHE_EXT}" ]; then PKG_BUILD=1 make BUILD_DEP_HASHES; if [ -f /tmp/bazel-cache/BUILD_DEP_HASHES ] && ! diff BUILD_DEP_HASHES /tmp/bazel-cache/BUILD_DEP_HASHES; then echo "Build dependencies have changed, clearing bazel cache"; rm -rf /tmp/bazel-cache/*; rm -rf /cilium/proxy/.cache/*; fi ; cp BUILD_DEP_HASHES ENVOY_VERSION /tmp/bazel-cache; fi &&     BAZEL_BUILD_OPTS="--define tcmalloc=gperftools --verbose_failures --disk_cache=/tmp/bazel-cache" PKG_BUILD=1 V=$V DEBUG=$DEBUG DESTDIR=/tmp/install make install &&     if [ -n "${COPY_CACHE_EXT}" ]; then cp -ra /tmp/bazel-cache /tmp/bazel-cache${COPY_CACHE_EXT}; ls -la /tmp/bazel-cache${COPY_CACHE_EXT}; fi:
11.79 _____ END BUILD LOGS _____
11.79 rules_foreign_cc: Build wrapper script location: bazel-out/aarch64-opt/bin/external/envoy/bazel/foreign_cc/gperftools_build_foreign_cc/wrapper_build_script.sh
11.79 rules_foreign_cc: Build script location: bazel-out/aarch64-opt/bin/external/envoy/bazel/foreign_cc/gperftools_build_foreign_cc/build_script.sh
11.79 rules_foreign_cc: Build log location: bazel-out/aarch64-opt/bin/external/envoy/bazel/foreign_cc/gperftools_build_foreign_cc/Configure.log
11.79 
12.04 Target //:cilium-envoy failed to build
12.05 INFO: Elapsed time: 8.387s, Critical Path: 3.38s
12.05 INFO: 16 processes: 16 internal.
12.05 FAILED: Build did NOT complete successfully
12.05 make: *** [Makefile:121: bazel-bin/cilium-envoy] Error 1
------
Dockerfile:69
--------------------
  68 |     RUN ./bazel/get_workspace_status
  69 | >>> RUN --mount=mode=0777,uid=1337,gid=1337,target=/cilium/proxy/.cache,type=cache,id=$TARGETARCH,sharing=private \
  70 | >>>     --mount=target=/tmp/bazel-cache,source=/tmp/bazel-cache,from=builder-cache,rw \
  71 | >>>     if [ -f /tmp/bazel-cache/ENVOY_VERSION ]; then CACHE_ENVOY_VERSION=`cat /tmp/bazel-cache/ENVOY_VERSION`; ENVOY_VERSION=`cat ENVOY_VERSION`; if [ "${CACHE_ENVOY_VERSION}" != "${ENVOY_VERSION}" ]; then echo "Building Envoy ${ENVOY_VERSION} with bazel archive from different Envoy version (${CACHE_ENVOY_VERSION})"; else echo "Building Envoy ${ENVOY_VERSION} with bazel cache of the same version"; fi; else echo "Bazel cache has no ENVOY_VERSION, it may be empty."; fi && \
  72 | >>>     touch /tmp/bazel-cache/permissions-check && \
  73 | >>>     if [ -n "${COPY_CACHE_EXT}" ]; then PKG_BUILD=1 make BUILD_DEP_HASHES; if [ -f /tmp/bazel-cache/BUILD_DEP_HASHES ] && ! diff BUILD_DEP_HASHES /tmp/bazel-cache/BUILD_DEP_HASHES; then echo "Build dependencies have changed, clearing bazel cache"; rm -rf /tmp/bazel-cache/*; rm -rf /cilium/proxy/.cache/*; fi ; cp BUILD_DEP_HASHES ENVOY_VERSION /tmp/bazel-cache; fi && \
  74 | >>>     BAZEL_BUILD_OPTS="${BAZEL_BUILD_OPTS} --disk_cache=/tmp/bazel-cache" PKG_BUILD=1 V=$V DEBUG=$DEBUG DESTDIR=/tmp/install make install && \
  75 | >>>     if [ -n "${COPY_CACHE_EXT}" ]; then cp -ra /tmp/bazel-cache /tmp/bazel-cache${COPY_CACHE_EXT}; ls -la /tmp/bazel-cache${COPY_CACHE_EXT}; fi
  76 |     #
--------------------
ERROR: failed to solve: process "/bin/sh -c if [ -f /tmp/bazel-cache/ENVOY_VERSION ]; then CACHE_ENVOY_VERSION=`cat /tmp/bazel-cache/ENVOY_VERSION`; ENVOY_VERSION=`cat ENVOY_VERSION`; if [ \"${CACHE_ENVOY_VERSION}\" != \"${ENVOY_VERSION}\" ]; then echo \"Building Envoy ${ENVOY_VERSION} with bazel archive from different Envoy version (${CACHE_ENVOY_VERSION})\"; else echo \"Building Envoy ${ENVOY_VERSION} with bazel cache of the same version\"; fi; else echo \"Bazel cache has no ENVOY_VERSION, it may be empty.\"; fi &&     touch /tmp/bazel-cache/permissions-check &&     if [ -n \"${COPY_CACHE_EXT}\" ]; then PKG_BUILD=1 make BUILD_DEP_HASHES; if [ -f /tmp/bazel-cache/BUILD_DEP_HASHES ] && ! diff BUILD_DEP_HASHES /tmp/bazel-cache/BUILD_DEP_HASHES; then echo \"Build dependencies have changed, clearing bazel cache\"; rm -rf /tmp/bazel-cache/*; rm -rf /cilium/proxy/.cache/*; fi ; cp BUILD_DEP_HASHES ENVOY_VERSION /tmp/bazel-cache; fi &&     BAZEL_BUILD_OPTS=\"${BAZEL_BUILD_OPTS} --disk_cache=/tmp/bazel-cache\" PKG_BUILD=1 V=$V DEBUG=$DEBUG DESTDIR=/tmp/install make install &&     if [ -n \"${COPY_CACHE_EXT}\" ]; then cp -ra /tmp/bazel-cache /tmp/bazel-cache${COPY_CACHE_EXT}; ls -la /tmp/bazel-cache${COPY_CACHE_EXT}; fi" did not complete successfully: exit code: 2
make: *** [Makefile.docker:221: docker-image-envoy] Error 1

@dioguerra
Copy link
Author

dioguerra commented Dec 15, 2024 •
edited
Loading

After analyzing this a bit. It seems that there is something happening with cross-compile and this gperftools options.

From the output it looks like when cross-compiling I need to use a --host flag

#17 11.79 configure: error: cannot run C compiled programs.
#17 11.79 If you meant to cross compile, use `--host'.
#17 11.79 See `config.log' for more details

I dont seem to be able to find this config.log file (host or buildx container) and also trying out to include --host in bazel configs throws an error. This time i changed the options to clear the cache, to make sure something weird was not happening:

NO_CACHE=1 NO_ARCHIVE=1 DOCKER_BUILD_OPTS="--output=out" ARCH=multi  BAZEL_BUILD_OPTS="--define tcmalloc=gperftools --verbose_failures --sandbox_debug" sudo -E make  docker-image-envoy

For reference, i'm trying to compile for v1.29, commit ID 2c13e1e.
Also noticed that new versions are breaking do to missing dependent containers

@dioguerra
Copy link
Author

dioguerra commented Dec 19, 2024 •
edited
Loading

My current process is to try and build the image(only arm) in the RPI4 directly, which will take some time...

BUILDKIT_PROGRESS=plain NO_CACHE=1 NO_ARCHIVE=1 DOCKER_BUILD_OPTS="--output=out" ARCH=arm64 BAZEL_BUILD_OPTS="--define tcmalloc=gperftools --verbose_failures --sandbox_debug" sudo -E make docker-image-envoy > build.log

Will then build the amd one and stich them with buildx or something if it works. 🤞
Edit: after writting this, dont think it really makes sense the arm only on arm processor build would work because the virtualization of buildx... Yeah... so kinda stuck until someone can help...

@sayboras
Copy link
Member

I personally rely on cross compilation, however, sometime local testing can be done with the below command directly.

If it helps, might need to bundle binary to docker image.

$ bazel build cilium-envoy --cpu aarch64

@dioguerra
Copy link
Author

I personally rely on cross compilation, however, sometime local testing can be done with the below command directly.

If it helps, might need to bundle binary to docker image.

$ bazel build cilium-envoy --cpu aarch64

Finally build after couple days, will give it a try in the next days

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@dioguerra @sayboras