Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: multus fails to reach api server when the old service account is out of data && update multus to v4 #4393

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

fix: multus fails to reach api server when the old service account is out of data && update multus to v4 #4393

wants to merge 1 commit into from

Conversation

cyclinder
Copy link
Collaborator

@cyclinder cyclinder commented Dec 13, 2024 •
edited
Loading

Thanks for contributing!

Notice:

What issue(s) does this PR fix:

Fixes #4436

Special notes for your reviewer:

kubectl logs -f -n kube-system kube-multus-ds-dnx5z
Defaulted container "kube-multus" out of: kube-multus, install-multus-binary (init)
INFO: 2024-12-16T09:07:40+00:00 ENV MULTUS_CLUSTER_NETWORK is empty, Detecting default cni in the /host/etc/cni/net.d
INFO: 2024-12-16T09:07:40+00:00 Found the default-cni file: 10-calico.conflist
INFO: 2024-12-16T09:07:40+00:00 cat /host/etc/cni/net.d/10-calico.conflist:
{
  "name": "k8s-pod-network",
  "cniVersion": "0.3.1",
  "plugins": [
    {
      "type": "calico",
      "log_level": "info",
      "log_file_path": "/var/log/calico/cni/cni.log",
      "datastore_type": "kubernetes",
      "nodename": "spider-control-plane",
      "mtu": 0,
      "ipam": {
        "type": "calico-ipam",
        "assign_ipv4": "true",
        "assign_ipv6": "true"
      },
      "policy": {
        "type": "k8s"
      },
      "kubernetes": {
        "kubeconfig": "/etc/cni/net.d/calico-kubeconfig"
      }
    },
    {
      "type": "portmap",
      "snat": true,
      "capabilities": {
        "portMappings": true
      }
    },
    {
      "type": "bandwidth",
      "capabilities": {
        "bandwidth": true
      }
    }
  ]
}
INFO: 2024-12-16T09:07:40+00:00 Updating the clusterNetwork of the multus-cni config to k8s-pod-network

INFO: 2024-12-16T09:07:41+00:00 multus kubeconfig is generated.
INFO: 2024-12-16T09:07:41+00:00 cat /tmp/00-multus.conf
{
  "cniVersion": "0.3.1",
  "name": "multus-cni-network",
  "type": "multus",
  "confDir": "/etc/cni/net.d/" ,
  "logLevel": "debug",
  "logFile": "/var/log/multus.log",
  "capabilities": {
    "portMappings": true,
    "bandwidth": true
  },
  "namespaceIsolation": false,
  "clusterNetwork": "k8s-pod-network",
  "defaultNetworks": [],
  "multusNamespace": "kube-system",
  "systemNamespaces": [],
  "kubeconfig": "/etc/cni/net.d/multus.d/multus.kubeconfig"
}
INFO: 2024-12-16T09:07:41+00:00 multus config file /tmp/00-multus.conf is copied to /host/etc/cni/net.d.
INFO: 2024-12-16T09:07:41+00:00 Entering watch loop...
  1. upgrade multus version to 4.1.4
  2. Provide a script (stored in configMap) to be mounted on the multus container that accomplishes two things:
  • Read the clusterNetwork and multusNamespace environment variables from the ENV of the multus container, automatically generate 00-multus.conf, and copy it to /host/etc/cni/net.d.
  • A for loop listens for changes in the serviceaccount and updates the multus kubeconfig automatically.
  1. multus init container is only responsible for copying the binary to the host, using the official image.
  2. multus container is responsible for multus configuration file generation and certificate update, using the spiderpoolAgent image, why? The official image uses the base-debian11 minimal image, which doesn't include sh, sed, cat, etc., as needed.

注意：

  1. 升级 multus 版本 到 4.1.4
  2. 提供一个脚本（存放与 configMap 中）挂载到 multus 容器中，完成两件事情：
    • 从 multus 容器的 ENV 中读取 clusterNetwork 和 multusNamespace环境变量，自动生成 00-multus.conf，并 copy 到 /host/etc/cni/net.d 下
    • 通过一个 for 循环不断监听 serviceaccount 是否变化，自动更新 multus kubeconfig.
  3. multus init container 只负责 copy 二进制到主机， 使用官方镜像
  4. multus container 负责 multus 配置文件生成和证书更新，这里使用 spiderpoolAgent 镜像 ，why? 官方镜像使用 base-debian11 最小镜像，该镜像不包括需要用到的 sh，sed ，cat 等命令，无法执行 entrypoint.sh 的操作
  5. 之前 multus 的配置文件是通过一个 configMap 声明，现在移除这个 configmap。现在新增 clusterNetwork 字段到 spiderpool-conf 中，配置默认的 CNI。避免 helm upgrade 时覆盖配置

@cyclinder cyclinder added the release/feature-new release note for new feature label Dec 13, 2024
@cyclinder cyclinder marked this pull request as draft December 13, 2024 11:02
@cyclinder cyclinder force-pushed the multus4.0 branch 3 times, most recently from 396ecad to 3947a43 Compare December 16, 2024 09:37
@cyclinder cyclinder marked this pull request as ready for review December 16, 2024 09:37
@cyclinder cyclinder force-pushed the multus4.0 branch 2 times, most recently from dca6d92 to 6b2b814 Compare December 16, 2024 10:19
@codecov Codecov
Copy link

codecov bot commented Dec 16, 2024 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 79.21%. Comparing base (7fd26d8) to head (7c6e42f).

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #4393      +/-   ##
==========================================
- Coverage   79.26%   79.21%   -0.05%     
==========================================
  Files          54       54              
  Lines        6283     6283              
==========================================
- Hits         4980     4977       -3     
- Misses       1108     1110       +2     
- Partials      195      196       +1
Flag Coverage Δ
unittests 79.21% <ø> (-0.05%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

see 1 file with indirect coverage changes

@cyclinder cyclinder force-pushed the multus4.0 branch 3 times, most recently from 3a62b5a to 4ce209b Compare December 19, 2024 02:50
@cyclinder cyclinder mentioned this pull request Dec 19, 2024
Closed
5 tasks
@cyclinder cyclinder added cherrypick-release-v0.9 cherrypick-release-v1.0 Cherry-pick the PR to branch release-v1.0. cherrypick-release-v0.8 Cherry-pick the PR to branch release-v0.8. labels Dec 19, 2024
@cyclinder
Copy link
Collaborator Author

wait #4420

@cyclinder
Copy link
Collaborator Author 

[Step 9] Install operator: kruise 
/etc/bash.bashrc: line 7: PS1: unbound variable
make[3]: Entering directory '/home/runner/work/spiderpool/spiderpool/test'
/etc/bash.bashrc: line 7: PS1: unbound variable
/etc/bash.bashrc: line 7: PS1: unbound variable
/etc/bash.bashrc: line 7: PS1: unbound variable
/etc/bash.bashrc: line 7: PS1: unbound variable
/etc/bash.bashrc: line 7: PS1: unbound variable
/etc/bash.bashrc: line 7: PS1: unbound variable
/etc/bash.bashrc: line 7: PS1: unbound variable
Makefile:410: warning: overriding recipe for target '@echo'
Makefile:206: warning: ignoring old recipe for target '@echo'
Makefile:410: warning: overriding recipe for target '"check'
Makefile:206: warning: ignoring old recipe for target '"check'
Makefile:410: warning: overriding recipe for target 'cni'
Makefile:206: warning: ignoring old recipe for target 'cni'
Makefile:410: warning: overriding recipe for target 'nodes'
Makefile:206: warning: ignoring old recipe for target 'nodes'
/etc/bash.bashrc: line 7: PS1: unbound variable
add openkruise charts repository...
"openkruise" already exists with the same configuration, skipping
v1.7.2: Pulling from openkruise/kruise-manager
Digest: sha256:15da38958bf9fee89d2730b937c5c3e291390f08dcc05467f737e5fcf6fecfb5
Status: Image is up to date for openkruise/kruise-manager:v1.7.2
docker.io/openkruise/kruise-manager:v1.7.2
Image: "docker.io/openkruise/kruise-manager:v1.7.2" with ID "sha256:414a6b7c8804fa65d20ae7cbcc85572c4181ea85c4240f7ad3bc8039e2929589" not yet present on node "spiderpool1224132049-worker", loading...
Image: "docker.io/openkruise/kruise-manager:v1.7.2" with ID "sha256:414a6b7c8804fa65d20ae7cbcc85572c4181ea85c4240f7ad3bc8039e2929589" not yet present on node "spiderpool1224132049-control-plane", loading...
v0.1.0: Pulling from openkruise/kruise-helm-hook
Digest: sha256:edc7cf9428fd72f9885431a4f0fe4e2e1724f6a8fbd4b592105fbdfdb2a9afdf
Status: Image is up to date for openkruise/kruise-helm-hook:v0.1.0
docker.io/openkruise/kruise-helm-hook:v0.1.0
Image: "openkruise/kruise-helm-hook:v0.1.0" with ID "sha256:ac7049d52f06d5f37229f6abea772d01a0badaee6cb3e71e118b73f98a04280e" not yet present on node "spiderpool1224132049-worker", loading...
Image: "openkruise/kruise-helm-hook:v0.1.0" with ID "sha256:ac7049d52f06d5f37229f6abea772d01a0badaee6cb3e71e118b73f98a04280e" not yet present on node "spiderpool1224132049-control-plane", loading...
# https://github.com/spidernet-io/spiderpool/issues/4396
/etc/bash.bashrc: line 7: PS1: unbound variable
make[3]: *** [Makefile:221: setup_kruise] Error 1
make[3]: Leaving directory '/home/runner/work/spiderpool/spiderpool/test'
make[2]: *** [Makefile:39: kind-init] Error 2
make[2]: Leaving directory '/home/runner/work/spiderpool/spiderpool/test'
make[1]: *** [Makefile:299: e2e_init] Error 2
make[1]: Leaving directory '/home/runner/work/spiderpool/spiderpool'
make: *** [Makefile:323: e2e_init_spiderpool] Error 2

@weizhoublue weizhoublue changed the title bump multus to v4 update multus config when service account is updated && update multus to v4 Dec 25, 2024
@weizhoublue
Copy link
Collaborator

good job , and we should add or attach an issue explaining the multus certificate issue for future reviewing

@cyclinder cyclinder force-pushed the multus4.0 branch 7 times, most recently from 921727c to 83e116a Compare December 25, 2024 13:23
@cyclinder cyclinder force-pushed the multus4.0 branch 8 times, most recently from 396b803 to 53350eb Compare December 26, 2024 09:38
@weizhoublue weizhoublue changed the title update multus config when service account is updated && update multus to v4 fix: multus fails to reach api server when the old service account is out of data && update multus to v4 Dec 27, 2024
@weizhoublue
Copy link
Collaborator

too much info log

{"level":"INFO","ts":"2024-12-26T09:40:38.158Z","logger":"spiderpool-controller","caller":"cmd/crd_manager.go:96","msg":"Webhook health check successful"}
{"level":"INFO","ts":"2024-12-26T09:40:40.770Z","logger":"spiderpool-controller","caller":"cmd/crd_manager.go:96","msg":"Webhook health check successful"}
{"level":"INFO","ts":"2024-12-26T09:40:48.158Z","logger":"spiderpool-controller","caller":"cmd/crd_manager.go:96","msg":"Webhook health check successful"}
{"level":"INFO","ts":"2024-12-26T09:40:50.770Z","logger":"spiderpool-controller","caller":"cmd/crd_manager.go:96","msg":"Webhook health check successful"}
{"level":"INFO","ts":"2024-12-26T09:40:58.158Z","logger":"spiderpool-controller","caller":"cmd/crd_manager.go:96","msg":"Webhook health check successful"}
{"level":"INFO","ts":"2024-12-26T09:41:00.770Z","logger":"spiderpool-controller","caller":"cmd/crd_manager.go:96","msg":"Webhook health check successful"}
{"level":"INFO","ts":"2024-12-26T09:41:08.158Z","logger":"spiderpool-controller","caller":"cmd/crd_manager.go:96","msg":"Webhook health check successful"}
{"level":"INFO","ts":"2024-12-26T09:41:10.769Z","logger":"spiderpool-controller","caller":"cmd/crd_manager.go:96","msg":"Webhook health check successful"}
{"level":"INFO","ts":"2024-12-26T09:41:18.158Z","logger":"spiderpool-controller","caller":"cmd/crd_manager.go:96","msg":"Webhook health check successful"}
{"level":"INFO","ts":"2024-12-26T09:41:20.770Z","logger":"spiderpool-controller","caller":"cmd/crd_manager.go:96","msg":"Webhook health check successful"}
{"level":"INFO","ts":"2024-12-26T09:41:28.158Z","logger":"spiderpool-controller","caller":"cmd/crd_manager.go:96","msg":"Webhook health check successful"}
{"level":"INFO","ts":"2024-12-26T09:41:30.770Z","logger":"spiderpool-controller","caller":"cmd/crd_manager.go:96","msg":"Webhook health check successful"}
{"level":"INFO","ts":"2024-12-26T09:41:38.158Z","logger":"spiderpool-controller","caller":"cmd/crd_manager.go:96","msg":"Webhook health check successful"}
{"level":"INFO","ts":"2024-12-26T09:41:40.770Z","logger":"spiderpool-controller","caller":"cmd/crd_manager.go:96","msg":"Webhook health check successful"}
{"level":"INFO","ts":"2024-12-26T09:41:48.158Z","logger":"spiderpool-controller","caller":"cmd/crd_manager.go:96","msg":"Webhook health check successful"}
{"level":"INFO","ts":"2024-12-26T09:41:50.770Z","logger":"spiderpool-controller","caller":"cmd/crd_manager.go:96","msg":"Webhook health check successful"}
{"level":"INFO","ts":"2024-12-26T09:41:58.158Z","logger":"spiderpool-controller","caller":"cmd/crd_manager.go:96","msg":"Webhook health check successful"}
{"level":"INFO","ts":"2024-12-26T09:42:00.770Z","logger":"spiderpool-controller","caller":"cmd/crd_manager.go:96","msg":"Webhook health check successful"}
{"level":"INFO","ts":"2024-12-26T09:42:08.158Z","logger":"spiderpool-controller","caller":"cmd/crd_manager.go:96","msg":"Webhook health check successful"}
{"level":"INFO","ts":"2024-12-26T09:42:10.770Z","logger":"spiderpool-controller","caller":"cmd/crd_manager.go:96","msg":"Webhook health check successful"}
{"level":"INFO","ts":"2024-12-26T09:42:18.157Z","logger":"spiderpool-controller","caller":"cmd/crd_manager.go:96","msg":"Webhook health check successful"}
{"level":"INFO","ts":"2024-12-26T09:42:20.770Z","logger":"spiderpool-controller","caller":"cmd/crd_manager.go:96","msg":"Webhook health check successful"}
{"level":"INFO","ts":"2024-12-26T09:42:28.157Z","logger":"spiderpool-controller","caller":"cmd/crd_manager.go:96","msg":"Webhook health check successful"}
{"level":"INFO","ts":"2024-12-26T09:42:30.769Z","logger":"spiderpool-controller","caller":"cmd/crd_manager.go:96","msg":"Webhook health check successful"}
{"level":"INFO","ts":"2024-12-26T09:42:38.158Z","logger":"spiderpool-controller","caller":"cmd/crd_manager.go:96","msg":"Webhook health check successful"}

@cyclinder
bump multus to v4
7c6e42f 
Signed-off-by: Cyclinder Kuo <[email protected]>
@cyclinder
Copy link
Collaborator Author

https://github.com/spidernet-io/spiderpool/actions/runs/12510892887/job/34902242465?pr=4393 此 PR removed 一些 init-pod 的 ENV， 会导致新 chart 使用 旧镜像 无法正常 running

@cyclinder cyclinder added the pr/ready-review This pull is ready for review label Dec 27, 2024
@cyclinder cyclinder mentioned this pull request Dec 27, 2024
Open
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@weizhoublue weizhoublue Awaiting requested review from weizhoublue weizhoublue is a code owner

At least 2 approving reviews are required to merge this pull request.

Assignees
No one assigned
Labels
cherrypick-release-v0.8 Cherry-pick the PR to branch release-v0.8. cherrypick-release-v0.9 cherrypick-release-v1.0 Cherry-pick the PR to branch release-v1.0. pr/ready-review This pull is ready for review release/feature-new release note for new feature
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Multus kubeconfig would be expired in a year
2 participants
@cyclinder @weizhoublue