Skip to content

Latest commit

 

History

History
182 lines (137 loc) · 8.87 KB

multi-interfaces-annotation.md

File metadata and controls

182 lines (137 loc) · 8.87 KB
Raw

Pod annotation of multi-NIC

When assigning multiple NICs to a Pod with Multus CNI, Spiderpool supports to specify the IP pools for each interface.

This feature supports to implement by annotation ipam.spidernet.io/subnets and ipam.spidernet.io/ippools

Get Started

The example will create two Multus CNI Configuration object and create two underlay subnets. Then run a Pod with two NICs with IP in different subnets.

Set up Spiderpool

Follow the guide installation to install Spiderpool.

Set up Multus Configuration

In this example, Macvlan will be used as the main CNI, Create two network-attachment-definitions，The following parameters need to be confirmed:

  • Confirm the host machine parent interface required for Macvlan. This example takes the ens192 and ens224 network cards of the host machine as examples to create a Macvlan sub interface for Pod to use.

  • In order to use the Veth plugin for clusterIP communication, you need to confirm the serviceIP CIDR of the cluster service, e.g. by using the command kubectl -n kube-system get configmap kubeadm-config -oyaml | grep service.

~# kubectl apply -f https://raw.githubusercontent.com/spidernet-io/spiderpool/main/docs/example/multi-interfaces-annotation/multus-conf.yaml

~# kubectl get network-attachment-definitions.k8s.cni.cncf.io -n kube-system
NAME                  AGE
macvlan-conf-ens192   20s
macvlan-conf-ens224   22s

multiple NICs by subnet

Create two Subnets to provide IP addresses for different interfaces.

~# kubectl apply -f https://raw.githubusercontent.com/spidernet-io/spiderpool/main/docs/example/multi-interfaces-annotation/different-segment-ipv4-subnets.yaml

~# kubectl get spidersubnet
NAME                 VERSION   SUBNET        ALLOCATED-IP-COUNT   TOTAL-IP-COUNT
subnet-test-ens192   4         10.6.0.1/16   0                    10
subnet-test-ens224   4         10.7.0.1/16   0                    10

In the following example Yaml, 2 copies of the Deployment are created：

~# kubectl apply -f https://raw.githubusercontent.com/spidernet-io/spiderpool/main/docs/example/multi-interfaces-annotation/subnet-test-deploy.yaml

Eventually, when the Deployment is created, Spiderpool will select random IPs from the specified subnet to create two fixed IP pools to bind to each of the Deployment Pod's two NICs.

~# kubectl get spiderippool
NAME                                 VERSION   SUBNET        ALLOCATED-IP-COUNT   TOTAL-IP-COUNT   DEFAULT   DISABLE
auto-test-app-v4-eth0-b1a361c7e9df   4         10.6.0.1/16   2                    3                false     false
auto-test-app-v4-net1-b1a361c7e9df   4         10.7.0.1/16   2                    3                false     false

~# kubectl get spiderippool auto-test-app-v4-eth0-b1a361c7e9df -o jsonpath='{.spec.ips}'
["10.6.168.171-10.6.168.173"]

~# kubectl get spiderippool auto-test-app-v4-net1-b1a361c7e9df -o jsonpath='{.spec.ips}'
["10.7.168.171-10.7.168.173"]

~# kubectl get po -l app=test-app -o wide
NAME                        READY   STATUS    RESTARTS   AGE   IP             NODE    NOMINATED NODE   READINESS GATES
test-app-6f4594ff67-fkqbw   1/1     Running   0          40s   10.6.168.172   node2   <none>           <none>
test-app-6f4594ff67-gwlx8   1/1     Running   0          40s   10.6.168.173   node1   <none>           <none>

~# kubectl exec -ti test-app-6f4594ff67-fkqbw -- ip a
3: eth0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether ae:fa:5e:d9:79:11 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.6.168.172/16 brd 10.6.255.255 scope global eth0
       valid_lft forever preferred_lft forever
4: veth0@if13: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 26:6f:22:91:22:f9 brd ff:ff:ff:ff:ff:ff link-netnsid 0
5: net1@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether d6:4b:c2:6a:62:0f brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.7.168.173/16 brd 10.7.255.255 scope global net1
       valid_lft forever preferred_lft forever

The following command shows the multi-NIC routing information in the Pod. The Veth plug-in can automatically coordinate the policy routing between multiple NICs and solve the communication problems between multiple NICs.

~# kubectl exec -ti test-app-6f4594ff67-fkqbw -- ip rule show
0:  from all lookup local
32764:  from 10.7.168.173 lookup 100
32765:  from all to 10.7.168.173/16 lookup 100
32766:  from all lookup main
32767:  from all lookup default

~# kubectl exec -ti test-app-6f4594ff67-fkqbw -- ip r show main
default via 10.6.0.1 dev eth0

~# kubectl exec -ti test-app-6f4594ff67-fkqbw -- ip route show table 100
default via 10.7.0.1 dev net1
10.6.168.123 dev veth0 scope link
10.7.0.0/16 dev net1 proto kernel scope link src 10.7.168.173
10.96.0.0/12 via 10.6.168.123 dev veth0

multiple NICs by IPPool

Create two IPPools to provide IP addresses for different interfaces.

~# kubectl apply -f https://raw.githubusercontent.com/spidernet-io/spiderpool/main/docs/example/multi-interfaces-annotation/different-segment-ipv4-ippools.yaml

~# kubectl get spiderippool
NAME                   VERSION   SUBNET        ALLOCATED-IP-COUNT   TOTAL-IP-COUNT   DEFAULT   DISABLE
ippool-test-ens192     4         10.6.0.1/16   0                    5                false     false
ippool-test-ens224     4         10.7.0.1/16   0                    5                false     false

In the following example Yaml, 1 copies of the Deployment are created：

~# kubectl apply -f https://raw.githubusercontent.com/spidernet-io/spiderpool/main/docs/example/multi-interfaces-annotation/ippool-test-deploy.yaml

Eventually, when the Deployment is created, Spiderpool randomly selects IPs from the two specified IPPools to form bindings to each of the two NICs.

~# kubectl get spiderippool
NAME                   VERSION   SUBNET        ALLOCATED-IP-COUNT   TOTAL-IP-COUNT   DEFAULT   DISABLE
ippool-test-ens192     4         10.6.0.1/16   1                    5                false     false
ippool-test-ens224     4         10.7.0.1/16   1                    5                false     false

~# kubectl get po -l app=ippool-test-app -o wide
NAME                               READY   STATUS    RESTARTS   AGE     IP             NODE    NOMINATED NODE   READINESS GATES
ippool-test-app-65f646574c-mpr47   1/1     Running   0          6m18s   10.6.168.175   node2   <none>           <none>

~#  kubectl exec -ti ippool-test-app-65f646574c-mpr47 -- ip a
...
3: eth0@tunl0: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue
    link/ether 2a:ca:ce:06:1e:91 brd ff:ff:ff:ff:ff:ff
    inet 10.6.168.175/16 brd 10.6.255.255 scope global eth0
       valid_lft forever preferred_lft forever
4: veth0@if15: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue
    link/ether 86:ba:6f:97:ae:1b brd ff:ff:ff:ff:ff:ff
5: net1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
    link/ether f2:12:b5:8c:ff:4f brd ff:ff:ff:ff:ff:ff
    inet 10.7.168.177/16 brd 10.7.255.255 scope global net1
       valid_lft forever preferred_lft forever

The following command shows the multi-NIC routing information in the Pod. The Veth plug-in can automatically coordinate the policy routing between multiple NICs and solve the communication problems between multiple NICs.

~# kubectl exec -ti ippool-test-app-65f646574c-mpr47 -- ip rule show
0: from all lookup local
32764: from 10.7.168.177 lookup 100
32765: from all to 10.7.168.177/16 lookup 100
32766: from all lookup main
32767: from all lookup default

~# kubectl exec -ti ippool-test-app-65f646574c-mpr47 -- ip r show main
default via 10.6.0.1 dev eth0

~# kubectl exec -ti ippool-test-app-65f646574c-mpr47 -- ip r show table 100
default via 10.7.0.1 dev net1
10.6.168.123 dev veth0 scope link
10.7.0.0/16 dev net1 scope link  src 10.7.168.177
10.96.0.0/12 via 10.6.168.123 dev veth0

Clean up

Clean the relevant resources so that you can run this tutorial again.

kubectl delete \
-f https://raw.githubusercontent.com/spidernet-io/spiderpool/main/docs/example/multi-interfaces-annotation/multus-conf.yaml \
-f https://raw.githubusercontent.com/spidernet-io/spiderpool/main/docs/example/multi-interfaces-annotation/different-segment-ipv4-subnets.yaml \
-f https://raw.githubusercontent.com/spidernet-io/spiderpool/main/docs/example/multi-interfaces-annotation/subnet-test-deploy.yaml \
-f https://raw.githubusercontent.com/spidernet-io/spiderpool/main/docs/example/multi-interfaces-annotation/different-segment-ipv4-ippools.yaml \
-f https://raw.githubusercontent.com/spidernet-io/spiderpool/main/docs/example/basic/custom-dual-ippool-deploy.yaml \
-f https://raw.githubusercontent.com/spidernet-io/spiderpool/main/docs/example/multi-interfaces-annotation/ippool-test-deploy.yaml \
--ignore-not-found=true