┌──[root@vms81.liruilongs.github.io]-[~/ansible] └─$ansible etcd -m shell -a "docker images" 192.168.26.102 | CHANGED |rc=0>> REPOSITORY TAG IMAGE ID CREATED SIZE quay.io/calico/node v2.6.12 401cc3e56a1a 3 years ago 281MB 192.168.26.100 | CHANGED |rc=0>> REPOSITORY TAG IMAGE ID CREATED SIZE quay.io/calico/node v2.6.12 401cc3e56a1a 3 years ago 281MB 192.168.26.101 | CHANGED |rc=0>> REPOSITORY TAG IMAGE ID CREATED SIZE quay.io/calico/node v2.6.12 401cc3e56a1a 3 years ago 281MB ┌──[root@vms81.liruilongs.github.io]-[~/ansible] └─$
calicoctl 工具导入
┌──[root@vms81.liruilongs.github.io]-[~/ansible] └─$ansible etcd -m copy -a "src=/root/calicoctl dest=/bin/ mode=+x"
开始建立 calico node 信息:每个主机上都部署了Calico/Node作为虚拟路由器
┌──[root@vms81.liruilongs.github.io]-[~/ansible] └─$ansible etcd -m shell -a "calicoctl node run --node-image=quay.io/calico/node:v2.6.12 -c /etc/calico/calicoctl.cfg"
查看node状态,通过Calico将宿主机组织成任意的拓扑集群
┌──[root@vms81.liruilongs.github.io]-[~/ansible] └─$ansible etcd -m shell -a "calicoctl node status" 192.168.26.102 | CHANGED |rc=0>> Calico process is running.
IPv4 BGP status +----------------+-------------------+-------+----------+-------------+ | PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO | +----------------+-------------------+-------+----------+-------------+ |192.168.26.100 | node-to-node mesh | up |14:46:35 | Established | |192.168.26.101 | node-to-node mesh | up |14:46:34 | Established | +----------------+-------------------+-------+----------+-------------+
IPv6 BGP status No IPv6 peers found. 192.168.26.101 | CHANGED |rc=0>> Calico process is running.
IPv4 BGP status +----------------+-------------------+-------+----------+-------------+ | PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO | +----------------+-------------------+-------+----------+-------------+ |192.168.26.100 | node-to-node mesh | up |14:46:31 | Established | |192.168.26.102 | node-to-node mesh | up |14:46:34 | Established | +----------------+-------------------+-------+----------+-------------+
IPv6 BGP status No IPv6 peers found. 192.168.26.100 | CHANGED |rc=0>> Calico process is running.
IPv4 BGP status +----------------+-------------------+-------+----------+-------------+ | PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO | +----------------+-------------------+-------+----------+-------------+ |192.168.26.101 | node-to-node mesh | up |14:46:31 | Established | |192.168.26.102 | node-to-node mesh | up |14:46:35 | Established | +----------------+-------------------+-------+----------+-------------+
IPv6 BGP status No IPv6 peers found. ┌──[root@vms81.liruilongs.github.io]-[~/ansible] └─$
┌──[root@vms81.liruilongs.github.io]-[~/ansible] └─$ansible etcd -m shell -a "docker network list" 192.168.26.100 | CHANGED |rc=0>> NETWORK ID NAME DRIVER SCOPE caa87ba3dd86 bridge bridge local 58121f89bcdd calnet1 calico global 1d63e3ad385f hosthostlocal adc94f172d5f none null local 192.168.26.102 | CHANGED |rc=0>> NETWORK ID NAME DRIVER SCOPE cc37d3c66e2f bridge bridge local 58121f89bcdd calnet1 calico global 3b138015d4ab hosthostlocal 7481614a7084 none null local 192.168.26.101 | CHANGED |rc=0>> NETWORK ID NAME DRIVER SCOPE d0cb224ed111 bridge bridge local 58121f89bcdd calnet1 calico global 106e1c9fb3d3 hosthostlocal f983021e2a02 none null local ┌──[root@vms81.liruilongs.github.io]-[~/ansible] └─$
查看节点中的网卡信息,这个时候没有容器运行,所以没有caliao网卡
┌──[root@vms81.liruilongs.github.io]-[~/ansible] └─$ansible etcd -m shell -a "ip a" 192.168.26.102 | CHANGED |rc=0>> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:0f:98:f1 brd ff:ff:ff:ff:ff:ff inet 192.168.26.102/24 brd 192.168.26.255 scope global ens32 valid_lft forever preferred_lft forever inet6 fe80::20c:29ff:fe0f:98f1/64 scope link valid_lft forever preferred_lft forever 3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN link/ether 02:42:c3:28:19:78 brd ff:ff:ff:ff:ff:ff inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 valid_lft forever preferred_lft forever 192.168.26.100 | CHANGED |rc=0>> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:8c:e8:1a brd ff:ff:ff:ff:ff:ff inet 192.168.26.100/24 brd 192.168.26.255 scope global ens32 valid_lft forever preferred_lft forever inet6 fe80::20c:29ff:fe8c:e81a/64 scope link valid_lft forever preferred_lft forever 3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN link/ether 02:42:f7:1a:2e:30 brd ff:ff:ff:ff:ff:ff inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 valid_lft forever preferred_lft forever 192.168.26.101 | CHANGED |rc=0>> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:3b:6e:ef brd ff:ff:ff:ff:ff:ff inet 192.168.26.101/24 brd 192.168.26.255 scope global ens32 valid_lft forever preferred_lft forever inet6 fe80::20c:29ff:fe3b:6eef/64 scope link valid_lft forever preferred_lft forever 3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN link/ether 02:42:70:a7:4e:7e brd ff:ff:ff:ff:ff:ff inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 valid_lft forever preferred_lft forever ┌──[root@vms81.liruilongs.github.io]-[~/ansible] └─$
┌──[root@vms81.liruilongs.github.io]-[~/ansible] └─$ansible etcd -m shell -a "docker ps" 192.168.26.102 | CHANGED |rc=0>> CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 82e4d6dfde5a busybox "sh" About a minute ago Up About a minute 192.168.26.102 c2d2ab904d6d quay.io/calico/node:v2.6.12 "start_runit"2 hours ago Up 2 hours calico-node 192.168.26.100 | CHANGED |rc=0>> CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 065724c073de busybox "sh" About a minute ago Up About a minute 192.168.26.100 f0b150a924d9 quay.io/calico/node:v2.6.12 "start_runit"2 hours ago Up 2 hours calico-node 192.168.26.101 | CHANGED |rc=0>> CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES cf2ff4b65e63 busybox "sh" About a minute ago Up About a minute 192.168.26.101 0e4e6f005797 quay.io/calico/node:v2.6.12 "start_runit"2 hours ago Up 2 hours calico-node ┌──[root@vms81.liruilongs.github.io]-[~/ansible] └─$
查看每个容器的内部网卡和IP
┌──[root@vms81.liruilongs.github.io]-[~/ansible] └─$ansible etcd -m shell -a "docker exec -it {{inventory_hostname}} ip a | grep cali0 -A 4" 192.168.26.100 | CHANGED |rc=0>> 4: cali0@if5: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff inet 192.168.239.128/32 scope global cali0 valid_lft forever preferred_lft forever 192.168.26.102 | CHANGED |rc=0>> 4: cali0@if5: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff inet 192.168.63.64/32 scope global cali0 valid_lft forever preferred_lft forever 192.168.26.101 | CHANGED |rc=0>> 4: cali0@if5: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff inet 192.168.198.0/32 scope global cali0 valid_lft forever preferred_lft forever ┌──[root@vms81.liruilongs.github.io]-[~/ansible] └─$
查看容器内的路由关系,即所有的出口都是通过cali0网卡来实现的
┌──[root@vms81.liruilongs.github.io]-[~/ansible] └─$ansible etcd -m shell -a "docker exec -it {{inventory_hostname}} ip route | grep cali0 " 192.168.26.101 | CHANGED |rc=0>> default via 169.254.1.1 dev cali0 169.254.1.1 dev cali0 scope link 192.168.26.102 | CHANGED |rc=0>> default via 169.254.1.1 dev cali0 169.254.1.1 dev cali0 scope link 192.168.26.100 | CHANGED |rc=0>> default via 169.254.1.1 dev cali0 169.254.1.1 dev cali0 scope link
┌──[root@vms81.liruilongs.github.io]-[~/ansible] └─$ansible etcd -m shell -a "ip a | grep -A 4 cali" 192.168.26.102 | CHANGED |rc=0>> 5: cali6f956c2ada9@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP link/ether 6a:65:54:1a:19:e6 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet6 fe80::6865:54ff:fe1a:19e6/64 scope link valid_lft forever preferred_lft forever 192.168.26.100 | CHANGED |rc=0>> 5: cali0b7f49da20a@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP link/ether 9e:da:0e:cc:b3:7e brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet6 fe80::9cda:eff:fecc:b37e/64 scope link valid_lft forever preferred_lft forever 192.168.26.101 | CHANGED |rc=0>> 5: calib6f7ddae7e3@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP link/ether 1e:e6:16:ae:f0:91 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet6 fe80::1ce6:16ff:feae:f091/64 scope link valid_lft forever preferred_lft forever ┌──[root@vms81.liruilongs.github.io]-[~/ansible] └─$
查看宿主机路由关系
┌──[root@vms81.liruilongs.github.io]-[~/ansible] └─$ansible etcd -m shell -a "ip route " 192.168.26.101 | CHANGED |rc=0>> default via 192.168.26.2 dev ens32 169.254.0.0/16 dev ens32 scope link metric 1002 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 192.168.26.0/24 dev ens32 proto kernel scope link src 192.168.26.101 192.168.63.64/26 via 192.168.26.102 dev ens32 proto bird blackhole 192.168.198.0/26 proto bird 192.168.198.1 dev cali2f9e2c68bad scope link 192.168.239.128/26 via 192.168.26.100 dev ens32 proto bird 192.168.26.100 | CHANGED |rc=0>> default via 192.168.26.2 dev ens32 169.254.0.0/16 dev ens32 scope link metric 1002 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 192.168.26.0/24 dev ens32 proto kernel scope link src 192.168.26.100 192.168.63.64/26 via 192.168.26.102 dev ens32 proto bird 192.168.198.0/26 via 192.168.26.101 dev ens32 proto bird 192.168.239.128 dev cali0b7f49da20a scope link blackhole 192.168.239.128/26 proto bird 192.168.26.102 | CHANGED |rc=0>> default via 192.168.26.2 dev ens32 169.254.0.0/16 dev ens32 scope link metric 1002 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 192.168.26.0/24 dev ens32 proto kernel scope link src 192.168.26.102 192.168.63.64 dev cali6f956c2ada9 scope link blackhole 192.168.63.64/26 proto bird 192.168.198.0/26 via 192.168.26.101 dev ens32 proto bird 192.168.239.128/26 via 192.168.26.100 dev ens32 proto bird ┌──[root@vms81.liruilongs.github.io]-[~/ansible] └─$
┌──[root@vms100.liruilongs.github.io]-[~] └─$ docker exec -it 192.168.26.100 /bin/sh / # ls bin dev etc home proc root sys tmp usr var / # ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 4: cali0@if5: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff inet 192.168.239.128/32 scope global cali0 valid_lft forever preferred_lft forever / # ping 192.168.63.64 PING 192.168.63.64 (192.168.63.64): 56 data bytes 64 bytes from 192.168.63.64: seq=0ttl=62time=18.519 ms 64 bytes from 192.168.63.64: seq=1ttl=62time=0.950 ms 64 bytes from 192.168.63.64: seq=2ttl=62time=1.086 ms 64 bytes from 192.168.63.64: seq=3ttl=62time=0.846 ms 64 bytes from 192.168.63.64: seq=4ttl=62time=0.840 ms 64 bytes from 192.168.63.64: seq=5ttl=62time=1.151 ms 64 bytes from 192.168.63.64: seq=6ttl=62time=0.888 ms ^C --- 192.168.63.64 ping statistics --- 7 packets transmitted, 7 packets received, 0% packet loss round-trip min/avg/max =0.840/3.468/18.519 ms / #
在K8s集群的中,有一个容器,就会生成一个calico网卡
┌──[root@vms81.liruilongs.github.io]-[~/ansible] └─$ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:ad:e3:93 brd ff:ff:ff:ff:ff:ff inet 192.168.26.81/24 brd 192.168.26.255 scope global ens32 valid_lft forever preferred_lft forever inet6 fe80::20c:29ff:fead:e393/64 scope link valid_lft forever preferred_lft forever 3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN link/ether 02:42:0a:9e:7d:44 brd ff:ff:ff:ff:ff:ff inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 valid_lft forever preferred_lft forever 4: tunl0@NONE: <NOARP,UP,LOWER_UP> mtu 1480 qdisc noqueue state UNKNOWN qlen 1 link/ipip 0.0.0.0 brd 0.0.0.0 inet 10.244.88.64/32 scope global tunl0 valid_lft forever preferred_lft forever 5: cali12cf25006b5@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet6 fe80::ecee:eeff:feee:eeee/64 scope link valid_lft forever preferred_lft forever 6: cali5a282a7bbb0@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 1 inet6 fe80::ecee:eeff:feee:eeee/64 scope link valid_lft forever preferred_lft forever 7: calicb34164ec79@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 2 inet6 fe80::ecee:eeff:feee:eeee/64 scope link valid_lft forever preferred_lft forever ┌──[root@vms81.liruilongs.github.io]-[~/ansible] └─$