你掌握v1.23.6 K8S集群二进制部署了吗？手把手教学，高可用集群部署攻略！

# 工作目录创建mkdir -vp /app/k8s-init-work && cd /app/k8s-init-work# cfssl 最新下载地址: github.com/cloudflare/cfssl/releases# cfssl 相关工具拉取 (如果拉取较慢，建议使用某雷下载，然后上传到服务器里)curl -L github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssl_1.6.1_linux_amd64 -o /usr/local/bin/cfsslcurl -L github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssljson_1.6.1_linux_amd64 -o /usr/local/bin/cfssljsoncurl -L github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssl-certinfo_1.6.1_linux_amd64 -o /usr/local/bin/cfssl-certinfo# 赋予执行权限chmod +x /usr/local/bin/cfssl*/app# cfssl version# Version: 1.2.0# Revision: dev# Runtime: go1.6

温馨提示:

• cfssl : CFSSL 命令行工具

• cfssljson : 用于从cfssl程序中获取JSON输出并将证书、密钥、证书签名请求文件CSR和Bundle写入到文件中，

步骤 02.利用上述 cfssl 工具创建 CA 证书。

# - CA 证书签名请求配置文件fssl print-defaults csr > ca-csr.jsontee ca-csr.json <<'EOF'{ "CN": "kubernetes", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "ChongQing", "ST": "ChongQing", "O": "k8s", "OU": "System" } ], "ca": { "expiry": "87600h" }}EOF# 关键参数解析:CN: Common Name，浏览器使用该字段验证网站是否合法，一般写的是域名，非常重要。浏览器使用该字段验证网站是否合法key：生成证书的算法hosts：表示哪些主机名(域名)或者IP可以使用此csr申请的证书，为空或者""表示所有的都可以使用(本例中没有`"hosts": [""]`字段)names：常见属性设置 * C: Country， 国家 * ST: State，州或者是省份 * L: Locality Name，地区，城市 * O: Organization Name，组织名称，公司名称(在k8s中常用于指定Group，进行RBAC绑定) * OU: Organization Unit Name，组织单位名称，公司部门# - CA 证书策略配置文件cfssl print-defaults config > ca-config.jsontee ca-config.json <<'EOF'{ "signing": { "default": { "expiry": "87600h" }, "profiles": { "kubernetes": { "expiry": "87600h", "usages": [ "signing", "key encipherment", "server auth", "client auth" ] }, "etcd": { "expiry": "87600h", "usages": [ "signing", "key encipherment", "server auth", "client auth" ] } } }}EOF# 关键参数解析:default 默认策略，指定了证书的默认有效期是10年(87600h)profile 自定义策略配置 * kubernetes：表示该配置(profile)的用途是为kubernetes生成证书及相关的校验工作 * signing：表示该证书可用于签名其它证书；生成的 ca.pem 证书中 CA=TRUE * server auth：表示可以该CA 对 server 提供的证书进行验证 * client auth：表示可以用该 CA 对 client 提供的证书进行验证 * expiry：也表示过期时间，如果不写以default中的为准# - 执行cfssl gencert 命令生成CA证书# 利用CA证书签名请求配置文件 ca-csr.json 生成CA证书和CA私钥和CSR(证书签名请求):cfssl gencert -initca ca-csr.json | cfssljson -bare ca # 2022/04/27 16:49:37 [INFO] generating a new CA key and certificate from CSR # 2022/04/27 16:49:37 [INFO] generate received request # 2022/04/27 16:49:37 [INFO] received CSR # 2022/04/27 16:49:37 [INFO] generating key: rsa-2048 # 2022/04/27 16:49:37 [INFO] encoded CSR # 2022/04/27 16:49:37 [INFO] signed certificate with serial number 245643466964695827922023924375276493244980966303$ ls # ca-config.json ca.csr ca-csr.json ca-key.pem ca.pem$ openssl x509 -in ca.pem -text -noout | grep "Not" # Not Before: Apr 27 08:45:00 2022 GMT # Not After : Apr 24 08:45:00 2032 GMT

温馨提示: 如果将 expiry 设置为87600h 表示证书过期时间为十年。

步骤 03.配置ETCD证书相关文件以及生成其证书,

# etcd 证书请求文件tee etcd-csr.json <<'EOF'{ "CN": "etcd", "hosts": [ "127.0.0.1", "10.10.107.223", "10.10.107.224", "10.10.107.225", "etcd1", "etcd2", "etcd3" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "ChongQing", "ST": "ChongQing", "O": "etcd", "OU": "System" } ]}EOF# 利用ca证书签发生成etcd证书cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=etcd etcd-csr.json | cfssljson -bare etcd$ ls etcd*etcd.csr etcd-csr.json etcd-key.pem etcd.pem$ openssl x509 -in etcd.pem -text -noout | grep "X509v3 Subject Alternative Name" -A 1 # X509v3 Subject Alternative Name: # DNS:etcd1, DNS:etcd2, DNS:etcd3, IP Address:127.0.0.1, IP Address:10.10.107.223, IP Address:10.10.107.224, IP Address:10.10.107.225

步骤 04.下载部署ETCD集群, 首先我们需要下载etcd软件包, 可以 Github release 找到最新版本的etcd下载路径(github.com/etcd-io/etcd/releases/)。

# 下载wget -L github.com/etcd-io/etcd/releases/download/v3.5.4/etcd-v3.5.4-linux-amd64.tar.gztar -zxvf etcd-v3.5.4-linux-amd64.tar.gzcp -a etcd* /usr/local/bin/# 版本 etcd --version # etcd Version: 3.5.4 # Git SHA: 08407ff76 # Go Version: go1.16.15 # Go OS/Arch: linux/amd64# 复制到其它master主机上scp -P 20211 ./etcd-v3.5.4-linux-amd64.tar.gz weiyigeek@master-223:~scp -P 20211 ./etcd-v3.5.4-linux-amd64.tar.gz weiyigeek@master-224:~# 分别在master-223与master-224执行, 解压到 /usr/local/ 目录同样复制二进制文件到 /usr/local/bin/tar -zxvf /home/weiyigeek/etcd-v3.5.4-linux-amd64.tar.gz -C /usr/local/cp -a /usr/local/etcd-v3.5.4-linux-amd64/etcd* /usr/local/bin/

温馨提示: etcd 官网地址 (etcd.io/)

步骤 05.创建etcd集群所需的配置文件。

# 证书准备mkdir -vp /etc/etcd/pki/cp *.pem /etc/etcd/pki/ls /etc/etcd/pki/ # ca-key.pem ca.pem etcd-key.pem etcd.pem# 上传到~家目录,并需要将其复制到 /etc/etcd/pki/ 目录中scp -P 20211 *.pem weiyigeek@master-224:~scp -P 20211 *.pem weiyigeek@master-223:~ # ****************** [ 安全登陆 (Security Login) ] ***************** # Authorized only. All activity will be monitored and reported.By Security Center. # ca-key.pem 100% 1675 3.5MB/s 00:00 # ca.pem 100% 1375 5.2MB/s 00:00 # etcd-key.pem 100% 1679 7.0MB/s 00:00 # etcd.pem 100% 1399 5.8MB/s 00:00# master-225 执行tee /etc/etcd/etcd.conf <<'EOF'# [成员配置]# member 名称ETCD_NAME=etcd1# 存储数据的目录(注意需要建立)ETCD_DATA_DIR="/var/lib/etcd/data"# 用于监听客户端etcdctl或者curl连接ETCD_LISTEN_CLIENT_URLS="10.10.107.225:2379,127.0.0.1:2379"# 用于监听集群中其它member的连接ETCD_LISTEN_PEER_URLS="10.10.107.225:2380"# [证书配置]# ETCD_CERT_FILE=/etc/etcd/pki/etcd.pem# ETCD_KEY_FILE=/etc/etcd/pki/etcd-key.pem# ETCD_TRUSTED_CA_FILE=/etc/kubernetes/pki/ca.pem# ETCD_CLIENT_CERT_AUTH=true# ETCD_PEER_CLIENT_CERT_AUTH=true# ETCD_PEER_CERT_FILE=/etc/etcd/pki/etcd.pem# ETCD_PEER_KEY_FILE=/etc/etcd/pki/etcd-key.pem# ETCD_PEER_TRUSTED_CA_FILE=/etc/kubernetes/pki/ca.pem# [集群配置]# 本机地址用于通知客户端，客户端通过此IPs与集群通信;ETCD_ADVERTISE_CLIENT_URLS="10.10.107.225:2379"# 本机地址用于通知集群member与member通信ETCD_INITIAL_ADVERTISE_PEER_URLS="10.10.107.225:2380"# 描述集群中所有节点的信息，本member根据此信息去联系其他memberETCD_INITIAL_CLUSTER="etcd1=10.10.107.225:2380,etcd2=10.10.107.224:2380,etcd3=10.10.107.223:2380"# 集群状态新建集群时候设置为new,若是想加入某个已经存在的集群设置为existingETCD_INITIAL_CLUSTER_STATE=newEOF# master-224 执行tee /etc/etcd/etcd.conf <<'EOF'# [成员配置]# member 名称ETCD_NAME=etcd2# 存储数据的目录(注意需要建立)ETCD_DATA_DIR="/var/lib/etcd/data"# 用于监听客户端etcdctl或者curl连接ETCD_LISTEN_CLIENT_URLS="10.10.107.224:2379,127.0.0.1:2379"# 用于监听集群中其它member的连接ETCD_LISTEN_PEER_URLS="10.10.107.224:2380"# [集群配置]# 本机地址用于通知客户端，客户端通过此IPs与集群通信;ETCD_ADVERTISE_CLIENT_URLS="10.10.107.224:2379"# 本机地址用于通知集群member与member通信ETCD_INITIAL_ADVERTISE_PEER_URLS="10.10.107.224:2380"# 描述集群中所有节点的信息，本member根据此信息去联系其他memberETCD_INITIAL_CLUSTER="etcd1=10.10.107.225:2380,etcd2=10.10.107.224:2380,etcd3=10.10.107.223:2380"# 集群状态新建集群时候设置为new,若是想加入某个已经存在的集群设置为existingETCD_INITIAL_CLUSTER_STATE=newEOF# master-223 执行tee /etc/etcd/etcd.conf <<'EOF'# [成员配置]# member 名称ETCD_NAME=etcd3# 存储数据的目录(注意需要建立)ETCD_DATA_DIR="/var/lib/etcd/data"# 用于监听客户端etcdctl或者curl连接ETCD_LISTEN_CLIENT_URLS="10.10.107.223:2379,127.0.0.1:2379"# 用于监听集群中其它member的连接ETCD_LISTEN_PEER_URLS="10.10.107.223:2380"# [集群配置]# 本机地址用于通知客户端，客户端通过此IPs与集群通信;ETCD_ADVERTISE_CLIENT_URLS="10.10.107.223:2379"# 本机地址用于通知集群member与member通信ETCD_INITIAL_ADVERTISE_PEER_URLS="10.10.107.223:2380"# 描述集群中所有节点的信息，本member根据此信息去联系其他memberETCD_INITIAL_CLUSTER="etcd1=10.10.107.225:2380,etcd2=10.10.107.224:2380,etcd3=10.10.107.223:2380"# 集群状态新建集群时候设置为new,若是想加入某个已经存在的集群设置为existingETCD_INITIAL_CLUSTER_STATE=newEOF

步骤 06.创建配置 etcd 的 systemd 管理配置文件，并启动其服务。

mkdir -vp /var/lib/etcd/cat > /usr/lib/systemd/system/etcd.service <<EOF[Unit]Description=Etcd ServerDocumentation=github.com/etcd-io/etcdAfter=network.targetAfter=network-online.targetwants=network-online.target[Service]Type=notifyWorkingDirectory=/var/lib/etcd/EnvironmentFile=-/etc/etcd/etcd.confExecStart=/usr/local/bin/etcd \ --client-cert-auth \ --trusted-ca-file /etc/etcd/pki/ca.pem \ --cert-file /etc/etcd/pki/etcd.pem \ --key-file /etc/etcd/pki/etcd-key.pem \ --peer-client-cert-auth \ --peer-trusted-ca-file /etc/etcd/pki/ca.pem \ --peer-cert-file /etc/etcd/pki/etcd.pem \ --peer-key-file /etc/etcd/pki/etcd-key.pemRestart=on-failureRestartSec=5LimitNOFILE=65535LimitNPROC=65535[Install]WantedBy=multi-user.targetEOF# 重载 systemd && 开机启动与手动启用etcd服务systemctl daemon-reload && systemctl enable --now etcd.service

步骤 07.查看各个master节点的etcd集群服务是否正常及其健康状态。

# 服务查看systemctl status etcd.service# 利用 etcdctl 工具查看集群成员信息export ETCDCTL_API=3etcdctl --endpoints=10.10.107.225:2379,10.10.107.224:2379,10.10.107.223:2379 \--cacert="/etc/etcd/pki/ca.pem" --cert="/etc/etcd/pki/etcd.pem" --key="/etc/etcd/pki/etcd-key.pem" \--write-out=table member list # +------------------+---------+-------+----------------------------+----------------------------+------------+ # | ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER | # +------------------+---------+-------+----------------------------+----------------------------+------------+ # | 144934d02ad45ec7 | started | etcd3 | 10.10.107.223:2380 | 10.10.107.223:2379 | false | # | 2480d95a2df867a4 | started | etcd2 | 10.10.107.224:2380 | 10.10.107.224:2379 | false | # | 2e8fddd3366a3d88 | started | etcd1 | 10.10.107.225:2380 | 10.10.107.225:2379 | false | # +------------------+---------+-------+----------------------------+----------------------------+------------+# 集群节点信息etcdctl --endpoints=10.10.107.225:2379,10.10.107.224:2379,10.10.107.223:2379 \--cacert="/etc/etcd/pki/ca.pem" --cert="/etc/etcd/pki/etcd.pem" --key="/etc/etcd/pki/etcd-key.pem" \--write-out=table endpoint status # +----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ # | ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | # +----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ # | 10.10.107.225:2379 | 2e8fddd3366a3d88 | 3.5.4 | 20 kB | false | false | 3 | 12 | 12 | | # | 10.10.107.224:2379 | 2480d95a2df867a4 | 3.5.4 | 20 kB | true | false | 3 | 12 | 12 | | # | 10.10.107.223:2379 | 144934d02ad45ec7 | 3.5.4 | 20 kB | false | false | 3 | 12 | 12 | | # +----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+# 集群节点健康状态etcdctl --endpoints=10.10.107.225:2379,10.10.107.224:2379,10.10.107.223:2379 \--cacert="/etc/etcd/pki/ca.pem" --cert="/etc/etcd/pki/etcd.pem" --key="/etc/etcd/pki/etcd-key.pem" \--write-out=table endpoint health # +----------------------------+--------+-------------+-------+ # | ENDPOINT | HEALTH | TOOK | ERROR | # +----------------------------+--------+-------------+-------+ # | 10.10.107.225:2379 | true | 9.151813ms | | # | 10.10.107.224:2379 | true | 10.965914ms | | # | 10.10.107.223:2379 | true | 11.165228ms | | # +----------------------------+--------+-------------+-------+# 集群节点性能测试etcdctl --endpoints=10.10.107.225:2379,10.10.107.224:2379,10.10.107.223:2379 \--cacert="/etc/etcd/pki/ca.pem" --cert="/etc/etcd/pki/etcd.pem" --key="/etc/etcd/pki/etcd-key.pem" \--write-out=tableendpoint check perf# 59 / 60 Boooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooom ! 98.33%PASS: Throughput is 148 writes/s# Slowest request took too long: 1.344053s# Stddev too high: 0.143059s# FAIL 5.Containerd 运行时安装部署

步骤 01.在各主机中安装二进制版本的containerd.io运行时服务，Kubernertes 通过 CRI 插件来连接 containerd 服务中, 控制容器的生命周期。

# 从 Github 中下载最新的版本的 cri-containerd-cni wget -L github.com/containerd/containerd/releases/download/v1.6.4/cri-containerd-cni-1.6.4-linux-amd64.tar.gz# 解压到当前cri-containerd-cni目录中。mkdir -vp cri-containerd-cnitar -zxvf cri-containerd-cni-1.6.4-linux-amd64.tar.gz -C cri-containerd-cni

步骤 02.查看其文件以及配置文件路径信息。

$ tree ./cri-containerd-cni/.├── etc│ ├── cni│ │ └── net.d│ │ └── 10-containerd-net.conflist│ ├── crictl.yaml│ └── systemd│ └── system│ └── containerd.service├── opt│ ├── cni│ │ └── bin│ │ ├── bandwidth│ │ ├── bridge│ │ ├── dhcp│ │ ├── firewall│ │ ├── host-device│ │ ├── host-local│ │ ├── ipvlan│ │ ├── loopback│ │ ├── macvlan│ │ ├── portmap│ │ ├── ptp│ │ ├── sbr│ │ ├── static│ │ ├── tuning│ │ ├── vlan│ │ └── vrf│ └── containerd│ └── cluster│ ├── gce│ │ ├── cloud-init│ │ │ ├── master.yaml│ │ │ └── node.yaml│ │ ├── cni.template│ │ ├── configure.sh│ │ └── env│ └── version└── usr └── local ├── bin │ ├── containerd │ ├── containerd-shim │ ├── containerd-shim-runc-v1 │ ├── containerd-shim-runc-v2 │ ├── containerd-stress │ ├── crictl │ ├── critest │ ├── ctd-decoder │ └── ctr └── sbin └── runc# 然后在所有节点上复制到上述文件夹到对应目录中cd ./cri-containerd-cni/cp -r etc/ /cp -r opt/ /cp -r usr/ /

步骤 03.进行containerd 配置创建并修改 config.toml .

mkdir -vp /etc/containerd# 默认配置生成containerd config default >/etc/containerd/config.tomlls /etc/containerd/config.toml # /etc/containerd/config.toml# pause 镜像源sed -i "s#k8s.gcr.io/pause#registry.cn-hangzhou.aliyuncs.com/google_containers/pause#g" /etc/containerd/config.toml# 使用 SystemdCgroupsed -i 's#SystemdCgroup = false#SystemdCgroup = true#g' /etc/containerd/config.toml# docker.io mirrorsed -i '/registry.mirrors]/a\ \ \ \ \ \ \ \ [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]' /etc/containerd/config.tomlsed -i '/registry.mirrors."docker.io"]/a\ \ \ \ \ \ \ \ \ \ endpoint = ["xlx9erfu.mirror.aliyuncs.com","docker.mirrors.ustc.edu.cn"]' /etc/containerd/config.toml# gcr.io mirrorsed -i '/registry.mirrors]/a\ \ \ \ \ \ \ \ [plugins."io.containerd.grpc.v1.cri".registry.mirrors."gcr.io"]' /etc/containerd/config.tomlsed -i '/registry.mirrors."gcr.io"]/a\ \ \ \ \ \ \ \ \ \ endpoint = ["gcr.mirrors.ustc.edu.cn"]' /etc/containerd/config.toml# k8s.gcr.io mirrorsed -i '/registry.mirrors]/a\ \ \ \ \ \ \ \ [plugins."io.containerd.grpc.v1.cri".registry.mirrors."k8s.gcr.io"]' /etc/containerd/config.tomlsed -i '/registry.mirrors."k8s.gcr.io"]/a\ \ \ \ \ \ \ \ \ \ endpoint = ["gcr.mirrors.ustc.edu.cn/google-containers/","registry.cn-hangzhou.aliyuncs.com/google_containers/"]' /etc/containerd/config.toml# quay.io mirrorsed -i '/registry.mirrors]/a\ \ \ \ \ \ \ \ [plugins."io.containerd.grpc.v1.cri".registry.mirrors."quay.io"]' /etc/containerd/config.tomlsed -i '/registry.mirrors."quay.io"]/a\ \ \ \ \ \ \ \ \ \ endpoint = ["quay.mirrors.ustc.edu.cn"]' /etc/containerd/config.toml

步骤 04.客户端工具 runtime 与 镜像 端点配置:

# 手动设置临时生效# crictl config runtime-endpoint /run/containerd/containerd.sock# /run/containerd/containerd.sock # 配置文件设置永久生效cat <<EOF > /etc/crictl.yamlruntime-endpoint: unix:///run/containerd/containerd.sockimage-endpoint: unix:///run/containerd/containerd.socktimeout: 10debug: falseEOF

步骤 05.重载 systemd自启和启动containerd.io服务。

systemctl daemon-reload && systemctl enable --now containerd.servicesystemctl status containerd.servicectr version # Client: # Version: 1.5.11 # Revision: 3df54a852345ae127d1fa3092b95168e4a88e2f8 # Go version: go1.17.8 # Server: # Version: 1.5.11 # Revision: 3df54a852345ae127d1fa3092b95168e4a88e2f8 # UUID: 71a28bbb-6ed6-408d-a873-e394d48b35d8

步骤 06.用于根据OCI规范生成和运行容器的CLI工具 runc 版本查看

runc -v # runc version 1.1.1 # commit: v1.1.1-0-g52de29d7 # spec: 1.0.2-dev # go: go1.17.9 # libseccomp: 2.5.1

温馨提示: 当默认 runc 执行提示runc: symbol lookup error: runc: undefined symbol: seccomp_notify_respond时，由于上述软件包中包含的runc对系统依赖过多，所以建议单独下载安装 runc 二进制项目(github.com/opencontainers/runc/)

wget github.com/opencontainers/runc/releases/download/v1.1.1/runc.amd64# 执行权限赋予chmod +x runc.amd64# 替换掉 /usr/local/sbin/ 路径原软件包中的 runcmv runc.amd64 /usr/local/sbin/runc

本文至此完毕，更多技术文章，尽情期待下一章节！

欢迎各位志同道合的朋友一起学习交流，如文章有误请在下方留下您宝贵的经验知识，个人邮箱地址或者个人公众号联系我。

更多文章来源于

个人主页:

博客地址:

专栏书写不易，如果您觉得这个专栏还不错的，请给这篇专栏(人间六大情)，这将对我的肯定，谢谢！。

• echo "，动动你那粗壮的拇指或者芊芊玉手，亲！"

• printf("%s", "，万水千山总是情，投个硬币行不行，亲！")

• fmt.Printf("，阅后即焚不吃灰，亲！")

• console.info("，让更多的志同道合的朋友一起学习交流，亲！")

• System.out.println("，后续浏览查看不迷路哟，亲！")

• cout << "，文章写得好不好、有没有错误，一定要留言哟，亲! " << endl;

往期相关文章

记一次在k8s集群搭建的Harbor私有仓库无法提供服务之镜像迁移恢复实践

K9s之Kubernetes集群管理交互工具实践

K9s之Kuberntes集群管理交互工具实践

3.Containerd容器运行时的配置浅析与知识扩充实践

4.如何使用nerdctl工具并配合Containerd容器运行时来替代Docker容器环境

WeiyiGeek

Always keep a beginner's mind, don't forget the beginner's mind. Blog :

174篇原创内容

更多网络安全、系统运维、应用开发、全栈文章，尽在站点，谢谢支持！

↓↓↓更多文章，请点击下方阅读原文。