K8s 安装包咋做？二进制制作全流程

简介

本章节将安装k8s-1.34需要的二进制文件、证书配置文件，脚本，以及etcd、master端组件、worker端组件所用到的配置文件、启动脚本都准备好。同时客户可以自行按照&34; 中的软件都下载准备好。该篇文章中的所有配置文件和脚本可以在git地址获取
:https://gitee.com/eeenet/k8s-install

== 文章导航===

【k8s-1.34.2安装部署】一.系统初始化及k8s集群规划
【k8s-1.34.2安装部署】二.kubernets软件、证书、配置、脚本等文件准备
【k8s-1.34.2安装部署】三.etcd-v3.6.6 TLS版集群安装
【k8s-1.34.2安装部署】四.kubernets master组件kube-apiserver，kube-controller-manager，kube-scheduler安装
【k8s-1.34.2安装部署】五.worker端containerd2.2.1、kubelet-1.34.2安装
【k8s-1.34.2安装部署】六.企业级部署cilium-1.18.4网络插件
【k8s-1.34.2安装部署】七.coredns-v1.13.1安装
【k8s-1.34.2安装部署】八.metric-server-0.8.0安装
【k8s-1.34.2安装部署】九.k8s管理平台xkube安装部署
【k8s-1.34.2安装部署】十.gateway Api v1.4.0安装
【k8s-1.34.2安装部署】十一.metallb-v0.15.2安装

一.软件下载及安装客户端工具

1.1.安装客户端工具

软件的下载地址参考上一章节:,将软件包解压提取出命令文件放到/usr/local/bin目录下并给予执行权限.
具体软件如下：

1.证书生成软件 cfssl-certinfo_1.6.5_linux_amd64 重命名为cfssljson cfssl_1.6.5_linux_amd64 这是cilium的客户端软件,解压提取出cilium cni-plugins-linux-amd64-v1.8.0.tgz 3.containerd 及运行时 containerd-2.2.0-linux-amd64.tar.gz runc.amd64-1.3.3 4.etcd etcd-v3.6.6-linux-amd64.tar.gz 5.k8s包管理工具,安装cilium helm-v4.0.0-linux-amd64.tar.gz 解压提取出kubectl 和 kubectl-convert 和 kubeadm

从上边的安装包上解压提取出如下文件，并给予执行权限: chmod +x *,然后拷贝到/usr/local/bin目录下

drwxr-xr-x 2 root root 4096 Dec 16 11:59 ./ drwxr-xr-x 9 root root 4096 Dec 16 11:59 ../ -rwxr-xr-x 1 root root 11890840 May 10 2024 cfssl* -rwxr-xr-x 1 root root 8413336 May 10 2024 cfssl-certinfo* -rwxr-xr-x 1 root root 6205592 May 10 2024 cfssljson* -rwxr-xr-x 1 root root 139694264 Oct 23 01:46 cilium* -rwxr-xr-x 1 root root 65491128 Nov 12 19:39 helm* -rwxr-xr-x 1 root root 74027192 Nov 12 03:26 kubeadm* -rwxr-xr-x 1 root root 60559544 Nov 12 03:26 kubectl* -rwxr-xr-x 1 root root 59642040 Nov 12 03:26 kubectl-convert*

二.准备证书配置文件并制作证书

2.1.证书配置文件列表如下

创建一个csr的目录，配置文件可以从git地址获取：
https://gitee.com/eeenet/k8s-install
将如下文件都放在这个csr目录下,如果不从git下载，可以拷贝2.3-2.10的证书配置。

-rw-r--r-- 1 root root 245 Feb 24 2023 admin-csr.json -rw-r--r-- 1 root root 330 Feb 23 2023 ca-config.json -rw-r--r-- 1 root root 284 Nov 21 17:24 ca-csr.json -rw-r--r-- 1 root root 410 Dec 5 12:16 etcd-csr.json -rw-r--r-- 1 root root 458 Dec 5 12:15 kube-apiserver-csr.json -rw-r--r-- 1 root root 292 Feb 23 2023 kube-controller-manager-csr.json -rw-r--r-- 1 root root 274 Feb 23 2023 kube-scheduler-csr.json -rw-r--r-- 1 root root 272 Feb 23 2023 proxy-client-csr.json

2.2.准备制作证书的脚本并制作证书：

脚本：create-cert.sh,拷贝并给予执行权限,和上边的csr文件夹在同一个目录下,准备好以上证书配置文件以后,执行该脚本，就会在当前目录创建cert目录，并将所有证书生成到cert下。
证书配置注意事项：1.证书中的hosts部分可以根据实际情况更换成自己需要的IP或域名。为了以后方便扩容，也可以规划一个泛域名。

34;cert&34;create ca.pem ca-key.pem=======&34;create etcd.pem etcd-key.pem=======&34;create kube-apiserver.pem kube-apiserver-key.pem=======&34;create kube-scheduler.pem kube-scheduler-key.pem=======&34;create kube-controller-manager.pem kube-controller-manager-key.pem=======&34;create proxy-client.pem proxy-client-key.pem=======&34;create admin.pem admin-key.pem=======&34;signing&34;default&34;expiry&34;175200h&34;profiles&34;kubernetes&34;expiry&34;175200h&34;usages&34;signing&34;key encipherment&34;server auth&34;client auth&34;CN&34;kubernetes&34;key&34;algo&34;rsa&34;size&34;names&34;C&34;CN&34;L&34;Guangzhou&34;ST&34;Guangdong&34;O&34;k8s&34;OU&34;System&34;CN&34;etcd&34;hosts&34;etcd01.my-k8s.local&34;etcd02.my-k8s.local&34;etcd03.my-k8s.local&34;*.my-k8s.local&34;127.0.0.1&34;key&34;algo&34;rsa&34;size&34;names&34;C&34;CN&34;L&34;Guangzhou&34;ST&34;Guangdong&34;CN&34;kubernetes&34;hosts&34;apiserver.my-k8s.local&34;*.my-k8s.local&34;127.0.0.1&34;10.96.0.1&34;kubernetes&34;kubernetes.default&34;kubernetes.default.svc&34;kubernetes.default.svc.cluster&34;kubernetes.default.svc.cluster.local&34;key&34;algo&34;rsa&34;size&34;names&34;C&34;CN&34;L&34;Guangzhou&34;ST&34;Guangdong&34;O&34;k8s&34;OU&34;system&34;CN&34;system:kube-controller-manager&34;hosts&34;127.0.0.1&34;key&34;algo&34;rsa&34;size&34;names&34;C&34;CN&34;ST&34;Guangdong&34;L&34;Guangzhou&34;O&34;system:kube-controller-manager&34;OU&34;system&34;CN&34;system:kube-scheduler&34;hosts&34;127.0.0.1&34;key&34;algo&34;rsa&34;size&34;names&34;C&34;CN&34;ST&34;Guangdong&34;L&34;Guangzhou&34;O&34;system:kube-scheduler&34;OU&34;system&34;CN&34;admin&34;hosts&34;key&34;algo&34;rsa&34;size&34;names&34;C&34;CN&34;ST&34;Guangdong&34;L&34;Guangzhou&34;O&34;system:masters&34;OU&34;system&34;CN&34;aggregator&34;hosts&34;key&34;algo&34;rsa&34;size&34;names&34;C&34;CN&34;ST&34;Guangdong&34;L&34;Guangzhou&34;O&34;system:masters&34;OU&34;System&!/bin/bash cert_dir=&34; kube_dir=&34; KUBE_APISERVER=&34; [ -d $kube_dir ] || mkdir -p $kube_dir echo &34; cat > $kube_dir/token.csv << EOF $(head -c 16 /dev/urandom | od -An -t x | tr -d &39;),kubelet-bootstrap,10001,&34; EOF echo &34; kubectl config set-cluster kubernetes \ --certificate-authority=$cert_dir/ca.pem \ --embed-certs=true \ --server=${KUBE_APISERVER} \ --kubeconfig=$kube_dir/kube-controller-manager.kubeconfig kubectl config set-credentials system:kube-controller-manager \ --client-certificate=$cert_dir/kube-controller-manager.pem \ --client-key=$cert_dir/kube-controller-manager-key.pem \ --embed-certs=true \ --kubeconfig=$kube_dir/kube-controller-manager.kubeconfig kubectl config set-context system:kube-controller-manager \ --cluster=kubernetes \ --user=system:kube-controller-manager \ --kubeconfig=$kube_dir/kube-controller-manager.kubeconfig kubectl config use-context system:kube-controller-manager --kubeconfig=$kube_dir/kube-controller-manager.kubeconfig echo &34; kubectl config set-cluster kubernetes \ --certificate-authority=$cert_dir/ca.pem \ --embed-certs=true \ --server=${KUBE_APISERVER} \ --kubeconfig=$kube_dir/kube-scheduler.kubeconfig kubectl config set-credentials system:kube-scheduler \ --client-certificate=$cert_dir/kube-scheduler.pem \ --client-key=$cert_dir/kube-scheduler-key.pem \ --embed-certs=true \ --kubeconfig=$kube_dir/kube-scheduler.kubeconfig kubectl config set-context system:kube-scheduler \ --cluster=kubernetes \ --user=system:kube-scheduler \ --kubeconfig=$kube_dir/kube-scheduler.kubeconfig kubectl config use-context system:kube-scheduler --kubeconfig=$kube_dir/kube-scheduler.kubeconfig echo &34; TOKEN=$(awk -F &34; &39; $kube_dir/token.csv) kubectl config set-cluster kubernetes \ --certificate-authority=$cert_dir/ca.pem \ --embed-certs=true \ --server=${KUBE_APISERVER} \ --kubeconfig=$kube_dir/kubelet-bootstrap.kubeconfig kubectl config set-credentials kubelet-bootstrap \ --token=${TOKEN} \ --kubeconfig=$kube_dir/kubelet-bootstrap.kubeconfig kubectl config set-context default \ --cluster=kubernetes \ --user=kubelet-bootstrap \ --kubeconfig=$kube_dir/kubelet-bootstrap.kubeconfig kubectl config use-context default --kubeconfig=$kube_dir/kubelet-bootstrap.kubeconfig echo &34; kubectl config set-cluster kubernetes \ --certificate-authority=$cert_dir/ca.pem \ --embed-certs=true \ --server=${KUBE_APISERVER} \ --kubeconfig=$kube_dir/kube.kubeconfig kubectl config set-credentials admin \ --client-certificate=$cert_dir/admin.pem \ --client-key=$cert_dir/admin-key.pem \ --embed-certs=true \ --kubeconfig=$kube_dir/kube.kubeconfig kubectl config set-context kubernetes \ --cluster=kubernetes \ --user=admin \ --kubeconfig=$kube_dir/kube.kubeconfig kubectl config use-context kubernetes --kubeconfig=$kube_dir/kube.kubeconfig

四.准备etcd配置文件及启动脚步

4.1.etcd.conf

etcd01机器的 的配置，每台机不一样

34;etcd01&34;/opt/etcd/default.etcd&34;https://0.0.0.0:2380&34;https://0.0.0.0:2379&[Clustering] ETCD_INITIAL_ADVERTISE_PEER_URLS=&34; ETCD_ADVERTISE_CLIENT_URLS=&34; ETCD_INITIAL_CLUSTER=&34; ETCD_INITIAL_CLUSTER_TOKEN=&34; ETCD_INITIAL_CLUSTER_STATE=&34; ETCD_LISTEN_METRICS_URLS=&34;

etcd02的配置，每台机不一样

34;etcd02&34;/opt/etcd/default.etcd&34;https://0.0.0.0:2380&34;https://0.0.0.0:2379&[Clustering] ETCD_INITIAL_ADVERTISE_PEER_URLS=&34; ETCD_ADVERTISE_CLIENT_URLS=&34; ETCD_INITIAL_CLUSTER=&34; ETCD_INITIAL_CLUSTER_TOKEN=&34; ETCD_INITIAL_CLUSTER_STATE=&34; ETCD_LISTEN_METRICS_URLS=&34;

etcd03 的配置，每台机不一样

34;etcd03&34;/opt/etcd/default.etcd&34;https://0.0.0.0:2380&34;https://0.0.0.0:2379&[Clustering] ETCD_INITIAL_ADVERTISE_PEER_URLS=&34; ETCD_ADVERTISE_CLIENT_URLS=&34; ETCD_INITIAL_CLUSTER=&34; ETCD_INITIAL_CLUSTER_TOKEN=&34; ETCD_INITIAL_CLUSTER_STATE=&34; ETCD_LISTEN_METRICS_URLS=&34;

4.2.etcd启动脚本

[Unit] Description=Etcd Server After=network.target After=network-online.target Wants=network-online.target [Service] Type=notify EnvironmentFile=-/opt/etcd/conf/etcd.conf WorkingDirectory=/opt/etcd/ ExecStart=/opt/etcd/bin/etcd \ --cert-file=/opt/etcd/ssl/etcd.pem \ --key-file=/opt/etcd/ssl/etcd-key.pem \ --trusted-ca-file=/opt/etcd/ssl/ca.pem \ --peer-cert-file=/opt/etcd/ssl/etcd.pem \ --peer-key-file=/opt/etcd/ssl/etcd-key.pem \ --peer-trusted-ca-file=/opt/etcd/ssl/ca.pem \ --peer-client-cert-auth \ --client-cert-auth Restart=on-failure RestartSec=5 LimitNOFILE=65536 [Install] WantedBy=multi-user.target

五.准备master相关的配置文件

5.1.kube-apiserver.conf

注意配置中的文件、证书路径。注意以下几个参数：
--etcd-servers： 配置etcd的主机名，需要apiserver能解析此主机名，在/etc/hosts中加入映射。

--service-cluster-ip-range：配置service的网段，用章节一：文章中规划的网段

KUBE_APISERVER_OPTS=&34;

5.2.kube-controller-manager.conf

注意配置中的文件、证书路径。注意以下几个参数：

--service-cluster-ip-range： service 网段
--cluster-cidr： pod网段

KUBE_CONTROLLER_MANAGER_OPTS=&34;

5.3.kube-scheduler.conf

注意配置中的文件路径

KUBE_SCHEDULER_OPTS=&34;

5.4.kubelet.yaml

注意几个参数：
clusterDNS: 定义coredns的服务IP。
/opt/kubernetes/ssl/ca.pem resolvConf：
/run/systemd/resolve/resolv.conf为系统的resolved的dns配置路径，不配置此项会导致读取/etc/resolv.conf,而/etc/resolv.conf是
/run/systemd/resolve/stub-resolv.conf的软连接，里面配置了本地缓存dns,127.0.0.1:53,会和k8s导致dns冲突

kind: KubeletConfiguration apiVersion: kubelet.config.k8s.io/v1beta1 address: 0.0.0.0 port: 10250 readOnlyPort: 10255 cgroupDriver: systemd clusterDNS: - 10.96.0.10 clusterDomain: cluster.local failSwapOn: false authentication: anonymous: enabled: false webhook: cacheTTL: 2m0s enabled: true x509: clientCAFile: /opt/kubernetes/ssl/ca.pem authorization: mode: Webhook webhook: cacheAuthorizedTTL: 5m0s cacheUnauthorizedTTL: 30s evictionHard: imagefs.available: 15% memory.available: 100Mi nodefs.available: 10% nodefs.inodesFree: 5% maxOpenFiles: 2048000 maxPods: 200 resolvConf: /run/systemd/resolve/resolv.conf

4.5.containerd配置文件

containerd的配置文件，可以从containerd命令导出默认配置，并修改里面的镜像地址和SystemdCgroup 为true,导出命令参考如下：

containerd config default | sudo tee /etc/containerd/config.toml sed -i &SystemdCgroup.*&39;ssandbox_image = &34;39; /etc/containerd/config.toml

config.toml 配置

4.6.kube-apiserver.service 启动脚本

kube-apiserver.service

[Unit] Description=Kubernetes API Server Documentation=https://github.com/kubernetes/kubernetes After=etcd.service Wants=etcd.service [Service] EnvironmentFile=-/opt/kubernetes/conf/kube-apiserver.conf ExecStart=/opt/kubernetes/bin/kube-apiserver $KUBE_APISERVER_OPTS Restart=on-failure RestartSec=5 Type=notify LimitNOFILE=65536 [Install] WantedBy=multi-user.target

4.7.kube-controller-manager.service 启动脚本

kube-controller-manager.service

[Unit] Description=Kubernetes Controller Manager Documentation=https://github.com/kubernetes/kubernetes [Service] EnvironmentFile=-/opt/kubernetes/conf/kube-controller-manager.conf ExecStart=/opt/kubernetes/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTS Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target

4.8.kube-scheduler.service 启动脚本

kube-scheduler.service

[Unit] Description=Kubernetes Scheduler Documentation=https://github.com/kubernetes/kubernetes [Service] EnvironmentFile=-/opt/kubernetes/conf/kube-scheduler.conf ExecStart=/opt/kubernetes/bin/kube-scheduler $KUBE_SCHEDULER_OPTS Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target

4.9.kubelet.service 启动脚本

kubelet.service

[Unit] Description=Kubernetes Kubelet Documentation=https://github.com/kubernetes/kubernetes After=containerd.service Requires=containerd.service [Service] ExecStart=/opt/kubernetes/bin/kubelet \ --hostname-override=node-hostname \ Copyright The containerd Authors. Licensed under the Apache License, Version 2.0 (the &34;); You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software 34;AS IS& WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. limitations under the License. [Unit] Description=containerd container runtime Documentation=https://containerd.io After=network.target dbus.service [Service] ExecStartPre=-/sbin/modprobe overlay ExecStart=/usr/local/bin/containerd Type=notify Delegate=yes KillMode=process Restart=always RestartSec=5 in the kernel. We recommend using cgroups to do container-local accounting. LimitNPROC=infinity LimitCORE=infinity Only systemd 226 and above support this version. TasksMax=infinity OOMScoreAdjust=-999 [Install] WantedBy=multi-user.target

六.制作etcd、master、containerd、node的安装包

6.1. etcd安装包

1.创建目录etcd,解压
etcd-v3.6.6-linux-amd64.tar.gz 将etcd、etcdctl、etcdutl 三个文件拷贝到etcd/bin下。
2.拷贝etcd.conf 到etcd/conf目录下,此处的配置文件，三台机分别拷贝etcd01.conf,etcd02.conf,etcd03.conf。
3.拷贝cert目录下生成好的证书到ssl目录下，注：该步骤是需要在前面已经在执行了create-cert.sh和create-kubeconfig.sh以后会生成，生成后不需要重复执行,不能将多次执行生成的cert和kubeconfig文件拷贝到不同的安装包内，一次执行，拷贝到不同的安装包。
4.etcd.service 是启动脚本，安装时需要拷贝到/usr/lib/systemd/system/

etcd目录结构如下: ├── bin │ ├── etcd │ ├── etcdctl │ └── etcdutl ├── conf │ └── etcd.conf ├── etcd.service ├── logs └── ssl ├── ca-key.pem ├── ca.pem ├── etcd-key.pem └── etcd.pem

6.2. 制作master节点安装包

1.master除了安装kube-controller-manager、kube-scheduler、kube-apiserver，以外还需要安装containerd和kubelet。其中四个.service文件是启动脚本,安装时需要拷贝到/usr/lib/systemd/system/

拷贝cert目录下生成好的证书到ssl目录下，需要拷贝的文件参考如下目录结构kubeconfig文件拷贝到conf目录下,需要拷贝的文件参考如下目录结构

master节点目录结构 ├── bin │ ├── kube-apiserver │ ├── kube-controller-manager │ ├── kubelet │ └── kube-scheduler ├── conf 在kubeconfig目录下 ├── kube-apiserver.service ├── kube-controller-manager.service ├── kubelet.service ├── kube-scheduler.service ├── logs └── ssl 从kubeconfig目录拷贝kubeconfig文件 │ ├── kubelet-bootstrap.kubeconfig │ ├── kubelet.yaml ├── kubelet.service ├── logs └── ssl #从cert目录拷贝 ├── ca-key.pem ├── ca.pem

6.4. containerd安装包

containerd2.0 开始安装时需要安装runc 和cni-plugins。该安装包是将runc 和 cni-plugins、containerd的执行文件、配置、启动脚本拷贝到一起。安装时：bin目录下的文件拷贝到/usr/local/bin下，cni目录拷贝到/opt/下，sbin/runc 拷贝到/usr/local/sbin 下，config.toml 拷贝到/etc/containerd，crictl.yaml 拷贝到/etc/ 下，containerd.service 拷贝到/etc/systemd/system/下，具体参考如下目录结构

下载地址:

https://gitee.com/eeenet/k8s-install

https://github.com/kubernetes-sigs/cri-tools/releases

https://github.com/containerd/containerd/releases

https://github.com/opencontainers/runc/releases

https://github.com/containernetworking/plugins/releases

├── etc │ ├── containerd │ │ └── config.toml │ ├── crictl.yaml │ └── systemd │ └── system │ └── containerd.service ├── opt │ └── cni │ └── bin │ ├── bandwidth │ ├── bridge │ ├── dhcp │ ├── dummy │ ├── firewall │ ├── host-device │ ├── host-local │ ├── ipvlan │ ├── LICENSE │ ├── loopback │ ├── macvlan │ ├── portmap │ ├── ptp │ ├── README.md │ ├── sbr │ ├── static │ ├── tap │ ├── tuning │ ├── vlan │ └── vrf └── usr └── local ├── bin │ ├── containerd │ ├── containerd-shim-runc-v2 │ ├── containerd-stress │ ├── crictl │ └── ctr └── sbin └── runc