二进制部署kubernetes集群

k8s 集群控制节点，对集群进行调度管理，接受集群外用户去集群操作请求

master节点上的主要组件包括：

1、kube-apiserver：集群控制的入口，提供 HTTP REST 服务，同时交给etcd存储，提供认证、授权、访问控制、API注册和发现等机制

2、kube-controller-manager：Kubernetes 集群中所有资源对象的自动化控制中心，理集群中常规后台任务，一个资源对应一个控制器

3、kube-scheduler：负责 Pod 的调度，选择node节点应用部署

4、etcd数据库（也可以安装在单独的服务器上）：存储系统，用于保存集群中的相关数据

node节点上的主要组件包括：
1、kubelet：master派到node节点代表，管理本机容器

一个集群中每个节点上运行的代理，它保证容器都运行在Pod中
负责维护容器的生命周期，同时也负责Volume(CSI) 和网络(CNI)的管理
2、kube-proxy：提供网络代理，负载均衡等操作

3、容器运行环境（Container runtime）如docker

容器运行环境是负责运行容器的软件
Kubernetes支持多个容器运行环境：Docker、containerd、cri-o、rktlet以及任何实现Kubernetes CRI (容器运行环境接口) 的软件。

1.准备工作

在开始之前，部署Kubernetes集群机器需要满足以下几个条件

一台或多台机器，操作系统CentOS 7.x
硬件配置：2GB ，2个CPU，硬盘30GB
集群中所有机器之间网络互通
可以访问外网，需要拉取镜像，如果服务器不能上网，需要提前下载镜像导入节点
禁止swap分区

2.准备虚拟机

首先我们准备了三台虚拟机，来进行安装测试

主机名	IP	组件
master1	172.16.1.125	kube-apiserver，kube-controller-manager，kube -scheduler，etcd
node1	172.16.1.216	kubelet，kube-proxy，docker，flannel，etcd
node2	172.16.1.242	kubelet，kube-proxy，docker，flannel，etcd

3.基础环境

#各个机器设置自己的域名
hostnamectl set-hostname master1
hostnamectl set-hostname node1
hostnamectl set-hostname node2

# 添加hosts
cat >> /etc/hosts << EOF
172.16.1.125 master1
172.16.1.216 node1
172.16.1.242 node2
EOF

# 将 SELinux 设置为 permissive 模式（相当于将其禁用）
sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

#关闭swap
swapoff -a  
sed -ri 's/.*swap.*/#&/' /etc/fstab

#允许 iptables 检查桥接流量
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sudo sysctl --system

# 时间同步
yum install ntpdate -y
ntpdate time.windows.com

4.部署Etcd集群

Etcd是一个分布式键值存储系统，Kubernetes使用Etcd进行数据存储，所以先准备一个Etcd数据库，为了解决Etcd单点故障，应采用集群方式部署，这里使用3台组建集群，可容忍一台机器故障，当然也可以使用5台组件集群，可以容忍2台机器故障

节点名称	IP
etcd1	172.16.1.125
etcd2	172.16.1.216
etcd3	172.16.1.242

1	`注：为了节省机器，这里与 K8s 节点机器复用。也可以独立于 k8s 集群之外部署，只要apiserver 能连接到就行`

4.1准备cfssl证书生成工具

cfssl是一个开源的证书管理工具，使用json文件生成证书，相比openssl 更方便使用。在Master节点进行操作

wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64
mv cfssl_linux-amd64 /usr/local/bin/cfssl
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo

4.2生成 Etcd证书

4.2.1自签证书颁发机构（CA）

1 2	`mkdir -p ~/TLS/{etcd,k8s} cd TLS/etcd`

4.2.2自签CA

ca-config.json

cat > ca-config.json << EOF
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "www": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}
EOF

ca-csr.json

cat > ca-csr.json<< EOF 
{
    "CN": "etcd CA",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing"
        }
    ]
}
EOF

生成证书：

[root@m1 etcd]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
----------------------------
2023/04/07 11:03:33 [INFO] generating a new CA key and certificate from CSR
2023/04/07 11:03:33 [INFO] generate received request
2023/04/07 11:03:33 [INFO] received CSR
2023/04/07 11:03:33 [INFO] generating key: rsa-2048
2023/04/07 11:03:34 [INFO] encoded CSR
2023/04/07 11:03:34 [INFO] signed certificate with serial number 174425569924111551053156001326556705558407151977

[root@m1 etcd]# ls
ca-config.json  ca.csr  ca-csr.json  ca-key.pem  ca.pem
[root@m1 etcd]# ls *pem
ca-key.pem ca.pem
[root@m1 etcd]# ls |wc -l
5

4.2.3使用自签 CA 签发 Etcd HTTPS 证书

创建证书申请文件：

#注意替换IP
cat > server-csr.json << EOF
{
  "CN": "etcd",
  "hosts": [
    "172.16.1.125",
    "172.16.1.216",
    "172.16.1.242"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "BeiJing",
      "ST": "BeiJing"
    }
  ]
}
EOF

生成证书：

[root@m1 etcd]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server
-------------------
2023/04/07 11:12:05 [INFO] generate received request
2023/04/07 11:12:05 [INFO] received CSR
2023/04/07 11:12:05 [INFO] generating key: rsa-2048
2023/04/07 11:12:05 [INFO] encoded CSR
2023/04/07 11:12:05 [INFO] signed certificate with serial number 164505293888830887708575372666336673001865598333
2023/04/07 11:12:05 [WARNING] This certificate lacks a "hosts" field. This makes its unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
[root@m1 etcd]# ls
ca-config.json  ca.csr  ca-csr.json  ca-key.pem  ca.pem  server.csr  server-csr.json  server-key.pem  server.pem
[root@m1 etcd]# ls server*.pem
server-key.pem  server.pem
[root@m1 etcd]# ls |wc -l
9

4.3部署 Etcd集群

4.3.1创建工作目录并下载二进制包

cd ~
wget https://github.com/etcd-io/etcd/releases/download/v3.4.9/etcd-v3.4.9-linux-amd64.tar.gz
mkdir /opt/etcd/{bin,cfg,ssl} -p    #bin里面存放的是可执行文件,cfg配置文件,ssl证书
tar zxvf etcd-v3.4.9-linux-amd64.tar.gz
mv etcd-v3.4.9-linux-amd64/{etcd,etcdctl} /opt/etcd/bin/

4.3.2创建 etcd配置文件

cat > /opt/etcd/cfg/etcd.conf << EOF
#[Member]
ETCD_NAME="etcd-1"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://172.16.1.125:2380" 
ETCD_LISTEN_CLIENT_URLS="https://172.16.1.125:2379" 
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://172.16.1.125:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://172.16.1.125:2379"
ETCD_INITIAL_CLUSTER="etcd-1=https://172.16.1.125:2380,etcd-2=https://172.16.1.216:2380,etcd-3=https://172.16.1.242:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" 
ETCD_INITIAL_CLUSTER_STATE="new"
EOF

ETCD_NAME：节点名称，集群中唯一
ETCD_DATA_DIR：数据目录
ETCD_LISTEN_PEER_URLS：集群通信监听地址
ETCD_LISTEN_CLIENT_URLS：客户端访问监听地址
ETCD_INITIAL_ADVERTISE_PEER_URLS：集群通告地址
ETCD_ADVERTISE_CLIENT_URLS：客户端通告地址

ETCD_INITIAL_CLUSTER：集群节点地址
ETCD_INITIAL_CLUSTER_TOKEN：集群 Token
ETCD_INITIAL_CLUSTER_STATE：加入集群的当前状态，new是新集群，existing表示加入已有集群

4.3.3systemd管理 etcd

cat > /usr/lib/systemd/system/etcd.service << EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
EnvironmentFile=/opt/etcd/cfg/etcd.conf
ExecStart=/opt/etcd/bin/etcd \
    --cert-file=/opt/etcd/ssl/server.pem \
    --key-file=/opt/etcd/ssl/server-key.pem \
    --peer-cert-file=/opt/etcd/ssl/server.pem \
    --peer-key-file=/opt/etcd/ssl/server-key.pem \
    --trusted-ca-file=/opt/etcd/ssl/ca.pem \
    --peer-trusted-ca-file=/opt/etcd/ssl/ca.pem \
    --logger=zap
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF

4.3.4生成的证书拷贝到配置文件中的路径

1	`cp ~/TLS/etcd/capem ~/TLS/etcd/serverpem /opt/etcd/ssl/`

4.3.4拷贝至node1,node2节点

scp -r /opt/etcd/ root@node1:/opt/
scp /usr/lib/systemd/system/etcd.service root@node1:/usr/lib/systemd/system/

scp -r /opt/etcd/ root@node2:/opt/
scp /usr/lib/systemd/system/etcd.service root@node2:/usr/lib/systemd/system/

4.3.5修改 etcd.conf 配置文件中的节点名称和当前服务器 IP

vim /opt/etcd/cfg/etcd.conf
#[Member]
ETCD_NAME="etcd-1" # 修改此处，节点 2 改为 etcd-2，节点 3改为 etcd-3
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://172.16.1.125:2380" # 修改此处为当前服务器 IP ETCD_LISTEN_CLIENT_URLS="https://172.16.1.125:2379" # 修改此处为当前服务器 IP
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://172.16.1.125:2380" # 修改此处为当前服务器 IP
ETCD_ADVERTISE_CLIENT_URLS="https://172.16.1.125:2379" # 修改此处为当前服务器IP
ETCD_INITIAL_CLUSTER="etcd-1=https://172.16.1.125:2380,etcd-2=htttps://172.16.1.216:2380,etcd-3=https://172.16.1.242:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

4.3.6启动并设置开机启动

systemctl daemon-reload 
systemctl start etcd 
systemctl enable etcd
systemctl status etcd

#查看集群状态
ETCDCTL_API=3 /opt/etcd/bin/etcdctl --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem --endpoints="https://172.16.1.125:2379,https://172.16.1.216:2379,https://172.16.1.242:2379" endpoint health
---------------------------
https://172.16.1.125:2379 is healthy: successfully committed proposal: took = 7.691741ms
https://172.16.1.216:2379 is healthy: successfully committed proposal: took = 7.842355ms
https://172.16.1.242:2379 is healthy: successfully committed proposal: took = 8.857985ms

报错查看日志：/var/log/message 或 journalctl -u etcd

5.docker安装

yum -y install yum-utils device-mapper-persistent-data lvm2 gcc
yum-config-manager	--add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
yum makecache fast
yum -y install docker-ce
mkdir -p /etc/docker
cat > /etc/docker/daemon.json <<EOF
{
  "registry-mirrors": ["https://hxqazmaw.mirror.aliyuncs.com"]
}
EOF
systemctl daemon-reload
systemctl start docker
systemctl enable docker
systemctl status docker

6.生成证书(只在master节点)

6.1生成 kube-apiserver 证书(用于部署master mode)

1 2	`# 切换目录 cd ~/TLS/k8s`

ca-config.json

cat > ca-config.json << EOF
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "kubernetes": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}
EOF

ca-csr.json

cat > ca-csr.json<< EOF 
{
    "CN": "kubernetes",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing",
            "O": "k8s",
            "OU": "System"
        }
    ]
}
EOF

生成证书

[root@m1 etcd]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
------------------
2023/04/07 14:11:58 [INFO] generating a new CA key and certificate from CSR
2023/04/07 14:11:58 [INFO] generate received request
2023/04/07 14:11:58 [INFO] received CSR
2023/04/07 14:11:58 [INFO] generating key: rsa-2048
2023/04/07 14:11:59 [INFO] encoded CSR
2023/04/07 14:11:59 [INFO] signed certificate with serial number 675942160448844902744791746300417237621797037302

[root@m1 k8s]# ls
ca-config.json  ca.csr  ca-csr.json  ca-key.pem  ca.pem
[root@m1 k8s]# ls *.pem
ca-key.pem  ca.pem
[root@m1 k8s]# ls |wc -l
5

6.2使用自签 CA 签发 kube-apiserver HTTPS 证书

创建证书申请文件：

cd ~/TLS/k8s
cat > server-csr.json << EOF
{
    "CN": "kubernetes",
    "hosts": [
      "10.0.0.1",
      "127.0.0.1",
      "172.16.1.125",
      "172.16.1.216",
      "172.16.1.242",
      "kubernetes",
      "kubernetes.default",
      "kubernetes.default.svc",
      "kubernetes.default.svc.cluster",
      "kubernetes.default.svc.cluster.local"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "BeiJing",
            "ST": "BeiJing",
            "O": "k8s",
            "OU": "System"
        }
    ]
}
EOF

生成证书：

[root@m1 etcd]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server
---------------
2023/04/07 14:17:52 [INFO] generate received request
2023/04/07 14:17:52 [INFO] received CSR
2023/04/07 14:17:52 [INFO] generating key: rsa-2048
2023/04/07 14:17:52 [INFO] encoded CSR
2023/04/07 14:17:52 [INFO] signed certificate with serial number 353411361046011027387548691075233577582166249129
2023/04/07 14:17:52 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
[root@m1 k8s]# ls
ca-config.json  ca.csr  ca-csr.json  ca-key.pem  ca.pem  server.csr  server-csr.json  server-key.pem  server.pem
[root@m1 k8s]# ls server*.pem
server-key.pem  server.pem
[root@m1 k8s]# ls |wc -l
9

6.3生成kube-proxy证书(用于部署worker node)

cd ~/TLS/k8s
# 创建证书请求文件
cat > kube-proxy-csr.json<< EOF
{
  "CN": "system:kube-proxy",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "BeiJing",
      "ST": "BeiJing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}
EOF

生成证书

[root@m1 k8s]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy
-----------------
2023/04/07 14:22:07 [INFO] generate received request
2023/04/07 14:22:07 [INFO] received CSR
2023/04/07 14:22:07 [INFO] generating key: rsa-2048
2023/04/07 14:22:08 [INFO] encoded CSR
2023/04/07 14:22:08 [INFO] signed certificate with serial number 702550285005427916593510235684249714244442590928
2023/04/07 14:22:08 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
[root@m1 k8s]# ls kube-proxy*.pem
kube-proxy-key.pem  kube-proxy.pem
[root@m1 k8s]# ls |wc -l
13

7. 部署Master Node

7.1 从 Github 下载二进制文件

下载地址：

https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.18.md#v1183

1
2
3

wget https://dl.k8s.io/v1.18.3/kubernetes-server-linux-amd64.tar.gz

5561483d796b124b8fe0e1cf5778ea03fec1e244ebc29f4b1b6c5ac93ab6bd10808d05da81b5f26426d51a3784c93ddf8b375ff971a78aebfd0ec7acac161365

7.2 解压二进制包

tar zxvf kubernetes-server-linux-amd64.tar.gz

mkdir -p /opt/kubernetes/{bin,cfg,ssl,logs} 
cd kubernetes/server/bin
cp kube-apiserver kube-scheduler kube-controller-manager /opt/kubernetes/bin
cp kubectl /usr/bin/

7.3 部署 kube-apiserver

7.3.1创建配置文件

cat > /opt/kubernetes/cfg/kube-apiserver.conf << EOF
KUBE_APISERVER_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/opt/kubernetes/logs \
--etcd-servers=https://172.16.1.125:2379,https://172.16.1.216:2379,https://172.16.1.242:2379 \
--bind-address=172.16.1.125 \
--secure-port=6443 \
--advertise-address=172.16.1.125 \
--allow-privileged=true \
--service-cluster-ip-range=10.0.0.0/24 \
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \
--authorization-mode=RBAC,Node \
--enable-bootstrap-token-auth=true \
--token-auth-file=/opt/kubernetes/cfg/token.csv \
--service-node-port-range=30000-32767 \
--kubelet-client-certificate=/opt/kubernetes/ssl/server.pem  \
--kubelet-client-key=/opt/kubernetes/ssl/server-key.pem \
--tls-cert-file=/opt/kubernetes/ssl/server.pem \
--tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \
--client-ca-file=/opt/kubernetes/ssl/ca.pem \
--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \
--etcd-cafile=/opt/etcd/ssl/ca.pem \
--etcd-certfile=/opt/etcd/ssl/server.pem \
--etcd-keyfile=/opt/etcd/ssl/server-key.pem \
--audit-log-maxage=30 \
--audit-log-maxbackup=3 \
--audit-log-maxsize=100 \
--audit-log-path=/opt/kubernetes/logs/k8s-audit.log"
EOF

– logtostderr：启用日志
– v：日志等级

– log-dir：日志目录
– etcd-servers：etcd集群地址
– bind-address：监听地址
– secure-port：https 安全端口
– advertise-address：集群通告地址
– allow-privileged：启用授权
– service-cluster-ip-range：Service虚拟 IP地址段
– enable-admission-plugins：准入控制模块
– authorization-mode：认证授权，启用 RBAC 授权和节点自管理
– enable-bootstrap-token-auth：启用 TLS bootstrap 机制
– token-auth-file：bootstrap token文件
– service-node-port-range：Service nodeport类型默认分配端口范围

– kubelet-client-xxx：apiserver 访问 kubelet客户端证书
– tls-xxx-file：apiserver https 证书
– etcd-xxxfile：连接 Etcd 集群证书
– audit-log-xxx：审计日志

7.3.2生成的证书拷贝到配置文件中的路径:

1	`cp ~/TLS/k8s/capem ~/TLS/k8s/serverpem /opt/kubernetes/ssl/`

7.3.3启用 TLS Bootstrapping 机制

TLS Bootstraping：Master apiserver 启用 TLS 认证后，Node 节点 kubelet 和 kube-proxy 要与 kube-apiserver 进行通信，必须使用 CA 签发的有效证书才可以，当 Node节点很多时，这种客户端证书颁发需要大量工作，同样也会增加集群扩展复杂度。为了简化流程，Kubernetes 引入了 TLS bootstraping 机制来自动颁发客户端证书，kubelet会以一个低权限用户自动向 apiserver 申请证书，kubelet 的证书由 apiserver 动态签署。所以强烈建议在 Node 上使用这种方式，目前主要用于 kubelet，kube-proxy 还是由我们统一颁发一个证书。

创建上述配置文件中 token 文件:

1
2
3

cat > /opt/kubernetes/cfg/token.csv << EOF
c47ffb939f5ca36231d9e3121a252940,kubelet-bootstrap,10001,"system:node-bootstrapper"
EOF

7.3.4systemd 管理 apiserver

 cat > /usr/lib/systemd/system/kube-apiserver.service << EOF
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-apiserver.conf
ExecStart=/opt/kubernetes/bin/kube-apiserver \$KUBE_APISERVER_OPTS 
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF

7.3.5启动并设置开机启动

systemctl daemon-reload
systemctl start kube-apiserver
systemctl enable kube-apiserver
systemctl status kube-apiserver

7.3.6授权 kubelet-bootstrap 用户允许请求证书

1
2
3

kubectl create clusterrolebinding kubelet-bootstrap \
--clusterrole=system:node-bootstrapper \
--user=kubelet-bootstrap

7.4部署 kube-controller-manager

7.4.1创建配置文件

cat > /opt/kubernetes/cfg/kube-controller-manager.conf << EOF
KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/opt/kubernetes/logs \
--leader-elect=true \
--master=127.0.0.1:8080 \
--bind-address=127.0.0.1 \
--allocate-node-cidrs=true \
--cluster-cidr=10.244.0.0/16 \
--service-cluster-ip-range=10.0.0.0/24 \
--cluster-name=kubernetes \
--cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \
--cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem  \
--root-ca-file=/opt/kubernetes/ssl/ca.pem \
--service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \
--experimental-cluster-signing-duration=87600h0m0s"
EOF

1
2
3

– master：通过本地非安全本地端口 8080 连接 apiserver。
– leader-elect：当该组件启动多个时，自动选举（HA）
– cluster-signing-cert-file/–cluster-signing-key-file：自动为 kubelet颁发证书的 CA，与 apiserver保持一致

7.4.2systemd 管理 controller-manager

cat > /usr/lib/systemd/system/kube-controller-manager.service << EOF
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-controller-manager.conf
ExecStart=/opt/kubernetes/bin/kube-controller-manager \$KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

7.4.3启动并设置开机启动

systemctl daemon-reload
systemctl start kube-controller-manager 
systemctl enable kube-controller-manager
systemctl status kube-controller-manager

7.5部署 kube-scheduler

7.5.1创建配置文件

cat > /opt/kubernetes/cfg/kube-scheduler.conf << EOF 
KUBE_SCHEDULER_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/opt/kubernetes/logs \
--leader-elect \
--master=127.0.0.1:8080 \
--bind-address=127.0.0.1"
EOF

1 2	`– master：通过本地非安全本地端口 8080 连接 apiserver。 – leader-elect：当该组件启动多个时，自动选举（HA）`

7.5.2systemd 管理 scheduler

cat > /usr/lib/systemd/system/kube-scheduler.service << EOF 
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-scheduler.conf
ExecStart=/opt/kubernetes/bin/kube-scheduler \$KUBE_SCHEDULER_OPTS 
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

7.5.3启动并设置开机启动

systemctl daemon-reload 
systemctl start kube-scheduler 
systemctl enable kube-scheduler
systemctl status kube-scheduler

7.6查看集群状态

所有组件都已经启动成功，通过 kubectl 工具查看当前集群组件状态：

[root@m1 ~]# kubectl get cs
NAME                 STATUS    MESSAGE             ERROR
scheduler            Healthy   ok                  
controller-manager   Healthy   ok                  
etcd-0               Healthy   {"health":"true"}   
etcd-1               Healthy   {"health":"true"}   
etcd-2               Healthy   {"health":"true"}

8. 部署Worker Node

8.1创建工作目录并拷贝二进制文件

在所有 worker node 创建工作目录：

mkdir -p /opt/kubernetes/{bin,cfg,ssl,logs}

#从master拷贝
cd kubernetes/server/bin
cp kubelet kube-proxy /opt/kubernetes/bin

scp kubelet kube-proxy root@node1:/opt/kubernetes/bin
scp kubelet kube-proxy root@node2:/opt/kubernetes/bin

8.2 创建bootstrap kubeconfig文件和kube-proxy kubeconfig文件

8.2.1创建kubeconfig文件

在生成kubernetes证书的目录下执行以下命令生成kubeconfig文件：
[root@m1 ~]# cd ~/TLS/k8s

#指定apiserver 内网负载均衡地址
[root@m1 k8s]# KUBE_APISERVER="https://172.16.1.125:6443"
[root@m1 k8s]# TOKEN=c47ffb939f5ca36231d9e3121a252940 #这个和上面创建token文件的一致
     
# 生成 kubelet bootstrap kubeconfig 配置文件

#设置集群参数
[root@m1 k8s]# kubectl config set-cluster kubernetes \
  --certificate-authority=/opt/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=bootstrap.kubeconfig

#设置客户端认证参数
[root@m1 k8s]# kubectl config set-credentials "kubelet-bootstrap" \
  --token=${TOKEN} \
  --kubeconfig=bootstrap.kubeconfig

#设置上下文参数
[root@m1 k8s]# kubectl config set-context default \
  --cluster=kubernetes \
  --user=kubelet-bootstrap \
  --kubeconfig=bootstrap.kubeconfig

#设置默认上下文
[root@m1 k8s]# kubectl config use-context default --kubeconfig=bootstrap.kubeconfig

8.2.2创建kube-proxy kubeconfig文件

[root@m1 k8s]# kubectl config set-cluster kubernetes \
  --certificate-authority=/opt/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=kube-proxy.kubeconfig

[root@m1 k8s]# kubectl config set-credentials kube-proxy \
  --client-certificate=./kube-proxy.pem \
  --client-key=./kube-proxy-key.pem \
  --embed-certs=true \
  --kubeconfig=kube-proxy.kubeconfig

[root@m1 k8s]# kubectl config set-context default \
  --cluster=kubernetes \
  --user=kube-proxy \
  --kubeconfig=kube-proxy.kubeconfig

[root@m1 k8s]# kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig

[root@m1 k8s]# ls *.kubeconfig
bootstrap.kubeconfig  kube-proxy.kubeconfig

!!!将这两个文件拷贝到Node节点/opt/kubernetes/cfg目录下
cp bootstrap.kubeconfig kube-proxy.kubeconfig /opt/kubernetes/cfg

scp bootstrap.kubeconfig kube-proxy.kubeconfig root@node1:/opt/kubernetes/cfg
scp bootstrap.kubeconfig kube-proxy.kubeconfig root@node2:/opt/kubernetes/cfg

8.3部署 kubelet

8.3.1创建配置文件

cat > /opt/kubernetes/cfg/kubelet.conf << EOF
KUBELET_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/opt/kubernetes/logs \
--hostname-override=k8s-m1 \
--network-plugin=cni \
--kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \
--bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \
--config=/opt/kubernetes/cfg/kubelet-config.yml \
--cert-dir=/opt/kubernetes/ssl \
--pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0"
EOF

– hostname-override：显示名称，集群中唯一
– network-plugin：启用 CNI
– kubeconfig：空路径，会自动生成，后面用于连接 apiserver
– bootstrap-kubeconfig：首次启动向 apiserver 申请证书
– config：配置参数文件
– cert-dir：kubelet证书生成目录
– pod-infra-container-image：管理 Pod 网络容器的镜像

8.3.2配置参数文件

cat > /opt/kubernetes/cfg/kubelet-config.yml << EOF
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
cgroupDriver: cgroupfs
clusterDNS: ["10.0.0.2"]
clusterDomain: cluster.local.
failSwapOn: false
authentication:
  anonymous:
    enabled: false 
  webhook:
    cacheTTL: 2m0s
    enabled: true
  x509:
    clientCAFile: /opt/kubernetes/ssl/ca.pem 
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 5m0s
    cacheUnauthorizedTTL: 30s 
evictionHard:
imagefs.available: 15%
memory.available: 100Mi
nodefs.available: 10%
nodefs.inodesFree: 5%
maxOpenFiles: 1000000 
maxPods: 110
EOF

8.3.3systemd管理kubelet组件

cat > /usr/lib/systemd/system/kubelet.service << EOF
[Unit]
Description=Kubernetes Kubelet
After=docker.service
Requires=docker.service

[Service]
EnvironmentFile=/opt/kubernetes/cfg/kubelet.conf
ExecStart=/opt/kubernetes/bin/kubelet \$KUBELET_OPTS
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

8.3.4启动并设置开机启动

systemctl daemon-reload 
systemctl start kubelet 
systemctl enable kubelet
systemctl status kubelet 

报错查看日志：journalctl -u kubelet --no-pager

8.4 批准 kubelet证书申请并加入集群

master节点：
在Master审批Node加入集群：
启动后还没加入到集群中，需要手动允许该节点才可以。
在Master节点查看请求签名的Node：

[root@m1 k8s]#  kubectl get csr
NAME                                                   AGE   SIGNERNAME                                    REQUESTOR           CONDITION
node-csr-LAbFoGwqhnHEeptAOdLAE8XmmJJ4VESdH6rVXAbyy6c   65s   kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   Pending

[root@m1 k8s]# kubectl certificate approve node-csr-LAbFoGwqhnHEeptAOdLAE8XmmJJ4VESdH6rVXAbyy6c
certificatesigningrequest.certificates.k8s.io/node-csr-LAbFoGwqhnHEeptAOdLAE8XmmJJ4VESdH6rVXAbyy6c approved

[root@m1 k8s]# kubectl get nodes
NAME     STATUS     ROLES    AGE   VERSION
k8s-m1   NotReady   <none>   25s   v1.18.3

注：由于网络插件还没有部署，节点会没有准备就绪 NotReady

8.5 部署 kube-proxy

8.5.1创建配置文件

cat > /opt/kubernetes/cfg/kube-proxy.conf << EOF
KUBE_PROXY_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/opt/kubernetes/logs \
--config=/opt/kubernetes/cfg/kube-proxy-config.yml" 
EOF

8.5.2配置参数文件

cat > /opt/kubernetes/cfg/kube-proxy-config.yml << EOF 
kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1 
bindAddress: 0.0.0.0
metricsBindAddress: 0.0.0.0:10249
clientConnection:
  kubeconfig: /opt/kubernetes/cfg/kube-proxy.kubeconfig 
hostnameOverride: k8s-m1
clusterCIDR: 10.0.0.0/24
EOF

8.5.3systemd 管理 kube-proxy

cat > /usr/lib/systemd/system/kube-proxy.service << EOF
[Unit]
Description=Kubernetes Proxy
After=network.target
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-proxy.conf 
ExecStart=/opt/kubernetes/bin/kube-proxy \$KUBE_PROXY_OPTS 
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF

8.5.4启动并设置开机启动

systemctl daemon-reload 
systemctl start kube-proxy 
systemctl enable kube-proxy
systemctl status kube-proxy

8.6 部署 CNI网络

wget https://github.com/containernetworking/plugins/releases/download/v0.8.6/cni-plugins-linux-amd64-v0.8.6.tgz
mkdir /opt/cni/bin -p
tar zxvf cni-plugins-linux-amd64-v0.8.6.tgz -C /opt/cni/bin

#部署 CNI 网络，安装flannel插件
wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

默认镜像地址无法访问，修改为 docker hub 镜像仓库。
sed -i -r "s#quay.io/coreos/flannel:.*-amd64#lizhenliang/flannel:v0.12.0-amd64#g" kube-flannel.yml

#部署拉去镜像需要一些时间
kubectl apply -f kube-flannel.yml
kubectl get pods -n kube-flannel
kubectl get node

8.7新增加 Worker Node

8.7.1拷贝已部署好的 Node 相关文件到新节点

在 master 节点将 Worker Node 涉及文件拷贝到新节点

# node1
[root@m1 ~]# scp -r /opt/kubernetes root@n1:/opt/
[root@m1 ~]# scp -r /usr/lib/systemd/system/{kubelet,kube-proxy}.service root@n1:/usr/lib/systemd/system
[root@m1 ~]# scp -r /opt/cni/ root@n1:/opt/
[root@m1 ~]# scp /opt/kubernetes/ssl/ca.pem root@n1:/opt/kubernetes/ssl

# node2
[root@m1 ~]# scp -r /opt/kubernetes root@n2:/opt/
[root@m1 ~]# scp -r /usr/lib/systemd/system/{kubelet,kube-proxy}.service root@n2:/usr/lib/systemd/system
[root@m1 ~]# scp -r /opt/cni/ root@n2:/opt/
[root@m1 ~]# scp /opt/kubernetes/ssl/ca.pem root@2:/opt/kubernetes/ssl

8.7.2删除 kubelet 证书和 kubeconfig 文件（新加的worker node）

rm -rf /opt/kubernetes/cfg/kubelet.kubeconfig
rm -rf /opt/kubernetes/ssl/kubelet*

#注：这几个文件是证书申请审批后自动生成的，每个 Node不同，必须删除重新生成。

8.7.3修改主机名

vim /opt/kubernetes/cfg/kubelet.conf
--hostname-override=k8s-node1

vim /opt/kubernetes/cfg/kube-proxy-config.yml
hostnameOverride: k8s-node1

8.7.4启动并设置开机启动

systemctl daemon-reload
systemctl start kubelet
systemctl enable kubelet
systemctl start kube-proxy
systemctl enable kube-proxy
systemctl status kubelet
systemctl status kube-proxy

8.7.5在 Master 上批准新 Node kubelet 证书申请

1 2	`kubectl get csr kubectl certificate approve 生成的name`

8.7.6查看 Node 状态

1	`kubectl get node`

Linux

二进制部署kubernetes集群

本博客所有文章除特别声明外，均采用 CC BY-SA 4.0 协议，转载请注明来自NoteBook！

s3fs挂载对象存储到本地上一篇

Centos8网卡配置下一篇