搜索到
199
篇与
cby
的结果
-
k8s加入新的master节点出现etcd检查失败 背景： 昨天在建立好新的集群后，出现了新的问题，其中的一台master节点无法正常工作。虽然可以正常使用，但是就出现了单点故障，今天在修复时出现了etcd健康检查自检没通过。Yesterday, after a new cluster was established, a new problem a problem occurred, and one of the master nodes did not work properly. Although can be used normally, but there is a single point of failure, today in the repair of the etcd health check self-test failed.对加入集群中时，出现如下报错：When you join a cluster, the following error occurs 提示 etcd 监控检查失败，查看一下Kubernetes 集群中的 kubeadm 配置信息。Prompt the etcd monitoring check to fail and review the kubeadm configuration information in the Kubernetes cluster.*\[root@master-01 ~\]# kubectl describe configmaps kubeadm-config -n kube-system ---- apiEndpoints: master-01: advertiseAddress: 10.0.0.11 bindPort: 6443 master-02: advertiseAddress: 10.0.0.12 bindPort: 6443 master-03: advertiseAddress: 10.0.0.13 bindPort: 6443 apiVersion: kubeadm.k8s.io/v1beta2 kind: ClusterStatus Events: <none> 因为集群搭建的时候，etcd是镜像的方式，在master02上面出现问题后，进行剔除完成后，etcd还是在存储在每个master上面，所以重新添加的时候会得知健康检查失败。Because when the cluster is built, etcd is mirrored, after the problem on master02, after the cull is completed, etcd is still stored on top of each master, so when you add again, you will learn that the health check failed.*这时就需要进入容器内部进行手动删除这个etcd了，首先获取集群中的etcd pod列表看一下，并进入内部给一个sh窗口。 At this point you need to go inside the container to manually delete this etcd, first get the list of etcd pods in the cluster to see, and go inside to give a sh window\[root@master-01 ~\]# kubectl get pods -n kube-system | grep etcd \[root@master-01 ~\]# kubectl exec -it etcd-master-03 sh -n kube-system 进入容器后，执行如下操作： After entering the container, do the following\## 配置环境 $ export ETCDCTL\_API=3 $ alias etcdctl='etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key' ## 查看 etcd 集群成员列表 $ etcdctl member list ## 删除 etcd 集群成员 master-02 $ etcdctl member remove ## 再次查看 etcd 集群成员列表 $ etcdctl member list ## 退出容器 $ exit查看列表并删除已不存在的masterView the list and remove the master that no longer exists*再次进行加入master，即可成功。Join master again and you'll be successful**高新科技园 -
Linux内核高性能优化 Linux内核高性能优化—内核优化开始——–# 内核panic时，1秒后自动重启kernel.panic = 1# 允许更多的PIDs (减少滚动翻转问题); may break some programs 32768kernel.pid_max = 32768# 内核所允许的最大共享内存段的大小（bytes）kernel.shmmax = 4294967296# 在任何给定时刻，系统上可以使用的共享内存的总量（pages）kernel.shmall = 1073741824# 设定程序core时生成的文件名格式kernel.core_pattern = core_%e# 当发生oom时，自动转换为panicvm.panic_on_oom = 1# 表示强制Linux VM最低保留多少空闲内存（Kbytes）vm.min_free_kbytes = 1048576# 该值高于100，则将导致内核倾向于回收directory和inode cachevm.vfs_cache_pressure = 250# 表示系统进行交换行为的程度，数值（0-100）越高，越可能发生磁盘交换vm.swappiness = 20# 仅用10%做为系统cachevm.dirty_ratio = 10# 增加系统文件描述符限制 2^20-1fs.file-max = 1048575# 网络层优化# listen()的默认参数,挂起请求的最大数量，默认128net.core.somaxconn = 1024# 增加Linux自动调整TCP缓冲区限制net.core.wmem\_default = 8388608 net.core.rmem\_default = 8388608 net.core.rmem\_max = 16777216 net.core.wmem\_max = 16777216# 进入包的最大设备队列.默认是300net.core.netdev_max_backlog = 2000# 开启SYN洪水攻击保护net.ipv4.tcp_syncookies = 1# 开启并记录欺骗，源路由和重定向包net.ipv4.conf.all.log\_martians = 1 net.ipv4.conf.default.log\_martians = 1# 处理无源路由的包net.ipv4.conf.all.accept\_source\_route = 0 net.ipv4.conf.default.accept\_source\_route = 0# 开启反向路径过滤net.ipv4.conf.all.rp\_filter = 1 net.ipv4.conf.default.rp\_filter = 1# 确保无人能修改路由表net.ipv4.conf.all.accept\_redirects = 0 net.ipv4.conf.default.accept\_redirects = 0 net.ipv4.conf.all.secure\_redirects = 0 net.ipv4.conf.default.secure\_redirects = 0# 增加系统IP端口限制net.ipv4.ip_local_port_range = 9000 65533# TTLnet.ipv4.ip_default_ttl = 64# 增加TCP最大缓冲区大小net.ipv4.tcp\_rmem = 4096 87380 8388608 net.ipv4.tcp\_wmem = 4096 32768 8388608# Tcp自动窗口net.ipv4.tcp_window_scaling = 1# 进入SYN包的最大请求队列.默认1024net.ipv4.tcp_max_syn_backlog = 8192# 打开TIME-WAIT套接字重用功能，对于存在大量连接的Web服务器非常有效。 net.ipv4.tcp\_tw\_recycle = 1 net.ipv4.tcp\_tw\_reuse = 0 # 表示是否启用以一种比超时重发更精确的方法（请参阅 RFC 1323）来启用对 RTT 的计算；为了实现更好的性能应该启用这个选项net.ipv4.tcp_timestamps = 0# 表示本机向外发起TCP SYN连接超时重传的次数net.ipv4.tcp\_syn\_retries = 2 net.ipv4.tcp\_synack\_retries = 2# 减少处于FIN-WAIT-2连接状态的时间，使系统可以处理更多的连接。 net.ipv4.tcp_fin_timeout = 10 # 减少TCP KeepAlive连接侦测的时间，使系统可以处理更多的连接。 # 如果某个TCP连接在idle 300秒后,内核才发起probe.如果probe 2次(每次2秒)不成功,内核才彻底放弃,认为该连接已失效.net.ipv4.tcp\_keepalive\_time = 300 net.ipv4.tcp\_keepalive\_probes = 2 net.ipv4.tcp\_keepalive\_intvl = 2# 系统所能处理不属于任何进程的TCP sockets最大数量net.ipv4.tcp_max_orphans = 262144# 系统同时保持TIME_WAIT套接字的最大数量，如果超过这个数字，TIME_WAIT套接字将立刻被清除并打印警告信息。net.ipv4.tcp_max_tw_buckets = 20000 # arp_table的缓存限制优化net.ipv4.neigh.default.gc\_thresh1 = 128 net.ipv4.neigh.default.gc\_thresh2 = 512 net.ipv4.neigh.default.gc\_thresh3 = 4096——内核优化结束——–
-
Docker启动MySQL、MongoDB、Redis、Elasticsearch、Grafana，数据库 前言： 临时使用数据库时可以使用docker运行，这样可以防止在系统上安装破坏环境，同时使用docker启动会比在系统中安装配置要快速，可以说是最快的方式安装部署并启动数据库。*docker配置启动运行MySQL 首先创建目录并进入sudo docker run -p 3306:3306 \\ --name mymysql \\ --restart=always \\ -v $PWD/conf:/etc/mysql/conf.d \\ -v $PWD/logs:/logs \\ -v $PWD/data:/var/lib/mysql \\ -e MYSQL\_ROOT\_PASSWORD=123456 \\ -d mysql:8--restart=always：在容器退出时总是重启容器MYSQL_ROOT_PASSWORD=123456：root密码123456mysql:8 使用MySQL8-v $PWD/conf:/etc/mysql/conf.d 配置文件-v $PWD/logs:/logs 日志-v $PWD/data:/var/lib/mysql 数据*docker配置启动运行phpMyAdmindocker run -d \\ -p 8001:80 \\ -e UPLOAD\_LIMIT=128M \\ -e MAX\_EXECUTION\_TIME=10000 \\ --name phpmyadmin \\ phpmyadmin/phpmyadminUPLOAD_LIMIT 和 MAX_EXECUTION_TIME 需要设置一下**docker配置启动运行MongoDB*docker run -d \\ -p 27017:27017 \\ -v mongo-data:/data/db \\ -v mongo-config:/data/configdb \\ --name mongo \\ -e MONGO\_INITDB\_ROOT\_USERNAME=mongoadmin \\ -e MONGO\_INITDB\_ROOT\_PASSWORD=123123 \\ -v /data:/mnt/data \\ mongoMONGO_INITDB_ROOT_USERNAME 用户名MONGO_INITDB_ROOT_PASSWORD 密码mongo-data 数据目录mongo-config 配置文件目录**docker配置启动运行Mongo Express* docker run -d \\ -p 8002:8081 \\ --name mongo-express \\ mongo-express**docker配置启动运行Redis*docker run -d \\ -p 6379:6379 \\ -v redis-data:/data \\ --name redis \\ redis**docker配置启动运行Elasticsearch*docker run -d \\ -p 9100:9100 -p 9200:9200 \\ -e discovery.type=single-node \\ -v es-data:/usr/share/elasticsearch/data \\ -v es-log:/usr/share/elasticsearch/logs \\ --name elasticsearch \\ elasticsearch**docker配置启动运行Grafana*docker run -d \\ -p 8003:3000 \\ --link mysql:mysql \\ --link mongo:mongo \\ --name grafana \\ grafana/grafana
-
利用 kubeadm 创建 kubernetes 的高可用集群 引言：kubeadm提供了两种不同的高可用方案。 堆叠方案：etcd服务和控制平面被部署在同样的节点中，对基础设施的要求较低，对故障的应对能力也较低堆叠方案 最小三个Master（也称工作平面），因为Etcd使用RAFT算法选主，节点数量需要为2n+1个。 外置etcd方案：etcd和控制平面被分离，需要更多的硬件，也有更好的保障能力外置etcd方案一、资源环境 下面采用的是kubeadm的堆叠方案搭建k8s集群，也就是说如果3台Master宕了2台时，集群将不可用，可能收到如下错误信息"Error from server: etcdserver: request timed out"。二、系统设置（所有主机） 设置主机名hostnamectl set-hostname master-\* hostnamectl set-hostname node-\* 设置静态IP\[root@localhost ~\]# vim /etc/sysconfig/network-scripts/ifcfg-ens18 \[root@localhost ~\]# \[root@localhost ~\]# \[root@localhost ~\]# \[root@localhost ~\]# cat /etc/sysconfig/network-scripts/ifcfg-ens18 TYPE=Ethernet PROXY\_METHOD=none BROWSER\_ONLY=no BOOTPROTO=static IPADDR=10.0.0.11 NETMASK=255.0.0.0 GATEWAY=10.0.0.1 DEFROUTE=yes IPV4\_FAILURE\_FATAL=no IPV6INIT=yes IPV6\_AUTOCONF=yes IPV6\_DEFROUTE=yes IPV6\_FAILURE\_FATAL=no IPV6\_ADDR\_GEN\_MODE=stable-privacy NAME=ens18 UUID=555fe27b-19eb-4958-aca7-c9c71365432f DEVICE=ens18 ONBOOT=yes \[root@localhost ~\]# reboot 配置主机名\[root@master-01 ~\]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 10.0.0.11 master-01 10.0.0.12 master-02 10.0.0.13 master-03 10.0.0.14 node-01 10.0.0.15 master-01 10.0.0.16 master-01 安装依赖\[root@node-01 ~\]# yum update -y Repository AppStream is listed more than once in the configuration Repository extras is listed more than once in the configuration Repository PowerTools is listed more than once in the configuration Repository centosplus is listed more than once in the configuration Last metadata expiration check: 0:19:42 ago on Sat 28 Nov 2020 04:25:04 PM CST. Dependencies resolved. Nothing to do. Complete! \[root@node-01 ~\]# yum install -y conntrack ipvsadm ipset jq sysstat curl iptables libseccomp bind-utils ... 关闭防火墙、swap、selinux\[root@master-01 ~\]# systemctl stop firewalld && systemctl disable firewalld Removed /etc/systemd/system/multi-user.target.wants/firewalld.service. Removed /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service. \[root@master-01 ~\]# swapoff -a \[root@master-01 ~\]# iptables -F && iptables -X && iptables -F -t nat && iptables -X -t nat && iptables -P FORWARD ACCEPT \[root@master-01 ~\]# sed -i '/swap/s/^\\(.\*\\)$/#\\1/g' /etc/fstab \[root@master-01 ~\]# cat /etc/fstab # # /etc/fstab # Created by anaconda on Mon Nov 23 08:19:33 2020 # # Accessible filesystems, by reference, are maintained under '/dev/disk/'. # See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info. # # After editing this file, run 'systemctl daemon-reload' to update systemd # units generated from this file. # /dev/mapper/cl-root / xfs defaults 0 0 UUID=46ea6159-eda5-4931-ae11-73095cf284c1 /boot ext4 defaults 1 2 #/dev/mapper/cl-swap swap swap defaults 0 0 \[root@master-01 ~\]# setenforce 0 \[root@master-01 ~\]# vim /etc/sysconfig/selinux \[root@master-01 ~\]# cat /etc/sysconfig/selinux # This file controls the state of SELinux on the system. # SELINUX= can take one of these three values: # enforcing - SELinux security policy is enforced. # permissive - SELinux prints warnings instead of enforcing. # disabled - No SELinux policy is loaded. SELINUX=disabled # SELINUXTYPE= can take one of these three values: # targeted - Targeted processes are protected, # minimum - Modification of targeted policy. Only selected processes are protected. # mls - Multi Level Security protection. SELINUXTYPE=targeted 高新科技园广东省深圳市南山区科文路4附近 系统参数设置\# 开启ipvs模块 \[root@master-01 ~\]# cat > /etc/sysconfig/modules/ipvs.modules <<EOF > #!/bin/bash > modprobe -- ip\_vs > modprobe -- ip\_vs\_rr > modprobe -- ip\_vs\_wrr > modprobe -- ip\_vs\_sh > modprobe -- nf\_conntrack\_ipv4 > modprobe br\_netfilter > EOF # 生效文件 \[root@master-01 ~\]# chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip\_vs -e nf\_conntrack\_ipv4 ip\_vs\_sh 16384 0 ip\_vs\_wrr 16384 0 ip\_vs\_rr 16384 0 ip\_vs 172032 6 ip\_vs\_rr,ip\_vs\_sh,ip\_vs\_wrr nf\_defrag\_ipv6 20480 2 nf\_conntrack\_ipv6,ip\_vs nf\_conntrack\_ipv4 16384 1 nf\_defrag\_ipv4 16384 1 nf\_conntrack\_ipv4 nf\_conntrack 155648 7 nf\_conntrack\_ipv6,nf\_conntrack\_ipv4,nf\_nat,nft\_ct,nf\_nat\_ipv6,nf\_nat\_ipv4,ip\_vs libcrc32c 16384 4 nf\_conntrack,nf\_nat,xfs,ip\_vs # 制作配置文件 \[root@master-01 ~\]# cat > /etc/sysctl.d/kubernetes.conf <<EOF > net.bridge.bridge-nf-call-iptables=1 > net.bridge.bridge-nf-call-ip6tables=1 > net.ipv4.ip\_forward=1 > net.ipv4.tcp\_tw\_recycle=1 # 表示开启TCP连接中TIME-WAIT sockets的快速回收，默认为0，表示关闭。 > net.ipv4.tcp\_keepalive\_time=600 # 超过这个时间没有数据传输，就开始发送存活探测包 > net.ipv4.tcp\_keepalive\_intvl=15 # keepalive探测包的发送间隔 > net.ipv4.tcp\_keepalive\_probes=3 # 如果对方不予应答，探测包的发送次数 > vm.swappiness=0 # 禁止使用 swap 空间，只有当系统 OOM 时才允许使用它 > vm.overcommit\_memory=1 # 不检查物理内存是否够用 > vm.panic\_on\_oom=0 # 开启 OOM > fs.inotify.max\_user\_instances=8192 > fs.inotify.max\_user\_watches=1048576 > fs.file-max=52706963 > fs.nr\_open=52706963 > net.ipv6.conf.all.disable\_ipv6=1 > net.netfilter.nf\_conntrack\_max=2310720 > EOF # 生效配置文件 \[root@master-01 ~\]# sysctl -p /etc/sysctl.d/kubernetes.conf net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-call-ip6tables = 1 net.ipv4.ip\_forward = 1 sysctl: cannot stat /proc/sys/net/ipv4/tcp\_tw\_recycle: No such file or directory net.ipv4.tcp\_keepalive\_time = 600 # 超过这个时间没有数据传输，就开始发送存活探测包 net.ipv4.tcp\_keepalive\_intvl = 15 # keepalive探测包的发送间隔 net.ipv4.tcp\_keepalive\_probes = 3 # 如果对方不予应答，探测包的发送次数 vm.swappiness = 0 # 禁止使用 swap 空间，只有当系统 OOM 时才允许使用它 vm.overcommit\_memory = 1 # 不检查物理内存是否够用 vm.panic\_on\_oom = 0 # 开启 OOM fs.inotify.max\_user\_instances = 8192 fs.inotify.max\_user\_watches = 1048576 fs.file-max = 52706963 fs.nr\_open = 52706963 net.ipv6.conf.all.disable\_ipv6 = 1 net.netfilter.nf\_conntrack\_max = 2310720 # 调整系统 TimeZone \[root@master-01 ~\]# timedatectl set-timezone Asia/Shanghai # 将当前的 UTC 时间写入硬件时钟 \[root@master-01 ~\]# timedatectl set-local-rtc 0 # 重启依赖于系统时间的服务 \[root@master-01 ~\]# systemctl restart rsyslog && systemctl restart crond # 关闭无关的服务 \[root@master-01 ~\]# systemctl stop postfix && systemctl disable postfix Failed to stop postfix.service: Unit postfix.service not loaded. # 设置 rsyslogd 和 systemd journald \[root@master-01 ~\]# mkdir /var/log/journal \[root@master-01 ~\]# mkdir /etc/systemd/journald.conf.d \[root@master-01 ~\]# cat > /etc/systemd/journald.conf.d/99-prophet.conf <<EOF > \[Journal\] > # 持久化保存到磁盘 > Storage=persistent > > # 压缩历史日志 > Compress=yes > > SyncIntervalSec=5m > RateLimitInterval=30s > RateLimitBurst=1000 > > # 最大占用空间 10G > SystemMaxUse=10G > > # 单日志文件最大 200M > SystemMaxFileSize=200M > > # 日志保存时间 2 周 > MaxRetentionSec=2week > > # 不将日志转发到 syslog > ForwardToSyslog=no > EOF \[root@master-01 ~\]# **三、安装docker **\[root@master-02 ~\]# wget https://download.docker.com/linux/centos/8/x86\_64/stable/Packages/containerd.io-1.3.7-3.1.el8.x86\_64.rpm --2020-11-28 17:47:12-- https://download.docker.com/linux/centos/8/x86\_64/stable/Packages/containerd.io-1.3.7-3.1.el8.x86\_64.rpm Resolving download.docker.com (download.docker.com)... 99.84.206.7, 99.84.206.109, 99.84.206.25, ... Connecting to download.docker.com (download.docker.com)|99.84.206.7|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 30388860 (29M) \[binary/octet-stream\] Saving to: ‘containerd.io-1.3.7-3.1.el8.x86\_64.rpm’ containerd.io-1.3.7-3.1 100%\[===============================>\] 28.98M 188KB/s in 3m 15s 2020-11-28 17:50:27 (153 KB/s) - ‘containerd.io-1.3.7-3.1.el8.x86\_64.rpm’ saved \[30388860/30388860\] \[root@node-02 ~\]# yum install ./containerd.io-1.3.7-3.1.el8.x86\_64.rpm Repository AppStream is listed more than once in the configuration Repository extras is listed more than once in the configuration ... \[root@node-01 ~\]# sudo yum -y install docker-ce Repository AppStream is listed more than once in the configuration Repository extras is listed more than once in the configuration Repository PowerTools is listed more than once in the configuration ... \[root@master-01 ~\]# systemctl start docker && systemctl enable docker四、安装必要工具，在主节点安装kubectl即可，其他节点无需进行安装kubectl**\[root@master-01 ~\]# cat <<EOF > /etc/yum.repos.d/kubernetes.repo \[kubernetes\] name=Kubernetes baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86\_64/ enabled=1 gpgcheck=1 repo\_gpgcheck=1 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg EOF \[root@master-01 ~\]# yum install -y kubelet kubeadm kubectl \[root@master-01 ~\]# systemctl enable kubelet && systemctl start kubelet五、安装LVS和keepalived\[root@vip ~\]# yum -y install keepalived # 备份并编辑 \[root@vip ~\]# cp /etc/keepalived/keepalived.conf{,.back} \[root@vip ~\]# vim /etc/keepalived/keepalived.conf \[root@vip ~\]# echo "" > /etc/keepalived/keepalived.conf \[root@vip ~\]# vim /etc/keepalived/keepalived.conf \[root@vip ~\]# systemctl enable keepalived && service keepalived start Created symlink /etc/systemd/system/multi-user.target.wants/keepalived.service → /usr/lib/systemd/system/keepalived.service. Redirecting to /bin/systemctl start keepalived.service \[root@vip ~\]# \[root@vip ~\]# \[root@vip ~\]# cat /etc/keepalived/keepalived.conf ! Configuration File for keepalived global\_defs { router\_id keepalived-master } vrrp\_instance vip\_1 { state MASTER ! 注意这是网卡名称，使用ip a命令查看自己的局域网网卡名称 interface ens18 ! keepalived主备router\_id必须一致 virtual\_router\_id 88 ! 优先级，keepalived主节点优先级要比备节点高 priority 100 advert\_int 3 ! 配置虚拟ip地址 virtual\_ipaddress { 10.0.0.99 } } virtual\_server 10.0.0.99 6443 { delay\_loop 6 lb\_algo rr lb\_kind DR persistence\_timeout 0 protocol TCP real\_server 10.0.0.12 6443 { weight 1 TCP\_CHECK { connect\_timeout 10 nb\_get\_retry 3 delay\_before\_retry 3 connect\_port 6443 } } real\_server 10.0.0.13 6443 { weight 1 TCP\_CHECK { connect\_timeout 10 nb\_get\_retry 3 delay\_before\_retry 3 connect\_port 6443 } } real\_server 10.0.0.11 6443 { weight 1 TCP\_CHECK { connect\_timeout 10 nb\_get\_retry 3 delay\_before\_retry 3 connect\_port 6443 } } } 添加本地回环\[root@master-01 rs\]# vim /opt/rs/rs.sh \[root@master-01 rs\]# cat /opt/rs/rs.sh #!/bin/bash # 虚拟ip vip=10.0.0.99 # 停止以前的lo:0 ifconfig lo:0 down echo "1" > /proc/sys/net/ipv4/ip\_forward echo "0" > /proc/sys/net/ipv4/conf/all/arp\_announce # 启动一个回环地址并绑定给vip ifconfig lo:0 $vip broadcast $vip netmask 255.0.0.0 up route add -host $vip dev lo:0 echo "1" >/proc/sys/net/ipv4/conf/lo/arp\_ignore echo "2" >/proc/sys/net/ipv4/conf/lo/arp\_announce echo "1" >/proc/sys/net/ipv4/conf/all/arp\_ignore echo "2" >/proc/sys/net/ipv4/conf/all/arp\_announce # ens33是主网卡名 echo "1" >/proc/sys/net/ipv4/conf/ens18/arp\_ignore echo "2" >/proc/sys/net/ipv4/conf/ens18/arp\_announce # 脚本不可以的话，使用命令吧 ifconfig lo:0 10.0.0.99 broadcast 10.0.0.99 netmask 255.255.255.255 up route add -host 10.0.0.99 dev lo:0 # 设置开机自启 \[root@vip ~\]# echo '/opt/rs/rs.sh' >> /etc/rc.d/rc.local \[root@vip ~\]# chmod +x /etc/rc.d/rc.local keepalived backup 设置\[root@vip ~\]# \[root@vip ~\]# vim /etc/keepalived/keepalived.conf \[root@vip ~\]# systemctl enable keepalived && service keepalived start Created symlink /etc/systemd/system/multi-user.target.wants/keepalived.service → /usr/lib/systemd/system/keepalived.service. Redirecting to /bin/systemctl start keepalived.service \[root@vip ~\]# \[root@vip ~\]# \[root@vip ~\]# cat /etc/keepalived/keepalived.conf ! Configuration File for keepalived global\_defs { router\_id keepalived-master } vrrp\_instance vip\_1 { state BACKUP ! 注意这是网卡名称，使用ip a命令查看自己的局域网网卡名称 interface ens18 ! keepalived主备router\_id必须一致 virtual\_router\_id 88 ! 优先级，keepalived主节点优先级要比备节点高 priority 99 advert\_int 3 ! 配置虚拟ip地址 virtual\_ipaddress { 10.0.0.99 } } virtual\_server 10.0.0.99 6443 { delay\_loop 6 lb\_algo rr lb\_kind DR persistence\_timeout 0 protocol TCP real\_server 10.0.0.12 6443 { weight 1 TCP\_CHECK { connect\_timeout 10 nb\_get\_retry 3 delay\_before\_retry 3 connect\_port 6443 } } real\_server 10.0.0.13 6443 { weight 1 TCP\_CHECK { connect\_timeout 10 nb\_get\_retry 3 delay\_before\_retry 3 connect\_port 6443 } } real\_server 10.0.0.11 6443 { weight 1 TCP\_CHECK { connect\_timeout 10 nb\_get\_retry 3 delay\_before\_retry 3 connect\_port 6443 } } } 六、kubeadm搭建集群（区分节点） master-01\[root@master-01 ~\]# cd /opt/kubernetes/ \[root@master-01 kubernetes\]# \[root@master-01 kubernetes\]# cat kubeadm-config.yaml apiVersion: kubeadm.k8s.io/v1beta2 kind: ClusterConfiguration # k8s的版本号，必须跟安装的Kubeadm版本等保持一致，否则启动报错 kubernetesVersion: v1.19.4 # docker镜像仓库地址，k8s.gcr.io需要翻墙才可以下载镜像，这里使用镜像服务器下载http://mirror.azure.cn/help/gcr-proxy-cache.html # imageRepository: k8s.gcr.io/google\_containers # 集群名称 clusterName: kubernetes # apiServer的集群访问地址，填写vip地址即可 # controlPlaneEndpoint: "10.0.0.99:6443" networking: # pod的网段 podSubnet: 10.10.0.0/16 serviceSubnet: 10.96.0.0/12 dnsDomain: cluster.local --- apiVersion: kubeproxy.config.k8s.io/v1alpha1 kind: KubeProxyConfiguration # kube-proxy模式指定为ipvs，需要提前在节点上安装ipvs的依赖并开启相关模块 mode: ipvs # 拉去镜像 \[root@master-01 kubernetes\]# kubeadm config images pull W1128 20:33:21.822265 4536 configset.go:348\] WARNING: kubeadm cannot validate component configs for API groups \[kubelet.config.k8s.io kubeproxy.config.k8s.io\] \[config/images\] Pulled k8s.gcr.io/kube-apiserver:v1.19.4 \[config/images\] Pulled k8s.gcr.io/kube-controller-manager:v1.19.4 \[config/images\] Pulled k8s.gcr.io/kube-scheduler:v1.19.4 \[config/images\] Pulled k8s.gcr.io/kube-proxy:v1.19.4 \[config/images\] Pulled k8s.gcr.io/pause:3.2 \[config/images\] Pulled k8s.gcr.io/etcd:3.4.13-0 # 记得： \[root@master-01 kubernetes\]# swapoff -a && kubeadm reset && systemctl daemon-reload && systemctl restart kubelet && iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X # 初始化 \[root@master-01 kubernetes\]# kubeadm init --config=kubeadm-config.yaml --upload-certs ... To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config You should now deploy a pod network to the cluster. Run "kubectl apply -f \[podnetwork\].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ You can now join any number of the control-plane node running the following command on each as root: kubeadm join 10.0.0.99:6443 --token dtkoyq.8ciqez70nj1ysdix \\ --discovery-token-ca-cert-hash sha256:f65ee972a9e9d0b8784f7db583a9cdf9865253459aa96a9b3529be2517570155 \\ --control-plane --certificate-key 0dc20030f8dfdede8cbb3b0906eda1a3a140e91f7e6ebb6eac1ad02ac65389d3 Please note that the certificate-key gives access to cluster sensitive data, keep it secret! As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use "kubeadm init phase upload-certs --upload-certs" to reload certs afterward. Then you can join any number of worker nodes by running the following on each as root: kubeadm join 10.0.0.99:6443 --token dtkoyq.8ciqez70nj1ysdix \\ --discovery-token-ca-cert-hash sha256:f65ee972a9e9d0b8784f7db583a9cdf9865253459aa96a9b3529be2517570155 # 安装网络组件 \[root@master-01 kubernetes\]# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml podsecuritypolicy.policy/psp.flannel.unprivileged created clusterrole.rbac.authorization.k8s.io/flannel created clusterrolebinding.rbac.authorization.k8s.io/flannel created serviceaccount/flannel created configmap/kube-flannel-cfg created daemonset.apps/kube-flannel-ds created \[root@master-01 kubernetes\]# \[root@master-01 kubernetes\]# \[root@master-01 kubernetes\]# \[root@master-01 kubernetes\]# kubectl get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE kube-system coredns-f9fd979d6-2hs76 0/1 Pending 0 5m18s kube-system coredns-f9fd979d6-5j4w8 0/1 Pending 0 5m18s kube-system etcd-master-01 1/1 Running 0 5m29s kube-system kube-apiserver-master-01 1/1 Running 0 5m30s kube-system kube-controller-manager-master-01 1/1 Running 0 5m30s kube-system kube-flannel-ds-grhh6 0/1 Init:0/1 0 5s kube-system kube-proxy-pl74w 1/1 Running 0 5m18s kube-system kube-scheduler-master-01 1/1 Running 0 5m30s 配置master-02和03# 加入master组 \[root@master-03 ~\]# kubeadm join 10.0.0.99:6443 --token dtkoyq.8ciqez70nj1ysdix --discovery-token-ca-cert-hash sha256:f65ee972a9e9d0b8784f7db583a9cdf9865253459aa96a9b3529be2517570155 --control-plane --certificate-key 0dc20030f8dfdede8cbb3b0906eda1a3a140e91f7e6ebb6eac1ad02ac65389d3 ... 等到pull镜像比较慢，耐心等待一下 ... # 加入完成后 \[root@master-03 ~\]# mkdir -p $HOME/.kube \[root@master-03 ~\]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config \[root@master-03 ~\]# sudo chown $(id -u):$(id -g) $HOME/.kube/config \[root@master-01 kubernetes\]# kubectl get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE kube-system coredns-f9fd979d6-2hs76 1/1 Running 0 45m kube-system coredns-f9fd979d6-5j4w8 1/1 Running 0 45m kube-system etcd-master-01 1/1 Running 0 45m kube-system etcd-master-02 1/1 Running 0 17m kube-system etcd-master-03 0/1 Running 0 49s kube-system kube-apiserver-master-01 1/1 Running 0 45m kube-system kube-apiserver-master-02 1/1 Running 0 17m kube-system kube-apiserver-master-03 1/1 Running 0 51s kube-system kube-controller-manager-master-01 1/1 Running 1 45m kube-system kube-controller-manager-master-02 1/1 Running 0 17m kube-system kube-controller-manager-master-03 0/1 Running 0 51s kube-system kube-flannel-ds-76vcb 0/1 Init:0/1 0 17s kube-system kube-flannel-ds-8tqlh 1/1 Running 0 17m kube-system kube-flannel-ds-fq8kz 0/1 Init:0/1 0 17s kube-system kube-flannel-ds-grhh6 1/1 Running 0 40m kube-system kube-flannel-ds-hqj25 1/1 Running 0 52s kube-system kube-flannel-ds-rlg4z 0/1 Init:0/1 0 17s kube-system kube-proxy-8kf2r 1/1 Running 0 17m kube-system kube-proxy-9n6p4 0/1 ContainerCreating 0 17s kube-system kube-proxy-9xdrl 1/1 Running 0 52s kube-system kube-proxy-pl74w 1/1 Running 0 45m kube-system kube-proxy-vtm97 0/1 ContainerCreating 0 17s kube-system kube-proxy-wdrpx 0/1 ContainerCreating 0 17s kube-system kube-scheduler-master-01 1/1 Running 1 45m kube-system kube-scheduler-master-02 1/1 Running 0 17m kube-system kube-scheduler-master-03 0/1 Running 0 51snode节点进行加入\[root@node-01 ~\]# kubeadm join 10.0.0.99:6443 --token dtkoyq.8ciqez70nj1ysdix \\ > --discovery-token-ca-cert-hash sha256:f65ee972a9e9d0b8784f7db583a9cdf9865253459aa96a9b3529be2517570155 \[root@node-01 ~\]# mkdir -p $HOME/.kube \[root@node-01 ~\]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config \[root@node-01 ~\]# sudo chown $(id -u):$(id -g) $HOME/.kube/config \[root@master-01 kubernetes\]# kubectl get nodes NAME STATUS ROLES AGE VERSION master-01 Ready master 46m v1.19.4 master-02 Ready master 18m v1.19.4 master-03 Ready master 107s v1.19.4 node-01 Ready <none> 72s v1.19.4 node-02 Ready <none> 72s v1.19.4 node-03 Ready <none> 72s v1.19.4 至此，高可用集群已部署完毕。七、部署Dashboard管理k8s集群\[root@master-01 ~\]# wget -P /etc/kubernetes/addons https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-rc1/aio/deploy/recommended.yaml && cd /etc/kubernetes/addons \[root@master-01 addons\]# kubectl apply -f recommended.yaml namespace/kubernetes-dashboard created serviceaccount/kubernetes-dashboard created service/kubernetes-dashboard created secret/kubernetes-dashboard-certs created secret/kubernetes-dashboard-csrf created secret/kubernetes-dashboard-key-holder created configmap/kubernetes-dashboard-settings created role.rbac.authorization.k8s.io/kubernetes-dashboard created clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created deployment.apps/kubernetes-dashboard created service/dashboard-metrics-scraper created deployment.apps/dashboard-metrics-scraper created 部署管理员\[root@master-01 addons\]# cat <<E0F > dashboard-adminuser.yaml > apiVersion: v1 > kind: ServiceAccount > metadata: > name: admin-user > namespace: kubernetes-dashboard > > --- > > apiVersion: rbac.authorization.k8s.io/v1 > kind: ClusterRoleBinding > metadata: > name: admin-user > roleRef: > apiGroup: rbac.authorization.k8s.io > kind: ClusterRole > name: cluster-admin > subjects: > - kind: ServiceAccount > name: admin-user > namespace: kubernetes-dashboard > E0F \[root@master-01 addons\]# kubectl apply -f dashboard-adminuser.yaml serviceaccount/admin-user created clusterrolebinding.rbac.authorization.k8s.io/admin-user created 查看token \[root@master-01 addons\]# kubectl -n kubernetes-dashboard describe secret $(kubectl -n kubernetes-dashboard get secret | grep admin-user | awk '{print $1}') | grep -E '^token' | awk '{print $2}' 高新科技园广东省深圳市南山区科文路4附近