3

关于Kubernetes中pod调度[选择器,指定节点,主机亲和性]和节点[coedon,drain,taint]标...

 2 years ago
source link: https://liruilongs.github.io/2022/05/29/K8s/%E5%85%B3%E4%BA%8E%20Kubernetes%E4%B8%ADpod%E8%B0%83%E5%BA%A6%5B%E9%80%89%E6%8B%A9%E5%99%A8%E3%80%81%E6%8C%87%E5%AE%9A%E8%8A%82%E7%82%B9%E3%80%81%E4%B8%BB%E6%9C%BA%E4%BA%B2%E5%92%8C%E6%80%A7%5D%E5%92%8C%E8%8A%82%E7%82%B9%5Bcoedon%E3%80%81drain%E3%80%81taint%5D%E6%A0%87%E8%AE%B0%E7%9A%84%E4%B8%80%E4%BA%9B%E7%AC%94%E8%AE%B0-md/
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.

关于Kubernetes中pod调度[选择器,指定节点,主机亲和性]和节点[coedon,drain,taint]标记的Demo.md

傍晚时分，你坐在屋檐下，看着天慢慢地黑下去，心里寂寞而凄凉，感到自己的生命被剥夺了。当时我是个年轻人，但我害怕这样生活下去，衰老下去。在我看来，这是比死亡更可怕的事。——–王小波

  • 嗯，整理K8s中pod调度相关笔记，这里分享给小伙伴
  • 博文内容涉及：
    • kube-scheduler组件的简述
    • Pod的调度(选择器、指定节点、主机亲和性)方式
    • 节点的coedon与drain标记
    • 节点的taint(污点)标记及pod的容忍污点(tolerations)定义
  • 食用方式：
    • 需要了解K8s基础知识
    • 熟悉资源对象pod，deploy的创建，了解资源对象定义yaml文件
    • 了解kubectl常用命令
  • 理解不足小伙伴帮忙指正

傍晚时分，你坐在屋檐下，看着天慢慢地黑下去，心里寂寞而凄凉，感到自己的生命被剥夺了。当时我是个年轻人，但我害怕这样生活下去，衰老下去。在我看来，这是比死亡更可怕的事。——–王小波

Pod调度

调度的三个对象

待调度Pod列表:有多少个pod需要调度，即创建的pod列表

可用node列表:有那些节点可以参与调度，排除有污点，端口的一些node

调度算法

  • 主机过滤
    + `NoDiskConflict`
    + `PodFitsResources`
    + `PodFitsPorts`
    + `MatchNodeSelector`
    + `HostName`
    + `NoVolumeZoneConflict`
    + `PodToleratesNodeTaints`
    + `CheckNodeMemoryPressure`
    + `CheckNodeDiskPressure`
    + `MaxEBSVolumeCount`
    + `MaxGCEPDVolumeCount`
    + `MaxAzureDiskVolumeCount`
    + `MatchInterPodAffinity`
    + `GeneralPredicates`
    + `NodeVolumeNodeConflic`
  • 主机打分
分数项 公式
LeastRequestedPriority score=cpu ( ( capacity - sum ( requested ) ) * 10 / capacity) + memory ( ( capacity - sum ( requested) ) * 10 / capacity )/2
BalanceResourceAllocation score = 10 -abs ( cpuFraction - memoryFraction ) * 10
CalculateSpreadPriority Score = 10 * ((maxCount -counts)/ (maxCount))

手动指定pod的运行位置:

可以给node设置指定的标签，然后我们可以在创建pod里指定node标签

标签设置 –
查看 kubectl get nodes –show-labels
设置 kubectl label node node2 disktype=ssd
取消 kubectl label node node2 disktype
所有节点设置 kubectl label node all key=vale

查看节点pod：kubectl get node --show-labels

给节点设置标签

┌──[[email protected]]-[~/ansible]
└─$kubectl label node vms82.liruilongs.github.io disktype=node1
node/vms82.liruilongs.github.io labeled
┌──[[email protected]]-[~/ansible]
└─$kubectl label node vms83.liruilongs.github.io disktype=node2
node/vms83.liruilongs.github.io labeled
┌──[[email protected]]-[~/ansible]
└─$kubectl get node --show-labels
NAME                         STATUS   ROLES                  AGE   VERSION   LABELS
vms81.liruilongs.github.io   Ready    control-plane,master   45d   v1.22.2   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=vms81.liruilongs.github.io,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=,node-role.kubernetes.io/master=,node.kubernetes.io/exclude-from-external-load-balancers=
vms82.liruilongs.github.io   Ready    <none>                 45d   v1.22.2   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,disktype=node1,kubernetes.io/arch=amd64,kubernetes.io/hostname=vms82.liruilongs.github.io,kubernetes.io/os=linux
vms83.liruilongs.github.io   Ready    <none>                 45d   v1.22.2   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,disktype=node2,kubernetes.io/arch=amd64,kubernetes.io/hostname=vms83.liruilongs.github.io,kubernetes.io/os=linux
┌──[[email protected]]-[~/ansible]
└─$

特殊的内置标签node-role.kubernetes.io/control-plane=,node-role.kubernetes.io/master=,用于设置角色列roles

┌──[[email protected]]-[~/ansible]
└─$kubectl get node
NAME                         STATUS   ROLES                  AGE   VERSION
vms81.liruilongs.github.io   Ready    control-plane,master   45d   v1.22.2
vms82.liruilongs.github.io   Ready    <none>                 45d   v1.22.2
vms83.liruilongs.github.io   Ready    <none>                 45d   v1.22.2

我们也可以做worker节点上设置

┌──[[email protected]]-[~/ansible]
└─$kubectl label nodes vms82.liruilongs.github.io node-role.kubernetes.io/worker1=
node/vms82.liruilongs.github.io labeled
┌──[[email protected]]-[~/ansible]
└─$kubectl label nodes vms83.liruilongs.github.io node-role.kubernetes.io/worker2=
node/vms83.liruilongs.github.io labeled
┌──[[email protected]]-[~/ansible]
└─$kubectl get node
NAME                         STATUS   ROLES                  AGE   VERSION
vms81.liruilongs.github.io   Ready    control-plane,master   45d   v1.22.2
vms82.liruilongs.github.io   Ready    worker1                45d   v1.22.2
vms83.liruilongs.github.io   Ready    worker2                45d   v1.22.2
┌──[[email protected]]-[~/ansible]
└─$

选择器(nodeSelector)方式

在特定节点上运行pod

┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$kubectl get nodes -l disktype=node2
NAME                         STATUS   ROLES     AGE   VERSION
vms83.liruilongs.github.io   Ready    worker2   45d   v1.22.2
┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$vim pod-node2.yaml
┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$kubectl apply  -f pod-node2.yaml
pod/podnode2 created
┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$kubectl get pods -owide
NAME       READY   STATUS    RESTARTS   AGE   IP               NODE                         NOMINATED NODE   READINESS GATES
podnode2   1/1     Running   0          13m   10.244.70.60     vms83.liruilongs.github.io   <none>           <none>

pod-node2.yaml

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: podnode2
  name: podnode2
spec:
  nodeSelector:
    disktype: node2
  containers:
  - image: nginx
    imagePullPolicy: IfNotPresent
    name: podnode2
    resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

指定节点名称(nodeName)的方式

┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$vim pod-node1.yaml
┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$kubectl apply -f pod-node1.yaml
pod/podnode1 created
┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$kubectl get pods -owide
NAME       READY   STATUS    RESTARTS   AGE   IP               NODE                         NOMINATED NODE   READINESS GATES
podnode1   1/1     Running   0          36s   10.244.171.165   vms82.liruilongs.github.io   <none>           <none>
podnode2   1/1     Running   0          13m   10.244.70.60     vms83.liruilongs.github.io   <none>           <none>

pod-node1.yaml

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: podnode1
  name: podnode1
spec:
  nodeName: vms82.liruilongs.github.io
  containers:
  - image: nginx
    imagePullPolicy: IfNotPresent
    name: podnode1
    resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

当pod资源文件指定的节点标签,或者节点名不存在时，这个pod资源是无法创建成功的

主机亲和性

所谓主机亲和性，即在满足指定条件的节点上运行。分为硬策略(必须满足)，软策略(最好满足)

硬策略(requiredDuringSchedulingIgnoredDuringExecution)

pod-node-a.yaml

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: podnodea
  name: podnodea
spec:
  containers:
  - image: nginx
    imagePullPolicy: IfNotPresent
    name: podnodea
    resources: {}
  affinity:
    nodeAffinity: #主机亲和性
      requiredDuringSchedulingIgnoredDuringExecution: #硬策略
        nodeSelectorTerms:
        - matchExpressions:
          - key: kubernetes.io/hostname
            operator: In
            values:
            - vms85.liruilongs.github.io
            - vms84.liruilongs.github.io
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

条件不满足，所以 Pending

┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$kubectl apply -f pod-node-a.yaml
pod/podnodea created
┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$kubectl get pods
NAME       READY   STATUS    RESTARTS   AGE
podnodea   0/1     Pending   0          8s

我梦修改一下

┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$sed -i  's/vms84.liruilongs.github.io/vms83.liruilongs.github.io/' pod-node-a.yaml
┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$kubectl apply -f pod-node-a.yaml
pod/podnodea created
┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$kubectl get pods -owide
NAME       READY   STATUS    RESTARTS   AGE   IP             NODE                         NOMINATED NODE   READINESS GATES
podnodea   1/1     Running   0          13s   10.244.70.61   vms83.liruilongs.github.io   <none>           <none>
┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$

软策略(preferredDuringSchedulingIgnoredDuringExecution)

pod-node-a-r.yaml

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: podnodea
  name: podnodea
spec:
  containers:
  - image: nginx
    imagePullPolicy: IfNotPresent
    name: podnodea
    resources: {}
  affinity:
    nodeAffinity: #主机亲和性
      preferredDuringSchedulingIgnoredDuringExecution: # 软策略
      - weight: 2
        preference:
          matchExpressions:
          - key: kubernetes.io/hostname
            operator: In
            values:
            - vms85.liruilongs.github.io
            - vms84.liruilongs.github.io
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

检查一下

┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$vim pod-node-a-r.yaml
┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$kubectl apply -f  pod-node-a-r.yaml
pod/podnodea created
┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$kubectl get pods -owide
NAME       READY   STATUS    RESTARTS   AGE   IP             NODE                         NOMINATED NODE   READINESS GATES
podnodea   1/1     Running   0          28s   10.244.70.62   vms83.liruilongs.github.io   <none>           <none>

运算符 描述
In 包含自, 比如上面的硬亲和就包含env_role=dev、env_role=test两种标签
NotIn 和上面相反，凡是包含该标签的节点都不会匹配到
Exists 存在里面和In比较类似，凡是有某个标签的机器都会被选择出来。使用Exists的operator的话，values里面就不能写东西了。
Gt greater than的意思，表示凡是某个value大于设定的值的机器则会被选择出来。
Lt less than的意思，表示凡是某个value小于设定的值的机器则会被选择出来。
DoesNotExists 不存在该标签的节点

Annotations 的设置

Annotations 即注释，设置查看方式很简单

┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$kubectl annotate nodes vms82.liruilongs.github.io "dest=这是一个工作节点"
node/vms82.liruilongs.github.io annotated
┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$kubectl describe nodes vms82.liruilongs.github.io
Name:               vms82.liruilongs.github.io
Roles:              worker1
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    disktype=node1
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=vms82.liruilongs.github.io
                    kubernetes.io/os=linux
                    node-role.kubernetes.io/worker1=
Annotations:        dest: 这是一个工作节点
                    kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
                    node.alpha.kubernetes.io/ttl: 0
                    projectcalico.org/IPv4Address: 192.168.26.82/24
                    projectcalico.org/IPv4IPIPTunnelAddr: 10.244.171.128
                    volumes.kubernetes.io/controller-managed-attach-detach: true

七、节点的coedon与drain

如果想把某个节点设置为不可用的话，可以对节点实施cordon或者drain

如果一个node被标记为cordon，新创建的pod不会被调度到此node上，已经调度上去的不会被移走

coedon用于节点的维护，当不希望再节点分配pod，那么可以使用coedon把节点标记为不可调度。

这里我们为了方便，创建一个Deployment控制器用去用于演示，关于Deployment，可以简单理解为他能保证你的pod保持在一定数量，当pod挂掉事，

┌──[[email protected]]-[~/ansible]
└─$kubectl create deployment nginx  --image=nginx  --dry-run=client -o yaml >nginx-dep.yaml
┌──[[email protected]]-[~/ansible]
└─$cp nginx-dep.yaml  ./k8s-pod-create/nginx-dep.yaml
┌──[[email protected]]-[~/ansible]
└─$cd k8s-pod-create/
┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$vim nginx-dep.yaml

nginx-dep.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: nginx
  name: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx
        name: nginx
        imagePullPolicy: IfNotPresent
        resources: {}
status: {}

创建 deploy资源

┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$kubectl apply  -f nginx-dep.yaml
deployment.apps/nginx created
┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$kubectl get pods -owide
NAME                     READY   STATUS    RESTARTS   AGE     IP               NODE
      NOMINATED NODE   READINESS GATES
nginx-7cf7d6dbc8-hx96s   1/1     Running   0          2m16s   10.244.171.167   vms82.liruilongs.github.io   <none>           <none>
nginx-7cf7d6dbc8-wshxp   1/1     Running   0          2m16s   10.244.70.1      vms83.liruilongs.github.io   <none>           <none>
nginx-7cf7d6dbc8-x78x4   1/1     Running   0          2m16s   10.244.70.63     vms83.liruilongs.github.io   <none>           <none>
┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$

节点的coedon

kubectl cordon vms83.liruilongs.github.io  #标记不可用
kubectl uncordon vms83.liruilongs.github.io #取消标记

通过cordon把vms83.liruilongs.github.io标记为不可调度

┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$kubectl cordon vms83.liruilongs.github.io  #通过cordon把83标记为不可调度
node/vms83.liruilongs.github.io cordoned

查看节点状态，vms83.liruilongs.github.io变成SchedulingDisabled

┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$kubectl get nodes
NAME                         STATUS                     ROLES                  AGE   VERSION
vms81.liruilongs.github.io   Ready                      control-plane,master   48d   v1.22.2
vms82.liruilongs.github.io   Ready                      worker1                48d   v1.22.2
vms83.liruilongs.github.io   Ready,SchedulingDisabled   worker2                48d   v1.22.2

修改deployment副本数量 –replicas=6 

┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$kubectl scale deployment nginx  --replicas=6
deployment.apps/nginx scaled

新增的pod都调度到了vms82.liruilongs.github.io 节点

┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$kubectl get pods -o wide
NAME                     READY   STATUS    RESTARTS   AGE     IP               NODE                         NOMINATED NODE   READINESS GATES
nginx-7cf7d6dbc8-2nmsj   1/1     Running   0          64s     10.244.171.170   vms82.liruilongs.github.io   <none>           <none>
nginx-7cf7d6dbc8-chsrn   1/1     Running   0          63s     10.244.171.168   vms82.liruilongs.github.io   <none>           <none>
nginx-7cf7d6dbc8-hx96s   1/1     Running   0          7m30s   10.244.171.167   vms82.liruilongs.github.io   <none>           <none>
nginx-7cf7d6dbc8-lppbp   1/1     Running   0          63s     10.244.171.169   vms82.liruilongs.github.io   <none>           <none>
nginx-7cf7d6dbc8-wshxp   1/1     Running   0          7m30s   10.244.70.1      vms83.liruilongs.github.io   <none>           <none>
nginx-7cf7d6dbc8-x78x4   1/1     Running   0          7m30s   10.244.70.63     vms83.liruilongs.github.io   <none>           <none>
┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$

把vms83.liruilongs.github.io节点上的Nginx都干掉，会发现新增pod都调度到了vms82.liruilongs.github.io

┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$kubectl delete pod nginx-7cf7d6dbc8-wshxp
pod "nginx-7cf7d6dbc8-wshxp" deleted
┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$kubectl get pods -o wide
NAME                     READY   STATUS    RESTARTS   AGE     IP               NODE                         NOMINATED NODE   READINESS GATES
nginx-7cf7d6dbc8-2nmsj   1/1     Running   0          2m42s   10.244.171.170   vms82.liruilongs.github.io   <none>           <none>
nginx-7cf7d6dbc8-5hnc7   1/1     Running   0          10s     10.244.171.171   vms82.liruilongs.github.io   <none>           <none>
nginx-7cf7d6dbc8-chsrn   1/1     Running   0          2m41s   10.244.171.168   vms82.liruilongs.github.io   <none>           <none>
nginx-7cf7d6dbc8-hx96s   1/1     Running   0          9m8s    10.244.171.167   vms82.liruilongs.github.io   <none>           <none>
nginx-7cf7d6dbc8-lppbp   1/1     Running   0          2m41s   10.244.171.169   vms82.liruilongs.github.io   <none>           <none>
nginx-7cf7d6dbc8-x78x4   1/1     Running   0          9m8s    10.244.70.63     vms83.liruilongs.github.io   <none>           <none>
┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$kubectl delete pod nginx-7cf7d6dbc8-x78x4
pod "nginx-7cf7d6dbc8-x78x4" deleted
┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$kubectl get pods -o wide
NAME                     READY   STATUS    RESTARTS   AGE     IP               NODE                         NOMINATED NODE   READINESS GATES
nginx-7cf7d6dbc8-2nmsj   1/1     Running   0          3m31s   10.244.171.170   vms82.liruilongs.github.io   <none>           <none>
nginx-7cf7d6dbc8-5hnc7   1/1     Running   0          59s     10.244.171.171   vms82.liruilongs.github.io   <none>           <none>
nginx-7cf7d6dbc8-chsrn   1/1     Running   0          3m30s   10.244.171.168   vms82.liruilongs.github.io   <none>           <none>
nginx-7cf7d6dbc8-hx96s   1/1     Running   0          9m57s   10.244.171.167   vms82.liruilongs.github.io   <none>           <none>
nginx-7cf7d6dbc8-lppbp   1/1     Running   0          3m30s   10.244.171.169   vms82.liruilongs.github.io   <none>           <none>
nginx-7cf7d6dbc8-m8ltr   1/1     Running   0          30s     10.244.171.172   vms82.liruilongs.github.io   <none>           <none>

通过 uncordon恢复节点vms83.liruilongs.github.io状态

┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$kubectl uncordon vms83.liruilongs.github.io #恢复节点状态
node/vms83.liruilongs.github.io uncordoned
┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$kubectl get nodes
NAME                         STATUS   ROLES                  AGE   VERSION
vms81.liruilongs.github.io   Ready    control-plane,master   48d   v1.22.2
vms82.liruilongs.github.io   Ready    worker1                48d   v1.22.2
vms83.liruilongs.github.io   Ready    worker2                48d   v1.22.2

删除所有的pod

┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$kubectl scale deployment nginx --replicas=0
deployment.apps/nginx scaled
┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$kubectl get pods -o wide
No resources found in liruilong-pod-create namespace.

节点的为drain

如果一个节点被设置为drain，则此节点不再被调度pod，且此节点上已经运行的pod会被驱逐(evicted)到其他节点

drain包含两种状态：cordon不可被调度，evicted驱逐当前节点所以pod

kubectl drain vms83.liruilongs.github.io   --ignore-daemonsets
kubectl uncordon vms83.liruilongs.github.io

通过deployment添加4个nginx副本--replicas=4

┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$kubectl scale deployment nginx --replicas=4
deployment.apps/nginx scaled
┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$kubectl get pods -o wide --one-output
NAME                     READY   STATUS    RESTARTS   AGE   IP               NODE                         NOMINATED NODE   READINESS GATES
nginx-7cf7d6dbc8-2clnb   1/1     Running   0          22s   10.244.171.174   vms82.liruilongs.github.io   <none>           <none>
nginx-7cf7d6dbc8-9p6g2   1/1     Running   0          22s   10.244.70.2      vms83.liruilongs.github.io   <none>           <none>
nginx-7cf7d6dbc8-ptqxm   1/1     Running   0          22s   10.244.171.173   vms82.liruilongs.github.io   <none>           <none>
nginx-7cf7d6dbc8-zmdqm   1/1     Running   0          22s   10.244.70.4      vms83.liruilongs.github.io   <none>           <none>

添加一下污点 将节点vms82.liruilongs.github.io设置为drain

┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$kubectl drain vms82.liruilongs.github.io --ignore-daemonsets --delete-emptydir-data
node/vms82.liruilongs.github.io cordoned
WARNING: ignoring DaemonSet-managed Pods: kube-system/calico-node-ntm7v, kube-system/kube-proxy-nzm24
evicting pod liruilong-pod-create/nginx-7cf7d6dbc8-ptqxm
evicting pod kube-system/metrics-server-bcfb98c76-wxv5l
evicting pod liruilong-pod-create/nginx-7cf7d6dbc8-2clnb
pod/nginx-7cf7d6dbc8-2clnb evicted
pod/nginx-7cf7d6dbc8-ptqxm evicted
pod/metrics-server-bcfb98c76-wxv5l evicted
node/vms82.liruilongs.github.io evicted
┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$kubectl get nodes
NAME                         STATUS                     ROLES                  AGE   VERSION
vms81.liruilongs.github.io   Ready                      control-plane,master   48d   v1.22.2
vms82.liruilongs.github.io   Ready,SchedulingDisabled   worker1                48d   v1.22.2
vms83.liruilongs.github.io   Ready                      worker2                48d   v1.22.2

查看节点调度，所有pod调度到了vms83.liruilongs.github.io这台机器

┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$kubectl get pods -o wide --one-output
NAME                     READY   STATUS    RESTARTS   AGE     IP            NODE                         NOMINATED NODE   READINESS GATES
nginx-7cf7d6dbc8-9p6g2   1/1     Running   0          4m20s   10.244.70.2   vms83.liruilongs.github.io   <none>           <none>
nginx-7cf7d6dbc8-hkflr   1/1     Running   0          25s     10.244.70.5   vms83.liruilongs.github.io   <none>           <none>
nginx-7cf7d6dbc8-qt48k   1/1     Running   0          26s     10.244.70.7   vms83.liruilongs.github.io   <none>           <none>
nginx-7cf7d6dbc8-zmdqm   1/1     Running   0          4m20s   10.244.70.4   vms83.liruilongs.github.io   <none>           <none>
┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$

取消污点：kubectl uncordon vms82.liruilongs.github.io

┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$kubectl uncordon vms82.liruilongs.github.io
node/vms82.liruilongs.github.io uncordoned
┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$

报错的情况

将节点vms82.liruilongs.github.io设置为drain

┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$kubectl drain vms82.liruilongs.github.io
node/vms82.liruilongs.github.io cordoned
DEPRECATED WARNING: Aborting the drain command in a list of nodes will be deprecated in v1.23.
The new behavior will make the drain command go through all nodes even if one or more nodes failed during the drain.
For now, users can try such experience via: --ignore-errors
error: unable to drain node "vms82.liruilongs.github.io", aborting command...

There are pending nodes to be drained:
 vms82.liruilongs.github.io
cannot delete DaemonSet-managed Pods (use --ignore-daemonsets to ignore): kube-system/calico-node-ntm7v, kube-system/kube-proxy-nzm24
cannot delete Pods with local storage (use --delete-emptydir-data to override): kube-system/metrics-server-bcfb98c76-wxv5l
┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$kubectl get nodes
NAME                         STATUS                     ROLES                  AGE   VERSION
vms81.liruilongs.github.io   Ready                      control-plane,master   48d   v1.22.2
vms82.liruilongs.github.io   Ready,SchedulingDisabled   worker1                48d   v1.22.2
vms83.liruilongs.github.io   Ready                      worker2                48d   v1.22.2

uncordon掉刚才的节点

┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$kubectl uncordon vms82.liruilongs.github.io
node/vms82.liruilongs.github.io uncordoned
┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$kubectl get nodes
NAME                         STATUS   ROLES                  AGE   VERSION
vms81.liruilongs.github.io   Ready    control-plane,master   48d   v1.22.2
vms82.liruilongs.github.io   Ready    worker1                48d   v1.22.2
vms83.liruilongs.github.io   Ready    worker2                48d   v1.22.2

八、节点taint(污点)及pod的tolerations(容忍污点)

给节点设置及删除taint，设置operator的值为Equal，以及设置operator的值为Exists

┌──[[email protected]]-[~/ansible]
└─$ansible master -m shell -a "kubectl describe  nodes vms81.liruilongs.github.io | grep -E '(Roles|Taints)'"
192.168.26.81 | CHANGED | rc=0 >>
Roles:              control-plane,master
Taints:             node-role.kubernetes.io/master:NoSchedule

master节点从来没有调度到pod，因为master节点设置了污点，如果想要在某个被设置了污点的机器调度pod，那么pod需要设置tolerations(容忍污点)才能够被运行。

taint（污点）的设置和查看

# 查看节点角色，和是否设置污点
┌──[[email protected]]-[~/ansible]
└─$kubectl describe  nodes vms82.liruilongs.github.io | grep -E '(Roles|Taints)'
Roles:              worker1
Taints:             <none>
# 给 vms83.liruilongs.github.io节点设置污点，指定key为key83
┌──[[email protected]]-[~/ansible]
└─$kubectl describe  nodes vms83.liruilongs.github.io | grep -E '(Roles|Taints)'
Roles:              worker2
Taints:             <none>
┌──[[email protected]]-[~/ansible]
└─$kubectl taint node vms83.liruilongs.github.io key83=:NoSchedule
node/vms83.liruilongs.github.io tainted
┌──[[email protected]]-[~/ansible]
└─$kubectl describe  nodes vms83.liruilongs.github.io | grep -E '(Roles|Taints)' # 从新查看污点信息
Roles:              worker2
Taints:             key83:NoSchedule
┌──[[email protected]]-[~/ansible]
└─$

重新通过deployment 创建pod，会发现pod都调度到82上面，因为83设置了污点

┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$kubectl scale deployment nginx --replicas=0
deployment.apps/nginx scaled
┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$kubectl scale deployment nginx --replicas=4
deployment.apps/nginx scaled
┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$kubectl get pods -o wide --one-output
NAME                     READY   STATUS              RESTARTS   AGE   IP       NODE                         NOMINATED NODE   READINESS GATES
nginx-7cf7d6dbc8-dhst5   0/1     ContainerCreating   0          12s   <none>   vms82.liruilongs.github.io   <none>           <none>
nginx-7cf7d6dbc8-j6g25   0/1     ContainerCreating   0          12s   <none>   vms82.liruilongs.github.io   <none>           <none>
nginx-7cf7d6dbc8-wpnhr   0/1     ContainerCreating   0          12s   <none>   vms82.liruilongs.github.io   <none>           <none>
nginx-7cf7d6dbc8-zkww8   0/1     ContainerCreating   0          11s   <none>   vms82.liruilongs.github.io   <none>           <none>
┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$kubectl delete deployment nginx
deployment.apps "nginx" deleted

取消污点设置

┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$kubectl taint node vms83.liruilongs.github.io key83-
node/vms83.liruilongs.github.io untainted
┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$kubectl describe  nodes vms83.liruilongs.github.io | grep -E '(Roles|Taints)'
Roles:              worker2
Taints:             <none>
┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$

设置operator的值为Equal

如果需要在有污点的节点上运行pod，那么需要在定义pod的时候指定toleration属性

在设置节点taint的时候，如果value的值为不为空，在pod里的tolerations字段只能写Equal，不能写Exists,

┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$kubectl taint nodes  vms82.liruilongs.github.io key82=val82:NoSchedule
node/vms82.liruilongs.github.io tainted
┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$kubectl describe  nodes vms82.liruilongs.github.io | grep -E '(Roles|Taints)'
Roles:              worker1
Taints:             key82=val82:NoSchedule
┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$

修改yaml文件 pod-taint3.yaml

┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$cat pod-taint2.yaml > pod-taint3.yaml
┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$vim  pod-taint3.yaml
┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$cat pod-taint3.yaml
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: pod1
  name: pod1
spec:
  nodeSelector:
    disktype: node2
  tolerations:
  - key: "key82"
    operator: "Equal"
    value: "val82"
    effect: "NoSchedule"
  containers:
  - image: nginx
    name: pod1
    resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}
┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$kubectl apply -f pod-taint3.yaml
pod/pod1 created
┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$kubectl get pods -o wide
NAME   READY   STATUS    RESTARTS   AGE   IP               NODE                         NOMINATED NODE   READINESS GATES
pod1   1/1     Running   0          11s   10.244.171.180   vms82.liruilongs.github.io   <none>           <none>
┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$

设置operator的值为Exists

如果使用Exists的话，那么pod中不能写value

设置vms83.liruilongs.github.io 节点污点标记

┌──[[email protected]]-[~/ansible]
└─$kubectl taint node vms83.liruilongs.github.io key83=:NoSchedule
node/vms83.liruilongs.github.io tainted
┌──[[email protected]]-[~/ansible]
└─$kubectl describe  nodes vms83.liruilongs.github.io | grep -E '(Roles|Taints)'
Roles:              worker2
Taints:             key83:NoSchedule
┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$kubectl get nodes --show-labels
NAME                         STATUS   ROLES                  AGE   VERSION   LABELS
vms81.liruilongs.github.io   Ready    control-plane,master   48d   v1.22.2   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=vms81.liruilongs.github.io,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=,node-role.kubernetes.io/master=,node.kubernetes.io/exclude-from-external-load-balancers=
vms82.liruilongs.github.io   Ready    worker1                48d   v1.22.2   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,disktype=node1,kubernetes.io/arch=amd64,kubernetes.io/hostname=vms82.liruilongs.github.io,kubernetes.io/os=linux,node-role.kubernetes.io/worker1=
vms83.liruilongs.github.io   Ready    worker2                48d   v1.22.2   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,disktype=node2,kubernetes.io/arch=amd64,kubernetes.io/hostname=vms83.liruilongs.github.io,kubernetes.io/os=linux,node-role.kubernetes.io/worker2=

pod-taint.yaml

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: pod1
  name: pod1
spec:
  nodeSelector:
    disktype: node2
  tolerations:
  - key: "key83"
    operator: "Exists"
    effect: "NoSchedule"
  containers:
  - image: nginx
    name: pod1
    resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

会发现节点调度到了有污点的vms83.liruilongs.github.io节点

┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$kubectl apply -f pod-taint.yaml
pod/pod1 created
┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$kubectl get pods -o wide
NAME   READY   STATUS    RESTARTS   AGE    IP            NODE                         NOMINATED NODE   READINESS GATES
pod1   1/1     Running   0          3m4s   10.244.70.8   vms83.liruilongs.github.io   <none>           <none>

当然，value没有值也可以这样使用Equal

┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$cp pod-taint.yaml pod-taint2.yaml
┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$vim pod-taint2.yaml

pod-taint2.yaml

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: pod1
  name: pod1
spec:
  nodeSelector:
    disktype: node2
  tolerations:
  - key: "key83"
    operator: "Equal"
    value: ""
    effect: "NoSchedule"
  containers:
  - image: nginx
    name: pod1
    resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

会发现节点还是调度到了有污点的vms83.liruilongs.github.io节点

┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$kubectl  delete -f pod-taint.yaml
pod "pod1" deleted
┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$kubectl apply -f pod-taint2.yaml
pod/pod1 created
┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$kubectl get pods -o wide
NAME   READY   STATUS              RESTARTS   AGE   IP       NODE                         NOMINATED NODE   READINESS GATES
pod1   0/1     ContainerCreating   0          8s    <none>   vms83.liruilongs.github.io   <none>           <none>
┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$kubectl taint nodes vms83.liruilongs.github.io key83-
node/vms83.liruilongs.github.io untainted
┌──[[email protected]]-[~/ansible/k8s-pod-create]
└─$

report

Recommend

About Joyk


Aggregate valuable and interesting links.
Joyk means Joy of geeK