12
docker18.03 hang at 'restoring container'
source link: https://zhangguanzhang.github.io/2020/12/04/docker1803-hang-container-restore/
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
起初是 k8s 有几个 node not ready，上去看了下 kubelet 日志刷 container runtime down，重启了下 docker 后还是没用，docker ps 命令都卡住。
环境信息
$ cat /etc/redhat-release Linux xxx-disk0 3.10.0-1127.13.1.el7.x86_64 #1 SMP Tue Jun 23 15:46:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux CentOS Linux release 7.4.1708 (Core) $ docker info Containers: 91 Running: 63 Paused: 0 Stopped: 28 Images: 539 Server Version: 18.03.0-ce Storage Driver: overlay2 Backing Filesystem: xfs Supports d_type: true Native Overlay Diff: true Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog Swarm: inactive Runtimes: runc Default Runtime: runc Init Binary: docker-init containerd version: cfd04396dc68220d1cecbe686a6cc3aa5ce3667c runc version: 4fc53a81fb7c994640722ac585fa9ca548971871 init version: 949e6fa Security Options: seccomp Profile: default Kernel Version: 3.10.0-1127.13.1.el7.x86_64 Operating System: CentOS Linux 7 (Core) OSType: linux Architecture: x86_64 CPUs: 8 Total Memory: 15.51GiB Name: xxx-disk0 ID: UZRM:KRSL:TYWM:VAQY:KWCX:AVFD:NP53:TC35:YHOC:TLLO:YGXO:RMYS Docker Root Dir: /app/kube/docker Debug Mode (client): false Debug Mode (server): false Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: treg.yun.xxx.cn reg.xxx.lan:5000 127.0.0.0/8 Registry Mirrors: https://registry.docker-cn.com/ https://docker.mirrors.ustc.edu.cn/ Live Restore Enabled: false
排查过程
先停掉docker，然后前台启动加 debug 参数启动
$ pgrep dockerd 4659 $ kill 4659 && > /var/run/docker.pid $ ps aux | grep dockerd root 5628 0.0 0.0 112708 980 pts/0 S+ 22:33 0:00 grep --color=auto dockerd $ ./dockerd -D WARN[0000] The "graph" config file option is deprecated. Please use "data-root" instead. WARN[2020-12-04T22:33:50.432804342+08:00] could not change group /var/run/docker.sock to docker: group docker not found DEBU[2020-12-04T22:33:50.432936283+08:00] Listener created for HTTP on unix (/var/run/docker.sock) INFO[2020-12-04T22:33:50.433612435+08:00] libcontainerd: started new docker-containerd process pid=5646 INFO[0000] starting containerd module=containerd revision=cfd04396dc68220d1cecbe686a6cc3aa5ce3667c version=v1.0.2 DEBU[0000] changing OOM score to -500 module=containerd INFO[0000] loading plugin "io.containerd.content.v1.content"... module=containerd type=io.containerd.content.v1 INFO[0000] loading plugin "io.containerd.snapshotter.v1.btrfs"... module=containerd type=io.containerd.snapshotter.v1 WARN[0000] failed to load plugin io.containerd.snapshotter.v1.btrfs error="path /app/kube/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter" module=containerd INFO[0000] loading plugin "io.containerd.snapshotter.v1.overlayfs"... module=containerd type=io.containerd.snapshotter.v1 INFO[0000] loading plugin "io.containerd.metadata.v1.bolt"... module=containerd type=io.containerd.metadata.v1 WARN[0000] could not use snapshotter btrfs in metadata plugin error="path /app/kube/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter" module="containerd/io.containerd.metadata.v1.bolt" INFO[0000] loading plugin "io.containerd.differ.v1.walking"... module=containerd type=io.containerd.differ.v1 INFO[0000] loading plugin "io.containerd.gc.v1.scheduler"... module=containerd type=io.containerd.gc.v1 INFO[0000] loading plugin "io.containerd.grpc.v1.containers"... module=containerd type=io.containerd.grpc.v1 INFO[0000] loading plugin "io.containerd.grpc.v1.content"... module=containerd type=io.containerd.grpc.v1 INFO[0000] loading plugin "io.containerd.grpc.v1.diff"... module=containerd type=io.containerd.grpc.v1 INFO[0000] loading plugin "io.containerd.grpc.v1.events"... module=containerd type=io.containerd.grpc.v1 INFO[0000] loading plugin "io.containerd.grpc.v1.healthcheck"... module=containerd type=io.containerd.grpc.v1 INFO[0000] loading plugin "io.containerd.grpc.v1.images"... module=containerd type=io.containerd.grpc.v1 INFO[0000] loading plugin "io.containerd.grpc.v1.leases"... module=containerd type=io.containerd.grpc.v1 INFO[0000] loading plugin "io.containerd.grpc.v1.namespaces"... module=containerd type=io.containerd.grpc.v1 INFO[0000] loading plugin "io.containerd.grpc.v1.snapshots"... module=containerd type=io.containerd.grpc.v1 INFO[0000] loading plugin "io.containerd.monitor.v1.cgroups"... module=containerd type=io.containerd.monitor.v1 INFO[0000] loading plugin "io.containerd.runtime.v1.linux"... module=containerd type=io.containerd.runtime.v1 DEBU[0000] loading tasks in namespace module="containerd/io.containerd.runtime.v1.linux" namespace=moby INFO[0000] loading plugin "io.containerd.grpc.v1.tasks"... module=containerd type=io.containerd.grpc.v1 INFO[0000] loading plugin "io.containerd.grpc.v1.version"... module=containerd type=io.containerd.grpc.v1 INFO[0000] loading plugin "io.containerd.grpc.v1.introspection"... module=containerd type=io.containerd.grpc.v1 INFO[0000] serving... address="/var/run/docker/containerd/docker-containerd-debug.sock" module="containerd/debug" INFO[0000] serving... address="/var/run/docker/containerd/docker-containerd.sock" module="containerd/grpc" INFO[0000] containerd successfully booted in 0.009604s module=containerd DEBU[2020-12-04T22:33:50.456534148+08:00] Golang's threads limit set to 113940 DEBU[2020-12-04T22:33:50.457345643+08:00] Using default logging driver json-file DEBU[2020-12-04T22:33:50.457466912+08:00] [graphdriver] priority list: [btrfs zfs overlay2 aufs overlay devicemapper vfs] DEBU[2020-12-04T22:33:50.457623030+08:00] processing event stream module=libcontainerd namespace=plugins.moby DEBU[2020-12-04T22:33:50.479691287+08:00] backingFs=xfs, projectQuotaSupported=false INFO[2020-12-04T22:33:50.479712832+08:00] [graphdriver] using prior storage driver: overlay2 DEBU[2020-12-04T22:33:50.479724151+08:00] Initialized graph driver overlay2 DEBU[2020-12-04T22:33:50.510882767+08:00] Max Concurrent Downloads: 10 DEBU[2020-12-04T22:33:50.510930407+08:00] Max Concurrent Uploads: 5 DEBU[0000] garbage collected d=24.493383ms module="containerd/io.containerd.gc.v1.scheduler" INFO[2020-12-04T22:33:50.608483121+08:00] Graph migration to content-addressability took 0.00 seconds INFO[2020-12-04T22:33:50.610430840+08:00] Loading containers: start. DEBU[2020-12-04T22:33:50.610704281+08:00] processing event stream module=libcontainerd namespace=moby DEBU[2020-12-04T22:33:50.611446797+08:00] Loaded container 027a389c8c1e93629cc5f68af8d023b2ecfe350d7771ba6b87598ff705f6c19f, isRunning: false DEBU[2020-12-04T22:33:50.611803503+08:00] Loaded container 24735e5aea2bd91b5fa5d729ca021a09532c2ea9b8b06f5171d0da23fc3bf4cc, isRunning: false DEBU[2020-12-04T22:33:50.612174253+08:00] Loaded container 487a8c2f30986796c3948d1469d506e1d3ab394e17533040ef7a5444a32be0fc, isRunning: false DEBU[2020-12-04T22:33:50.612494092+08:00] Loaded container 52d32b0e03c957b6cb9b4d793c47900e689a29d9ae0d63703ea29073a352fbe5, isRunning: false DEBU[2020-12-04T22:33:50.612816495+08:00] Loaded container 5b7b0b52c71a14164f269853679211b3823e9eecc2d3829bf2db10c9b720217d, isRunning: false DEBU[2020-12-04T22:33:50.613447082+08:00] Loaded container 62b049d16d1fe03c193295329e7055b3e675a5e94b9566eee6accc35820530be, isRunning: true DEBU[2020-12-04T22:33:50.613769649+08:00] Loaded container 68ba211ec7328bebd3b241631a703639447c05056ffe07ed633b72d0bc210938, isRunning: false DEBU[2020-12-04T22:33:50.614756585+08:00] Loaded container 73cfe941e7a948a77783c77f963efc66327323c2603e058e7ab61f85f8e98212, isRunning: true DEBU[2020-12-04T22:33:50.615381990+08:00] Loaded container 7a2a75c8a0c8dac8c5773ad92fa45ac7e1d33c9be85ecb65eb147929955dca50, isRunning: false DEBU[2020-12-04T22:33:50.616222796+08:00] Loaded container 81095c01c4b99c7d2cc9e6bee8726c11f16d27204523727e7d067d980c26ac64, isRunning: false DEBU[2020-12-04T22:33:50.616569394+08:00] Loaded container 94098167eb466dbf1a454f5491a162488d1fdb1eebe804c5c1f403f7fce62dc4, isRunning: false DEBU[2020-12-04T22:33:50.616981038+08:00] Loaded container 9d4cbcce43b0262d972e73b2770f26ca762e2fa86f0de88a7909b8e59c0b805a, isRunning: false DEBU[2020-12-04T22:33:50.617460452+08:00] Loaded container aecde8eb18924d8548d79d5e0383baa7ac3ab1cfc4c55e1f32c4089dfc153071, isRunning: false DEBU[2020-12-04T22:33:50.617908975+08:00] Loaded container aed0618a325b4b84363357c1830515048d23af6afd79606cbb0ad64bf5f226a2, isRunning: false DEBU[2020-12-04T22:33:50.618252961+08:00] Loaded container b43e4995720f235c40ffd60bde1fb54e87ece3598f8bd625996042f637896687, isRunning: false DEBU[2020-12-04T22:33:50.618557604+08:00] Loaded container c1e6a1de9b9c2fd420e718c405c114e726ec5531561a4caf662b757a3724711e, isRunning: false DEBU[2020-12-04T22:33:50.618942417+08:00] Loaded container c5eb3c941e562153e0cf0af738f1cb43f34591f0b48ad5458ab2002f5be9e0a8, isRunning: false DEBU[2020-12-04T22:33:50.619380785+08:00] Loaded container e211ffccacb8f7982899097fbd0f9ce1d95f8f31f290fb10baf40d00f4980bc9, isRunning: false DEBU[2020-12-04T22:33:50.619831551+08:00] Loaded container ef547d238cd01ff7ec048de3442fe9293aa1d5d932ea66c5aed34bfff014182b, isRunning: false DEBU[2020-12-04T22:33:50.620192032+08:00] Loaded container f3bb916ec5d7847c3be4341975c47f4e2fe587fc726ca7d76e3dca15cb8dd21d, isRunning: false DEBU[2020-12-04T22:33:50.620438678+08:00] Loaded container fa6de6f4aa8894c18a9737bac462f57c69893eca5e4b58bc3bd793a76b252951, isRunning: false DEBU[2020-12-04T22:33:51.379861237+08:00] restoring container container=ef547d238cd01ff7ec048de3442fe9293aa1d5d932ea66c5aed34bfff014182b paused=false running=false DEBU[2020-12-04T22:33:51.379910464+08:00] restoring container container=e211ffccacb8f7982899097fbd0f9ce1d95f8f31f290fb10baf40d00f4980bc9 paused=false running=false DEBU[2020-12-04T22:33:51.379994141+08:00] restoring container container=7a2a75c8a0c8dac8c5773ad92fa45ac7e1d33c9be85ecb65eb147929955dca50 paused=false running=false DEBU[2020-12-04T22:33:51.380029802+08:00] restoring container container=c1e6a1de9b9c2fd420e718c405c114e726ec5531561a4caf662b757a3724711e paused=false running=false DEBU[2020-12-04T22:33:51.380084763+08:00] restoring container container=5b7b0b52c71a14164f269853679211b3823e9eecc2d3829bf2db10c9b720217d paused=false running=false DEBU[2020-12-04T22:33:51.380127006+08:00] restoring container container=9d4cbcce43b0262d972e73b2770f26ca762e2fa86f0de88a7909b8e59c0b805a paused=false running=false DEBU[2020-12-04T22:33:51.380121758+08:00] restoring container container=fa6de6f4aa8894c18a9737bac462f57c69893eca5e4b58bc3bd793a76b252951 paused=false running=false DEBU[2020-12-04T22:33:51.380163318+08:00] restoring container container=52d32b0e03c957b6cb9b4d793c47900e689a29d9ae0d63703ea29073a352fbe5 paused=false running=false DEBU[2020-12-04T22:33:51.380310029+08:00] restoring container container=027a389c8c1e93629cc5f68af8d023b2ecfe350d7771ba6b87598ff705f6c19f paused=false running=false DEBU[2020-12-04T22:33:51.380382722+08:00] restoring container container=68ba211ec7328bebd3b241631a703639447c05056ffe07ed633b72d0bc210938 paused=false running=false DEBU[2020-12-04T22:33:51.380419320+08:00] restoring container container=487a8c2f30986796c3948d1469d506e1d3ab394e17533040ef7a5444a32be0fc paused=false running=false DEBU[2020-12-04T22:33:51.380433522+08:00] restoring container container=73cfe941e7a948a77783c77f963efc66327323c2603e058e7ab61f85f8e98212 paused=false running=true DEBU[2020-12-04T22:33:51.380459224+08:00] restoring container container=c5eb3c941e562153e0cf0af738f1cb43f34591f0b48ad5458ab2002f5be9e0a8 paused=false running=false DEBU[2020-12-04T22:33:51.380525276+08:00] restoring container container=b43e4995720f235c40ffd60bde1fb54e87ece3598f8bd625996042f637896687 paused=false running=false DEBU[2020-12-04T22:33:51.380563957+08:00] restoring container container=81095c01c4b99c7d2cc9e6bee8726c11f16d27204523727e7d067d980c26ac64 paused=false running=false DEBU[2020-12-04T22:33:51.380586567+08:00] restoring container container=62b049d16d1fe03c193295329e7055b3e675a5e94b9566eee6accc35820530be paused=false running=true DEBU[2020-12-04T22:33:51.380599061+08:00] restoring container container=94098167eb466dbf1a454f5491a162488d1fdb1eebe804c5c1f403f7fce62dc4 paused=false running=false DEBU[2020-12-04T22:33:51.380616220+08:00] restoring container container=aecde8eb18924d8548d79d5e0383baa7ac3ab1cfc4c55e1f32c4089dfc153071 paused=false running=false DEBU[2020-12-04T22:33:51.380641090+08:00] restoring container container=f3bb916ec5d7847c3be4341975c47f4e2fe587fc726ca7d76e3dca15cb8dd21d paused=false running=false DEBU[2020-12-04T22:33:51.380825356+08:00] restoring container container=24735e5aea2bd91b5fa5d729ca021a09532c2ea9b8b06f5171d0da23fc3bf4cc paused=false running=false DEBU[2020-12-04T22:33:51.380953092+08:00] restoring container container=aed0618a325b4b84363357c1830515048d23af6afd79606cbb0ad64bf5f226a2 paused=false running=false
然后发现卡在这，正常是会像 gin 那样启动输出支持的 http api 路由信息的。开一个窗口，发送 SIGUSR1 信号打印 goroutine 堆栈信息看看卡在哪儿：
$ pgrep dockerd 3085 $ kill -USR1 3085
docker 的日志和系统日志都会有下面的类似输出：
Dec 04 22:33:52 xxxx dockerd[3085]: time="2020-12-33T58:15:52.906433650+08:00" level=info msg="goroutine stacks written to /var/run/docker/goroutine-stacks-2020-12-04T223358+0800.log"
查看了下下面这段比较可疑， daemon/daemon.go:364
附近
goroutine 1 [semacquire, 5 minutes]: sync.runtime_Semacquire(0xc4204de73c) /usr/local/go/src/runtime/sema.go:56 +0x3b sync.(*WaitGroup).Wait(0xc4204de730) /usr/local/go/src/sync/waitgroup.go:131 +0x74 github.com/docker/docker/daemon.(*Daemon).restore(0xc42009a480, 0x190c3a6, 0x4) /go/src/github.com/docker/docker/daemon/daemon.go:364 +0xfeb github.com/docker/docker/daemon.NewDaemon(0xc42018d200, 0x2ec75c0, 0xc4201be410, 0x2ea89e0, 0xc420087d40, 0xc4201323c0, 0x0, 0x0, 0x0) /go/src/github.com/docker/docker/daemon/daemon.go:894 +0x258d main.(*DaemonCli).start(0xc42051da40, 0xc4201c5d50, 0x0, 0x0) /go/src/github.com/docker/docker/cmd/dockerd/daemon.go:223 +0x1320 main.runDaemon(0xc4201c5d50, 0xc42044b3b0, 0x0) /go/src/github.com/docker/docker/cmd/dockerd/docker.go:78 +0x78 main.newDaemonCommand.func1(0xc420176000, 0xc4201359e0, 0x0, 0x1, 0x0, 0x0) /go/src/github.com/docker/docker/cmd/dockerd/docker.go:29 +0x5d github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).execute(0xc420176000, 0xc42000c090, 0x1, 0x1, 0xc420176000, 0xc42000c090) /go/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:646 +0x44f github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc420176000, 0x2194e40, 0x2419c01, 0xc420135980) /go/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:742 +0x310 github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).Execute(0xc420176000, 0xc420135980, 0x190fa00) /go/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:695 +0x2d main.main() /go/src/github.com/docker/docker/cmd/dockerd/docker.go:105 +0xe3
按照docker info 的信息去找了下对应的 分支代码
。364 行一个 wg.Wait()
，得看前面的 goroutine 是卡在哪儿，根据前面的堆栈信息，应该是卡在 github.com/docker/docker/daemon.(*Daemon).restore
，也就是 238 行的 daemon.containerd.Restore 方法
，卡在 wg.Wait()
说明有协程没释放锁，这里 containerd.Restore方法
的第一行就是锁，里面有个方法 c.remote.LoadContainer
，实际上是和docker-containerd通信的。
查看下 docker-containerd 进程：
$ ps aux | grep containerd root 5646 0.2 0.0 606436 14048 ? Ssl 22:33 0:00 docker-containerd --config /var/run/docker/containerd/containerd.toml appuser 6261 0.0 0.0 112708 984 pts/1 S+ 22:36 0:00 grep --color=auto containerd root 8355 0.1 0.0 9052 4308 ? Sl Dec03 2:56 docker-containerd-shim -namespace moby -workdir /app/kube/dockercontainerd/daemon/io.containerd.runtime.v1.linux/moby/62b049d16d1fe03c193295329e7055b3e675a5e94b9566eee6accc35820530be -address /var/run/docker/containerd/docker-containerd.sock -containerd-binary /app/kube/bin/docker-containerd -runtime-root /var/run/docker/runtime-runc root 11171 0.0 0.0 9052 4052 ? Sl Dec03 0:18 docker-containerd-shim -namespace moby -workdir /app/kube/dockercontainerd/daemon/io.containerd.runtime.v1.linux/moby/73cfe941e7a948a77783c77f963efc66327323c2603e058e7ab61f85f8e98212 -address /var/run/docker/containerd/docker-containerd.sock -containerd-binary /app/kube/bin/docker-containerd -runtime-root /var/run/docker/runtime-runc
有残留的，杀掉一个试试
$ kill 11171
然后原窗口有输出了一些日志，实际上是执行了 244行的 daemon.containerd.DeleteTask方法 ，说明思路是对的，进程通信有问题。
ERRO[0172] connecting to shim error=<nil> id=73cfe941e7a948a77783c77f963efc66327323c2603e058e7ab61f85f8e98212 module="containerd/io.containerd.runtime.v1.linux" namespace=moby DEBU[2020-12-04T22:36:42.690975930+08:00] restored container alive=false container=73cfe941e7a948a77783c77f963efc66327323c2603e058e7ab61f85f8e98212 module=libcontainerd namespace=moby pid=0 DEBU[2020-12-04T22:36:42.701154551+08:00] Trying to unmount /app/kube/docker/containers/73cfe941e7a948a77783c77f963efc66327323c2603e058e7ab61f85f8e98212/mounts DEBU[2020-12-04T22:36:42.707909556+08:00] Unmounted /app/kube/docker/containers/73cfe941e7a948a77783c77f963efc66327323c2603e058e7ab61f85f8e98212/mounts DEBU[0172] event published module="containerd/io.containerd.runtime.v1.linux" ns=moby topic="/tasks/exit" type=containerd.events.TaskExit DEBU[0172] event published module="containerd/io.containerd.runtime.v1.linux" ns=moby topic="/tasks/delete" type=containerd.events.TaskDelete DEBU[2020-12-04T22:36:42.947670205+08:00] event module=libcontainerd namespace=moby topic=/tasks/exit
接着处理另一个：
$ ps aux | grep containerd root 5646 0.2 0.0 606692 14048 ? Ssl 22:33 0:00 docker-containerd --config /var/run/docker/containerd/containerd.toml root 6461 0.0 0.0 112708 984 pts/1 S+ 22:37 0:00 grep --color=auto containerd root 8355 0.1 0.0 9052 4260 ? Sl Dec03 2:56 docker-containerd-shim -namespace moby -workdir /app/kube/dockercontainerd/daemon/io.containerd.runtime.v1.linux/moby/62b049d16d1fe03c193295329e7055b3e675a5e94b9566eee6accc35820530be -address /var/run/docker/containerd/docker-containerd.sock -containerd-binary /app/kube/bin/docker-containerd -runtime-root /var/run/docker/runtime-runc $ kill 8355
然后前台 debug 的日志没有卡住，正常启动了：
ERRO[0249] connecting to shim error=<nil> id=62b049d16d1fe03c193295329e7055b3e675a5e94b9566eee6accc35820530be module="containerd/io.containerd.runtime.v1.linux" namespace=moby
DEBU[2020-12-04T22:37:59.709825146+08:00] restored container alive=false container=62b049d16d1fe03c193295329e7055b3e675a5e94b9566eee6accc35820530be module=libcontainerd namespace=moby pid=0
DEBU[2020-12-04T22:37:59.710064357+08:00] event module=libcontainerd namespace=moby topic=/tasks/delete
INFO[2020-12-04T22:37:59.710093459+08:00] ignoring event module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
WARN[2020-12-04T22:37:59.710215638+08:00] Ignoring Exit Event, no such exec command found container=73cfe941e7a948a77783c77f963efc66327323c2603e058e7ab61f85f8e98212 exec-id=73cfe941e7a948a77783c77f963efc66327323c2603e058e7ab61f85f8e98212 exec-pid=11197
DEBU[2020-12-04T22:37:59.719102521+08:00] Trying to unmount /app/kube/docker/containers/62b049d16d1fe03c193295329e7055b3e675a5e94b9566eee6accc35820530be/mounts
DEBU[2020-12-04T22:37:59.722934436+08:00] Unmounted /app/kube/docker/containers/62b049d16d1fe03c193295329e7055b3e675a5e94b9566eee6accc35820530be/mounts
DEBU[0249] event published module="containerd/containers" ns=moby topic="/containers/delete" type=containerd.events.ContainerDelete
DEBU[2020-12-04T22:37:59.978450001+08:00] container mounted via layerStore: &{/app/kube/docker/overlay2/97a09a97cf8c3ae835fb0ca6526c0282b26379942dfb49081189a39ce0400596/merged 0x2f42600 0x2f42600}
DEBU[0249] event published module="containerd/io.containerd.runtime.v1.linux" ns=moby topic="/tasks/exit" type=containerd.events.TaskExit
DEBU[2020-12-04T22:38:00.169804015+08:00] event module=libcontainerd namespace=moby topic=/tasks/exit
DEBU[0249] event published module="containerd/io.containerd.runtime.v1.linux" ns=moby topic="/tasks/delete" type=containerd.events.TaskDelete
WARN[2020-12-04T22:38:00.169915440+08:00] Ignoring Exit Event, no such exec command found container=62b049d16d1fe03c193295329e7055b3e675a5e94b9566eee6accc35820530be exec-id=62b049d16d1fe03c193295329e7055b3e675a5e94b9566eee6accc35820530be exec-pid=8396
DEBU[2020-12-04T22:38:00.170037297+08:00] event module=libcontainerd namespace=moby topic=/tasks/delete
INFO[2020-12-04T22:38:00.170061445+08:00] ignoring event module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
DEBU[0249] event published module="containerd/containers" ns=moby topic="/containers/delete" type=containerd.events.ContainerDelete
DEBU[2020-12-04T22:38:00.199820461+08:00] container mounted via layerStore: &{/app/kube/docker/overlay2/2abb109b107ef7f0e5c31b1a100b446234118ae38afe43977c8c718f115cdfd6/merged 0x2f42600 0x2f42600}
DEBU[2020-12-04T22:38:00.208519823+08:00] Option Experimental: false
DEBU[2020-12-04T22:38:00.208542167+08:00] Option DefaultDriver: bridge
DEBU[2020-12-04T22:38:00.208549815+08:00] Option DefaultNetwork: bridge
DEBU[2020-12-04T22:38:00.208557480+08:00] Network Control Plane MTU: 1500
DEBU[2020-12-04T22:38:00.245647071+08:00] /sbin/iptables, [--wait -t nat -D PREROUTING -m addrtype --dst-type LOCAL -j DOCKER]
DEBU[2020-12-04T22:38:00.247719844+08:00] /sbin/iptables, [--wait -t nat -D OUTPUT -m addrtype --dst-type LOCAL ! --dst 127.0.0.0/8 -j DOCKER]
DEBU[2020-12-04T22:38:00.249828613+08:00] /sbin/iptables, [--wait -t nat -D OUTPUT -m addrtype --dst-type LOCAL -j DOCKER]
DEBU[2020-12-04T22:38:00.251439314+08:00] /sbin/iptables, [--wait -t nat -D PREROUTING]
...
服务器上有安全狗，可能和安全狗有关系。
Recommend
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK