Docker can become unresponsive, but containers continue to run

Docker containers wrap a piece of software in a complete filesystem that contains everything needed to run: code, runtime, system tools, system libraries – anything that can be installed on a server. This guarantees that the software will always run the same, regardless of its environment.

Moderator: Lillian.W@AST

User avatar
Nazar78
Posts: 2002
youtube meble na wymiar Warszawa
Joined: Wed Jul 17, 2019 10:21 pm
Location: Singapore
Contact:

Re: Docker can become unresponsive, but containers continue to run

Post by Nazar78 »

You're still having issues after ADM 4.2.0.RE71? I still can't seems to replicate this.

I'm interested to know when you stop using the start-stop.sh, were there any docker processes left? Should be none `ps -ef|grep docker`. How about the mounts? `mount|grep docker` There should be only one left: nsfs on /var/run/docker/netns/default type nsfs (rw). Anything left hanging here could be the issue.

As for the script, temporarily replace all the ">/dev/null 2>&1" with ">>/root/debug.txt 2>&1" so the stdout/err goes into /root/debug.txt for your findings when it happens.
AS5304T - 16GB DDR4 - ADM-OS modded on 2GB RAM
Internal:
- 4x10TB Toshiba RAID10 Ext4-Journal=Off
External 5 Bay USB3:
- 4x2TB Seagate modded RAID0 Btrfs-Compression
- 480GB Intel SSD for modded dm-cache (initramfs auto update patch) and Apps

When posting, consider checking the box "Notify me when a reply is posted" to get faster response
ilike2burnthing
Posts: 379
Joined: Thu Apr 09, 2020 8:01 pm

Re: Docker can become unresponsive, but containers continue to run

Post by ilike2burnthing »

Yes, that last time was after having updated to ADM 4.2.0.RE71.

I'll have to wait for the next time Docker becomes unresponsive to check on the processes and mounts. That's assuming the script doesn't solve the issue for me, but if it does at least I have changed the error location, so I'll keep an eye on debug.txt as well.
ilike2burnthing
Posts: 379
Joined: Thu Apr 09, 2020 8:01 pm

Re: Docker can become unresponsive, but containers continue to run

Post by ilike2burnthing »

Had another failure, script got to it before I could. Content of debug.txt:

Code: Select all

Starting Docker daemon...
rpcbind: another rpcbind is already running. Aborting
time="2023-03-09T06:30:05.952320168Z" level=info msg="Starting up"
failed to start daemon: pid file found, ensure docker is not running or delete /var/run/docker.pid
mkdir: can't create directory '/sys/fs/cgroup/systemd': File exists
mount: mounting cgroup on /sys/fs/cgroup/systemd failed: Device or resource busy
Containers not accessible and docker not responding so ran your commands (not quite what you wanted to see I know, as the script had already run):

Code: Select all

/volume1/.@root$ docker ps -a
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

/volume1/.@root$ ps -ef|grep docker
 4248 root      0:00 grep docker

/volume1/.@root$ mount|grep docker
/dev/md1 on /volume1/.@plugins/AppCentral/docker-ce/docker_lib type btrfs (rw,relatime,ssd,discard,space_cache,subvolid=259,subvol=/.@plugins)
/dev/md1 on /volume1/.@plugins/AppCentral/docker-ce/docker_lib/btrfs type btrfs (rw,relatime,ssd,discard,space_cache,subvolid=259,subvol=/.@plugins)
nsfs on /var/run/docker/netns/2ea2113c8b08 type nsfs (rw)
nsfs on /var/run/docker/netns/eda0074c67b8 type nsfs (rw)
nsfs on /var/run/docker/netns/69288b1921c9 type nsfs (rw)
nsfs on /var/run/docker/netns/793151b05e17 type nsfs (rw)
nsfs on /var/run/docker/netns/fdfd16ffd65d type nsfs (rw)
nsfs on /var/run/docker/netns/415945e347ba type nsfs (rw)
nsfs on /var/run/docker/netns/6c20777f76b7 type nsfs (rw)
nsfs on /var/run/docker/netns/93d371c29138 type nsfs (rw)
nsfs on /var/run/docker/netns/f5474e4b393d type nsfs (rw)
5mins later, cron job ran again and debug.txt was added to:

Code: Select all

sh: can't kill pid 6745: No such process
kill: you need to specify whom to kill
Starting Docker daemon...
rpcbind: another rpcbind is already running. Aborting
time="2023-03-09T06:35:05.881651305Z" level=info msg="Starting up"
time="2023-03-09T06:35:05.885742205Z" level=warning msg="could not change group /var/run/docker.sock to docker: group docker not found"
time="2023-03-09T06:35:05.890629605Z" level=info msg="libcontainerd: started new containerd process" pid=7026
time="2023-03-09T06:35:05.890759268Z" level=info msg="parsed scheme: \"unix\"" module=grpc
time="2023-03-09T06:35:05.890798618Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
time="2023-03-09T06:35:05.890892855Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}" module=grpc
time="2023-03-09T06:35:05.890944155Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
time="2023-03-09T06:35:05Z" level=warning msg="containerd config version `1` has been deprecated and will be removed in containerd v2.0, please switch to version `2`, see https://github.com/containerd/containerd/blob/main/docs/PLUGINS.md#version-header"
time="2023-03-09T06:35:05.955861943Z" level=info msg="starting containerd" revision=78f51771157abb6c9ed224c22013cdf09962315d version=v1.6.13
time="2023-03-09T06:35:05.991446430Z" level=info msg="loading plugin \"io.containerd.content.v1.content\"..." type=io.containerd.content.v1
time="2023-03-09T06:35:05.991689880Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.aufs\"..." type=io.containerd.snapshotter.v1
time="2023-03-09T06:35:06.003002755Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.aufs\"..." error="aufs is not supported (modprobe aufs failed: exit status 1 \"modprobe: module 'aufs' not found\\n\"): skip plugin" type=io.containerd.snapshotter.v1
time="2023-03-09T06:35:06.003163243Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." type=io.containerd.snapshotter.v1
time="2023-03-09T06:35:06.004263055Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.devmapper\"..." type=io.containerd.snapshotter.v1
time="2023-03-09T06:35:06.004402480Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.devmapper" error="devmapper not configured"
time="2023-03-09T06:35:06.004453105Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.native\"..." type=io.containerd.snapshotter.v1
time="2023-03-09T06:35:06.004581605Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.overlayfs\"..." type=io.containerd.snapshotter.v1
time="2023-03-09T06:35:06.005037380Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.zfs\"..." type=io.containerd.snapshotter.v1
time="2023-03-09T06:35:06.005925630Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.zfs\"..." error="path /volume1/.@plugins/AppCentral/docker-ce/docker_lib/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
time="2023-03-09T06:35:06.006051105Z" level=info msg="loading plugin \"io.containerd.metadata.v1.bolt\"..." type=io.containerd.metadata.v1
time="2023-03-09T06:35:06.006169905Z" level=warning msg="could not use snapshotter devmapper in metadata plugin" error="devmapper not configured"
time="2023-03-09T06:35:06.006233793Z" level=info msg="metadata content store policy set" policy=shared
time="2023-03-09T06:35:06.006766018Z" level=info msg="loading plugin \"io.containerd.differ.v1.walking\"..." type=io.containerd.differ.v1
time="2023-03-09T06:35:06.006847268Z" level=info msg="loading plugin \"io.containerd.event.v1.exchange\"..." type=io.containerd.event.v1
time="2023-03-09T06:35:06.006908943Z" level=info msg="loading plugin \"io.containerd.gc.v1.scheduler\"..." type=io.containerd.gc.v1
time="2023-03-09T06:35:06.007010793Z" level=info msg="loading plugin \"io.containerd.service.v1.introspection-service\"..." type=io.containerd.service.v1
time="2023-03-09T06:35:06.007065205Z" level=info msg="loading plugin \"io.containerd.service.v1.containers-service\"..." type=io.containerd.service.v1
time="2023-03-09T06:35:06.007147968Z" level=info msg="loading plugin \"io.containerd.service.v1.content-service\"..." type=io.containerd.service.v1
time="2023-03-09T06:35:06.007199243Z" level=info msg="loading plugin \"io.containerd.service.v1.diff-service\"..." type=io.containerd.service.v1
time="2023-03-09T06:35:06.007855243Z" level=info msg="loading plugin \"io.containerd.service.v1.images-service\"..." type=io.containerd.service.v1
time="2023-03-09T06:35:06.007941080Z" level=info msg="loading plugin \"io.containerd.service.v1.leases-service\"..." type=io.containerd.service.v1
time="2023-03-09T06:35:06.008009180Z" level=info msg="loading plugin \"io.containerd.service.v1.namespaces-service\"..." type=io.containerd.service.v1
time="2023-03-09T06:35:06.008063180Z" level=info msg="loading plugin \"io.containerd.service.v1.snapshots-service\"..." type=io.containerd.service.v1
time="2023-03-09T06:35:06.008174730Z" level=info msg="loading plugin \"io.containerd.runtime.v1.linux\"..." type=io.containerd.runtime.v1
time="2023-03-09T06:35:06.008424255Z" level=info msg="loading plugin \"io.containerd.runtime.v2.task\"..." type=io.containerd.runtime.v2
time="2023-03-09T06:35:06.010013680Z" level=warning msg="cleaning up after shim disconnected" id=01be38262ce1c1f707e7e923644c7ac58228f0149645492fc089972f5f29bed0 namespace=moby
time="2023-03-09T06:35:06.010079980Z" level=info msg="cleaning up dead shim"
time="2023-03-09T06:35:06.160800805Z" level=warning msg="cleanup warnings time=\"2023-03-09T06:35:06Z\" level=info msg=\"starting signal loop\" namespace=moby pid=7036 runtime=io.containerd.runc.v2\n"
time="2023-03-09T06:35:06.164331655Z" level=warning msg="cleaning up after shim disconnected" id=08da28bbb15d30de0ad97ebd48d42333c3e83ecff4a3e7b50f8e5b8a1b4ae849 namespace=moby
time="2023-03-09T06:35:06.164411143Z" level=info msg="cleaning up dead shim"
time="2023-03-09T06:35:06.205154468Z" level=warning msg="cleanup warnings time=\"2023-03-09T06:35:06Z\" level=info msg=\"starting signal loop\" namespace=moby pid=7076 runtime=io.containerd.runc.v2\n"
time="2023-03-09T06:35:06.207630855Z" level=warning msg="cleaning up after shim disconnected" id=35826350d1429f60f11032f7c21fb816e89a472d2722503ea5ce30b9321e59e6 namespace=moby
time="2023-03-09T06:35:06.207717493Z" level=info msg="cleaning up dead shim"
time="2023-03-09T06:35:06.239441130Z" level=warning msg="cleanup warnings time=\"2023-03-09T06:35:06Z\" level=info msg=\"starting signal loop\" namespace=moby pid=7092 runtime=io.containerd.runc.v2\n"
time="2023-03-09T06:35:06.241514718Z" level=warning msg="cleaning up after shim disconnected" id=38fe954eaf5abb78d9aed4e9ec7efc790486d4ff6d5e38cca2197cf8cc3f10c5 namespace=moby
time="2023-03-09T06:35:06.241588943Z" level=info msg="cleaning up dead shim"
time="2023-03-09T06:35:06.378694455Z" level=warning msg="cleanup warnings time=\"2023-03-09T06:35:06Z\" level=info msg=\"starting signal loop\" namespace=moby pid=7107 runtime=io.containerd.runc.v2\n"
time="2023-03-09T06:35:06.380388080Z" level=warning msg="cleaning up after shim disconnected" id=4ff27400801a009cc9a923ac346e625dfeadceb19c12538445bf3cc010d1293a namespace=moby
time="2023-03-09T06:35:06.380492743Z" level=info msg="cleaning up dead shim"
time="2023-03-09T06:35:06.510076205Z" level=warning msg="cleanup warnings time=\"2023-03-09T06:35:06Z\" level=info msg=\"starting signal loop\" namespace=moby pid=7123 runtime=io.containerd.runc.v2\n"
time="2023-03-09T06:35:06.511739105Z" level=warning msg="cleaning up after shim disconnected" id=76e95f3cdfc354a588077b19bb16d016916551e34b7b3830f24eaf656133d520 namespace=moby
time="2023-03-09T06:35:06.511840705Z" level=info msg="cleaning up dead shim"
time="2023-03-09T06:35:06.541164593Z" level=warning msg="cleanup warnings time=\"2023-03-09T06:35:06Z\" level=info msg=\"starting signal loop\" namespace=moby pid=7137 runtime=io.containerd.runc.v2\n"
time="2023-03-09T06:35:06.542696218Z" level=warning msg="cleaning up after shim disconnected" id=8e4d71b8a034a3b13dc50a3677dc2aec630599ed01c61b53763aa0ed3a84e81e namespace=moby
time="2023-03-09T06:35:06.542762155Z" level=info msg="cleaning up dead shim"
time="2023-03-09T06:35:06.675839905Z" level=warning msg="cleanup warnings time=\"2023-03-09T06:35:06Z\" level=info msg=\"starting signal loop\" namespace=moby pid=7151 runtime=io.containerd.runc.v2\n"
time="2023-03-09T06:35:06.685608668Z" level=warning msg="cleaning up after shim disconnected" id=e9415dae7d140251b5dac30d8334d55b9150e76e22bf5252226ec55e9f2e33e8 namespace=moby
time="2023-03-09T06:35:06.685686618Z" level=info msg="cleaning up dead shim"
time="2023-03-09T06:35:06.822445330Z" level=warning msg="cleanup warnings time=\"2023-03-09T06:35:06Z\" level=info msg=\"starting signal loop\" namespace=moby pid=7172 runtime=io.containerd.runc.v2\n"
time="2023-03-09T06:35:06.824177093Z" level=warning msg="cleaning up after shim disconnected" id=e9b93c4a6b52ccb4028d5f59e63960db064ae0635a6badd89430f446e132e181 namespace=moby
time="2023-03-09T06:35:06.824277530Z" level=info msg="cleaning up dead shim"
time="2023-03-09T06:35:06.891622030Z" level=warning msg="grpc: addrConn.createTransport failed to connect to {unix:///var/run/docker/containerd/containerd.sock  <nil> 0 <nil>}. Err :connection error: desc = \"transport: error while dialing: dial unix:///var/run/docker/containerd/containerd.sock: timeout\". Reconnecting..." module=grpc
time="2023-03-09T06:35:06.957871480Z" level=warning msg="cleanup warnings time=\"2023-03-09T06:35:06Z\" level=info msg=\"starting signal loop\" namespace=moby pid=7194 runtime=io.containerd.runc.v2\n"
time="2023-03-09T06:35:06.959308730Z" level=info msg="loading plugin \"io.containerd.monitor.v1.cgroups\"..." type=io.containerd.monitor.v1
time="2023-03-09T06:35:06.960761180Z" level=info msg="loading plugin \"io.containerd.service.v1.tasks-service\"..." type=io.containerd.service.v1
time="2023-03-09T06:35:06.961158905Z" level=info msg="loading plugin \"io.containerd.grpc.v1.introspection\"..." type=io.containerd.grpc.v1
time="2023-03-09T06:35:06.961308443Z" level=info msg="loading plugin \"io.containerd.internal.v1.restart\"..." type=io.containerd.internal.v1
time="2023-03-09T06:35:06.961494580Z" level=info msg="loading plugin \"io.containerd.grpc.v1.containers\"..." type=io.containerd.grpc.v1
time="2023-03-09T06:35:06.961565018Z" level=info msg="loading plugin \"io.containerd.grpc.v1.content\"..." type=io.containerd.grpc.v1
time="2023-03-09T06:35:06.961635243Z" level=info msg="loading plugin \"io.containerd.grpc.v1.diff\"..." type=io.containerd.grpc.v1
time="2023-03-09T06:35:06.961699180Z" level=info msg="loading plugin \"io.containerd.grpc.v1.events\"..." type=io.containerd.grpc.v1
time="2023-03-09T06:35:06.961771105Z" level=info msg="loading plugin \"io.containerd.grpc.v1.healthcheck\"..." type=io.containerd.grpc.v1
time="2023-03-09T06:35:06.961839068Z" level=info msg="loading plugin \"io.containerd.grpc.v1.images\"..." type=io.containerd.grpc.v1
time="2023-03-09T06:35:06.961904343Z" level=info msg="loading plugin \"io.containerd.grpc.v1.leases\"..." type=io.containerd.grpc.v1
time="2023-03-09T06:35:06.961969893Z" level=info msg="loading plugin \"io.containerd.grpc.v1.namespaces\"..." type=io.containerd.grpc.v1
time="2023-03-09T06:35:06.962044193Z" level=info msg="loading plugin \"io.containerd.internal.v1.opt\"..." type=io.containerd.internal.v1
time="2023-03-09T06:35:06.962640518Z" level=info msg="loading plugin \"io.containerd.grpc.v1.snapshots\"..." type=io.containerd.grpc.v1
time="2023-03-09T06:35:06.962723080Z" level=info msg="loading plugin \"io.containerd.grpc.v1.tasks\"..." type=io.containerd.grpc.v1
time="2023-03-09T06:35:06.962841205Z" level=info msg="loading plugin \"io.containerd.grpc.v1.version\"..." type=io.containerd.grpc.v1
time="2023-03-09T06:35:06.962934255Z" level=info msg="loading plugin \"io.containerd.tracing.processor.v1.otlp\"..." type=io.containerd.tracing.processor.v1
time="2023-03-09T06:35:06.963016705Z" level=info msg="skip loading plugin \"io.containerd.tracing.processor.v1.otlp\"..." error="no OpenTelemetry endpoint: skip plugin" type=io.containerd.tracing.processor.v1
time="2023-03-09T06:35:06.963077405Z" level=info msg="loading plugin \"io.containerd.internal.v1.tracing\"..." type=io.containerd.internal.v1
time="2023-03-09T06:35:06.963171968Z" level=error msg="failed to initialize a tracing processor \"otlp\"" error="no OpenTelemetry endpoint: skip plugin"
time="2023-03-09T06:35:06.964816730Z" level=info msg=serving... address=/var/run/docker/containerd/containerd-debug.sock
time="2023-03-09T06:35:06.965001018Z" level=info msg=serving... address=/var/run/docker/containerd/containerd.sock.ttrpc
time="2023-03-09T06:35:06.965144880Z" level=info msg=serving... address=/var/run/docker/containerd/containerd.sock
time="2023-03-09T06:35:06.965197843Z" level=info msg="containerd successfully booted in 1.014801s"
time="2023-03-09T06:35:07.898163605Z" level=info msg="parsed scheme: \"unix\"" module=grpc
time="2023-03-09T06:35:07.898231905Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
time="2023-03-09T06:35:07.898286480Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}" module=grpc
time="2023-03-09T06:35:07.898323330Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
time="2023-03-09T06:35:07.902586480Z" level=info msg="parsed scheme: \"unix\"" module=grpc
time="2023-03-09T06:35:07.902659018Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
time="2023-03-09T06:35:07.902714518Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}" module=grpc
time="2023-03-09T06:35:07.902751618Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
time="2023-03-09T06:35:07.907164443Z" level=info msg="[graphdriver] using prior storage driver: btrfs"
time="2023-03-09T06:35:07.957068118Z" level=warning msg="Your kernel does not support cgroup blkio weight"
time="2023-03-09T06:35:07.957161255Z" level=warning msg="Your kernel does not support cgroup blkio weight_device"
time="2023-03-09T06:35:07.957486605Z" level=info msg="Loading containers: start."
time="2023-03-09T06:35:08.229917955Z" level=warning msg="Running modprobe bridge br_netfilter failed with message: modprobe: invalid option -- 'a'\nBusyBox v1.31.1 (2023-02-07 03:01:23 CST) multi-call binary.\n\nUsage: modprobe [-rq] MODULE [SYMBOL=VALUE]...\n\n\t-r\tRemove MODULE\n\t-q\tQuiet\n, error: exit status 1"
time="2023-03-09T06:35:08.260204218Z" level=warning msg="Failed to find ip6tables: exec: \"ip6tables\": executable file not found in $PATH"
time="2023-03-09T06:35:08.448845418Z" level=warning msg="Could not load necessary modules for Conntrack: Running modprobe nf_conntrack failed with message: `modprobe: invalid option -- 'a'\nBusyBox v1.31.1 (2023-02-07 03:01:23 CST) multi-call binary.\n\nUsage: modprobe [-rq] MODULE [SYMBOL=VALUE]...\n\n\t-r\tRemove MODULE\n\t-q\tQuiet`, error: exit status 1"
time="2023-03-09T06:35:11.287788355Z" level=info msg="Removing stale sandbox eda0074c67b8a25f8396ca6c773deb2723af5450e0f7c431995bd29db16f161a (35826350d1429f60f11032f7c21fb816e89a472d2722503ea5ce30b9321e59e6)"
time="2023-03-09T06:35:11.440865618Z" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint 94128d0ef41ccf94ff2c46aaba654db098580a00ea5af84437533fc0d7a96e89 ec495054a6f6d489d5ae97dc3ba5f9e3ff619f987668275d420e233cdf03f46a], retrying...."
time="2023-03-09T06:35:11.941250743Z" level=info msg="Removing stale sandbox f5474e4b393d12650a02041b2bf3542a8e3190b68abfa0022eda3750832b8493 (e9415dae7d140251b5dac30d8334d55b9150e76e22bf5252226ec55e9f2e33e8)"
time="2023-03-09T06:35:12.095887955Z" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint 94128d0ef41ccf94ff2c46aaba654db098580a00ea5af84437533fc0d7a96e89 9762410bdee322de70c78767df38c07716a402cdbd734bdd9d9cd42b21e9ba45], retrying...."
time="2023-03-09T06:35:12.505974218Z" level=info msg="Removing stale sandbox fdfd16ffd65d6283e58c68aa6de8156945739fd71f5537fe66e67911ca26377f (4ff27400801a009cc9a923ac346e625dfeadceb19c12538445bf3cc010d1293a)"
time="2023-03-09T06:35:12.675373018Z" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint 94128d0ef41ccf94ff2c46aaba654db098580a00ea5af84437533fc0d7a96e89 7e2ea7f5d1b7fddcfef3d0650bfc2b7ba0dd1d6b9191d934f6b88aa92fc71a76], retrying...."
time="2023-03-09T06:35:13.088777093Z" level=info msg="Removing stale sandbox 2ea2113c8b08acdeb809f6be3b98e2835a829382c26936d52850fac695a07e97 (08da28bbb15d30de0ad97ebd48d42333c3e83ecff4a3e7b50f8e5b8a1b4ae849)"
time="2023-03-09T06:35:13.107176105Z" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint 90ef2ca1a67898ca5111a23df2ce373ae497eae4c8dea001275201a6cbb2ce49 1d44ac8db092403f7e04fa097abb68bbae0f5b2516be8cf226735151c5ce7f60], retrying...."
time="2023-03-09T06:35:13.519772530Z" level=info msg="Removing stale sandbox 69288b1921c9e2a891204ee74c8c456499bb1b8730c7e10426e1c89dcbea5582 (76e95f3cdfc354a588077b19bb16d016916551e34b7b3830f24eaf656133d520)"
time="2023-03-09T06:35:13.926835480Z" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint 90ef2ca1a67898ca5111a23df2ce373ae497eae4c8dea001275201a6cbb2ce49 8dc89f3557ff3d002fd1a247bab4c01dc5984f6a087e262ecd27ebe844a4ccc3], retrying...."
time="2023-03-09T06:35:14.408280768Z" level=info msg="Removing stale sandbox 93d371c29138d5b5040b3bae313936316e478df5a83ae3f3e58b2034eee415b0 (38fe954eaf5abb78d9aed4e9ec7efc790486d4ff6d5e38cca2197cf8cc3f10c5)"
time="2023-03-09T06:35:14.578800668Z" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint 94128d0ef41ccf94ff2c46aaba654db098580a00ea5af84437533fc0d7a96e89 6277dbb651229089911aca44941cca3a5a657bc74d787c868527a9f36813cb7b], retrying...."
time="2023-03-09T06:35:15.053580230Z" level=info msg="Removing stale sandbox 415945e347ba5036454f7ccecaec001e8c58fe922d7b09ce8423bc7da3dda41e (8e4d71b8a034a3b13dc50a3677dc2aec630599ed01c61b53763aa0ed3a84e81e)"
time="2023-03-09T06:35:15.204900643Z" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint 94128d0ef41ccf94ff2c46aaba654db098580a00ea5af84437533fc0d7a96e89 50fb391758e5f2f57b341b5b02f113509f6eacdc5f358160903013c524b98006], retrying...."
time="2023-03-09T06:35:15.613840105Z" level=info msg="Removing stale sandbox 6c20777f76b73eb7695b094c0a5f0ebc8ba228f97d98818cf08cf37496ff11f0 (e9b93c4a6b52ccb4028d5f59e63960db064ae0635a6badd89430f446e132e181)"
mkdir: can't create directory '/sys/fs/cgroup/systemd': File exists
mount: mounting cgroup on /sys/fs/cgroup/systemd failed: Device or resource busy
time="2023-03-09T06:35:15.768228193Z" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint 94128d0ef41ccf94ff2c46aaba654db098580a00ea5af84437533fc0d7a96e89 df118316e6f8a42ce5a57856f89905e56ba8642e19606b79c50c80cf4b23aa0c], retrying...."
time="2023-03-09T06:35:16.121719068Z" level=info msg="Removing stale sandbox 793151b05e17235549e07eff248248a319444b53fdc87b6bc7844079e01d6b15 (01be38262ce1c1f707e7e923644c7ac58228f0149645492fc089972f5f29bed0)"
time="2023-03-09T06:35:16.295440680Z" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint 94128d0ef41ccf94ff2c46aaba654db098580a00ea5af84437533fc0d7a96e89 796a0259f95dad3b31bb00248281cd3e89bca1db4e63a82b0e3a3a8ae446b8cb], retrying...."
time="2023-03-09T06:35:16.561516430Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
time="2023-03-09T06:35:18.055153068Z" level=warning msg="Failed to allocate and map port 8191-8191:  (iptables failed: iptables -t filter -A DOCKER ! -i br-94128d0ef41c -o br-94128d0ef41c -p tcp -d 172.18.0.8 --dport 8191 -j ACCEPT: Another app is currently holding the xtables lock. Perhaps you want to use the -w option?\n (exit status 4))"
time="2023-03-09T06:35:18.183797580Z" level=warning msg="Failed to allocate and map port 17878-17878:  (iptables failed: iptables -t nat -A DOCKER -p tcp -d 0/0 --dport 17878 -j DNAT --to-destination 172.18.0.6:7878 ! -i br-94128d0ef41c: Another app is currently holding the xtables lock. Perhaps you want to use the -w option?\n (exit status 4))"
time="2023-03-09T06:35:18.370834780Z" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
time="2023-03-09T06:35:18.371162393Z" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
time="2023-03-09T06:35:18.371838193Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
time="2023-03-09T06:35:18.376553043Z" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/08da28bbb15d30de0ad97ebd48d42333c3e83ecff4a3e7b50f8e5b8a1b4ae849 pid=8809 runtime=io.containerd.runc.v2
time="2023-03-09T06:35:18.555330805Z" level=error msg="01be38262ce1c1f707e7e923644c7ac58228f0149645492fc089972f5f29bed0 cleanup: failed to delete container from containerd: no such container"
time="2023-03-09T06:35:18.555428543Z" level=error msg="failed to start container" container=01be38262ce1c1f707e7e923644c7ac58228f0149645492fc089972f5f29bed0 error="driver failed programming external connectivity on endpoint FlareSolverr (7e09820a28b7400d5367d180cb08c7d2c538583be36a7338664bae7807d5b538):  (iptables failed: iptables -t filter -A DOCKER ! -i br-94128d0ef41c -o br-94128d0ef41c -p tcp -d 172.18.0.8 --dport 8191 -j ACCEPT: Another app is currently holding the xtables lock. Perhaps you want to use the -w option?\n (exit status 4))"
time="2023-03-09T06:35:18.985085368Z" level=warning msg="Failed to allocate and map port 9091-9091:  (iptables failed: iptables -t nat -A DOCKER -p tcp -d 0/0 --dport 9091 -j DNAT --to-destination 172.18.0.2:9091 ! -i br-94128d0ef41c: Another app is currently holding the xtables lock. Perhaps you want to use the -w option?\n (exit status 4))"
time="2023-03-09T06:35:19.039482418Z" level=error msg="e9415dae7d140251b5dac30d8334d55b9150e76e22bf5252226ec55e9f2e33e8 cleanup: failed to delete container from containerd: no such container"
time="2023-03-09T06:35:19.039567905Z" level=error msg="failed to start container" container=e9415dae7d140251b5dac30d8334d55b9150e76e22bf5252226ec55e9f2e33e8 error="driver failed programming external connectivity on endpoint Radarr (a7c340bd4fe771a010c9787256baa33667364f1150a03c0f419bb4166791d800):  (iptables failed: iptables -t nat -A DOCKER -p tcp -d 0/0 --dport 17878 -j DNAT --to-destination 172.18.0.6:7878 ! -i br-94128d0ef41c: Another app is currently holding the xtables lock. Perhaps you want to use the -w option?\n (exit status 4))"
time="2023-03-09T06:35:19.557417355Z" level=warning msg="Failed to allocate and map port 18989-18989:  (iptables failed: iptables -t nat -A DOCKER -p tcp -d 0/0 --dport 18989 -j DNAT --to-destination 172.18.0.5:8989 ! -i br-94128d0ef41c: Another app is currently holding the xtables lock. Perhaps you want to use the -w option?\n (exit status 4))"
time="2023-03-09T06:35:19.602951530Z" level=error msg="35826350d1429f60f11032f7c21fb816e89a472d2722503ea5ce30b9321e59e6 cleanup: failed to delete container from containerd: no such container"
time="2023-03-09T06:35:19.603036530Z" level=error msg="failed to start container" container=35826350d1429f60f11032f7c21fb816e89a472d2722503ea5ce30b9321e59e6 error="driver failed programming external connectivity on endpoint TransmissionVPN (e65fa20c45cc9e3f0b217f9f1319a39b569f2d05c7b5072022f3bd09a0d2cc56):  (iptables failed: iptables -t nat -A DOCKER -p tcp -d 0/0 --dport 9091 -j DNAT --to-destination 172.18.0.2:9091 ! -i br-94128d0ef41c: Another app is currently holding the xtables lock. Perhaps you want to use the -w option?\n (exit status 4))"
time="2023-03-09T06:35:20.117623230Z" level=warning msg="Failed to allocate and map port 19900-19900:  (iptables failed: iptables -t nat -A DOCKER -p tcp -d 0/0 --dport 19900 -j DNAT --to-destination 172.17.0.2:9000 ! -i docker0: Another app is currently holding the xtables lock. Perhaps you want to use the -w option?\n (exit status 4))"
time="2023-03-09T06:35:20.141795243Z" level=error msg="4ff27400801a009cc9a923ac346e625dfeadceb19c12538445bf3cc010d1293a cleanup: failed to delete container from containerd: no such container"
time="2023-03-09T06:35:20.141884530Z" level=error msg="failed to start container" container=4ff27400801a009cc9a923ac346e625dfeadceb19c12538445bf3cc010d1293a error="driver failed programming external connectivity on endpoint Sonarr (d55ec85b9c5d5e44e054b5bb211352f9cbf60b18800ca58f4768d09176b34457):  (iptables failed: iptables -t nat -A DOCKER -p tcp -d 0/0 --dport 18989 -j DNAT --to-destination 172.18.0.5:8989 ! -i br-94128d0ef41c: Another app is currently holding the xtables lock. Perhaps you want to use the -w option?\n (exit status 4))"
time="2023-03-09T06:35:20.818917530Z" level=error msg="76e95f3cdfc354a588077b19bb16d016916551e34b7b3830f24eaf656133d520 cleanup: failed to delete container from containerd: no such container"
time="2023-03-09T06:35:20.819000893Z" level=error msg="failed to start container" container=76e95f3cdfc354a588077b19bb16d016916551e34b7b3830f24eaf656133d520 error="driver failed programming external connectivity on endpoint PortainerCE (15e8c3c4b450c787bf94e20785be0a3bedea2b94a41642b41234ba27e1c0e514):  (iptables failed: iptables -t nat -A DOCKER -p tcp -d 0/0 --dport 19900 -j DNAT --to-destination 172.17.0.2:9000 ! -i docker0: Another app is currently holding the xtables lock. Perhaps you want to use the -w option?\n (exit status 4))"
time="2023-03-09T06:35:21.216938368Z" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
time="2023-03-09T06:35:21.217304118Z" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
time="2023-03-09T06:35:21.217357730Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
time="2023-03-09T06:35:21.219485005Z" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/38fe954eaf5abb78d9aed4e9ec7efc790486d4ff6d5e38cca2197cf8cc3f10c5 pid=9447 runtime=io.containerd.runc.v2
time="2023-03-09T06:35:21.799080630Z" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
time="2023-03-09T06:35:21.799419343Z" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
time="2023-03-09T06:35:21.799455593Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
time="2023-03-09T06:35:21.799859455Z" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/8e4d71b8a034a3b13dc50a3677dc2aec630599ed01c61b53763aa0ed3a84e81e pid=9562 runtime=io.containerd.runc.v2
time="2023-03-09T06:35:22.110183993Z" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
time="2023-03-09T06:35:22.110400068Z" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
time="2023-03-09T06:35:22.110442968Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
time="2023-03-09T06:35:22.116797955Z" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/e9b93c4a6b52ccb4028d5f59e63960db064ae0635a6badd89430f446e132e181 pid=9652 runtime=io.containerd.runc.v2
time="2023-03-09T06:35:22Z" level=warning msg="Failed to find ip6tables: exec: \"ip6tables\": executable file not found in $PATH"
time="2023-03-09T06:35:23Z" level=warning msg="Failed to find ip6tables: exec: \"ip6tables\": executable file not found in $PATH"
time="2023-03-09T06:35:23Z" level=warning msg="Failed to find ip6tables: exec: \"ip6tables\": executable file not found in $PATH"
time="2023-03-09T06:35:23.723425805Z" level=info msg="Loading containers: done."
time="2023-03-09T06:35:23.727979655Z" level=warning msg="Could not get operating system name: Error opening /usr/lib/os-release: open /usr/lib/os-release: no such file or directory"
time="2023-03-09T06:35:23.729559455Z" level=warning msg="Could not get operating system version: Error opening /usr/lib/os-release: open /usr/lib/os-release: no such file or directory"
time="2023-03-09T06:35:23.761703480Z" level=info msg="Docker daemon" commit=42c8b31 graphdriver(s)=btrfs version=20.10.22
time="2023-03-09T06:35:23.761946468Z" level=info msg="Daemon has completed initialization"
time="2023-03-09T06:35:23.930223068Z" level=info msg="API listen on /var/run/docker.sock"
ilike2burnthing
Posts: 379
Joined: Thu Apr 09, 2020 8:01 pm

Re: Docker can become unresponsive, but containers continue to run

Post by ilike2burnthing »

All containers working except Portainer (shows as running but not accessible). Stopping and starting the container doesn't work. Enabling and disabling Portainer in App Central doesn't work.

debug.txt has also been added to every 5 mins:

Code: Select all

time="2023-03-09T06:45:52.411566168Z" level=info msg="ignoring event" container=e9b93c4a6b52ccb4028d5f59e63960db064ae0635a6badd89430f446e132e181 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2023-03-09T06:45:52.415459493Z" level=info msg="shim disconnected" id=e9b93c4a6b52ccb4028d5f59e63960db064ae0635a6badd89430f446e132e181
time="2023-03-09T06:45:52.415617005Z" level=warning msg="cleaning up after shim disconnected" id=e9b93c4a6b52ccb4028d5f59e63960db064ae0635a6badd89430f446e132e181 namespace=moby
time="2023-03-09T06:45:52.415656305Z" level=info msg="cleaning up dead shim"
time="2023-03-09T06:45:52.452957205Z" level=warning msg="cleanup warnings time=\"2023-03-09T06:45:52Z\" level=info msg=\"starting signal loop\" namespace=moby pid=6101 runtime=io.containerd.runc.v2\n"
time="2023-03-09T06:45:57.942159193Z" level=info msg="ignoring event" container=38fe954eaf5abb78d9aed4e9ec7efc790486d4ff6d5e38cca2197cf8cc3f10c5 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2023-03-09T06:45:57.943426005Z" level=info msg="shim disconnected" id=38fe954eaf5abb78d9aed4e9ec7efc790486d4ff6d5e38cca2197cf8cc3f10c5
time="2023-03-09T06:45:57.943548855Z" level=warning msg="cleaning up after shim disconnected" id=38fe954eaf5abb78d9aed4e9ec7efc790486d4ff6d5e38cca2197cf8cc3f10c5 namespace=moby
time="2023-03-09T06:45:57.943584880Z" level=info msg="cleaning up dead shim"
time="2023-03-09T06:45:57.986240743Z" level=warning msg="cleanup warnings time=\"2023-03-09T06:45:57Z\" level=info msg=\"starting signal loop\" namespace=moby pid=6506 runtime=io.containerd.runc.v2\n"
time="2023-03-09T06:46:00.372510218Z" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
time="2023-03-09T06:46:00.372696980Z" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
time="2023-03-09T06:46:00.372732630Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
time="2023-03-09T06:46:00.373193543Z" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/28a0f835136cd908532ea546e92ffed53d925a1995e40ac2fe3fbf1c3e07ceea pid=6769 runtime=io.containerd.runc.v2
time="2023-03-09T06:46:01Z" level=warning msg="Failed to find ip6tables: exec: \"ip6tables\": executable file not found in $PATH"
time="2023-03-09T06:46:05.822378793Z" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
time="2023-03-09T06:46:05.822673355Z" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
time="2023-03-09T06:46:05.822753218Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
time="2023-03-09T06:46:05.823297805Z" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/dce04991c68337127fb7e345704035d736798c4199a1c24f1e9d9f4bb9a5ac28 pid=7380 runtime=io.containerd.runc.v2
time="2023-03-09T06:46:06Z" level=warning msg="Failed to find ip6tables: exec: \"ip6tables\": executable file not found in $PATH"
time="2023-03-09T06:52:22.308360493Z" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
time="2023-03-09T06:52:22.308590455Z" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
time="2023-03-09T06:52:22.308642680Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
time="2023-03-09T06:52:22.309068468Z" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/e9415dae7d140251b5dac30d8334d55b9150e76e22bf5252226ec55e9f2e33e8 pid=24592 runtime=io.containerd.runc.v2
time="2023-03-09T06:52:22Z" level=warning msg="Failed to find ip6tables: exec: \"ip6tables\": executable file not found in $PATH"
time="2023-03-09T06:52:23.700894105Z" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
time="2023-03-09T06:52:23.701082393Z" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
time="2023-03-09T06:52:23.703241055Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
time="2023-03-09T06:52:23.704251993Z" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/4ff27400801a009cc9a923ac346e625dfeadceb19c12538445bf3cc010d1293a pid=24890 runtime=io.containerd.runc.v2
time="2023-03-09T06:52:24Z" level=warning msg="Failed to find ip6tables: exec: \"ip6tables\": executable file not found in $PATH"
time="2023-03-09T06:52:25.494281643Z" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
time="2023-03-09T06:52:25.494522180Z" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
time="2023-03-09T06:52:25.494573893Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
time="2023-03-09T06:52:25.495724080Z" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/01be38262ce1c1f707e7e923644c7ac58228f0149645492fc089972f5f29bed0 pid=25229 runtime=io.containerd.runc.v2
time="2023-03-09T06:52:26Z" level=warning msg="Failed to find ip6tables: exec: \"ip6tables\": executable file not found in $PATH"
time="2023-03-09T06:52:28.332775405Z" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
time="2023-03-09T06:52:28.333113693Z" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
time="2023-03-09T06:52:28.333196743Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
time="2023-03-09T06:52:28.336416530Z" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/76e95f3cdfc354a588077b19bb16d016916551e34b7b3830f24eaf656133d520 pid=25646 runtime=io.containerd.runc.v2
time="2023-03-09T06:52:30.165355330Z" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
time="2023-03-09T06:52:30.165709093Z" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
time="2023-03-09T06:52:30.165809193Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
time="2023-03-09T06:52:30.166431718Z" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/35826350d1429f60f11032f7c21fb816e89a472d2722503ea5ce30b9321e59e6 pid=25868 runtime=io.containerd.runc.v2
time="2023-03-09T06:52:30.926838893Z" level=warning msg="Could not get operating system name: Error opening /usr/lib/os-release: open /usr/lib/os-release: no such file or directory"
time="2023-03-09T06:52:30.927442005Z" level=warning msg="Could not get operating system version: Error opening /usr/lib/os-release: open /usr/lib/os-release: no such file or directory"
time="2023-03-09T06:52:30.965996680Z" level=warning msg="Could not get operating system name: Error opening /usr/lib/os-release: open /usr/lib/os-release: no such file or directory"
time="2023-03-09T06:52:30.966178555Z" level=warning msg="Could not get operating system version: Error opening /usr/lib/os-release: open /usr/lib/os-release: no such file or directory"
time="2023-03-09T06:52:31Z" level=warning msg="Failed to find ip6tables: exec: \"ip6tables\": executable file not found in $PATH"
time="2023-03-09T06:57:30.953820505Z" level=warning msg="Could not get operating system name: Error opening /usr/lib/os-release: open /usr/lib/os-release: no such file or directory"
time="2023-03-09T06:57:30.954066505Z" level=warning msg="Could not get operating system version: Error opening /usr/lib/os-release: open /usr/lib/os-release: no such file or directory"
time="2023-03-09T06:58:31.521318543Z" level=info msg="ignoring event" container=76e95f3cdfc354a588077b19bb16d016916551e34b7b3830f24eaf656133d520 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2023-03-09T06:58:31.522472118Z" level=info msg="shim disconnected" id=76e95f3cdfc354a588077b19bb16d016916551e34b7b3830f24eaf656133d520
time="2023-03-09T06:58:31.522620505Z" level=warning msg="cleaning up after shim disconnected" id=76e95f3cdfc354a588077b19bb16d016916551e34b7b3830f24eaf656133d520 namespace=moby
time="2023-03-09T06:58:31.522679005Z" level=info msg="cleaning up dead shim"
time="2023-03-09T06:58:31.552252818Z" level=warning msg="cleanup warnings time=\"2023-03-09T06:58:31Z\" level=info msg=\"starting signal loop\" namespace=moby pid=10729 runtime=io.containerd.runc.v2\n"
time="2023-03-09T06:58:52.119623280Z" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
time="2023-03-09T06:58:52.119957093Z" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
time="2023-03-09T06:58:52.120066818Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
time="2023-03-09T06:58:52.120866780Z" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/76e95f3cdfc354a588077b19bb16d016916551e34b7b3830f24eaf656133d520 pid=11881 runtime=io.containerd.runc.v2
time="2023-03-09T06:58:52.973637443Z" level=warning msg="Could not get operating system name: Error opening /usr/lib/os-release: open /usr/lib/os-release: no such file or directory"
time="2023-03-09T06:58:52.973908555Z" level=warning msg="Could not get operating system version: Error opening /usr/lib/os-release: open /usr/lib/os-release: no such file or directory"
time="2023-03-09T06:58:52.973997343Z" level=warning msg="Could not get operating system name: Error opening /usr/lib/os-release: open /usr/lib/os-release: no such file or directory"
time="2023-03-09T06:58:52.974169905Z" level=warning msg="Could not get operating system version: Error opening /usr/lib/os-release: open /usr/lib/os-release: no such file or directory"
time="2023-03-09T07:03:31.885940618Z" level=info msg="shim disconnected" id=76e95f3cdfc354a588077b19bb16d016916551e34b7b3830f24eaf656133d520
time="2023-03-09T07:03:31.886112080Z" level=warning msg="cleaning up after shim disconnected" id=76e95f3cdfc354a588077b19bb16d016916551e34b7b3830f24eaf656133d520 namespace=moby
time="2023-03-09T07:03:31.886162043Z" level=info msg="cleaning up dead shim"
time="2023-03-09T07:03:31.891721443Z" level=info msg="ignoring event" container=76e95f3cdfc354a588077b19bb16d016916551e34b7b3830f24eaf656133d520 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2023-03-09T07:03:31.942646568Z" level=warning msg="cleanup warnings time=\"2023-03-09T07:03:31Z\" level=info msg=\"starting signal loop\" namespace=moby pid=25017 runtime=io.containerd.runc.v2\n"
time="2023-03-09T07:03:48.058414805Z" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
time="2023-03-09T07:03:48.058749643Z" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
time="2023-03-09T07:03:48.058833668Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
time="2023-03-09T07:03:48.059527843Z" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/76e95f3cdfc354a588077b19bb16d016916551e34b7b3830f24eaf656133d520 pid=26140 runtime=io.containerd.runc.v2
time="2023-03-09T07:03:48.903140155Z" level=warning msg="Could not get operating system name: Error opening /usr/lib/os-release: open /usr/lib/os-release: no such file or directory"
time="2023-03-09T07:03:48.903307143Z" level=warning msg="Could not get operating system version: Error opening /usr/lib/os-release: open /usr/lib/os-release: no such file or directory"
time="2023-03-09T07:03:48.920993780Z" level=warning msg="Could not get operating system name: Error opening /usr/lib/os-release: open /usr/lib/os-release: no such file or directory"
time="2023-03-09T07:03:48.921200118Z" level=warning msg="Could not get operating system version: Error opening /usr/lib/os-release: open /usr/lib/os-release: no such file or directory"time="2023-03-09T07:08:48.902679055Z" level=warning msg="Could not get operating system name: Error opening /usr/lib/os-release: open /usr/lib/os-release: no such file or directory"
time="2023-03-09T07:08:48.902848668Z" level=warning msg="Could not get operating system version: Error opening /usr/lib/os-release: open /usr/lib/os-release: no such file or directory"
ilike2burnthing
Posts: 379
Joined: Thu Apr 09, 2020 8:01 pm

Re: Docker can become unresponsive, but containers continue to run

Post by ilike2burnthing »

Decided to disable and enable Docker in App Central again, seems to be working. I renamed the script so it wouldn't run while this was happening. However, I noticed that debug.txt was added to while doing so:

Code: Select all

time="2023-03-09T07:10:38.796527793Z" level=info msg="ignoring event" container=76e95f3cdfc354a588077b19bb16d016916551e34b7b3830f24eaf656133d520 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2023-03-09T07:10:38.797804955Z" level=info msg="shim disconnected" id=76e95f3cdfc354a588077b19bb16d016916551e34b7b3830f24eaf656133d520
time="2023-03-09T07:10:38.797944705Z" level=warning msg="cleaning up after shim disconnected" id=76e95f3cdfc354a588077b19bb16d016916551e34b7b3830f24eaf656133d520 namespace=moby
time="2023-03-09T07:10:38.797988318Z" level=info msg="cleaning up dead shim"
time="2023-03-09T07:10:38.830762293Z" level=warning msg="cleanup warnings time=\"2023-03-09T07:10:38Z\" level=info msg=\"starting signal loop\" namespace=moby pid=12850 runtime=io.containerd.runc.v2\n"
time="2023-03-09T07:10:57.240001968Z" level=info msg="shim disconnected" id=8e4d71b8a034a3b13dc50a3677dc2aec630599ed01c61b53763aa0ed3a84e81e
time="2023-03-09T07:10:57.240165593Z" level=warning msg="cleaning up after shim disconnected" id=8e4d71b8a034a3b13dc50a3677dc2aec630599ed01c61b53763aa0ed3a84e81e namespace=moby
time="2023-03-09T07:10:57.240203568Z" level=info msg="cleaning up dead shim"
time="2023-03-09T07:10:57.241168930Z" level=info msg="ignoring event" container=8e4d71b8a034a3b13dc50a3677dc2aec630599ed01c61b53763aa0ed3a84e81e module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2023-03-09T07:10:57.273309355Z" level=warning msg="cleanup warnings time=\"2023-03-09T07:10:57Z\" level=info msg=\"starting signal loop\" namespace=moby pid=14009 runtime=io.containerd.runc.v2\n"
time="2023-03-09T07:11:06.077368505Z" level=info msg="shim disconnected" id=4ff27400801a009cc9a923ac346e625dfeadceb19c12538445bf3cc010d1293a
time="2023-03-09T07:11:06.077491093Z" level=warning msg="cleaning up after shim disconnected" id=4ff27400801a009cc9a923ac346e625dfeadceb19c12538445bf3cc010d1293a namespace=moby
time="2023-03-09T07:11:06.077527493Z" level=info msg="cleaning up dead shim"
time="2023-03-09T07:11:06.081788555Z" level=info msg="ignoring event" container=4ff27400801a009cc9a923ac346e625dfeadceb19c12538445bf3cc010d1293a module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2023-03-09T07:11:06.106617493Z" level=warning msg="cleanup warnings time=\"2023-03-09T07:11:06Z\" level=info msg=\"starting signal loop\" namespace=moby pid=14711 runtime=io.containerd.runc.v2\n"
time="2023-03-09T07:11:14.327026280Z" level=info msg="shim disconnected" id=dce04991c68337127fb7e345704035d736798c4199a1c24f1e9d9f4bb9a5ac28
time="2023-03-09T07:11:14.327196105Z" level=warning msg="cleaning up after shim disconnected" id=dce04991c68337127fb7e345704035d736798c4199a1c24f1e9d9f4bb9a5ac28 namespace=moby
time="2023-03-09T07:11:14.327233430Z" level=info msg="cleaning up dead shim"
time="2023-03-09T07:11:14.328558218Z" level=info msg="ignoring event" container=dce04991c68337127fb7e345704035d736798c4199a1c24f1e9d9f4bb9a5ac28 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2023-03-09T07:11:14.351042718Z" level=warning msg="cleanup warnings time=\"2023-03-09T07:11:14Z\" level=info msg=\"starting signal loop\" namespace=moby pid=15365 runtime=io.containerd.runc.v2\n"
time="2023-03-09T07:11:23.416237230Z" level=info msg="ignoring event" container=e9415dae7d140251b5dac30d8334d55b9150e76e22bf5252226ec55e9f2e33e8 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2023-03-09T07:11:23.419318368Z" level=info msg="shim disconnected" id=e9415dae7d140251b5dac30d8334d55b9150e76e22bf5252226ec55e9f2e33e8
time="2023-03-09T07:11:23.419759880Z" level=warning msg="cleaning up after shim disconnected" id=e9415dae7d140251b5dac30d8334d55b9150e76e22bf5252226ec55e9f2e33e8 namespace=moby
time="2023-03-09T07:11:23.419808468Z" level=info msg="cleaning up dead shim"
time="2023-03-09T07:11:23.470673118Z" level=warning msg="cleanup warnings time=\"2023-03-09T07:11:23Z\" level=info msg=\"starting signal loop\" namespace=moby pid=16062 runtime=io.containerd.runc.v2\n"
time="2023-03-09T07:11:28.029970468Z" level=info msg="shim disconnected" id=01be38262ce1c1f707e7e923644c7ac58228f0149645492fc089972f5f29bed0
time="2023-03-09T07:11:28.031535680Z" level=info msg="shim disconnected" id=08da28bbb15d30de0ad97ebd48d42333c3e83ecff4a3e7b50f8e5b8a1b4ae849
time="2023-03-09T07:11:28.031706218Z" level=warning msg="cleaning up after shim disconnected" id=08da28bbb15d30de0ad97ebd48d42333c3e83ecff4a3e7b50f8e5b8a1b4ae849 namespace=moby
time="2023-03-09T07:11:28.031742143Z" level=info msg="cleaning up dead shim"
time="2023-03-09T07:11:28.033838468Z" level=warning msg="cleaning up after shim disconnected" id=01be38262ce1c1f707e7e923644c7ac58228f0149645492fc089972f5f29bed0 namespace=moby
time="2023-03-09T07:11:28.033943180Z" level=info msg="cleaning up dead shim"
time="2023-03-09T07:11:28.035482630Z" level=info msg="ignoring event" container=08da28bbb15d30de0ad97ebd48d42333c3e83ecff4a3e7b50f8e5b8a1b4ae849 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2023-03-09T07:11:28.035625643Z" level=info msg="ignoring event" container=01be38262ce1c1f707e7e923644c7ac58228f0149645492fc089972f5f29bed0 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2023-03-09T07:11:28.154663105Z" level=warning msg="cleanup warnings time=\"2023-03-09T07:11:28Z\" level=info msg=\"starting signal loop\" namespace=moby pid=16493 runtime=io.containerd.runc.v2\n"
time="2023-03-09T07:11:28.212159505Z" level=warning msg="cleanup warnings time=\"2023-03-09T07:11:28Z\" level=info msg=\"starting signal loop\" namespace=moby pid=16500 runtime=io.containerd.runc.v2\n"
time="2023-03-09T07:11:30.183436605Z" level=info msg="ignoring event" container=35826350d1429f60f11032f7c21fb816e89a472d2722503ea5ce30b9321e59e6 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2023-03-09T07:11:30.184192530Z" level=info msg="shim disconnected" id=35826350d1429f60f11032f7c21fb816e89a472d2722503ea5ce30b9321e59e6
time="2023-03-09T07:11:30.184738305Z" level=warning msg="cleaning up after shim disconnected" id=35826350d1429f60f11032f7c21fb816e89a472d2722503ea5ce30b9321e59e6 namespace=moby
time="2023-03-09T07:11:30.184796093Z" level=info msg="cleaning up dead shim"
time="2023-03-09T07:11:30.256374468Z" level=warning msg="cleanup warnings time=\"2023-03-09T07:11:30Z\" level=info msg=\"starting signal loop\" namespace=moby pid=16832 runtime=io.containerd.runc.v2\n"
time="2023-03-09T07:11:32.267432668Z" level=info msg="ignoring event" container=28a0f835136cd908532ea546e92ffed53d925a1995e40ac2fe3fbf1c3e07ceea module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2023-03-09T07:11:32.267785880Z" level=info msg="shim disconnected" id=28a0f835136cd908532ea546e92ffed53d925a1995e40ac2fe3fbf1c3e07ceea
time="2023-03-09T07:11:32.267912368Z" level=warning msg="cleaning up after shim disconnected" id=28a0f835136cd908532ea546e92ffed53d925a1995e40ac2fe3fbf1c3e07ceea namespace=moby
time="2023-03-09T07:11:32.267948805Z" level=info msg="cleaning up dead shim"
time="2023-03-09T07:11:32.297936243Z" level=warning msg="cleanup warnings time=\"2023-03-09T07:11:32Z\" level=info msg=\"starting signal loop\" namespace=moby pid=17037 runtime=io.containerd.runc.v2\n"
time="2023-03-09T07:11:32.626789980Z" level=info msg="Processing signal 'terminated'"
time="2023-03-09T07:11:32.642978443Z" level=info msg="stopping event stream following graceful shutdown" error="<nil>" module=libcontainerd namespace=moby
time="2023-03-09T07:11:32.651476530Z" level=info msg="Daemon shutdown complete"
time="2023-03-09T07:11:32.651813868Z" level=info msg="stopping healthcheck following graceful shutdown" module=libcontainerd
time="2023-03-09T07:11:32.652170430Z" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=libcontainerd namespace=plugins.moby
Nothing has been added to debug.txt since re-enabling Docker.

I'll keep the script renamed for now, so I can get a chance to run those commands next time with everything still down (yes, I should just remove the cron job, but effort...).

If you see anything in the logs which might help, let me know. Thanks as always!
User avatar
Nazar78
Posts: 2002
Joined: Wed Jul 17, 2019 10:21 pm
Location: Singapore
Contact:

Re: Docker can become unresponsive, but containers continue to run

Post by Nazar78 »

/volume1/.@root$ docker ps -a
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

/volume1/.@root$ ps -ef|grep docker
4248 root 0:00 grep docker

/volume1/.@root$ mount|grep docker
/dev/md1 on /volume1/.@plugins/AppCentral/docker-ce/docker_lib type btrfs (rw,relatime,ssd,discard,space_cache,subvolid=259,subvol=/.@plugins)
/dev/md1 on /volume1/.@plugins/AppCentral/docker-ce/docker_lib/btrfs type btrfs (rw,relatime,ssd,discard,space_cache,subvolid=259,subvol=/.@plugins)
nsfs on /var/run/docker/netns/2ea2113c8b08 type nsfs (rw)
nsfs on /var/run/docker/netns/eda0074c67b8 type nsfs (rw)
nsfs on /var/run/docker/netns/69288b1921c9 type nsfs (rw)
nsfs on /var/run/docker/netns/793151b05e17 type nsfs (rw)
nsfs on /var/run/docker/netns/fdfd16ffd65d type nsfs (rw)
nsfs on /var/run/docker/netns/415945e347ba type nsfs (rw)
nsfs on /var/run/docker/netns/6c20777f76b7 type nsfs (rw)
nsfs on /var/run/docker/netns/93d371c29138 type nsfs (rw)
nsfs on /var/run/docker/netns/f5474e4b393d type nsfs (rw)
One thing stood out is I'm using overlay2 driver, ext4 and you're using btrfs. When I stop dockerd, there's no other docker mounts left except for "/var/run/docker/netns/default type nsfs". Do you happen to have btrfs snapshots scheduled? Coincidentally it coincide with the docker hangups?

I'm not sure if this is the reason as I have little experience with btrfs, Docker stated in: Use the BTRFS storage driver,
Note: The btrfs storage driver is only supported on Docker Engine - Community on SLES, Ubuntu or Debian.
The package is maintained by Asustor so there could be things missing out, (edited: example errors thrown by busybox which works fine on vanilla SUSE/Ubuntu/Debian). Also take a look at Btrfs and Docker performance.

I'll try to setup some btrfs mounts then let run docker in it see if I have any issues...
AS5304T - 16GB DDR4 - ADM-OS modded on 2GB RAM
Internal:
- 4x10TB Toshiba RAID10 Ext4-Journal=Off
External 5 Bay USB3:
- 4x2TB Seagate modded RAID0 Btrfs-Compression
- 480GB Intel SSD for modded dm-cache (initramfs auto update patch) and Apps

When posting, consider checking the box "Notify me when a reply is posted" to get faster response
ilike2burnthing
Posts: 379
Joined: Thu Apr 09, 2020 8:01 pm

Re: Docker can become unresponsive, but containers continue to run

Post by ilike2burnthing »

I have snapshots scheduled, but they're for midday and midnight, so they don't coincide with the issue.
ilike2burnthing
Posts: 379
Joined: Thu Apr 09, 2020 8:01 pm

Re: Docker can become unresponsive, but containers continue to run

Post by ilike2burnthing »

While Docker is unresponsive:

Code: Select all

/volume1/.@root$ ps -ef|grep docker
 1427 root      0:00 docker ps -a
 3862 root      0:00 grep docker
 7169 root      0:00 /usr/local/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 18989 -container-ip 172.18.0.5 -container-port 8989
 7186 root      0:02 /volume1/.@plugins/AppCentral/docker-ce/bin/containerd-shim-runc-v2 -namespace moby -id 6120c597ddfb0c74d53d81e1f1477a19f039448bd9ea0ce95597a17917278c03 -address /var/run/docker/containerd/containerd.sock
11274 root      0:00 /usr/local/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 38181 -container-ip 172.18.0.7 -container-port 8181
11301 root      1:07 /volume1/.@plugins/AppCentral/docker-ce/bin/containerd-shim-runc-v2 -namespace moby -id 9ec13083a55b9896219d4a66f14cb25943f1f29eb7c27998d946fd21c99ee81b -address /var/run/docker/containerd/containerd.sock
11627 root      2:12 /volume1/.@plugins/AppCentral/docker-ce/bin/containerd-shim-runc-v2 -namespace moby -id 6ecf6e7fb217d14de459af522b8bfbebecccbab37e688eac325e1e5a1015f942 -address /var/run/docker/containerd/containerd.sock
12479 root      0:00 /usr/local/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 9091 -container-ip 172.18.0.2 -container-port 9091
12494 root      1:41 /volume1/.@plugins/AppCentral/docker-ce/bin/containerd-shim-runc-v2 -namespace moby -id 151e8565bcbdf41cdd742fd0d42f11514fe19f5a0c37e4c041c755b3ef28136a -address /var/run/docker/containerd/containerd.sock
16412 root      0:00 /usr/local/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 8191 -container-ip 172.18.0.8 -container-port 8191
16427 root      0:15 /volume1/.@plugins/AppCentral/docker-ce/bin/containerd-shim-runc-v2 -namespace moby -id 06e66bb0332003a1fc70856153f80b393bc1675c949d5ef0df96dcdde00a2b2e -address /var/run/docker/containerd/containerd.sock
16818 root      0:00 /usr/local/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 17878 -container-ip 172.18.0.6 -container-port 7878
16846 root      2:09 /volume1/.@plugins/AppCentral/docker-ce/bin/containerd-shim-runc-v2 -namespace moby -id d27ef4bf12f9a432f9b3b0ad5c87919c8e18bef919cc176ad55658eb6e2f2b8c -address /var/run/docker/containerd/containerd.sock
21463 root      0:00 /usr/local/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 19117 -container-ip 172.18.0.3 -container-port 9117
21492 root      0:12 /volume1/.@plugins/AppCentral/docker-ce/bin/containerd-shim-runc-v2 -namespace moby -id e17f42f06ad883c1a406199ea15d41e007123804bee1c42aee86e4c54ca115dd -address /var/run/docker/containerd/containerd.sock
21854 root      0:00 /usr/local/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 19118 -container-ip 172.18.0.4 -container-port 9117
21876 root      0:11 /volume1/.@plugins/AppCentral/docker-ce/bin/containerd-shim-runc-v2 -namespace moby -id 9b7ade4d6a1cbb89d79c8a4faf6d466369db232847ccb7e36050b0210fa835b9 -address /var/run/docker/containerd/containerd.sock
22902 root     37:17 /usr/local/AppCentral/docker-ce/bin/dockerd --data-root /usr/local/AppCentral/docker-ce/docker_lib/
22910 root     10:50 containerd --config /var/run/docker/containerd/containerd.toml --log-level info
31542 root      0:00 /usr/local/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 19943 -container-ip 172.17.0.3 -container-port 9443
31590 root      0:00 /usr/local/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 19900 -container-ip 172.17.0.3 -container-port 9000
31643 root      0:00 /usr/local/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 19800 -container-ip 172.17.0.3 -container-port 8000
31661 root      2:48 /volume1/.@plugins/AppCentral/docker-ce/bin/containerd-shim-runc-v2 -namespace moby -id 76e95f3cdfc354a588077b19bb16d016916551e34b7b3830f24eaf656133d520 -address /var/run/docker/containerd/containerd.sock

Code: Select all

/volume1/.@root$ mount|grep docker
/dev/md1 on /volume1/.@plugins/AppCentral/docker-ce/docker_lib type btrfs (rw,relatime,ssd,discard,space_cache,subvolid=259,subvol=/.@plugins)
/dev/md1 on /volume1/.@plugins/AppCentral/docker-ce/docker_lib/btrfs type btrfs (rw,relatime,ssd,discard,space_cache,subvolid=259,subvol=/.@plugins)
nsfs on /var/run/docker/netns/ccec822d0a17 type nsfs (rw)
nsfs on /var/run/docker/netns/7059d189af74 type nsfs (rw)
nsfs on /var/run/docker/netns/6086e23c40ec type nsfs (rw)
nsfs on /var/run/docker/netns/1e4db016f43d type nsfs (rw)
nsfs on /var/run/docker/netns/af8b1bb4cec0 type nsfs (rw)
nsfs on /var/run/docker/netns/9a8df3764918 type nsfs (rw)
nsfs on /var/run/docker/netns/02c2fd9d67d0 type nsfs (rw)
nsfs on /var/run/docker/netns/a01210693e56 type nsfs (rw)
nsfs on /var/run/docker/netns/cc11f29537d5 type nsfs (rw)
containerd-shim and docker-proxy processes were sleeping
User avatar
Nazar78
Posts: 2002
Joined: Wed Jul 17, 2019 10:21 pm
Location: Singapore
Contact:

Re: Docker can become unresponsive, but containers continue to run

Post by Nazar78 »

Just a wild thought, try docker with tcp instead of socket? Edit the start-stop append "-H tcp://127.0.0.1:2375" (2376 for encrypted) to the DOCKERD_OPT then restart dockerd. You'll have to also change portainer to connect to tcp.

I have yet to test running docker on btrfs. Since I don't have any bay slots left, a way for me to do this is to make a loop mount of a btrfs image then migrate docker to run on it or just wait for my new 1TB ssd upgrade to arrive.
AS5304T - 16GB DDR4 - ADM-OS modded on 2GB RAM
Internal:
- 4x10TB Toshiba RAID10 Ext4-Journal=Off
External 5 Bay USB3:
- 4x2TB Seagate modded RAID0 Btrfs-Compression
- 480GB Intel SSD for modded dm-cache (initramfs auto update patch) and Apps

When posting, consider checking the box "Notify me when a reply is posted" to get faster response
User avatar
Nazar78
Posts: 2002
Joined: Wed Jul 17, 2019 10:21 pm
Location: Singapore
Contact:

Re: Docker can become unresponsive, but containers continue to run

Post by Nazar78 »

Code: Select all

root@Nimbustor4:~# mount|grep docker|grep -v overlay
/dev/loop3 on /volume1/.@plugins/AppCentral/docker-ce type btrfs (rw,relatime,ssd,space_cache,subvolid=5,subvol=/)
/dev/loop3 on /volume1/.@plugins/AppCentral/docker-ce/docker_lib type btrfs (rw,relatime,ssd,space_cache,subvolid=5,subvol=/)
nsfs on /var/run/docker/netns/default type nsfs (rw)
nsfs on /var/run/docker/netns/57d03ed5b7fd type nsfs (rw)
Ok so I created a btrfs image to run docker, however the Asustor's build of the storage driver is still using overlay2, only the backing fs is btrfs. Does yours says the same as below? Else I'll need to dig further as I can't seems to find the daemon.json to change the storage driver.

Code: Select all

root@Nimbustor4:~# docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  compose: Docker Compose (Docker Inc., v2.15.1)

Server:
 Containers: 8
  Running: 8
  Paused: 0
  Stopped: 0
 Images: 8
 Server Version: 20.10.22
 Storage Driver: overlay2
  Backing Filesystem: btrfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 78f51771157abb6c9ed224c22013cdf09962315d
 runc version: v1.1.4-0-g5fd4c4d1
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.13.0.x
 Operating System: Linux
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 15.41GiB
 Name: Nimbustor4
 ID: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
 Docker Root Dir: /volume1/.@plugins/AppCentral/docker-ce/docker_lib
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
AS5304T - 16GB DDR4 - ADM-OS modded on 2GB RAM
Internal:
- 4x10TB Toshiba RAID10 Ext4-Journal=Off
External 5 Bay USB3:
- 4x2TB Seagate modded RAID0 Btrfs-Compression
- 480GB Intel SSD for modded dm-cache (initramfs auto update patch) and Apps

When posting, consider checking the box "Notify me when a reply is posted" to get faster response
Post Reply

Return to “Docker”