Docker can become unresponsive, but containers continue to run

Docker containers wrap a piece of software in a complete filesystem that contains everything needed to run: code, runtime, system tools, system libraries – anything that can be installed on a server. This guarantees that the software will always run the same, regardless of its environment.

Moderator: Lillian.W@AST

ilike2burnthing
Posts: 380
youtube meble na wymiar Warszawa
Joined: Thu Apr 09, 2020 8:01 pm

Re: Docker can become unresponsive, but containers continue to run

Post by ilike2burnthing »

That's the weird thing, the containers seem to work fine while this is all going on, it's just docker that's non-responsive, it's only when I kill docker that they're not accessible.

I'm also not really sure how I'd script the disabling and re-enabling of Docker Engine in App Central.
User avatar
Nazar78
Posts: 2003
Joined: Wed Jul 17, 2019 10:21 pm
Location: Singapore
Contact:

Re: Docker can become unresponsive, but containers continue to run

Post by Nazar78 »

I just quickly wrote these, tested and it's seems good. Put it in a script named say checkDocker.sh, chmod +x then schedule every 5 minutes in cron as root and results in /var/log/messages filtered by the script name. Feel free to modify like adding notifications etc.

Code: Select all

#!/bin/sh

waitSecsInterval=5

restartDocker() {
	# This doesn't work when docker is already not responding. /usr/local/AppCentral/docker-ce/CONTROL/start-stop.sh stop>/dev/null 2>&1
	ps -ef|grep docker|grep -v grep|awk '{print $1}'|xargs kill -9>/dev/null 2>&1
	/usr/local/AppCentral/docker-ce/CONTROL/start-stop.sh start>/dev/null 2>&1
	sleep 15
	docker container ls -a|awk 'NR>1{print $NF}'|xargs docker start>/dev/null 2>&1
}

RES=/tmp/${0##*/}.$$
rm -f $RES>/dev/null 2>&1
(curl -so $RES --unix-socket /var/run/docker.sock docker/containers/json>/dev/null 2>&1)&
PID=$!

while :;do
	sleep 1
	COUNT=$((COUNT+1))
	if [ -e $RES ];then
		# "All good! Need not fill the logs..."
		rm -f $RES>/dev/null 2>&1
		break
	elif [ $COUNT -ge $waitSecsInterval ];then
		logger -st ${0##*/} "Process hung? Trying to restart..."
		restartDocker &
		kill -9 $PID>/dev/null 2>&1
		break
	fi
done

Code: Select all

*/5 * * * * /root/checkDocker.sh>/dev/null 2>&1
Edited: Add to force start all containers.
Last edited by Nazar78 on Wed Mar 29, 2023 3:55 am, edited 2 times in total.
AS5304T - 16GB DDR4 - ADM-OS modded on 2GB RAM
Internal:
- 4x10TB Toshiba RAID10 Ext4-Journal=Off
External 5 Bay USB3:
- 4x2TB Seagate modded RAID0 Btrfs-Compression
- 480GB Intel SSD for modded dm-cache (initramfs auto update patch) and Apps

When posting, consider checking the box "Notify me when a reply is posted" to get faster response
ilike2burnthing
Posts: 380
Joined: Thu Apr 09, 2020 8:01 pm

Re: Docker can become unresponsive, but containers continue to run

Post by ilike2burnthing »

Thanks, this looks great, I'll give it a shot tomorrow.
User avatar
Nazar78
Posts: 2003
Joined: Wed Jul 17, 2019 10:21 pm
Location: Singapore
Contact:

Re: Docker can become unresponsive, but containers continue to run

Post by Nazar78 »

There's a new version 20.10.22.r2. Hope it could fix your issue.
AS5304T - 16GB DDR4 - ADM-OS modded on 2GB RAM
Internal:
- 4x10TB Toshiba RAID10 Ext4-Journal=Off
External 5 Bay USB3:
- 4x2TB Seagate modded RAID0 Btrfs-Compression
- 480GB Intel SSD for modded dm-cache (initramfs auto update patch) and Apps

When posting, consider checking the box "Notify me when a reply is posted" to get faster response
ilike2burnthing
Posts: 380
Joined: Thu Apr 09, 2020 8:01 pm

Re: Docker can become unresponsive, but containers continue to run

Post by ilike2burnthing »

Thanks, updating now. I didn't get round to adding the cron job (busy week), but I'll see how it goes without it, then add it if the issue returns in a week or so.
ilike2burnthing
Posts: 380
Joined: Thu Apr 09, 2020 8:01 pm

Re: Docker can become unresponsive, but containers continue to run

Post by ilike2burnthing »

First failure since the Docker update (and then ADM update).

I ran /usr/local/AppCentral/docker-ce/CONTROL/start-stop.sh stop and it just hung on Stopping Docker daemon... until it timed out.

Running the other two commands worked, though I had to start Portainer manually.

Anything needing to be tweaked in the script, or should I just implement that cron as is?
User avatar
Nazar78
Posts: 2003
Joined: Wed Jul 17, 2019 10:21 pm
Location: Singapore
Contact:

Re: Docker can become unresponsive, but containers continue to run

Post by Nazar78 »

Sorry to hear that the new updates didn't work out for you. You can append to the restartDocker function to sleep 15 seconds, then issue a restart on all your containers. I've edited the script on the previous post.
AS5304T - 16GB DDR4 - ADM-OS modded on 2GB RAM
Internal:
- 4x10TB Toshiba RAID10 Ext4-Journal=Off
External 5 Bay USB3:
- 4x2TB Seagate modded RAID0 Btrfs-Compression
- 480GB Intel SSD for modded dm-cache (initramfs auto update patch) and Apps

When posting, consider checking the box "Notify me when a reply is posted" to get faster response
ilike2burnthing
Posts: 380
Joined: Thu Apr 09, 2020 8:01 pm

Re: Docker can become unresponsive, but containers continue to run

Post by ilike2burnthing »

Well crap... :/

So turns out my earlier attempts did restart Docker and the containers were 'running' (I was able to access their respective UIs in browser), but none of them had internet access. Restarting or stopping/starting the containers didn't resolve this.

I tried disabling and re-enabling Docker in App Central like before, but I couldn't access any of the containers and when running docker ps I got the error Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

So I tried starting dockerd, and the process was then running, but docker ps returned nothing.

I reran start-stop.sh stop and start-stop.sh start, restarted the containers, and then I was finally able to get access again.

No idea what the issue is now :S
User avatar
Nazar78
Posts: 2003
Joined: Wed Jul 17, 2019 10:21 pm
Location: Singapore
Contact:

Re: Docker can become unresponsive, but containers continue to run

Post by Nazar78 »

No sure either, just give the script a try. Then monitor the containers' uptime and/or /var/log/messages to see if it works.
AS5304T - 16GB DDR4 - ADM-OS modded on 2GB RAM
Internal:
- 4x10TB Toshiba RAID10 Ext4-Journal=Off
External 5 Bay USB3:
- 4x2TB Seagate modded RAID0 Btrfs-Compression
- 480GB Intel SSD for modded dm-cache (initramfs auto update patch) and Apps

When posting, consider checking the box "Notify me when a reply is posted" to get faster response
ilike2burnthing
Posts: 380
Joined: Thu Apr 09, 2020 8:01 pm

Re: Docker can become unresponsive, but containers continue to run

Post by ilike2burnthing »

Unfortunately the cron job/didn't work. There were several entries in messages for checkDocker.sh: Process hung? Trying to restart..., but nothing happens. Running the script manually doesn't work either.

The only thing which has worked so far is a restart of the NAS, or kill -HUP $(ps -ef|grep docker|grep -v grep|awk '{print $1}') and then disable and re-enable Docker in App Central.

I'm going to try removing the start-stop.sh stop line and see if it works next time (I'm only thinking of this now, having resolved the issue already).
Post Reply

Return to “Docker”