Docker can become unresponsive, but containers continue to run

[ilike2burnthing]

Been running into an issue for the past couple of months, it happens every 1-2 weeks, and is only resolved by a restart of the NAS. It happened again about an hour ago, so I thought I was overdue asking for help.

I notice it each time when opening Portainer, usually having done so several times in that same hours long browser session. I can access Portainer's settings, but if I try to access the environment dashboard it just loads forever. I can log out and back in, try from a different browser, from a private/incognito window with no addons/extensions enabled, using the local IP rather than DDNS, etc., nothing fixes it.

It seems to be a Docker issue though, rather than Portainer, because if I try to run `docker ps` or any other Docker commands, nothing happens and I have to Ctrl+C out of them. However, all the while my containers are running fine.

Trying to disable Docker through App Central doesn't work either, no matter how long I wait it's just a spinning wheel of doom.

Obviously I'd like to prevent the problem if possible, or at least fix it without a restart of the NAS, so any help or suggestions would be appreciated.

Feel free to ask for any information, but I'll just throw the basics here:
AS3102T
ADM 4.2.0.RC81
Docker 20.10.17.r1

[Nazar78]

Try:

Code: Select all

root@Nimbustor4:~# docker system prune -a
WARNING! This will remove:
  - all stopped containers
  - all networks not used by at least one container
  - all images without at least one container associated to them
  - all build cache

Are you sure you want to continue? [y/N] y
Deleted build cache objects:
urvxgudcf32glhbqfkzbq9k07
93r7jisvxz6wjjx75pw0nvi86
9dbj5ixqd1unbo60s1hkr1nfg
kqi9d417ro9n863jf3pdiqgoz
5ivo4zvwkzlsp13rpyo9bef29
pxiygqx2f89rgt2r2cfjn823j
ul1h8nihrbffyxbusmhk25l6a
whzcl2wobdo88fjnuojya1f5o
tl7g38hqmkeagrxxmq6xr5a8l
fpz8l6p7qvps5mu7su8gscfnz
vj4emp8epb64nr7g1c6wy30ea
k551v9692ix6v0mjd2tsyez51
5erpkh17bu3iaoddqzrfb6cnb
aekdwesmoiq15gdsg5gzxicbq

Total reclaimed space: 22.09MB

And/or:
Since you said it happened every 1-2 weeks, it could be one of the containers affecting the daemon, especially portainer as it's using the unix socket to communicate with dockerd. I'm not sure but try disabling portainer or similar containers for the time being see if it helps.

[ilike2burnthing]

I keep my Docker very tidy, so `prune` didn't remove anything.

Unfortunately I rely pretty heavily on Portainer, so I don't think I could really go without it for the next 2 weeks (or not at all if that did end up being the cause).

The rest of the containers are fairly normal - Jackett, *arrs, Transmission with VPN, Tautulli, FlareSolverr, and Watchtower.

Are you aware of any logs which might give some insight, either now or the next time this happens?

[Nazar78]

Disabling portainer is just for troubleshooting.

For logging, you could try run the dockerd with "-D" and "-l debug" switches, by editing the start-stop script docker opts. I'm not sure though how Asustor build the app because the above doesn't show any more details in the /var/log/messages but I haven't look deeper.

[ilike2burnthing]

Unless I've misunderstood, to troubleshoot it I'd need to disable Portainer for at least 2 weeks, which isn't really a viable option for me.

I'll definitely try fishing around in the logs the next time it happens.

Another scrap of information, no matter what I was trying to do, ADM's Activity Manager just showed Docker's processes as sleeping.

[Nazar78]

Just curious if you don't mind, what is it that you do with portainer? Are you utilizing its API? I did this long ago for some automation but decided to cron script via console instead. Now I seldom open portainer.

I do recall having similar issues, the docker commands never returned (via default socket not TCP) even though the containers are running OK. But it was me who's messing with some ethernet interfaces trying to make VPN work in LXC (succeeded eventually). Not sure why it affected docker instead. I had to kill every instance of the docker*/container* to avoid restarting the whole system.

[ilike2burnthing]

Just monitoring/restarting/editing containers, accessing logs, and occasionally trying out test builds.

I know it's possible to do all that without Portainer, but I'm faster and more comfortable with a GUI than a CLI.

Docker has worked fine with roughly the same setup for the past couple of years, it's only in the last couple of months that this problem has popped up, and other than the usual updates to ADM, Plex, Docker, and images, I can't really point to any particular changes as the cause.

[ilike2burnthing]

Well that was quick, just 4 days this time.

Nothing I can see in /var/log/messages but not really sure where else to look.

Going to leave it running overnight, see if there's any change.

If there's anything you want me to try running or looking at, let me know.

---

UPDATE: damn, I had to restart the NAS, a container had an issue (I assume unrelated), but let me know for next time.

[ilike2burnthing]

Well I may be getting somewhere. I've had another 2 cases of this since, one about a week ago and another one today.

When it became unresponsive today, I decided to adapt some earlier help from you and ran:

Code: Select all

kill -HUP $(ps -ef|grep dockerd|grep -v grep|awk '{print $1}')

This didn't seem to achieve anything, so then I ran:

Code: Select all

kill -HUP $(ps -ef|grep docker|grep -v grep|awk '{print $1}')

All containers were now unreachable and docker did not seem to restart (I also had a bunch of docker-proxy zombie processes). Trying to kill docker or dockerd again didn't help.

Disabling Portainer CE in ADM's App Central started docker again, and re-enabling Portainer CE allowed me to access it. All containers were shown to be running, however I couldn't access any in browser. Restarting them through Portainer fixed this and also got rid of the zombie processes.

I repeated this again, this time running the kill command for docker and disabling Docker Engine in App Central, then re-enabling it and the associated 'apps' for each container. This was a little quicker and worked just as well.

So, is there an easier way of doing this, e.g. better command(s) to run?

[Nazar78]

You could probably cron a script to monitor the apps, probe the ports, if they aren't accessible, trigger the kill, restart then recheck. Do this kill-restart-recheck in a finite loop break when success.

Add notifications so you can keep track until at least there's a possible fix in next release.

I have something similar implemented but without the kill part.