Docker can become unresponsive, but containers continue to run

[ilike2burnthing]

Ah apologies, I've obviously misunderstood you somewhere. I'm currently getting the following error for my looping containers:

Code: Select all

/run/s6/basedir/scripts/rc.init: line 67: --ulimit nofile=10240:10240: not found

Created /usr/local/etc/init.d/S00ulimit.sh with that copied and pasted code, ran chmod +x on it, added '--ulimit nofile=10240:10240' to the Command section of each container using Portainer and redeployed them. Finally I restart my NAS for good measure, but I'm now getting the above.

I also ran grep -E '(^Limit|open)' /proc/$(pidof dockerd)/limits and it returned:

Code: Select all

Limit                     Soft Limit           Hard Limit           Units     
Max open files            1024                 4096                 files

[Nazar78]

added '--ulimit nofile=10240:10240'

Not to the app command but docker container itself where when you first create the container either by docker create or docker run or docker-compose https://docs.docker.com/compose/compose ... 3/#ulimits. But I can't seem to find a way to do this in Portainer though you can inspect the value under HostConfig after using the ulimit switch to create the container. Actually you don't need to do this part unless your app (like a busy public webserver) has issues with the default ulimit.

I also ran grep -E '(^Limit|open)' /proc/$(pidof dockerd)/limits and it returned:

Erm this part let me test it later tonight because I'm still using the default limit and it's the first day of the month, schedules running for badblocks x8 hdd + raid5 scrubbing.

[ilike2burnthing]

K, ta. I'll hold fire. No rush.

Opening an issue on moby/moby at the moment.

You're tagged anyway, but for the sake of anyone else finding this - https://github.com/moby/moby/issues/45253

[Nazar78]

Ok about the ulimit init.d script, sorry it doesn't work ... I've updated previous posts to remove those comments. Please revert all the changes, if any.

Seems for Asustor, there's no way to change the global ulimit at boot unless you patch the initramfs, or inject into commands (unethical), which will also be lost at firmware upgrade. The only way I could think of to make it somewhat permanent is in each of the start-stop scripts themselves. Hence yet another simple script below. It will check and add the ulimit for the defined apps at boot time into the start-stop script. This will also take effect when you toggle the apps in the App Central. The ulimit command will be added just after the top shebang. However if those apps gets updated, you'll need to either run the ulimit command manually (or put it in your ~/.profile) then restart the app or wait till next boot to take effect which the script will modify the start-stop before it runs.

Code: Select all

vi /usr/local/etc/init.d/S00ulimit.sh

Code: Select all

#!/bin/sh

# S00ulimit.sh ################
# Check and set the ulimit of defined apps in /usr/local/etc/init.d/ during Asustor boot.
# For ulimit details, refer to https://linuxcommand.org/lc3_man_pages/ulimith.html.
###############################
# One line each, apps name in lowercase + ulimit command(s) separated by semicolon ';'.
# Use '#' to comment out.
APPS="
# myapp:ulimit -Sn 10240;ulimit -Hn 20480
docker-ce:ulimit -SHn 10240
"
###############################

case $1 in
	start)
		echo "$APPS"|while read i;do
			i=$(echo $i|xargs)
			APP=${i%%:*}
			ULIMIT=${i##*:}
			if [ ! -z "$APP" ] && [ ! -z "$ULIMIT" ] \
				&& [ "$(echo $ULIMIT|grep ^ulimit)" != "" ] \
				&& ls /usr/local/etc/init.d/S*$APP>/dev/null 2>&1 \
				&& ! grep -q ^ulimit /usr/local/etc/init.d/S*$APP;then
				echo -n "Changing ulimit for $APP..."
				sed -ri "s/(^#!\/bin\/sh)/\1\n\n$ULIMIT/" \
					/usr/local/AppCentral/$APP/CONTROL/start-stop.sh
				if [ $? -eq 0 ];then
					echo "SUCCESS!"
				else
					echo "FAILED!"
				fi
			fi
		done
	;;
	*)
		echo "Usage: $0 {start}"
	;;
esac

Code: Select all

root@Nimbustor4:~# sh /usr/local/etc/init.d/S00ulimit.sh start
Changing ulimit for docker-ce...SUCCESS!

Code: Select all

root@Nimbustor4:~# head /usr/local/etc/init.d/S50docker-ce
#!/bin/sh

ulimit -SHn 10240

DOCKERD_DIR="/usr/local/AppCentral/docker-ce"
DOCKER_LIB=$DOCKERD_DIR/docker_lib/
# disable iptables forward for vpn connection
DOCKERD_OPT="--data-root /usr/local/AppCentral/docker-ce/docker_lib/"
DOCKERD_PID="/var/run/docker.pid"
# DOCKER_RAMDISK muse be true

[Nazar78]

K, ta. I'll hold fire. No rush.

Opening an issue on moby/moby at the moment.

You're tagged anyway, but for the sake of anyone else finding this - https://github.com/moby/moby/issues/45253

It's just my hunch that the btrfs driver uses more resources but as per the conversation in that post, yeah the limit is not sufficient so it's up to the maintainer (Asustor) or admin user to raise this limit accordingly to their needs.

[ilike2burnthing]

Thanks again! Seems to be working well, hard and soft limits now both show as 10240.

I also removed checkDocker.sh and its cron job.

I'll open a ticket with Asustor, see if there's anything they can do.

Update: opened, and they're on holiday until the 5th, because of course they are

[ilike2burnthing]

I didn't update with their reply as it was just the generic, "This issue has been forwarded to the concerned department." Having not heard any further update though, I asked for one and got a reply today.

Apparently, a fix for the issue was added in the last ADM update; I'm guessing it fell under 'miscellaneous bug fixes'.

I've removed S00ulimit.sh and restarted the NAS, and the hard and soft limits still show as 10240. However, I'm not sure if that's just because of the existing additions of 'ulimit -SHn 10240' to start-stop.sh, K50docker-ce, and S50docker-ce, or if they would have made that their fix.

Are you seeing these on your end, or is there a way for me to check (short of uninstalling and reinstalling Docker)?

[Nazar78]

Ulimit is inheritance from its parent process and the parent is still the default for open files, 1024 (from the first init to your current shell, unless you made changes by adding it to the ~/.profile for the current session only or globally in the initramfs by rebuilding the firmware for ADM - other systems from /etc/security/limits.conf which is not implemented in ADM):

Code: Select all

root@Nimbustor4:~# ulimit -n
1024

I personally don't use that script and I'm still seeing 1024 for open files in the latest 4.2.1.RGE2. You can revert by disabling the boot script and removing or commenting out the ulimit line which was added by the boot script to the start-stop.sh, then restart the docker-ce or just reboot. Note, the K50docker-ce and S50docker-ce are just symlinks to the start-stop.sh, so just modify any one of them or start-stop.sh itself.

ps: I'm looking at an unexpected first time encounter issue with the 4.2.1.RGE2. The system crashed not responding after few days upgrading during a midnight heavy load backup period. Had few "system" mods done; apps running on external USB SSD, read caching to external USB SSD, raid5 on external USB, OS running in ram and built some kernel modules for ipset/ip6tables/USBIP. All these has been working flawlessly until this 4.2.1.RGE2. Due to how ADM works, the logs gets lost when I had to hard restart, so I had to do some tweaks to try catch the crash next time. Will post on a different thread if necessary.

[Nazar78]

Apparently I hit this ulimit issue with ext4 while using MongoDB and self-hosted Rocket.Chat. So end up using the S00ulimit.sh myself

[ilike2burnthing]

Sent that as an update to support.