constant disk activity

[lunix]

I see - thanks for the warning. Maybe I will still invest some time in tracing before moving all apps. Will report how it goes some time next week.

[lunix]

I fatraced only a little bit and that showed me already some apps reading from the AppCentral directory that I had not seen before. Enough reason to follow your advice and move all apps in bulk to SSD. This together with the symlinked data directories (apps & docker containers), reduced swappiness, symlinked ADM logging directories symlinks and other tweaks mentioned in my first post may do the trick - let's see (too early to report final success ).

Since I do not trust crontab @reboot entirely I modified the approach slightly: In order to be 100% certain that the bind mount happens before any of the apps is started after reboot I not only start the below script "S00-apps2ssd.sh" as a cron job but also in /volume1/.@plugins/etc/init.d on the hard disk where it persists. To avoid mounting multiple times:

Code: Select all

#!/bin/sh
if [ ! -f /volume1/.@plugins/ssd_mounted ]
then
	# Need to bind mount
	mount -o bind /volume2/ssdAppData/.@plugins /volume1/.@plugins
	echo "# apps mounted to ssd" > "/volume1/.@plugins/ssd_mounted"
fi

If there is still anything left from ADM or Apps going to volume0 (e.g. apps maybe reading etc/localtime frequently which links to /volume0/usr/builtin/share/zoneinfo ?) I will hopefully catch these with fatrace and then see how to deal with them. Thanks again, Nazar, for the insight you provided.

Edit: There is something ugly too with the bind mount: At every reboot the NAS complains it was not shut down properly last time. While this does not cause actual functional degradation it is annoying that ADM restores all sort of things because of this assessment. I suspect the ADM code which checks if e.g. apps were not properly shut down happens before the bind mount script runs. Maybe this could be dealt with via the rcK scripts, e.g. rsync the SSD folder back to the hard drive as the last K99... script? Do you see that behavior too?

[lunix]

tested further and found that ADM really does not like the bind mount method - I was not able to get rid of the "The NAS did not shut down properly last time" message. While it does not seem to cause any harm it is ugly.

So I went back and used a symlink instead, this time however for the Apps root folder (not just single apps like previously):

Code: Select all

#!/bin/sh

if [ ! -f /volume1/.@plugins/ssd_mounted ]
then
	# Need to symlink
	mv -f /volume1/.@plugins /volume1/.@plugins_OLD
	ln -s /volume2/ssdAppData/.@plugins /volume1/.@plugins
	echo "# apps SYMLINKED to ssd from cron job" > "/volume1/.@plugins/ssd_mounted"
fi

Now there are no more complaints from ADM and it seems that the cron job gets done before any process uses the directory - allowing the rename before creating the symlink.

[Nazar78]

The reason the NAS nags about not shutting down properly is because the NAS tries to unmount the mounted paths during reboot/shutdown. And if the paths are still being in use by the apps (apps which did not exit in a timely manner), the unmount will fail which will result in the NAS process "stormand" hanging that will need to be killed after a 60 secs timeout. This "stormand" is responsible for assembling the RAID and mounting the volumes, do a reverse on reboot/shutdown. If it's not exited properly, you'll get that "nag".

The proper way is to unmount the paths that you mounted during reboot/shutdown. I also added some steps to detect those unmountables that are still in use using mount grep after an umount command, kill the associated apps clinging onto those mounts, then unmount them again.

Usually the "nag" doesn't do any harm but I have certain situations (even without using this mount bind tweaks) that the NAS doesn't proceed with the reboot/shutdown that I have to force it using the physical button. This would be an issue for a remote restart/shutdown. Imagine the auto firmware upgrade fail to reboot due to this. And because of this too, I've added another custom 45 secs timer to detect such occurrence to force reboot/shutdown the NAS before the 60 secs timeout, because if the reboot/shutdown failed, it's already too late to intercept to force reboot/shutdown automatically.

And the reason I use bind mount instead of symlink, is due to the reason I've mentioned earlier, cross-device link. Some apps doesn't transverse hard links into symlinks that's located on another physical location, thus a recursive move operation within that app could fail or symlinked paths not visible in the apps. It's sometimes by default or you need to edit certain configs within the app to allow this to happen. If you don't have such setup, then symlinks are ok. E.g. I have some paths that I want to backup using the ADM Backup & Restore. Using symlinks you won't be able to see the paths while bind mount does.

[lunix]

Update: So, after a couple of days of testing I am quite happy with the symlink method. The symlinks are persistent over reboot and I have a cron job @ reboot just in case firmware updates spoil the symlinks - they will then be re-created automatically. I have also discovered a few more files on volume0 that are being accessed occasionally leading to disk wake ups. Symlinked those too. Hibernation times are quite decent now (typically hours).

Nonetheless there will always be moments when the disks are woken if a file on volume0 is not in cache. This is pretty random due to the Kernel's algorithm and depends on unpredictable factors like prior RAM use etc. Even if I added RAM to increase the chances of keeping files in cache this would still rely on the magic mercy of the Kernel's cache strategy. I am wondering if I should also go down the loop device route for the entire volume0 (on SSD). What I do not like about it is mostly the fear of unintended shutdowns (power grid failures happening quite often here). I am worried about always having to boot with a live linux every time to restore hard drive links (or mounts) if there was no proper shutdown... I figure you went through similar thought before finally moving it completely into RAM on every reboot. It is ironic, since my device actually has the built in M2 NVMe slots. Just too late without a backup. It would be nice if Asustor would add a migration tool rather than having to re-initialize everything. Oh well...

[Nazar78]

Unfortunately even with SSD or NVMe, ADM will still create the OS/SWAP RAID1 partitions on them similar to the HDDs. This is mainly for redundancy, in case any of the media failed or removed, the remaining media will take over.

I'm not actually concerned about the unintended shutdowns. Data wise nothing really gets updated frequently in the volume0 except for ADM loggings or if you manually made changes in the settings ADM or firmware upgrade. If the NAS did crashed before my last nightly resync (usually happens when I did something out of the ordinary i.e. recently trying to compile a patched MongoDB 6.3.1 from source using all CPUs), no I don't have to boot it into Live Linux, the previous state old will take over. On top of the nightly resync, I also use that same function to trigger resync on demand in case I'm planning doing something out of the ordinary again.

Additionally I have another nightly schedule doing tar gzip of the volume0 retaining them for a week. This is due to an issue that I had not long ago where the volume0 file system gets corrupted upon hard shutdowns and the NAS reboots into initialization mode. This can even happen when the volume0 is at its default state running on the HDDs with no mods done. With the backups, I can easily restore them without booting into Live USB. You can read about it here, [Tips] ADM system or data volume recovery.

Also if you don't have enough RAM to turn off the swap, you can swap it as a file onto the SSD/NVMes then turn it off from the HDDs partitions. Just ensure you have a dedicated SSD/NVMe space/partition for this 2GB volume0 and 2GB swap that never gets removed.