AS-608T file system resize issue / Is it possible to boot ADM without mounting /dev/md1?

[Odeen]

Hello!

I have a 9-year-young AS-608T that I recently upgraded from 4TB drives running in RAID6 to 18TB.

The migration went fine up to a point -- I was able to swap in new drives one at a time, recover, and, eventually, resync /dev/md1 successfully. It's now 108TB (!!)

However, resizing the EXT4 file system fails. I believe, since I created the file system in December 2013, the file system is affected by this bug: https://bugzilla.redhat.com/show_bug.cgi?id=982871. Since my file system is already 24TB, it cannot be resized online. ADM doesn't allow for unmounting the file system from the GUI.

Since the HDMI output on the AS-60x series does not initialize until after the OS boots, it's virtually impossible to boot it from a Linux USB drive -- you can't see the BIOS screen unless you connect to the serial header on the motherboard.

I'm curious, is it possible to edit some kind of startup configuration files to prevent MDADM from starting the MD1 array? Alternately, can we boot into some kind of ADM "safe" or "recovery" -- bare minimum of services, just telnet or SSH access. Either way, if the array never mounts, an offline resize should work.

The last thing I can think of is connecting all 8 drives to a desktop PC, booting into Linux, assembling the array manually, and doing an offline resize there. That seems..... fraught -- I mostly have SFF PC's around the house, so I'm looking at purchasing an 8-port SATA card and setting up a rat's nest of power and data cables. And, with my luck and level of Linux experience, I'd probably end up destroying the array and have to restore from backups.

If you guys have any thoughts or ideas, I'm all ears. I'd rather not reformat and restore from backups, unless there's a compelling reason to switch to BTRFS


Thanks!

[Nazar78]

Connecting to another PC would work. I don't have your model but hitting the del key while booting you won't get to see the BIOS?

If so try this, never at any point attempt to do a hot swap as it could destroy your array. Always power off before physically handling the disks. And warning, do this at your own risk as no others shall be responsible of any damages.

• 1. Ensure your NAS has the lastest firmware.
  
  2. Shutdown the NAS, remove all its disks, boot it back up without the disks. You should be greeted with the typical initialization webUI when you access from the browser.
  
  3. Try ssh root@nas-host-or-ip, login with the password "admin".
  
  4. If the above doesn't work, you're left with the PC option. Need not continue further.
  
  5. Else if the above works, shutdown again then boot with all the disks.
  
  6. Ssh as admin then edit this file using `sudo vi /usr/etc/volume.conf`, you can choose to create a backup copy of this file, rename the INI header [volume1] to something else i.e. [volume1-bak]. Or just simply rename this file to volume.conf.bak. Then reboot.
  
  7. The NAS should no longer assemble the /dev/md1 and IIRC you'll also again see the initialization webUI as even the /dev/md0 will not get assembled (where the volume.conf resides). Use the method in point #3 to do your magical stuffs, you'll have to manually assemble the arrays md0/md1 then temporarily mount them i.e. to /mnt/md0 if you need access to the FS especially restoring the volume.conf, when done, revert everything back.

[Odeen]

Thank you for the detailed reply! As an abovementioned Linux newbie, I appreciate the pointer to the configuration files.

According to https://github.com/mafredri/asustor_as- ... er/boot.md, the AS-60x series firmware doesn't have HDMI support, so no video until ADM or an alternate OS actually boots. Also, I recall reading some forum posts discussing HDMI on the AS-60x that confirmed that. Sigh.

I did buy an 8-port SATA card and I'm playing around with the old 4tb drives to get a sense of how MDADM works. If nothing else, this project has been... educational. If I had just replaced the drives and recreated everything from scratch, I wouldn't have learned about superblocks.

Anyways, I'm going to take one more backup of my data, verify that I can access those configuration files, and report back.



Random thought. In late 2013, when the AS-6xx series came out, 4 TB drives were common enough to get discounted for Black Friday / Cyber Monday So, an 8x4 / 24TB AS-608T or 9x4 / 28TB AS-609Rx configuration was eminently reasonable, as was the possibility of upgrading from those to, say, 8-12tb drives in 2018. Given that this bug was discovered in 2013 and fixed in 2014, it feels like Asustor should have handled it and added an "offline expansion" feature to the firmware:

"Your NAS must be rebooted to complete the capacity expansion. While we expand the volume, you will not be able to access your files. After the expansion is complete, your NAS will reboot again into regular operating mode. Press OK to continue or schedule the expansion for a later time."

Modify the boot files temporarily, reboot, run resize2fs while the file system is offline, show status on the LCD if we're feeling fancy, reboot again, done.

It seems like implementing the procedure you described could have been a tiny matter of programming for Asustor. Oh well, we'll do it manually.


Thanks again! Off to test things.

[Nazar78]

NP, you should send your suggestion to Asustor, it's a good one and yes it can be easily programmed. I've sent a few about recovery features but seems to fall on deaf ears. One of those *rant* I mean suggestions, if you have spare time to read, https://forum.asustor.com/viewtopic.php?f=240&t=12860.

[Odeen]

So, update time!

So I thought that I could swap my drives out one at a time, rebuild the array, and still have a copy of my data on the OLD drives after I'm done. Well, with a few writes happening during some part of the rebuild, drives I swapped out earlier were QUITE a bit out of sync with the later drives. MDADM tried its best, and assembled an unusable volume. I realize *now* how much of a mess I created, and that I was wrong to expect the old array to be functional. Oops.

I ended up creating a new RAID6 array using my old disks, then using a combination of rsync and straight file copying to actually make my backup, which is why it took me so long to update. Weirdly enough, rsync alone was SLOOOOOW (28mb/sec), straight file copying via Samba was faster (40-45mb/sec), but running 2-3 file copies + rsync actually ut out the gigabit connection. I'm blaming my hastily-thrown-together Linux install and RAID array, along with me being an utter Linux n00b.

Once I had a backup, I followed your instructions. Thank you they worked virtually perfectly. The were only two interesting things:
1) If you have an IP specified in ADM, booting the NAS with no drives (and thus no ADM) means it gets a different IP from DHCP server.
1a) When booting with no drives, the NAS *briefly* displays a error message and its current IP address, *then turns off the LCD*
1ai) Watch the LCD carefully when booting with no drives. I imagine assigning the NAS a static DHCP lease on your router should also work, since ADM doesn't set the NICs MAC addresses. (I just forgot to do that when I set up a new router 2 years ago )

2) Renaming [volume1] header in volume.conf didn't work -- ADM simply created a new [volume1] header with a full set of configuration parameters. My volume.conf ended up looking like:

Code: Select all

[volume1]
<correct RAID config>

[volume1-bak]
<correct RAID config>

ADM showed two volumes -- functional volume1 and volume1-bak with a yellow question mark.

What *worked*, though, was deleting volume.conf entirely. ADM booted up, Storage Manager Overview showed a "System error" error message, and I was able to

• Telnet in,
  MDADM assemble volume1
  Try resize2fs
  Get an error message about running e2fsck first
  e2fsck
  Try resize2fs
  *Have resize2fs FAIL with a memory allocation error* -- because it's still the old resize2fs 1.42.6
  Be glad I took that backup
  e2fsck again -- seemingly no errors

So, unfortunately, the outdated software on the NAS can't resize such a large volume, even offline.

That said, I bit the bullet, pulled the drives, and hooked them up to my PC. Given my experience with futzing around with the old array, I was a little hesitant about unhooking my old drives to connect the new ones. I was *pleasantly* surprised to find that mdadm on the computer had no issues reassembling the NAS-created and resized array, resize2fs *1.45.6* took 25 minutes of nearly 100% CPU load but only 92mb RAM, I mounted the volume, restored volume.conf and put the drives back in the NAS.

Aaaand... Storage Manager shows a 97.82TB volume, all my data is there, and the Seagate Exos drives are definitely a bit more chattery than the old 5900RPM Barracudas but they're definitely faster.

So, thank you again for your help. It looks like the old NAS can't quite handle large file system resizes, but it seems to have no issues with running that large file system now.