Migrating RAID volumes

[Nazar78]

And RAID1 + RAID1 to RAID 10

RAID1 to RAID10 yes but not RAID1 + RAID1 to RAID10.

I'm sure I could help you speed up the process (USB3->NAS vs NAS<->LAN<->PC+RAID migration) but please do proceed with your plan and I hope all goes well, do let us know.

I have/had an active webserver for educational purposes, Emby and Roon running.

I'm running a few sites and there's few users login remotely so I can't turn off the NAS. I ran my custom monitoring to find any rogue process trying to wake the disks while they're supposed to be asleep, not the 5mins diagnostics tool provided by Asustor which to me doesn't provide any relevant info but I do find the hibernate log useful. I do this whenever I notice the disks are not hibernating. It could take analyzing a whole day worth of logs just to pin the issue, could be a recently updated apps or ADM itself.

From what I experienced, the disks will never sleep or NAS enter S3 as long as there's apps actively running off the internal disks. Not only apps, the OS and SWAP also resides on the same data volume disks only on different partitions. The reason why I moved the OS to the external SSD and turned off SWAP completely.

[helgrimm]

And RAID1 + RAID1 to RAID 10

I'm sure I could help you speed up the process (USB3->NAS vs NAS<->LAN<->PC+RAID migration) but please do proceed with your plan and I hope all goes well, do let us know.

Thanks again for your invaluable sparring on this one. Progress has been made, even though it's a bit early for the celebratory dance.

1) Data has been copied one way from NAS->Windows and onto one of the 4 disks in the NAS RAID10 to be. Some data has also been copied to an external USB3 disk I have.
2) After pulling the 2 disk RAID0 system volume, a new system volume has been created on one of the 4 disks
3) Data has been copied the other way back onto the new system volume from the Windows volume and the USB disk
4) All 4 disks have been installed and the new volume is being converted to a 4 disk RAID10 volume.
5) After about 22 hours of synchronizing the new RAID10 was sitting at 0% progress, as I was writing this, it's now at 21% - so the progress updates are quite slow.

So everything points to a successful conclusion in the coming days. During RAID migration the system is read only, so I can access data, but not install apps.

After this experience, I still have several questions as to what the best future course of action will be when it comes to backing up data from the NAS. I've realized the NAS locks up my data volumes up in ways that may make my data inaccessible in certain situations and I'm not quite sure what to make of it.

[Nazar78]

5) After about 22 hours of synchronizing the new RAID10 was sitting at 0% progress, as I was writing this, it's now at 21% - so the progress updates are quite slow.

So everything points to a successful conclusion in the coming days. During RAID migration the system is read only, so I can access data, but not install apps.

If you want to speed things up, ssh into the NAS, then run:

Code: Select all

sysctl dev.raid.speed_limit_min
sysctl dev.raid.speed_limit_max

Take note of its default value. Then run:

Code: Select all

sysctl -w dev.raid.speed_limit_min=5000000
sysctl -w dev.raid.speed_limit_max=5000000

It'll boost the sync to full disk speed but could affect normal usage, hence the reason why it's lowered in the first place. After done, re-run replace the value with default, or simply reboot the NAS as the settings are not persistent.

After this experience, I still have several questions as to what the best future course of action will be when it comes to backing up data from the NAS. I've realized the NAS locks up my data volumes up in ways that may make my data inaccessible in certain situations and I'm not quite sure what to make of it.

Emm I've never encountered this and I have lots of applications/tasks running at about 50% usage of 16GB RAM, well that's for another day to troubleshoot. For backup up, it's up to end-users to plan. Usually the backup rule 1 2 3 should suffice. For me, important backups from devices to the NAS, internal NAS backups utilizing both Backup & Restore plus some custom tar-gzip scripts i.e. OS/settings/containers/chroot/SQLs, then NAS to external USB RAID enclosure, finally rsync a copy to external location. All these done automatically every midnight. I'm actually quite pleased with this AS5304T. Maybe later I'll just get another USB enclosure to expand the RAID arrays, unofficially you can run split RAID array, partial on internal bays and partial on external USB and even on TCPIP

[helgrimm]

Thanks for the tips about ssh and syncing. It turns out it hadn't completed making the file system yet. As soon as that completed I had full access and have installed a few apps. I don't mind the synchronization takes a few days, because it doesn't affect my usage. I'm up and running with ROON and listening to music.

[Emm I've never encountered this and I have lots of applications/tasks running at about 50% usage of 16GB RAM

Not about errors. What I tried to formulate was about the realization that the volumes have limited life outside of the NAS. And that I'm not completely sure what the consequences could be in case of various problems.

I couldn't move the volumes about and simply copy the data. I'd have needed another Asustor NAS for that purpose. And the extension cabinet - the AS6004U - has lost in standing. The volumes I create on that one - and I still have 4 spare disks with respectable capacity that I could use for backing up data - would be completely depending on the NAS. I want the backup I make to an external cabinet to be able to connect freely to virtually any other computer for the best options in case of trouble. I'm already looking at a few 4 bay options and I'm sorry I didn't purchase one to begin with.

What happens if the NAS wipes the installation for some reason? It always happens to a few. Would the backup to the AS6004U be any good? Probably not, as they're tied to the installation.

It seems the more money you spend the more money you have to spend

running at about 50% usage of 16GB RAM

The max. capacity in the 5304T is 8 gigs is it not? It comes with 4 and is expandable to 8 is what I got from their spec sheets. That's why I expanded it with 4.

Regards

[Nazar78]

Yes for sure you can recover provided it wasn't initialized.

Officially 8GB max ram but unofficially 16GB.

Sorry outside overseas on mobile, short replies...

[helgrimm]

Yes for sure you can recover provided it wasn't initialized.

So if I escaped a burning building only saving the AS6004U, the data could be accessed? On a Windows machine or other?

I get the sense that by "you" you mean somebody with the particular skills over ssh, and not your average plug-and-play type of user (or owner in this case )

I still feel I'd be better off with a backup in a USB volume that could simply be attached to any machine - perhaps even to my router.

[helgrimm]

[img]Raid10.jpg[/img]

Interesting process. First it was silent for 22 hours. Then it updated its activities at about 80 percent of building the ext4 file system and concurrently "synchronizing". Then it's been synchronizing for a few days and now that process is done, but is now apparently "rebuilding" (could it have something to do with the fact that I replaced the drives of the previous system?).

And it still says "Faulty drives tolerated: 0" which I certainly hope will change when it's done, because it should with RAID10.

[Nazar78]

Yes for sure you can recover provided it wasn't initialized.

So if I escaped a burning building only saving the AS6004U, the data could be accessed? On a Windows machine or other?

On Windows using Linux VM like VirtualBox with its USB filters exposed from host to client or boot to a Linux recovery USB/CD/DVD/Ethernet to assemble back the array.

I get the sense that by "you" you mean somebody with the particular skills over ssh, and not your average plug-and-play type of user (or owner in this case )

Users with basic Linux background should be able to accomplish this simple recovery. It's not that difficult actually now a days one can just Google. You can play around in VM, dual boot or just boot using Linux recovery/installer media then create hdd images to get the hang of it and you'll realize the NAS is nothing other than a stripped down version of Linux (busybox).

I still feel I'd be better off with a backup in a USB volume that could simply be attached to any machine - perhaps even to my router.

Yes it's simpler but it lacks software RAID. That said, I'm also using RAID running in Debian 11 chroot on my Asus RT-AX86U router.

[Nazar78]

[img]Raid10.jpg[/img]

Interesting process. First it was silent for 22 hours. Then it updated its activities at about 80 percent of building the ext4 file system and concurrently "synchronizing". Then it's been synchronizing for a few days and now that process is done, but is now apparently "rebuilding" (could it have something to do with the fact that I replaced the drives of the previous system?).

And it still says "Faulty drives tolerated: 0" which I certainly hope will change when it's done, because it should with RAID10.

Yup2 it will change to one disk tolerance once done. Rsync is when you first migrate, rebuild is when one of the disks got replaced. As mentioned previously you can speed up the process however be careful not to remove any other disks while at this stage else render the array useless.

Edited: From my honest opinion, it's better to speedup the rsync/build process in exchange for less usability because if another disk fail during this process, unless the array is designed to tolerate 2 disks failures, the array will fail and there goes the data. I had RAID10 new disk failure which happened twice over two different new disks in a period of few months.

[helgrimm]

Users with basic Linux background should be able to accomplish this simple recovery. It's not that difficult actually now a days one can just Google. You can play around in VM, dual boot or just boot using Linux recovery/installer media then create hdd images to get the hang of it and you'll realize the NAS is nothing other than a stripped down version of Linux (busybox).[/color]

Thanks for that information. I do actually have a PC running Ubuntu on an Asus Mini PC that I use as a media center. But say the situation was real - how would I connect the disks? The mini PC only has one SATA connection for the system SSD (2½ inch). Or if I booted my laptop into Linux from USB - again, how to get the disks connected.

On the AS5304T things are a bit different in case you can still SSH into it.

Yes it's simpler but it lacks software RAID. That said, I'm also using RAID running in Debian 11 chroot on my Asus RT-AX86U router.[/color]

I've found a product 2 thirds the price of the AS6004U that has 4 slots and offers JBOD, RAID0. RAID1 in 2 pairs and with the lastest USB3. I'm going to go with that one. I'll rest more assured when I know that my data can be moved and accessed independently on almost any external hardware and software. You can have a look if you like - https://www.fantec.de/fileadmin/user_upload/Fantec-QB-35U31R-Manual_EN.pdf

my Asus RT-AX86U router.

I'm going off topic a bit

My NAS before the Asustor was a WD MyCloud (I had 2x4TB). Actually one of those 2 volumes I've been talking about copying data from in this thread consists of the two 4TB disks I've yanked from the WD MyCloud enclosures. I have the same NAS as you and my router? You guessed right, it's an Asus RT-AX86U. Now I've read on your homepage that your previous NAS was a WD MyCloud as well. Do you also have a sofa from IKEA?