Noticeable general performance degradation (guidance on troubleshooting)

Moderator: Lillian.W@AST

Post Reply
jfgratton
Posts: 24
youtube meble na wymiar Warszawa
Joined: Sun Jun 13, 2021 3:44 am

Noticeable general performance degradation (guidance on troubleshooting)

Post by jfgratton »

Hi,

I've an AS6604T unit with 2 drives (RAID1, btrfs), and 2 empty bays. Lately there has been a huge performance degradation that I cannot pinpoint. No changes were made in the config, apps, data that would explain it.

I see spikes of IOwait with not much CPU or memory use. The disks turn non-stop. I'm at the point to that I have taken one of the disks offline (removed it) to save it of excessive wear.
Performance degradation varies: docker stop/start taking way too much time, ssh connections the same, shutdown/restart taking the double of time it used to (nothing in /var/log/messages hinting at issues), even the webUI takes some time going from a blank desktop to showing icons. Accessing files to/from the NAS takes forever, etc.

I'm not aware of many tools that I could use on the system to troubleshoot. The usual swiss knife (iotop, iostat, strace, etc) are not present in this stripped-down (Alpine ?) OS. All I can see is tailing /var/log/messages, htop/top, and the webui widget that does not show much.

I've ran the diagnostic tools, no red flag, it says the drives are healthy and good working order. I'm stumped. The idea of a UI on top the NAS is to allow beginning users not to be overwhelmed in the gritty details of administering such a beast, but for experienced users, it becomes an hindrance.

Any ideas of what I could do to troubleshoot ? I've opened a support ticket to Asustor, but it looks like their SLA is not that tight, the ticket has been sitting un-triaged for 36hrs+

I'm at the point that I'm looking for a solution on how to factory reset my unit while keeping the data intact (see https://forum.asustor.com/viewtopic.php?f=240&t=12398 if you'd wish to comment on my idea for that and let me know if it's doable or not).

Thanks !
User avatar
father.mande
Posts: 1809
Joined: Sat Sep 12, 2015 2:55 am
Location: La Rochelle (France)

Re: Noticeable general performance degradation (guidance on troubleshooting)

Post by father.mande »

Hi,

First impress ... it's perhaps a network problem (if docker use network) ... all the other are linked to network.
NB to get extra tools (not all but lot) install Entware APKG that add up to +2500 packages including full GNU version of tools offer by Asustor as limited applet of busybox.

Philippe.
Last edited by father.mande on Mon Nov 29, 2021 12:38 am, edited 1 time in total.
AS6602T / AS5202T /AS5002T / AS1002T / FS6706T
jfgratton
Posts: 24
Joined: Sun Jun 13, 2021 3:44 am

Re: Noticeable general performance degradation (guidance on troubleshooting)

Post by jfgratton »

father.mande wrote:Hi,

First impress ... it's perhaps a network problem (if docker use network) ... all the other are linked to network.

Philippe.

Good afternoon,

I've ruled out docker for many reasons (no container changed : contents or config), and most importantly, the issue is still present even when all containers are stopped or the docker daemon stopped.
User avatar
father.mande
Posts: 1809
Joined: Sat Sep 12, 2015 2:55 am
Location: La Rochelle (France)

Re: Noticeable general performance degradation (guidance on troubleshooting)

Post by father.mande »

Hi,

to get extra tools (not all but lot) install Entware APKG that add up to +2500 packages including full GNU version of tools offer by Asustor as limited applet of busybox.

Philippe.
AS6602T / AS5202T /AS5002T / AS1002T / FS6706T
jfgratton
Posts: 24
Joined: Sun Jun 13, 2021 3:44 am

Re: Noticeable general performance degradation (guidance on troubleshooting)

Post by jfgratton »

father.mande wrote:Hi,

to get extra tools (not all but lot) install Entware APKG that add up to +2500 packages including full GNU version of tools offer by Asustor as limited applet of busybox.

Philippe.

Just did, thanks :-) I see you as the maintainer of the package, nice to know ;)

I see iostat there, so it's a beginning ! Unsure on how to list a package contents (basically know where the tools are) ?

--JFG
User avatar
father.mande
Posts: 1809
Joined: Sat Sep 12, 2015 2:55 am
Location: La Rochelle (France)

Re: Noticeable general performance degradation (guidance on troubleshooting)

Post by father.mande »

Hi,
Entware is developed by Zyxmon (aka Andrey Sechin) ... I am only the packager for Asustor

You can get information in Github (search Entware) and read Wiki ... but the base is OpenWrt .
some list of packages available exist (I am out of my office so I don't remember the URL)
EDIT it's : https://bin.entware.net/
when a packages is member of a "group" in Debian or other Linux, generally the tool is prefixed by group name, ex. coreutils

normally using opkg list | grep some_selection, you can reduce the output to the concerned packages
ex. if you do : opkg list | grep iostat ... the result is 3 packages : dstat ifstat sysstat with comment on what are inside.

Philippe.
AS6602T / AS5202T /AS5002T / AS1002T / FS6706T
jfgratton
Posts: 24
Joined: Sun Jun 13, 2021 3:44 am

Re: Noticeable general performance degradation (guidance on troubleshooting)

Post by jfgratton »

Nice for the extra info, I'll look it up !

This became kind of moot, really. My problems keep piling up. After you made me realise that removing disk1 to preserve its data (confirmed by Asustor's support during the night), I reinserted it. It started rebuilding the array, with an ETA of 3 days. I left it running, went to bed. I came back this morning, it was showing 82% done, and close to finish. ... but ... everything NAS-related was frozen (my linux X11 session frozen, too, as I authenticate throught an LDAP, sitting in a container on the NAS).

Rebooting the NAS (what else to do, if you cannot connect ?), it came back with all RAID information gone. So I have 2 disks in the unit, they seem healthy, but no RAID.

I've ordered an USB SATA dock on Amazon, I'll have it tomorrow. I'll plug a drive on it and see what can be recovered. Cross fingers.....
Post Reply

Return to “[Official] For AS52xx/53xx/66xx Series”