Lockerstor 6 - poor network performance due to scheduling issues?

This is where you can find the latest ADM official release. Feel free to discuss any questions regarding it here.

Moderator: Lillian.W@AST

Post Reply
reese
Posts: 3
youtube meble na wymiar Warszawa
Joined: Sat Jun 18, 2022 8:30 pm

Lockerstor 6 - poor network performance due to scheduling issues?

Post by reese »

I have an AS6704T with 4 drives in raid5 configuration that I purchased to replace a Synology DS418 - also with 4 drives in Raid5.

The AS6704 has the AST10-G2 10Gbit card and is connected directly to a fast PC also with a 10Gbit card.

I purchased the AS6704 to increase the backup speed over the DS418, but unfortunately the AS6704 is barely any faster and does not acheive more than 1.1Gbs transfer rate on either the 10Gbit card or when connecting via the internal 2.5Gbit ports (RTL8125 based).

I have tried NFS, SMB, MTU settings or 1500, 4000, 9000 and 16000.
I have tried fixing network speeds to 2.5, 5 and 10 Gbit.
I have tried setting all combinations of LTO / GRO and they basically made marginal if any difference.

The AS6704T raid5 array copies data at 2.36Gbs locally but slows to 1.1Gbs over any network link.
The drives on the receiving PC can write data at a sustained rate of 3.4Gbs and are not the limiting factor.

When using iperf between the NAS and the PC - the throughput is 7Gbs - so there isn't a problem with TCP/IP throughput.

The fundamental difference between the Asustor NAS and the Synology NAS is that:

the Synology NAS (which is only 1Gbit ethernet) transfers at a Solid 1Gbs for all large file copies. The bandwidth is a solid consistent line at 1Gbs

The Asustor transfers at upto 2Gbs and then pauses, then transfers again, then pauses. The data is only copied in bursts and the resulting overall speed never exceeds 1.1Gbs.
Even when the AS6704T is set down to 1Gbs network speed, it still only copies in bursts and only achieves 580Mbs. which is 30% slower than the Synology NAS.

I have requested support from Asustor - including details graphs and settings, but not received a reply.

So my summary is as follows:

- The AS6704T appears to have OS level scheduling issues which basically limited data transfer rate.
- The appears to be no point in purchasing or using a 10Gbit card (at least with RAID5 drives) as the NAS cannot stream data faster than 1.1Gbs

So far I am disappointed that I have had no appreciable performance improvement from the AS6704T compared to the DS418 - despite that Asustor device having faster drives, a far faster CPU and 10Gbit ethernet compared to 1Gbit in the Synology.

I am hoping that they can isolate the cause of the poor performance and address it in a software update.
User avatar
orion
Posts: 3482
Joined: Wed May 29, 2013 11:09 am

Re: Lockerstor 6 - poor network performance due to scheduling issues?

Post by orion »

That's definitely something wrong. I think you should send a support ticket to asustor to find the root causes. http://support.asustor.com/
User avatar
snapshot
Posts: 129
Joined: Sat Mar 16, 2013 6:58 am
Location: Wiltshire, England

Re: Lockerstor 6 - poor network performance due to scheduling issues?

Post by snapshot »

I have a Lockerstor 6 (AS6706T, the AS6704T is a Lockerstor 4) but I don't have the 10Gbps card yet. Copying a multi-GB file from the NAS to my PC (which does have a 10Gbps card) runs at a steady 2.3Gbps. I have 6 drives in RAID 10 so it could be a RAID 5 artefact. But have you checked the health status of the individual drives? What you're describing could just be a dodgy drive.
What version of ADM do you have? Mine is 4.1.0.RHU2
utalevy
Posts: 1
Joined: Wed Jun 22, 2022 2:25 pm

Re: Lockerstor 6 - poor network performance due to scheduling issues?

Post by utalevy »

You should send a support ticket to asustor.
reese
Posts: 3
Joined: Sat Jun 18, 2022 8:30 pm

Re: Lockerstor 6 - poor network performance due to scheduling issues?

Post by reese »

Hi I did send a support ticket to Asustor - and three follow ups. They have not responded :(

I have provided them with full logs, a spreadsheet with testing every combination, throughput graphs showing the difference in performance between Synology (consistent throughput) and Asustor - bursty traffic.

The drives are all brand new and I have tested the drive throughput locally on the NAS to avoid the networking. Locally the array copies at 2.4Gbps.

If you are getting 2.3Gbps with raid 10, I would be curious to see if you are getting bursty traffic still?
Attachments
throughput profile - ds418 vs AS6704T
throughput profile - ds418 vs AS6704T
Screenshot from 2022-06-22 19-23-38.png (118.04 KiB) Viewed 4377 times
User avatar
Nazar78
Posts: 2002
Joined: Wed Jul 17, 2019 10:21 pm
Location: Singapore
Contact:

Re: Lockerstor 6 - poor network performance due to scheduling issues?

Post by Nazar78 »

Don't touch the MTUs, jumbo frames will not work correctly in this kind of environment. Also just to be sure, there's no aggregation happening between the links. Use SMB(multichannel off)/NFS IPs from its single ethernet to test.

Have you tried isolating the links?

Three scenarios taken from your draft:
1. Break the 10G link, what is the 1Gbps speed graph like between PC Linux and AS6704T over the 1G switch?

2. While the 10G link is still down, swap the DS418 link with the AS6704T. Now what's the speed graph like between PC Windows and AS6704T over the 1G switch?

3. Connect only the 2.5G or 10G directly between PC Linux and AS6704T bypassing the switch, assign them static IPs. Now what's the speed graph like for each 2.5G and 10G?

While I don't have a 10G card/switch nor a AS6704T, my 2.G network could also sustain 290-295MB/s (2.3xGbps) on both RAID5 and RAID10 during off-peak times.
AS5304T - 16GB DDR4 - ADM-OS modded on 2GB RAM
Internal:
- 4x10TB Toshiba RAID10 Ext4-Journal=Off
External 5 Bay USB3:
- 4x2TB Seagate modded RAID0 Btrfs-Compression
- 480GB Intel SSD for modded dm-cache (initramfs auto update patch) and Apps

When posting, consider checking the box "Notify me when a reply is posted" to get faster response
reese
Posts: 3
Joined: Sat Jun 18, 2022 8:30 pm

Re: Lockerstor 6 - poor network performance due to scheduling issues?

Post by reese »

Thankyou for your feedback.

Today I went to re-test in order to validate the whole environment. I re-started the NAS (no other changes at all). And the AS6704T is now copy at near twice the speed. The bandwidth graph no longer shows the bursty traffic transfer.

I am not sure why it changed, but this speed is now close to what the NAS gets copying directly to /dev/null on the device locally - so it is performing as well as can be expected.

So I will do some more tesitng, but what ever the issue was seems to have resolved itself :)
Attachments
Screenshot from 2022-06-23 07-41-24.png
Screenshot from 2022-06-23 07-41-24.png (36.64 KiB) Viewed 4364 times
Post Reply

Return to “[Official] For AS-60X Series”