12 Dec 2024
I use a bunch of 60-bay DAS units for my spinning rust at home. The performance difference between the units with 6Gbps SAS and 12Gbps SAS is definitely notable. With four lanes per SFF-8644 and two cables on each of the pair of controllers, you end up with 240Gbps (SAS 6Gbps) or 480Gbps (SAS 12Gbps) per DAS. You'd think 28GB/s with 6Gbps would be more than enough since the units usually max out at two or three GB/s during zpool scrubs in practice. Even with 30MiB/s per disk as I usually get, that's only 1.75GiB/s per 60-Bay DAS.
Best guess about reason is the SAS-level and/or host-level path level multipathing not being 100% efficient. That and maybe the backplanes-to-controllers interconnects or some other DAS level bottleneck.