Patriot Pyro 120G SSD RAID
While I periodically write about technology, I try do so in a business context. I will make an exception for this entry though as I want to document the gotchas we've ran into for others to benefit from our experiences. I myself Google'd and Google'd to no avail trying to find answers to these questions while researching what to buy and analyzing rest results.
SSD
Let's first start with this premise. If you want fast storage, you want SSDs. They will wear out fast from over-use but to be honest, that's not much different than hard drives dying from mechanical failure from over-use. And after burning out our previous SSDs, we picked 5 of the following drive as replacements:
And since we process a lot of data every week, we aimed high. The data processing server motherboard has 6 x SATA3 (6Gbps) connectors -- allocate the 1st for the boot OS leaving 5 for data drives. 5 500MB/s SSDs striped together? 2500MB/s! Woohoo!
CPU & Motherboard
After all the hardware arrived, we built our replacement data processing server on the following platform:
While there are far more comprehensive disk benchmarking tools, I started with HDPARM for simplicity purposes to test the upper bounds. The command used was "hdparm -t /dev/sdX" producing the following results:
The key part of the above diagram is the connection between the 890GX and SB850 chips where it says Alink Express III. So there we have the limitation -- 2 GB/s. And because the arrow is a 2-headed arrow, that number says you can push 1 GB/s one way and 1 GB/s the other way at the same time for an aggregate total of 2 GB/s. 1 GB/s is 1024 MB/s which is close enough for government work to our 1120MB/s number if the latest SB850 revision is now 10% faster that originally depicted in this diagram.
Highpoint Rocket 620
As it seems like a waste to run our SSDs at less than half speed, we want to squeeze more speed out. And from looking at that diagram, I noticed the PCIe connections to the northbridge have 16 GB/s bandwidth. What if we put SATA3 6Gbps adaptors there and moved 3 of the SSDs over? This way the SB850 could handle 2xSSDs for 1000 MB/s with the 1500 MB/s of SSDs connected to the northbridge. We pulled a Highpoint Rocket 620 card from another server and hooked up 2 SSDs it. The results:
PostgreSQL SSD Bandwidth Tests
But since our old SSDs were now unsable, we had to go live with the new hardware regardless of sub-optimal performance. After copying over the databases, I started with real world testing running PostgreSQL queries while tuning disk configuration options. What I found was the maximum data we could push through any single query was about 800 MB/s on a Phenom II X6 3.3GHz. As PostgreSQL cannot use use multiple processors/cores to speed up a single query, it turns out the SB850 1100 MB/s limit is not as great as it seems.
Hardware Upgrade Options
PostgreSQL can assign different queries to different processors/cores so if I had 2 queries hitting 2 different giant tables, that would in theory need 1600 MB/s for maximum speed -- 2400 MB/s for 3 queries. For these scenarios, it is "possible" high-end 6Gbps SAS/SATA RAID cards that can push all these SSDs. However, I'm beyond trusting specifications only and buying $1000 adaptors sight unseen for us to be the guinea pigs is not very appealing.
The best option appears to be a different CPU/MB platform not only to increase total disk I/O bandwidth but to supply a faster CPU to drive single-threaded PostgreSQL queries beyond 800 MB/s. The options I discarded after doing research:
Woo! 4 GB/s for SB950 Alink Express III. That could mean 2.5 mega PostgreSQL queries could run at maximum speed. Of course, this is just a diagram -- who knows if this is true or not in the real world.
Conclusions
Beware if you plan on creating a mega SSD RAID0 array in hopes of going above 1 GB/s transfer rates. This is the bleeding edge as system platforms were mostly designed for the previous generation of SSDs where you were expected to stuff at most 4 x 300 MB/s SATA2 SSDs. Luckily, there is not much price difference between using a 2 x 240G SSD array versus a 4 X 120G SSD array. What you gain in possible future speed enhancements as ecosystems evolve to catch up, you do lose in possible future disk space expansion (by going from 2 x 240G to 4 x 240G).
Our plan for now is to live with the mere 1100 MB/s disk subsystem limit and wait for the FX-8170 3.9GHz (4.5GHz turbo) to be released. At that point, we'll upgrade the CPU and motherboard -- if the disk performance doesn't pan out, at least we might get a faster processor to push a single-threaded PostgreSQL query closer to the limit.
SSD
Let's first start with this premise. If you want fast storage, you want SSDs. They will wear out fast from over-use but to be honest, that's not much different than hard drives dying from mechanical failure from over-use. And after burning out our previous SSDs, we picked 5 of the following drive as replacements:
Patriot Pyro 120G SSDBy comparison, the latest spinning magnetic hard can transfer data at 150MB/s when on the outer edge (new drive), 100MB/s on the inner edge (filled up hard drive) with a drive latency of 5ms (15K RPM) to 12ms (7200RPM). In short, SSDs of this class are mind-bogglingly faster. But if 500MB/s is not fast enough for you, you can put 2 drives together in a RAID0 array for 1000MB/s transfer rates.
SandForce SF2281 controller
SATA 3.0 6Gbps
Sequential Read Speed: 550MB/s
Sequential Write Speed: 515MB/s
Random Write IOPS: 85,000
Read Latency: 0.05ms
Write Latency: 0.25ms
And since we process a lot of data every week, we aimed high. The data processing server motherboard has 6 x SATA3 (6Gbps) connectors -- allocate the 1st for the boot OS leaving 5 for data drives. 5 500MB/s SSDs striped together? 2500MB/s! Woohoo!
CPU & Motherboard
After all the hardware arrived, we built our replacement data processing server on the following platform:
AMD Phenom II X6 1100TSimple SSD Bandwidth TestsASUS M4A87TD
- 3.2GHz / 3.7GHz turbo
- 6-core
- AMD 870 northbridge
- AMD SB850 southbridge
While there are far more comprehensive disk benchmarking tools, I started with HDPARM for simplicity purposes to test the upper bounds. The command used was "hdparm -t /dev/sdX" producing the following results:
1 x Patriot Pyro: 520 MB/sIt doesn't take much analysis to see what we ran into. My initial guess was you simply cannot shovel more than 1100 MB/s from either the SATA subsystem or the overall southbridge. Now to confirm my guess, I read through a ton of reviews for hardware using the SB850 chipset but could not find anybody who was mad enough to stack 2.5GB/s worth of SSDs into a system. I did run into the following image though as a common graphics filler material:
2 x Patriot Pyro RAID0: 1040 MB/s
3 x Patriot Pyro RAID0: 1120 MB/s
4 x Patriot Pyro RAID0: 1120 MB/s
5 x Patriot Pyro RAID0: 1120 MB/s
The key part of the above diagram is the connection between the 890GX and SB850 chips where it says Alink Express III. So there we have the limitation -- 2 GB/s. And because the arrow is a 2-headed arrow, that number says you can push 1 GB/s one way and 1 GB/s the other way at the same time for an aggregate total of 2 GB/s. 1 GB/s is 1024 MB/s which is close enough for government work to our 1120MB/s number if the latest SB850 revision is now 10% faster that originally depicted in this diagram.
Highpoint Rocket 620
As it seems like a waste to run our SSDs at less than half speed, we want to squeeze more speed out. And from looking at that diagram, I noticed the PCIe connections to the northbridge have 16 GB/s bandwidth. What if we put SATA3 6Gbps adaptors there and moved 3 of the SSDs over? This way the SB850 could handle 2xSSDs for 1000 MB/s with the 1500 MB/s of SSDs connected to the northbridge. We pulled a Highpoint Rocket 620 card from another server and hooked up 2 SSDs it. The results:
Highpoint R620, 1 x Patriot Pyro: 200 MB/sRather underwhelming. Even though the R620 is a SATA3 6Gbps card, the R620's Marvel chipset is just not fast enough to drive latest, greatest SSDs. I'm sure this card works perfectly fine for SATA3 hard drives though.
Highpoint R620, 2 x Patriot Pyro RAID0: 400 MB/s
PostgreSQL SSD Bandwidth Tests
But since our old SSDs were now unsable, we had to go live with the new hardware regardless of sub-optimal performance. After copying over the databases, I started with real world testing running PostgreSQL queries while tuning disk configuration options. What I found was the maximum data we could push through any single query was about 800 MB/s on a Phenom II X6 3.3GHz. As PostgreSQL cannot use use multiple processors/cores to speed up a single query, it turns out the SB850 1100 MB/s limit is not as great as it seems.
Hardware Upgrade Options
PostgreSQL can assign different queries to different processors/cores so if I had 2 queries hitting 2 different giant tables, that would in theory need 1600 MB/s for maximum speed -- 2400 MB/s for 3 queries. For these scenarios, it is "possible" high-end 6Gbps SAS/SATA RAID cards that can push all these SSDs. However, I'm beyond trusting specifications only and buying $1000 adaptors sight unseen for us to be the guinea pigs is not very appealing.
The best option appears to be a different CPU/MB platform not only to increase total disk I/O bandwidth but to supply a faster CPU to drive single-threaded PostgreSQL queries beyond 800 MB/s. The options I discarded after doing research:
Intel i7 Sandy Bridge: 2 GB/s to P55 SB, no ECC memory supportAfter tossing out everything else, the only option left was the AMD SB950 chipset. It's interesting how the SB850 diagram was plastered in almost every review but I had a hard time finding a similar one for the SB950. I finally found it after clicking through a lot of links for motherboard reviews.
Intel i7 Nehalem: 2 GB/s to ICH10 SB, no ECC memory support
Intel i7 Xeon: seems like 2 GB/s non-video to H55 SB, nosebleed pricing for 6-core
AMD Opteron C32: 6Gbps SATA support only through LSI SAS, 2.8GHz max speed
AMD Opteron G34: 6Gbps SATA support only through LSI SAS, 2.4GHz max speed
Woo! 4 GB/s for SB950 Alink Express III. That could mean 2.5 mega PostgreSQL queries could run at maximum speed. Of course, this is just a diagram -- who knows if this is true or not in the real world.
Conclusions
Beware if you plan on creating a mega SSD RAID0 array in hopes of going above 1 GB/s transfer rates. This is the bleeding edge as system platforms were mostly designed for the previous generation of SSDs where you were expected to stuff at most 4 x 300 MB/s SATA2 SSDs. Luckily, there is not much price difference between using a 2 x 240G SSD array versus a 4 X 120G SSD array. What you gain in possible future speed enhancements as ecosystems evolve to catch up, you do lose in possible future disk space expansion (by going from 2 x 240G to 4 x 240G).
Our plan for now is to live with the mere 1100 MB/s disk subsystem limit and wait for the FX-8170 3.9GHz (4.5GHz turbo) to be released. At that point, we'll upgrade the CPU and motherboard -- if the disk performance doesn't pan out, at least we might get a faster processor to push a single-threaded PostgreSQL query closer to the limit.
(Filed in technology)
this is a really interesting article.
I appreciated the hdparm command info.
How did you measure your PostgreSQL query throughput? I would like to try measuring some different configurations and think about best practices.
thanks,
Greg