User:Tim/NFS server build

From Van Essen Lab

(Difference between revisions)
Jump to: navigation, search
Line 32: Line 32:
== Drive bays ==
== Drive bays ==
-
12 of [http://www.newegg.com/Product/Product.aspx?Item=N82E16817990001 Kingwin hotswap rack] - $19, has activity and power lights for each disk, activity light will help for locating a failed disk.
+
10 or 12 of [http://www.newegg.com/Product/Product.aspx?Item=N82E16817990001 Kingwin hotswap rack] - $19, has activity and power lights for each disk, activity light will help for locating a failed disk.
== Power supply ==
== Power supply ==
[http://www.newegg.com/Product/Product.aspx?Item=N82E16817703026 PC Power and Cooling "Silencer mkII" 650W] - $95, single high current 12V rail at up to 46 amps, WD Caviar green only needs 21 amps if all spin up at once
[http://www.newegg.com/Product/Product.aspx?Item=N82E16817703026 PC Power and Cooling "Silencer mkII" 650W] - $95, single high current 12V rail at up to 46 amps, WD Caviar green only needs 21 amps if all spin up at once
 +
 +
 +
== System configuration ==
 +
 +
Instead of a proprietary embedded OS with a web interface, this machine will run a full-fledged desktop operating system, for ease of maintenance.  Hardware raid controllers are expensive, and tend to use proprietary methods to label disks, so the plan is to use linux's built in software raid drivers, which are more transparent and portable.  As such, the disks only need to be connected to a standard SATA port, everything else is done in software.  However, most motherboards don't have 12-14 SATA connectors, so a SATA controller card will be required for some of the disks.
 +
 +
The most robust and high performance configuration would be to have two SSD drives, connected directly to the motherboard, in software raid 1, for booting and operating system, with as many 3TB drives as can fit in a RAID 6 array (two drives from this array can fail without losing any data), with a hot spare (a disk in the machine, but unused until a disk fails, used immediately to rebuild when a new disk is required to keep the array fully redundant).  Assuming only 10 bays will be available to the 3TB drives, this gives capacity of 7 disks, or 21TB of raw space for storage.  Without the hot spare, this increases to 24TB, and assuming we can put the SSDs somewhere other than a drive bay, it could give 27 or 30TB of raw space (depending on the hot spare).
 +
 +
Alternate configurations include using LVM to put the root filesystem on the same array as the main storage, with the drawback that an event that takes the storage partition down also takes the entire operating system down (such as failure of the controller card), but allows us to use 12 3TB drives with no separate boot drives, for 27 or 30TB of raw space, with no real performance hit.
 +
 +
A configuration that would keep all disks the same without sacrificing performance or bootability in the event of the storage array going down would be to use 2 3TB disks in a separate software raid 1 from the motherboard, for 21 or 24TB again.
 +
 +
One of the concerns for a raid 6 array on such a large amount of storage is that it may take a long time to build or rebuild, that is, to make the array able to suffer 2 disk failures without losing data.  This must be done when the array is first created, or a disk is replaced (including being "replaced" by the hot spare).  There is another more exotic setup that would alleviate this concern, though not without some drawbacks.  Instead, the array could be a raid 1+0 array, where first the disks are paired, and each pair contains a perfect copy of the other in the pair, and then the data is written across these disk pairs with blocks being written to different pairs in a round robin fashion.  Because all of the redundancy is simply in disks being mirror images, it is relatively fast to build or rebuild the array.  This setup is currently the highest general performance raid setup, but you only get half the space, in our case, 15TB for 10 disks, 18TB for 12 (depending again on boot drive setup).  The other main drawback is that failure of two drives in the same pair will cause data loss, so 2 failed disks has a chance to cause the array to become irretrievable (raid 6 requires 3 disks lost before it is irretrievable, but absolutely any 3 lost causes this, while raid 1+0 can tolerate one lost from each pair, if you are lucky).
 +
 +
== Backup plan ==
 +
 +
The plan is to build two of these for a very specific reason: in case one somehow manages to die, even temporarily, the other should keep a copy of everything on it.

Revision as of 01:38, 14 June 2011

The goal of this project is to create 2 identical NFS servers from standard parts, having roughly 20-30TB of robust storage each, for a cost of under $4000 each. One of the two will be built first to ensure compatibility of parts, and to make any necessary adjustments to the build, following which the other will be built to the final working specifications of the first.

Contents

Drives

Storage: 10 or 12 of one of the following:

Western Digital Caviar Green 3TB - 5400rpm, $150, cheapest 3TB from Western Digital

Hitachi Deskstar 3TB - 7200rpm, $180, cheapest 7200rpm 3TB on newegg

Hitachi Deskstar 3TB - 5400rpm, $130, cheapest 3TB on newegg

Seagate Barracuda XT 3TB - 7200rpm, $215, cheapest Seagate 3TB on newegg


Boot drives: 0 or 2 of one of the following:

OCZ Vertex 30GB SSD - $85, cheap but reputable solid state drive with okay performance (again, per Tom's Hardware Guide) and enough size for OS, will outperform rotational disks for booting

OCZ Vertex 2 40GB SSD - $105, slightly more space and significantly higher specifications than the above OCZ Vertex

OCZ Vertex 2 60GB SSD - $120, slightly more space, and better sustained write, nearly equivalent to Vertex 2E series

OCZ Agility 3 60GB SSD - $135, SATA III boosts speed to nearly double the Vertex 2, if motherboard supports SATA III

Seagate Barracuda 250GB - $37, cheap (slow) hard drive to let the OS boot without the SATA controller expansion card operating

Case

Lian Li aluminum ATX full tower - $320, toolless case, recommended by Jon S

Drive bays

10 or 12 of Kingwin hotswap rack - $19, has activity and power lights for each disk, activity light will help for locating a failed disk.

Power supply

PC Power and Cooling "Silencer mkII" 650W - $95, single high current 12V rail at up to 46 amps, WD Caviar green only needs 21 amps if all spin up at once


System configuration

Instead of a proprietary embedded OS with a web interface, this machine will run a full-fledged desktop operating system, for ease of maintenance. Hardware raid controllers are expensive, and tend to use proprietary methods to label disks, so the plan is to use linux's built in software raid drivers, which are more transparent and portable. As such, the disks only need to be connected to a standard SATA port, everything else is done in software. However, most motherboards don't have 12-14 SATA connectors, so a SATA controller card will be required for some of the disks.

The most robust and high performance configuration would be to have two SSD drives, connected directly to the motherboard, in software raid 1, for booting and operating system, with as many 3TB drives as can fit in a RAID 6 array (two drives from this array can fail without losing any data), with a hot spare (a disk in the machine, but unused until a disk fails, used immediately to rebuild when a new disk is required to keep the array fully redundant). Assuming only 10 bays will be available to the 3TB drives, this gives capacity of 7 disks, or 21TB of raw space for storage. Without the hot spare, this increases to 24TB, and assuming we can put the SSDs somewhere other than a drive bay, it could give 27 or 30TB of raw space (depending on the hot spare).

Alternate configurations include using LVM to put the root filesystem on the same array as the main storage, with the drawback that an event that takes the storage partition down also takes the entire operating system down (such as failure of the controller card), but allows us to use 12 3TB drives with no separate boot drives, for 27 or 30TB of raw space, with no real performance hit.

A configuration that would keep all disks the same without sacrificing performance or bootability in the event of the storage array going down would be to use 2 3TB disks in a separate software raid 1 from the motherboard, for 21 or 24TB again.

One of the concerns for a raid 6 array on such a large amount of storage is that it may take a long time to build or rebuild, that is, to make the array able to suffer 2 disk failures without losing data. This must be done when the array is first created, or a disk is replaced (including being "replaced" by the hot spare). There is another more exotic setup that would alleviate this concern, though not without some drawbacks. Instead, the array could be a raid 1+0 array, where first the disks are paired, and each pair contains a perfect copy of the other in the pair, and then the data is written across these disk pairs with blocks being written to different pairs in a round robin fashion. Because all of the redundancy is simply in disks being mirror images, it is relatively fast to build or rebuild the array. This setup is currently the highest general performance raid setup, but you only get half the space, in our case, 15TB for 10 disks, 18TB for 12 (depending again on boot drive setup). The other main drawback is that failure of two drives in the same pair will cause data loss, so 2 failed disks has a chance to cause the array to become irretrievable (raid 6 requires 3 disks lost before it is irretrievable, but absolutely any 3 lost causes this, while raid 1+0 can tolerate one lost from each pair, if you are lucky).

Backup plan

The plan is to build two of these for a very specific reason: in case one somehow manages to die, even temporarily, the other should keep a copy of everything on it.

Personal tools
Sums Database