If I were to go with Lustre

The network side of my rack redo is almost done. Once the two SRX650s on top are removed it'll be good to go. 

Gear

• 4x SRX650s w/ 10GbE
• 2x RackSwitch G8264
• 2x RackSwitch G8000 w/ 10GbE
• 1x Digi console serve
• 1x Avocent KVM
• 2x Voltaire Infiniband 4036 - 4xQDR (aka 40Gbps)

IB Network

The IB network will be used for Lustre. 

These two NDS-4600-JD-05 units each have space for 60 3.5" drives with four 6 Gbps SAS ports on each of the two controllers. The plan is to connect two R610s (eventually R620s) to each of them with the DAS units partitioned so that each of the R610s/R620s has 30 disks (well, 15-disks on each of a pair of redundant 6 Gbps SAS lines). There will be a Ceph OSD per disk on each of the 30 disks. Half of the 30 disks will be 8TB and half will be either 3TB or 2TB disks. 

Updated Plans!

First up, the door install is set for Tuesday, February 11th. Once the door is in two of the libraries (the two working ones) will come inside. I'll probably end up using movers to handle getting the boxes, shelving, tool boxes, and other stuff out to the garage. Although I'm not sure yet if I'll try to get the libraries inside before, during, or after the movers come. 

First it's a single drive, then a small 16-slot library. Then you get into the TS3200 territory. Before you know it you have three Scalar i6000s...

Next up in the z uber move...Getting the tape libraries from the garage to the server room. This will require installing a much larger door (two doors really), building a ramp, and moving a metric ton of boxes of gear/cables/parts/etc.

On Tuesday 2019-10-29 at 11:01 AM US/East a team of amateur radio operations, myself included, made contact with the International Space Station. Thirteen students from Farmwell Station Middle School (FSMS), part of Loudoun County Public Schools (LCPS), were able to ask questions of Dr. Drew Morgan.

This script helps identify what disks are in use (or not in use) by ZFS. It's a stupid-simple python 3 script that requires only 'sh'. You can get that via 'pip3 install sh'. 

#!/usr/bin/python3
import re
from sh import zpool,lsblk,ls
from pprint import pprint

print("Finding all disks")
sdisks = str(lsblk('-d').stdout)
# sdbl  67:240  0   7.3T  0 disk
alldisks = re.findall(r'(sd[a-z]+)\s+\d+:\d+\s+\d+\s+(.*?)\s+\d+\s+disk',sdisks,re.MULTILINE)
print("Found %d disks on system"%len(alldisks))

Check `pvdisplay` as Ceph can use LVM. 

zfs destroy data/osd 
cannot destroy 'data/osd': dataset is busy