Overview
There are two schools of thought with Lightroom Classic: One uber catalog file vs one per {year, shoot, project, etc}. This post covers how my one uber catalog setup works and the problems faced.
The Goal
One Adobe Lightroom Classic catalog file to rule them all. This means storing my current photo library of 276,503 images taking up over 20TiB of disk. Near real-time offsite storage (via Dropbox) is preferred.
Biggest Problems
Photo Storage
With 20TiB (and growing) of photos, storage is more than just having a single SSD with all of the data. The storage, access, and sequential access speeds of this storage is important for each time you need to either view without a LR preview (stored with catalog) or edit the file.
Catalog Performance
The 5GiB SQLite3 database file that is the Lightroom Classic catalog storage is the most critical part of day-to-day Lightroom performance. About all you can do without getting very hacky is put it on local NVMe.
Lightroom Preview Storage
Storing Lightroom's previews - that's the {Smart,Full,etc} Previews locations next to the catalog file - is a bit harder than you'd think. This is because it needs both high speed access and, depending on your habits, a lot of space. I usually dedicated a 512GB NVMe drive to the catalog and the previews. This provides it both a reasonable amount of storage space plus great random access performance.
Future Growth
Most new images are DNGs with a size between 100MiB and 175MiB. Those DNGs are normally a 50 megapixel RAWs (Fuji RAF or Nikon NEF) losslessly compressed with an embedded preview plus any change history. Between a trio of moderate FPS cameras (D850, GFX 50R) shooting at up to 10fps and heavy use of those speeds, I can take a lot of photos per year. Although since moving out of the DC Metro Area the rate per year has been reduced from a five year average of 30k photos/year (+- 15k) to 14k photos in 2023.
My Solution
Platform
Since Adobe Creative Cloud didn't want to install in either Wine or Proton, I'm stuck using W10 to run the suite. But my main workstation - the only box I have that has a serious amount of horsepower - runs an Arch based distro called EndeavourOS.
Virtual Machine Hardware
To work around this I run a W10 virtual machine via KVM/QEMU with 64GiB of RAM, 48 vCPU, and a 3070 TI using hardware passthrough. I should point out my workstation is backed by a 2990wx (older Threadripper) with 32/64 cores, 128GiB of RAM, three local NVMe drives, and dual 10GbE links to my core network. The physical host also has a pair of GPUs. The VM also has three drives: One 96GiB "sata" boot drive backed by NVMe, one 450GiB virtio backed by NVMe, and a 30TiB virtio backed by NFS to a large file server.
Virtual Machine Access
While SPICE based access works well in most situations, there are better, more interesting options. Short term I'm using a mix of SPICE via Virt-Manager for day-to-day activities. For remote or performance critical usage I'll use Moonlight and Sunshine. They're basically a client and server, respectively, that runs on any OS and are both OSS. They're a lot like Parsec except they run on more than just Windows. It's worth pointing out that Sunshine on the VM can make use of the 3070 TI for hardware graphics streaming. Just one of a few reasons for passing the machine's second GPU through to the VM.
File Server
The 30TiB VirtIO disk that the W10 VM uses for RClone's cache of the photo library is stored on an NFS volume that's backed by a box with 35 HDDs each 8TB in size. They're in split into a ZFS pool in three vdevs each at RAIDz3. This pool, while not overly busy, does have other uses. Overall performance is in the 1-2 GiB/s range for reads or writes. IOPS with random IO is surprisingly good too.
Storage
Photo Storage
The actual DNGs/JPEGs/TIFFs/PSDs/etc are all stored in Dropbox. A local drive, W:\, is a virtual device created/managed by an RClone mount. This drive is basically the photo library folder in Dropbox. There is a 30TiB read/write cache stored on the 30TiB virtio disk. A number of RClone flags are used to make heavy use of this cache. While not fully populated, it's not overly fast for first-reads. But once the cache has all of the photos in it, I strongly suspect performance will be only a little less than the backing volume.
RClone Flags
Coming Soon!
Catalog & Previews
The Lightroom Classic sqlite3 based catalog file, catalog settings, and all Lightroom Smart/Full/etc previews are stored on a 450GiB virtio volume. The file for that volume on the host OS is stored on a BTRFS volume made from a pair of 512GB NVMe disks.
OS & App
The OS and apps, including the Adobe Suite, are installed on a 96GiB drive that the VM sees as a SATA disk. This is slower than VirtIO but allows booting. The host OS file is stored on the same BTRFS volume as the catalog and previews file.
Updates!
2024-05-08
Between Win stability issues (mostly a bad GPU cable) and Rclone taking its time to mirror, I think I'm moving this from an rclone mount to an rclone sync.
2024-05-16
Been using `rclone copy` to download all of the data to a NFS-mounted, ZFS-backed share with a 32TiB virtual disk on it. The vdesk with "writethrough" set for caching mode had long hang ups that'd mess with windows every 10-20 minutes of heavy writes. Changing it to "writeback" fixed this. Been working great for the copy/download since.