> Besides, I don't think anyone really has a local network which is faster than their SSD. Even a 4-year-old consumer Samsung 970 Pro can sustain full-disk writes at 2.000M Byte/s, easily saturating a 10Gbit connection.
You might be surprised if you take a look at how cheap high speed NICs are on the used market. 25G and 40G can be had for around $50, and 100G around $100. If you need switches things start to get expensive but for the "home lab" crowd since most of these cards are dual port a three-node mesh can be had for just a few hundred bucks. I've had a 40G link to my home server for a few years now mostly just because I could do it for less than the cost of a single hard drive.
Depending on your sensitivity to power/noise a 40Gb switch can be had somewhat inexpensively too - something like the Brocade ICX6610 costs <200$ on eBay.
I'm using Mellanox ConnectX-3 cards, IIRC they're HP branded. They shipped in Infiniband mode and required a small amount of command line fiddling to put them in ethernet mode but it was pretty close to trivial.
They're PCIe 3.0 x8 cards so they can't max out both ports, but realistically no one who's considering cheap high speed NICs cares about maxing out more than one port.
You might be surprised if you take a look at how cheap high speed NICs are on the used market. 25G and 40G can be had for around $50, and 100G around $100. If you need switches things start to get expensive but for the "home lab" crowd since most of these cards are dual port a three-node mesh can be had for just a few hundred bucks. I've had a 40G link to my home server for a few years now mostly just because I could do it for less than the cost of a single hard drive.