> For those not keeping count, total hardware spend is in the 13k-20k USD ballpark, by my count.
Yep! Right around the $13.5k mark.
> This homelab does seem compute focused, which might be right for OP but is normally a mistake that people make when they build their first homelab.
Very compute focused for the specific projects that I intend to work on.
> I'm wondering what OP's home internet bandwidth is. It seems odd to have so much compute behind a network bottleneck unless OP has high-compute-small-output workloads (ml training, data searching/analysis, NOT video encoding)
1gbps symmetric + a failover Starlink connection. Not a ton of large output workloads at the moment.
> However, keep in mind that a single machine will make it hard to tinker with large parts of the stack.
Very much in agreement here. This is one of the reasons I went with multiple machines.
> I'm willing to pay a premium on compute to have less commitment and physical footprint.
I also like this mindset, but my other hobbies are piano (and I'm very sensitive to the way that the keys are weighted, so I prefer playing a real grand piano vs a portable/mini electronic piano) and woodworking (even more bulky equipment), so I'm already pretty committed to putting down roots.
This is definitely one of the purchasing decisions that I regret. My backups are robust and trustworthy enough that I don't have data loss concerns, but the software is atrocious and the customizability is extremely limited.
e.g. I wanted to serve tftp directly from the NAS. I can log in and `apt install tftpd-hpa`, but that package has to be reinstalled every time the NAS updates.
I'll be replacing this in the medium term, but I'm not buying more hardware for a little while lol
How do you know if it's overkill without knowing anything about what I am/will be running on it? Certainly, there are projects for which this setup might be appropriately sized, wouldn't you think?
Incidentally, this setup does leave me some significant headroom in terms of compute resources, but that's by design.
Talos is absolutely incredible. There's a learning curve to it, but it's not as steep as it seems.
I started with Ansible, but found myself getting really annoyed at the amount of YAML I was shoveling around for basic things, so I ended up writing a series of bash scripts that rsync files out to where they need to go, run actions when a particular file changes, etc. Provisioning scripts are bundled up and piped to the remote machine over SSH. It's been pretty nice. I'm thinking about building that out into a separate project somewhere.
I'd love to check out what you're working on! The link seems to be broken though.
Talos (and Talhelper) seem pretty reasonable so far. Digging Sops too for managing secrets. I was using Ansible Vault before which worked but was weirdly cumbersome to automate (go figure), and Sops seems to Just Work™.
> The link seems to be broken though.
Yeah, I'm a world-class infra engineer. smdh. Changed how the DNS record was created but didn't push my changes so they were reverted by a scheduled job facepalm
> Digging Sops too for managing secrets. I was using Ansible Vault before which worked but was weirdly cumbersome to automate (go figure), and Sops seems to Just Work™.
I considered both of those and ended up using the External Secrets Operator + 1Password for my secrets. Maybe not the _best_ solution, but it saved a fair amount of effort on my part.
Oh, that's super cool! I somehow missed that ESO and 1Password would work together. I set up a 1Password Connect Server at one point but ended up not doing much with it. Hmm...
I'm currently sitting at < 200W, but I expect that to go up with a higher workload. The SER9s idle at 5-7W, but they can run at 50-60W sustained without thermal throttling. Some reviewers have claimed that they can run at 75-80W sustained for 10-15 minutes, but I think that's pretty unlikely.
We may be optimizing for different needs. For instance, while I was able to get a significant amount of extra height, I didn't have a lot of cabinet depth to work with, which is somewhat limiting for traditional server hardware. There are short-depth options out there, but I also wanted at lease some GPU capability. The integrated GPUs in the SER9s are not top of the line by any means, but they're more than capable for what I want to be working on.
You read that right. Currently, the DR plan is "replace/repair the machine and bring everything back up from a backup". It's not a good plan, but it's also only the short term plan. Longer term, I'll likely add another control plane node or two.
I am very much on board with this line of thinking. Because things are still somewhat in flux, it was much easier to plan for excess cabling and have a place for that cabling to live in the rack so that things can be moved if needed. I'll probably re-cable it with cut-to-length cables in the future.
One thing that I haven't found a solution for though: I have a lot of USB and HDMI cable coiled up behind the Beelink boxes (for KVM connectivity). I've found the normal length cables (1', 3', 6', etc), but I haven't been able to find custom length cables for those specific connections. Do you happen to know anywhere I can find those?
Yep! Right around the $13.5k mark.
> This homelab does seem compute focused, which might be right for OP but is normally a mistake that people make when they build their first homelab.
Very compute focused for the specific projects that I intend to work on.
> I'm wondering what OP's home internet bandwidth is. It seems odd to have so much compute behind a network bottleneck unless OP has high-compute-small-output workloads (ml training, data searching/analysis, NOT video encoding)
1gbps symmetric + a failover Starlink connection. Not a ton of large output workloads at the moment.
> However, keep in mind that a single machine will make it hard to tinker with large parts of the stack.
Very much in agreement here. This is one of the reasons I went with multiple machines.
> I'm willing to pay a premium on compute to have less commitment and physical footprint.
I also like this mindset, but my other hobbies are piano (and I'm very sensitive to the way that the keys are weighted, so I prefer playing a real grand piano vs a portable/mini electronic piano) and woodworking (even more bulky equipment), so I'm already pretty committed to putting down roots.