I agree that empirical data in programming is difficult, but i’ve used many of those languages personally, so I can say for myself at least that I’m far more productive in Go than any of those other languages.
> As an aside, I’d even go so far as to say that the main problem with C++ is not that it has so many features in number, but that its features interact with each other in unpredictable ways. Said another way, it’s not the number of nodes in the graph, but the number of edges and the manner of those edges.
I think those problems are related. The more features you have, the more difficult it becomes to avoid strange, surprising interactions. It’s like a pharmacist working with a patient who is taking a whole cocktail of prescriptions; it becomes a combinatorial problem to avoid harmful reactions.
I’ve been using Python since 2008, and I don’t feel like I understand very much of it at all, but after just a couple of years of using Go in a hobby capacity I felt I knew it very well.
When Go was new, having better package management than Python and C++ was saying a lot. I’m sure Go wasn’t the first, but there weren’t many mainstream languages that didn’t make you learn some imperative DSL just to add dependencies.
I picked up Go precisely in 2012 because $GOPATH (as bad as it was) was infinitely better than CMake, Gradle, Autotools, pip, etc. It was dead simple to do basic dependency management and get an executable binary out. In any other mainstream language on offer at the time, you had to learn an entire programming language just to script your meta build system before you could even begin writing code, and that build system programming language was often more complex than Go.
The fact that virtualenv exists at all should be viewed by the python community as a source of profound shame.
The idea that it's natural and accepted that we just have python v3.11, 3.12, 3.13 etc all coexisting, each with their own incompatible package ecosystems, and in use on an ad-hoc, per-directory basis just seems fundamentally insane to me.
It's still pretty mid and still missing basic things like sets.
But mid is not all that bad and Go has a compelling developer experience that's hard to beat. They just made some unfortunate choices at the beginning that will always hold it back.
Similar story for me. I was looking for a language that just got out of the way. That didn’t require me to learn a full imparable DSL just to add a few dependencies and which could easily produce some artifact that I could share around without needing to make sure the target machine had all the right dependencies installed.
I've often wondered why there isn't a simpler identity provider service that does the thing that ~90% of applications need without all of the complex configuration.
The world of Auth has been made miserable with everything having to support OAuth2/LDAP/SSO/SAML etc., plus a million versions of access control, session configs, yadda yadda.
Each of these has their own (usually legitimate) purpose, but also each one has to integrate with other providers that each don't follow and/or extend the spec in their own special way. And the pain goes on and on.
Obviously you can make a product that only does really good username/password auth for example, but there's always more pressure to implement more things for another use case.
Another problem is also that "standards" like OAuth2/OIDC are used for a thousand use cases that weren't intended by the authors, so people get really creative with them.
Plus the spec itself is vague on many essential things, for example how logout should work.
Thankfully I never had to implement SAML but I would guess it's even worse there...
Ironically, their hard dependency on Docker is a showstopper for me - none of my systems run Docker Engine, they use containerd and Podman, neither of which are supported.
I hadn't heard of them, but I'm looking at their GitHub page now and they seem to support Kubernetes, which makes me think they must support containerd, right?
> One bonus (for us) for Keycloak was that it was JVM-based, meaning it was easier to integrate our existing JVM libraries. Though its use of Hibernate was frustrating at times, heh
I'm pretty frightened of running Java services, not because of the JVM, but because every Java app I've had to operate is infinitely configurable via some poorly documented XML file, and trying to reverse engineer the XML file is often difficult because you have to route through a bunch of Spring Boot magic (preventing an easy grep for configuration options). And on top of that the defaults are rarely system defaults, so even figuring out _where_ the application expects to find its configuration file is nontrivial and logging by default is separated into some unknown number of log streams which each go to a completely different path on disk by default and each one has its own configuration option for telling it to log to stderr.
By contrast, Go services are pretty explicit about where they expect their configuration, they usually log to stderr by default, you can pretty much drop them into any Docker image and run them without issue (no need to custom tune the JVM or bundle up dependencies and ensure the right runtime version). I'm told that the Java world is changing, but in the mean time I will put up with _a lot_ in the way of missing features in order to avoid running a Java application.
The nice thing about the Java base here was that instead of trying to solve problems with a mess of configuration, we could just write our own code plugging directly into / replacing parts of Keycloak. Definitely don't disagree with you about the pain of XML, but that wasn't an issue for us here at all
Yeah, I fully believe that there are advantages for your team, and even that Keycloak is much better than the Java apps I have had to operate. I'm just traumatized. :)
Even if all parties involved were Jewish, I still don’t understand why that would make it not of interest to others that a corporation is blacklisting people on the basis of a mainstream political viewpoint.
As a Jellyfin user, this hasn’t been my experience. I needed to do a fair bit of work to make sure Jellyfin could access its database no matter which node it was scheduled onto and that no more than one instance ever accessed the database at the same time. Jellyfin by far required more work to setup maintainably than any of the other applications I run, and it is also easily the least reliable application. This isn’t all down to SQLite, but it’s all down to a similar set of assumptions (exactly one application instance interacting with state over a filesystem interface).
Jellyfin isn’t meant to be some highly available distributed system, so of course this happens when you try to operate it like one. The typical user is not someone trying to run it via K8s.
Yeah, I agree, though making software that can run in a distributed configuration is a matter of following a few basic principles, and would be far less work than what the developers have spent chasing down trying to make SQLite work for their application.
The effort required to put an application on Kubernetes is a pretty good indicator of software quality. In other words, I can have a pretty good idea about how difficult a software is to maintain in a single-instance configuration by trying to port it to Kubernetes.
Most of the issues with the database are old sins from Emby. With 10.11 the Jellyfin team finally managed to clean up that mess so they can move forward with a clean implementation.
Their blog post on moving to EFCore [1] and version 10.11 release post [2] have more details.
Yes, but you have to go out of your way when writing software to make it so the software can only run on one node at a time. Or rather, well-architected software should require minimal, isolated edits to run in a distributed configuration (for example, replacing SQLite with a distributed SQLite).
That's just not true. Distributed software is much more complicated and difficult than non-distributed software. Distributed systems have many failure modes that you don't have to worry about in non-distributed systems.
Now maybe you could have an abstraction layer over your storage layer that supports multiple data stores, including a distributed one. But that comes with tradeoffs, like being limited to the least common denominator of features of the data stores, and having to implement the abstraction layer for multiple data stores.
I’m a distributed systems architect. I design, build, and operate distributed systems.
> Distributed systems have many failure modes that you don't have to worry about in non-distributed systems.
Yes, but as previously mentioned, those failure modes are handled by abiding a few simple principles. It’s also worth noting that multiprocess or multithreaded software have many of the same failure modes, including the one discussed in this post. Architecting systems as though they are distributed largely takes care of those failure modes as well, making even single-node software like Jellyfin more robust.
> Now maybe you could have an abstraction layer over your storage layer that supports multiple data stores, including a distributed one. But that comes with tradeoffs, like being limited to the least common denominator of features of the data stores, and having to implement the abstraction layer for multiple data stores.
Generally I just target storage interfaces that can be easily distributed—things like Postgres (or maybe dqlite?) for SQL databases or an object storage API instead of a filesystem API. If you build a system like it could be distributed one day, you’ll end up with a simpler, more modular system even if you never scale to more than one node (maybe you just want to take advantage of parallelism on your single node, as was the case in this blog post).
> just target storage interfaces that can be easily distributed—things like Postgres
But as I mentioned above, that makes the system more complicated for people who don't need it to be distributed.
Setting up separate db software, configuring the connection, handling separate updates, etc. is a lot more work for most users than Jellyfin just using a local embedded sqlite database. And it would probably make the application code more complicated as well.
> But as I mentioned above, that makes the system more complicated for people who don't need it to be distributed. Setting up separate db software, configuring the connection, handling separate updates, etc. is a lot more work for most users than Jellyfin just using a local embedded sqlite database.
You can package a Postgres database with your app just like SQLite. Users should not have to know that they are using Postgres much less configuring connections, handling updates, etc.
> And it would probably make the application code more complicated as well.
Not at all, this is an article about the hoops the application has to jump through to make SQLite behave well with parallel access. Postgres is designed for parallel access by default. It’s strictly simpler from the perspective of the application.
> You can package a Postgres database with your app just like SQLite
You technically can. But that is much more difficult to do than including sqlite, and isn't how postgresql was meant to be used. And what happens when you want to upgrade the major version of postgresql? Do you now include two versions of postgresql so that you can convert old databases to the new postgresql format? I certainly wouldn't say it is "just like SQLite".
I can believe the packaging step is more difficult on some platforms, but that’s a one time cost per platform.
Upgrades are more complex because Postgres (for some reason) doesn’t endure backwards compatibility with its disk format like sqlite across major versions (minor versions work fine though). For major version upgrades you ship both binaries and use pg_upgrade or pg_dump/pg_restore. It’s annoying, but it’s a one time cost to automate. It’s not like battling concurrency bugs in sqlite.
Jellyfin isn't a Netflix replacement, it's a desktop application that's a web app by necessity. Treat it like a desktop app and you won't have these issues.
It's a local media library manager in the same vein as media servers that came before it that were intended to run on desktops and serve up content to consoles and whatever on your LAN back when that was the thing to do.
My point is to treat it like software from that lineage and you won't have a problem, trying to treat it like something it's not, like a distributed web app, will lead to issues.
It feels like we’re saying similar things. We both agree that its architecture makes it difficult to run with high availability, although I’ll point out that the issues documented in the article apply to single nodes and even on a single node it has pretty specific hardware requirements. I think we just disagree about whether “you have to hold it very carefully and it works just fine” is a good thing or not.
Presently I’m running my media directory and sqlite database on NFS (one big single-point-of-failure). My Kubernetes Deployment resource is configured to use the “Replace” rollout strategy (at least I think that’s what it’s called if i’m not misremembering) so there are never two concurrent instances. This means I take downtime during rollouts, but it’s fine for my use case.
One of the more difficult bits (which is not really Jellyfin’s fault) is that the application must run on nodes with access to an adequate GPU to handle any on demand transcode tasks, which requires making the GPU available to them Kubernetes pod and also telling the scheduler which nodes have a GPU and which do not. For that I used node feature discovery along with some intel specific plugin for the GPU (my GPU was an integrated intel GPU).
As a user of Jellyfin, I’m very sad that it doesn’t just use Postgres. I basically have to run an NFS system just for Jellyfin so that its data can be available to it no matter which node it gets scheduled on and also that there are never multiple instances running at the same time, even during deployments (e.g., I need to take care that deployments completely stop the first Jellyfin instance before starting the subsequent instance). There are so many unnecessary single points of failure, and Postgres would make a pretty big one go away (never mind addressing the parallelism problems that plague the developers).
Jellyfin is by far the least reliable application I run, but it also seems to be best in class.
You're from the current generation of selfhosters, which culturally is very similar to kit car builders. The next generation of selfhosters/indiehosters just want a car to get from point A to point B. Sqlite is better for those people.
That’s a bit of a strange argument considering all the hoops one needs to jump through to make Jellyfin work on account of Sqlite. I just want to run the software I use on the computers I have.
You're having issues because you're trying to shoehorn it into your desired architecture. Most people just want to run an app on their Windows laptop and start streaming their videos.
Maybe that’s what most users want, but that’s not what the software was designed to target, judging from all of the documentation and marketing. But yes, clearly the software wasn’t designed to run in a distributed fashion, and that’s kind of the point of my criticism—they had to go out of their way to couple their application in a way that precludes distributed execution. Well designed server software is trivial to distribute, and even if you never run it in a distributed configuration it makes it easy to do the basic parallelism described in this article.
I have the same experience. SQLite has been a source of most Jellyfin problems, and Jellyfin has more problems than the rest of the ~ 150 containers I run regularly.
A stateless design where a stateless jellyfin server talks to a postgres database would be simpler and more robust.
Yeah, honestly I’m kind of thinking about a media server architecture that has a stateless media server that vends links to pre-transcoded media in object storage (which video players would source from), since pretty much anything can handle mp4/h264/acc video. Maybe in the future I could add on some on-the-fly transcoding (which would happen on a dedicated cluster, reading and writing to object storage), but that seems like a pretty big undertaking.
That’s kind of an interesting and profound question. I’m inclined to answer that it’s a way of efficiently allocating resources without making decisions in other words, there is no group voting about allocating resources. Of course, “efficiency” is probably in the eye of the beholder.
I would argue that the decisions are still made, first at the individual level choices, then as the larger behavior that emerges from them. The strange part is that you can't point to any specific entity making those larger choices (aside from "the economy" or whatever) but I'd say they're still being made. This ultimately comes down to how you want to define "decision" though.
Yep, I agree with this. No one is making macro scale resource allocation decisions and yet resources are allocated reasonably efficiently (for some value of “efficiency”). The collective “decision” is an emergent property of countless individual decisions, much like an ant colony exhibits complex and sophisticated decision making as an emergent property of simple individual decisions.
> As an aside, I’d even go so far as to say that the main problem with C++ is not that it has so many features in number, but that its features interact with each other in unpredictable ways. Said another way, it’s not the number of nodes in the graph, but the number of edges and the manner of those edges.
I think those problems are related. The more features you have, the more difficult it becomes to avoid strange, surprising interactions. It’s like a pharmacist working with a patient who is taking a whole cocktail of prescriptions; it becomes a combinatorial problem to avoid harmful reactions.