I actually like using systemd. I took the time to learn how to use it, and as a desktop Linux user, it's honestly quite nice.

But the one thing that still really pisses me off about systemd-the-project was the fact that they ate udev-the-project. In my view, that decision was unnecessary and was done for purely anti-competitive reasons.

If you're not familiar, udev is basically the user-facing device manager for Linux. It's what allows you to easily configure rules and permissions for all the USB devices that you plug into your computer. These devices still need kernel drivers, but udev is how you tell your system "Please let user ohazi use this device, and also give it a convenient name like /dev/someDevice0"

By devouring the udev project, the systemd maintainers have guaranteed that dealing with USB devices on non-systemd systems was going to be a giant pain.

Then came the forks -- Gentoo maintains eudev, which is a systemd-free fork of udev. But really, this shouldn't be a fork. Udev should be independent and available on all systems. If systemd wants to do something special with devices, they should use udev APIs like everybody else. In my view, this is what finally allowed systemd to win the init war, despite all the protest. Ideological arguments about how best to start daemons is one thing, but you simply can't use a modern system without a sane approach to USB.

Edit: It seems that some of my concerns were overblown, e.g. you apparently can run udev without systemd as pid 1. Udev has a build-time dependency on systemd, but not a run-time dependency. The eudev fork removes that build-time dependency, and (maybe?) also papers over some other inconsistencies. I'm pleased that this is the case. I'm annoyed that the discourse around systemd/udev has been muddy enough to lead me to incorrect conclusions (and I'm aware that the original version of this comment likely added fuel to this particular fire). Oh well... live and learn.

The person who started udev (i.e. me), and the person who did the majority of the work on udev to make it into the proper solution for everyone (i.e. Kay Sievers), both agreed that it made more sense to move it into the systemd codebase in order for lots of duplicated code and functionality to be removed.

By doing this, we have made the maintenance and support for this core userspace tool much easier for everyone who was involved in working on it.

If the developers involved in a project do not do what you want with that project, feel free to fork the project as that's the beauty of open source! And hey, that's exactly what the eudev developers did, go use their fork if you want to, no one is forcing you to use the version from systemd, just like no one is forcing you to use any other open source program. It's your choice to do so or not :)

Why did you chose udev using libsystemd and not systemd using libudev (or made a different shared library, I'm not familiar with what happened)? Seems like this would have been a more widely accepted change and achieved the same thing.

I'm not understanding the question here at all, sorry, can you please rephrase it?

And if you have questions about how this happened, look at the source for when it was merged, it's all there for everyone to review :)

Also, udevd is a program, not a library.

As I understand there was some shared code between the two projects and udev was refactored into using systemd's code. Why not make systemd re-use udev code instead? Aka either expand udev's library or make a library out of the shared code that's used by both systemd and udev.

That way udev stays a separate project and you don't have to bring in a big library to use it. Systemd needs udev both ways so it changes nothing for it apart perhaps a bit of maintenance for a separate library for the shared code.

I don't want to put words in Greg's mouth, but if you check the code for udevd at this moment you can see a lot of use of utility headers that are not really appropriate for a libudev: https://github.com/systemd/systemd/blob/c54cfef3968613f9e86e...

Expanding udev's library would probably end up a decent size library of other functionality, I don't think there is any good way around needing this. This is not code that could be deleted from systemd if it was separated. The way eudev has handled it is by doing what you describe and copying those shared functions out from systemd: https://github.com/gentoo/eudev/tree/34b2037d379e33f1cf79a34...

The result of that being there are two copies of the same shared code floating around, which is what they were trying to avoid.

>but if you check the code for udevd at this moment you can see a lot of use of utility headers that are not really appropriate for a libudev:

That's fair.

>This is not code that could be deleted from systemd if it was separated.

Why not? You could make a library out of those utility functions and use it from both systemd and udev (without having to include most of libsystemd which doesn't seem useful for udev).

I don't have a quote handy, but systemd maintainers have previously said they are not interested in also maintaining a giant public API of utility functions, on top of everything else.

What particularly grated was that they managed to introduce bugs into udev and then say they didn't have the "expertise or understanding" to fix them: https://github.com/systemd/systemd/issues/7157#issuecomment-...

(That bug was causing udev to give multiple disks the same name in /dev/disk/by-path, which I think is pretty much core udev functionality, given that udev was intended to supersede devfs.)

I'm just curious, how would you suggest they debug and maintain support for that hardware, without even owning that particular device? It seems like anything they do would just be a wild guess as to whether it actually works or not. In my experience with this specialized datacenter hardware, if people want any of this stuff fixed (in userspace or in the kernel) it usually falls on the hardware vendor to hire some developers to fix it and upstream it, or at the very least donate some hardware to the upstream project so they can test properly.

If they felt they needed information from, say, the Linux scsi maintainers then they could have gone out and asked for it. If they thought that in order for udev to do a proper job with these devices they needed the sg3-utils maintainers to write something new, they could have gone and asked for that. As it was they just stuck a "needs-new-home" label on the bug and ignored it.

I think useful action would have been more likely if udev had still been maintained by a team who believed they'd taken on the responsibility to make /dev/disk/by-path work, rather than maintained as a small part of a project whose maintainers are mostly interested in other things.

(Also, IIRC from what I had to do to work around it, they could have fixed that bug by reverting the change that introduced it. It wasn't a case of existing udev code not working with new hardware; it was caused by someone making changes without understanding the consequences.)

That's what I mean though, all of that seems to be just wild guessing as to what will actually work and sending random emails out to random people who may or may not be able to fix it (Disclaimer: I don't know anything about this specific bug or the specific hardware). Wouldn't someone who actually owns the device and knows who to contact on the kernel side be a better person to lead the effort on that?

This was a bug in udev.

The right people to lead the effort on fixing bugs in udev are the udev maintainers.

If people did not wish to have the responsibility for leading the effort to fix bugs in udev, they had the option of not taking over the maintenance of udev.

I agree with ohazi, above, that the world would be a better place if the systemd people had availed themselves of that option.

(To be fair, it looks like this issue was fixed a few months later by someone who knew what they were doing. So the main lesson here may be something more like "@poettering should not be triaging udev bugs".)

For what it's worth, I didn't mean that the systemd people shouldn't triage the bug.

I meant that I suspect @poettering was mistaken when he said the current maintainers didn't collectively have "the expertise and understanding to maintain this properly."

As far as I can make out, there was nothing particularly hardware-specific about the bug. The bug was that udev was assuming that there could be no more than one disk per SATA host node, which turns out not to be the case.

`udevadm info` output would have been enough to see what was going on, which I'm sure the reporter would have happily supplied.

I think their point is: with smaller project scope, this bug wouldn’t have existed in the first place.

And they will continue to be bitten by problems specific to large-scope projects. Wanting something, no matter how much, does not let them escape engineering reality.

Apparently not. If they had not signed up for it, in this case they would not have ended up triaging a bug they did not have the skills or hardware to tackle.

No, the alternative is that, with no clear owner of the bug, the ownership problem gets dealt with early, rather than having the bug go stale in a can't-fix-won't-fix state for months because the wrong people claimed ownership of the subsystem.

This wasn't a hardware bug and it didn't materialize only when some particular hardware was plugged in. Were they just supposed to open a phone book, pick a random hard drive manufacturer, and say "hey this is your problem now"?

systems/udev was spearheaded by employees getting paid to write it, not enthusiastic gnuers doing it for love.

My biggest gripe is that the logs are in a binary format.

You use a tool like cat, vim , emacs, or tail to see logs. You can't physically read the file off the disk, how is this different than using journalctl ?

# journalctl -u ssh

I think the issue is not that it's binary, but that it's a custom half-assed format. It's actually pretty slow because there is not enough indexing, this is why `systemctl status` can takes seconds to show the last 10 log lines[1]. Even worse, systemd can detect corruption but can't repair the logs[2], in practice a power failure can mean months of logs thrown away. Thankfully it can skip over corruption, so logging continues to work.

I would have no problem if systemd used a database like sqlite, lmdb or anything else with some valid tooling, indexing, caching and proper recovery solutions.

[1]: https://github.com/systemd/systemd/issues/2460

[2]: https://unix.stackexchange.com/questions/86206

I looked into this a while ago -- a traditional transactional CRUD database is totally unnecessary here, these are not files that you want to be editable by random tools. The journald log files have the important property that they are append-only by a single-writer. I agree there are some outstanding issues but those issues should just be fixed, instead of throwing it all out and moving to an even more complicated database layer that brings with it all its own additional problems. I think it would be perfectly feasible for someone to improve the indexing to show the latest 10 lines, or to write a fsck-type tool to repair damaged logs.

> instead of throwing it all out and moving to an even more complicated database layer that brings with it all its own additional problems.

Would SQLite be any more complicated than what they have now? Or OpenLDAP's Lightning Memory-Mapped Database (LMDB)? OpenLDAP/Symas' Howard Chu on the latter:

* https://www.linkedin.com/pulse/20140924071300-170035-why-you...

Conventional logs are files, the fundamental unifying abstraction of Unix, and so you can read them with any tool, even a tool that was written before your logging system and that your logging system's author didn't know about. Having to take them just as piped input is a much more limited interface that doesn't allow you to do all the things you can do with a real file.

> so you can read them with any tool

what tool can't you read them with? Why is the fs the fundamental unifying abstraction, as opposed to pipes?

> doesn't allow you to do all the things you can do with a real file

like what?

You can't do random access through a pipe, so you can't binary search for it, or if you have some kind of probabilistic skip-sampling tool that won't work. Pipes don't have names, so anything that expects to work with them won't work. You can't re-read the same file. I don't know all the things you might want to do, but that's kind of my point - most things work with and expect files, and so if you want to be able to count on being able to use other unknown tools, a file is what you need.

Pipes are files. From the point of view of the consuming process, you're just reading from a file descriptor. Files are more fundamental than pipes.

If lmm shared this opinion, then there should be no problem, as systemd logs can be piped.

All pipes are files. Not all files are pipes.

So, back to the original question: what's the problem here? what is disallowed?

Also: If all pipes are files, but not all files are pipes, it would seems to me that files are more restrictive. That, and the extra steps you need to take to avoid needless IO.

I recommend https://www.amazon.co.uk/dp/013937681X/ref=cm_sw_r_cp_apa_gl... as an introduction to the fundamental concepts of the Unix filesystem. Certainly a better education than I could give here.

The original post was:

  that doesn't allow you to do all the things you can do with a real file

I'm not interested in what the differences between a pipe and a file are (I know) - I'm interested in why OP thinks they are relevant in this specific context/case; particularly in context of my statement:

  seems to me that files are more restrictive

Is, for example, a sticky bit relevant to systemd log files?

But allows you to do things like properly filter based on multiple properties, and you can still use it to output text logs as well.

There are plenty of unix tools for filtering files based on all manner of things. That's the unix way of doing things, and for many users the ability to mix and match unrelated tools is where a lot of the value of linux-like systems comes from.

I don’t find grepping text logs for a service file name with many false positives better in any way than just filtering on an actual column of a quasi-db.

Also, you can pipe the output of journalctl to do whatever you want with it with those unix tools.

At the risk of uttering a "you're holding it wrong" defense, it feels to me the issue isn't with logging format, but the stubbornness of *nix admins used to working with text. For better or worse, software moves in the direction of things large amounts of developers like the most, whether it makes sense or not.

Amarok 2 moved in the direction that the developers wanted, but the users hated it. Proves your point though!

the systemd stars-to-issues ratio is worrying to say the least

What do you imagine that ratio shows? Different projects use issues differently, so that number isn't super comparable...

And I've also noticed that low-level systems projects generally attract relatively few stars (i.e. a javascript library or go library will have more stars than a well used c library or systemd). My theory on that is that the people involved in projects like systemd, gnu, etc, see github as mostly just a git repository host, and don't bother with stars etc.

I guess from my perspective, I put zero weight into the stars on a project, and issue count only matters in the context of how the project deals with issues. What do you see as "worrying" in there? Do you see more value in those numbers than me?

Github stars are just bookmarks. They're not an expression that someone loves the project, or are using it, or even very interested in it - they're just a way to bookmark something that piqued your interest, so that you can find it later.

This tells you why "low-level systems projects" attract less stars. It's because well-known, long-lasting projects attract less stars. There's a little point of starring a repo of a project that you recognize and use. You remember its name already. Meanwhile, a random JS library that looks like it could come in handy in the future? These things come and go, and you'll forget its name 5 minutes from now anyway, so you star it.

Well, to you. I use GitHub stars as a way to mark projects that I use or particularly appreciate.

I think it generally shows how active, responsive and sufficiently staffed the maintainers are. should projects be on github if the issues aren't used as intended? the worry comes from the number of systems in production relying on systemd

> should projects be on github if the issues aren't used as intended?

What do you mean by "not used as intended"? systemd has a mailing list too, but they also respond to github issues. Does having multiple different forms of interaction with your community mean you're not using issues as intended?

Some projects use github issues as a sorta backlog, or in conjunction with milestones and github projects to track future work. Some don't. Are projects that don't do that not using issues as intended?

As far as I know, github gives no general guidance on how to use github issues, and it's up to each project to setup issue templates and/or contributing.md files to explain that project's expectations.

> generally shows how active, responsive and sufficiently staffed the maintainers are

Okay, so if a project has a bunch of maintainers and is so well staffed they close all issues immediately, that means they have 0 issues, so that's good, right?

Except many well-maintained projects on github have hundreds to thousands of open issues, like nodejs with >1k, rust-lang with 7k, etc.

From where I sit, how well-maintained a project is has almost nothing to do with how many issues it has. Issues are a function of number of users (which is much different from level of maintenance), and maintainer policy on whether they close issues without enough information, close stale issues automatically, use issues to track backlog work, etc.

So in your metric, all those projects that use a bot to close issues after 30 days as "stale" are using Github better?

it's not a metric, just an indicator

There is for example nixpkgs, which uses issues very differently to a conventional repo, and you can’t really argue whether the project is over/under stuffed based on that, since it packages almost everything. Some of the issues may be upstream bugs, etc.

More stars wouldn't decrease the number of issues though.

The criteria for a person to star a repo likely varies person to person. Therefore I’m not sure it’s a useful metric of anything in particular? Some crude popularity metric maybe? I don’t star repos myself. Feels more like a social networking feature.

it would decrease the ratio

The number of issues has some relevance. The number of stars doesn't.

not following

For what it's worth, I also don't think comparing issues between different projects works.

If you have project A, run by a company, which uses an internal Jira instance to track backlogs and developer's selected tasks, but uses github issues to track all user's reported bugs, well, that will have fewer issues than project B, which uses github issues to track backlogs and sprints in addition to bugs.

There's a lot of variability in this. The kubernetes project (~2k issues) has a bot that closes issues after 90 days by default (~9k closed by the bot so far), and without that bot, they'd have way more.

> A large number of open issues means the maintainers can't keep up.

That's not necessarily true. Github doesn't prescribe that every issue is immediately actionable. Perhaps the issue was tagged as "feature-request / help-wanted". Perhaps it is marked as "awaiting additional reporter info" (like logs, or OS), and there's nothing the maintainers can do until the reporter gets them more info.

Whether every issue gets triaged tells you something about whether maintainers are keeping up, but issues that are triaged aren't then always closed, so the number of open issues doesn't tell you much about how many issues are triaged and how quickly.

> By devouring the udev project...

That's the main issue (not udev specifically but the overall philosophy of trying to make systemd mandatory for as many things as possible).

Any distro adopting systemd is basically forever giving up any hope of moving away from systemd in the future.

I'm still on Debian (Debian user since "Buzz") but I already tried Devuan (Debian fork without systemd) once and may switch to it at some point.

I simply don't like the overall mindset of those pushing systemd.

I had a hard time trying Devuan because all the package mirrors are really far away from me unfortunately, but I don't have any issues at all just removing it from whenever I am working with debian servers. It all used to work through some shell script called systemd-shim, I'm not entirely sure now though, maybe someone else has an idea off hand

https://sysdfree.wordpress.com/2020/07/02/319/

>that decision [...] was done for purely anti-competitive reasons.

This is not correct, I don't know where you got this idea. The systemd-udevd daemon still runs without systemd. The code has moved into the systemd repository, and it has a build dependency on libsystemd, but otherwise it has no runtime dependency; your distro should be able to package it separately if it wants.

Can you point to a distro that does this vs. using eudev? I mean, you might be right, but I haven't found any, and I think the reason for this is that there are subtler inter-dependencies that require additional workarounds:

> > Also, AFAIR the connection between systemd and udev doesn't really go deeper than the fact that they're sharing the same upstream tarball. It is still possible to build and use udev without systemd

> But that's not really the case operationally, and it's why eudev was forked away from it for Gentoo.

https://lwn.net/Articles/778547/

The udevd package in gentoo does exactly this, which is supported as an alternative to eudev: https://packages.gentoo.org/packages/sys-fs/udev

I believe the reason for eudev is to avoid some of those other unwanted changes from upstream udev that broke udev scripts. Which to me is a fine technical justification, but not really related to another udev implementation having a build dependency on libsystemd. The udevd package seems maintained so I really don't understand what operational issues that comment is getting at.

Interesting. I didn't realize this was a supported option, but the Gentoo wiki also seems to agree with you [1], so I may need to walk back some of my frustration around this.

I still have concerns that having a shared codebase makes it easier for interdependencies to materialize later, but if they've managed to keep it as decoupled as is claimed here, then I'm pleasantly surprised, and also annoyed that the communication around this issue has been confusing enough to lead me to an apparently incorrect conclusion.

[1] https://wiki.gentoo.org/wiki/Gentoo_Without_systemd#The_udev...

Most of the systemd components actually work without systemd as the init. I've used udev and nspawn without systemd-init.

As long as you don't have a problem with libsystemd existing, most things work fine without systemd-init. Mainly systemd-journal and systemd-oomd both require systemd-init.

The people that makes distro without systemd have users that scan the filesystem and get crazy if they found the name systemd in the filesystem. I can say the same for the devs also. So we have hardcore-hate distros that "liberates" packages and the dependency of the unused library. Better give that time to make the distro better imho

In practice I think systemd-udevd probably still runs without systemd, but as I understand it that's not an officially supported solution and the systemd developers have said they reserve the right to make changes that break it at any time. (The main reason why they haven't yet is most likely that their attempts to integrate dbus into the kernel and systemd more tightly were rejected by the kernel devs; if I remember rightly that was expected to be the point at which you could not use it without systemd.)

Using the kernel interfaces that udev relies on without using udev itself is also unsupported and the developers consider it within their rights to break those as well.

It has been supported for at least the last 9 years now, I don't see why they would have any reason to break it.

>The main reason why they haven't yet is most likely that their attempts to integrate dbus into the kernel and systemd more tightly were rejected by the kernel devs; if I remember rightly that was expected to be the point at which you could not use it without systemd.

That seems quite dubious, udev has never depended on dbus, and there really would be no reason for it to ever do that.

>Using the kernel interfaces that udev relies on without using udev itself is also unsupported and the developers consider it within their rights to break those as well.

This I know is true, the netlink stuff varies really wildly from driver to driver. The alternative would be to put udevd and the other userspace bits into the kernel, which I really doubt anyone wants to do.

I have a feeling if this was as simple as you say, distro maintainers would be doing it.

They do. The few distributions that still support non-systemd init systems do split out udev: https://packages.debian.org/sid/udev

Which is not that uncommon, FWIW.

binutils and gdb were also moved into a common repository and gcc merged some repositories as well.

IT's not my area of expertise, but the eudev page does speak of restoring some functionality that it lacks without systemd running alongside it, as well as removing glibc and GCC specific dependencies.

There is probably a reason why even many binary systems adopted eudev instead. — the maintainers often justify this with “problems” with udev but never go into much detail about what that is but the language does keep suggesting that it restored some functionality that is missing if it run sans systemd.

> but the language does keep suggesting that it restored some functionality that is missing if it run sans systemd.

People often misunderstand what libsystemd does. See the many reactions here, but also e.g. the response by Devuan around libsystemd.

Too often people assume things around systemd, or too heavily rely on incorrect information found via Google.

That things are being done this way by a few projects to me isn't enough. Nowadays people did not thoroughly investigate things before they take a decision. See e.g. the various responses in this thread.

>and it has a build dependency on libsystemd

That does seem kind of unfortunate for anyone that wants to bootstrap a systemd-less distibution.

Libsystemd is just a shared library with some common code used by various systemd daemons (an epoll-based event loop, the dbus protocol implementation, uuid generation, parsers for the binary logs, etc). I can understand why people don't want to run systemd as pid1, but having some other external daemon depend on this library doesn't make it any harder to bootstrap that type of distribution.

That doesn't really make any sense, libsystemd is just a library with some fairly generic functionality.

I cannot shake off the feeling that the core developers standing behind systemd, pulseaudio, Gnome desktop spend a lot of time using macOS, and like it better than Linux.

This would explain why they imitate or copy many of its features (and misfeatures), and seemingly tend to replace the Unix ways with entirely different approaches. Effectively forcing things on users, as it was with pulseaudio and systemd, is also a very macOS thing to do.

I'm not opposed to replacing Unix with different approaches, as long as they keep playing on its key strengths, and do not remove modularity and composability. That is, Plan 9 is fine with me. Much of what Red Hat does, sadly, no.

> [Gnome developers imitating/coping MacOS features]

I've heard this a lot, but having used Gnome briefly and MacOS fairly extensively, I just don't see it. Also, have you seen KDE's Dolphin? If ever there was a Finder clone, it's Dolphin. With regards to copying MacOS, Dolphin devs have wiped the floor clean with Gnome.

Besides Dolphin specifically, KDE in general is flexible enough that making it look and feel like MacOS is pretty easy. Gnome though? It's not particularly MacOS-y by default, and Gnome devs have naked contempt for customization, so good luck if you want to try anyway.

However, I share your suspicion that most Gnome developers do not actually use Gnome as their daily driver.

Yes, that naked contempt to customization is what I find more macOS-y than any exact copy of the visuals. KDE in this regard is pretty safe: you can bend and shape it to taste pretty thoroughly.

Interestingly, classic MacOS, at the times of 6.5 or 7, was somehow more customizable, and seemed to have more power-user features. Just like, well, Gnome back in the day %)

So, my daily driver is Xfce.

Interesting that you found Dolphin to be a Finder clone. It may well be, I just didn't see it besides some superficial UI similarities.

I think Dolphin is one of the very best Linux apps out there while I found Finder to be very clunky, often so unintuitive and useless that I resorted to command line instead. To be honest, it would probably have been fine if I took the time to learn the Mac idiosyncrasies like missing cut/paste, but Finder was one of my main gripes when I had to use Mac for work.

To be frank I find both Finder and Dolphin to be generally unsatisfactory. Finder is worse though, it's riddled with bugs and weird shit. I could never get Finder to show preview thumbnails of symlinks pointed to files on network mounts (while the Dock's folder view could always thumbnail those same symlinks just fine.) Or, when you mount a network drive, Finder can take dozens of seconds to recognize the mount has succeeded, while in a terminal next to finder you can clearly see it succeeded instantly. In Finder, thumbnailing sometimes breaks and completely stops working, with no indication of why. `qlmanage -r` usually fixes it, but what is the point of using macos if I have to waste my time fixing janky shit in a terminal? It's worse than the modern linux desktop experience.

Dolphin just has weird omissions; for instance try to sort a directory of symlinks by the time those symlinks were created. As far as I can tell, you can't do this because Dolphin follows the links and sorts by the dates of targets (contrast with `ls -ltr` and Finder, which sort by the dates of the links.)

Gnome is one of the most customizable DEs around. You can customize almost every aspect of it via the extension mechanism. I'm even running a full tiling WM right now that is coded as a Gnome extension (PaperWM).

The customization in the desktop of GNOME itself is still there, but it has mostly moved to shell extensions. If you're willing to deal with that, you can change quite a bit of functionality.

The apps are a different story, those tend not to go for a lot of customization and will usually just focus on optimizing for one or two use cases. If you have other use cases, the suggestion there would probably be to make a separate app.

GNOME would be fine and was fine up until the attempt at tablet integration happened, and it's been total chaos since then.

The problem is they hopped aboard the same bandwagon Microsoft did with Windows 8, where every started throwing tablet/touch style UI onto desktop interfaces.

For the life of me, I don't understand why there wasn't a happy middle ground of "or we'll render the main menu as a regular menu bar".

But at this point it's not a GNOME criticism so much as an entire UI/UX industry criticism.

>The hamburger menus were done to save space and reduce clutter, not out of any desire to be a tablet app.

"Save space"? That's a laugh. Reducing clutter is ill defined and probably subjective, but saving space? Saving it for what? The default window decorations, widgets, etc are bloated, not slim and compact (as they would be, if they were saving space.) Everything has huge margins, big buttons, and empty voids. "Saving space" is not a principle Gnome cares about.

Do you know what huge margins and big buttons are actually good for? Touchscreen interfaces designed for fat fingering. That's what Gnome cares about.

Again, what you are saying is not correct. You may be comparing their designs to other apps developed for other desktops using other toolkits, but that's not what is influencing GNOME. The space is saved compared to previous designs that were used by GNOME. Please consider comparing an old style GTK app with a new style GTK app:

Inkscape, old style: https://media.inkscape.org/media/resources/file/inkscape-0.4...

GNOME builder, new style: https://gitlab.gnome.org/GNOME/gnome-builder/raw/master/doc/...

The old style has four bars at the top (titlebar, menubar, toolbar, secondary toolbar). The new style merges them all into one bar and moves the menus into the hamburger menu. The margins may be bigger in the new style but ultimately, space is saved and the UI chrome is reduced. Also, GNOME Builder is not an application designed totally around touch screens -- it's a text editor, for typing into.

I sympathize if you don't like this style of app (and I encourage you to use different apps if you were used to having a lot of menus and toolbars, and you don't like the new style of GNOME apps), but what you are saying about the design goals and motivations does not match what the developers are actually doing.

If plan 9 is what you want, I would encourage you to use 9front. I can't see why you would use Linux if you didn't like Red Hat, they have been a key contributor in Linux land for a long time, employing many long-time kernel developers.

I'm not sure if 9front can be a daily driver yet; say, it does not seem to be able to run Emacs. Would be nice though.

Not that I don't like Red Hat. They are the poster child of a successful open-source company, and did and keep doing a ton of great things. I only don't like certain directions of development of certain userspace things which happen to be done under their corporate umbrella. These things, systemd in particular, were quite noticeable on the Linux landscape as of recently, both in the tiny desktop niche, and the huge server sector. Nobody's perfect, you know.

I have to say, that seems like a strange requirement: I have never really considered Emacs (or Lisp) to be particularly Unix-like at all! I think the plan 9 people would probably tell you to ditch Emacs and use acme (Me personally, I say use whatever you want).

Emacs inside is very unixy, under a particular angle. Everything is a list, a ton of small composable functions doing one thing well, one common language that unifies all key interfaces, and the general lack of a predefined final construction, but rather a bucket of Lego blocks around a kernel which does the heavy lifting.

Acme us interesting, but pretty different; its automation is apparently written in the shell language using normal OS commands. AFAICT the Plan 9 shell lacks structured data types comparable to Lisp's: it has lists, but not nested lists, so such constructs are a bit less robust. Also, AFAICT, Acme actively wants mouse operations; I hope reasonable keyboard equivalents exist.

I'll explore more of it in my copious free time (sigh).

Having a lot of small composable functions and a lot of singly linked lists describes most functional programming languages though. I think that's only one part of the traditional Unix design. Though to me it was always seemed like the inspiration goes the opposite way -- it seems like Unix was in some ways designed as a "Lisp-like," where the shell works as a somewhat restricted version of functional programming that only operates on one data type.

I don't blame them. Linux desktop is still a mess.

systemd is great though.

If you like (the mess) that is OSX, use OSX.

No.

Wasn't it because Kay Sievers was a major developer on both udev and systemd?

I can image someone thinking then "I'm implementing everything twice, both for udev and systemd. I'll just merge them"

According to Lennart that's correct, the reason it was done was to reduce the amount of duplicate code: https://lwn.net/Articles/490441/

I realize that I don't really have any right to complain about this. I don't contribute to either project, and I think that maintainers should feel free to do whatever they think is best for their projects, including things that help reduce burdensome things that they find annoying.

However, as a heavy user of both projects, I see what those decisions have led to in practice, and have opinions about what this might mean for the community in the medium/long term. The reality on the ground today is that you either run systemd+udev, or you run OpenRC/something else + eudev. And having observed other projects that went down the "we forked, but merge regularly to try and keep the forks in sync" rabbit hole (e.g. ffmpeg and libav), this almost always ends badly.

The goal may be to keep the two separate projects compatible, but inevitably environment differences, bugs, refactors, etc. cause the two projects to diverge. This sucks for everybody, but it sucks a lot more for people using the less popular fork, and this can eventually kill the fork. I worry about eudev long-term.

So that really seems to be the core of the problem to me: nobody really wants to seriously maintain udev as an individual project. It's not a fun or glamorous project to work on, it's boring, nobody will notice it unless it breaks, and nobody really seems to care that it depends on libsystemd. For most people this is just an implementation detail. For an embedded distro I can't see why you would use udevd or eudev, you probably want a much simpler device manager without the giant hwdb.

> I realize that I don't really have any right to complain about this. I don't contribute to either project

This is a strange mindset. Do Star Wars fans not have a right to complain about bad movies just because they weren't part of the cast or crew?

That's what code libraries were intended for.

Not sure why you're saying that -- moving it to a library is basically what happened. The libudev code got moved to libsystemd, and now libudev is a wrapper around that.

> Not sure why you're saying that -- moving it to a library is basically what happened. The libudev code got moved to libsystemd, and now libudev is a wrapper around that.

Oh, so now one has to include "libsystemd" if one wants to use "udev" - is that what you're writing? Hence the original comment:

> By devouring the udev project, the systemd maintainers have guaranteed that dealing with USB devices on non-systemd systems was going to be a giant pain.

I don't understand, is this not what you were asking for initially? libsystemd.so is just a shared library, it's not any more of a giant pain than any other library.

Why not just include libudev, in systemd instead?

Because that would not reduce code duplication. I don't know if you have ever worked on any of the low-level Linux libraries written in C, but the amount of unnecessary code duplication across them is awful. The standard library features provided by glibc are extremely inadequate for modern applications, every non-trivial C project I've seen starts to include their own private implementations of various C++ things like hashmaps, binary trees, dynamically sized strings, unicode support, async event loops, etc.

For utilities that have to interact with low level kernel APIs it's even worse, every library seems to have to reimplement their own parsing of various other random things like netlink, or in the various pseudo filesystems like sysfs, procfs, cgroupfs, etc etc, the situation is really way out of hand. I don't know how to solve this in a reasonable manner beyond what systemd is already doing. Yes people will complain that they have a systemd dependency now but what else can you do? This is the exact reason the BSDs update the kernel, libc, init and core utils in tandem and consider a lot of the kernel API to be private, Linux was just slow to catch on in that regard.

> Because that would not reduce code duplication.

I don't want to call you out for making a disingenuous argument, but it was either subsumed by systemd for code duplication reasons, or it wasn't?!

Duplicate code, could be refactored out to a shared library, that could then be incorporated in both udev, and systemd. That would mean, anyone looking to incorporate udev into a system, could do so without depending on libsystemd.

Instead, it would seem udev code, has been subsumed by libsystemd (in your own words) - which would appear to the sceptical eye, as a power play on the part of red hat - to force other distributions into using libsystemd, which would logically end with them also using systemd itself.

> I don't know if you have ever worked on any of the low-level Linux libraries written in C [...]

Yes, I have.

The issue is not with the actual udev device event logic itself, but with all the other bits I talked about. Those are the things that would need to be duplicated.

>to force other distributions into using libsystemd, which would logically end with them also using systemd itself.

As I said elsewhere, libsystemd is just a library with some generic functions provided for convenience. This is like saying that installing python libraries on your system logically means that the PSF is trying to take over your system and forcibly rewrite everything in python, it doesn't make any sense.

Does it really make a difference whether it's called libsystemd or something else? If you're talking about the other functionality that's private and unstable, it's not included in a separate library for exactly that reason: it's considered private and unstable (Also I'm not sure if you're joking with that name but I really doubt there would ever be something like this literally called "libredhat," that makes about as much sense as putting a random b-tree implementation in a library called libubuntu or a libgentoo).

>Does it really make a difference whether it's called libsystemd or something else?

Probably not. But when I think of something named "libsystemd" I think of a library to interact with systemd, not a collection of random hashmap and B-tree implementations.

> If you're talking about the other functionality that's private and unstable

Why would I talk about factoring out private and unstable code into a shared library.

But if udev is depending on that private and unstable code I do have a lot more sympathy for the packagers who are wary of the merger. It's kind of disengenuois to claim that they're totally separate projects, and can reliably be deployed separately when they both depend on some private special sauce. Even if the sauce is open sourced.

> Also I'm not sure if you're joking with that name

Half a joke and half trying to avoid the "I hate the name" problem.

You have clarified a little bit of what that library is actually doing. And why it seems to make sense to have udev pick up the dependency. Thank you.

Sorry I wasn't clear -- most of the stuff I was talking about (hashmaps, b-trees, parsers, low-level utility functions, etc) is the private and unstable stuff, in the sense that it's a little too specialized towards systemd's style of C coding to warrant making it a public API, but it's useful enough to be shared between all the systemd components including udev. That's all the "private special sauce" is, it really doesn't make it any harder to deploy separately. They're not totally separate projects but there also is nothing really in common between them besides the build time dependency. Does that make sense?

Yeah it does.

It seems to be a symptom of the kind of social and political problems that appears to be the root of quite a lot of systemd hate.

Ultimately those who dislike it are probably better of maintaining some kind of alternative than writing rants on forums though.

I really appreciate your time helping me understand this particular issue.

You're assuming libsystemd does things which it doesn't. Saying there should've been a libudev says enough. Too much assumption the lib does loads of things around the init system part of systemd. The unneeded dislike of the library is exactly why I used to make fun of the Devuan project. Loads of decisions that pretend to be made on a technical basis, but are actually mostly emotional and "gut"-feeling.

I think you're confusing me with another poster in this thread.

I'm mildy "not a fan of systemd" in that my minimalist sense of what a "good" system is is bothered by it and it doesn't pass my "gut-feeling" but I run way more systems with systemd than without so...

The way this whole thing is divisive fascinates me though. Systemd seems to work, and work fairly well, but also attract phenomenally loud detractors. I can't think of any other piece of software that does that. Not even PHP triggers as much back and forth. This whole udev and libsystemd thing is one of the only arguments that seems technical to me.

> Duplicate code, could be refactored out to a shared library, that could then be incorporated in both udev, and systemd.

That's basically what has been done. The library was named libsystemd.

You seem to have a problem with the name "libsystemd".

Feel free to rename it libNotSystemdButAGeneralSharedLib and link that.. it is named systemd, not systemd-the-init-system itself.

I'd expect the opposite, if someone said they moved something to a library.

I love Gentoo and really like they chose to make eudev and a elogind so I can continue to use OpenRC, which has been working great for me since forever.

Don't get me wrong -- I too am extremely grateful to the Gentoo devs for maintaining eudev. It's also the obvious choice for highly memory constrained embedded Linux systems that can't fit systemd. I just think it's absolutely bonkers that it has to be a fork... eudev should really just be udev.

Credit where it's due, the GNU Guix folks made elogind and Gentoo picked it up.

> But the one thing that still really pisses me off about systemd-the-project was the fact that they ate udev-the-project. In my view, that decision was unnecessary and was done for purely anti-competitive reasons.

The developers of udev at the time agreed to merge to the project into systemd, as it made a lot of sense to them.

There was no "devouring", no "anti-competitive" bullshit.

Given that at the time of the merge the developers of udev were the systemd developers, "agreed to merge" seems like an odd presentation of that particular piece of history.

Why? You're saying it was pushed. But actually the maintainers agreed. Further, the maintainers overlapped. Instead of acknowledging that it wasn't pushed because the maintainers overlapped you make up "anti-competitive" behaviour.

I work for a company where every year there is a mandatory refresher on anti competition laws / behaviour. The suggestion that this is done for malicious purposes is highly offensive, yet so easily stated. This without actually going into a lengthy and thorough explanation. I think you're failing to understand how offensive you're being towards these developers. So easily using very harsh words, so easily dismissing what anyone else says, plus what the developers said. Yet not actually proving any good proof for your statements.

> Why?

Because saying that they were separate groups implies a level of propriety to the decision that doesn't hold when the groups "overlap".

> You're saying it was pushed.

Am I? Where?

> But actually the maintainers agreed.

At the risk of repeating myself, this implies a level of propriety to the decision that doesn't hold when the maintainers are the same in each group.

> I work for a company where every year there is a mandatory refresher on anti competition laws / behaviour.

Didn't take, did it? Just for giggles, see if you can identify the common factor in these classes of anticompetitive behaviour:

  - Cartel price-fixing
  - Refusal to deal
  - Conspiracy to monopolise
  - Dividing territories
  - Regulatory capture

> I think you're failing to understand how offensive you're being towards these developers.

I don't. I also don't think they need you to defend them.

> But the one thing that still really pisses me off about systemd-the-project was the fact that they ate udev-the-project. In my view, that decision was unnecessary and was done for purely anti-competitive reasons.

Such are all the common criticisms on systemd.

It was never a technical objection; it was a political objection to the fact that systemd unnecessarily coupled things that could and should be separated, but were coupled not for technical reasons, but for anticompetitive ones, and this goes beyond systemd.

Many Red Hat projects have a way of finding themselves to be dependent on one another in ways that are often hard to justify from a technical standpoint, and often even use unstable, undocumented interfaces that they make available for one another.

Unnecessary coupling of components is a technical objection. It's poor engineering, and that criticism was framed by systemd supporters as political.

By that argument, all criticism of software is technical.

The argument is that it's not done due to incompetence, but commercial strategies.

How about the technical standpoint of having loose coupling everywhere is bug prone and hard to maintain? Will you create a public API that you will break at each update? That’s not too different to what is the state now. Should you carry around 10 years old baggage APIs or what exactly would you propose?

systemd still hasn’t properly replaced ISC dhclient so I dumped the entire systemd and went both Gentoo and Denuvan (systemd-free Debian).

Are you being sarcastic? The DHCP bits are entirely optional. You can either use it, or not use it. The code is useful in some cases, not useful in others.

Saying that some optional code has issues as a reason to not just use something else, but as a reason to not use the project: not logical.

Try getting Nintendo to do a DHCP with systemd, you can’t. Nor with Juniper DHCP server configured by Comcast and Verizon, you can’t. Hence, ISC dhclient remains.

I have a system where systemd decides, about half the times I boot it, to immediately unmount all of the filesystems listed in fstab literally right after having finished mounted them as part of the regular boot process. All of this with other daemons starting in parallel.

It will fail to unmount most of the filesystems, since they are of course busy, but often it will succeed in unmounting /var, /tmp, /home and others. Then it will continue starting on further services as if nothing happened and even proceed to gdm. But of course without /home I can't even login.

No idea how to even start debugging it.

That's super annoying. I've experienced similarly frustrating boot problems when I had some FS corruption causing /var fsck to fail @ boot.

Despite dropping me to a rescue shell, systemd entered a continuous loop of retrying fscking /var while I was attempting to identify and fsck the problematic filesystem.

Few things are more infuriating than having your system drop you to a rescue shell, while it continues to endlessly change the state of the system - and not just any state, but the very area you're in the rescue shell to investigate and resolve. It just kept making /var busy while I kept attempting to manually fsck it...

I would run systemd-analyze critical-chain and then systemd-analyze blame to see if there is a service that could do this. And then I would look at the detailed boot logs. If nothing can be found, I'd look for some way to increase the log level.

If you are not curious, reinstalling everything could be quite fast too, depending on what you have installed.

The logs are mostly useless and random. There is nothing that consistently appears between the point where it finishes mounting (lots of "Mouting...", then "Reached target Local File Systems") and where it starts unmounting ("Stopped target Local File Systems", then lots of "Unmouting ..."). Sometimes these two lines are less than one second apart. But I haven't found any message which is consistently between these two points that would point to a suspect.

And this is not a problem that happens on every boot, so it doesn't show up on analysis... not to mention that most of the analysis is designed on trying to reduce boottime, which obviously is no help (thought it is rather fast already).

It is rather easy to guess why a service is starting, but not why it decides to stop 'orderly', much less why a .target decides to stop...

Most certainly you can only analyze things on boots where the problem appears, unless logs are persisted across boots.

Indeed systemd-analyze focuses on boot times, though I find it is useful to get a list of services that are run.

If you are adventurous you could probably replace the umount (resp. the systemd-umount) binary by a script that runs umount (resp. systemd-umount), and also prints the process tree (and its PID) in some file so you can get insight on what ran it.

You can check in journal which service startup does the unmounting align with. Maybe generate a bootchart http://manpages.ubuntu.com/manpages/bionic/man1/systemd-boot...

You could also replace umount with a wrapper that logs the parent process tree on execution.

And of course you can open an issue on GitHub.

> And of course you can open an issue on GitHub.

Only do this if you've reproduced what seems to be a bug in one of the latest two releases, otherwise contact your distro.

The systemd github issue tracker is not a support forum, you'll find this stated in the issue submission form.

Fair advice, and they’re probably right to redirect most users that would file a bug with them.

But on a normative level, perhaps a project that doesn’t want to support users shouldn’t have subsumed a huge swath of user land things?

> But on a normative level, perhaps a project that doesn’t want to support users shouldn’t have subsumed a huge swath of user land things?

It's not that the project doesn't want to support users in some absolute sense, it just doesn't want it happening in github issues.

The project provides the systemd-devel mailing list for that, but it's still preferred that users stuck on old versions determined by their distro's release cadence seek support from their distro vendor. It's likely such slow-moving distros are patching their LTS versions anyways, and are best positioned to support what they ship.

If this is a bug in systemd it belongs at systemd no matter which version it is. If systemd doesn't want to do support that is their choice but distros aren't the right place to post systemd bugs. It only belongs there if it is either a bug in the distro itself or a support request for distro specific problems.

If you go filing bugs against old versions of systemd in the github issues, you're likely just going to receive a canned reply along the same lines of what I've already said.

You may not like it, I'm just playing messenger here, to try save some bother on both sides of this subject. Use the mailing list for support questions, or contact your distro vendor for support.

We had that kind of problem because we relied on /dev/disks/by-label

Can you tell us more about this? How did you determine that that was the problem? What was your solution?

I ask because I use /dev/disks/by-label and have never had this problem, and if I ever had to experience it, it would drive me insane.

This is a common error even back in the Sysvinit days. /dev/disks/by-label is not guaranteed to map the name labels to disks on every boot, and it can change with boot timing based on the storage controller behavior. Use /dev/disks/by-uuid instead.

/dev/disks/by-label is a legacy from the old days of systems with static disks wired in a specific, unchanging configuration.

You've confused by-label with the kernel's device nodes (e.g. /dev/sda1). Kernel names are assigned in order. Labels are just like UUIDs and part of the filesystem.

However, unlike UUIDs they don't necessarily exist, are not as likely to be unique (you should only get duplicate UUIDs by cloning filesystems) and are easier to change. This makes them less suitable for mount configuration.

I've solved a similarly sounding issue by using the UUID. Get it with blkid, then use UUID="<uuid goes here>" in fstab instead of LABEL="<label>".

But this was a long time ago, on a much older systemd (like Fedora 17 or 18).

Long time ago now, but we had disks just unmounting and mounting at free will.

As someone said by-label is not stable and could be changed, causing problems. We still use it though but not in customer facing servers where these problems arise. Maybe related to lvm/snapshot etc.

Deactivate graphical boot screens (often just by pressing Esc), increase the log level[0] and try to read the logs somehow. Maybe you can still log in as root. If you cannot log in as root even, but networking works, configure rsyslog to send your logs to another host and read them there.

[0] https://freedesktop.org/wiki/Software/systemd/Debugging/

if you can't login as root, you can try to put bash as init (with init=/bin/bash in the kernel boot command line) to inspect the system. Might be difficult on some systems (with SELinux?)

if you have access to kernel command line to add init= you can also disable selinux there.

I'd rather recommend booting a liveCD. grml, knoppix, whatever floats your boat. An init=/bin/bash shell needs quite a bit of knowledge to get working (mount rw, get rid of selinux, mount the real rootfs), is possibly just busybox and generally a pain. Even more so if you need the net to copy something over.

It isn't impossible, and OK in a pinch, but why make it hard on yourself?

You can actually specify systemd.unit=rescue.target nowadays.

https://ostechnix.com/how-to-boot-into-rescue-mode-or-emerge...

You forgot mounting /proc and /sys :) Anyway, it's the quickest way to recover if all you need is to edit some config file or run passwd.

Have you tried increasing the log-level in /etc/systemd/system.conf? You can also set LogTarget=kmsg if you have issues with the journal. You can also control this with the kernel command line if you prefer https://www.freedesktop.org/software/systemd/man/systemd.htm... You could try a kernel command line like this 'systemd.log_level=debug systemd.log_target=kmsg' for instance

You can also look at systemctl list-dependencies local-fs.target to see if it has any failed dependencies

You can also use systemctl show local-fs.target to sanity-check it to see if there are any local modifications to the target that are breaking it

I'd start by looking at the systemd unit files in /etc/systemd/system. Beware of service wants.

I had straight up errors in a network unit file that caused weird behavior (IP address disappearing). This was a long time ago though. Sometimes moving a unit from one target (graphical) to an earlier (multiuser) solves the issue.

You might also have "defaults" in /usr/lib/systemd/ to compare with.

systemctl and friends are really good tools for troubleshooting but takes a little while to get used to. Duck duck go is your friend.

Finally, some volume technologies like VDO requires a x-systemd.requires statement in fstab.

Systemd generates mount units for the filesystems listed in fstab. Maybe check those with systemctl status? (Like `systemctl status -- -.mount` for my root partition)

Sadly no. The problem is that the entire local-fs.target is stopped 'cleanly' but with no clear reason. i.e.

    $ systemctl status local-fs.target
    Active: inactive (dead) since [...]
    21:12:56 ... systemd[1]: Reached target Local File Systems.
    21:12:57 ... systemd[1]: Stopped target Local File Systems.

Yes, the log just shows it stopping one second after being started, nothing happening inbetween, and no reason given for it being stopped.

Incredibly enough, local-fs.target is a dependency for graphical.target, so the system should not have continued booting. But not only it has continued booting, systemd even thinks that it finished booting all-OK. State is "running" (not "degraded" as it would be if any service/mount failed), with 0 failed services. Even though both graphical.target and local-fs.target are 'dead'.

Candidly: that command looks like the least discoverable command I’ve seen in the last year.

It is at least built in a logical way and once you understand that logic then related commands will be discoverable.

`systemctl` is the systemd command line tool for inspecting and manipulating services/units. `systemctl status` is the command for showing the status of a service or unit.

The `--` part is not specific to systemd at all, it's used in many commands to separate flags from positional arguments. This lets you use positional arguments that might happen to start with a dash without them being interpreted as flags. [1].

systemd manages mounts, they're named according to the mount point, with a transformation to turn it into a filename (mostly converting slashes to dashes, plus a .mount suffix). Hence, the root mount ('/') is named '-.mount'.

[1] See, e.g., https://unix.stackexchange.com/questions/11376/what-does-dou...

The annoying part is that now that '-' stands for '/', you need to use '\x2d' for '-'. Which is pretty annoying when manipulating mount names during the shell.

I wish they had picked ':' for '/' instead of '-'. That one would be pretty uncommon to find in a mount path, and has a bit of precedent on OSX.

(Who would ever mount anything on a path with a colon anyway? That would be as silly as using slashes instead of dashes for options)

Have you seen system-escape?

https://www.freedesktop.org/software/systemd/man/systemd-esc...

C:\ much?

I’m pretty sure whoever came up with that was also using slashes for options. Someone was gonna bite :)

Fair. I think it's because systemd lets you use slashes in unit names, but you can't represent those as files so you have to use a dash instead. This is the worst for the root filesystem mounted at / since that's the entire name. In systemd's defense, it warns you when you try systemctl status the normal way.

  Hint: to specify units starting with a dash, use "--":
        systemctl [OPTIONS...] COMMAND -- -.mount ...

I found my mount names by running systemctl list-units, so it's not something I had to look up.

You can use systemd-escape(1) to have it generate the correct unit name for you, e.g.: systemctl status $(systemd-escape --path /home --suffix mount) Yeah, complicated and verbose, but reliable and integrates into systemd mindset very well.

The command

    systemctl status *.mount

Just shows the status of all mount units.

It's the same command you'd use to check any other systemd unit, for example, if you want to check Docker's status:

   systemctl status docker.service

Seems consistent and discoverable as long as you have a surface understanding of systemd units.

That to me reads as "show me the status of all files ending in .mount in the current directory". Once you start hijacking shell metacharacters you're on dodgy ground.

It's not using the shell glob syntax, it's just a regex compatible string. The find utility does the same thing:

    find ~/Code -type f -name *.js

Where *.js isn't using the shell glob feature, it's just a regex compatible string.

You could put quotes around it to make it clearer:

    systemctl status "*.mount"

The string is matching on unit names, and the mount units have the .mount suffix.

> Where *.js isn't using the shell glob feature, it's just a regex compatible string.

Except it isn't, because a `*` at the start of a regex is invalid. `-name` patterns given to `find` are explicitly shell glob patterns. You might be thinking of `-regex` patterns, which GNU find also has.

It's a bit shonky when `find` does it because it's not saying "the shell pattern applies in this directory", and I regularly trip over it. It's something special you've got to know about how `find` works, but at least it's still anchored to the idea of being searched in a path that's also part of the command. There's a semantic link there.

With `systemctl status *.mount` you've not even got that. What path is being searched for `*.mount` files? Is that path part of the `systemctl` interface? Or is this just another way that `systemctl` insists on being a special snowflake that someone's decided I now need to devote neurons to?

Would

  systemctl status -- $(systemd-escape --suffix=mount --path /)

be simpler to read?

If you use a wildcard operator, you can list all the mount statuses: systemctl status *.mount

Complex systems fail in complex ways.

If that happens again I'd try opening a virtual terminal, login as root and check logs.

> If that happens again I'd try opening a virtual terminal, login as root and check logs.

If the system has unmounted all the drives, there won't be any logs. This happens often enough with systemd / journald to be a big source of annoyance.

The only option you get is to tell systemd to boot in debug mode but that fundamentally alters the timing and behaviour of so much that it often stops the weird behaviour from ever happening (similar fun can be had with `dracut` where enabling debug tends to stop any race condition from ever happening)

Note systemd does dump its own log to the kernel ring buffer until journald starts (the last message I see on the kernel ring buffer is Started Journal Service). It's no help though when (as in my case) journald starts but then has /var unmounted below its feet. I can write a script that keeps /var busy though, preventing it from being unmounted.

> I can write a script

There's some delicious irony in having to write a shell script to combat the init system that was designed to replace all the shell scripts.

I don't agree with her sentiment that alpine should adopt something similar. The whole reason I use alpine is because it doesn't use systemd.

I like alpine because it uses a simpler, lighter approach. It's not for everyone, and I'm pretty sure alpine maintainers don't intend it to be for everyone. It serves its niche really well as it is. For those who like systemd there's many distro choices already.

Edit: As detaro mentioned in another comment ( https://news.ycombinator.com/item?id=27176391 ) they are actually working on something, but they aim to solve the various issues around alpine that don't fit well with alpine's philosophy. It sounds like a good story and I'm open to this idea. The main reasons I don't like systemd are its complexity and its dbus reliance, basically bringing too many desktop paradigms into all Linux systems, even servers and containers.

So about my above point: I'm open to it adopting a service manager but not too similar to systemd :)

> By not having something competitive Alpine is less and less attractive for newer production deployments.

Alpine uses OpenRC (https://wiki.gentoo.org/wiki/OpenRC). IMHO Alpine and OpenRC tend to make production deployments more predictable and reliable, because of deliberately simple init ordering, as well as a smaller surface area as compared to systemd.

There is some work on a replacement that does more while fitting the requirements and philosophy of alpine: https://skarnet.com/projects/service-manager.html

According to that page "the main reason Alpine aims to replace" OpenRC is they want the init system to "interface with external events". Huh? The thing I like about OpenRC is is if I `grep -R socket` on its codebase then it comes up empty. Alpine seems to be leaning towards s6 according to that page, which appears to have a treasure trove of network services. Last thing I want is to wake up one morning after updating Alpine and have all these scampering daemons like skadnsd, s6-fdholderd, s6-sudod, s6-ipcserverd, s6-ftrigrd, s6lockd, and ucspilogd skulking in the background. Also the boldness of the name. If you name it System 6 then you're claiming you've built something better than Bell System V. The init process has total power over the rest of userspace. I'm sorry but I don't think your system administration paradigm that MiTMs DNS deserves to be /sbin/init. I just want a UNIX-like system with few opinions that stays out of the way.

Those last four s6-* programs are not daemons. They are chain loading programs that modify permissions or the environment then exec into the next program in the chain.

To be frank s6 and its cousins like runit have so much fewer lines of code -- I'm pretty sure it's fewer than OpenRC and traditional SysV both -- because of simple and thought-through architecture. If you're gonna do a systemd alternative at all, ever, it'd have to be on one of these sane foundations.

Interesting, I have a lot more faith in the alpine team implementing something like this than in RedHat. Even when RedHat made consumer OSes (before spinning them off to Fedora) I didn't like them.

What they're doing looks like a great alternative. I also hope they will not have any reliance on dbus as systemd does. Reliance on dbus is one of the main issues I have with systemd (besides its size and complexity).

The problem I have with it is that it's not easy to interact with, and something that's very desktop focused.

I think it's one of the examples of Systemd pulling in way too wide a scope.

Can you elaborate? I found it very easy to interact with once I learned how to use the tooling (busctl, d-feet, etc). I also don't think I would say it's desktop-focused, it was designed for that use case originally, but it seems it can be used by anything else that happens to use a similar OOP-based RPC design (and it seems systemd is one of those things). Is there another replacement that's of a smaller scope that would cover all the use cases that systemd needs?

Question is if an init system needs all the use cases dbus can support. Dbus always feels unwieldy to work with, especially once you leave specific desktop stacks, and for me personally, the biggest failures of systemd-based systems have been issues with dbus not working properly for whatever reason.

I don't know the status of this on a practical level (e.g. in any given distro) but systemd does support daemonless dbus access over a private socket (/var/run/systemd/private) in some scenarios. Would that help?

Good read, thank you!

I think given these lines in the article (and the fact that she mentions OpenRC):

> I have been an Alpine user for almost a decade and it's one of my favorite linux distributions.

> This talk is intended to show how green the grass is on the other side and how Alpine can benefit from these basic ideas.

She's probably familiar with it.

Okay, I'll admit I was hoping for a blank web page :-). Not a fan.

I always ask ""What problem is this trying to solve?" and then "Is this a good solution for that?"

I appreciate the "startup dependency problem" and when I was in the Systems Group at Sun we worked with AT&T on their solution which was the whole "run level" thing.

And I appreciate the whole "shell scripts can do anything" problem that just using a naming scheme on shell scripts to run can lead to unexpected behaviors.

From an architectural taste perspective, I prefer systems which are more comprehensible in parts and thus don't try to abstract out semantics to the point of incomprehension.

So at the end of the day, something with a better model, better visibility into its operation and configuration, with a way to validate and debug dependencies would be good. But I haven't been impressed with systemd.

It solves a whole lot of problems to me.

* It gets rid of a system cobbled together from init, inittab (which people rarely remember exists), monit, cron, and inetd/xinetd.

* It removes a bunch of horrible hacks in init scripts such as calls to sleep.

* It removes the need for pid files.

* It removes the need for a lot of stupid boilerplate, like the whole start-stop-daemon stuff in scripts.

* It allows precisely tracking what belongs to each service. I can easily find out what a random process belongs to.

* It captures all the log output of each process

* It drastically improves the upgrade process -- I never need to look at a 3-way diff of /etc/init.d/apache2 again

* As a developer it means I don't need to write a slightly different script for every distro

* It makes logging a whole lot pleasant

* It allows having user services without having to hack around it with cron

* It allows a whole bunch of settings to be applied to any random service, without having to edit a shell script and figure out how to stick it in there.

* It means services run identically when started by hand, without the confusion of a service inheriting my local environment.

* It makes my VM servers boot faster. And if you're going to pipe up with "I reboot my servers once a year", that doesn't cut it for me. I boot my servers on demand, when they're needed. They adapt to the load.

I had to chime in here and say I really like this post, thank you for taking the time to write it out. I would give it extra up votes if I could.

It is a nicely concise summary of the problems. I'd disagree with the characterizations of the previous system as "cobbled" or "horrible hacks" but there are solid things that need to be solved well by any system initialization architecture/service.

For me the key idea is that they don't all need to be solved in the same place, by the same project. Implications that the systemd approach is the only way to do so really rub me up the wrong way.

Out of curiosity, what would you do if you had the inclination and the know how for solving all of these issues? Would you create a project for each of them or unite them under a single one like Lennart did?

Which of these can be decoupled in your opinion, and isn’t very close to system init?

> It allows having user services without having to hack around it with cron

Recently, I started using pipewire, which is designed to rely on a user-level service to start and stop the pipewire process.

To bypass this requirement, the slackware guys came up with an elaborate solution that wraps the pipewire command with a daemonize tool that is supposed to cleanup after itself: https://www.linuxquestions.org/questions/slackware-14/using-...

Meanwhile, the gentoo guys just did the simplest thing possible: https://github.com/gentoo/gentoo/blob/f788f38/media-video/pi...

Of course, with both of these workarounds, the semantic changes from "start this process once the user logs in" to "start this process once the desktop environment starts", but the end results are identical for almost all users.

> the slackware guys came up with an elaborate solution that wraps the pipewire command with a daemonize tool that is supposed to cleanup after itself

You mean Slackware users on some random forum. Besides, the solution they came up with uses XDG autostart which has nothing to do with systemd. Not to mention that it's not even doing the exact same thing as the Gentoo solution and running two more commands in addition to pipewire. On top of that, Gentoo also seems to be using XDG autostart for Pipewire[1] so it's basically the same approach.

[1]: https://github.com/gentoo/gentoo/blob/f788f389419678d2f2399e...

> You mean Slackware users on some random forum.

Believe it or not, that's actually the official slackware forum. And whatever solution those guys come up with, it will likely become the official solution.

> Besides, the solution they came up with uses XDG autostart which has nothing to do with systemd.

The slackware solution involves a project that nobody has heard of before, just so it can imitate the "user-level service" feature provided by systemd: https://github.com/raforg/daemon

> Not to mention that it's not even doing the exact same thing as the Gentoo solution and running two more commands in addition to pipewire.

The slackware solution requires starting those 3 processes (pipewire, pipewire-media-session, pipewire-pulse) separately from 3 different .desktop files, likely because the daemon tool above can't properly reap the pipewire-pulse process (not sure whose fault is this though).

On the other hand, the gentoo solution can start all 3 processes with just 1 .desktop files, because `pkill` takes care of it. Simple and effective.

I think the key difference, in this case, is that the slackware guys are trying their best to imitate a systemd feature, while the gentoo guys seem to focus more on finding the best way to allow users to enjoy pipewire.

> Believe it or not, that's actually the official slackware forum.

My bad. TIL.

> The slackware solution involves a project that nobody has heard of before, just so it can imitate the "user-level service" feature provided by systemd

Daemonizing processes has been a thing far before systemd even existed. Solutions similar to this "daemon" project are commonly employed in SysV init scripts. If anything, systemd made those things irrelevant by offering a more simple configuration to daemonize programs.

> The slackware solution requires starting those 3 processes

> On the other hand, the gentoo solution can start all 3 processes with just 1 .desktop files

It looks to me Gentoo doesn't start the other two processes. I don't see how "daemonize" or pkill is relevant here.

> slackware guys are trying their best to imitate a systemd feature, while the gentoo guys seem to focus more on finding the best way to allow users to enjoy pipewire.

With respect, this is a unsound take. The explanation is more simple and it's that Gentoo does not currently handle pipewire cleanup.

> * It drastically improves the upgrade process -- I never need to look at a 3-way diff of /etc/init.d/apache2 again

lol. I had to trash an Arch Linux box because it failed to upgrade systemd from 208 to 211. Everytime I upgraded the system hung trying to mount filesystems so I had to roll back.

You’re complaining about something different. If there’s a bug in systemd itself that makes your system unbootable then not much you can do except not upgrade. What the parent is talking about is the fact that services are organized into base service files and overrides. Base files live in /usr and are to be modified only by the package that owns them. Override files or full-replacements live in /etc which is for the local administrator. Service updates are now always a safe operation.

Perhaps arch broke something? How is it the failing of systemd without further verification??

How much faster do your VM servers start with systemd?

On the desktop, I feel like boot time is slightly longer than it was 20 years ago, yet hardware is faster now. I used to shut down my computer every night back then; now I leave it running, because it takes so long to get back to where I left off.

> now I leave it running, because it takes so long to get back to where I left off.

I think it takes my Linux system ~13 seconds to boot? That is definitely less time than it took 20 years ago, but I think that's mostly b/c the storage is no longer slow spinning rust running on IDE cables more than anything else…

Returning to state is what takes time, not the boot process to login.

It's Monday morning at 10:30, I've already got 29 tabs on 3 windows across 2 virtual desktops (2 monitors each) running on my main desktop. I've trimmed down to just 30 various rxvt windows with ssh or vim open, with files and notes at various stages of completion.

Some tabs may restore next time I load my browser, some won't.

It's because of this that I've started to do everything in VMs. I can always save state and pick up where I left off without worrying about losing anything.

Don't you have the same problem of having to reboot the VM occasionally?

Haven't compared because I'd have to try SysV and see how much longer that takes.

My current server boots in 10 seconds, which isn't bad.

SystemD solved enough problems that such a large number of distro owners switched to it that one of the biggest complains against it is how prevalent it is.

> when I was in the Systems Group at Sun we worked with AT&T on their solution which was the whole "run level" thing

Interesting that you mention Sun. Didn't SMF precisely replace runlevels with SMF milestones[1] in much the same way systemd replaces runlevels with target units?

[1] https://docs.oracle.com/cd/E23824_01/html/821-1451/hbrunleve...

1. You don't have to clone then modify a 100 line bash script for a normal daemon's /etc/rc.d/init.d script.

The OpenBSD init system and OpenRC init system haven't had this problem for years -- even before systemd became a thing.

OpenRC is the only init system I've ever used on Linux that I actually like, rather than just tolerating.

Similarly runit, s6, and Upstart have never had that issue. It is the oddest complaint to come up with when considering life without systemd.

Upstart was good and probably paved the way to systemd.

No, Upstart’s design was so bad that even Scott James Remnant (the creator of Upstart) admitted it’s design was inherently flawed.

The biggest problem with Upstart was that it put the dependency chain upside down and just started a service whenever it’s dependencies were fulfilled, no matter whether it actually made sense to start the service.

I remember reading about it and thought it seemed wrong, but presumably the author did his research and knew what he was doing. Oh well. His attitude and the readability of Upstart code were very good, though.

The Upstart design was probably easier to implement. With the systemd design, the dependency tree needs to be traversed bottom up (figure out what needs to be started to start the targeted set of services), then top down (start dependees when dependencies are ready). With Upstart, top down was enough. Or so it seems.

Do you have a source on SJR stating that the event based design was inherently flawed?

I think the comments towards the bottom of this thread is what is being referred to: https://web.archive.org/web/20140928104327if_/https://plus.g...

I wonder if systemd is influencing a new init right now

I for one have great sympathy for people who are placed in the position of having to give good reasons for using systemd. "All your distro are belong to us" was the original plan but it didn't quite work out. The shills had to scramble but they managed to make the best of a bad situation.

I've not seen even sysvinit based systems with this problem. Every service manager I know of solves this problem. This is as much an argument in favour of systemd as "don't have to clean up horse excrement" is an argument in favour of owning a fiat 500.

Most init scripts are in that ballpark. See openssh for example (88 lines) https://github.com/openssh/openssh-portable/blob/master/open...

And yet, it still tries to use pidfiles (which systemd has a better alternative for), and notes "We really should check if the service is already going" which they don't do because it's an actual non-trivial issue if you're writing a standalone init script.

Does anyone use these init scripts? Every popular sysvinit based init system uses short declarative service scripts for most services. Every non-sysvinit init system seems to mostly manage with 2 line scripts.

And yes pidfiles are crap and I don't like them either (a genuine criticism of sysvinit and advantage of systemd) but systemd isn't the first init system to do away with pidfiles so it's hardly a unique strength.

Could you show an example of a declarative sysv-init script? To be honest, I'm not aware of it being possible and ddg doesn't bring up anything.

> Does anyone use these init scripts?

By default every popular distro before systemd/upstart did. Many services still use them anyways (systemd just generates a shim to run the start/stop/status operations on them)

> By default every popular distro before systemd/upstart did. Many services still use them anyways (systemd just generates a shim to run the start/stop/status operations on them)

Honestly I wouldn't be surprised if much of the negativity surrounding systemd (other than people who don't like the architectural approach) is from not being able to use a clean startup configuration, instead having to use a bolted-together compilation of startup methods under the systemd "brand".

It's been a while since I've confirmed this with a new install, but while my Arch Linux setup is very clean, Ubuntu has always been a mess. It still has compatibility with the old "service" command (from Upstart I think?) for goodness sake. My Ubuntu install still has a ton of scripts running under the systemd shim, while as far as I can tell everything that is running on Arch has a .service file. If you're going to appreciate the value of systemd, the latter is a much better experience.

> This is OpenBSD's sshd init script:

It's highly abstracted with macros, but that's not the same as declarative. It's still using full shell evaluation and any service can complicate it or make it slightly different. It's pretty close though.

> I see that the equivalent of the rc_reload function is "ExecReload=/bin/kill -HUP $MAINPID" (which is actually less safe...)

> man: "rc_reload - Send a SIGHUP signal using pkill(1) on the regular expression given in the pexp variable."

I don't understand why it would be less safe. $MAINPID is well defined because systemd knows the top process in that service's group. A pkill/regex by definition matches by text, so it can HUP any process with matching name - whether it was started by this service or not.

> I mean is anyone CURRENTLY using them?

Not that I'm aware of. They're still installed for people who don't have a different supported init system, but that's just edge cases these days.

It's less safe because it reloads sshd regardless of if the reload would cause sshd to fail to reload. It runs a config check of sshd before it goes ahead and asks sshd to reload its config.