This is the approach used by Kata Containers/Firecracker. It's not much heavier than the shared kernel approach, but has significantly better security. An bug in the container runtime doesn't immediately break the separation between containers.
The performance overhead of the VM is minimal, the main tradeoffs is container startup time.
I wonder why Apple cared so much about the security aspect to take the isolated VM approach versus shared VM approach. Seems unlikely that Apple hardware is going to be used to host containerized applications in production where this would be more of a concern. On the other hand, it's more likely to be used for development purposes where the memory overhead could be a bigger concern.
On a non Linux OS that should be offset by being able to allocate RAM separately to each container instead of the current approach in Docker Desktop where a static slice of your system memory is always allocated to the Docker VM.
This a feature targeting developers or perhaps apps running on end-user machine where page cache sharing between applications or container does not typically get much of RAM saving.
Linux kernel overhead itself while non-trivial is still very manageable in those settings. AWS Nitro stripped down VM kernel is about 40 MB, I suppose for Apple solution it will be similar.
No it's the opposite, the entire premise of Docker over VMs is that you run one instance of all the OS stuff that's shared so it takes less resources than a VM and the portable images are smaller because they don't contain the OS image.
The premise is containerization, not necessarily particular resource usage by the host running the containers.
For hosted services, you want to choose - is it worth running a single kernel with a lot of containers for the cost savings from shared resources, or isolate them by making them different VMs. There are certainly products for containers which lean towards the latter, at least by default.
For development it matters a lot less, as long as the sum resources of containers you are planning to run don't overload the system.
The VM option is relatively new and the original idea was to provide that isolation without the weight of a VM. Also I'm not sure that docker didn't coin the word containerization, I've alway associated it with specifically the kind of packaging docker provides and don't remember it being mentioned around VMs.
> On non-Linux, you obviously need an additional kernel running (the Linux kernel).
That seems to be true in practice, but I don't think it's obviously true. As WSL1 shows, it's possible to make an emulation layer for Linux syscalls on top of quite a different operating system.
I would draw the opposite conclusion from the WSL1 attempt.
It was a strategy that failed in practice and needed to be replaced with a vm based approach.
The Linux kernel have a huge surface area with some subtle behavior in it. There was no economic way to replicate all of that and keep it up to date in a proprietary kernel. Specially as the VM tech is well established and reusable.
> an emulation layer for Linux syscalls on top of quite a different operating system.
My point was that, in principle, it could be possible to implement Linux containers on another OS without using VMs.
However, as you said (and so did I), in practice no one has. Probably because it's just not worth the effort compared to just using a VM. Especially since all your containers can share a single VM, so you end up only running 2 kernels (rather than e.g. 11 for 10 containers). That's exactly how Docker on WSL2 works.
I think that's the point. You don't have to run the full kernel to run some linux tools.
Though I don't think it ever supported docker. And wasn't really expected to, since the entire namespaces+cgroup stuff is way deeper than just some surface level syscall shims.
> On non-Linux, you obviously need an additional kernel running (the Linux kernel)
Only "obvious" for running Linux processes using Linux container facilities (cgroups)
Windows has its own native facilities allowing Windows processes to be containerised. It just so happens that in addition to that, there's WSL2 at hand to run Linux processes (containerised or not).
There is nothing preventing Apple to implement Darwin-native facilities so that Darwin processes would be containerised. It would actually be very nice to be able to distribute/spin up arbitrary macOS environments with some minimal CLI + CLT base† and run build/test stuff without having to spawn full-blown macOS VMs.
I could imagine one Linux kernel running in a VM (on top of MacOS) and then containers inside that host OS. So 1 base instance (MacOS), 1 hypervisor (Linux L0), 12 containers (using that L0 kernel).
Shoutout to Michael Crosby, the person in this video, who was instrumental in getting Open Containers (https://opencontainers.org) to v1.0. He was a steady and calm force through a very rocky process.
"A new report from Protocol today details that Apple has gone on a cloud computing hiring spree over the last few months... Michael Crosby, one of a handful of ex-Docker engineers to join Apple this year. Michael is who we can thank for containers as they exist today. He was the powerhouse engineer behind all of it, said a former colleague who asked to remain anonymous."
We can thank the linux kernel developers for implementing namespaces and overlayfs.
And we can thank predecessor systems like BSD jails, Solaris zones, as well as Virtuozzo/openVZ and lxc as previous container systems on linux.
Docker's main improvements over lxc, as I understand it, were adding a layered, immutable image format (vs. repurposing existing VM image formats) and a "free" public image repository.
But the userspace implementation isn't exactly rocket science, which is why we periodically see HN posts of tiny systems that can run docker images.
I would assume that "lightweight" in this case means that they share a single Linux kernel. Or that there is an emulation layer that maps the Linux Kernel API to macOS. In any case, I don't think that they are running a Linux kernel per container.
interesting choice - doesn't that then mean that container to container integration is going to be harder and a lot of overhead per-container? I would have thought a shared VM made more sense. I wonder what attracted them to this.
Note that they didn't "do it" for WSL1, they started doing it, realized it is far too much work to cover eveything, and abandoned the approach in favor of VMs. It's not like WSL1 was a fully functioning Linux emulator on top of Windows, it was still very far from it, even though it could do many common tasks.
I've always wondered why only Linux can do 'true' containers without VMs. Is there a good blog post or something I can read about the various technical hurdles?
BSD can do BSD containers with Jails for more than a decade now?
Due to innate features of a container, it can be of the same OS of the host running on the system, since they have no kernel. Otherwise you need to go the VM route.
> Containers build on top of the host operating system's kernel (...), and contain only apps and some lightweight operating system APIs and services that run in user mode
> You can increase the security by using Hyper-V isolation mode to isolate each container in a lightweight VM
I'm not sure about MacOS, but otherwise all major OSs today can run containers natively. However, the interest in non-Linux containers is generally very very low. You can absolutely run Kubernetes as native Windows binaries [0] in native Windows containers, but why would you?
Note that containers, by definition, rely on the host OS kernel. So a Windows container can only run Windows binaries that interact with Windows syscalls. You can't run Linux binaries in a Windows container anymore than you can run them on Windows directly. You can run Word in a Windows container, but not GCC.
Is there any limitation in running older.NET Framework on current Windows? Back when I was using it, you could have multiple versions installed at the same time, I think.
Containers are essentially just a wrapper tool for a linux kernel feature called cgroups, with some added things such as layered fs and the distribution method.
You can also use just use cgroups with systemd.
Now, you could implement something fairly similar in each OS, but you wouldn't be able to use the vast majority of contained software, because it's ultimately linux software.
FreeBSD has linuxulator and illumos comes with lx-zones that allow running some native linux binaries inside a "container". No idea why Apple didn't go for similar option.
FreeBSD Linux emulation is being developed for 20 (may be even 30) years. While Apple can throw some $$$ to get it implemented in a couple years using virtualisation requires much less development time (so it’s cheaper).
It puts them on par with Windows that has container support with a free option, plus I imagine it's a good way to pressure test swift as a language to make sure it really can be the systems programming language they are betting that it can and will be.
OrbStack has a great UX and experience, so I imagine this will eat into Docker Desktop on Mac more than OrbStack.
syscalls are just a fraction of the surface area. There are many files in many different vfs you need to implement, things like selinux and ebpf, iouring, etc. It's also a constantly shifting target. The VM API is much simpler, relatively stable, and already implemented.
Emulating Linux only makes sense on devices with constrained resources.
Looks like each container gets its own lightweight Linux VM.
Can take it for a spin by downloading the container tool from here: https://github.com/apple/container/releases (needs macOS 26)