This article is a little misleading in that it seems to indicate that FB will trash Xeon servers and drop in ARM processors. Current ARM processors (SOC and standalone silicon) have poor data throughput. You can mask this using media accelerators, e.g. decode x264 video in the GPU so you have smooth playback while saving some data bandwidth. ARMv8 (allegedly sampling in 2014-2015) will double the AHB/CoreLink width and instruction size to 64 bits, but we are not there yet. What is much more likely is that they will incorporate specialized hardware using FPGA/ASIC and use ARM cores as supervisory modules. That way, they can use a general purpose OS and use ARM tools to develop for the supervisor module, and still get the data throughput of the custom controller. This is how data processing is already done on 1080/2k/4k video streams. Xilinx and likely Altera already provide FPGAs with hard ARM cores (in addition to IP for soft cores) which makes it very easy to roll your own data controller with an ARM core managing it. Makes a lot more sense for FB to go this route for custom NAS boxes, possibly network switch/router hardware to follow in Google's path.
I didn't get the impression at all that Facebook was "dumping" Xeon, merely that they were adding ARM.
Throughput is not really relevant. Throughput per unit power is better. ARM manufacturers have been optimizing for power consumption and unit cost for a long time, Intel has been optimizing for throughput and speed for a long time. Both sides are aiming for the same target, from different angles. A "low power" Xeon might be 40W for four cores, but the typical range is 60W - 140W. Calxeda sells a sixteen-core, 20W ARM board. By these numbers, Xeon needs to have 8x throughput of the ARM to beat it in throughput per watt.
Now, this is not entirely unreasonable. I've done benchmarking of my Core i5 against my Atom on an audio sample rate conversion library I wrote, and the Core i5 has 6-7x the throughput of the Atom. So, I would believe you if you said that a Xeon core is more than 8x as fast as an ARM core, and therefore the throughput per watt is better. This is all guessing, based on one benchmark I did for a library I wrote. I'm sure that Facebook has done some experiments with the applications they'll actually be running on the ARM.
I'm not sure where you get the idea that these ARM computers will be centered around specialized hardware, Calxeda's website only mentions general-purpose hardware built around ARM cores.
If your business is largely shuffling data around then your machines may be DRAM speed limited. Sounds like a lower power processor is going to make sense even if it has smaller, slower caches.
I hear Applied Micro will sample their custom ARMv8 chip by the end of this quarter. It's likely we'll see at least the custom ARMv8 chips start shipping in the beginning of 2014. The stock Cortex A57-based chips like what AMD and I think Calxeda, too, are using will probably arrive late 2014.
I think Nvidia might target Project Denver for servers, too, and that's also a custom ARMv8 chip, but I don't think it will be ready until the 2nd half of 2014 (even though they initially promised late 2013). They might be waiting on the Maxwell GPU architecture to be ready.
As far as I can work out from the details released so far, this is basically a standard backplane for "blade" (or microserver as they are now known) servers, based on PCIe x8 so it is simple and cheap, which just happens to be more of less what Calxeda was using anyway.
I can't work out yet if there is a standard way of attaching RAM as well I don;t think so (yet).
It has taken an open source project to make blade servers standardised by the look of it, should make the whole ecosystem much cheaper than the proprietary systems.
It is not actually clear if Facebook has committed to ARM, although it would not surprise me if they use some. But they have the option.
Although this project is vaguely similar to blades, it isn't hot-swap, it can't hold 2S/4S servers, and it's not clear how I/O consolidation will work. So it's pretty different from existing blades.
Hot swap would not be that hard to add to pcie but I don't think they care. Sure there is ram per soc but the discussion implied they wanted to break it up more longer term...
This reminded me of James Hamilton's blog post [1], about 48 Atom servers in 3U, or 624 servers into a standard 42U rack [2]. Shows you where the industry is headed.
Compared with virtualization, it would be more secure. If a dedicated ARM (or Atom) microserver were competitive with mid-range VPS, people would buy it.
Virtualization is "leaky", there are esoteric attacks against other guests on the same host that allow attackers to obtain encryption keys. Last time I checked, the mitigation strategy was to move to dedicated hardware.
I doubt dedicated microservers will be competetive with VPS for a while yet because you can't oversell them.
