We did something similar at MSN in the 90s - once a dialup ISP was spending so much on transit to us, we’d install a dedicated circuit directly from MSN to whatever mom & pop ISP. It was nice for everyone.
It's a net win for the ISP due to cost savings from not having to haul that traffic across their network/to the Internet.
You're right that powering racks isn't free either though: if the cache operator doesn't utilize the rack well and isn't offloading enough traffic to it, the ISP will give them the boot (as in, "we wheeled your rack out to our docking bay, come pick up your piece of junk if you want it back"). It's uncommon since _most_ of the orgs running these racks are competent, but does happen from time to time.
Also, even though it's a net win for the ISPs, there are still cases where the operator ends up paying them a fee. This has less to do with the economics of edge caching within an ISP network and more to do with the bargaining power certain ISPs have. The 2014 Netflix/Comcast peering agreement is a good example of how those things sometimes pan out.
MSN paid. We also carried all traffic for Microsoft properties; if you’re an ISP, whatever you were paying for Microsoft data went to zero and your other links had more capacity.
A neat secret about these cache devices is ISPs that charge for usage. The data the customers use comes straight from the ISP's Central Office, not from "The internet" at all.
My parents have a (crappy) wireless internet provider. During most evenings and weekend, they're lucky to get even 0.5Mbps. At the same time, Netflix works fine, and the speed on fast.com (Netflix-run) will be a few Mbps (I don't remember exactly). In the daytime on weekdays, they usually get a few Mbps from any speed test and generally everything feels fast.
This lets me deduce the ISP is way over-subscribed on their upstream internet connection, but also has a Netflix appliance.
What's frustrating is if they complain enough, the company will send a tech out who will "adjust" their antenna, or suggest it's a line-of-sight problem and they need a bigger tower, should cut down some trees, etc.
> "My parents have a (crappy) wireless internet provider. During most evenings and weekend, they're lucky to get even 0.5Mbps."
Sounds like my experience on Three UK here in London. Just awful! Vodafone and O2 are at least 20X faster at peak times despite Three actually having a significantly stronger 4G signal at my flat. Absolute joke of a network. I'm so angry that I was basically tricked into a 12 month contract with them.
The problem with Three is they don't have enough spectrum especially in densely populated areas, they definitely have invested a lot in backhaul capacity. 99% of cellular data problems are nothing to do with backhaul or peering, it's spectrum usage.
Some good news: they are aggressively rolling out SDL (supplementary downlink) which will really improve things. They also announced today their 5G network (which has by far the most spectrum of all UK carriers) has went live. I would expect insanely good speeds on that network.
But yeah, turns out selling unlimited data/tethering/roaming/text/minutes for as low as £11/month isn't a good business strategy.
Generally though: EE is good everywhere. Vodafone is good in North London, O2 better in South London. Three generally awful everywhere.
Three’s 5G has actually been running since last summer : I should know, I was in one of the pilot areas in London and signed up for their home broadband offering on day one.
No complaints, 100-300Mbps on average.
They seem to have done great at the 5G spectrum auctions, they’re the only UK carrier with 100+ Mhz.
I’m also using them for mobile 4G and yes that’s often awful, as others are reporting. I should probably switch to giffgaff (safer to use different providers anyway in case Three has a catastrophic outage), but Three’s "Feel at home" free roaming in many countries beyond the EU (including the US) is super handy and has no equivalent that I know of.
> "Three’s 5G has actually been running since last summer : I should know, I was in one of the pilot areas in London and signed up for their home broadband offering on day one."
But this is not really the Three network. It's a separate network with a different network ID and cannot be accessed with ordinary Three devices/SIM cards.
It's based on the network formerly known as "Relish", which was bought by Three in 2017. The areas covered by Three Broadband's 5G are basically exactly the same areas which were always covered by Relish. They just upgraded the equipment to use 5G radio.
Relish held a lot of spectrum in the 3.5 Ghz (n78) band. Three, by arrangement with OfCom, added Relish's holding to their own 3.5 Ghz spectrum won at auction to give them the contiguous 100 Mhz.
There is an additional 200 Mhz of 5G spectrum being auctioned this year, which should bring the other operator's 5G holdings up to comparable levels with Three's:
> But this is not really the Three network. It's a separate network with a different network ID and cannot be accessed with ordinary Three devices/SIM cards.
I'm not in the UK and ask purely out of curiosity, but what SIM/configuration do you use then?
I've found 5G speeds on Vodafone to be quite variable. I have a weak 5G (Vodafone) signal at my flat, and at times I've seen it as high as 130 Mbit, but at other times it drops to 1-2 Mbit or vanishes completely. Where as on 4G I get a consistent 20-30 Mbit all day long.
I suppose this will improve when they build out the network more, but even in locations with strong 5G signal the speed seems to vary a lot.
Seems like 3's 5G network has still not actually launched, but they're now promising it "by the end of February".
Is 3's SDL in addition to their carrier aggregation ("4G+") rollout? Because I already have that and it doesn't seem to have helped much!
Keep in mind that current 5G deployments in the UK (and I think the world) bond 4G and 5G together. This often causes problems if the 4G is heavily congested. Vodafone's London 5G deployment is still pretty spotty, it will get a lot better. Plus it uses a very high frequency band (3.5GHz). When the 700MHz band gets added in a couple of years it will be a lot more consistent.
Yes, SDL is on LTE1500, LTE Band 32. It is another carrier to aggregate with the other ones. Both 3 and Vodafone have some spectrum in that band, IIRC 3 has a lot more. It will really help with overloaded cells in the short term.
My router (Huawei E6878-870) apparently supports LTE band 32, but I've never seen it used on either Three or Vodafone. At my flat, Three seems to use band 3 (1800 Mhz) always.
Vodafone floats between bands 7 (20 Mhz channel!), 20, and 1, but gets good speeds on any of them.
Three always slow at peak times, sometimes unusably slow, despite having by far the strongest signal.
This sounds a lot like Freedom mobile/Wind here in Canada. Although it was the opposite, the cities had the good coverage but if you drove an hour out of town it was either offline or really slow.
The networked sucked at first but they are slowly improving things and offered far better customer service and contracts than the existing 2-3 monopolies.
But sadly if you really want the cutting edge of speed and spectrum you need to use one of the monopolies.
That's interesting. In Australia, if you happen to use Telstra 4G (the only provider that is reliable for most of the country) I've found that speed in remote towns can be a lot faster than in any of the capital cities. It's surprising as the remote towns can either on satellite link or at the end of 1000km of fibre.
> The problem with Three is they don't have enough spectrum especially in densely populated areas
Isn’t this argument basically debunked? Places like South Korea and Japan have no issues whatsoever with extremely fast wireless internet at a fraction of the price.
It's pretty clear that Three's spectrum is woefully lacking compared to the others, except on 5G and the LTE SDL band 32 (both not widely deployed yet).
Especially problematic when you consider that Three are the ones who have been selling super cheap unlimited data packages for the last couple of years!
On one hand you have people saying 4G is good enough, and say 5G is hype only ( It is over hype, but surely not hype ), on the other hand you have people complaining about 4G capacity.
It seems most people when discussing 4G or 5G have absolutely no idea what Capacity and Bandwidth is about. They only care about absolute speed.
I've heard that in the olden days European ISPs would sometimes charge different rates for transatlantic traffic and other traffic and similar in Australia for local and overseas, but generally nobody breaks out traffic charges by destination; it's difficult to manage, it's confusing for customers, and it's hard for customers to find out which IPs will get preferred rates or how to direct their traffic there.
Charging different rates for different traffic happens here in New Zealand. I generally see variation on a theme of ‘bonus free social media traffic’. It’s all very depressing. Eg: https://www.spark.co.nz/shop/mobile-plans/socialiser/
Well, as a subscriber everything beyond your cable modem is "the internet" and you don't care how the ISP's network is set up internally. CDNs, netflix, etc have all been embedding devices in carrier networks for years.
Some of your ISP's connectivity probably comes via settlement-free peering, yet we pay for that too. None of this is a secret, it's just how the internet's plumbing works.
Qwilt and similar products are really neat, but I don't think they've entirely panned out. The idea to do transparent caching at the network edge was hurt by the move to TLS everywhere, and there's a lot less cost incentive when things like Open Connect exist (eg. if Netflix will give me a rack that offloads XX% of my total traffic, how much additional traffic do I need this transparent cache to offload before it's cost effective for me?). Without transparent caching the economics get trickier.
It seems like their niche is as an easy to deploy, more traditional looking CDN that can run out in the RAN, which does still have value.
For now, I think the majority of bandwidth would be the zillions of iPhone users doing backups and app updates all the time. However, this is definitely a way to pave the path for Apple TV+.
That’s a sure-fire way of getting dataloss. Wouldn’t it be much easier to just upload the backups from users devices when, idk, at night? iCloud requires your device to be locked, connected to WiFi, and charging, so I imagine that already shifts the traffic to off-peak hours.
Not sure it is still the case. MacOS Server have this function if you set it up your network, it will cache all your backup, App Update, iCloud Files and Photos on your Mac.
Yes, but the peering/traffic teams are the ones that generally manage these embedded cache deployments, hence why related info are on the same pages as peering details or in PeeringDB. All the ones I linked do actually have edge cache racks, not just peering arrangements.
Do you have a rough overview of how the requirements differ across the similar edge programs linked above? If there's one thing Apple does better, it's providing a clean straight forward page with all the relevant info and little fluff.
There are some facility and org requirements depending on the various company's rack setups, but most of that's standard.
The main requirement is how much traffic you need to be doing with the peer before they give you a rack - Apple states 25Gbps; more mature programs like Netflix are 5Gbps. Sometimes that's negotiable and they'll hand them out for even smaller traffic amounts (I've heard less than 1Gbps for some of those listed, though won't mention which ones). Like most things in this area of networking, there's a lot of variability and personal contacts involved.
That's not really suprising. I imagine setting up a new deployment isn't terribly expensive (now the R&D is done), but given you are going to be spending a few weeks working with the ISP to get it up and running, you probably want it to be with someone you get on well with.
It's not only that. The community is comparatively small and tight-knit, one's technical abilities can be judged relatively comfortably from afar, and social currency matters a lot. Everyone with a stake in The Network already has to trust and be trusted by their peers. In that situation, informal agreements seem to work out well.
The website isn't really targeted at the people deploying these. If you are pushing more than 10 Gbps into another ASN you'll start getting emails from their peering coordinators.
Simpler tricks but along the same lines have been described on, IIRC The Old New Thing (Raymond Chen's blog for Microsoft) although it's possible it was Michael Kaplan in which case you'd have to dredge the Internet Archive or something because Microsoft deleted Kaplan's blog years ago, then they fired him, and then he died (he had a degenerative disease, although getting fired basically as an exercise in plumping the numbers for the stock market probably isn't good for you either).
If you're interested in similar edge cache programs:
https://openconnect.netflix.com/en/
https://www.facebook.com/peering/ (though I don't see FNA specifically mentioned there)
https://peering.google.com/#/options/google-global-cache
https://www.akamai.com/us/en/products/network-operator/akama...
https://peering.azurewebsites.net/peering/Caching
https://www.cloudflare.com/partners/peering-portal/