Hacker Newsnew | past | comments | ask | show | jobs | submitlogin
Several Apple Services Down (apple.com)
257 points by jader201 on Sept 30, 2020 | hide | past | favorite | 126 comments


As an aside, that's quite a nice and concise status page, considering the number of discrete services they provide.


And super easy for at least this colorblind individual to parse.

I'm off to see if I can get Greasemonkey to swap out the icons for icons like these on the AWS status page.


Outage is a red triangle for anyone wondering. Apple doesn't show non-current states in their legend.

There's a screenshot in the comments of https://www.macrumors.com/2020/09/29/apple-icloud-services-i...


Here are the four possibilities:

https://imgur.com/a/3Cl5567


Sadly, an individual on an older device who is browsing with JS disabled for accessibility reasons (else most of the web freezes up the device and takes down the web browser) is not able to access this basic information.


Have you ever checked out uMatrix? It's a chrome plugin that lets you control which javascript can run on a page (and allow some by default across the web based on where it's loaded from).


That's a nice idea, but it's not a solution. The real solution is to make content accessible, rather than depending on JavaScript. Apple has terrible form in this regard.


Pragmatically, I think that ship sailed a decade+ ago. You're not wrong though...


developement of uMatrix is no more.


There is a fork 'nuTensor' with stated intention of continuing development/maintenance (with a Firefox focus):

https://github.com/geekprojects/nuTensor


uMatrix has been discontinued.


It still works though. I realize it's not a perfect solution, but I was just trying to offer another option.


does not work on ios 7.


The most important feature (and most difficult to implement) of a status page is that it works.



Cook: “I’m so glad that really can’t happen.”


Perhaps instead of Tim meaning that its impossible for Apple to have a service outage, he was instead talking about the incredibly unrealistic things that were happening in the video. 1 guy accidentally unplugs 1 cable and brings all of Apple down, which results in applications closing out on everyone's devices and the apps just start magically uninstalling themselves.


Hah this was great


Developers & product/project managers across the industry are desperately trying to hit their Q3 release targets.


This, after yesterday Microsoft's auth servers and other services went down. Are the citadels under attack?


There have been quite a few high profile outages recently, although I don’t think correlated.

At the suggestion of several kind people here, I started publishing quarterly round ups of all the major outages on the StatusGator blog. We have lots of data about outages, though the last post took a mostly qualitative approach. Hopefully we can get more sophisticated with it in the future and be more quantitative, perhaps funding outages that affect or cause other outages.


More like cascading effects from the extended work from home and extended quarantine orders.

Ops people also get tired of video conferences.

Ops people are used to working together in a room with stats, runbooks, subject matter experts at hand for consults, going to the next office over to get the people who wrote the code for the service to pop by and consult on how/why a service went down.

All of those procedures are no longer useful.

It's getting to all of us.


Then what happened last year that made services go down?


Maybe they just all went the wrong path when it comes to managing complexity : trying to much to abstract it away with abstractions like kubernetes.

https://www.protocol.com/apple-hires-cloud-open-source-engin...


Jeez, AWS, Azure, Google, and now iCloud. I don’t generally align myself with conspiracies, but it is very interesting to me that all 4 of these tech giants had severe issues recently


Everybody in a rush trying to ship their stuff before the end of the quarter ... before holiday shopping season code freezes? ️


I suspect that this is all a symptom of increased load since the world stayed home and went online.

Remember years ago when apple first shipped the iphone on AT&T and many cellphone calls just couldn't connect? I think it happened for multiple reasons all along the food chain when so many demanding phones came online. It took a while to build things out so things were less fragile.


Gotta hit those OKRs...


it's as if centralizing everything on a system designed for decentralization for precisely this reason, was not a great idea


Happens every year around this time. Teaching events.

People who left university and are now starting to work autonomously make more mistakes, it’s only natural.


if you want to put your tin foil hat on, add the recent Huawei's R&D lab explosion.


All of these have particularly large datacenter footprints in places where fires are ravaging the countryside.


Can you point me to the AWS and Google issues? I know about the Azure Active Directory failures yesterday.




Perhaps the NSA is acting like a digital immune system by provoking an immune response in our tech giants.


> an immune response

Immune system response? In another analogy, Snowden showed us that the American gov + the NSA has literally monopolized their Corporate-DNA Crispr gene editing tool + corrupt laws spreading the disease of ‘Patents” and invention monopolizing ‘intellectual property’. It’s literally withholding the ability for people to take the steps they need to learn. Contrived scarcity [1]. I don’t understand how anyone can see this monopolizing system where Silicon Valley is the place of inherited privilege as anything besides the hypocritical shitshow that it is.

[1] "They assert a belief in ‘free markets’ and want us to believe that economic policies are extending them. That is untrue. Today we have the most unfree market system ever created [...]

How can politicians look into TV cameras and say we have a free market system when patents guarantee monopoly incomes for twenty years, preventing anyone from competing? How can they claim there are free markets when copyright rules give a guaranteed income for seventy years after a person’s death? How can they claim free markets exist when one person or company is given a subsidy and not others, or when they sell off the commons that belong to all of us, at a discount, to a favoured individual or company, or when Uber, TaskRabbit and their ilk act as unregulated labour brokers, profiting from the labour of others?

Far from trying to stop these negations of free markets, governments are creating rules that allow and encourage them."

https://www.resilience.org/stories/2017-08-03/book-day-corru...


Also worth looking at their developer-facing status page, where a couple of their developer products are currently facing issues: https://developer.apple.com/system-status/


This page (as of writing) says iMessage isn't affected, but anecdotally, while I've been able to send text over it, images have been failing to send for about an hour.


I wonder if they leverage iCloud services under the hood for images.


Yes

> APNs can only relay messages up to 4KB or 16KB in size, depending on iOS or iPadOS version. If the message text is too long or if an attachment such as a photo is included, the attachment is encrypted using AES in CTR mode with a randomly generated 256-bit key and uploaded to iCloud. The AES key for the attachment, its Uniform Resource Identifier (URI), and a SHA-1 hash of its encrypted form are then sent to the recipient as the contents of an iMessage, with their confidentiality and integrity protected through normal iMessage encryption

https://support.apple.com/guide/security/how-imessage-sends-...


the images are all stored on cdns.


My images are being delivered but very very very very slowly.


The page now tells me that everything is operational.

However, I cannot get this information without enabling JS and image display.

All for a small page with a couple lines of text.

Wrong.


I'm an App Store developer, and every time that App Store downloads fail, users email me instead of contacting Apple, even though Apple controls the entire installation process, and there's nothing developers can do. It's so frustrating.

Apple charges 30%, offers crap service, and developers shoulder the entire support burden.


Correlation doesn’t mean causation, but given the recent string of service interruptions it seems like there’s at least some increased indication of a cohesive/targeted cyber attack campaign here right?


I think this would be quite low on my list of assumptions of the cause of downtime. Issues might cause a domino effect that may take days or even weeks to surface, or the mitigations to original problems might introduce issues of their own.

Last I checked, Apple runs a lot through AWS which has a few DDoS resilient interfaces to the web very commonly being used by its customers.

As for other cloud provider outages, I doubt it's more than a coincidence if nothing has come out about the issue weeks after it first started - still possible there's some bad actor causing outages in ways that aren't detectable as botnet attacks.


Do this kind of Apple incidents happen more often than before? Coz recently I had a bad experience for my MacBook (to an extent that I wrote a blog post about it), which made me feel that Apple is getting less committed to its products and service level.


In general their cloud offerings have got a lot better over recent years. They had a lot of problems in the early days.


Apple software was never any good but their hardware was great. The quality of both has been in decline for at least five years and in complete freefall for at least two. I'm not sure what changed. Frustratingly they are still the best option.

I have lost all faith in Apple. I just hope their engineers have the decency to be ashamed of themselves.

At this point I skip updates until forced and I'm dreading the day my '14 RMBP finally spins it's last fan.


If they are still the best option, why should their engineers be ashamed? Should all other engineers be even more ashamed?


They should be ashamed because they keep shipping broken products that are worse than their predecessors.


Ignore the downvotes. You are right. I jumped ship recently because of endless problems. The last straw was the 2 week old magic keyboard that the space bar failed on.

I’ve had three failed MacBooks in recent years to the point I stopped buying their laptops. Now using a custom built PC which is about the same grunt as a mid range Mac Pro but 1/5th of the cost. Oh did I mention Apple being too expensive?


What would you choose for a laptop? Seriously considering to jump ship as well, because my MacBook Pro 2016 got BOTH a flexgate[0] screen and a swollen battery[1] on the same MacBook, and yet Apple refused to fix them for free, because they found a little corrosion on the logic board and concluded it as a "liquid damage". I was not given the option to fix the logic board only and get the rest free. Tried to dispute but there was no such channel (expect the general feedback form https://www.apple.com/feedback/macbookpro.html)...

It was a very frustrating experience in terms of the product quality, as well as the way they try to pick on customer's "fault" in order to cover the repair cost of their manufacturing defects.

[0] https://support.apple.com/en-hk/13-inch-macbook-pro-display-... [1] https://support.apple.com/en-hk/13inch-macbookpro-battery-re...


I’m using a Lenovo Thinkpad T495s at the moment as a laptop. The screen isn’t retina grade but it’s cheap and it works perfectly and the keyboard is excellent.


I don't blame the engineers on this but you are right on the situation. They are shipping shittier stuff every passing year.

This is gotta be a management issue though. However, this trend has not reflected on their revenue negatively, management gotta be happy too.


For all the downvoters: I changed my iCloud password two days ago. Messages stopped syncing so I did the same thing I always have to do which is disable the account in Messages, restart the app and re-enable the account. Now for the last 12 hours the app is using 100% CPU and is still not caught up with all messages.

But yeah I'm sure that's my fault.

After all syncing messages from iCloud is a highly CPU intensive operation. It's a miracle we can do it at all.


What is 'Good' software in your book, let alone great?


Software that works as advertised and lets me get my work done.

My work '17 MBP video driver crashes every time it resumes from sleep. The laptop kernel panics at least once a week. This has been true for the three years I have used it. I plan an extra 20 minutes on Monday mornings to unfuck my laptop.

My '14 RMBP can't run some resolutions in MacOS but they work fine in Windows on the same hardware. This is a known driver issue in MacOS and Apple refuses to fix it.

mail.app is a flaky mess as always. I added a fastmail account and my laptop became unusable until I removed it. Still no idea what that's about.

I just double clicked an image in the MacOS messages app because I honestly had no idea what would happen. Will it be a reaction or open a preview? It opened a preview (good) but of a completely different image (not good).


To be honest, those seem very specific. Some issues on your side (like remembering a password).

My experience, I have multiple macs in the house (7) with different hardware and OS versions and haven’t experienced none of these issues.

I use mail.app (multiple accounts including gmail, google apps, exchange, and others using smtp+imap), iWork, iCloud, and messages without issues. With the exception of iWork, the rest I use daily.

The only annoying thing is a delay on FaceTime which means that multiple devices might ring once more even though I already answered a call.


Remembering a password is my failing, I admit to that.

What stinks is how hard it is to successfully change in MacOS. Messages never seems to pick up the change to my iCloud account in settings. I have to go in and fiddle some settings to make it work again.

At my old job I had a '14 RMBP with a Thunderbolt display and everything worked flawlessly. The newer USB-C stuff is a major headache.

I use mail.app and messages every day as well. It mostly works but it still requires fiddling to make it work properly from time to time. Moving large volumes of emails is a major challenge. It requires several restarts of the app and a lot of fan spinning.


> My work '17 MBP video driver crashes every time it resumes from sleep. The laptop kernel panics at least once a week.

This is not normal and will be fixed under warranty like any other hardware defect.

> mail.app is a flaky mess as always. I added a fastmail account and my laptop became unusable until I removed it. Still no idea what that's about.

This is similarly not normal - possibly file corruption due to those panics?


I'm not sure if it is hardware or drivers causing the crashes, it appears to be multiple causes for multiple issues. The laptop is three years old, there is no warranty.

The kernel panics are on a different laptop ('17 MBP) than the mail.app system instability issues ('14 RMBP).

Moving large volumes of mail has been a problem in mail.app for at least six years and remains a problem on both.


No informed scuttlebut on causes yet?


[flagged]


What a stupid post. What gives you any indication that this is the cause?


It seemed like a joke to me. Why meet it with such vitriol?


This is a good time to point out that simple home servers to replace some of these megacorp walled gardens are coming. e.g. https://start9labs.com/


Lost my attention when I saw the first feature listed is “Bitcoin”.


also loads a bunch of trackable stuff from google and such. not a good look when privacy is your main selling point.


Yeah. "Bitcoin", "crypto-", and "block-" are instant tab closers.


Unraid+docker has made self-hosting tons of services super easy. Pretty liberating until you're the one who fucked up the "up-time" and the family starts texting you about Plex being down.


What kind of spam is this?


Someone got lost on their way to their 2017 reenactment session


“No technical expertise required”

Nice. Would love these things to be as easy to set up as a new Apple device. Feel like that’s the 20 part of the 80-20 that these projects tend to leave out.


Lol, the landing page is so dense only an IT guy can make heads or tails out of it


> Embassy > Privacy without trust

no kidding


> A radical, uncompromising, plug and play personal server, offering one-click installation and simple configuration of open source software services that run over Tor V3. The Embassy is your sovereign territory in the land of the Internet. No technical expertise required.

I have technical expertise and that paragraph goes down like molasses.

Imagine if instead it just said

> Plug this in and you have a cloud in your own home.

Sure HN types might nitpick the use of cloud for self-hosted hardware but at least there'd be some hope for non-technical users


I have to say that I'll never download or install anything from that webpage.


could this happen with bitcoin?


I mean not conceptually, but with major traders then yeah it could happen, and that would be where the value of cryptocurrency comes from.


Only if you lose your private key.


No. It couldn't happen, because there would be no services to report a status.


yes.

Because bitcoin still needs infrastructure to be useful to normal humans.

Think of bitcoin like pay pal[1]. Paypal or bitcoin itself doesn't help you if store is down.

[1] Not really but its 3am and that's the best i can come up with


def not. if Apple built on the bitcoin cloud we would never see this mess :p


Of course because it wouldn’t run in the first place.


Of course we wouldn’t see it, it would be closed source


This is good for bitcoin.


"Find My device" is down too. Imagine if I lose my iPhone at the moment and if I cannot find it, will Apple reimburse my loss?


Obviously not. What makes you think this is in any way uncertain?


You should read the terms and conditions for your updates more thoroughly. They are mor liable for you losing your phone. The Find My app is an added bonus to help you find it but not a legal obligation for doing so.


That was more of a question - I was not making a statement. You seem to be rude with your answer.


When multiple major tech companies go down in short order, it's generally reasonable to assume a state actor is responsible.

There are only 3 countries with the resources to mount attacks of this magnitude...


No I don't think that's reasonable at all. What time frame is "short order"? Pareidolia seems like a better explanation.


I only recently learned that term extended more broadly to pattern recognition, in addition to “seeing faces in stuff.”


What time frame is "short order"?

Three days.

Pareidolia

Given that Russia and China have been identified as behind multiple previous outages, it's reasonable to assume they are behind these as well.


I bet if you made a list of all the major tech companies, estimated the average time between failure, and did a little math then you'd find that, statistically, you should expect to see multiple failures within a few days of each other just by chance.


A lot more likely to be the NSA.


This is patently false and paranoid. Any one at most big tech companies can look into the internal report to see exactly which code change took down services by accident, and their remediation steps.

For example, the Google issue this week was an internal routing issue caused by some SRE modifying some config files on load balancers.


I agree its reasonable to speculate, given its a particularly vulnerable time in democracy with US election in 35 days and there's a lot to gain in disruption. Obviously MS/Apple/Google are all the juiciest targets for acquiring private communications, something that can (and possibly did) change the outcome of an election. I would expect this is an extremely active time in attack/defense in cybersecurity regardless of this incident.

IF this is the case, what do we think this could be? What's common among the different outages?

Google says 9/24's outage was traffic routing hardware. [1]

Microsoft says 9/28's outage was authentication error due to a bug in a service update, deployment tooling error, and a bug in rollback [2]

Apple's has affected many services including Apple ID but it isn't clear what root cause is.

[1] https://twitter.com/uhoelzle/status/1309313556328841216

[2] https://status.azure.com/en-us/status/history/


I promise you this is a self-inflicted wound.


Yeah, of the end-of-a-quarter kind.


IDK if its even that.


It's just as silly to promise something you can't possibly know as it is to assume this is a state sponsored attack


You think state-sponsored attacks are just as likely a source of outages as programmer error? Either you think the world is a much more perilous place than it is or you give programmers much more credit than they deserve.


I never said they were equally likely. I said it is just as silly to promise something you can't possibly promise as it is to suggest it is a state-sponsored attack. They easily could have said, "it is much more likely programmer error", and I would have 100% agreed.

> Either you think the world is a much more perilous place than it is or you give programmers much more credit than they deserve

It generally isn't a good idea to make this many assumptions like this about people you don't know. I think neither of those things.


> you can't possibly know

I promise you I can possibly know this.


Or a major network interchange pushed a shitty routing change.


Canada, Australia, South Korea? UK, Russia, German? France, Poland, Singapore? I can think of a lot of countries who have great technology resources.


Canada, Australia, SK, UK, Germany, France, Poland, and Singapore have middling-to-great tech resources, but not the resources to take down global access to the 3 biggest companies in the world.

Only 3 countries have the resources for that: the US, Russia, and China.


I'm super curious why you think that? A 17 year old from Florida tweeted as Elon Musk, Bill Gates or Barack Obama. Why couldn't the Canadian government do it?


Obtaining Admin access on Twitter probably can't take down Twitter. Although, an admin might be able to run heavy tasks that a normal user can't access to DOS the service. That hypothetical would be even easier for sysadmin's to track down though since I believe only a single admin account was compromised.


Not really. The Twitter hack was a community admin (account data, customer service rep), not a sysadmin (software config)


Twitter isn't anywhere in the same league as Apple/Google/Microsoft in terms of security.


It was an anecdote. If a 17 year old can socially engineer twitter, I don't think it's crazy to think that a room of comp sci PhDs in South Korea couldn't figure out how to mess with infrastructure, in fact I'm quite sure they could. There are not 3 countries in the world who could cause this type of outage, there are many.


Do we know a lot about Apple's internal reliability/security engineering? Google writes books about it and has things like Project Zero. Windows is used by governments everywhere and so Microsoft probably has some contacts in high places. I have never heard anything about Apple.


Anybody with access to a large enough botnet can take down services, it isn't rocket science.


True. But a botnet large enough to take down Microsoft, Apple, and Google without being detected would have to be extremely large or well-hidden, so if it is a botnet, then it's even stronger evidence of a state actor.


And yet many such botnets exist and are actively attacking all day long, yet aren't taking down the major professional services outside of tiny temporary corners.


Or, that when one goes down, people overuse the services of the others, causing them to go down, causing a chain reaction. Not denying the existence of state actors but they are not a given.


Sure, that would be a possibility for a different set of tech companies going down, such as if they all shared the same network backbone or overlapping users.

But they don't. We're talking about the largest companies in the world, with facilities geo-located around the world, all going down within a few days of each other.


Yet an attack that large was noticed by none of the major network providers or internet exchange points?


An attack on a large infrastructure is not necessarily a large attack. Social engineer into one account and disable some cron job somewhere and the infrastructure falls over.


Who are the others that are down?


How do you figure? Seems very optimistic to assume that these companies are incapable of this type of failure themselves.


Hope I don't get downvoted, but I like to entertain plausible but low probability ideas because my creative mind goes there:

What if some outages like this are the US government performing security attack testing to test infrastructure of PRISM-like partners (or just anyone economically important to the US), so that when real attacks come from cyber enemy states like China, things have already been probed to see what needs to be fixed.

Kind of like national security HIIT, so your body politic is ready for war. National security is about economic health too. It's all part of the one picture.

</creative thinking>


Not a impossibility but there's nothing to support this. Unsupported accusations are just conspiracy theories without a cult following behind them [yet].




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: