Wow, mad jelly their CI/CD and monitoring proceses are robust enough to trust a major rollout in December. That's a pretty badass eng culture
That being said, still some unanswered questions:
- If the issue was ipv6 configuration breaking automated cert renewals for ipv4, wouldn't they have hit this like.. a long time ago? Did I miss something here?
- Why did this take 90 minutes to resolve? I know it's like a blog post and not a real post-mortem, but some kind of timeline would have been nice to include in the post.
- Why not move to DNS provider that natively supports ipv6s?
Also I'm curious if it's worth the overhead to have a dedicated domain for scripts/packages? Do other folks do this? (excluding third-parties like package repositories).
>- If the issue was ipv6 configuration breaking automated cert renewals for ipv4, wouldn't they have hit this like.. a long time ago? Did I miss something her
AIUI, they switched to their current setup 90 days prior to the outage. The initial cert they installed during their migration lasted 90 days. So 90 days after the migration, they had an outage.
> and learned an enormous amount about what it means to build and run a people-first company
He's talking about Auth0.
> And then it happened, I got the email. Slack didn't work. My laptop restarted and came back with accounts missing. It really, actually happened. And even though the writing seemed to be on the wall, I was still caught a bit off guard.
Talking about how that culture no longer existed -- it's fully Okta now. The Auth0 culture essentially erroded until it finally just no longer existed. This of course happened at a different pace across differnt organizations. Unfortunately, I'd say ours was probably one of the earliest to get hit.
---
As an aside, I was part of same team (also laid off), and had an especially unique viewpoint.
I joined Auth0 in 2020 -- months before the acquisition. I joined because of the culture and the amazing people I get to work with and learn from. I had a lot of fun and we built some amazing things. After about 18 months I decided to join a local startup, itching to get back to a much smaller arena, building something from the ground up. Applying all the great things I've learned in my career thus far. Long story short, as most starups go, especially during COVID, it didn't pan out.
I kept in touch with my old colleagues, now my friends, who approached me with a potential new opportunity. They did warn me things were different now, but I was excited to get to work with great people again.
Coming back it was starkly different. Gone was the magic that was once there. A lot of familiar faces were still around, but so too were a lot gone now. I can feel the beating of the Okta drums more loudly now.. there just felt a lot of separation between leadership and us dreamers at the bottom. I felt like I was just back at a corporate company now, more worried about writing OKRs and how they made my boss/division look good verses actually making our customers lives better. There was a constant dread in the air.. verses excitement.
It's strange, but in some ways its almost like a feeling of mourning. That meme about not knowing when you're in the good old days is very true. Goodbye Auth0 <3
> Whitehurst, on the other hand, has a history of strong execution across multiple industries, and built a reputation as someone who protected Red Hat's culture against attempts from within IBM to "Big Blueify" it (possibly to the detriment of his own role within IBM). Even as an interim, having him onboard is a good sign for how Unity is looking to repair its relationships with developers.
100%
I was at IBM at the time. We really hoped he would eventually take over from Ginni once she left... nope. We really could have used someone who wasn't drinking the blue koolaid. Well.. the rest is history.
All this other crap about Silver Lake being a giant POS is concerning though.
Arvind Krishna seems like a pretty cool guy though, and definitely has engineering background.
It's interesting that everybody focuses on Red Hat after acquisition but no one ever asks how IBM is doing now. Maybe it's delusional to think that Red Hat could change something of its new overlord, but perhaps the acquisition was already showing the desire to change IBM from within?
IBM is doing okay, but not great. Their fiscal peak was a decade ago, and in terms of real dollars they aren't even at pre covid numbers yet. And on what little I know of the mainframe side, IBM is still very IBM-like, except they will now occasionally concur with consultants who point out places mainframe consumers are needlessly spending money.
It just felt like more of the same. I left soon after (2020) so perhaps things are changing. Was gutted to leave, IBM was always one of those mythical places to work for me, but in the end I made the right choice. Perhaps one day I'll go back, ha who knows!
Oh man, shocked. Sadly, never met her in personally, but have always admired her work in the field and for just generally being an inspiring person. Huge loss. My sincerest condolences to her family and friends :(
