On June 18, 2024, I was an SRE at a major enterprise software company when a ransomware gang encrypted our systems. What followed was the most intense month of my career - 16 to 18 hour days running a 24/7 war room, rebuilding servers as fast as humanly possible, and learning lessons about teamwork that no training could ever teach.
This isn't a technical post-mortem. This is about what I learned about teams, leadership, and what happens when thousands of customers are counting on you to bring their operations back online.
What the World Saw
The attack hit on June 18, 2024. The next day, during initial recovery efforts, we were hit again. Our customers were forced to revert to pen and paper. Operations stopped. The industry estimated losses in the billions.
By early July, services were restored. From the outside, it looked like a two-week outage. From the inside, the recovery effort continued for weeks after that.
The First Days: Chaos
I won't sugarcoat it - the first days were rough. Not because people weren't working hard. Everyone was working harder than they'd ever worked. The problem was that we were pulling in different directions.
When you're in crisis mode, everyone has ideas. Everyone has theories about what to prioritize, what to rebuild first, how to approach the problem. In normal times, that diversity of thought is valuable. In a crisis, it can paralyze you.
We had competing theories about how to handle the situation. Smart people, all with valid perspectives, advocating for different approaches. Meetings ran long. Decisions got revisited. Progress was slower than it needed to be.
The spreadsheets were a perfect example. Every leader had ideas for tracking the recovery, so someone created a spreadsheet. Then another leader needed different information, so we had two spreadsheets. Then three. Eventually we realized the sprawl was causing confusion, so we combined them into one master sheet. But then too many people were updating it at once and it kept crashing. So they locked it down so only a few people could edit. And suddenly nobody could update their status and everything ground to a halt.
That's what uncoordinated crisis response looks like. Good intentions, smart people, total gridlock.
We were a crew trying to row a boat with everyone pulling their oar in a different direction.
The Turning Point
Then Sean Fretenborough stepped up.
Sean is a 24-year veteran of enterprise IT. He'd seen crises before. But more importantly, he understood something critical: in a crisis, a good decision made quickly beats a perfect decision made slowly.
He didn't claim to have all the answers. What he did was establish a clear direction and get everyone aligned. He made the calls that needed to be made. When people had concerns, he listened - briefly - then decided. And once a decision was made, it was made. We moved forward.
The shift was immediate. Suddenly, we weren't debating. We were executing. The war room transformed from a place of competing theories to a well-coordinated rebuild operation.
What 16-Hour Days Teach You
When you're running a war room for 16-18 hours a day, week after week, you learn things about yourself and your team that you can't learn any other way.
1. A team rowing in the same direction is unstoppable
Once we aligned, servers started coming back online faster than anyone expected. Not because we suddenly got smarter or worked harder - we were already working as hard as possible. We just stopped wasting energy on friction. Every hour of effort went directly toward recovery.
2. Leadership isn't about having all the answers
Sean didn't know exactly how everything would play out. Nobody did. But he made decisions with incomplete information and adjusted as we learned more. That's what leadership looks like in a crisis. Waiting for certainty means waiting too long.
3. You find out who people really are
Crisis strips away everything superficial. Some people who seemed like superstars crumbled under pressure. Others who had been quiet contributors became the backbone of the operation. You can't fake it when you're exhausted and the stakes are real.
4. Take care of your people
Nobody can sustain 18-hour days indefinitely. We learned to rotate people out, force breaks, and watch for signs of burnout. A rested engineer working 12 hours is more valuable than a burnt-out engineer making mistakes at hour 17.
5. Communication is everything
We had to keep leadership informed, coordinate across teams, and manage expectations - all while actually doing the work. The teams that communicated well recovered faster. The ones that went heads-down without updates created confusion and duplicate effort.
The Lesson I'll Carry Forever
The biggest lesson from that month wasn't technical. It wasn't about backup strategies or incident response procedures, though those matter.
It was this: a team that rows in the same direction can accomplish almost anything.
Before alignment, we had talented individuals working hard but canceling each other out. After alignment, we had a coordinated force that rebuilt an enterprise platform in record time.
The difference wasn't skill. It wasn't resources. It was direction.
What This Means for Your Team
You probably won't face a ransomware attack of this scale. But you will face crises. Systems will go down. Deadlines will loom. Pressure will mount.
When that happens, remember:
- Get aligned first. Speed comes from coordination, not just effort.
- Someone needs to make decisions. If that's you, make them. If it's not, support the person who does.
- Debate before the crisis, execute during it.
- Take care of your team. They're the only thing that will get you through.
That month was the hardest of my career. It was also the most formative. I learned more about what it means to be part of a team - a real team, under real pressure - than I had in my entire career before that.
I hope you never have to learn these lessons the way I did. But if you do find yourself in a crisis, remember: get everyone rowing in the same direction. Everything else follows from that.
- Seth Black, Co-Founder, The SRE Project