Getting Out of an Operational Rut

Problem

One of my teams was bogged down by daily incidents that had a production incident every day, and it consumed a lot of their time. These incidents became taxing for the team, and it caused them to feel pretty demoralized.

Think of it as a leaky boat. All boats leak to some degree. But there is a big difference between a hole above or below the waterline.

If the boat has a hole leaking below the waterline, you are probably struggling to keep afloat. If you leave that leaking hole unattended, the boat will start to sink, and you have to expend a lot of effort actively pumping water out of the boat to just stay afloat. This can be very demoralizing on the team.

If the hole is above the waterline, it will sometimes let water get into the boat. Although this is inconvenient, it is seldom life-threatening. You have just to dry it up, and it is mostly just annoying.

The daily incidents my team dealt with were similar to a leaky boat with a hole below the waterline. As a result, it was quite challenging for them to perform at a high level.

Of course, having production incidents is a part of owning a production system. You're never going to have zero incidents. However, the severity of these incidents depends on where the hole in the boat is. If the hole is below the waterline, how do you get back to a point where you can perform at a high level?

Actions taken

To solve this problem, I decided to stop all new product development and make a list of things we need to do to fix the hole below the water.

While we were sinking, we could not suggest brand new product development or even add new features to any existing products. If we had, the team would have revolted because they would have said we're only even staying afloat. They would have been fighting to survive, not working to create new value.

With that, instead of working on new products, we spent two months getting things to a state where we didn't feel we were sinking. Interestingly, the act of fixing the root of these problems allowed us to gel better as a team.

After these two months, we started feeling better about the system. Despite the new sense of courage and synergy as a team, we found ourselves slightly bored. Ironically, even with our fresh technical courage to tackle new things, we did not have any ideas for what to do next. We weren't in the thinking mode of value creation. We were still very much in the mode of value protection.

We asked ourselves, "How do we get into value creation mode?"

To answer this question, we discussed a few things and came up with some ideas we liked. When it came time to plan and execute, we realized that our ideas were not ambitious enough. They were too grounded, so we decided to make them a little bit more ambitious.

So we added a bit of crazy to our ideas, causing the team to become apprehensive about accomplishing them. I reminded them that we were just dealing with computers and, through the code we write, it'll do whatever we tell it to. With that, we set up a hack week for the team, and they took a bunch of those crazy ideas and developed them into proof of concepts.

We showed our proof of concept to the product and the solution teams. Some of them found immediate traction and we knew we were onto something. We took one of those ideas and developed it into an MVP. Then we tested it with some customers and iterated until we found good product-market fit.

At this point, we went all in. We brought in resources from another team to increase development velocity and released a brand new product, after two quarters, at our annual user conference.

Within nine months to a year, we went from a problematic leaky boat situation all the way to a brand new product released that created value and helped us restart our fire.

Lessons learned

Simply put, one of the big takeaways is that you cannot innovate when your boat is sinking. The goal is to fix those leaky holes. Otherwise, everyone is distracted, and nobody can think creatively.

Once you remove the distractions of the leaky boat, you can go into value creation mode instead of sitting in value protection mode. You can raise the bar and dream big.

We probably wouldn't have landed on the product we did if we had acted safely. We would not have delivered something that the market was looking for, and it would not have been something that genuinely excited us.

In essence, when you do go into value creation mode, dream big. Dream big on how to iteratively innovate and go to market.

I remembered the moment when we realized we have this leaky boat, and I realized we should stop everything right now. We should do nothing else but fix this thing. 

That was like a hard fork in the road for the product, but we said we're not going to speak to product for six weeks. We don't exist until we are in a better place. Even though it was a hard line to draw, if we did not do that and kept working on product, we weren't going to have a team left to work in the first place.

However, once we made that hard decision, and we made the fixes, we were able to come up with ideas. Even though the ideas weren't big enough, we had a foundation that allowed us to dream big.

With that, we raised the bar, and we aimed high. Our courage and aspirations allowed us to be deliberate with our actions and create something that allowed us to feel incredibly proud.