From firefighting to a rhythm

Some teams you join are in a constant state of emergency. The sprint plan is out of date by Tuesday, and engineers keep half an eye on the support channel all day. People are tired, and they've stopped expecting it to change.

It's tempting to see this as a capacity problem and wish for two more people. Two more people would help, but they'd be interrupted too. Interruptions arrive from everywhere, all at once, and land on whoever happens to be looking. What helps more is giving them a place to go.

Why it keeps happening

Firefighting also keeps itself going. Every incident pulls people off planned work, including the work that would stop the next incident, so the causes stay. And when any engineer can be interrupted at any time, the whole team works in short, broken stretches, and the longer work that would fix things for good never gets the hours it needs.

Breaking the loop takes a few simple structures. None of them are new. What matters is putting them in place on purpose and keeping them.

A support rotation

The single most useful change is a support rotation. One person each week takes support requests and interruptions, and everyone else is protected. In the DSP Processing and DSP Reporting teams, we call that person the dev on duty. Whoever is dev on duty knows that week is for support, and the rest of the team knows their sprint work is safe.

When a team is deep in firefighting, it helps if the manager takes some of that first line too, at least at the start. With the Reportal & CUED team, before I moved to Rights & Royalties, I answered the support desk myself for a long time, and only passed a request on when it needed an engineer. It also showed me where the requests were coming from.

In the DSP teams, the support rotation did more than shield the rest of the team. Support requests come from all over the system, one after another, so the dev on duty ends up in parts of the codebase they would normally never touch. Engineers told me it showed them how little they knew outside their own area. People who were experts in one domain found themselves working in corners of the repo they had never needed to learn. And because the scope of support is written down in Confluence, the dev on duty doesn't get pulled into sprint tickets on top of it. Their week is for support, and only support.

Decide what counts as an incident

A team can't stop firefighting until it agrees what a fire is. I wrote an incident process for the DSP teams that answers a few questions up front:

• When does support become an incident? If it can't be solved within two hours, or within the day.
• Who decides? Anyone can raise an incident. The leads set its priority, so the person who shouts loudest doesn't set it by default.
• Who's in charge? The dev on duty is the incident commander unless they hand it over. One person leads, everyone else helps.
• What does each priority cost? A P3 is negotiated and normal support carries on. A P2 takes one person full time. A P1 takes everyone and is allowed to break the sprint.

Spelling out the cost matters as much as the labels. Once everyone knows a P1 stops the sprint, fewer things get called P1.

The real fix goes into a sprint

During an incident, the only goal is getting things stable. The permanent fix goes into planning like any other piece of work, sized and prioritised next to everything else.

Fixes made in a hurry during an incident tend to be partial, and nobody comes back to them. Fixes that go through planning get finished. Every incident ends with a follow-up ticket and a short, blameless review of what in the system allowed it and what we'll change.

Planning each sprint

A firefighting team has no idea what next week looks like. A fixed planning rhythm gives them that. With the DSP teams it runs over the end of each sprint:

• Thursday: Operations and Business go through the candidate tickets, and the proposed scope for next week is shared.
• Thursday afternoon: engineering estimates in detail, splitting or sending back anything too big or unclear.
• Friday: Operations, Engineering and Business agree the final priorities together, and cut scope where the estimates don't fit.
• Monday: the new sprint starts with the work shared out.

We used to run one-week sprints, and have since moved to two-week sprints. It made the team's working life better. Some ceremonies now happen once every two weeks instead of every week, so there are fewer meetings. People have more time to prepare their tasks, and the weekend in the middle of the sprint gives everyone a pause to step back and come back with a fresh view. With one-week sprints, every Friday was a rush of pull requests trying to get merged before the sprint closed. Our velocity per week went up too, and the team takes on more work each sprint.

If something urgent has to come into the sprint, something else comes out, and the person asking helps choose what. With the Reportal & CUED team we agreed in a retro that a blocker raised in the first half of a sprint could swap out a planned ticket, while one raised in the second half waited for the next sprint.

Where I'd start

Don't roll all of this out in your first week. A team that's exhausted doesn't need a new process document. It needs one thing to get easier. The first thing I'd suggest is a postmortem after each incident, run as a short retrospective. Then start the support rotation and a priority conversation before each sprint, and add the incident definitions once people trust that the dev on duty really does take the interruptions.

A postmortem doesn't need to be a big meeting. Half an hour with the people involved is enough, in three parts: Five Whys, lessons learned and action points. Kept blameless, that half hour makes everyone aware of four things:

• How did we end up here? The Five Whys walk back from the symptom to the cause. The first answer is usually a person or a single mistake. By the fifth, it's usually a gap in the system, like a missing check, alert or test, that let the mistake through.
• How do we find our way if it happens again? Write down what worked during the fix: where you looked first, who you needed, which dashboards and commands helped. That becomes a runbook, so the next dev on duty doesn't start from zero.
• How do we make sure it doesn't happen again? Each lesson becomes an action point with an owner, and goes into planning like any other ticket.
• How can we improve this part of the process altogether? Some incidents point past the bug to the way the team works, like how changes get reviewed or how requests reach the team. Those go to the next team retrospective, where everyone can decide what to change.

Held after every incident, postmortems build up a record of what keeps going wrong and why. That record tells you where to spend the time a calmer week frees up.