Qualities to look for in your Incident Commander
tl;dr We have compiled 10 attributes for success in the role of Incident Commander. This list of qualities is derived from Incident Commanders who have inspired us along the way, and we are deeply grateful and thankful for those who have set the bar high. Worth noting, these 10 qualities certainly are not comprehensive nor intended to be limiting. For instance, we’ve assumed a threshold and sufficient level of knowledge and subject matter expertise on your systems and incident process and technical fluency.
Hopefully this list provides guidance as you build out the IC functionality for your team or continue to train your IC’s.
- Ownership mentality
- Bias for swift decision making and action
- Ability to facilitate thoughtful debate across the team
- Capabality to design the plan and delegate responsibility
- Problem Solving
- Common Sense
- Accurate synthesizer
- Ability to understand problems (impact/surface area) and prioritize accordingly
- Strong communication skills
- Expertise on the different services and systems and how they interact with each other
- Involved in prior incidents
Origins of the *Incident Commander
The role of an “Incident Commander” can be traced to the “Incident Command System,” (“ICS”); a framework applied to emergency response situations. ICS actually originated back in 1968 as a means for firefighters to contain wildfires. Many engineering teams have also adopted this system for handling incidents that occur.
Role of the Incident Commander
During an incident, the Incident Commander (“IC”) plays an essential role for the team. The IC leads the charge on incident management efforts; serving as the single source of truth on what is happening and who needs to do what next.
An important distinction to keep in mind is the IC should not be doing, but rather absorbing information, making decisions, and delegating action. Ideally, the IC is looking from above at the system at play and directing the action to resolve the incident at hand.
On a team, you may have a designated IC or set of IC’s trained to serve in the role. Often the IC for a particular incident is the one who spotted the issue. Alternatively, for a more complex incident, the IC can be a pre-assigned member of the team.
The role of Incident Commander is critical and complex, and when you layer in the high stress and high pressure environment of an incident, it becomes even more intense. In light of the importance and difficulty of the role, it can be helpful to articulate the qualities that make a successful Incident Commander. That you can match the person to the role design (vs. the other way around!). In service of providing guidance to help you select and train IC’s for your team, we’ve outlined some of the qualities we find in the best Incident Commanders.
The greatest Incident Commanders have a strong ownership mindset. Said differently, the IC feels high responsibility and accountability for successfully navigating the team through incidents.
Further, the IC is reliable and can be trusted to come through, even in the depths of night when the pager goes off at 3am.
Bias for swift decision-making and action
Critical to successful incident collaboration is the ability to act thoughtfully and swiftly. To achieve this, the IC needs to have a propensity for action.
Often this manifests in the ability to make difficult decisions, on the fly, with low signal or data.
During an incident, there are often many possible directions and a team can easily find itself down a rabbit hole leading nowhere. As IC, it is important to successfully harness individual team member’s viewpoints and perspectives and engage in productive debate. By facilitating thoughtful disagreement (a diversity of high value perspectives), a team can increase its probability and velocity of identifying and mitigating an incident.
A great Incident Commander knows the difference between unproductive and productive conflict and encourages the latter.
Empathy is the ability to connect with and understand another’s perspective and thinking. During incidents, empathy is an important and, sometimes, overlooked attribute.
Incidents are stressful and often mentally taxing. Empathy yields better decision making. The ability to accurately understand the situation at play is key to incident resolution. To do so you need to understand the what and the who. An IC who has empathy for all involved (customers, team members, other stakeholders) can better understand what is happening and to whom, and therefore has a higher probability of getting to the right answer, faster.
Empathy for the customer is key. Understanding how the incident impacts your customer drives how to best serve them, what to focus on in resolution, and lead the strategy for customer communication.
Further, an IC with empathy can navigate the team through the chaos and mental burden of on-call and incident response; facilitating strong collaboration and mitigating burnout and negative team dynamics.
Design and Delegation
An IC is a leader of organizational design and incident collaboration, kind of like a grandmaster overseeing a chess game. By understanding the different players available and their respective strengths and weaknesses, the IC can assign roles responsibly and to the best and highest use.
The ability to design the set of plays and delegate responsibilities effectively is crucial to success. The right design can be broken down into the right people acting in the right ways. The challenge is being able to see the system as a whole and its individual parts at the same time. On top of that, the fact that these are complex systems only compounds the challenge of nailing the design and resulting delegation.
Look for an IC who can accurately synthesize the nature of the system and the individual forces at play.
Creative Visualization and Problem Solving
Creative problem solving is key to incident collaboration. An effective IC can connect information or systems and creatively devise an action plan. The path to resolution is often murky and hard to sort through. The capacity to visualize different scenarios can speed up and ensure successful resolution.
To (re)state the obvious, incidents are stressful. They can last hours, sometimes days. They require mental and physical stamina and the ability to push through to results no matter how long you and the team have been at it.
It takes someone with determination and resilience to keep themselves and the team motivated.
A healthy dose of practicality is necessary. As the IC, you are faced with a multitude of information. You need to be able to separate the signals from the noise to make good calls. Common sense helps you accurately weigh the different possibilities to devise not only a successful solution but one that takes into account the resources, time, and complexity of the incident at hand.
An outlandish solution, even one that would eventually lead to resolution, can come at massive cost: unnecessarily wasting resources, engineering and customer time.
A great IC does not just make the call that eventually leads to mitigation but rather makes the call that minimizes risk and waste and maximizes impact, efficiency, and thoughtful velocity.
Faced with a deluge of Slack threads, graphs, screenshots, and information, an Incident Commander needs to be able to synthesize *what is happening*. This entails sorting important from unimportant, relevant from irrelevant. Essentially wading through the noise to find the signal(s) and accurately communicate what is happening to the team so that the right people can act in the right ways.
Beyond accuracy, speed of synthesis is key. Not only does an IC need to present what is happening but they need to do so on the fly -- even a few seconds can make a difference.
Problem differentiation and prioritization
As the team works to resolve the incident, they may encounter additional problems, (related or unrelated to the incident). The ability to differentiate between an important problem and an important problem is critical. If not, the team can find themselves distracted, solving the wrong problems.
An IC needs to be able to look at these problems and work with the team to assess their magnitude, scope, and level of pain to determine which problems to go after and in what priority order. Now, 3 months from now, never? This involves an IC’s own capacity to understand problems as well as the ability to listen well to their team and their perspectives.
Experience and pattern recognition can be quite helpful in building this muscle.