Hello, my name is David Battersby. I work for a small company named Tree Snake in Scotland and I'm a Software Engineer. I have been working in the industry as a web developer for over 10 years and through those years I gained a reputation as being a good "emergency guy"; what I mean by that is colleagues will often ask for my help or a manager will often get me to join projects which are headed to or actually in a disaster state. I would like to extend thanks to Jan and Carole for putting me up to write about this. I would also like to extend thanks to Panczek, Ewan, Mhairi, Greg and Baz for their reviews. 

What I will attempt to offer you here is a methodology and some tools which I have found to be practical and effective and I will also offer you some "soft" advice on how to deal with people during this kind of a situation. This article is for everyone who works in software development (and perhaps beyond). The focus will be on the Senior Developer/Team Lead who is attempting to resolve the crisis but I will attempt to bring in perspective for Project Managers, Junior and Mid Level Developers as well as Directors.

The Crisis

So let's examine the situation, I used a comical description in the teaser but unfortunately sometimes that is not far from the truth. There are jobs on the line, people have been working very hard and despite their best efforts the whole thing has come down like a house of cards. It's pretty normal for people to cry in these situations, if you've been working long enough you will know how horrible it is when this happens; it's even worse when you don't know how to fix it.

Now you don't know any of that yet; you're sitting there at 15:30 on a Friday tying up some loose ends and sending some morale boosting memes to your team on the Slack channel. At some point the hairs on your neck stand up and you notice one of your pals on another project has their face in their hands, an equally stressed out Project Manager approaches the face holding friend and during their interaction one of them points at you. It's on.

Should I be doing this?

There's plenty of reasons why you shouldn't, for example you've just worked a full week and you might burn out or maybe you have something else you need to be doing. These are valid reasons, and I can tell you that there will be overtime involved. However there is a great amount of satisfaction involved in solving a crisis, especially when it really pushes you to your limits, and if your boss has any sense then there will be some kind of reward for it (you may negotiate in advance). It's your call and don't let anyone convince you otherwise.

I'm doing this.

Alright. Your objective as the assigned Emergency Guy (officially I suppose that would be Crisis Manager, so let's use that going forward) is to prevent the company from haemorrhaging money and losing customers which in turn can mean saving jobs.  

Before you even look at the project you must have these three things:

1. Official Control - The person who has overall authority of the project must give you absolute control of the project and the team carrying out the work. This is required so that you can work freely to get the project done without chasing people around to get authorisation to change bits technical and/or functional aspects of the product your project is currently failing to deliver. This is also useful for dealing with combative team members, if they refuse to carry out what is asked for them then they must answer to the person signing their paychecks.
2. Ensure you have access to funds - You don't want to end up sitting in your office at 3am and hitting a roadblock because you need a license for something. You do not need direct access to the company cards however the likelihood of a 3am phone call will probably result in a card being given to you (do not pay for things with your own money, your company pays less tax and getting it refunded is painful). Also, considering several of you will be in the office late, the boss will usually offer to feed the team (if not then suggest it - it's good for morale).
3. Gather ALL contacts - You need to know everyone's phone number: the team, internal stakeholders and the client. Contacting the client is a contentious topic of conversation but you must press your point; someone in this team must have a direct line to the client. Why? Because the client wants their finished product: they have probably made plans for having it and not having it is going to be a really big deal for them. If you have to expose your crisis to a client OR not deliver then, in my opinion, you must expose your crisis and work with your client to produce the best possible outcome.

If you cannot gain these things then don't even start. If you have managed to gain these things then here's what to do next: Stop.

1) Stop

"Where there is discord, may we bring harmony. Where there is error, may we bring truth." - Margaret Thatcher

From your perspective you are now in control of the project. Ensure that this information is passed on to all the team members. The first thing you need to do is get everyone to stop, there may be objections and if there are then you must take each one individually and consider it. There are very few valid reasons for people not to stop at this stage but there are a few obvious ones: scheduled meeting with the client, an essential component of the product nearing completion and other things of that nature. However, stopping is essential because before you even consider your next steps you need to take stock of what you have, what you will need and the state of your team.

From the perspective of everyone else, it might seem a bit odd that a tech lead that you've possibly never met has just come over and told you to stop. You're going hell for leather to get this project finished so why on earth should you stop? Well, this is because within the next hour the work your doing may become irrelevant. There might be huge changes afoot and you might need a break even if you do not feel like one.

I try to do the following things in order where possible and relevant.

The Team

"if you know your enemies and know yourself, you will not be imperiled in a hundred battles; if you do not know your enemies but do know yourself, you will win one and lose one; if you do not know your enemies nor yourself, you will be imperiled in every single battle." - Sun Tzu

You need to check the condition of the team, doing this personally is the best approach but you may need to delegate if the team is sufficiently large. You must be empathetic to them, they are probably feeling pretty awful. Some people will attempt to conceal their condition and you may need to consider compelling them to rest. Ask them if they are willing and able to help resolve the crisis; simply dismiss them if they aren't, do not punish those who leave but rather ensure that those who do help resolve the crises are rewarded. I personally prefer to have my team in one place, however if people wish to work from home then it's fine I find it just makes communication a bit more difficult.

After you figure out how they are you need to take stock of who they are. If you are fortunate enough to know them all this should be fairly easy, if not you may need to consult with the Project Managers or with the person directly. You need to quickly get an idea of their skills as this will direct how you approach the crises from a technical viewpoint.

You must also identify any possible problematic people; there may be someone there who you don't want to be there and they may insist on staying. It's tough but you must dismiss them too (provided you can do without their skills) because if that person becomes disruptive or creates further tension you may lose your entire team to it. You'd be better off getting 90% of the way there without them than 0% of the way there and a handful of resignations.

Finally, you must do all of the above in a calm and reassuring way whilst also moving with efficiency, so speak gently but with authority and be concise.

The Client

Hopefully you will have met the client, but if not then you should make contact with them. At this point it is a good idea to explain the situation and that you're trying to resolve it, although this may initially be met with negativity it will be better in the long run. Ask the client about their availability for contact and if anyone else at their end is contactable. Most of the clients I have been in this situation with care about their business enough to make themselves available.

Technical Details

You've done the rounds with the humans, you need to now figure out what they are trying to do and what they are trying to do that with. At this point you need to lock yourself in the room with the project manager(s) and get as short an explanation of what it is they are trying to do as you can. If there is a Functional Specification document and it is still valid then this can also be very useful, so you may need to absorb this. Anything that is a known-unknown you should get on to the phone with the client about, they generally prefer to receive phone calls when they are awake.

As you collate the tasks you should prioritise them and highlight any which stand above the others in importance or risk - these ones should be taken by the senior developers (which may be yourself at this stage) and tackled first whilst you are freshest. For example in an eCommerce project the payment provider modules will rank first and be taken first and the terms and conditions text formatting that nobody looks at can be done at 4am when you're starting to hallucinate.

Also very important at this stage is to track down every license key, find every server and login (or cloud logins) you're going to need. This doesn't need to be completed before the next stage but should be started and can be delegated.    

Deadlines

If relevant, find out when the client requires each part of the project's product. Obviously if you are delivering a complete eCommerce solution then they may simply say "everything". What you're trying to achieve here is figuring out if there are any parts which can be built and released after the deadline or could be temporarily simplified and then upgraded later. I've found it's best to have this conversation with the client directly.

When you have this information, you start to look at the next stage of the process.

2) Stage

Plans are useless, but planning is indispensable. - Dwight Eisenhower

During this part of the process you will be planning the tasks to complete the project from the information you gathered in stage 1. This is where the who, what and when is decided. Ideally you want the whole team in one room for this, however for very large teams or teams with mostly remote members you may need to coordinate this differently. I have found that people in a room results in a quicker meeting and higher morale where as a conference call causes irritation and takes a long time and for that reason it may make more sense to limit the decision making to a smaller group.

There are two important points to make here:

Make sure things are done the way you want them

There's not a lot of time to discuss alternatives, the caveat here is if someone strongly believes that your idea will not work in that case hear them out. If someone thinks they have a better idea than yours but you believe your idea will work then you must remind them that you have authority. Obviously, under normal circumstances ideas must be engaged and remind them of that, however this is a crisis.

Technical: Code readability trumps everything right now

Ensure that everyone gets this message. There might be super efficient ways to write something or perhaps shorter ways but right now you need to ensure that the least experienced member of the team can read something written by the most senior person. Exceptions are obviously made here where performance is an essential part of the component but it is important to balance that with being able to actually deliver the project.

From the perspective of others here it may appear that the Crisis Manager is being overly assertive or dismissive of other members of the team. It is easy in these situations to feel like you're being shoved aside but try to recognise that the Crisis Manager may simply be working efficiently.

I make two recommendations for planning here: Agile Boards and Eisenhower Matrices.

Agile Board

There's a very good chance you already know what this is and a full explanation of Agile Methodology is beyond the scope of this article. A good primer and tool can be found here: Agile and Scrum.

I personally prefer to use a physical board but it's just preference. Online tools are also much better if your team is distributed.

Eisenhower Matrix

An Eisenhower Matrix is a simple and extremely effective tool for personal organisation. It can be used for teams but I find it works best for organising the individual. Take a small whiteboard (or chalkboard, or even simply use post-its and paper) and divide it into quarters. The top left is the DO NOW box, these are things you must do immediately, the top right box is SCHEDULE and these are things you must do but can (or must) do later, the bottom left box is DELEGATE and in here goes tasks which you could send to others, the bottom right box is DON'T DO and in here goes anything you don't need to do but are aware of (from experience, rename this "DO LATER" if other people can see it as they sometimes get upset if their request ends up here).

The Eisenhower Matrix is active all the time and you must keep it updated.

Here's a good how-to on the Eisenhower Matrix

So, now you have your plan and your team is briefed it's time to Start.

Start

There's actually not much to say about this part - this is what you do most of the time. You have your work and you go through it. You make sure you are open and approachable to your colleagues and it is prudent to seek regular updates from the other people on the project. How often you seek updates can depend on how large or small the components are, however on your agile board you will have an estimate of the complexity of a task so you can guess how long you should wait before you need to start asking questions.

To your team provide encouragement and periodically check if they need anything: Pizza? Coffee? A Hug? Also; you should continue to keep an eye on the condition of your team members. Again some may need compelled to rest, others may be apprehensive about taking a break unless you ask if they need one.

With regards to the plan, you should consider it a "living document" and if you find it requiring tweaking then go ahead and tweak it and ensure that all affected team members are aware of the change.

Delivery

As the deadline approaches you will have a good idea of where you are with everything. Sometimes everything works out great and everything is ready, other times you still need to put more work in. Either way by now the project will be a lot further along than it was before and your client is aware of the situation and aware that you've gone the whole nine yards to get it over the line.

The first thing to do is update all the internal stakeholders on the situation. If it's good news then they will be overjoyed, if it's not quite finished they will understand and will need to prepare to ameliorate the situation with the client.

The next part will be a call with the client, if it's gone great then everything will be fine. If it's still in progress then provided communication has been good I've found most clients will be more interested in handling the situation than being punitive.

Lastly it's time to talk to the team. Obviously, first thing is to thank and praise the members of the team who went to extra mile to help dig the project out of the mud and be sure to highlight their efforts to management in the company. You will have to provide an update from the call with the client and perhaps a more extensive update if it's Monday and team members are returning to the project.

At this point either the project (or the part in crisis) is done and you can catch a break, or it's time to crack on and get it finished in which case you have a plan so keep going.

Post-mortem

This article would not be complete without a mention of what happens when the crisis event is over. It is tremendously important that you insist upon and contribute to a fact finding (not blame giving) exercise following the crisis so that it may be avoided in future. How this should be carried out is beyond the scope of this article however there are many great articles on this subject. Again I repeat, approach with empathy because this is a jarring enough experience already.

Closing Remarks

This article is quite lengthy and if you have made it this far then thank you! This is quite a broad subject and obviously not everything can fit into one article but hopefully I've provided enough of a framework for you to be able to tackle (or for someone not in the lead role, understand) a crisis in a software project. Another thing to note is that these are techniques which I have found to be effective for me when I am the lead, you may need to alter the framework to best match your personality, approach, company and client. No two projects or crises will be exactly the same so always approach it with an open mind.