Software development is a complex process, and few organizations manage to handle it properly from the outset. It’s no surprise to anyone in the IT industry that software project failure rates still hover near the 40 percent mark even after more than 60 years of industry experience in the field.
To outsiders, though, not only is that failure rate remarkable, but so is the fact that it has come down to that still-atrocious level from a high closer to 60 percent in the late 1990s. That decrease is no accident, and many industry experts attribute it primarily to one thing: agile software development practices.
The most popular of these, hands-down, is called “Scrum.” Forrester Research has determined that more than 90 percent of organizations that identify as “agile” practitioners use Scrum as their methodology of choice. And Scrum’s appeal extends beyond software development, these days, finding a home in lean organizations in roles as diverse as developing new radio programs (at National Public Radio) or building a new hyper fuel-efficient car (at Wikispeed).
The principles behind Scrum will not be foreign to any student of business familiar with conventional lean manufacturing or development practices. By using loosely coupled iterations of rapid development, Scrum seeks to manage risk by reducing wasted effort and maximize value by delivering working software rapidly. Whenever the details of the process start to look obtuse or confusing, return to those basic precepts: they are the heart of the practice.
You can use Scrum to manage almost any sort of project.
The heart of Scrum is the sprint. A sprint is simply a predefined segment of time, during which a subset of product features (called backlog items) will be built by the development team. The most common length for a sprint is about two weeks, although it can be as short as a day or as long as a month. You’ll have to determine the appropriate window for your team and project.
The sprint is part of a loop, a continuing set of iterations of feature definition, development, and acceptance which continue until the product is complete. The goal of each sprint, however, is to have a working piece of software at the end–even if not all desirable features are in place, those that are in place should function correctly and provide value to the user.
The trick to achieving this feat lies in the definition of the backlog items. The product backlog is the master list, created, maintained, and organized in order of priority by the product owner. The owner is the customer or customer surrogate, and is the only person allowed to add or remove items from the product backlog. Each item, however, must be discrete enough to be built entirely within the sprint timeline and valuable enough to make it worth finishing. To this end, the product owner will work with the development team to determine the technical feasibility of the item as well as a cost, which corresponds to the percentage of effort the team will require to complete the item in a single sprint. Some sprints might have only one major item in them. More frequently, a collection of items, possibly unrelated, will be put on the current sprint backlog.
Often, these will be jotted down on 3×5 cards; a rule of thumb is that when an item can’t be described concisely in that amount of space, it’s probably also too large to be completed in a single sprint and must be broken into several different items.
In the same way that the product owner is the only person allowed to put items on the product backlog, only development team members are allowed to add or remove items from the sprint backlog, although they often must do so in order of the priority assigned by the product owner. This allows them to control the pace and scale of their own work. At the same time, it puts the onus of completing them entirely on the team. They must deliver what they sign up for; there is no finger-pointing allowed.
Once an item passes into the sprint backlog, the item is frozen to the outside world: the product owner may not longer tinker with it. The development team, as they dive into the implementation, may request changes be made to incorporate their improved understanding of how to feature fits together. They, or the product owner, can also request that an item be dropped mid-sprint if conditions change. Only the development team can request that items be added mid-sprint, which they may do if they are otherwise ahead of schedule.
Each day during the sprint, the team will come together in a quick, 15-minute meeting called the “Daily Scrum” or “Daily Standup.” It’s called a “standup” because no one is allowed to sit down–it’s not a coffee-and-doughnuts meeting, it’s a rapid-fire coordinating event during which each team member is required to individually answer three questions:
- What did you do yesterday?
- What will you do today?
- What obstacles are preventing you from doing it?
At the end of the sprint, the team and product owner get together for the sprint review. This is a longer meeting during which the development team demonstrates the working features created during the sprint. The product owner validates the completion of the items, and may ask questions or provide input on them and discuss how they could impact other items. Outside stakeholders may come to the sprint review, but only the product owner is allowed to sign off on completed backlog items. The sprint review is followed by a sprint retrospective, in which only the Scrum team participates. While the product was the focus of discussion at the review, the retrospective is intended to examine the team itself. It’s about discussing how the work was done rather than the items themselves. The idea is that only by continually examining and attempting to improve on the process used to build the product can the team learn to build it better and faster.
There is a shadowy figure standing behind the other parts of the Scrum team: the Scrum Master. This role is the hardest fill for dabblers, and often the most misunderstood for all Scrum practitioners. Despite being called the “master,” the role has no supervisory or administrative purpose. Instead, the master acts like a coach and mentor. His or her role is to offer advice and suggestions, to stand slightly outside the process to observe roadblocks and help participants find ways to route around them. The detachment allows a fair hearing of disputes and fair resolutions. The Scrum master can also help participants trim inefficiencies from their process and accomplish their work faster and better.
Because the role is so poorly defined, in concrete terms, many dabblers in Scrum attempt to make do without it, or to double it up with other roles, but that badly misunderstands the value and purpose of the Scrum master. Ideally, you should find someone who has been part of a Scrum team before to fill this role.
The Scrum master frequently facilitates the sprint retrospective, offering observations and asking keen, insightful, Socratic questions that can lead other participants to a better understanding of their own roles in the process.
Another process that happens intermittently during and around the sprint is called grooming. Grooming is the process of defining and honing backlog items. It is primarily the responsibility of the product owner, but the development team also participates frequently to help the owner define items in such a way that they are technically viable for development in a single sprint, and to estimate the effort required to complete them.
Finally, the tail of the sprint becomes the head at the sprint planning meeting. It is during this meeting that all the sprint backlog items are decided on from the possible list of outstanding product backlog items.
If you look hard at all the various rules and aspects of Scrum you will start to notice that they are all oriented at forcing you to face facts and deal with hard problems. The time limitations, the restrictions on who is allowed to define and deliver features, the mandatory reviews and checkpoints… they are all in place to deal with scope creep and procrastination.
Scrum does not eliminate any of these classic problems of project management, but it does force you to deal with them, collectively, in ways that more traditional project management methodologies do not.
Other than organization, Scrum has minimal overhead associated with it. It’s easy to dip your toes in and give it a shot, and the rapid iterations also make it easy to abandon if it turns out to not be the right fit for your company.
But Scrum isn’t going to go away. Increasingly, your new hires are going to be familiar with Scrum or some other school of iterative development. They’re likely to bring some pressure, and provide some institutional knowledge, to help you get your own Scrum projects off the ground!