Agile Methods are Disruptive

Now I might actually start a fight with that title, but at least it got you here. I recently just finished reading the "Inventor's Dilemma" by Clayton Christensen. It's an amazing book that focuses on the difference between disruptive technologies and sustaining technologies. This distinction is important because it changes how your organization should attempt to develop and manage these types of technologies. Within the book he discusses some of the social forces at play that make developing disruptive technologies different from sustaining ones. I believe these same dynamics are at work with adopting agile development processes.

In order to understand this you have to understand the difference between what is disruptive and what is sustaining. Sustaining technologies are complimentary to existing technologies your customers use. Sustaining technologies will be easily accepted by your existing customer base. That property makes it very easy for you to develop within product line using your existing resources and process. Disruptive technologies are the opposite. They most likely won't be accepted by your customers at first and your organization will find it extremely hard to develop them in house. Christensen's argument is that disruptive technologies only work if they are spun off into a separate organization independent from your own. They must be quarantined away from the core or else your organization will kill them at all costs.

It's a fascinating phenomenon, but at the end of the book he gives hints as to why this is so. He starts talking about the three components that make up any organization: resources, processes, and values. This is where the book starts to sound more like a sociology study on business structure. He points out that resources are portable: people, assets, software, money, etc. They can be fired, hired, moved, procured, sold, bought, etc. They don't care where they are, and they can be applied anywhere you want. This the key difference between the other two because processes and values are NOT portable. It's much harder to move processes and values to between organizations. These are very important properties because without them the organization would disappear. Remember resources move in and out of an organization but it's these processes and values that stay behind and keep it alive. This also means that changing these processes and values is next to impossible. Why? Well because they are what define the organization if you change them then the organization dies, and a new one comes into being.

This got me thinking about agile environments and how they try to affect the later two components. Agile is a process, but mainly targeted at software development. And with it comes a certain set of values you must adopt or else you're going to find it very hard to follow the process. If you don't accept the idea that high levels of communication and collaboration are much better than comprehensive documentation then you'll find agile methods very hostile.

The other day a group of developers were all talking about agile development. Eventually we drifted towards the difficultly we were all having trying to convert an organization into an agile one. Almost all of us felt like it was somewhere between limited success to impossible. It finally hit me. We're trying to do the exact thing Christensen says you can't do. Change a company's process AND values! Not so much a company, but a development team which like a company has processes and values.

We all had anecdotal evidence of a lack of success in doing so. In fact of all the organizations I know that have successfully adopted agile development were green field starts, or they were able to convert everyone in their organization to it all at once. This normally meant small shops or isolated teams. And, in fact my only successful attempt was when I was on a team that was separate from the rest of the development organization that had virtually no dependencies on non-agile groups. My other attempts were very large groups, or groups that had lots of dependencies on other non-agile groups. No surprise those all failed to reap the benefits of agile development.

Why do groups with dependencies fail? It seemed obvious at the time, but I think another idea Christiansen mentions is to blame. And, that is Resource Dependence Theory. In the book Christiansen explains a theory of management that says something like the following. Employees (e.g. CEO, the board, VPs, managers, etc) aren't in control of the decisions in a company. The customers they serve are. I would actually add to this that it's not just the customers, but suppliers, partners, etc. For example, think of the car dealers for GM and how they have crippled GM's ability to cut costs by closing dealerships through the years. The CEO could do nothing to change this until they almost went bankrupt. That's how entrenched customers can make an organization.

This same idea of resource dependence comes into play with agile teams. If you have a lot of dependency between you another team then you will find it increasingly difficult to be agile yourself. Why? Just like company heads aren't in charge of their companies you aren't in charge of your group.

Just like when you develop a new disruptive technology you can't stop at the product development teams. You have to break everything off: sales, marketing, etc. Agile is much the same as it requires you to adopt a new set of processes and values in how you build your products.

Agile processes' values are in conflict with traditional development values. For one, agile can't predict what will be in the release and when it will be done. It can only predict one or the other, but not both. Traditional development thinks it can do both, but really it can't. However, this points out a key difference in values between the two. Traditional shops like both pieces, or they like the idea that they might know both pieces. Traditional development believes you can reliably predict outcomes and plan for long term success. Agile shops believe predictions are unreliable and reject the idea of long term planning in terms of project management. Traditional shops call for loads of documentation and check points. Agile groups reject documentation as wasteful and unproductive in favor of collaboration and high level of communication. These differences in values makes traditional shops find reasons to reject agile, or at best neuter it into submission.

I remember one such conversation that illuminated how ferocious this difference in values can be. We were in a meeting trying to explain agile development practices work. It turned into a huge argument with one of the vice presidents about why agile practices would never work for product development. He fixated on the lack of "robust" agile procedures. He claimed that they might work in the consulting world, but don't apply to product development because product development need more "robust" procedures. His main evidence was this problem of predictability. Agile development could not predict both features and delivery schedule. The VP insisted agile development would not deliver quality software that a product company needs. In many ways it sounded exactly like an existing customer might come after a disruptive technology. Disruptive technologies typically don't have the same level of performance the entrenched technology does at first. The VP was making an argument over quality much in the same way Christensen says existing customers will make over disruptive technologies. Disruptive technologies, at first, don't perform as well, scale as well, or meet the high end needs of the existing customer base. So they usually find their foothold in smaller markets with lower margins. The incumbents are often all to happy to let the disruptive technology provider enter them because they don't make much money from these lower level markets anyway. I think it's interesting how agile development found it's foothold in consulting and small teams first.

If you want to succeed with agile development in your organization don't try and change your existing development process into an agile one. It won't work or if you are able to do it it will be a very frustrating and tiresome process. Better is to start an agile organization. Separate them from other non-agile groups, give them autonomy to affect the processes outside your development organization. Don't see agile as just something your engineers do.