Anatomy of a Deployment Bottleneck

What causes the deployment slow down? To answer this question, let’s look at what must occur to get an application from development to production in a large enterprise. We’ll do that with a fictional but not uncommon scenario.
A big brand national retailer needs to improve its e-commerce site by making it easy for customers to personalize their shopping experience. This is considered a business imperative as competitors have rolled out robust personalization and social media features that have led to increased sales. Ted, the director of IT, has agreed to a hard deadline of six months to go live with this new functionality.
Ted’s development team launches the project using PHP because it allows rapid application development. It will also allow the team to roll out features incrementally on a weekly or daily basis, feature by feature. This agility has a number of advantages over the traditional approach of building a whole project that launches on a “grand opening” release date:
1. Customers will see immediate improvements in their online experience
2. Upper management will quickly see positive results from the e-commerce improvement
3. Small iterative deployments are easier to address if something goes wrong
4. All features of the new application will be live within the required time frame
The first features are ready for deployment according to the plan timeline. While the development team moves on to the next set of features, Ted transitions the tested feature set to Mark, a project lead on the operations side of the IT department, so that he and his team can prepare and implement the first round of changes and be ready for the next round.
Issues crop up almost immediately.
Mark and his team attempt to keep pace with the speed of the development team, but the lack of processes leads to errors. Deployment steps are run in the wrong order. The database schema doesn’t get synched with the new version of the code. The project slows as the operations team goes back and corrects errors. In the meantime, the development team is ready to transition the next set of features for roll out.
The original goal of deploying features in incremental fashion, along with the benefits that this approach would create, begins to evaporate as the operations team gets bogged down. Ted and Mark seriously consider adapting a deployment technology originally designed for use with a different language. However, nobody on the team has the expertise to adapt any of those technologies to a PHP platform, and hiring the needed talent will take time and money. This would simply add to the delay, so Ted and Mark decide not to pursue the idea.
In order to avoid more process errors, the operations team is forced to slow down the pace of deployment further. This effort, designed to make a strategic difference to the corporation in a short space of time, has soaked up far more resources than intended without reaching a resolution. Ted dreads C-suite meetings because he will not only have to report continued delay, he will have to listen to the problems the delay is causing elsewhere in the business:
• Sales will not meet its targets, which were based in part on the ecommerce improvement going live on time
• Marketing must keep revising its campaigns, which had centered on getting maximum brand leverage from the “rolling improvements” Ted initially promised that aren’t showing up on time
• Finance points angrily to both sides of the ledger, as Ted’s department once again spends more than allocated without any revenues to justify the extra expense
On top of everything else, Ted considers the application’s future scalability requirements, which leads to more concern. If the operations team is running into problems from manual errors and lack of processes as they deploy onto a single server, how bad will things be when they need to deploy exactly the same way onto an eight-server cluster?