How we manage large projects

One of our recently completed projects was a relatively complex web application. Here are some interesting facts:

• 3 primary developers, 2 others involved
• 8 months
• 90 MB of data
• 41,000+ lines of code in 510+ separate files

With projects like this one, there’s always a risk that the size of the project itself will actually hinder our progress and ability to meet the client’s needs quickly and efficiently. As a project gets larger and lasts longer, the effort required to keep things organized and moving forward increases dramatically. In fact, the chances that the project will be finished at all actually goes down substantially. For projects over 6 months in duration, the average success rate (completed on time, under budget) has been shown to be below 50%. [ Agile and Iterative Development by Craig Larman, p. 71 ]

That’s not a statistic we want our clients to have to deal with.

1. Prioritize features and functions

One of the most common reasons for failed development projects is having too broad a scope. After all, the duration of a project is directly related to the scope of its features. A recent study of large projects and their implemented features showed that over 45% of the features initially requested are never used in practice, with another 35% seldom used. Shockingly, only 20% of features in initial project scopes are actually used regularly. [ Agile and Iterative Development by Craig Larman, p. 73 ] In other words, for many projects, a large portion of the time, energy and money invested in them is essentially wasted.

We combat this danger by spending time early on helping our clients analyze their needs and choose only the most important features and functions. If the project is coming in on time and under budget, there’s always room to add those extra touches from the wish list later on. But the reality is, if we don’t intelligently pick and choose, the project is on rough ground right from the start.

2. Short development cycles

Developing a large website or web application is a lot like constructing a building. You need plans before you begin if you want to obtain a satisfactory result. And if those plans are going to change significantly, you want it to happen as early on as possible. Changing the floor plan after sheetrock is on the walls would be costly and time-consuming. In the world of software development, this risk is exacerbated. The longer a project takes, the more likely things are to change along the way. And when they do, it can cause major delays.

There are a few things we can do to lower the chances of budget-breaking changes late in the development cycle. First and foremost among these is “iterative development”. In simple terms, we break the project into smaller, more manageable pieces. Instead of tackling the entire feature set from day one, we take things in bite-size chunks of 2-3 weeks in length. During each of these development cycles, we examine the most important features we’d like to implement, work with the client to make sure we’re meeting their needs, and finish by testing and reviewing the outcome of that particular piece. This lowers the risk of failure by reducing the feature scope, shortening the duration, and giving us a chance to quickly adjust to any changes as we go along.

3. Source code management

Dealing with project source code is a regular task for developers, but one that can lead to serious problems if not handled correctly. If you’ve ever collaborated with others on a single document, you’ll know that information can be lost if more than one user is editing simultaneously. The easy way around this is to restrict editing to one user at a time. For a large project however, this limitation is unacceptable. For the project I mentioned at the beginning, we often had 3 developers working with dozens of files at any given time, and regularly needing to work on the same files simultaneously. Coordinating with each other to make sure we had the latest versions and weren’t overwriting each other’s changes would have slowed progress to a crawl.

The solution? Source control management (SCM). SCM software stores source code in a simple database called a “repository” and tracks each of the changes made to the code as development progresses. Because the SCM is keeping track, it can intelligently merge changes from multiple developers within a single source code file. If something goes wrong, it can even help undo any changes made by mistake. There are several SCM tools available, but our favorite is Git. Git is a “distributed” SCM, meaning each developer has his own code repository that he makes changes to. When we’re ready, a simple command merges all our code into the master Git repository for the project. A wonderful benefit to using an SCM is that making quick changes to a project is trivial. Any one of the developers has full access to the latest copy of the project code, and any changes he makes are instantly available to all the others. An SCM is essential for any project involving more than one developer.

4. Centralized project testing

Another area that can often slow down project teams is the test environment setup. Each developer working on a project needs to be able to quickly test the code he’s writing, often in multiple browsers to ensure compatibility. The usual way to do this is to set up a local web server on every developer’s system. Aside from the time this consumes every time a developer is added to the project, it can cause further delays any time a dependency or server extension is required in the project code. This then has to be properly installed or upgraded on each developer’s machine independently. Having multiple server environments also increases the general amount of effort required to keep things running smoothly.

At Brighten Labs, we got tired of dealing with these headaches, and decided to set up a single, central server environment for testing. All the code being edited by our developers resides on this central system. By giving the server a public IP address and each project its own subdomain, a developer can even connect securely and continue working when outside the office. Any changes can be tested instantly on this subdomain address, without even being committed to our Git repository. This takes half a dozen steps out of the usual process, speeding up development and testing substantially.

—

With over 50% of larger projects (over 6 months duration) going beyond deadlines and over budget, we want to do whatever we can to make sure our clients don’t have to feel that pain. I’ve mentioned four of the major ways we’ve found to combat “large project syndrome”, but there are many other things that can help.

Do you have a story involving trouble during a large development project? Maybe you have some strategies you’ve discovered along the way? Feel free to sound off in the comments.

Post a comment:

Name:
Email:
URL:

Click here to try another image.

Type the code shown: