Using Critical Chain for software projects

Earlier, I had posted how the concept theory of constraints (ToC) can be used for effective offshore communication. Those familiar with ToC might wonder why not apply the Critical Chain methodology, which is a project methodology offshoot of the ToC, for the entire software project itself. I've seen this implemented in some companies fairly rigorously for software projects and think that it's a great idea, but with a few caveats.

Critical Chain Methodology
First, a quick primer on CCM. The project methodology that most of us are familiar with, thanks to Microsoft Project, is called critical path - where the time needed to complete a project is based on the longest sequence of events. The focus in this methodology is on tasks and their schedule. Any unknowns are typically factored into each task by means of a slight buffer. For example, if a task would take 2 days to complete, it's typically calculated as 3 days, adding 1 day for potential delays or distractions.

On the other hand critical chain methodology focuses on the resources involved in the project and on the longest chain of resource constraints that would be need to complete a project. The concept is quite good and has been reported to help projects get completed 10% - 20% before the estimated date.

The primary difference between the two is that with critical chain you pool the buffer time for each task rather than include it with the task itself. So, essentially, the schedule is created based on the 'pure' time needed to complete a task (called 'focus' time) and not on the 'buffered' time. All the buffers are then pooled into a 'project' buffer (at the end of the project) or a 'feeding' buffer (at the end of each chain of tasks). Thus, you don't essentially say you'll complete the project at a certain date, but rather within a range, where the end date equals the date you'd calculate using critical path.

The catch - Consultants and Offshore teams
This all sounds good. So, what's the catch? If your project is completely onshore and does not use any consultants, then there is almost no catch. The concept works very well once people are trained to understand the methodology -which is crucial to the success of the project. The methodology itself has its roots in manufacturing, where these assumptions are mostly true.

The problem comes when you introduce either consultants or offshore teams, which is mostly true in case of software projects. As I mentioned earlier, CCM is a resource-based methodology and hence, has a strong emphasis on the resources and their linkages. When offshore teams are included in the mix, the problem arises due to time difference. Since the project plan itself is time-based and on dependencies between resources, it is difficult to capture the dependencies incorporating the time difference as well. While this is not impossible to capture, it just needs a bit more planning than usual to include this. This is in some sense true even for Critical Path plans, but becomes a bit more apparent here due to the emphasis on resources.

The second issue is when you include consultants. In CCM, resources would constantly shift tasks based on what is more important at a given point in time (as the critical chain would keep moving depending on the constraints at any given day). This is fine if all the resources are part of the same company.

However, let's say you have one consultant and one employee working on two dependent tasks. If the employee decides to move to another critical task for a day or two and not worry about the task that the consultant is dependent upon, what should the consultant do for the two days? Therein lies the issue. Ideally, you have to pay the consultant for sitting around because he's just dependent on another resource, but it may be hard to digest for the sponsor and the knee jerk reaction might be to ask them to do something 'useful', which may in turn, affect some other activity. This becomes more complex when multiple resources and projects are involved.

The other issue on the consultant's side is to create an invoice for the project. Usually, project plans are created where resources start low, ramp up, and finally ramp down close to launch, so as to minimize the overall cost of the project. This is fine as long as the project length is fixed and you know roughly when to on-board or roll-off a resource. However, with critical chain, the project timeline is a range and is not fixed. So, you'd have to create a range in your invoice potentially with an 'early completion' bonus or something similar, because you won't be sure exactly when to on-board or roll-off resources. This becomes messy when some high-value resources are in demand on other projects that don't use Critical Chain.

So, the bottom line is that Critical Chain is a great methodology and one that works really well. However, when brought into a software project, the three constraints - 1) consultants 2) offshore team, and 3) training (for both employees and consultants) must be considered before implementing it. Otherwise, you'll have a lot of headaches.

Software (and life) lessons we can learn from Pixar

Watched Pixar's latest film "UP" yesterday. Over the last years, Pixar has become the undisputed leader in making cartoon movies (or what we now more sophisticatedly call as animated movies). Their claim to fame is that they haven't produced a single bad film till now, which is pretty amazing in the movie industry. When you look a little deep into how they've managed to pull this off, some consistent ideals surface, which I think is very apt for the software industry, presentations, and to an extent, to our daily activities.

Pace yourself: Pixar releases films once 1-2 years, fairly lengthy compared to other movie studios (while somewhat comparable to Dreamworks). Pixar seems to focus on the long-term gains than short-term ones, something that almost no one does in any industry. Their conviction that a great product, even if released after a long period, can produce greater returns, has paid off time and again.

Contrarily, software giants seem to focus on the short-term gains, releasing version after version in 6-month or sometime even 3-month gaps, little realizing that people want stable products than frequent releases.
I guess we can apply the same in life - no, not getting babies every 2 years - but to essentially 'take time and smell the roses'. I remember when I was on a trek at Kilimanjaro, the guide constantly used to tell us 'pole pole' (e pronounced as ey) meaning 'slowly slowly', meaning don't walk too fast or you're going to get exhausted fast. In most cases, we don't realize this until its too late.

Having stable, timed releases has enabled Pixar to make more money with less releases and increase credibility with its viewers.

Have a Story: Pixar doesn't produce cartoons - they make stories. More than the animation wizardry, what makes the movie stick is the story that it conveys. And in almost all cases, the story is very simple, is unexpected (or at least has opposing characters), and is emotional. Without the story, a movie falls flat (something that most Indian filmmakers can learn from!). When you watch a movie, you are so engrossed in the story that you barely notice the animation - and I think that's essential to the success of Pixar.

In software, this translates into business functions. If the software is not functional, any amount of technical wizardry is not going to help an application to succeed.

Be detail-oriented, but don't show it: I heard that in their latest film Up, Pixar implemented a special algorithm that makes the 10,000 balloons that powers the house move as if they would in real life (bumping, moving, etc.). While the viewer may never even notice this nuance, having these details taken care of, somehow completes the picture more. The key here is that even if the effort that went in to do this was tremendous, they did not make that feature prominent, primarily because it's supposed to be in the background, as a prop to the story.
Similarly, in a software, one needs to take care of all the nuances (error messages, logging, connection retries, etc.) so that it in essence, is not visible to the user but is there doing its job.
Contextual user interfaces effectively achieve this - by giving you only the actions you need based on the context and not the entire set.

Don't succumb to too much technology: This is probably a negative lesson we can possibly learn from "Up". It looks like Pixar seems to have focused a little too much on the 3D than the story by keeping a lot of action sequences, which I feel diluted the film's message. It's always tempting to use the latest and greatest technology whether it's needed or not. At the end, users are going to be happy not because it has AJAX, but because it meets all their needs, in a friendly way.

I am sure if I dig deeper, there will be a few more prominent lessons, but these are the ones I consider to be the most imprtant. In any case, this is a blog and not an essay - so I'll stop here!

Effective Offshore Communication

A while back, my sister-in-law, who was doing her MBA at that time, recommended me to read a book called The Goal by Eliyahu Goldratt. I started reading it just before bedtime and ended up finishing it well into the night in around 4-5 hours straight. It's a fascinating book and written like a story (more on that in another blog later). Anyhow, in the book, the author proposes the idea of 'Theory of Constraints', which in roughly put, says that you have to identify the bottlenecks or constrants in your system and then address them first so that your throughput (or in some sense productivity) increases. The author mainly talks about manufacturing assembly-line style systems (this was written circa 1980).
However, I found that most of the concepts applied very well to the software industry, especially when it comes to onshore-offshore communication. I happened to be managing an offshore team in China at that time and thought I'd apply some of the concepts and ended up finding that it worked extremely well. Inspired, I wrote a white paper (or what I thought was an awesome white paper), only to find out that my company had a long process to actually publish it. I thought I'll share it with the world through my website as an alternative. You can read the full white paper Effective Offshore Communication using Theory of Constraints in my website.
Meanwhile, here's a small gist to get you interested.
In a team that's comprised of onshore and offshore resources, the biggest bottleneck is distance. This bigger bottleneck in turn leads to others such as turnaround time (due to time difference) and cultural differences (leading to misunderstanding). Per ToC, this can be handled using the following techniques.

1. Move the bottleneck to the top: As communication is the primary issue, make sure that is addressed first. For example, if you have a junior resource who knows Chinese, first get him to understand what needs to be done onshore and let him be the liaison to talk to the offshore folks than letting the Technical Lead trying to convey the same in English. You are not there to propagate English but rather to get the work done.
   
2. Increase capacity: Offshore resources are cheaper. If you can, hire a good technical writer on your offshore team (even an English Litt. would do) and get them to do all documentation and additional testing than spend scarcer resources onshore. You can also try to bring in a few junior resources offshore to get some of the daily tasks out of the hands of the more senior developers. The slight increase in cost will be well compensated by the increased productivity from your core team
   
3. Provide smaller chunks: Don't give a month's work to the offshore team and expect that everything will be great when you get it back. It's better to have weekly deliverables so that you can monitor the progress more effectively. More importantly, you can also confirm that what you told them as business requirements are what they understood. The tricky piece here is that too small a chunk would be counter-productive due to the time difference. Asking them to deliver something every day or every other day would make them spend more time in packaging the product than get any good work done. A good balance would be to ask the offshore team to get them to show progress over a video conference once a week and a deliverable every other week.
4. Quality control before a bottleneck: This is probably the biggest timesaver. A day or two before getting a build, get your offshore team to give you a demo over a video conference such as Webex. This way you can do a quick smoke test and visually confirm if what you will be getting is what you're expecting. If you run your tests AFTER you get the build, you might end up losing 3-4 days in sending the build back, getting them to fix, getting it back, and testing it again.
5. Redistribute and parallelize work: I've seen some managers or even tech leads push everything offshore. Just because they are the developers mean they should develop everything. Sometimes it may make sense to do a few prototypes onshore or even a small chunk of work onshore, especially ones that connect to the client systems or need to be validated with the clients frequently.
6. Prioritize work: While this sounds obvious, it's quite important to make sure that the offshore team focuses on the most important work first. It also helps to have a few low-priority items as fillers - stuff that the developers can do to take a break or clear their head.

Hope you get time to read the full white paper. If you do, please do post a comment on your thoughts and your experiences either here or in my site.

Process of candidate selection

I recently attended an interview for a Technical Lead position. The interviewer (a very bright intellectual, especially if he's reading this blog!) asked me a series of questions around software development in general and specific questions around theory behind exceptions and threads. While I could answer most of the standard questions along with what I've been doing, I failed to impress him on the theoretical questions. Not that I did not know the answer or was not familiar with the concepts, but that it has been a while since I've read about them or used them. As a technical lead, combined with offshore development, the focus nowadays I believe has shifted more to design, process, and governance as opposed to hands-on development (which is where the offshore piece comes in). This made me wonder how we select people for a particular task.

It's interesting how your career or future depends on how you perform at a single instance regardless of how better or worse you've performed in the past. Our entire system seems to have been built around this concept - be it education, sports, politics, or job. Exams, match formats, and interviews all favor how alert you are on a given moment in time with respect ot the opposition compared to how you've been doing over time.

While I can understand the current rationale behind this (it's not feasible to keep track of everyone at all times to see how they do, especially those who you don't know, such as potential candidates for job), I think it's quite out-dated and incorrect.

Most US universities at least at the collegiate level use a combination of your final exam marks along with how you've done in all your assignments and interim exams, with varied weights. I fail to see why the same concept cannot be applied to other areas. Let me take the specific scenarios I mentioned above and provide some alternate examples.

Sports

This is probably the more controversial of the lot since it's not just the process, but also the thrill provided by the finish. Nevertheless, instead of providing a point win-lose scenario, why not have a weighted system, where, say, the number of wins up to finals accounts for, say, 40% of the final game, while the remaining 60% goes to the final game itself? Such a mechanism will provide a fair ground to a team that, say, has been winning for the whole tournament but ends up losing in the last match. In such a case, their past record will provide an additional boost for them to keep up the momentum to win the final piece.

Game Shows

Game shows are no different from sports in that they are a type of a sport, with similar format. A recent example which added to my thoughts is the latest season of the Biggest Loser, a game show in USA where the person who loses the most weight by the end of the season gets a grand prize. 

In the show, an ex-model Tara, won all the rounds handily including all additional challenges, but lost the final weigh-in by 5 pounds. I thought it was a bit unfair and sends a relatively wrong message, since her opponent seemed to have lost a little too much weight! With a weighted system, her past victories would have contributed to her final score and would have helped her, especially since she lost by such a small margin.

Politics

Another controversial topic I am sure, and definitely harder to implement, but think about this. Why not have a system where, in an election, the winner is selected not just by votes on how they did in the election process but also on how they performed during the previous tenure? While I am no political expert, I can think of some fairly reasonable metrics to measure the performance during a tenure such as the attendance of the politicians in the parliament procedures, the percentage of growth in development in various sectors, relative stability of the country - both economically and militarily, potentially calculated by level of threat, people confidence, GDP growth, and inflation rate. I am sure more metrics can be added.

Such a system would ensure that only a candidate, either ruling or opposition, who has really worked for his/her people is selected for another term, without being swayed by last minute sympathy waves or other external criteria. It also tends to be more meritocratic and will tend to keep the politicians on bay WHILE they are in power and not after.

Job

Finally, to something that made me write this blog. I think the evolution of Internet has provided some excellent opportunities to change the way we select candidates. Instead of deciding the worth of a candidate in a 30 minute interview, why not combine the responses with other factors such as what they have done outside their core skillset? This could include how active they are in social networking sites, what sites they are active on, what they have contributed, and so on. This is similar to valuing PhD candidates based on the articles they have written on conferences they've attended, for example. 

Such a weighted system would not only help a team lead pick the candidate with the right technical background (based on the interview) but also with the right passion, interest, and intellect, given that you need more than just technical smarts to execute a project.

Is this a gripe because the interviewer didn't consider my candidate worthy? No. I honestly believe that human nature is more complicated than a binary solution and that a weighted system that evaluates a person over time (what we would call experience) is a more accurate measure.