What your grandma knew but you don't - The power of stories

Stories are an incredible form of communication and probably the most powerful. Numerous studies have re-affirmed the fact that our brain can understand visuals better than words. Stories are unique in that they are words portrayed visually, thereby bridging the gap between the two.

Since our brain has to process numerous things every second of the day and it is impossible and inefficient to record everything verbatim, it has chosen to record things in an abstract manner such that the gist is preserved without the entire content. Simple examples such as Gestaldt images as you see below are examples that our brain is excellent in forming the whole using incomplete parts.

 Can you see the dog in the picture?

Obviously, such condensation of information is not without its disadvantages. The first one is misinterpretation which occurs when the brain tries to create the reality from its gist but does so in a way that is different from the original reality. An example of this is the classic game that shows how rumors are created. The second and more subtle issue is that of creating a non-existent reality. The power of stories is so strong that when the brain is given a bunch of data points that do not necessarily fit together, it still tries to make up a story that makes it fit somehow, kind of like pushing a square peg in a round hole. This has been demonstrated by neuroscientists like Oliver Sacks in his book "The man who mistook his wife for a hat" and V S Ramachandran in his book "Phantoms in the Brain". Even so, such issues are overlooked by the brain in favor of the efficiency it gains by condensing information as stories.

Maybe realizing this tremendous power of stories, our ancestors across the world have chosen fables, parables, stories, dramas, and epics as the means to capture the wisdom gathered over the years in lieu of raw facts - or maybe it was the most natural form of information storage. Over the last few decades however, we seem to have ignored this idea of storing information within stories in favour of storing real facts. With increasingly cheaper storage mediums, it is probably logical for us to conclude that we should be storing raw data as opposed to processed information to avoid the fallacies of condensed information. However, we are overlooking the fact that even though the raw data is stored as is, our brain still processes information in a more abstract story format. Thus every person who reads the raw data has to create a story within his brain to make sense, which can still lead to misinterpretation. Moreover, such storage of raw data is typically done without the context in which it existed, thereby robbing crucial information that would be required for the reader to process the data later. This is where the story overpowers raw data in that it can also codify the context within which the data was created.

Such zeal in trying to store raw data in favour of stories can be seen everywhere. Presentations stuffed with bullet points are one such example. As many of the visual presenters such as Garr Reynolds in his "Presentation Zen" and more recently Carmine Gallo in "Presentation Secrets of Steve Jobs" will tell you, slides with a single "story line" coupled with a visual is far more appealing and impactful than a slide full of bullet points. The underlying principle however, is in trying to give raw facts as opposed to a story. You can see which one is more powerful. For example, here are two slides that summarizes this blog. Which one sticks in your mind better?

(You can purchase the image at http://www.istockphoto.com/stock-photo-3201212-amusement.php)

Google's philosophy is somewhat similar. It encourages users to store all the raw data in the world so that it can be retrieved easily at a future date. While one can argue that the 'more than 200 parameters' it uses to rank the results is equivalent of processing the data, it still does not come close to the effectiveness of a story, primarily because the context is lost along the way.

Recently, there seems to realization in the IT/Business sector of the power of stories. Books such as "The Story Factor" and "Made to Stick" are good examples. Even with such compelling evidence in favor of using stories as a medium of communication as opposed to raw data, we still tend to choose the latter, probably because it is the easy way out. It is far less time consuming to dump information in bullet points than to hunt for a visual and try to condense your message into a single line. However, we are making the mistake of allowing the users to create their own interpretation instead of providing ours to them.

The problem seems more intense in the IT industry compared to other areas. The whole movie industry thrives on stories. Industries related to arts such as cooking rely on apprenticeship, which is in effect, creating your personal story by observing data in context. Another known attempt was done a decade or so back by Peter Naur (famous for his Backus-Naur Form notation) in his article 'Programming as Theory Building' where he touched upon the notion that software design is in some sense akin to building a story.

However, when it comes to IT or even Business in general, we seem to ignore this powerful medium in favor of the much less efficient raw data. Those who break away from this mould such as Steve Jobs are seen as charismatic and visionary. I wonder if we can take a cue from our past and focus more on creating better stories than trying to deluge people with information.

How many points am I worth?

A few months back, I had developed an interest in game development - primarily driven by my mobile device interests. Not having had a prior experience in gaming (except in playing), I thought it would be wise to read up on the subject. After some research, I honed in on The Art of Game Design by Jesse Schell. I am glad I did.

The book itself is not about a how-to on gaming that provides code snippets and exercises but really more on what gaming is and how to develop a mindset towards gaming. Jesse, I learned later, is a professor in Carnegie Mellon University and teaches gaming there. The book is quite fascinating and builds incrementally. Jesse also throws in a number of 'lenses' through which one should look at a gaming problem, with each lens providing a different perspective to the challenge of creating a game. I would strongly suggest that you pick a copy even if you have no inclination to develop games, as many of the concepts transcend gaming.

The fondest memories of my childhood always involved one game or the other, be it cricket with the local kids, pretending like Captain Kirk and playing Star Trek with clay models of "transponders" that me and my cousin used to make, or my and my friend inventing our own games (like 'Alphacross') and spending countless hours in the patio playing it. I think games add an element of fun to learning that is otherwise absent, and for some reason, we seem to forget this fundamental concept along the way and treat games as distractions instead.

Cut to present time - I got a weekly update from TED on non-TED presentations around the web last week and one of the updates was a presentation by Jesse Schell at the DICE 2010 conference about gaming in our lives.

DS Games - E3 2010 - Guitar Hero 5

I think it is a fascinating speech. While it starts a bit slowly, he ends with a bang that I think is more profound than what it seems to be on the outside. Essentially, Jesse claims that many concepts of gaming are already a part of our lives whether we like it or not, and that we are seeing games integrated with our lives in more ways than one and in not-so-obvious forms. He foresees a day when pretty much all our activities will be games where we will earn points in one way or the other by playing 'life' games and more importantly, corporations have an interest in us doing so.

While we are while away from the slightly horrific world of us getting points for brushing well due to the gaps in technology required, I think we are already well into this world and much more than we would like to admit. We get points for traveling around the world to drinking coffee at the local Starbucks.

And I don't think it is just a vision for the future as well. The idea of using gaming can be used in the current real world in more ways than one - think of how much more your children will be interested in doing something if you can mimic the virtual world they are already more familiar with. Same can be done in companies to allow employees to open up and share their knowledge with fellow team members. Another major area where gaming can have a direct application, as Jesse mentions, is in health care. Building on what Wii Fit has shown us, we can leverage games and social media to have a healthy competition regarding our health and fitness.

At the end of the day, everyone wants to be recognized and feel special. Giving a game setting to a challenge can help achieve that easily and cheaply.

I can also not help make a connection between his future vision and an article that Vannevar Bush wrote in the 1940s. The article, "As we may think", is considered a seminal article on personalization. In the article, Bush envisions a time when we will have devices or mechanisms that will augment our memory and keep track of pretty much everything that we do. 70 years later, this still seems to hold true and we seem to be getting closer to his vision than ever!

Bad Pronunciation and Good Presentation

A quick note to non-(South) Indian readers: The first part of this blog is a little topical and you may not relate to it. I have tried to provide an analogy to relate better, but I hope you can relate to the second part regardless of where you are from.

On my recent trip to India, there was one song that was played repeatedly in all the TV channels. It's a pretty nice, melodious number from one of the recent Tamil movies named Pasanga (Boys). It's been ages since I've seen any of the new movies, but the song stuck in my head. I think the song is about two young folks falling in love. Unlike many of the Indian film songs, thankfully the actors don't tend to sing and dance the number here and it is rendered more as a background number.

I forgot about it after I left.

Bad Pronunciation

Few months later, I was sampling through all the songs that I had gotten from India on my way to the office. Two songs stood out. One was a song called Maalai Neram (Evening Time) from a film called Ayirathil Oruvan (One in a thousand) and the other one, the song "Oru Vetkam Varudhe" (I am getting shy) - the one I mentioned earlier.

Maalai Neram (Evening Time) by Andrea Jeremiah in Ayirathil Oruvan



Oru Vetkam Varudhe (I feel shy) by Shreya Ghoshal in Pasanga



The songs stood out to me because they had a pleasing melody and because they both had unique voices (compared to others in the genre). As I started listening a bit more closely, I was all the more captivated by the second one and slowly started disliking the first. The reason - the singer of the first song, while having an excellent voice, butchered the diction. Her pronunciation was way off and it constantly distracted me from trying to enjoy the song itself, especially since I knew how it was supposed to be pronounced. On the other hand, the second singer rendered the song perfectly - with barely any noticeable variation in diction.

I know this sounds nitpicky. After all, music is supposed to be beyond language, right? Why not just forget about diction and enjoy the music itself? While agree to this philosophy and endorse it, I feel it is fine as long as you do not know the language. Knowing the language and understanding the meaning, I feel adds a few more layers of depth that you do not get with just the music.

For example, if you are a non-English speaker "Hotel California" by Eagles may be pleasing to you at first because of the guitar riffs at the beginning and the end as well as how the song progresses. For an English speaker, the song adds an extra depth not just because of the lyrics, but also because of the diction. Imagine the same song sung in a Karaoke fashion by Apu from Simpsons! The song goes on to add even more depth if you were a child of the mid '70s.

The irony in this case is that Andrea Jeremiah is a native Tamil speaker, while Shreya Ghoshal is not!

The trend of bringing non-native singers to sing in film songs especially because of their unique voice in Tamil is a new trend started a few years ago by A R Rahman (the 'Jai Ho' guy). While this is great and it added a new depth to the songs, not many have taken this task to heart. Many of the famous singers (Udit Narayan and Adnan Sami being the biggest of the culprits) have sung the song pretty much however they want with scant regard to the intricacies of the language in which it is sung, which I feel is a shame, as it robs the listener off a perception level. This cannot be just brushed aside as something that they do not know or are familiar with. There are others (notably Shreya Ghoshal and S P Balasubramaniyam) who have made a sincere effort to sing the song with as much accuracy as possible, trying to preserve the language of the song in the process. This, to me, makes them stand apart from others.

Good presentation

So far, whatever I've said may border along a rant. While it may be true, there are a number of other places where the same issue can have a more tangible impact. I will expand on two.

The first obvious one is when you travel to another country. While people may be curious about a tourist and may try to be helpful, it only goes so far. The curiosity will go away very quickly if the tourist does not respect the local culture. While some allowance may be given for a short while, it is not sustainable. On the other hand, if you attempt to understand and respect the others' culture even to a small degree and try to speak their language, it can go a long way in building trust (Avatar movie has this theme). I have experienced this in practice when I have worked with my China colleagues. While I don't understand the culture or speak the language, they become quite enthusiastic even if I say a "Hello" or a "Thank you" in their language. It has also helped 'break the ice' at times during conversations.

The second one is a little bit more subtle and one I can bet many of us do not do. It's about presenting our work better. As an IT professional, I end up creating a lot of documentation, typically in PowerPoint, Excel, or Word. For a long time, my focus has been on getting the documentation done (since it's done at the end of a project typically and I can't wait to get a break!) and not necessarily in making it pretty. However, over the last few years, I have come to realize that the style of presentation can have a subtle but significant impact on both your audience as well as on you.

Especially for Microsoft Office documents, there are some simple steps that you can take to make the documentation go from average to impressive. It takes probably a few hours at first to get this, but as time goes on, it becomes second nature. I have tried to explain this in a presentation that I put in Slideshare sometime back.

Better Documentation

View more presentations from Sathya Srinivasan.

To me the content of the presentation is like the unique voice a singer brings to a song. It's what she was paid for. No one is going to complain as long as the singer does not screw up badly and sing the song with her special take on it.

The style of presentation (layout, typography, and use of visuals) is the pronunciation of the singer. It reflects how well the singer understands the target language (how relevant the graphics and typography are to the content) and how much effort she has put in trying to know the language better. When this caring shows up on your presentation, the presentation on the whole becomes well-crafted. There are fewer distractions in the document that takes the reader's mind away from the content and the reader will have the intangible feeling that it is appealing - and that could make a big difference in how well your idea is accepted, and is well worth the effort.

Can you think of a few other scenarios where such additional 'style' on top of 'substance' makes a noticeable intangible impact? If so, feel free to post a comment.

Searching for a way to Web 3.0

For the last few years, I have been involved professionally in the "Information Management" space. Much has been written, blogged, and tweeted about managing information, ironically just resulting in more information to manage (job security!). Having been involved with computers since the late 80s, I have witnessed the explosive growth of computing power, the Internet, and the information that goes with the Internet. It makes sense since as social beings, we seem to have the fundamental tendency to share, regardless of whether someone is willing to receive or not.

One of the most significant changes that has happened during the last couple of decades is the widespread adoption of the Internet. While we haven't found a parallel physical universe, it looks like we have been able to create a parallel Earth in a sense with the Internet. I had blogged earlier about how the one key feature that seems to distinguish human behavior against God-like behavior is our limits of perception. To me, the Internet is our attempt to break the sensory limitations that we have and go beyond. Because of the Internet, our overall memory has increased and our inherent limitations on what we can understand and comprehend have started diminishing. Ironically the same break in the perception barrier has also imposed new challenges. We now have too much information to deal with and it's becoming harder to distinguish meaningful information (such as this blog) and meaningless drivel (such as this blog). Maybe it's God's way of punishing us for trying to break the barrier!

So, for sake of better understanding this universe, it may be worth understanding how it came into being in the first place.


Web 1.0
Around the beginning of Internet time, most content processed through the computers was structured data, such as tables of numbers. We were primarily using the computers as number crunching machines and it served our purpose just fine. Then someone had the bright idea to also share the documentation that goes with it (something that never happens in the real world - have you ever written good documentation for your code so that others can understand your mess?!) and thus started the great Internet revolution. The intent was mainly to push the information one-way, more like shouting from a soap box or giving a lecture, probably because the initial intent was to share scientific work.

This suited many people just fine and hence we had a proliferation of websites or soap boxes and everyone was voicing their opinion happily. This is probably the highest form of democracy - you can voice your opinion as much as you want without fear of censorship (except in some places) or getting reprimanded. However, as we have seen in many democracies (especially in India), democratic behavior can also lead to chaos if not controlled properly. You might end up having too many people voicing too many opinions resulting in more noise than music. For a country, this manifests in lack of decisive action (such as Obama not being able to pass a health care bill or India not being able to focus on long-term goals). For the Internet, this manifests in lack of focus from the user's perspective, such as not being able to find the right piece of information on time.

Web 2.0
As the information being shared grew larger and larger, so did the difficulty in finding the right information quickly.

Search engines came along to help alleviate this problem. They can probably be equated to a coordinating committee that tries to pass a bill. The purpose was to get the information and present the findings in some sort of an order so as to make the job of finding information easier. Most search engines were focused primarily on getting to the information based on the search being performed. It is kind of like finding a book in the library if you know the title (or at best, the category).

Due to the increasing reliance and relative ease of use of finding the information that was needed on the Internet rather than making a personal visit or a phone call, we seem to have started spending more and more time in this parallel universe than the real one. Maybe this increased presence led to the need for personalization, kind of like how your doctor or grocer starts greets you personally if you go to him more often. This increased presence also potentially led the "soap box" shouters to realize that there actually is an audience to hear their rant. This new audience also indicated potential opportunities, which in turn, led to increased advertisements and increased need to personalize the message to attract more audience towards one soap box as opposed to another.

Then Google came along to create a shift in the search. Instead of providing results based on just what the users entered, the search engine giant pulled parameters related to the information that could help contextualize the result. This shift in thinking converted Internet from being an information source to being a knowledge source, as knowledge is primarily information that is put in context. We are currently here.

Knowledge can be obtained in one of three ways - analyzing and understanding the content itself and putting it in perspective, prioritizing the information based on the users who read it, and comparing the information with other related information in order to create a context. Google has succeeded in the second and third areas primarily - it prioritizes the result not just by what the user is looking for but also by how others perceive that information (by way of inbound and outbound likes and supposedly 200 other parameters). However, I think we are still a bit off on the other aspect - understanding the information itself.

The parallel progress of increased communication in the Internet space as opposed to the physical space seems to have helped Google as well. Thanks to Social Networking such as Facebook, Twitter, and the like, we are actively providing more feedback to Google by means of linking content. In essence, we are the minions of Google!

Web 3.0
The next evolution potentially lies in the first aspect of getting knowledge - understanding and analyzing the information to find meaning. This issue has been complicated primarily due to the inherent unstructured nature of the primary language of the Internet itself - English. English by nature is more unstructured than structured, potentially due to its organic evolution. In a sense, it is kind of like the Internet in that it has grown in all directions with a simple core grammar. The grammar itself has a lot of rules and exceptions. To compound the problem, the words are heavily dependent on context, with a single word having multiple meanings depending on the context. Such ambiguity is bad for a structured lookup, which is why finding results by just keywords in a search engine is difficult.

Restructuring the web
Since this is a hard nut to crack, we have to look for alternatives. There are two. First is to convert the way information is stored into something more structured. The most concerted effort being made in this space is that of Semantic web, or web of meaning. In principle, semantic web attempts to redo how information is stored and shared. Current document formats, be it HTML that powers the Internet or Word, PDF, etc. use a convention that is more focused on presentation rather than information.

Semantic web attempts to change this by means of new standards such as RDF (Resource Description Framework) and OWL (Ontology Web Language) aim to get users to provide structure around otherwise unstructured information. This is great, except that it is difficult to get existing content providers to convert to this format quickly. The push can potentially happen if major content providers conform to these standards and eventually others are forced to follow suit for fear of being left out.

The second option is to obtain this contextual information from the attempts made to get to the information itself, by understanding the intent of the user's request. I feel this area is probably more immediately solvable than the previous one and hence can potentially provide a more short-term solution and augment the long-term solution later. For this to work, we need to go one step beyond looking at converting information into knowledge and start focusing on converting knowledge into wisdom. Wisdom, to me, is knowledge distilled over time. Thus instead of the current search behavior of returning results based on a single request, what is required is to understand user's behavior over time.

Knowledge by filter
This can happen in one of two ways - first is to store all the searches the user performs, and essentially glean information off that information provide relevant results. This sort of a brute force method is probably underway in Google already, since it pretty much keeps track of all the searches stored. This method has decent benefits. For example, if I search for the word "ball" and my last 100 website visits or searches were around football, the system can make an assumption that "ball" means "football" (or "soccer" - this is where semantic nets come in as well). Given the current scenario, this is probably the next step in search.

This concept was even attempted by a few systems. Yahoo! for example, introduced a beta version of its website for a short period where one could provide some clues to the search engine on whether they are shopping for an item or doing research. Unfortunately, it looks like it never made it to the main search engine and the site is currently inactive (as far as I know).

Google seems to have incorporated the same concept in its website as well, although it is not readily apparent (you have to click on the "More Options" link in the search results window to see the option.


Wisdom by conversation
Another way to understand user's intent is to have a conversation with the user. In real life, understanding occurs in this fashion. We have a conversation with the other person and by asking questions in different ways, we eventually figure out the underlying intention. It's kind of like the "blind men and the elephant" story but with a twist that there is a 7th person who pulls in the input from the six blind men to come to the realization that they are talking about an elephant. Similarly, it would be interesting if a search engine has a conversation rather than ask a single keyword and then analyze the information to get to the right result.

A few years back, there used to be a website called Active Buyer's Guide. Using the Wayback machine, I found that it became inactive around 2006 or so. Before Google and Amazon became mainstream, I used to use this site (probably around 2000 - 2002) to drill down to a specific product.

Source: http://faculty.ksu.edu.sa/2921/Lectures%202/Uses%20of%20Digital%20Camera.ppt

The site was different from other typical product sites (like Newegg, which by the way, has a much better filtering mechanism than Amazon). Instead of just giving a bunch of options that I may or may not know anything about, it used to strike a conversation with me in plain English.

Source: http://faculty.ksu.edu.sa/2921/Lectures%202/Uses%20of%20Digital%20Camera.ppt

On top of that, it also used to ask me how important one feature was compared to another, which was cool because not all features are given the same priority.

Source: http://faculty.ksu.edu.sa/2921/Lectures%202/Uses%20of%20Digital%20Camera.ppt

Thus, the results would be a combination of what I want in a product and how important I consider different features to be with respect to one another. This resulted in an amazingly close approximation to what I wanted.

Source: http://faculty.ksu.edu.sa/2921/Lectures%202/Uses%20of%20Digital%20Camera.ppt

The searches we perform our searches are similar to this behavior. We first do a search, look at the results, and if we are not happy with the results, perform another search potentially with some keywords found in the first search, till we get to what we desire. It should not be that difficult for the search engine to emulate this behavior. I believe this is where the next revolution in search engine lies and it will be interesting to see if this becomes a reality soon.

The developer's blame hierarchy

Having worked on a reasonable number of software projects, I come to realize that a sizable chunk of productivity loss is due to inappropriate placement of blame. In a team setting, it is natural to have confrontations and consequently, the mentality to be defensive, even if everyone is working towards the same common goal.

As a result, I developed a blame hierarchy that I normally mandate to my team before the start of a project and ask them to follow the hierarchy before raising the issue further. While it's true that there are some exceptions where the hierarchy fails, it works more often than not.

As per the hierarchy, if a developer finds an issue, the blame should be placed at a step only after clearing all the previous steps.

1. Your module
2. Your automated tests (especially if you didn't update the tests when the logic changed)
   
3. Your team mate's module
4. Stable custom framework
5. 3rd party framework
6. 3rd party libraries
7. Network connectivity/Firewall rules
   
8. Database
9. JVM / Virtual Machine
10. Operating System
11. Infrastructure / Environment

Failure to follow this hierarchy, mostly because of "My code cannot be wrong" mentality, will result in significant loss of productivity, not to mention irritation, annoyance, and outright anger.

Also note that the hierarchy should be followed only after due diligence has been performed, such as checking logs and looking for error messages.

How Sun could have made money off Java

I started out my career as a Java developer. While I have branched out to Microsoft SharePoint recently, I still do a lot of architecture and development in Java and consider it to be quite handy.
I have always wondered why Sun could not make money off Java after having come up with such a cool and now ubiquitous software. True, they tried their luck with optimizing their servers for Java and so on, but none ever stuck and they have ultimately got themselves acquired by Oracle.
Recently, I started working on a project that essentially is a desktop application (Swing components) with Webstart as the deployment mechanism. I have used JGoodies intermittently for creating rich layouts before and got a chance to use it more thoroughly this time around. As a quick aside, JGoodies is an extremely well-written library and makes your screens look a lot nicer. Combined with a professional tool like JFormDesigner for visually editing the screens, I was able to cut down my design time from roughly 2 hours to 10 minutes!
While I was developing the application, I was able to think of two ways in which Sun could have really leveraged its expertise and thought leadership around Java without irking the open-source community that it has come to rely on so much.
Meta-libraries
First way in which Sun could've made quite a bit of money is by developing a set of higher-level libraries and selling them commercially at a moderate price. For example, developing database applications and managing objects in Java was cumbersome at best until Hibernate and Spring frameworks came into the picture. Sun could've come up with such high-level libraries on top of the core language and sold them commercially. Even now, Swing is relatively bare bones in nature and you have to do a lot of plumbing before anything meaningful is accomplished.
Ideally, if Sun had created such meta-libraries around various scenarios (such as database, desktop, web, etc.) and had developer and commercial licenses (say, $10-$20 for developers and $100-$200 for enterprise deployment), it could have formed a great revenue stream. Optionally they could have milked some more money for any major upgrades (say, from Java 5 to Java 6). I think these price-points would've been more than acceptable for both developers and companies. The core libraries and JVM themselves would still be free and available for all those who want to start from scratch.
Marketing via App Store
Recently Sun released the Java App Store taking cue from Apple and after sitting idle for almost a decade. The App Store is based on the Java WebStart technology, a core component of the Java Runtime Environment that has been around pretty much since the beginning (around 1999, I believe). WebStart is probably the most under-appreciated concept within Java, thanks in part due to the Internet boom that happened soon after its release.
Essentially, WebStart allows you to deploy Java applications via Internet on to your system without needing any .EXE files. Better, applications deployed via WebStart are Java applications that are handled and run by the JRE in the client computer and hence, can be run on almost all platforms that have JRE (which is pretty much every OS out there).
Sun could have used this technology to build an eco-system around the Java technology while marketing the virtues of the language more successfully. Eventually they could've built premium services around the app store creating another revenue stream.
Oh, well!

Transformation complete - Windows on Ubuntu

As I had blogged a couple of months back, I installed Ubuntu 9.04 on my server and have been quite happy since. The system is very stable, I get all the applications I need for free and don't have to worry about updating them (Ubuntu shows me updates periodically for all applications installed through its package manager - like Windows Updates) and have no dearth of functionality.
However, I still had my Windows installation on another partition - just in case - because, much as Ubuntu is cool, there are still a few applications that need Windows, and my wife hasn't warmed up completely to Ubuntu due to occasional hardware glitches (mouse stuttering for example).
Today I was generally browsing through the applications available via the "Add/Remove..." option in Ubuntu and came across a little piece of heaven called VirtualBox from Sun.
In the Windows world, the one application that I used fairly often was Microsoft Virtual PC. I believe it's one of the better products from M$ and more importantly, it was free. With Virtual PC, you can install any other OS to run 'virtually' on Windows, such as Ubuntu (although it had its issues).
VirtualBox is essentially Virtual PC for Ubuntu, only an even better version. It installed in less then a minute. Then I created a simple Virtual Machine and a Virtual Hard Disk using a fairly familiar user interface, pointed it to my Windows ISO image, and lo and behold, I had a Windows XP running on my Ubuntu in less than 20 minutes. Ironically, I felt XP installed faster on VirtualBox than even on Virtual PC - and the start-up time is roughly 10 seconds!
With this, I think my transformation to Ubuntu is finally complete, with the last piece of the puzzle tightly in place. Now, I can have Ubuntu and Windows harmoniously running. Even better, with Ubuntu's multiple desktop feature, I can simply full-screen Windows XP on another desktop. With that, all I have to do is to flick my mouse to go back and forth between the two OS.
Now, that's what I call comfort!

Using Critical Chain for software projects

Earlier, I had posted how the concept theory of constraints (ToC) can be used for effective offshore communication. Those familiar with ToC might wonder why not apply the Critical Chain methodology, which is a project methodology offshoot of the ToC, for the entire software project itself. I've seen this implemented in some companies fairly rigorously for software projects and think that it's a great idea, but with a few caveats.

Critical Chain Methodology
First, a quick primer on CCM. The project methodology that most of us are familiar with, thanks to Microsoft Project, is called critical path - where the time needed to complete a project is based on the longest sequence of events. The focus in this methodology is on tasks and their schedule. Any unknowns are typically factored into each task by means of a slight buffer. For example, if a task would take 2 days to complete, it's typically calculated as 3 days, adding 1 day for potential delays or distractions.

On the other hand critical chain methodology focuses on the resources involved in the project and on the longest chain of resource constraints that would be need to complete a project. The concept is quite good and has been reported to help projects get completed 10% - 20% before the estimated date.

The primary difference between the two is that with critical chain you pool the buffer time for each task rather than include it with the task itself. So, essentially, the schedule is created based on the 'pure' time needed to complete a task (called 'focus' time) and not on the 'buffered' time. All the buffers are then pooled into a 'project' buffer (at the end of the project) or a 'feeding' buffer (at the end of each chain of tasks). Thus, you don't essentially say you'll complete the project at a certain date, but rather within a range, where the end date equals the date you'd calculate using critical path.

The catch - Consultants and Offshore teams
This all sounds good. So, what's the catch? If your project is completely onshore and does not use any consultants, then there is almost no catch. The concept works very well once people are trained to understand the methodology -which is crucial to the success of the project. The methodology itself has its roots in manufacturing, where these assumptions are mostly true.

The problem comes when you introduce either consultants or offshore teams, which is mostly true in case of software projects. As I mentioned earlier, CCM is a resource-based methodology and hence, has a strong emphasis on the resources and their linkages. When offshore teams are included in the mix, the problem arises due to time difference. Since the project plan itself is time-based and on dependencies between resources, it is difficult to capture the dependencies incorporating the time difference as well. While this is not impossible to capture, it just needs a bit more planning than usual to include this. This is in some sense true even for Critical Path plans, but becomes a bit more apparent here due to the emphasis on resources.

The second issue is when you include consultants. In CCM, resources would constantly shift tasks based on what is more important at a given point in time (as the critical chain would keep moving depending on the constraints at any given day). This is fine if all the resources are part of the same company.

However, let's say you have one consultant and one employee working on two dependent tasks. If the employee decides to move to another critical task for a day or two and not worry about the task that the consultant is dependent upon, what should the consultant do for the two days? Therein lies the issue. Ideally, you have to pay the consultant for sitting around because he's just dependent on another resource, but it may be hard to digest for the sponsor and the knee jerk reaction might be to ask them to do something 'useful', which may in turn, affect some other activity. This becomes more complex when multiple resources and projects are involved.

The other issue on the consultant's side is to create an invoice for the project. Usually, project plans are created where resources start low, ramp up, and finally ramp down close to launch, so as to minimize the overall cost of the project. This is fine as long as the project length is fixed and you know roughly when to on-board or roll-off a resource. However, with critical chain, the project timeline is a range and is not fixed. So, you'd have to create a range in your invoice potentially with an 'early completion' bonus or something similar, because you won't be sure exactly when to on-board or roll-off resources. This becomes messy when some high-value resources are in demand on other projects that don't use Critical Chain.

So, the bottom line is that Critical Chain is a great methodology and one that works really well. However, when brought into a software project, the three constraints - 1) consultants 2) offshore team, and 3) training (for both employees and consultants) must be considered before implementing it. Otherwise, you'll have a lot of headaches.

Three Box Principle

As a software architect, my primary job is to define an architecture, framework, or a platform for my clients. The expectation is that the framework or platform will be generic, modular, and flexible as soon as it's created. It does not work that way.

For the sake of this article, I'll use the term 'framework' to define frameworks, architectures, and platforms and the implementations you would do on top of them as 'applications'.

When I started my career as a software consultant, my first gig was at Bell Labs - the famed R&D wing of Lucent (now Alcatel) technologies. As a young Java programmer imbibed with GoF patterns and other similar literature, I was ready to spit out architecture all around me! When I attempted one such 'framework' for the project I was working on, I soon realized that my framework was not as flexible or as modular as I had hoped it would be.

One day, while sitting at the library in Bell Labs, I came across a book (or article) that talked about the three box principle. The principle essentially is that it takes at least three attempts or revisions to make a framework generic.

I have attempted to reproduce the principle in the diagram below.

The principle has an important corollary - "You cannot build a framework without building sample applications that utilize or implement the system." Most IT teams tend to create an "architecture" team who would create architecture by considering the overall goal of the company and then impose it on the applications used within the company. Such elitist attitude is doomed to fail, as it does not consider the ground realities of the applications and eventually the maintainers of the applications will tend to find ways to circumvent the architecture rather than use it, beating the purpose of the architecture in the first place.

In contrast, the best way to build a framework is to work with the applications, including them iteratively while developing the framework itself, which is what the principle suggests.
By the principle, in the first box or phase, you pick one or two candidate applications and build them. At this point, you are not worried about the framework. You are just developing the applications with a vision of the framework in the background, but not really doing anything conscious about it. This phase should then be deployed and field-run for at least a few cycles (months, weeks, etc.)

In the second box/phase, you pick few more applications and try to add them to the first set of applications. At this point, some patterns will begin to emerge due to the commonalities between the applications. You still don't consciously build the framework, but rather refactor the common libraries and features so that they are more modular. You might end up following principles of inversion of control around this time. By this time, you would have also seen the feedback from real customers and noticed the pain points of the initial applications and would adjust the features accordingly.

In the third phase, you add some more applications to the system. At this stage, there will be enough common functions/features that you can modularize further and see a much stronger pattern that would enable you to separate out the framework and setup the applications on top.

I have personally seen both sides of this principles - the success stories when this was followed and the failures when it was not. So, next time when someone asks you to build a framework, try to set the expectation that frameworks are more like a pastry from a master chef rather than a frozen microwave dinner. It takes time and multiple tries to make the end-product perfect.

PS: I believe the book I read was Implementing Application Frameworks by Mohammed Al Fayad, although I couldn't find a reference in the book. Maybe it was an article from the same author - I am not sure. If anyone has read this article, I would greatly appreciate it if you can drop in a comment with the right source.

Java Store - Good concept but late arrival

Sun unveiled Java Store yesterday at the JavaOne conference - yet another example of playing catch-up. As Java user for more than 10 years now, I think it's a great concept, but can't help but feel that this would've been a great hit probably circa 2004.
It was a time when the Java brand was getting beaten up by almost everyone as being slow and cumbersome compared to native applications. A Java Store with some really good applications would have gone a long way in establishing trust and keeping the web insanity that happened at that time (almost everyone wanted a web app at that time - whether it fit the scenario or not).
Now, it just seems like Java is copying Apple's idea and desparately trying to hold some ground on their own. Given the popularity of iPhone applications and their foray more and more into non-trival application space, they probably are. Sun missed this exact boat a few years back when they introduced J2ME. While half the blame should be shared by the telephone companies that wanted to monopolize the mobile application arena, Sun could've come out with an 'app store' that would've lent a lot of credibility to mobile applications. But they didn't and it was up to Apple to come up with the concept.
I wonder if Sun will get any mileage out of this. I have always felt that Java WebStart was a great concept and probably a very under-appreciated concept within the Java stack and am very happy to see that brought to fore with this initiative. But the question is, "Is it too little, too late?"

Software (and life) lessons we can learn from Pixar

Watched Pixar's latest film "UP" yesterday. Over the last years, Pixar has become the undisputed leader in making cartoon movies (or what we now more sophisticatedly call as animated movies). Their claim to fame is that they haven't produced a single bad film till now, which is pretty amazing in the movie industry. When you look a little deep into how they've managed to pull this off, some consistent ideals surface, which I think is very apt for the software industry, presentations, and to an extent, to our daily activities.

Pace yourself: Pixar releases films once 1-2 years, fairly lengthy compared to other movie studios (while somewhat comparable to Dreamworks). Pixar seems to focus on the long-term gains than short-term ones, something that almost no one does in any industry. Their conviction that a great product, even if released after a long period, can produce greater returns, has paid off time and again.

Contrarily, software giants seem to focus on the short-term gains, releasing version after version in 6-month or sometime even 3-month gaps, little realizing that people want stable products than frequent releases.
I guess we can apply the same in life - no, not getting babies every 2 years - but to essentially 'take time and smell the roses'. I remember when I was on a trek at Kilimanjaro, the guide constantly used to tell us 'pole pole' (e pronounced as ey) meaning 'slowly slowly', meaning don't walk too fast or you're going to get exhausted fast. In most cases, we don't realize this until its too late.

Having stable, timed releases has enabled Pixar to make more money with less releases and increase credibility with its viewers.

Have a Story: Pixar doesn't produce cartoons - they make stories. More than the animation wizardry, what makes the movie stick is the story that it conveys. And in almost all cases, the story is very simple, is unexpected (or at least has opposing characters), and is emotional. Without the story, a movie falls flat (something that most Indian filmmakers can learn from!). When you watch a movie, you are so engrossed in the story that you barely notice the animation - and I think that's essential to the success of Pixar.

In software, this translates into business functions. If the software is not functional, any amount of technical wizardry is not going to help an application to succeed.

Be detail-oriented, but don't show it: I heard that in their latest film Up, Pixar implemented a special algorithm that makes the 10,000 balloons that powers the house move as if they would in real life (bumping, moving, etc.). While the viewer may never even notice this nuance, having these details taken care of, somehow completes the picture more. The key here is that even if the effort that went in to do this was tremendous, they did not make that feature prominent, primarily because it's supposed to be in the background, as a prop to the story.
Similarly, in a software, one needs to take care of all the nuances (error messages, logging, connection retries, etc.) so that it in essence, is not visible to the user but is there doing its job.
Contextual user interfaces effectively achieve this - by giving you only the actions you need based on the context and not the entire set.

Don't succumb to too much technology: This is probably a negative lesson we can possibly learn from "Up". It looks like Pixar seems to have focused a little too much on the 3D than the story by keeping a lot of action sequences, which I feel diluted the film's message. It's always tempting to use the latest and greatest technology whether it's needed or not. At the end, users are going to be happy not because it has AJAX, but because it meets all their needs, in a friendly way.

I am sure if I dig deeper, there will be a few more prominent lessons, but these are the ones I consider to be the most imprtant. In any case, this is a blog and not an essay - so I'll stop here!

Effective Offshore Communication

A while back, my sister-in-law, who was doing her MBA at that time, recommended me to read a book called The Goal by Eliyahu Goldratt. I started reading it just before bedtime and ended up finishing it well into the night in around 4-5 hours straight. It's a fascinating book and written like a story (more on that in another blog later). Anyhow, in the book, the author proposes the idea of 'Theory of Constraints', which in roughly put, says that you have to identify the bottlenecks or constrants in your system and then address them first so that your throughput (or in some sense productivity) increases. The author mainly talks about manufacturing assembly-line style systems (this was written circa 1980).
However, I found that most of the concepts applied very well to the software industry, especially when it comes to onshore-offshore communication. I happened to be managing an offshore team in China at that time and thought I'd apply some of the concepts and ended up finding that it worked extremely well. Inspired, I wrote a white paper (or what I thought was an awesome white paper), only to find out that my company had a long process to actually publish it. I thought I'll share it with the world through my website as an alternative. You can read the full white paper Effective Offshore Communication using Theory of Constraints in my website.
Meanwhile, here's a small gist to get you interested.
In a team that's comprised of onshore and offshore resources, the biggest bottleneck is distance. This bigger bottleneck in turn leads to others such as turnaround time (due to time difference) and cultural differences (leading to misunderstanding). Per ToC, this can be handled using the following techniques.

1. Move the bottleneck to the top: As communication is the primary issue, make sure that is addressed first. For example, if you have a junior resource who knows Chinese, first get him to understand what needs to be done onshore and let him be the liaison to talk to the offshore folks than letting the Technical Lead trying to convey the same in English. You are not there to propagate English but rather to get the work done.
   
2. Increase capacity: Offshore resources are cheaper. If you can, hire a good technical writer on your offshore team (even an English Litt. would do) and get them to do all documentation and additional testing than spend scarcer resources onshore. You can also try to bring in a few junior resources offshore to get some of the daily tasks out of the hands of the more senior developers. The slight increase in cost will be well compensated by the increased productivity from your core team
   
3. Provide smaller chunks: Don't give a month's work to the offshore team and expect that everything will be great when you get it back. It's better to have weekly deliverables so that you can monitor the progress more effectively. More importantly, you can also confirm that what you told them as business requirements are what they understood. The tricky piece here is that too small a chunk would be counter-productive due to the time difference. Asking them to deliver something every day or every other day would make them spend more time in packaging the product than get any good work done. A good balance would be to ask the offshore team to get them to show progress over a video conference once a week and a deliverable every other week.
4. Quality control before a bottleneck: This is probably the biggest timesaver. A day or two before getting a build, get your offshore team to give you a demo over a video conference such as Webex. This way you can do a quick smoke test and visually confirm if what you will be getting is what you're expecting. If you run your tests AFTER you get the build, you might end up losing 3-4 days in sending the build back, getting them to fix, getting it back, and testing it again.
5. Redistribute and parallelize work: I've seen some managers or even tech leads push everything offshore. Just because they are the developers mean they should develop everything. Sometimes it may make sense to do a few prototypes onshore or even a small chunk of work onshore, especially ones that connect to the client systems or need to be validated with the clients frequently.
6. Prioritize work: While this sounds obvious, it's quite important to make sure that the offshore team focuses on the most important work first. It also helps to have a few low-priority items as fillers - stuff that the developers can do to take a break or clear their head.

Hope you get time to read the full white paper. If you do, please do post a comment on your thoughts and your experiences either here or in my site.