Software (and life) lessons we can learn from Pixar

Watched Pixar's latest film "UP" yesterday. Over the last years, Pixar has become the undisputed leader in making cartoon movies (or what we now more sophisticatedly call as animated movies). Their claim to fame is that they haven't produced a single bad film till now, which is pretty amazing in the movie industry. When you look a little deep into how they've managed to pull this off, some consistent ideals surface, which I think is very apt for the software industry, presentations, and to an extent, to our daily activities.

Pace yourself: Pixar releases films once 1-2 years, fairly lengthy compared to other movie studios (while somewhat comparable to Dreamworks). Pixar seems to focus on the long-term gains than short-term ones, something that almost no one does in any industry. Their conviction that a great product, even if released after a long period, can produce greater returns, has paid off time and again.

Contrarily, software giants seem to focus on the short-term gains, releasing version after version in 6-month or sometime even 3-month gaps, little realizing that people want stable products than frequent releases.
I guess we can apply the same in life - no, not getting babies every 2 years - but to essentially 'take time and smell the roses'. I remember when I was on a trek at Kilimanjaro, the guide constantly used to tell us 'pole pole' (e pronounced as ey) meaning 'slowly slowly', meaning don't walk too fast or you're going to get exhausted fast. In most cases, we don't realize this until its too late.

Having stable, timed releases has enabled Pixar to make more money with less releases and increase credibility with its viewers.

Have a Story: Pixar doesn't produce cartoons - they make stories. More than the animation wizardry, what makes the movie stick is the story that it conveys. And in almost all cases, the story is very simple, is unexpected (or at least has opposing characters), and is emotional. Without the story, a movie falls flat (something that most Indian filmmakers can learn from!). When you watch a movie, you are so engrossed in the story that you barely notice the animation - and I think that's essential to the success of Pixar.

In software, this translates into business functions. If the software is not functional, any amount of technical wizardry is not going to help an application to succeed.

Be detail-oriented, but don't show it: I heard that in their latest film Up, Pixar implemented a special algorithm that makes the 10,000 balloons that powers the house move as if they would in real life (bumping, moving, etc.). While the viewer may never even notice this nuance, having these details taken care of, somehow completes the picture more. The key here is that even if the effort that went in to do this was tremendous, they did not make that feature prominent, primarily because it's supposed to be in the background, as a prop to the story.
Similarly, in a software, one needs to take care of all the nuances (error messages, logging, connection retries, etc.) so that it in essence, is not visible to the user but is there doing its job.
Contextual user interfaces effectively achieve this - by giving you only the actions you need based on the context and not the entire set.

Don't succumb to too much technology: This is probably a negative lesson we can possibly learn from "Up". It looks like Pixar seems to have focused a little too much on the 3D than the story by keeping a lot of action sequences, which I feel diluted the film's message. It's always tempting to use the latest and greatest technology whether it's needed or not. At the end, users are going to be happy not because it has AJAX, but because it meets all their needs, in a friendly way.

I am sure if I dig deeper, there will be a few more prominent lessons, but these are the ones I consider to be the most imprtant. In any case, this is a blog and not an essay - so I'll stop here!

Plone ecosystem, terms, and definitions

Plone is one of the few fairly robust open source Web Content Management Systems out there, along with the likes of Drupal, Joomla, and Alfresco. While most WCMs are typically built on PHP or Java, Plone is based on Python. While this maybe one of the reasons why it is mentioned relatively less in the market (PHP is by far, more dominant in web-based applications, compared even to Java, I'd say), the features are quite impressive and on par with the other systems.

Here's a quick concept map that can help you become familiar with the terms and definitions around the Plone ecosystem, and trust me, there are quite a few because it has its own application server, database, and search engine! So far, the popularity seems to be more with educational institutions (although a few commercial case studies are mentioned in the Plone site).



I will be attending the Plone Symposium that's being held from May 26 - 31 shortly, and hope to blog more once I get back.

Effective Offshore Communication

A while back, my sister-in-law, who was doing her MBA at that time, recommended me to read a book called The Goal by Eliyahu Goldratt. I started reading it just before bedtime and ended up finishing it well into the night in around 4-5 hours straight. It's a fascinating book and written like a story (more on that in another blog later). Anyhow, in the book, the author proposes the idea of 'Theory of Constraints', which in roughly put, says that you have to identify the bottlenecks or constrants in your system and then address them first so that your throughput (or in some sense productivity) increases. The author mainly talks about manufacturing assembly-line style systems (this was written circa 1980).
However, I found that most of the concepts applied very well to the software industry, especially when it comes to onshore-offshore communication. I happened to be managing an offshore team in China at that time and thought I'd apply some of the concepts and ended up finding that it worked extremely well. Inspired, I wrote a white paper (or what I thought was an awesome white paper), only to find out that my company had a long process to actually publish it. I thought I'll share it with the world through my website as an alternative. You can read the full white paper Effective Offshore Communication using Theory of Constraints in my website.
Meanwhile, here's a small gist to get you interested.
In a team that's comprised of onshore and offshore resources, the biggest bottleneck is distance. This bigger bottleneck in turn leads to others such as turnaround time (due to time difference) and cultural differences (leading to misunderstanding). Per ToC, this can be handled using the following techniques.

1. Move the bottleneck to the top: As communication is the primary issue, make sure that is addressed first. For example, if you have a junior resource who knows Chinese, first get him to understand what needs to be done onshore and let him be the liaison to talk to the offshore folks than letting the Technical Lead trying to convey the same in English. You are not there to propagate English but rather to get the work done.
   
2. Increase capacity: Offshore resources are cheaper. If you can, hire a good technical writer on your offshore team (even an English Litt. would do) and get them to do all documentation and additional testing than spend scarcer resources onshore. You can also try to bring in a few junior resources offshore to get some of the daily tasks out of the hands of the more senior developers. The slight increase in cost will be well compensated by the increased productivity from your core team
   
3. Provide smaller chunks: Don't give a month's work to the offshore team and expect that everything will be great when you get it back. It's better to have weekly deliverables so that you can monitor the progress more effectively. More importantly, you can also confirm that what you told them as business requirements are what they understood. The tricky piece here is that too small a chunk would be counter-productive due to the time difference. Asking them to deliver something every day or every other day would make them spend more time in packaging the product than get any good work done. A good balance would be to ask the offshore team to get them to show progress over a video conference once a week and a deliverable every other week.
4. Quality control before a bottleneck: This is probably the biggest timesaver. A day or two before getting a build, get your offshore team to give you a demo over a video conference such as Webex. This way you can do a quick smoke test and visually confirm if what you will be getting is what you're expecting. If you run your tests AFTER you get the build, you might end up losing 3-4 days in sending the build back, getting them to fix, getting it back, and testing it again.
5. Redistribute and parallelize work: I've seen some managers or even tech leads push everything offshore. Just because they are the developers mean they should develop everything. Sometimes it may make sense to do a few prototypes onshore or even a small chunk of work onshore, especially ones that connect to the client systems or need to be validated with the clients frequently.
6. Prioritize work: While this sounds obvious, it's quite important to make sure that the offshore team focuses on the most important work first. It also helps to have a few low-priority items as fillers - stuff that the developers can do to take a break or clear their head.

Hope you get time to read the full white paper. If you do, please do post a comment on your thoughts and your experiences either here or in my site.

Americanization of Cricket

Born and raised in India, I am a big fan of the sport of Cricket, not unlike how most Americans swear by Baseball. As of my colleagues told me earlier, Baseball, despite its boring pace, is the favorite of most Americans because it's a part of their childhood. Same goes with cricket in India. My fondest memories are of me playing in the streets with stumps drawn on the wall, playing with tennis balls, and occassionally dipping my hands into the open gutters to get the ball that went out of bounds.

When I came to America first, the first thing that struck me the most was the complete commercialization and capitalism of the sports industry. Almost all sports are governed by money than by talent (some would argue that the money is given for the talent, but I don't agree fully with that). 

As a cricket fan, I always knew who to root for - India at an international level and Tamilnadu at a national level. Loyalty was more to the geography than an individual. While I admired exceptional sportsmen from other countries, my loyalty was always unwavering and didn't have to, since the players stuck to the geography. In contast, I did not see any strong reason for loyalty in sports in USA, because players keep shifting between teams, except possibly at the college level. If I root for NJ for example in one season, I find that half the players are dispersed in other teams in the next season because they got better lucrative contracts.

As with all other sectors, it looks like the wave of captialism and American values has hit the Indian sports industry, and like in most cases, they have taken the worst of the wave than the best. Yes, I am talking about the T20 matches and the IPL league. You have everything there now - half-time shows, cheerleaders, and not to forget - million-dollar contracts. BCCI - the cricket board in India is swimming in cash and they want more. Their interest seems to be in getting more money than improving the sport or building talent within the country. 

I am not saying that the new format is bad - it's short, and has lots of thrills. But my gripe is more about the complete disregard they have for the county cricket - the Ranji trophy matches - which aims to build regional teams that would eventually make it into the national team. Why can't they both co-exist together? Why can't the Ranji trophy be made as interesting as the T20 matches or as prestigious as the British county cricket? Is it because they cannot make money by building new talent? Possibly, but I don't think it's a valid reason. You can bring the same T20 format into country cricket (which is roughly what the IPL format is like, but not quite). 

The short-term gains seems to be clouding the judgement of long-term stability, much like the Indian polity. And no one seems to be looking back into history and see how that attitude has fared till now.

Hunt for the next tablet

I think the time is getting ripe for a consolidation, with rumors of an iPad from Apple. There have been a lot of super-similar products in the market now around the concept of the old Tablet PC.

• Tablet PC - The age-old giant which has still not caught the fancy and for some reason hasn't gone down in price as well
• Touch-sensitive phones - iPhone and Blackberry Storm are leading the pack with the way we interact with touch screens
• Netbook - The recent entrant in the notebook market, which for some reason that I cannot imagine, does not have a foldable display OR a touch-sensitive screen
• E-book Reader - An anomaly in evolution, Kindle and Sony are good for one thing - reading books, and that too in black and white.

It doesn't really take too much smartitude to find that all these products revolve around one uber-product - an A4-sized tablet PC with a multi-touch LED-based screen with nice, readable fonts.

Bigger question is, why are other companies waiting for Apple to innovate and take the market in this area and not beating it to the punch themselves?

Exception handling/safety mindmap

I have been following Jessica Hagy's Indexed for a while now. While some of the graphs may be a bit too esoteric for me, I am amazed at the way she converts fairly complex issues into simple graphs.

This, combined with my *conversation* about exceptions and threads I blogged about earlier, made me think. Why not attempt something similar, not with graphs, but small, manageable mind maps on core concepts that I can later use to refresh my memory?

I think mind maps, if used properly, are a great way to crunch knowledge into visual chunks that are easy to remember. After all, our brain is supposed to work this way, so why not remember it in a way it's easy for the brain?

So, here's my first attempt. This is about the concepts around exceptions.

Don't whine about Java being slow

I came across the article Java is too slow! via The Server Side today. I think most Java programmers in the late 90's and early 00's have seen one Dick or the other in their career. I made a post there on why you can't whine about Java being slow and thought of expanding on that here.

I think Java got a bad rep mainly becuase of yuppie programmers (and yes, many from India with more interest in catching on to the next wave than learning fundamentals of programming) spewing out junk code. The big firms were no exception either. I vividly remember IBM's Visual Age for Java, the pre-cursor to the now popular Eclipse - a kludgy, super-slow IDE that made users cringe and forced them away from Swing and over to writing web apps for desktop systems.

I have implemented a number of highly transactional systems as well as GUI applications (yes, Swing) in Java that have run extremely fast. The top reasons I have found for Java being slow are Lack of application of fundamentals of programming (such as taking care of the number of objects you create). Spring to some extent reduces this risk, if used correctly. The impact is the highest in Swing applications where layouts are nested too much just because someone didn't spend enough time to do the design properly. In some cases, even moving some unwanted "new"s out of loops will help things significantly. Not using the right software and instead reinventing the wheel. I have used MQ Series for a distributed system that happily processed around 100,000 records or more per hour without any complaints. The average transaction was around 7ms. And it was all written in Java. Not using threads optimally. One program that used the Java Advanced Imaging packaged processed and uploaded 50,000 images (each image around 200K) per hour easily when multiple threads were used (you have to of course check how many threads are optimal). Not using a profiling tool to check your code. In the glory days of Borland, I used their OptimizeIt software - an excellent tool to detect object leaks, something that's ignored due to Java's garbage collection. Finally, not tuning the JVM. After optimizing your code, you need to tune the GC so that it works for your application. Just leaving the defaults as they are (or just modifying the ms/mx values doesn't cut it). It's like expecting an auto-gear car to win in a drag race. I am sure there are a number of other reasons I can come up with, but these 5 tend to be the most common. It looks like Java itself succumbed to the slowness by creating stupidities such as the original J2EE, where there was way too much architecture/framework and way little code. Thankfully, the open source community - with Spring, Hibernate, and others - seem to have brought some sanity back and hope it continues before another wave of stupidity (possibly int he form of overuse of annotations) sets in.

Mapping your mind - or at least concepts

I was first introduced to the idea of mind maps when I saw the Java Concept Map. While it itself is not a mind map in the strictest sense, the idea is the same and made me explore this new way of capturing information.

Since then, I have heard rants and raves about mind maps - how crazy minded lunatics try to apply mind maps to everything on one hand and how procedure-oriented people write pages of documentation without a simpler visual representation. I think mind maps can be used and abused in a way similar to UML diagrams. While use case diagrams, class diagrams, and state charts can certainly be useful to represent the flow of a program nicely, over-use of the diagrams (such as strictest implementation of Model-Driven-Architecture or using Rational Architect) almost always ends in a disaster. I should know, I've lived through a few of them!

Back to the concept map, I love the use of typography there - big fonts for important concepts and decreasing font sizes for less important topics - kind of like tag clouds. Simple but effective.

Interestingly, I haven't seen any mind mapping software, including the most popular MindManager from MindJet, or the other free open source versions, such as XMind or FreeMind, provide this type of functionality out of the box. 

While no doubt these folks have done way more research in this area than I, my feeling is that the following two features that I have seen missing so far would be very useful.

• Each topic should optionally have a definition that will be displayed in a smaller font right next to the topic. Having it as a tooltip is not good enough as it will not be visible in the diagram.
• The connecting line should be 'describable'. In other words, I should be able to explain why two topics are related. This helps to read the relations in a meaningful sentence.

The closest that comes to achieving this is a nice piece of free software called Cmap Tools. The only downside I have seen in this one is that it by default is intended for a more collaborative environment than individual use.

Maybe it's time for me to reinvent the wheel! After all, that's how most open source projects start :)

Process of candidate selection

I recently attended an interview for a Technical Lead position. The interviewer (a very bright intellectual, especially if he's reading this blog!) asked me a series of questions around software development in general and specific questions around theory behind exceptions and threads. While I could answer most of the standard questions along with what I've been doing, I failed to impress him on the theoretical questions. Not that I did not know the answer or was not familiar with the concepts, but that it has been a while since I've read about them or used them. As a technical lead, combined with offshore development, the focus nowadays I believe has shifted more to design, process, and governance as opposed to hands-on development (which is where the offshore piece comes in). This made me wonder how we select people for a particular task.

It's interesting how your career or future depends on how you perform at a single instance regardless of how better or worse you've performed in the past. Our entire system seems to have been built around this concept - be it education, sports, politics, or job. Exams, match formats, and interviews all favor how alert you are on a given moment in time with respect ot the opposition compared to how you've been doing over time.

While I can understand the current rationale behind this (it's not feasible to keep track of everyone at all times to see how they do, especially those who you don't know, such as potential candidates for job), I think it's quite out-dated and incorrect.

Most US universities at least at the collegiate level use a combination of your final exam marks along with how you've done in all your assignments and interim exams, with varied weights. I fail to see why the same concept cannot be applied to other areas. Let me take the specific scenarios I mentioned above and provide some alternate examples.

Sports

This is probably the more controversial of the lot since it's not just the process, but also the thrill provided by the finish. Nevertheless, instead of providing a point win-lose scenario, why not have a weighted system, where, say, the number of wins up to finals accounts for, say, 40% of the final game, while the remaining 60% goes to the final game itself? Such a mechanism will provide a fair ground to a team that, say, has been winning for the whole tournament but ends up losing in the last match. In such a case, their past record will provide an additional boost for them to keep up the momentum to win the final piece.

Game Shows

Game shows are no different from sports in that they are a type of a sport, with similar format. A recent example which added to my thoughts is the latest season of the Biggest Loser, a game show in USA where the person who loses the most weight by the end of the season gets a grand prize. 

In the show, an ex-model Tara, won all the rounds handily including all additional challenges, but lost the final weigh-in by 5 pounds. I thought it was a bit unfair and sends a relatively wrong message, since her opponent seemed to have lost a little too much weight! With a weighted system, her past victories would have contributed to her final score and would have helped her, especially since she lost by such a small margin.

Politics

Another controversial topic I am sure, and definitely harder to implement, but think about this. Why not have a system where, in an election, the winner is selected not just by votes on how they did in the election process but also on how they performed during the previous tenure? While I am no political expert, I can think of some fairly reasonable metrics to measure the performance during a tenure such as the attendance of the politicians in the parliament procedures, the percentage of growth in development in various sectors, relative stability of the country - both economically and militarily, potentially calculated by level of threat, people confidence, GDP growth, and inflation rate. I am sure more metrics can be added.

Such a system would ensure that only a candidate, either ruling or opposition, who has really worked for his/her people is selected for another term, without being swayed by last minute sympathy waves or other external criteria. It also tends to be more meritocratic and will tend to keep the politicians on bay WHILE they are in power and not after.

Job

Finally, to something that made me write this blog. I think the evolution of Internet has provided some excellent opportunities to change the way we select candidates. Instead of deciding the worth of a candidate in a 30 minute interview, why not combine the responses with other factors such as what they have done outside their core skillset? This could include how active they are in social networking sites, what sites they are active on, what they have contributed, and so on. This is similar to valuing PhD candidates based on the articles they have written on conferences they've attended, for example. 

Such a weighted system would not only help a team lead pick the candidate with the right technical background (based on the interview) but also with the right passion, interest, and intellect, given that you need more than just technical smarts to execute a project.

Is this a gripe because the interviewer didn't consider my candidate worthy? No. I honestly believe that human nature is more complicated than a binary solution and that a weighted system that evaluates a person over time (what we would call experience) is a more accurate measure.

Better Documentation

One of the interesting things in IT I have seen so far is how people tend to learn less of what they use more. I call this Inverse law of IT learning. For example, we take training courses, read books, and participate in discussions about say, Weblogic or WebSphere or SAP or any other technology we tend to use sporadically in different projects. However, we barely tend to read a book or take a course about Microsoft Word or PowerPoint. We have come to see them equivalent to a DVD Player or a TV where we're just supposed to 'get' it without RTFM.

I don't think the analogy we developed applies much. Desktop publishing software such as the Microsoft Office Suite are much more complicated than Notepad for example. They are excellent tools to deliver quality documentation, if only we spend a little bit of time understanding them a bit more. I have seen experienced professionals spend minutes (which adds up significantly over time as we use Word almost every day) in cutting/pasting/selecting by fumbling with the mouse without knowing some basic keyboard shortcuts that can let them do the same in seconds.

We are more happy to crib about the 'user unfriendliness' of Microsoft Office than trying to spend a few minutes to understand the design and concepts involved, such as the Style library and Paragraphs.

I made a presentation (in the traditional death-by-bullets style) a while ago for my colleagues on how small tricks in Microsoft Office suite can significantly improve documentation that is produced daily by teams - be they defect reports, project status, or design documents.

While writing the presentation, I ended up reading Presentation Zen and Non-designers design book, which completely changed my perception on how to create presentations. You can see the final version in SlideShare. Please comment!

Better Documentation

View more presentations from Sathya Srinivasan.

Blogging again!

Not that anyone cares, but it took me ages (as can be seen from my last post) to get back on track to start blogging again. I have lived through one of the biggest hurdles to blogging that one can go through - I am not doing anything interesting, so what exactly can I blog, and that too daily?!

It's a fairly easy state of mind to get into, especially if you are working constantly. I got a chance to take a breather and realized that I have quite a bit to blog actually. I read a lot, I surf a lot (and no, I don't spam-a-lot!). That's good fodder for the blog.

As a software consultant, I always end up getting another doubt. Consultants by profession, are supposed to get into the heart of the problem, understand it quickly, and provide a solution. In most cases, we have to pretend that we know all the answers to build customer's confidence and then, in the background, cram as much information as possible to be one-step ahead of the questioners. This poses a problem with blogging. Let's say I get into a new domain or technology. It's hard to blog in the process of learning for the fear of letting the customer, who may potentially stumble upon the blog, that I am learning. And once I am done learning, I move on to other things, and end up not blogging about what I learned. A kind of a catch-22 situation I guess.

Well, in any case, I have taken my keyboard up again and hope to put it to the page. Let's see how long it lasts this time.