My Second Blog: tomlooy.wordpress.com

Good morning Andrew. Just thought I would let you know that I have started a second blog at http://tomlooy.wordpress.com/. I'll keep ConversationsWithAndrew going, continuing to focus on ideas that I think are relevant to share with you. It will start to become a little more personal and range beyond just what I am learning as a PM and more into matters of being successful life in general.

The new blog will be focused more on software development process. I'll continue to talk about Agile project management but I will also write about what I am learning about Lean, Theory of Constraints, Servant Leadership, Deming and others and how it relates to organizations in general.

Lessons for Corporate Leaders from the New Testament Local Church

Hi Andrew. I have been reading a lot of books on Leadership as I am motivated to becoming a better Servant Leader. You can see many of these books on our bookshelves at home like:

• Leadership and the New Science
• The Contrarian's Guide to Leadership
• The Serving Leader
• The Leader's Handbook
• Great Boss Dead Boss
• Solving Tough Problems

These are all great books and I would recommend you read them regardless of whether or not you choose to be a manager or corporate leader. Somewhere in your lifetime you will become a leader and you will find these books to be of value to you.

I recently finished a leadership book that I found to be very helpful and enlightening: Spiritual Leadership. The book covers a lot of the leadership principles from the books I have listed above and it confirmed for me that even in a spiritual context they are valid principles. The author does go one step further by defining how spiritual leadership extends beyond corporate leadership. You can read my review of the book on Amazon to see what I mean.

But rather than seeing how secular leadership principles apply in the functioning of a local church, I would like to show how the organization and leadership principles defined for the local church in the New Testament 2000 years ago are the principles that are making today's great corporations great. I believe that the local church, when operating at its fullest potential, is the most effective organizational structure in history. Therefore, we have much to learn from highly functioning New Testament local churches.

Let's start by looking at five leadership tools that we are instructed to use to serve members of the church in assisting them in become more mature Christians. Although they might seem very similar, they are significantly different. We need to understand these differences and effectively implement each, whether in the church or in our corporations if we are going to have great organizations.

(Please scroll down to the table below. Blogger is inserting excessive line breaks into the HTML, creating this big gap in the posting.)

Leadership Tool	Purpose in the Church	Purpose in the Corporation
Preaching
Teaching
Training
Mentoring
Discipling

In my next several postings I will fill out this table and expand on each of these tools of leadership, starting with Preaching.

Preaching? In corporate America? You'll see what I mean...but I don't mean anything that resembles Steve Balmer's 'Developers Developers Developers Developers'.

My Book Shelf from Shelfari

Hi Andrew - I just added the My Book Shelf widget to our blog. These are books that I have read recently that I have found great value in. I would suggest reading any of these books, even if you don't become an Agile PM.

Enjoy!

Agile Project Management: Working Within Complex Adaptive Systems

Hi Andrew. If anyone asks you what I do for a living you, it's pretty simple: I'm a Project Manager, an Agile Project Manager to be a little more precise. The Agile part of the title is important in that it lets people know that I choose to manage projects differently than other more traditional managers. It's also important to note this distinction: I don't manage Agile Projects - I am an Agile Manager of projects.

So what is the distinction in being an Agile Manager? Let's start with this premise: software development is hard to do well. We know this because our success record for projects is pretty dismal and this is not because we haven't put our best people on it. Great developers and analysts and managers and architects have been working on software development efforts for years and we still have a dismal record of success.

One of the reasons that software development is difficult is that software is complex and the development process is equally complex. So in trying to manage the complexity of software and software development one of two approaches can be taken.

The first approach to managing complexity in a software development project is to try to control the complexity in the project by keeping things as static as possible. This is one of the fundamental tenets of the waterfall approach to project management that results in the following practices:

• Get the requirements right from the start so they won't change during the course of the project;
• Define your architecture up front and try to anticipate anything that the application may be required to handle in the future;
• Commit to the estimates that your team makes at the beginning of the project and stick to them throughout the project;
• Establish standard methodologies in your organization to assure that each project is done the same way.

This approach did have some success early on in bringing 'cowboy' software projects under control. But software development has become much more complex and volatile with rapid changes in technology as well as changing business requirements and demands for faster times to market. These new conditions have strained the waterfall approach beyond its limits.

So a new approach to software development has emerged - Agile Project Management. Agile does not attempt to keep things static but rather embraces change. Agile manages complexity by keeping things as simple as responsible and by constant learning and adapting to the changing conditions surrounding the project.

As an Agile Manager of projects I attempt to create environments where complex systems can adapt as the organization, customer and development team learns. Agile uses the following practices in managing these Complex Adaptive Systems:

• User Stories are gathered just-in-time during development of the software so we get the freshest requirements for the system
  
• The highest priority User Stories are developed first so we don't over-complicate the software with features that are not really needed
  
• Architectures emerge through coding, not by big upfront design
• Working software is delivered early and often for feedback from the customer
• Development teams self-organize around what works best for their team

So Andrew, at the heart of what I do as an Agile Manager of projects is attempt to optimize the work being done in Complex Adaptive Systems. This involves recognizing the systems involved in software development, understanding their complexities and interdependencies, and allowing for teams to adapt their processes to optimize their results. This is a lot more fun than the old command and control oversight of projects I did before becoming an Agile Manager of projects.

Not making much sense to you yet? Maybe in a couple of years...

Martin Fowler May Have Slept Here

Andrew - a little lesson in Physics for you today. Trust me that this will eventually lead to another favorite subject of mine and the theme of our blog - Agile Software Delivery.

Let's say you have an environment that is really cold, potentially at Absolute Zero. You decide that you want to measure the temperature of the environment to see if it really is at Absolute Zero. But you quickly realize that if you introduce a measuring device into the environment, the warmth of the measuring device actually affects the temperature of the environment because it will transfer some of its heat into the environment. Your conclusion - you cannot accruately measure something that is really cold without affecting its temperature by trying measuring it.

This same problem occurs in other areas of Physics, most noteably in the area of quantum Physics. The Heisenberg Uncertainty Principle states that there is an observer effect on the measurement of the position or the measurement of the momentum of a sub-atomic particle. Stated differently, you can know either the position of a particle or its momentum but you can't know both. Measuring one affects the other. (The Heisenberg Uncertainty Principle has led to the most famous of all Physics bumper stickers - "Heisenberg May Have Slept Here".)

This Observer Effect also has a place in software development. The following is a quote from Scaling Software Agility by Dean Leffingwell:

"We also discovered an even more pernicious aspect of providing software solutions: delivery of a new system changes the basic requirements of the business and the behavior of the system, so the actual act of deliverying a system causes the requirements of the system to change." (The Italics are the author's.)

Agile's preference for early and frequent delivery of working software allows for the requirements of a system to emerge and the business process to evolve as the system is being coded and delivered incrementally. In Agile Software Development we are constantly asking the question "What can we code next that adds the most business value?". In contrast, complete up front gathering of requirements for a new software solution is more an exercise of the memory, not the imagination. "Let's make sure we don't forget anything" is the mantra of Functional Spec contributors.

So let's complete our analogy: in Agile Software Development the Observer is our Agile enabled development team (aptly portrayed by Martin Fowler here); the thing that is measured is business value at the end of each sprint/iteration; the thing that is affected by our measurement is the backlog of story cards as we re-prioritize and add or remove (yes remove!) story cards based on feedback from our users. Also affected by our measure of business value is the user's business process - it will become better optimized as the software is continually optimized for greater business value.

In contrast, in the Waterfall approach the Observer is the keeper of the project plan (or occasionally a requirements traceability matrix); what is measured is progress to the plan; what is affected is...hmmm...this is interesting...what is effected is quality. A poor quality software solution will have an effect on the business process but negatively as the business process must adapt to deal with the shortcomings of a badly designed and buggy software solution rather than adapting to provide greater business value.

The Information Age Is Keeping Me Stupid

Good afternoon Andrew. Sorry for not posting in awhile. Let me explain why.

About 6 months ago I read The Contrarian's Guide to Leadership. In the chapter "You Are What You Read" the author suggests that a leader should read the classics, great books that have passed the test of time (i.e. Plato's Republic, Machiavelli's The Prince, as well as the texts from the major world religions). He also suggests that a lot of books that we read today may not be relevant in as short as a couple of years so he encourages selectiveness in what we read. As a result, I have become more selective in my reading, especially when it comes to blogs. This has also made me much more selective about about my own postings.

But that was also a wake up call for me that I have lost focus of what this blog is about. I have been too concerned about what my colleagues think about my postings, forgetting that this blog is really about sharing with you, my son, about what is important in life. So now that I think I am back on track I am ready to focus on more important things, like passing on worthwhile advise to you.

On to the topic at hand - The Information Age Is Keeping Me Stupid.

The book that I am currently reading is The Leader's Handbook by Peter Scholtes. The author is a disciple of W. Edwards Deming whose work in resurrecting the Japanese economy after World War II has stood the test of time so I figured learning from Deming would be worthwhile.

In the The Leader's Handbook I found an interesting diagram that I have recreated below. The diagram illustrates that from 1800 to 1950 the average time that a person worked was approximately 25 years due to a short lifespan of approximately 40 - 45 years. At the same time technology was not changing at a very fast rate. Therefore, a person was sure to learn their craft at an early age and not have to learn anything new to stay employed. But in the middle of the last century something interesting happened - life spans doubled and the rate of technology changes began to accelerate. In essense what has happened is that in the duration of our new working lifespan of 40 - 45 years, we will have several significant technology changes of which we have to stay current with. Scholtes point is that now more than ever, we are compelled to learn how to learn to stay gainfully employed.

For me this is fine because I enjoy the learning of new technologies and new ideas in Project Management. But this also means I don't have the time to read the classics as suggested in The Contrarian Leader. I may be staying technically viable but am I getting any wiser? Am I equiped to discuss the most significant matters of life, things much more relavant than the merits of Ruby on Rails or Agile Project Management?

What I really don't want is to be satisfied with what are culture considers a dialog on ideas. Today we see the shouting down of each other on TV in what is supposed to be debate. I don't want to rely to getting my political views from comedian talk show hosts or Hollywood stars. I also don't want to resort to such simplist arguments as 'Flying Spagetti Monsters' in rebuttal of ideas related to important topics such as orgins. I don't want the pre-thought thoughts of the media, conservative or liberal, to shape my thinking or the 15 second sound bites from our politicians to influence my vote.

Charlie Munger of Berkshire Hathaway gave the Commencement Speech at the USC Law School in May of 2007. Here is an excerpt from his speech:

"I have what I call an iron prescription that helps me keep sane when I naturally drift toward preferring one ideology over another and that is: I say that I’m not entitled to have an opinion on this subject unless I can state the arguments against my position better than the people who support it. I think only when I’ve reached that state am I qualified to speak."

Charlie Munger also said "Nothing has served me better in my long life than continuous learning." Somehow I don't think he was referring to staying current in technology.

Has the demands of staying current in technology left me with no time left for learning from the classics? It's time for a shift in my priorities.

Andrew - make time to read important stuff. Learn to respect others and their ideas, even if they are different than the ideas that you formulate. And then find good people that you can have deep meaningful conversations with. Learn to articulate your ideas in the company of these good people.

I have been fortunate to be around good people with whom I can have these kinds of meaningful discussions regarding the important things in life: Matt Gelbwaks of Globant, Chris Andrasick, my CEO at Tacit Knowledge, Amit Rathore of ThoughtWorks, and most recently my good friend Jay Padinjaredath, also from Tacit Knowledge.

Read important books, make good friends and discuss important ideas.

Servant Leadership is great, but does Ricardo Semler go too far with it?

Hi Andrew. As you will recall, I have been talking to you about Servant Leadership for a long time now. Its principles resonate well with me and have validated my preferred leadership style.

One of the fundamental principles of Servant Leadership is to 'Upend the Pyramid' and is described nicely in The Serving Leader by Ken Jennings. Essentially 'Upend the Pyramid' says a Serving Leader is to invert the hierarchy within an organization - put the executive leadership of an organization at the bottom of the pyramid, focusing on serving the management layer above them. The management layer is to be focused on serving the layer above them, the people within the organization that are actually getting things done - the factory line workers, software developers, etc. All great stuff which I heartily embrace.

Lately I have been reading about Richardo Semler as several friends have suggested I read his books including Maverick and The Seven-Day Weekend: Changing the Way Work Works. In my research I have learned that Semler embraces the concept of upending the pryamid to a degree far beyond my wildest imagination. I found this on CNN.com's profile of Semler: "Workers set their own salaries, share company profits and hire and fire their own managers. " (Emphasis mine!)

Is Ricardo Semler going to far with the empowering of his employees to fire their own managers? If you look at his results at Semco SA you would have to say 'No'. How do I feel personally about being a manager working in an environment where I could be fired by my own development team? Well, if the development team is made up of the caliber of developers that I get to work with today at Tacit Knowledge or those that I had the priviledge of working with at ThoughtWorks, I'd say I'm fine with it. Their focus is on adding value for their customer and not their own personal agendas. If I am not doing my job of creating an environment where they can be successful in adding value, then I want them to let me know it, but hopefully before it gets to the point where they would be handing me a pink slip.

ADD (Accountability Driven Development) - The Original Software Development Driver

Hi Andrew.

Lately the Agile community has been inundated with acronyms and abbreviations describing what should be driving software development or design. First it was TDD (Test Driven Development), then FDD (Feature Driven Development) and BDD (Behavior Driven Design). My favorite is ATDD for Acceptance Test Driven Development which I learned from Richard Watt at Thoughtworks. Most recently Alistair Cockburn has introduced us to XXD, pronounced Dos Equis Driven, for eXecutable eXample Driven Design.

I really don't get annoyed by these variations on a theme as I learn something from each one of them. I just hope that they don't create confusion for those that are trying to embrace Agile in their organizations or cause schisms within the Agile community. But what is yet to be acknowledged is the granddaddy of them all, a driving principle of software development that has been around for decades and continues to drive software development in many organizations today: ADD or Accountability Driven Development.

With ADD, you make sure you have someone to be held accountable during each phase of the software development process. You get someone to be accountable for getting the requirements right from the start. And then you have someone who is held accountable for getting the architecture of the software right before development begins. Accountability is then thrust upon the Project Manager and developers for delivery of the software and eventually onto QA for testing it. I call this Role Based Accountability.

Tell me Andrew, doesn't it seem like a reasonable thing to hold people accountable for getting their job done? Actually son, Role Based Accountability isn't a reasonable thing at all. Let me explain.

First of all, Role Based Accountability is based on the mistaken notion that specialization of skills is a good thing, that somehow it makes the process more efficient. Agile software development has learned from Lean Manufacturing and from the Theory of Constraints that specialization does not improve quality or time to market or even profitability. Specialization leads to Local Optimization which creates unnecessary bottlenecks within the system. Global Optimization or Systemic Thinking should be the goal. (We have already talked quite a bit about Local Optimization so I won't go into more detail here. I will direct you back to Eli Goldratt's book The Goal or Peter Senge's The Fifth Discipline if you need a refresher on the subject.)

Secondly, Role Based Accountability makes people hesitant on moving the development process along. For example, Business Sponsors will want to spend weeks reviewing a requirements document before approving it to make sure that every possible feature they may want is included in the document. Software Architects will want to research new technologies and design architectures for the application before they let developers proceed with development. QA will not want to start testing until the software is at a 'code complete' state.

Third, Role Based Accountability tends to create 'contracts' between Roles, the kind of contracts that are designed to protect an organization from a self serving vendor or a vendor from scope creep from its customer. The contracts between customer and vendor are understandable as the two organizations typically don't have the same goals. But within an organization, having a Functional Requirements Document as a contract between the business and IT often does not foster a trusted relationship between business and IT and it definitely does not promote development optimization.

Finally, Accountability Driven Development is often not about getting software into production but about making sure that there is someone to be held accountable in the case of failure, someone that is either downstream from you in the development process or below you in the org chart. Read Machiavelli's The Prince and you will see this medieval principle that is applied throughout business today.

All too often IT projects are not organized around getting quality software quickly into production that adds value to their customer. Even as organizations try to adopt Agile principles I see them getting too fixated on defining Roles within the Agile process. Agile is not about Roles and creating a process around these Roles. Agile is about maximizing throughput – the right requirements to code as fast as possible. Therefore, any Role that gets between the customer and the developer will diminish your throughput.

Follow Up on Agile and Trust: My Strengths Test Results

Andrew and Chelsea on a beach in Ft. Bragg, CA

Andrew, as a follow up to my earlier posting I thought it might be appropriate for me to share the results of my strengths test:

Learner - You love to learn. The subject matter that interests you most will be determined by your other themes and experiences, but whatever the subject, you will always be drawn to the process of learning...

Connectedness - Things happen for a reason. You are sure of it. You are sure of it because in your soul you know that we are all connected...

Positivity - You are generous with praise, quick to smile, and always on the lookout for the positive in the situation...

Includer - "Stretch the circle wider." This is the philosophy around which you orient your life. You want to include people and make them feel part of the group...

Restorative - You love to solve problems. Whereas some are dismayed when they encounter yet another breakdown, you can be energized by it. You enjoy the challenge of analyzing the symptoms, identifying what is wrong, and finding the solution...

Agile and Trust

[Updated - corrected name of the book referenced in the first paragraph.]

Andrew, you may recall that the other night I told you I was in the process of taking an online test and I could not be disturbed. Sorry about not being able to hang out with you but the test was something important for me to do. It was a test to help me determine my core strengths. It is based on the book Now, Discover Your Strengths, a book recommended by a good friend of mine, Adam Monago.

After I finished the test I was given my results - they were interesting but pretty much what I expected. I immediately sent my results to Adam and he confirmed that my strengths assessment results were dead on. And then he sent me his results. That’s when my results really got my attention. You see, Adam’s results were completely different than mine and he is also a ThoughtWorks project manager (and a very good one at that). Fundamentally, the difference between Adam’s results and mine is that his strengths are very goal oriented and mine are very relationship oriented.

Comparing Adam’s results to mine was a very tramatic moment for me. My first reaction was that if I were a client I would want Adam as my PM rather than me. I would want someone goal oriented and driven in managing my projects to successful completion. I wouldn’t want some relationship building pansy going around giving hugs and trying to make everyone feel good. There’s work to be done so everybody get to work!

But as I reflect back on my career as a somewhat successful project manager I now see that my management ‘style’ has always been based on relationship building. But this is not the kind of relationship building of going out after work with the guys, but rather the building of relationships based on trust. Let me explain further.

There are three different groups of people that I interact with as a project manager: my team, my peers, and my customer or boss. Each requires me to build a level of trust in the relationship in order for me to be effective.

• For the team I want to build a trusted relationship where they know that they are empowered to do good work and that I will be fair and honest with them;

• For my peers (other PMs) it is a trusted relationship that I will be supportive of their projects and be globally focused on what is best for the organization and not what is only best for me or my project;

• For my customer or boss it is a trusted relationship of transparency into the health and status of my project.

I have recently recognized that when I do not have the opportunity to establish these kinds of trusting relationships that I resort to a different type of project management, one that is more goal oriented and less personal. But this is not where my strengths are so I am not as affective a project manager and I find my work much less satisfying.

Knowing this now, it’s not at all suprising that my preferred software development methodology is Agile where one of the cornerstone principles is ‘People over Process’.

So Andrew, you may not end up being a project manager and you may not even have relationship building as one of your core strengths. What is important is for us to discover your strengths and build on them. I'm looking forward to it.

Agile versus Stubborn Methodologies

I was reading the Sept 2006 issue of Dr Dobb’s Journal today, especially interested in seeing what Ivar Jacobson has come up with in his new and improved Essential Unified Process. The article starts out pretty well with an acknowledgment that his “Unified Process has become too heavy”. But two paragraphs later he says his new process “must provide agility with discipline”. That very statement tells me that Jacobson does not get Agile. Agile is the most disciplined software development process I have used to date. (There is enough material out there regarding the disciplines of Agile that I won’t spend the time here discussing it. If you need further details check out Jim Highsmith’s excellent book Agile Project Management.)

So if I believe that Agile is disciplined, more disciplined than the heavyweight processes like the Unified Process, what word would I use to describe these heavyweight waterfall processes? The word I came up with is stubborn. Here’s why:

• Waterfall has the project’s requirements defined up front. A request to change a requirement during the development phase can be an arduous and slow process of change requests and approvals. Heavyweight processes do not promote change but are designed to stubbornly resist it.

• Waterfall projects have delivery dates set up early on in the development process, often by people too far removed from the actual software development and without the scope of the project well defined. Then the project is stubbornly managed against those early and ignorantly derived dates, even when scope increases.

• Waterfall projects often spend a significant amount of time in ‘Big Upfront Design’. Then this design is stubbornly enforced by the architecture police, even if it is proven to not add any business value and causes delays in delivery.

Discipline means you do what you say you are going to do. On many of the waterfall projects that I have experienced in my IT career, the early rigor (stubbornness) applied to the development process got abandoned when it was realized that the project was in trouble. Specs were left incomplete; testing time was traded for more development time; quality was sacrificed to meet the schedule. I have seen waterfall projects too often end up in this undisciplined state. Is more stubbornness to adhering to the waterfall process the answer? I don’t think so because this stubbornness is a big reason that waterfall projects get into trouble in the first place.

I have found the adaptability of Agile and the disciplines of XP (test driven design, continuous integration, relentless refactoring, early and often delivery of working code) to be a much more satisfying solution for effective software development. And so have my customers.

The Failure of Rotisserie Baseball Logic

Andrew -

I recently completed reading two books that before I started reading either of them I had no idea how profoundly related they were. The first book was The Logic of Failure. It has been on my reading list for quite a while, recommended by Rob, a former colleague and now friend. The second book, Fantasyland (about Fantasy Baseball), I picked up on a whim at the recommendation of a friend (Michael) while riding home with him on the train. (You’re smart enough to already see the connection between these two books, aren’t you…)

The Logic of Failure is a clinical study into the psychology of why people make bad decisions that ultimately lead to catastrophic failures. The disaster at Chernobyl is one example that he uses in the book.

To study people’s decision making processes the author created a computer simulation game of two complex adaptive systems (a village in West Africa and a small town in Germany) and asked the study participants to try to improve the lives of the people in the simulation through decision making. More often than not, the simulation ended up with the people in worse shape than before the participants ‘started meddling’. Poor goal setting, focusing on incidentals, not addressing problems soon enough, lack of experience in the domain and cynicism are among the reasons for failure identified and discussed at length for the bad decisions people were making.

The Logic of Failure was a hard book for me to read for a couple of reasons, the first being that it is a very clinical study of why people were making bad decisions that ultimately lead to failure. But it was also difficult to read as I clearly saw several patterns in my own decision making processes that have not served me well.

Well, after beating myself over my poor decisions making skills I decided that I wanted to pick up something to read that would be just for fun - try to lighten my mood a little. I needed something to read that was not work related, not a self-improvement book, not technical - just something for pure entertainment. Well, I hit the jackpot with Fantasyland.

Fantasyland is the story of a sportswriter’s first attempt at playing Rotisserie Baseball during the 2004 Baseball season. Sam Wallace from the Wall Street Journal joined Tout Wars, the premier Rotisserie Baseball league, for his inaugural season of Fantasy Baseball. With his in depth knowledge of baseball, ready access to players and coaches for their insights (using his press pass to get into each team’s club house) and two full time advisors he put on his personal payroll (and an occasional consult from an Astrologer) he was convinced that he could outdo the others in the league. It’s compelling reading as he takes you through the months of his decision making process in preparation for selecting his team during draft day and then toiling over managing the team through trades during the season. (And, it is a very funny book. My favorite line from the book is how he described feeling after being taken advantage of by a seasoned Tout Wars competitor in a lopsided trade: “He worked me over like a drunken chiropractor.”)

Well, with me associating The Logic of Failure with Fantasyland in this blog posting you probably have already concluded that Sam Wallace didn’t do well in his first season in Rotisserie Baseball and you would be correct: eighth place out of 12 teams. The same patterns in decision making documented in The Logic of Failure that led to failure were being made by Sam Wallace in Fantasyland. But the real brilliance of Fantasyland is the epilogue, on the last page and last paragraph of the book:

"Sam Wallace returned to Tout Wars in 2005 to draft a second incarnation of the Streetwalkers Baseball Club. With only two nights to spare on the evaluation of American League ballplayers, he arrived at the draft in New Your fully expecting to be thumped like a traffic cone. Six months later, he won..."

Andrew - once you get some experience in a domain, whether it be Baseball or Project Management, don't over-analyze the problems within the domain. Apply the sound fundamentals that you have learned and trust them work for you. Always be passionate about your domain and never become cynical over the effects of your decisions. Love what you do and success will follow.

"It's all about Version 2.0"

Andrew - don't take shortcuts, they don't payoff in the long run and often not in the short run either. That is what Dave Stanton, my CTO when I was at Sage IT Partners, was conveying when he said that good software development "is all about Version 2.0".

We have a phrase in my business for taking shortcuts. We call it technical debt. A software development project goes into technical debt by borrowing time by not allowing for quality work today in hopes of getting more functionality completed for the impending release. But we eventually will be forced to pay back the time, the technical debt, if the project makes it into production (and therefore into maintenance mode) or if the business chooses to invest in a Version 2 of the application. There is no getting around it. We either have to take the time to fix the stuff that has been done hastily in Version 1 or we will struggle with modifications to the application because of a fragile code base. Each takes time. This will result in either fewer features in the next release or a delay in releasing Version 2. Payback is a bit...oops, sorry, I'm your dad and I shouldn't talk like that in front of you.

I think we (IT Project Managers) have done a poor job of communicating the concept of technical debt to our business sponsors. We continue to get asked to accelerate schedules, add scope without changing deadlines, work longer hours or even stop writing tests for our code, all of which incurs technical debt. All we seem to do is complain about having to work harder which falls on deaf ears as our business sponsors are also being asked to work harder by their bosses.

But what if we were able to quantify technical debt? Can we put a dollar amount on a decision to increase scope in an upcoming release? Can we quantify how much it will cost by not having a suite of unit tests and functional tests supporting our code? Our business sponsors most likely have gone through a cost justification for the project so costs rather than effort would mean more to them.

The best metric that I have seen to date has been the waterfall methodology's quantification of the costs to a project by introducing changes in requirements late in the release cycle of a project. The graph below illustrates how the cost of a change request grows exponentially throughout the Software Development Lifecycle (SDLC).



This chart illustrates how a change request during Pre-Production can cost as much as 100x more to the project than if it would have cost the project by introducing it during requirements gathering. This high cost of change requests comes from:

• Needing to update documentation
• Needing to update acceptance tests
• Complexity of the change within an established code base, therefore needing more time
  
• Time to fix defects introduced to the code base as a result of changes
• Needing to re-run acceptance tests after the change has been implemented
  

In a waterfall managed project, it is seen as a success to defer a change request to a subsequent release. But one thing that is hidden in the chart above is that the cost of the change request that has been deferred to Version 2 in a waterfall project can be as expensive as incurring the cost late in the initial release. The only difference in Version 2 is that you have more time to put in the change. But it is still costly. All the things that make change requests costly late in a project are still there during development of Version 2. Therefore, Version 2 is often delayed or reduced in scope because of incurred technical debt inherent within the waterfall approach.

The Agile community has come up with a solution for technical debt - don't pay it. Keep your documentation light and write tests first. Continuously refactor and build/test the application early and often. What we haven't done well is quantify or document our successes to show how effective these practices are in releasing subsequent versions of an application.

One of the development teams at ThoughtWorks that I had the pleasure of serving as Project Manager and Iteration Manager (along with some talented folks from the client's IT staff) has recently delivered Version 2 of a successful application for a Fortune 50 company. With Version 2, the team went through a significant refactoring of the object model, added a significant amount of new functionality and kept the code base approximately the same size as Version 1. There were virtually no defects found during testing of Version 2. In other words, technical debt was not incurred during Version 1 and therefore the overall quality of the code was allowed to improve during Version 2 development. The client received the functionality they expected when they expected it. We owe this to the team's diligent application of XP development practices of test first, continuous integration, simple design, well written story cards, constant collaboration with the customer and fearless refactoring.

How Long is a Piece of String?

Good morning Andrew.

So, how long is a piece of string? Do you remember when Mom asked you that question last week? Your response was pretty funny - that quizical look with your head cocked to the side and your response of 'huh?' was rather amusing.

It's a pretty silly question, isn't it? But would you believe that questions like that are asked all the time regarding IT projects? And what is even more amazing is that people actually give answers to such vague questions and then they make important business decisions based on them.

Getting back to the string for a minute, if I show you a piece of string and ask you how long it is you can give me a pretty good estimate because you can see the string. If you want to do just a little more work you can get out your ruler and measure the string and give me a pretty precise answer. The frustration of the original question is that there is just not enough information to give a meaningful response.

So how does this question about the length of a string relate to me as a Project Manager? Well, in this analogy the string is my project and the length of the string is the cost and/or duration of my project. There is always an original estimate for the cost and duration of a project and as a PM it is my job to mitigate the risks of the project overrunning these estimates and to validate the accuracy of the original estimates. It is pretty well documented how inaccurate original project estimates are in my business so I won't belabor that point here. But the real problem is that we often don't know how bad our estimates are until way too late in the project.

So how do I know how my project is progressing if estimates can't be trusted? What I have learned as a Project Manager is to stop estimating and start measuring progress on my projects as soon as possible. How soon? Within weeks of the project start date.

The problem with using measurements rather than estimating is most projects are not set up for adequately measuring progress. For example, it is extremely difficult to get meaningful progress measurements during a requirements gathering phase of a project. How do you know when all requirements have been captured? How do you know that requirements won't change? Most often a project will have a requirements gathering phase that is timeboxed, say 3 or 6 months. But 100% complete of the requirements phase does not necessarily mean that all requirements gathering is done. If you end the requirements gathering phase on time just to statisfy the estimates or the project plan, your developers will end up burning their cycles during your development phase chasing down missing requirements, something that you probably did not include in your development estimates.

Another example of what some people think is a measurement is the 90% complete status on a task. This is rarely used as a measure of work complete but is just another estimate of work required to complete the task. And I have seen in my years as a PM that 90% complete often equates to about 50% work left.

Here are some suggestions on how to start measuring a project's progress and stop relying on estimates:

- Start measuring progress as soon as possible. Gather just enough requirements to have developers start coding and continue to gather requirements in a just-in-time fashion to keep developers productive;

- Prioritize requirements so you can work on the most important stuff first and get feedback from the users on the most important stuff right away. Don't wait for months to get their feedback and find out you have missed the mark. That will really blow your original estimates;

- Use working software as the only measure of the progress on a project (no more 90% completed tasks);

- Measure using smaller sized tasks. Smaller tasks give you better measures than larger ones and will also give you a larger sampling of measures that can be used for meaningful statistical evaluations. Smaller tasks are also completed sooner and therefore give you measurements sooner that you can begin to use to extrapolate to a meaningful project complete date.

Just remember this Andrew - estimates are predictions, often with a lot of unknowns and nobody gets all their predictions correct. Completed work is real and measurable and should be used to validate original estimates and then to adapt the plan to reality.