Software Development Methods

Building a House

Building a house is still one of the most common analogies I hear and use for discussing software development with customers.  The problem is that the architecture of the house is left out; and unless you’re a developer, it’s hard to understand why a stack of wireframes isn’t the comprehensive description of the system needed to provide accurate cost and schedule estimates.

These wireframes are like the diagrams a first-time home owner might sketch on the back of a napkin.  In order to know what must be built, and to determine a realistic sense of scope and cost, those rough ideas need to be developed into something much more substantial by a professional.  For our house building scenario, this would involve hiring an architect to transform those ideas into a set of drawings which include such things as plumbing, heat and air conditioning ducts, electrical access panels, drains, the appropriate foundation and structural support elements, and so on.  The customer asks for a “nice room with a window looking to the back yard”, but all of these other hidden structural elements must be taken into consideration as well.

Unfortunately, determining the actual cost and size of a large project with any accuracy requires having this detailed design; and worse, doing this detailed design can take weeks or even months.  By the time development is underway, the customer will inevitably change many of the requirements anyway.  In fact, the very process of evolving such a large and detailed design brings many issues to the surface to discuss, debate, and make decisions about.  It’s easy to see how this process could carry on for quite a while, and there are still changes in the market to consider and adapt to, both during and after requirements definition.

The house building analogy is useful, but it is also misused.  To be fair, a typical waterfall software project put into house building terms would go like this.

1. Customer sends sketches to Architect and asks for plan including schedule and cost.
2. Architect works with Customer, being paid for his services to arrive at detailed design, which provides the project schedule and cost.
3. Builder works with the Architect’s plan to build exactly what’s in the requirements.
4. Days or weeks later, Customer makes changes to requirements.  This requires the Architect to get involved again, update the plan, work with the Builder to make physical adjustments, which may include tearing down existing structures and modifying others.  Plan changes need to be approved by government (verified with tests), documentation updated, and Customer agrees to pay for the changes.
5. * Customer requests to now connect their house to other houses near by, the shapes and layouts of which may change over time.
6. Repeat #4.

However, this is completely at odds with how homes are normally built.  People typically choose a previously-implemented design, and only customize superficial features like countertops, cabinets, floors, and railings.  Houses from this plan have been built before, and the labor and materials cost are known from previous experience.

Building software is usually more like constructing something that’s never been built before: the first sky scraper, the Golden Gate Bridge, or the Hoover Dam.  The requirements are unique, the pieces have never been assembled in such a way before, and there’s an inherent level of risk in creating something new.  When this is the case, the Customer needs the services of an Architect, not just a Builder who stamps out deliverables in a cookie-cutter style.

Waterfall methodologies have been dramatically in decline over the years in favor of so-called Agile methods.  There are several problems with a Waterfall approach:

• Because of requirements churn, attempting to write all the requirements up-front is incredibly wasteful.  Your team will end up doing a lot of design work that is eventually thrown away.  When I hear about churn numbers as high as 80%, I shudder to think of all the time and money spent refining details that will never make it into the production system.
• It tends to take a blind approach to requirements change, working under the assumption that change is bad and should be prevented.  Many waterfall methodologies erect barriers to change, making Customers jump through hoops to push changes through.
• Customers typically don’t want to pay for this architecture phase.  They want to pay for actual work on their system, and the cost of defining the actual scope and cost of their system seems steep.  “Why should I pay you $<large number> just for you to tell me how much you’ll charge me for the final system?”

An Agile Solution: Efficiency and Uncertainty

The Agile community emerged largely as a reaction to the disappointing results of Waterfall projects.  Customers were sick of schedules and budgets that were consistently and grossly overrun (despite all the up-front design), and wanted to more easily change requirements without being put through the wringer to do so.  Developers were tired of spending their valuable time writing detailed design documents that kept changing throughout the project, and the burden to update that documentation as the project goals changed focus or priorities shifted made development a very inefficient and slow process.

Essentially, as is often the case, a trade-off was made.  Some stakeholders realized that the predictability they were attempting to achieve by spelling out everything in detail was delusional: they were aiming at a moving target regardless of their efforts to pin things down.  Project costs rose precipitously because the scope of the system grew.  They needed a process that acknowledged and accepted these dynamic project forces instead of one that denied and suppressed them.  They needed a process that was efficient, and they gave up the illusion of certainty to get it.

The cone of uncertainty is one of the most valuable tools for visualizing and understanding uncertainty in development projects.  From that article:

Research has found that the accuracy of the software estimate depends on the level of refinement of the software’s definition. The more refined the definition, the more accurate the estimate. The reason the estimate contains variability is that the software project itself contains variability. The only way to reduce the variability in the estimate is to reduce the variability in the project itself.

An important—and difficult—concept is that the Cone of Uncertainty represents the best case accuracy it’s possible to have in software estimates at different points in a project. The Cone represents the error in estimates created by skilled estimators [which is up to 16x error at the beginning of a project]. It’s easily possible to do worse. It isn’t possible to be more accurate; it’s only possible to be more lucky.

Commitments made too early in a project undermine predictability, increase risk, increase project inefficiencies, and impair the ability to manage a project to a successful conclusion.

Meaningful commitments are not possible in the early, wide part of the Cone. Effective organizations delay their commitments until they have done the work to force the Cone to narrow. Meaningful commitments in the early-middle of the project (about 30% of the way into the project) are possible and appropriate.

Or as noted by the MSDN documentation for Composite Client Application Guidance:

Application requirements can change over time. New business opportunities and challenges may present themselves, new technologies may become available, or even ongoing customer feedback during the development cycle may significantly affect the requirements of the application. Therefore, it is important to build the application so that it is flexible and can be easily modified or extended over time.

Agile methods have always dealt with small iterations and early, frequent builds to evolve product functionality, but the Lean Development method especially has an explicit focus on making meaningful commitments at the latest possible responsible moment.  From Wikipedia:

As software development is always associated with some uncertainty, better results should be achieved with an options-based approach, delaying decisions as much as possible until they can be made based on facts and not on uncertain assumptions and predictions. The more complex a system is, the more capacity for change should be built into it, thus enabling the delay of important and crucial commitments. The iterative approach promotes this principle – the ability to adapt to changes and correct mistakes, which might be very costly if discovered after the release of the system.

While the methodology pundits debate the relative merits of development processes, Tom DeMarco, author of the fantastic books Peopleware and Controlling Software Projects, challenges us to consider the value of our projects and to re-evaluate elaborate control measures in light of the bigger picture.  From his article:

To understand control’s real role, you need to distinguish between two drastically different kinds of projects:

• Project A will eventually cost about a million dollars and produce value of around $1.1 million.
• Project B will eventually cost about a million dollars and produce value of more than $50 million.

What’s immediately apparent is that control is really important for Project A but almost not at all important for Project B.  This leads us to the odd conclusion that strict control is something that matters a lot on relatively useless projects and much less on useful projects.  It suggests that the more you focus on control, the more likely you’re working on a project that’s striving to deliver something of relatively minor value.

This adds additional credence to the idea of agile development, since the Big Design Up-Front (BDUF) of Waterfall is obsessed with detailed definitions to make accurate schedule promises.  If the business model is sound, lots of control isn’t needed.  It’s better to navigate with a good compass (the product vision) than an incorrect and constantly changing map.

So What Will I Get?

At the end of the day, customers want to know what they’re going to get.  There are tradeshow and conference dates to meet, beta and v1.0 launch targets to hit, release cycles to determine, and so on.  Ultimately, some up-front architecture and design is usually required.  But instead of trying to pin down every detail, it’s actually best in terms of efficiency to share the vision, provide direction, and define the technical scope only in a loose way.  This functional scope definition should reflect the reality of the cone of uncertainty at that point in the development process.

At the beginning of the project, items to include in the project should be broadly defined, having a granularity appropriate for the actual uncertainty.  As the project progresses and work is done, the scope becomes better defined, risks are determined benign or neutralized, and the cadence of that specific team working on that customer’s specific product can be measured.

This is successfully done with Lean and Kanban approaches, providing an empirical method for measuring productivity.  Planning can then be adjusted to match the regular rhythm of production that develops.  This creates a semblance of real predictability.  You can’t predict any one thing with absolute certainty, but if you can measure the general rhythm at which work is done, you will very quickly learn to recognize how long different kinds of tasks actually take.  Because you’re focusing on measured results rather than predictions, it’s easier to normalize task size and estimate larger functional units.

If you’re interested in learning more about Lean development, Mary and Tom Poppendieck’s book Implementing Lean Software Development is a fantastic background and explanation of the Lean process.  They also have a new book coming out this November, exploring the same topics more deeply.

When a good process is followed by hard-working, talented developers, the customer will end up with what they need to satisfy their business goals, often in surprising ways, ways that will be radically different from what they initially asked for.

What Can Be Promised?

The customer has their vision, beginning with the end in mind as they do; and they are asked to break it apart, prioritize individual features, and stretch their end-goal image across a long and expensive tunnel that separates them from their goal.  They usually want some kind of promise of what they’re going to get at the end, even though they’re going to keep adding and changing features throughout its development.

What can be promised?  It would be dishonest to promise customers what they’re asking for, unless they promise not to change their minds as the project progresses, but the only certainty in software development is that requirements will change.

Development isn’t about creating flawless, predictable plans that eliminate business risk.  Development is an unpredictable journey to create tremendous value.  To achieve great reward, risk is a necessary part of the equation.  Instead of coming up with creative ways to hide real project risk, it’s best to have a solid process for detecting and managing that risk.

Important Professional Skills for Software Architects

I was recently asked which professional skills were most important for a software architect, and the answers I came up with, I believe, apply to professionals in general.  Some of the following descriptions are slanted toward the software architect, but can be applied quite easily to other professions.

Customer advocacy.  First and foremost, we have an ethical obligation to our clients to discover and represent their best interests.  There are unscrupulous consultants who will milk contracts for as long as they can for their own financial survival, and this is a serious problem because it creates mistrust in our industry and problems for those who are honest.  As this problem may be outside of our immediate sphere of influence, I would also point out that even with the best intentions, those who are honest may not be serving the customer’s real needs due to hyperfocus on the detail and technology, and unfamiliarity with the requirements or lack of focus on business needs.  Some of this can only come with experience, but I believe we can all learn what our limitations are and to admit when we aren’t seeing enough of the big picture to make a confident recommendation.  This is an important focus of systems architecture organizations such as IASA and WWISA.  Dan Appleman in a recent Hanselminutes podcast episode covers this responsibility of the Software Architect very well.  I had the opportunity to meet Dan briefly at DevConnections, and consider him to be a great role model.

Articulate and organized written and verbal communication skills.  Communication skills are the glue that hold teams together and allow a single coherent vision of a project or product to be shared.  The most important of these skills is listening, or observation in general, as decisions can only be as good as the information they’re based on.  Communication skills can be further broken down for more intense self-examination and improvement: the ability to compose meaningful and relevant messages; delivery of those messages via writing, impromptu speaking, and prepared presentations all require different skills; and observance of social protocol to appear responsive, considerate, etc.  When I hear people saying that someone has “poor communication skills”, they are most often referring to a failure to observe social protocol, not that the person isn’t capable of composing and articulating their thoughts well.

Negotiation and conflict resolution.  Everything that involves communication contains the potential for conflict.  Knowing how to negotiate can prevent daily micro-conflcits from escalating out of control.  When we understand that conflict is normal and comes from the interaction of different perspectives, requiring a dynamic, shifting dance of give and take, we can embrace and work with it, using that energy and the diverse perspectives involved to our advantage, fueling our creative processes instead of letting them degenerate into unhealthy ego battles.

Problem-space modeling and creative problem solving.  The effectiveness of the solutions we put in place depend primarily on the way we view the problem.  Modeling a problem (seeing) is a very creative process, and the ability to be flexible and think outside the box when looking at a situation can lead to much simpler and more effective solutions.  Too often, designers assume the first model they imagine “is” the problem, instead of one of many ways to view it, and build their systems with this limiting and often dangerous assumption.  A willingness to play and refactor ideas, coming up with alternatives through brainstorming, and synthesizing conflicting approaches, all add to one’s repertoire of visualization techniques, and therefore problem-solving skills.

Risk identification and management.  Software development (as with business in general) involves a great deal of risk, with pitfalls around every corner.  Incorporating new tools, processes, and team members, solving problems in unfamiliar domains, company acquisitions, etc., can make a project miss schedule and budget targets.  Identifying the risks involved requires project experience and good observation skills.  Managing them requires organization and a proactive, prioritized approach, as well as planning for several alternatives and mitigations.  We start with awareness, then we identify, and finally we can manage and plan around them.

Critical thinking and analysis.  Complex software systems are designed and developed too frequently with a casual, shoot-from-the-hip mentality that endangers projects.  By thinking through issues with a critical and analytical approach, we can reduce risk substantially and ensure success.  The greatest barrier to proper critical thinking is impatience, the urge to jump to conclusions to avoid thought.  As David Allen says in Getting Things Done, “You have to think about your things more than you realize, but not as much as you’re afraid you might.”

Strategic (long-term) and tactical (short-term) project planning.  Short-sighted design and development, and lack of planning, leads to systems that are incapable of growing with or adapting to changing business needs.  Keeping long-term goals in mind ensures longevity and relevance.  This depends, of course, upon the resources and leadership of the company you’re working for.  You may be working on a project with a short lifetime, or your boss may not be persuaded to invest in strategic moves now to save time and money later.  But for most projects, development costs are too great to be wasted on short-sighted expediency.  In any case, long-term and short-term priorities and forces must be balanced against one another along the way.  Every decision and trade-off will have consequences down the line.

Continuous research on trends and technologies.  Long-term strategic planning can’t be effective without knowing the future state of the industry or its many technologies.  This is all about “knowing the territory” as Sun-Tzu has described in The Art of War.  We must often plan the future state of our software systems to take advantage of technologies that aren’t yet available.  This requires keeping ourselves in the loop and on top of our game at all times.  This is difficult in a world that is expanding exponentially, and often requires that we find a niche specialization both to differentiate ourselves and to provide more focused value.

Whereas a developer is concerned with doing things right and being efficient, an architect must focus on doing the right thing from a larger point of view.  As Peter Drucker writes, “Efficiency is doing things right; effectiveness is doing the right things.”  An architect must be effective.

Or as paraphrased by the VisualSVN tag line (quite brilliantly): “Right thing.  Done right.”

Couldn’t have said it better myself.

Tags: Risk+Identification, Risk+Management, Critical+Thinking, Analysis, Strategy, Long+Term+Planning, Continuous+Research, Ethics, Customer+Advocacy, Communication+Skills, Professional+Skills, Conflict+Resolution, Negotiation, Problem+Modeling

Setting & Managing Goals – Modeling the Approach

I’ve been thinking about this question: how do I set and manage goals?  What is the most effective way?  If there were a tool to maximize our effectiveness in achieving goals, how would it operate?

My first thought is that it has to start with a definition of goals.  What are my goals?  Although it would be valuable to examine my personal goals, I will narrow the focus for the moment and consider only my career goals.  So what do I want to accomplish?  Since I believe that being satisfied with my work will maximize the value I provide, maximizing my career satisfaction sounds like a fair answer.  The natural avenue to explore is thus my interests, abilities, etc.  I need stimulation, love a challenge, am a proficient software developer, and after some reflection, my answer is: to continuously advance the science and art of software development.

Once I can identify a guiding principle, many goals fall into place.  Successful software development, like other occupations, requires many skills: communication and collaboration, analysis and critical thinking, risk management, strategic planning, creative problem modeling, organization and prioritization of work, continuously learning new technologies, etc.  Improving the effectiveness of each of these creates yet more goals, which keep getting broken down until we identify actions that we can execute immediately.  This is the top-down approach, or “beginning with the end in mind”.

Tactically–from the bottom up–we adjust to the changing landscape, taking advantage of whatever opportunities come along, all the while keeping in mind our long-term vision.  While the high-level goals are more general and universally applicable for career paths, short-term goals diverge dramatically.  While business analysts and software developers may both benefit by learning to model problems creatively, the two professionals will use very different methods to represent their domains, and will implement those designs in completely different ways, utilizing different specific tricks to overcome different specific obstacles.

One question that arises: how can I achieve the optimal balance of focus between long-term and short-term goal-seeking activity?

Also, if I were to examine my goals in great detail, or if I had to track a complicated set of goal seeking behavior across a team, what would be the best way to model this to take full advantage of the information?  Perhaps by modeling this goal-seeking-achieving behavior, we can arrive at some insightful answers.  I’ve thought about this before, specifically in terms of tracking the goals of developers building a software system, but I want to model it in a way that is generic enough to be reused in other domains outside of software development because there’s a massive core of shared goal-oriented behavior that most industries have in common.

In development, there is the idea of a software defect, or bug.  Typically a “bug” gets saved in a bug-tracking database, and when the bug is fixed, it gets checked off in the list, or in some other way marked as “resolved”.  On a small, simple project, this may be sufficient.  However, one thing I’ve seen in larger projects is bugs being closed and then reopened a lot because the real cause wasn’t understood.  The bug recorded in the system was really just a symptom of a larger problem.  Sometimes, a single bug can cause multiple symptoms, such that by making one fix, several unrelated “bugs” are closed all at once with no time associated with them.

If we really want to deeply understand the dynamics of problem solving, we might model the situation with a bit more epistemological integrity.  For example, in the diagram below, you’ll see that some problem (which is a cause) produces one or more symptoms (which are perceptions).  In terms of software, a software defect causes some misbehavior.  Someone who is not a developer, such as a support or help desk person, or perhaps even a customer, can save information about the misbehavior without knowing what it’s cause is.  They will likely be unqualified to know that anyway.  Unhandled bug reports (misbehaviors) can be pulled out of a queue by a developer, who can reproduce and analyze the problem, do some troubleshooting, and upon figuring out what’s wrong, create a software defect record and save it, associating the misbehaviors with it.  And if there’s a simple way to tell the system “and here’s another misbehavior that is probably also caused by this problem”, all the better (such as an attribute describing the degree of certainty).

Now that we’ve identified one or more misbehaviors, and identified their root cause, we are now in a good position to recommend a solution.  There may be more than one way to solve the problem.  So the engineer (in this case) can create an set of ordered actions describing approximately how the problem should be solved.  As the problem definition matures and evolves, which may involve other engineers (and possibly some re-design meetings depending on the scope of the problem), other alternatives can be proposed, recorded, and evaluated.  Once a solution is selected and approved, those actions become task items which are assigned to one or more engineers.

This scales up better for tracking detailed information about complex projects and scenarios, but it would also be nice if such a system could work on a small scale.  If an engineer defines a symptom, he should be able to quickly and effortlessly move into problem definition (if the root cause is obvious and he has permission to do so), and then just as quickly define one or two steps for correction, which get checked off ten minutes later when the problem is fixed and the solution verified.  It should be possible to present a user interface that can glue all of these pieces together in a simple way visually, and then expand into a more complex visual structure as the scope of the problem evolves.

In the same way that bugs can be tracked, use cases and requirements could also be defined, executed with a set of actions, and verified.  And while there are some specific pieces of information to track in software use cases versus more general business opportunities, there is enough overlap that mechanisms could be shared not only in the persistent data model, but also in a creatively-designed user interface.  Ideally, the same system should be usable by many different groups in an organization, including executives for strategic planning, lower level business managers and analysts, as well as developers and others in the IT department.  Imagine the level of business intelligence that would be possible if an organization’s strategies could be traced from the CEO or CIO all the way down to specific features of an in-house, custom software project, in a way that made sense for each team.

Ultimately, this model mimics the brain’s own organization as well as the relationship between cause and effect.  We become aware of problems (negative, reactive perceptions) and opportunities (positive, proactive perceptions).  In analyzing problems, we converge on a cause, whereas in analyzing opportunities, we diverge into a set of goals.  But whether we’re talking about a problem (or really, a desire to rid ourselves of that problem) or a goal, we’re talking about intention: a cause for action.  Our brains perceive the world, turn that raw data into information, process that into intention, and the output is always action.  The better this functional circle operates, the more effective we’ll be at setting and achieving goals.

Another important aspect of goal-achieving behavior is risk management.  Where, then, do risks fit into this picture?  My first reaction is to think of risks as problems, where the problem has some attribute indicating that it hasn’t happened yet, or where one could specify the conditions under which it would occur.  Different sets of actions would be plans for avoidance or mitigation, some of which can’t be executed until some condition is met.  Or is a risk a different type of thing altogether?

Tags: Problem+Modeling, Software+Development, Goal+Setting