Animate.NET: Fluent Animation Library for Silverlight & WPF

Overview

The basic idea—in Silverlight and WPF—that an animation is just a change in some DependencyProperty over time is simple and powerful.  However, at that level of detail, the API for defining and managing complex animations involves writing a ton of code.  There are code-less animations, of course, such as those created in the Visual State Manager, but when you want to perform really dynamic animations, state-based animations can become impractical or outright impossible.

In response to this, I’ve published my fluent-style code-based animation library for Silverlight and WPF on CodePlex at http://animatedotnet.codeplex.com.  This is an API for making code-based animations intuitive and simple without having to write dozens or even hundreds of lines of code to create and configure storyboards, keyframes, perform repetitive math to calculate alignment, rotation, and other low-level details that distract one from the original purpose of the animation.  In one example, I counted over 120 lines of standard storyboard code, and with the abstractions and fluent API I’ve come up with, reduced that down to half a dozen lines of beautiful, pure intent.  As a result, it’s much more readable and faster to write.

I was initially inspired by Nigel Sampson from his blog article on building a Silverlight animation framework.  The code on his site was a good first step in creating higher-level abstractions, going above DoubleAnimation to define PositionAnimation and RotationAnimation, and I decided to build on top of that, adding other abstractions as well as a fluent-style API in the form of extension methods that hide even those classes.

Concepts

All Animate.NET animations derive from the Animation class which tracks the UI element being modified, the duration of animation, whether it has completed, and fires an event when the animation completes.  It manages building and executing the Storyboard object so you don’t have to.

Subclasses of Animation currently include OpacityAnimation, PositionAnimation, RotationAnimation, SizeAnimation, TransformAnimation, and GroupAnimation.  TransformAnimation is the parent class of RotateAnimation, and in the future ScaleAnimation and TranslateAnimation may also be included.

GroupAnimation is special because it allows you to combine multiple animations.  These groups can be nested and each group can include a wait time before starting (to stagger animations).

The Animate static class includes all of the extension methods that make up the fluent API, and the intention is for this to be the master class for building complex group animations.  Most of these methods come in pairs: you can RotateTo a specific angle or RotateBy relative to your current angle; MoveTo a specific location or MoveBy relative to your current position, etc.

Here’s the list so far:

• Group and Wait
• Fade, FadeIn, FadeOut, and CrossFade
• RotateTo and RotateBy
• ResizeTo and ResizeBy
• MoveTo and MoveBy

Examples

Animate.NET can best be understood and appreciated with examples.

Basic Animations

Let’s say you want to resize an element to a new size.  Normally you’d need a storyboard and two DoubleAnimations: one for x and another for y, and for each you’d need to set several properties.  With Animate.NET, you can define and execute your animation beginning with a reference to the element you want to animate:

var rect = new Rectangle()
{
    Height = 250,
    Width = 350,
    Fill = new SolidColorBrush(Colors.Blue)
};
MainStage.Children.Add(rect);
rect.SetPosition(50, 50);

rect.ResizeTo(150, 150, 1.5.seconds()).Begin();

Only a single line of code, the last one, is needed to resize the rect element.

Note the call to Begin.  Without this, the ResizeTo (and all other fluent API calls) will return an object that derives from Animation but will not run.  We can, if needed, obtain a reference to the animation and begin the animation separately, like this:

var anim = rect.ResizeTo(150, 150, 1.5.seconds());
anim.Begin();

This allows us to compose animations into groups and manipulate animations after they’ve started, and is very similar to how LINQ queries are composed and later executed.

You’ll also notice the use of several other extension methods:

• SetPosition – sets Left and Top currently.  In future versions, you’ll be able to define a registration point for positioning that may be located elsewhere, such as the center of the element.
• seconds() – along with milliseconds, minutes, etc., allows you to specify a TimeSpan object more fluently.  I saw this in some Ruby code and loved it.  If only the C# team would implement extension properties, it would look even cleaner (eliminate the need for parentheses).
• Center() and GetCenter() – centers an element immediately, and gets a Point object representing the center of the object respectively.  Not used in these examples, but worth mentioning.

Group Animations

Next I’ll show an example of a group animation using the Animate class’s Group method:

Animate.Group(
    rect.RotateBy(rect.GetCenter(), -90, 1.seconds()),
    rect.FadeOut(1.seconds())
    ).Begin();

This group animation contains two child animations: one to rotate the rectangle 90 degrees counterclockwise, and the other to fade the rectangle out (make it completely transparent).  The method takes a params array, so you can include as many animations as you like.

Because the animations listed are peers in the group, they begin running at the same time.  Often you will want to stagger animations, however.  You can accomplish this with the Wait method, which is the Group method in disguise (it simply includes an additional TimeSpan parameter).

Animate.Group(
    rect.RotateBy(rect.GetCenter(), -90, 1.5.seconds()),
    rect.FadeIn(0.5.seconds()),
    Animate.Wait(1.seconds(),
        rect.FadeOut(0.5.seconds())
        )
    ).Begin();

This animation rotates the rectangle for 1.5 seconds.  During the first 0.5 seconds, it fades in; during the last 0.5 seconds, it fades out.  Only one element, rect, was used in this example, but any number of UI elements can participate.

Animations can be nested and staggered to arbitrary complexity.  Because all animations derive from the Animation class, you can write properties or methods to encapsulate group animations, and assemble them programmatically before executing them.  Because all the ceremony of storyboards and keyframes is abstracted away, it’s very easy to see what’s happening in this code in terms of the end result.

Method Chaining

One of the benefits of a fluent API is the ability to chain together methods that modify a primary object.  For example, the Animation class defines a WhenComplete method that can be used to respond to the completion of an animation.  In the samples project on CodePlex, I create new UI objects at the beginning of each animation, and remove them afterward:

rect.ResizeTo(150, 150, 1.5.seconds())
    .WhenComplete(a =>
    {
        Thread.Sleep(2000);
        MainStage.Children.Remove(rect);
    })
    .Begin();

I pause for a couple seconds after displaying the final result before removing that object from its container.

Extension methods will be used more in the future for this library.  Uses will include modifying the animation to apply easing functions, responding to collision detection (by stopping or reversing), and so on.  This might end up looking something like:

rect.ResizeTo(150, 150, 1.5.seconds())
    .Ease(EasingFunction.Cubic(0.5))
    .StopIf(a => Animate.Collision(GetCollisionObjects()))
    .Begin();

Feedback and Future Direction

I’m releasing this as a very early experiment, and I’m interested in your feedback on the library and its API.

What kind of functionality would you like to see added?  Do the method names and syntax feel right?  What major, common animation scenarios have I omitted?  What other kinds of samples would you like to see?

Download the library and samples and give it a try!

AssignAsync Extension Method for ADO.NET Data Services

ADO.NET Data Services is a rapidly evolving set of tools that provides data access to remote clients through a set of REST-based services.  The Data Services Client Library for .NET performs the magic of translating your Linq queries to URLs and passing them to the data service back-end, as well as retrieving results and hydrating objects in the client to represent them.

After running into a number of problems with the current CTP of RIA Services (see my article), I decided to fall back on Data Services to provide data access in my newest project.  Data Services has the advantage of allowing you to write fairly normal Linq queries against Entity Framework entity sets, and entity data models can reside in a dedicated data model assembly (instead of requiring them to be part of the web project).

One of the differences that remain when using Data Services in Silverlight—as opposed to accessing an Entity Framework ObjectContext directly—is that Silverlight doesn’t allow asynchronous calls.  So code like this, which would force a synchronous call (with FirstOrDefault), will fail in Silverlight:

var result = (from p in context.Properties
              where p.Required
              select p).FirstOrDefault();

This forces us to adopt some new patterns for data access.  This isn’t a bad thing, however.  And it’s an inevitable transition we’re making to asynchronous, concurrent program logic.

Here’s a typical example of querying data with Data Services in Silverlight:

var RequiredProperties = from p in context.Properties
                         where p.Required
                         select p;

var dsq = RequiredProperties as DataServiceQuery<Node>;
dsq.BeginExecute(ar =>
    {
        var result = dsq.EndExecute(ar);
        // do something with the the result
    }, null);

When using a lambda statement for brevity, the syntax isn’t too bad, but the pattern gets a little more involved when you include error handling logic.  If EndExecute fails, you’ll need the ability to perform some compensating action.

So what I’ve done to keep my client code simple is to define an extension method called AssignAsync that encapsulates this whole pattern.

public static class DataServicesExtensions
{
    public static void AssignAsync<T>(this IEnumerable<T> expression,
        Action<IEnumerable<T>> Assignment,
        Action<Exception> Fail)
    {
        var dsq = expression as DataServiceQuery<T>;
        dsq.BeginExecute(ar =>
            {
                IEnumerable<T> result = null;
                try
                {
                    result = dsq.EndExecute(ar) as IEnumerable<T>;
                }
                catch (Exception ex)
                {
                    Fail(ex);
                    return;
                }
                Assignment(result);
            }, null);
    }
}

This enables me to write the following code:

var RequiredProperties = from p in context.Properties
                         where p.Required
                         select p;

RequiredProperties.AssignAsync(result => properties = result,
    ex => Debug.WriteLine(ex));

In other words: if the query succeeds, assign the result to the properties collection; if it fails, send the exception object to Debug output.  Either action can be used to send signals to other parts of your application that will respond appropriately.  Instead of Debug.WriteLine, you might add the exception object to some collection that triggers an error dialog to appear and your logging framework to record the event.  Instead of assigning the result to a simple collection, you could convert it to an ObservableCollection and assign it to an ItemsControl in WPF or Silverlight.  Anything is possible.

As I explore Data Services further, I will be looking for ways to share query and other model-centric logic between Silverlight and non-Silverlight clients.  I suspect that the same asynchronous patterns can be used in non-Silverlight projects as well, and that those projects will benefit from this query style.

Problems with RIA Services (Feedback for July 2009 CTP)

RIA Services (new home page) is a collection of tools and libraries for making Rich Internet Applications, especially line of business applications, easier to develop.  Brad Abrams did a great presentation of RIA Services at MIX 2009 that touches on querying, validation, authentication, and how to share logic between the server and client sides.  Brad also has a huge series of articles (26 as I write this) on using Silverlight and RIA Services to build a realistic application.

I love the concept of RIA Services.  Brad and his team have done a fantastic job of identifying the critical issues for LOB systems and have the right idea to simplify those common data access tasks through the whole pipeline from database to UI controls, using libraries, Visual Studio tooling, or whatever it takes to get the job done.

So before I lay down some heavy criticism of RIA Services, take into consideration that it’s still a CTP and that my scenario pushes the boundaries of what was likely conceived of for this product, at least for such an early stage.

Shared Data Model with WPF & Silverlight Clients

The cause of so much of my grief with RIA Services has been my need to share a data model, and access to a shared database, across WPF as well as Silverlight client applications.  Within the constraints of this situation, I keep running into problem after problem while trying to use RIA Services productively.

The intuitive thing to do is: define a single data model project that compiles to a single assembly, and then reference that in my Silverlight and non-Silverlight projects.  This would be a 100% full-fidelity shared data model.  As long as the code I wrote was a subset of both Silverlight and normal .NET Frameworks (an intersection), we could share identical types and write complex validation and model manipulation logic, all without having to constrain ourselves to work within the limitations of a convoluted code generation scheme.  Back when I wrote Compact Framework applications, I did this with great success despite the platform gap, and I didn’t have anything like RIA Services to help.

Incompatible Assemblies

Part of the problem arises because Silverlight assemblies are incompatible with non-Silverlight assemblies.  A lot of what RIA Services is doing is trying to find a way around this limitation: picking up attributes and code files from one project and inserting that code into the Silverlight project with a build action.  This Visual Studio “magic” has been criticized for its weakness in dealing with multiple-solution systems where Visual Studio can’t update the client because it’s not loaded, and I’ve heard there’s work being done to address this, but for my current needs, this magic aspect of it isn’t a problem.  The specifics of how it works, however, are.

Different Data Access APIs

Accessing entities requires a different API in Silverlight via RiaContextBase versus ObjectContext elsewhere.  Complex logic in the model (for validation and other actions against the model) requires access to other entities and therefore access to the current object context, but the context APIs for Silverlight and WPF are very different.  Part of this has to do with Silverlight’s inability to make synchronous calls to the server.

In significantly large systems that I build, I use validation logic such as “this entity is valid if it’s pointing to an entity of a different type that contains a PropertyX value of Y”.  One of my tables stores a tree of data, so I have methods for loading entire subtrees and ensuring that no circular references exist.  For these kinds of tasks, I need access to the data context in basic validation methods.  When I delete nodes from a tree, I need to delete child nodes, so update logic is part of the model that needs to be the same in every client.  I don’t want to define that multiple times for multiple clients.  I like to program very DRY.  In other words, I find myself in need of a shared model.

RIA Services doesn’t provide anything like type equivalence for a shared model, however.  Data model classes in Silverlight inherit from Entity, but EntityObject in WPF.  In the RIA Services domain context, we RaiseDataMemberChanged, but in a normal EF object context, we need to ReportPropertyChanged.  In WPF, I can call MyEntity.Load(MergeOption.PreserveChanges), but in Silverlight there’s no Load method on the entity and no MergeOption enum.  In WPF I can query against context.SomeEntitySet, but in Silverlight you would query against context.GetSomeEntitySetQuery() and then execute the query with another method call.

This chasm of disparity makes all but the simplest shared model logic impractical and frustrating.  The code generation technique, though good in principle, keeps getting in the way.  For example, I have both parameterless and parameterized constructors in my entity classes.  This works great in my WPF client, but when this code is synchronized to my Silverlight client, I get an error because the Silverlight-side entity class is generated in two parts: in the hidden partial class, a parameterless constructor is generated which calls partial method OnCreated; and in the visible partial class, the constructor method I defined on the server is dumped into another file, so I have duplicate constructors.  If I remove the parameterless constructor from the server side, I get an error because my entity class requires a parameterless constructor (and defining a non-default constructor effectively removes the default one from the resulting type unless it’s explicitly defined).  I thought I could define the partial method OnCreated and put my construction logic in there, but the partial method is only defined on the client side.  That means sharing construction logic consists of copying and pasting the OnCreated method across the various clients—far from an ideal solution.

Entity Data Model Required to be in Web Project

Another strategy I attempted was to define the .edmx file and my partial class extensions in a class library, and then reference that from the web project.  I could define the LinqToEntitiesDomainService<MyDataContext>, but sharing entity class code (by generating code in the Silverlight project) isn’t possible unless the .edmx file and partial class extensions are defined in the web project itself.  This would mean that my WPF client would have to reference a web project for data access, which by itself seems wrong.  (Or making a copy of the data model, which is worse.)  It would be better for the WPF client to talk to the same domain service as the Silverlight client, but RIA Services doesn’t give you an option to link that web project to a non-Silverlight project, so again I ran into a brick wall.

So Don’t Do That

The kind of advice I’m getting for this is, “so don’t do that”.  In other words, don’t write complex validation logic in the model or otherwise try to access the data context; don’t write parameterized constructors; don’t aim for 100% type fidelity across all endpoints of a system; don’t try to share data models with Silverlight and non-Silverlight projects, etc.  But I see the potential for RIA Services, so I have to push for these things unless I hear really convincing arguments against them (or compelling alternatives).

Conclusion

The fact that there are different data contexts and data item definitions within those contexts imposes a burden on the developer to use different techniques for each environment, and creates challenges for centralizing data model logic and reusing equivalent logic across different kinds of clients.  My gut feeling is that RIA Services in its current form has some fundamental design flaws that will need to be addressed, taking into consideration systems with a mix of Silverlight, WPF, and other clients, before it becomes a truly robust data access platform.

Phidgets Robotics Project – Source Code Posted

By request, I’ve made available the source code to my Wii remote controlled pan-tilt camera system using Phidgets components.  See my original article here.

Warning: It’s messy in there.  Don’t look to this as a good example of how robotics applications should be developed.  There are far better patterns and libraries to use.  This was merely my “get it up and running as quickly as possible” approach, to prove out the concept of getting all of the components to work together correctly.

Enjoy!

The Wonders of Aruba

 

This morning, after being awoken at 5:30am by a rooster living nearby, I went for a walk to the northern tip of the island where beautiful homes are surrounded by lush tropical flowers and various palm trees, ferns, and cacti.  Clouds with serious character muddied the early morning sky, and large birds hovered playfully in the air above the beautiful homes on J. E. Irasquin Blvd—not covering any ground, simply enjoying the feeling of the strong ocean wind, gliding without effort or purpose, hovering in place just above the tallest trees.

It’s surprising to me that humans consider such aimless delight a luxury.  I’m in Aruba for the month September in part because I disagree; I think from time to time, it’s an absolute necessity to stay sane and keep a healthy perspective and sense of balance.  When so many of our moments are goal-directed and serious, and as Americans we have less time off work than virtually every country on Earth, it’s only a matter of time before the intelligence of our own bodies revolts against us in protest, a petition against the undue stress and unrealistic expectations we often have of ourselves.

An hour later, I was following the winding road past Arushi beach, onto the part of Aruba that isn’t polluted much by light at night, where you can see thousands of stars and galaxies and the colorful dust of the Milky Way.  The road curves back and forth several times and climbs steeply toward the California Lighthouse where I normally turn around and head back.  Except today, to my surprise, I came across a herd of goats!

I first spotted them on the road and let them cross in front of me.  A baby lagged behind, and I followed as closely as possible to get some better pictures.  When I got within 20 feet, the little one bolted ahead, sprinting over ground that was treacherously uneven volcanic rock.  The goats didn’t seem to have any problem running over this terrain, however, nor did they seem to mind me following them around for a half hour.  Here you can see the little one in mid stride of a dashing pace, and notice how well it blends in with the ground’s color.  I also enjoy seeing all the lizards here.  I’ve seen several kinds and most of them are small, but this large one was hanging out at the Raddisson hotel by the pool.

I had the pleasure of going to a huge DJ party called something like Maj 4 Stix.  The DJ rig was enormous, with thick outdoor smoke effects, blasts of fire and bright lights of every color, and thumping dance music.  There were acrobats running in translucent plastic balls in the water that surrounded the dance stage like a moat, and hundreds of people dancing to really great music.

Every few days, I head to Oranjestad to work: the capital of Aruba.  The best shopping seems to be there, since that’s where the cruise ships stop.  The pictures below are of a shopping area in Oranjestad, and a church and graveyard in Noord where many people are buried in elaborate above-ground stone tombs.

Finally, here are two pictures of me: one in front of the rock waterfalls at the Raddisson Hotel from a video I made to wish my niece Ava a happy birthday, and a fun picture of me at Confession Club in Palm Beach.

I’ve hiked through the unpopulated countryside of Aruba; I’ve gone to the big parties and night clubs, spent a lot of time tanning on the beaches, enjoyed Dutch food (Cafe Rembrandt is my favorite), and went on a Jeep tour (through ABC Tours) to the natural pools, the gold mine buildings, and the old Indian-painted caves; and somehow have still managed to be very productive writing software for my current client as well as some personal projects I have in the works.  I don’t often give advice, but I would definitely recommend enjoying life as much as possible while it lasts.  Travel, work remotely, start a business, or do whatever makes sense in your life to follow your dreams, but don’t wait to do it!

Filtering with Metadata in the Managed Extensibility Framework

The Managed Extensibility Framework (MEF) is the new extensibility framework from Microsoft.  Pioneered by Glenn Block in the patterns & practices group, and leveraged by the behemoth Visual Studio 2010, it has a striking resemblance to my own Inversion of Control (IoC) and Dependency Injection (DI) framework—which led to me to have a couple great conversations about IoC with Glenn at Tech Ed 2008 and then again at PDC 2008.

But MEF isn’t really written to be your IoC.  Instead, the IoC engine and DI aspects are implementation details, allowing you to do really no more than “MEF things together”.  The core concept of MEF is to provide very simple and powerful application composability.  Not in the user interface composition sense—for that, see Prism for WPF and Silverlight (explained in MSDN Magazine, September 2008)—but for virtually all other dynamic component assembly needs, MEF is your best friend.

The two things I like most about MEF is its simplicity as its lack of presumption on how it will be used.  Compose collections of strings, single method delegates, or implementations of complex services.  All you’re doing is importing and exporting things, with little code required to wire things up.

MEF is currently in its seventh preview release, so expect beta-like quality.  My own experience with it has been very positive, but there are a number of shortcomings in the API.  This article is about a few of them and what can be done to add some much-needed functionality.

System.AddIn vs. MEF

There’s been some confusion with Microsoft coopetition among products with similar aims, and extensibility and composition are no exception.  The AddIn API (team blog) serves a similar purpose as MEF.  (See this two-part MSDN article on System.AddIn: first and second.)  The primary differentiator, from my understanding, is that the AddIn API is a bit more robust and a lot more complicated, and supports such things as isolating extensions in separate AppDomains.

With Visual Studio siding with MEF, it’s personally hard for me to imagine using the AddIn API.  If MEF is flexible and robust enough for Visual Studio, is it really likely to fall short for my own much smaller software systems?  Krzysztof Cwalina suggests they are complementary approaches, but I find that hard to swallow.  Why would I want to use two different extensibility frameworks instead of one coherent API?  If anything, I imagine that the lessons learned from the AddIn API will eventually migrate to MEF.

Daniel Moth notes that with the AddIn API, “there are many design decisions to make and quite a few subtleties in implementing those decisions in particular when it comes to discovering addins, version resiliency, isolation from the host etc.”  A customer of mine using the AddIn API was using a Visual Studio plug-in to manage pipelines, and things were a real mess.  There were a bunch of assemblies, a lot of generated code, and not much clarity or confidence that it was all really necessary.

MEF: Import & ImportMany

In MEF, the Import attribute allows you to inject a value that is exported somewhere else using the Export attribute—typically from another assembly.  There is also an ImportMany attribute which is useful when you expect several exports that use the same contract.  By defining an IEnumerable<T> field or property and decorating it with the ImportMany attribute, all matching exports will be added to an enumerable type.

[ImportMany]
public IEnumerable<IVehicle> Vehicles;

What if you want to filter the exported vehicle types by some kind of metadata, though?  Let’s take a look at the IVehicle contract and some concrete classes that implement the contract.

public interface IVehicle { }

[Export(typeof(IVehicle))]
[ExportMetadata("Speed", "Slow")]
public class ToyotaPrius : IVehicle
{
    public ToyotaPrius() { }
}

[Export(typeof(IVehicle))]
[ExportMetadata("Speed", "Fast")]
public class LamborghiniDiablo : IVehicle
{
    public LamborghiniDiablo() { }
}

The object model isn’t very interesting, but that’s not the point.  What is interesting is that MEF allows us to supply metadata corresponding to our exports.  In this case, my contrived example has defined a metadata variable of “Speed”, with two possible values: “Fast” and “Slow”.  The variable name must be a string, but its value can be any value; that is, any value that’s supported from within an attribute, which means string literals and constants, type objects, and the like.

Filtering Imports on Metadata

What if you want to ImportMany for all exports that have a particular metadata value?  Unfortunately, there are no such options in the ImportMany attribute class.

In my scenario, I’ve defined a static factory class called VehicleFactory, which at some imaginary point in the future will be responsible for building a city full of trafic.

public static class TrafficFactory
{
    // type initialization fails without a static constructor
    static TrafficFactory() { }

    public static IEnumerable<IVehicle> SlowVehicles =
        App.Container.GetExportedValues<IVehicle>(metadata => metadata.ContainsKeyWithValue("Speed", "Slow"));

    public static IEnumerable<IVehicle> FastVehicles =
        App.Container.GetExportedValues<IVehicle>(metadata => metadata.ContainsKeyWithValue("Speed", "Fast"));

    public static IDictionary<object, IVehicle> AllVehicles =
        App.Container.GetKeyedExportedValues<IVehicle>("Speed");
}

This is what I want to do, but there is no overload of GetExportedValues that supplies a metadata-dependent predicate function.  Adding one is easy, though.  While we’re at it, we’ll also add the ContainsKeyWithValue which I borrow from The Code Junky article also on MEF container filtering.

public static class IDictionaryExtensions
{
    public static bool ContainsKeyWithValue<KeyType, KeyValue>(
        this IDictionary<KeyType, ValueType> Dictionary,
        KeyType Key, ValueType Value)
    {
        return (Dictionary.ContainsKey(Key) && Dictionary[Key].Equals(Value));
    }
}

public static class MEFExtensions
{
    public static IEnumerable<T> GetExportedValues<T>(this CompositionContainer Container,
        Func<IDictionary<string, object>, bool> Predicate)
    {
        var result = new List<T>();

        foreach (var PartDef in Container.Catalog.Parts)
        {
            foreach (var ExportDef in PartDef.ExportDefinitions)
            {
                if (ExportDef.ContractName == typeof(T).FullName)
                {
                    if (Predicate(ExportDef.Metadata))
                        result.Add((T)PartDef.CreatePart().GetExportedValue(ExportDef));
                }
            }
        }

        return result;
    }
}

Now we can test this logic by wiring up MEF and then accessing the two filtered collections of cars, which will each contain a single IVehicle instance.

class App
{
    [Export]
    public CompositionContainer Container;

    static void Main(string[] args)
    {
        AssemblyCatalog catalog = new AssemblyCatalog(Assembly.GetExecutingAssembly());
        Container = new CompositionContainer(catalog);
        Container.ComposeParts();

        var FastCars = TrafficFactory.FastVehicles;
        var SlowCars = TrafficFactory.SlowVehicles;
    }
}

Viola!  We have metadata-based filtering.

You’ll also noticed that I added an Export attribute to the Container itself.  By doing this, you can Import the container into any module that gets dynamically loaded.  It’s not used in this article, but getting to the container from a module is otherwise impossible without some kind of work-around.  (Thanks for pointing out the problem, Damon.)

Using Metadata to Assign Dictionary Keys

Let’s take this one step further.  Let’s say you want to import many instances of MEF exported values into a Dictionary, using one of the metadata properties as the key.  This is how I’d like it to work:

public static IDictionary<object, IVehicle> AllVehicles =
    App.Container.GetKeyedExportedValues<IVehicle>("Speed");

Again, the current MEF Preview doesn’t support this, but another extension method is all we need.  We’ll add two, so that one version gives us all exported values and the other allows us to filter that selection based on other metadata.

public static IDictionary<object, T> GetKeyedExportedValues<T>(this CompositionContainer Container,
    string MetadataKey, Func<IDictionary<string, object>, bool> Predicate)
{
    var result = new Dictionary<object, T>();

    foreach (var PartDef in Container.Catalog.Parts)
    {
        foreach (var ExportDef in PartDef.ExportDefinitions)
        {
            if (ExportDef.ContractName == typeof(T).FullName)
            {
                if (Predicate(ExportDef.Metadata))
                    result.Add(ExportDef.Metadata[MetadataKey],
                        (T)PartDef.CreatePart().GetExportedValue(ExportDef));
            }
        }
    }

    return result;
}

public static IDictionary<object, T> GetKeyedExportedValues<T>(this CompositionContainer Container,
    string MetadataKey)
{
    return GetKeyedExportedValues<T>(Container, MetadataKey, metadata => true);
}

Add an assignment to TrafficFactory.AllVehicles in the App.Main method and see for yourself that it works.

If you’re using metadata values as Dictionary keys, it’s probably important for you not to mess them up.  I recommend using enum values for both metadata property names as well as valid values if it’s possible to enumerate them, and string const values otherwise.

Now go forth and start using MEF!

Better Tool Support for .NET

Productivity Enhancing Tools

Visual Studio has come a long way since its debut in 2002.  With the imminent release of 2010, we’ll see a desperately-needed overhauling of the archaic COM extensibility mechanisms (to support the Managed Package Framework, as well as MEF, the Managed Extensibility Framework) and a redesign of the user interface in WPF that I’ve been pushing for and predicted as inevitable quite some time ago.

For many alpha geeks, the Visual Studio environment has been extended with excellent third-party, productivity-enhancing tools such as CodeRush and Resharper.  I personally feel that the Visual Studio IDE team has been slacking in this area, providing only very weak support for refactorings, code navigation, and better Intellisense.  While I understand their desire to avoid stepping on partners’ toes, this is one area I think makes sense for them to be deeply invested in.  In fact, I think a new charter for a Developer Productivity Team is warranted (or an expansion of their team if it already exists).

It’s unfortunately a minority of .NET developers who know about and use these third-party tools, and the .NET community as a whole would without a doubt be significantly more productive if these tools were installed in the IDE from day one.  It would also help to overcome resistance from development departments in larger organizations that are wary of third-party plug-ins, due perhaps to the unstable nature of many of them.  Microsoft should consider purchasing one or both of them, or paying a licensing fee to include them in every copy of Visual Studio.  Doing so, in my opinion, would make them heroes in the eyes of the overwhelming majority of .NET developers around the world.

It’s not that I mind paying a few hundred dollars for these tools.  Far from it!  The tools pay for themselves very quickly in time saved.  The point is to make them ubiquitous: to make high-productivity coding a standard of .NET development instead of a nice add-on that is only sometimes accepted.

Consider just from the perspective of watching speakers at conferences coding up samples.  How many of them don’t use such a tool in their demonstration simply because they don’t want to confuse their audience with an unfamiliar development interface?  How many more demonstrations could they be completing in the limited time they have available if they felt more comfortable using these tools in front of the masses?  You know you pay good money to attend these conferences.  Wouldn’t you like to cover significantly more ground while you’re there?  This is only likely to happen when the tool’s delivery vehicle is Visual Studio itself.  Damon Payne makes a similar case for the inclusion of the Managed Extensibility Framework in .NET Framework 4.0: build it into the core and people will accept it.

The Gorillas in the Room

CodeRush and Resharper have both received recent mention in the Hanselminutes podcast (episode 196 with Mark Miller) and in the Deep Fried Bytes podcast (episode 35 with Corey Haines).  If you haven’t heard of CodeRush, I recommend watching these videos on their use.

• CodeRush Express (which is free)
• CodeRush with Refactor!
• DXCore (the library you can use to build your own add-ins, and on which CodeRush and Refactor! are built)

For secondary information on CodeRush, DXCore, and the principles with which they were designed, I recommend these episodes of DotNetRocks:

• Tool Development
• Discoverability
• The Science of Good UI

I don’t mean to be so biased toward CodeRush, but this is the tool I’m personally familiar with, has a broader range of functionality, and it seems to get the majority of press coverage.  However, those who do talk about Resharper do speak highly of it, so I recommend you check out both of them to see which one works best for you.  But above all: go check them out!

Refactor – Rename

Refactoring code is something we should all be doing constantly to avoid the accumulation of technical debt as software projects and the requirements on which they are based evolve.  There are many refactorings in Visual Studio for C#, and many more in third-party tools for several languages, but I’m going to focus here on what I consider to be the most important refactoring of them all: Rename.

Why is Rename so important?  Because it’s so commonly used, and it has such far-reaching effects.  It is frequently the case that we give poor names to identifiers before we clearly understand their role in the “finished” system, and even more frequent that an item’s role changes as the software evolves.  Failure to rename items to accurately reflect their current purpose is a recipe for code rot and greater code maintenance costs, developer confusion, and therefore buggy logic (with its associated support costs).

When I rename an identifier with a refactoring tool, all of the references to that identifier are also updated.  There might be hundreds of references.  In the days before refactoring tools, one would accomplish this with Find-and-Replace, but this is dangerous.  Even with options like “match case” and “match whole word”, it’s easy to rename the wrong identifiers, rename pieces of string literals, and so on; and if you forget to set these options, it’s worse.  You can go through each change individually, but that can take a very long time with hundreds of potential updates and is a far cry from a truly intelligent update.

Ultimately, the intelligence of the Rename refactoring provides safety and confidence for making far-reaching changes, encouraging more aggressive refactoring practices on a more regular basis.

Abolishing Magic Strings

I am intensely passionate about any tool or coding practice that encourages refactoring and better code hygiene.  One example of such a coding practice is the use of lambda expressions to select identifiers instead of using evil “magical strings”.  From my article on dynamically sorting Linq queries, the use of “magic strings” would force me to write something like this to dynamically sort a Linq query:

Customers = Customers.Order("LastName").Order("FirstName", SortDirection.Descending);

The problem here is that “LastName” and “FirstName” are oblivious to the Rename refactoring.  Using the refactoring tool might give me a false sense of security in thinking that all of my references to those two fields have been renamed, leading me to The Pit of Despair.  Instead, I can define a function and use it like the following:

public static IOrderedEnumerable<T> Order<T>(this IEnumerable<T> Source,
    Expression<Func<T, object>> Selector, SortDirection SortDirection)
{
    return Order(Source, (Selector.Body as MemberExpression).Member.Name, SortDirection);
}

Customers = Customers.Order(c => c.LastName).Order(c => c.FirstName, SortDirection.Descending);

This requires a little understanding of the structure of expressions to implement, but the benefit is huge: I can now use the refactoring tool with much greater confidence that I’m not introducing subtle reference bugs into my code.  For such a simple example, the benefit is dubious, but multiply this by hundreds or thousands of magic string references, and the effort involved in refactoring quickly becomes overwhelming.

Coding in this style is most valuable when it’s a solution-wide convention.  So long as you have code that strays from this design philosophy, you’ll find yourself grumbling and reaching for the inefficient and inelegant Find-and-Replace tool.  The only time it really becomes an issue, then, is when accessing libraries that you have no control over, such as the Linq-to-Entities and the Entity Framework, which makes extensive use of magic strings.  In the case of EF, this is mitigated somewhat by your ability to regenerate the code it uses.  In other libraries, it may be possible to write extension methods like the Order method shown above.

It’s my earnest hope that library and framework authors such as the .NET Framework team will seriously consider alternatives to, and an abolition of, “magic strings” and other coding practices that frustrate otherwise-powerful refactoring tools.

Refactoring Across Languages

A tool is only as valuable as it is practical.  The Rename refactoring is more valuable when coding practices don’t frustrate it, as explained above.  Another barrier to the practical use of this tool is the prevalence of multiple languages within and across projects in a Visual Studio solution.  The definition of a project as a single-language container is dubious when you consider that a C# or VB.NET project may also contain HTML, ASP.NET, XAML, or configuration XML markup.  These are all languages with their own parsers and other language services.

So what happens when identifiers are shared across languages and a Rename refactoring is executed?  It depends on the languages involved, unfortunately.

When refactoring a C# class in Visual Studio, the XAML’s x:Class value is also updated.  What we’re seeing here is cross-language refactoring, but unfortunately it only works in one direction.  There is no refactor command to update the x:Class value from the XAML editor, so manually changing it causes my C# class to become sadly out of sync.  Furthermore, this seems to be XAML specific.  If I refactor the name of an .aspx.cs class, the Inherits attribute of the Page directive in the .aspx file doesn’t update.

How frequent do you think it is that someone would want to change a code-behind file for an ASP.NET page, and yet would not want to change the Inherits attribute?  Probably not very common (okay, probably NEVER).  This is a matter of having sensible defaults.  When you change an identifier name in this way, the development environment does not respond in a sensible way by default, forcing the developer to do extra work and waste time.  This is a failure in UI design for the same reason that Intellisense has been such a resounding success: Intellisense anticipates our needs and works with us; the failure to keep identifiers in sync by default is diametrically opposed to this intelligence.  This represents a fragmented and inconsistent design for an IDE to possess, thus my hope that it will be addressed in the near future.

The problem should be recognized as systemic, however, and addressed in a generalized way.  Making individual improvements in the relationships between pairs of languages has been almost adequate, but I think it would behoove us to take a step back and take a look at the future family of languages supported by the IDE, and the circumstances that will quickly be upon us with Microsoft’s Oslo platform, which enables developers to more easily build tool-supported languages (especially DSLs, Domain Specific Languages). 

Even without Oslo, we have seen a proliferation of languages: IronRuby, IronPython, F#, and the list goes on.  A refactoring tool that is hard-coded for specific languages will be unable to keep pace with the growing family of .NET and markup languages, and certainly unable to deal with the demands of every DSL that emerges in the next few years.  If instead we had a way to identify our code identifiers to the refactoring tool, and indicate how they should be bound to identifiers in other languages in other files, or even other projects or solutions, the tools would be able to make some intelligent decisions without understanding each language ahead of time.  Each language’s language service could supply this information.  For more information on Microsoft Oslo and its relationship to a world of many languages, see my article on Why Oslo Is Important.

Without this cross-language identifier binding feature, we’ll remain in refactoring hell.  I offered a feature suggestion to the Oslo team regarding this multi-master synchronization of a model across languages that was rejected, much to my dismay.  I’m not sure if the Oslo team is the right group to address this, or if it’s more appropriate for the Visual Studio IDE team, so I’m not willing to give up on this yet.

A Default of Refactor-Rename

The next idea I’d like to propose here is that the Rename refactoring is, in fact, a sensible default behavior.  In other words, when I edit an identifier in my code, I more often than not want all of the references to that identifier to change as well.  This is based on my experience in invoking the refactoring explicitly countless times, compared to the relatively few times I want to “break away” that identifier from all the code that references.

Think about it: if you have 150 references to variable Foo, and you change Foo to FooBar, you’re going to have 150 broken references.  Are you going to create a new Foo variable to replace them?  That workflow doesn’t make any sense.  Why not just start editing the identifier and have the references update themselves implicitly?  If you want to be aware of the change, it would be trivial for the IDE to indicate the number of references that were updated behind the scenes.  Then, if for some reason you really did want to break the references, you could explicitly launch a refactoring tool to “break references”, allowing you to edit that identifier definition separately.

The challenge that comes to mind with this default behavior concerns code that spans across solutions that aren’t loaded into the IDE at the same time.  In principle, this could be dealt with by logging the refactoring somewhere accessible to all solutions involved, in a location they can all access and which gets checked into source control.  The next time the other solutions are loaded, the log is loaded and the identifiers are renamed as specified.

Language Property Paths

If you’ve done much development with Silverlight or WPF, you’ve probably run into the PropertyPath class when using data binding or animation.  PropertyPath objects represent a traversal path to a property such as “Company.CompanyName.Text”.  The travesty is that they’re always “magic strings”.

My argument is that the property path is such an important construct that it deserves to be an core part of language syntax instead of just a type in some UI-platform-specific library.  I created a data binding library for Windows Forms for which I created my own property path syntax and type, and there are countless non-UI scenarios in which this construct would also be incredibly useful.

The advantage of having a language like C# understand property path syntax is that you avoid a whole class of problems that developers have used “magic strings” to solve.  The compiler can then make intelligent decisions about the correctness of paths, and errors can be identified very early in the cycle.

Imagine being able to pass property paths to methods or return then from functions as first-class citizens.  Instead of writing this:

Binding NameTextBinding = new Binding("Name") { Source = customer1; }

… we could write something like this, have access to the Rename refactoring, and even get Intellisense support when hitting the dot (.) operator:

Binding NameTextBinding = new Binding(@Customer.Name) { Source = customer1; }

In this code example, I use the fictitious @ operator to inform the compiler that I’m specifying a property path and not trying to reference a static property called Name on the Customer class.

With property paths in the language, we could solve our dynamic Linq sort problem cleanly, without using lambda expressions to hack around the problem:

Customers = Customers.Order(@Customer.LastName).Order(@Customer.FirstName, SortDirection.Descending);

That looks and feels right to me.  How about you?

Summary

There are many factors of developer productivity, and I’ve established refactoring as one of them.  In this article I discussed tooling and coding practices that support or frustrate refactoring.  We took a deep look into the most important refactoring we have at our disposal, Rename, and examined how to get the greatest value out of it in terms of personal habits, as well as long-term tooling vision and language innovation.  I proposed including property paths in language syntax due to its general usefulness and its ability to solve a whole class of problems that have traditionally been solved using problematic “magic strings”.

It gives me hope to see the growing popularity of Fluent Interfaces and the use of lambda expressions to provide coding conventions that can be verified by the compiler, and a growing community of bloggers (such as here and here) writing about the abolition of “magic strings” in their code.  We can only hope that Microsoft program managers, architects, and developers on the Visual Studio and .NET Framework teams are listening.

Living & Working in Sunny Aruba

I am thrilled to finally be living in Aruba, at least for the month of September.  This is an experiment in remote working, and an experiment in living outside the United States.  “Why Aruba?” you ask.  Why not?  Aruba has weather that’s perfect for the beach year round, lies safely outside the hurricane belt, and has one of the highest per capita incomes in the Caribbean, making it a very safe and happy place.  In fact, their license plate tag line is “One Happy Island”.

 

Indeed it is!  Everyone here has been extremely friendly.  The population is ethnically diverse and many languages can be heard.  Residents speak Papiamento, Spanish, English, and Dutch generally, and I often hear German, French, Japanese, and other languages I can’t yet identify.  It seems common for people living here to speak six or more languages.  Being a lover of languages, I hope to pick up as much as I can while I have the opportunity.

I planned many months ahead of time, but found a paucity of information available online and have had to wing-it for many aspects of the trip, which just makes it more of an adventure.  Aruban websites are geared toward mainstream tourism and high-profile resort hotel-casinos (many of which are beautiful), but I was looking for longer-term residency, and a bargain at that.  I settled for a cheap room off the beaten path, which was about the same rate for a month as a hotel room would be for a week.  As it turned out, I was upgraded for free to a nice two-bedroom condominium due to last-minute rescheduling of my original room.  I’m a ten-minute walk from Palm Beach, a two-hour walk from the capital of Oranjestad, and at about 20 miles by 5 miles, Aruba is large enough to keep me busy exploring but small enough to make exploring most areas of it possible within my month here.

Do You Ever Work?

Yes, I work on projects for several customers while I’m here.  I found a fantastic free-Internet Dutch cafe called Cafe Rembrandt with a wonderful staff.  I have plugs to power my laptop, and use Skype or iCall to make calls to customers.  Both of these have applications for iPhone.  With them, I pay $0.20 – $0.27 per minute for calls.  Without them, through AT&T (and through SETAR, who is the cell and wi-fi provider for the island), I’d be paying an outrageous $1.69 per minute.  This limits me to making calls from free Internet hotspots, or I could pay SETAR $70 per month for unlimited access to their wireless access that blankets the popular parts of the island.

From a technical communication perspective, it’s all working well so far.  Because I’m working on smaller projects and my customers are geographically distributed anyway, I’m not running up against many of the hurdles that would appear on larger projects, so it’s a good way to dip one foot into the water without jumping in all the way on day one.  Working side-by-side in person with other members on larger projects is always the highest-bandwidth method of communicating, but remote working scenarios are becoming more and more common and have many benefits.  The only real way to identify the challenges these scenarios impose is to put yourself into them again and again, and deal with the issues as they come up, finding solutions to problems, working around limitations, and exploiting the advantages that decentralization provides.

Getting Around & Communicating

Being an avid running and hiker, I’ve walked about four hours a day since I’ve been here, pushing myself as I usually do.  The busses, however, are air-conditioned, cheap (about $1.30 per trip just about anywhere), clean and safe, so I always have an easy way home when I’m completely exhausted.  They’ll go anywhere you need them to, so renting a vehicle is unnecessary, but car rentals are reasonable if you need one.  If you want to rent one, make sure to go to your local AAA and get an International Driver’s License before coming.  Also check AAA and tourist books for coupons, which can be 10-20% off of listed rates.

Phones are available for rent, or you can use your existing phone as long as your carrier allows international roaming (you may have to call them to authorize that feature).  AT&T customers need to sign up for their World Traveler plan.  I use mine only for Google Maps to navigate and to check for email periodically, as the data plans are outrageously expensive if you go over your limit (over $5 per MB).

Wrapping Up

I could write many pages more about my few days here already, but instead I’ll conclude with a few of the pictures I’ve taken from my iPhone.  If you have your own stories about Aruba, or living and working abroad or remotely in general and the lessons you learned, I’d love to hear about them.

   

Strongly-Typed, Dynamic Linq Order Operator

A Community Solution

I love social technologies like Stack Overflow, where people can collaborate loosely to share knowledge and help get things done.  Stack Overflow does on a large scale what developer blogs like mine have been doing on a smaller scale: creating a community around the sharing of ideas and methods.

Every once in a while, I get some great feedback that includes a fix for one of my bugs, a performance tweak I didn’t realize was possible, or an extension to some library I left unfinished.

This morning I ran into a question about my dynamic Linq sort, solved and answered by “Ch00k”, allowing one to get compile-time checking of identifier names.  Well done!

(It’s too bad Stack Overflow doesn’t promote the use of real names for professional developers to better market themselves with skill and reputation.)

My original article (with source code) is here.  All I added to the library was this:

public static IOrderedEnumerable<T> Order<T>(this IEnumerable<T> Source,
    Expression<Func<T, object>> Selector, SortDirection SortDirection)
{
    return Order(Source, (Selector.Body as MemberExpression).Member.Name, SortDirection);
}

To test it, I used this code:

IEnumerable<Customer> Customers = new Customer[] { new Customer("Dan", "Vanderboom"), new Customer("Steve", "Vanderboom"),
    new Customer("Tracey", "Vanderboom"), new Customer("Sarah", "Barkelew") };

Customers = Customers.Order(c => c.LastName, SortDirection.Ascending);
Customers = Customers.Order(c => c.FirstName, SortDirection.Descending);

foreach (var cust in Customers)
{
    Console.WriteLine(cust.FirstName + " " + cust.LastName);
}

Now I can refactor these data model classes with a tool and all my dynamic sorting queries will stay in sync!

Software Development Methods

Building a House

Building a house is still one of the most common analogies I hear and use for discussing software development with customers.  The problem is that the architecture of the house is left out; and unless you’re a developer, it’s hard to understand why a stack of wireframes isn’t the comprehensive description of the system needed to provide accurate cost and schedule estimates.

These wireframes are like the diagrams a first-time home owner might sketch on the back of a napkin.  In order to know what must be built, and to determine a realistic sense of scope and cost, those rough ideas need to be developed into something much more substantial by a professional.  For our house building scenario, this would involve hiring an architect to transform those ideas into a set of drawings which include such things as plumbing, heat and air conditioning ducts, electrical access panels, drains, the appropriate foundation and structural support elements, and so on.  The customer asks for a “nice room with a window looking to the back yard”, but all of these other hidden structural elements must be taken into consideration as well.

Unfortunately, determining the actual cost and size of a large project with any accuracy requires having this detailed design; and worse, doing this detailed design can take weeks or even months.  By the time development is underway, the customer will inevitably change many of the requirements anyway.  In fact, the very process of evolving such a large and detailed design brings many issues to the surface to discuss, debate, and make decisions about.  It’s easy to see how this process could carry on for quite a while, and there are still changes in the market to consider and adapt to, both during and after requirements definition.

The house building analogy is useful, but it is also misused.  To be fair, a typical waterfall software project put into house building terms would go like this.

1. Customer sends sketches to Architect and asks for plan including schedule and cost.
2. Architect works with Customer, being paid for his services to arrive at detailed design, which provides the project schedule and cost.
3. Builder works with the Architect’s plan to build exactly what’s in the requirements.
4. Days or weeks later, Customer makes changes to requirements.  This requires the Architect to get involved again, update the plan, work with the Builder to make physical adjustments, which may include tearing down existing structures and modifying others.  Plan changes need to be approved by government (verified with tests), documentation updated, and Customer agrees to pay for the changes.
5. * Customer requests to now connect their house to other houses near by, the shapes and layouts of which may change over time.
6. Repeat #4.

However, this is completely at odds with how homes are normally built.  People typically choose a previously-implemented design, and only customize superficial features like countertops, cabinets, floors, and railings.  Houses from this plan have been built before, and the labor and materials cost are known from previous experience.

Building software is usually more like constructing something that’s never been built before: the first sky scraper, the Golden Gate Bridge, or the Hoover Dam.  The requirements are unique, the pieces have never been assembled in such a way before, and there’s an inherent level of risk in creating something new.  When this is the case, the Customer needs the services of an Architect, not just a Builder who stamps out deliverables in a cookie-cutter style.

Waterfall methodologies have been dramatically in decline over the years in favor of so-called Agile methods.  There are several problems with a Waterfall approach:

• Because of requirements churn, attempting to write all the requirements up-front is incredibly wasteful.  Your team will end up doing a lot of design work that is eventually thrown away.  When I hear about churn numbers as high as 80%, I shudder to think of all the time and money spent refining details that will never make it into the production system.
• It tends to take a blind approach to requirements change, working under the assumption that change is bad and should be prevented.  Many waterfall methodologies erect barriers to change, making Customers jump through hoops to push changes through.
• Customers typically don’t want to pay for this architecture phase.  They want to pay for actual work on their system, and the cost of defining the actual scope and cost of their system seems steep.  “Why should I pay you $<large number> just for you to tell me how much you’ll charge me for the final system?”

An Agile Solution: Efficiency and Uncertainty

The Agile community emerged largely as a reaction to the disappointing results of Waterfall projects.  Customers were sick of schedules and budgets that were consistently and grossly overrun (despite all the up-front design), and wanted to more easily change requirements without being put through the wringer to do so.  Developers were tired of spending their valuable time writing detailed design documents that kept changing throughout the project, and the burden to update that documentation as the project goals changed focus or priorities shifted made development a very inefficient and slow process.

Essentially, as is often the case, a trade-off was made.  Some stakeholders realized that the predictability they were attempting to achieve by spelling out everything in detail was delusional: they were aiming at a moving target regardless of their efforts to pin things down.  Project costs rose precipitously because the scope of the system grew.  They needed a process that acknowledged and accepted these dynamic project forces instead of one that denied and suppressed them.  They needed a process that was efficient, and they gave up the illusion of certainty to get it.

The cone of uncertainty is one of the most valuable tools for visualizing and understanding uncertainty in development projects.  From that article:

Research has found that the accuracy of the software estimate depends on the level of refinement of the software’s definition. The more refined the definition, the more accurate the estimate. The reason the estimate contains variability is that the software project itself contains variability. The only way to reduce the variability in the estimate is to reduce the variability in the project itself.

An important—and difficult—concept is that the Cone of Uncertainty represents the best case accuracy it’s possible to have in software estimates at different points in a project. The Cone represents the error in estimates created by skilled estimators [which is up to 16x error at the beginning of a project]. It’s easily possible to do worse. It isn’t possible to be more accurate; it’s only possible to be more lucky.

Commitments made too early in a project undermine predictability, increase risk, increase project inefficiencies, and impair the ability to manage a project to a successful conclusion.

Meaningful commitments are not possible in the early, wide part of the Cone. Effective organizations delay their commitments until they have done the work to force the Cone to narrow. Meaningful commitments in the early-middle of the project (about 30% of the way into the project) are possible and appropriate.

Or as noted by the MSDN documentation for Composite Client Application Guidance:

Application requirements can change over time. New business opportunities and challenges may present themselves, new technologies may become available, or even ongoing customer feedback during the development cycle may significantly affect the requirements of the application. Therefore, it is important to build the application so that it is flexible and can be easily modified or extended over time.

Agile methods have always dealt with small iterations and early, frequent builds to evolve product functionality, but the Lean Development method especially has an explicit focus on making meaningful commitments at the latest possible responsible moment.  From Wikipedia:

As software development is always associated with some uncertainty, better results should be achieved with an options-based approach, delaying decisions as much as possible until they can be made based on facts and not on uncertain assumptions and predictions. The more complex a system is, the more capacity for change should be built into it, thus enabling the delay of important and crucial commitments. The iterative approach promotes this principle – the ability to adapt to changes and correct mistakes, which might be very costly if discovered after the release of the system.

While the methodology pundits debate the relative merits of development processes, Tom DeMarco, author of the fantastic books Peopleware and Controlling Software Projects, challenges us to consider the value of our projects and to re-evaluate elaborate control measures in light of the bigger picture.  From his article:

To understand control’s real role, you need to distinguish between two drastically different kinds of projects:

• Project A will eventually cost about a million dollars and produce value of around $1.1 million.
• Project B will eventually cost about a million dollars and produce value of more than $50 million.

What’s immediately apparent is that control is really important for Project A but almost not at all important for Project B.  This leads us to the odd conclusion that strict control is something that matters a lot on relatively useless projects and much less on useful projects.  It suggests that the more you focus on control, the more likely you’re working on a project that’s striving to deliver something of relatively minor value.

This adds additional credence to the idea of agile development, since the Big Design Up-Front (BDUF) of Waterfall is obsessed with detailed definitions to make accurate schedule promises.  If the business model is sound, lots of control isn’t needed.  It’s better to navigate with a good compass (the product vision) than an incorrect and constantly changing map.

So What Will I Get?

At the end of the day, customers want to know what they’re going to get.  There are tradeshow and conference dates to meet, beta and v1.0 launch targets to hit, release cycles to determine, and so on.  Ultimately, some up-front architecture and design is usually required.  But instead of trying to pin down every detail, it’s actually best in terms of efficiency to share the vision, provide direction, and define the technical scope only in a loose way.  This functional scope definition should reflect the reality of the cone of uncertainty at that point in the development process.

At the beginning of the project, items to include in the project should be broadly defined, having a granularity appropriate for the actual uncertainty.  As the project progresses and work is done, the scope becomes better defined, risks are determined benign or neutralized, and the cadence of that specific team working on that customer’s specific product can be measured.

This is successfully done with Lean and Kanban approaches, providing an empirical method for measuring productivity.  Planning can then be adjusted to match the regular rhythm of production that develops.  This creates a semblance of real predictability.  You can’t predict any one thing with absolute certainty, but if you can measure the general rhythm at which work is done, you will very quickly learn to recognize how long different kinds of tasks actually take.  Because you’re focusing on measured results rather than predictions, it’s easier to normalize task size and estimate larger functional units.

If you’re interested in learning more about Lean development, Mary and Tom Poppendieck’s book Implementing Lean Software Development is a fantastic background and explanation of the Lean process.  They also have a new book coming out this November, exploring the same topics more deeply.

When a good process is followed by hard-working, talented developers, the customer will end up with what they need to satisfy their business goals, often in surprising ways, ways that will be radically different from what they initially asked for.

What Can Be Promised?

The customer has their vision, beginning with the end in mind as they do; and they are asked to break it apart, prioritize individual features, and stretch their end-goal image across a long and expensive tunnel that separates them from their goal.  They usually want some kind of promise of what they’re going to get at the end, even though they’re going to keep adding and changing features throughout its development.

What can be promised?  It would be dishonest to promise customers what they’re asking for, unless they promise not to change their minds as the project progresses, but the only certainty in software development is that requirements will change.

Development isn’t about creating flawless, predictable plans that eliminate business risk.  Development is an unpredictable journey to create tremendous value.  To achieve great reward, risk is a necessary part of the equation.  Instead of coming up with creative ways to hide real project risk, it’s best to have a solid process for detecting and managing that risk.