Critical Development

Language design, framework development, UI design, robotics and more.

Posts Tagged ‘MSBuild’

Visual Studio 2008 – Project Reference Oddness

Posted by Dan Vanderboom on December 11, 2007

I can’t count the number of hours I’ve spent over the past few years fighting with references.  In the old days of C, Pascal, and other languages, you told the compiler to look here for a source or header file, and I’ll be damned if that’s not where it looked.  But in working with Visual Studio and the .NET Framework, those days are forever gone.

Project Reference Paths – Hints and Clues

The picture is a great deal more complicated with things like “hint paths” and directory searching algorithms.  The name hint path should be evidence enough of the absurdity of the situation.  We’re not telling you where something is.  Instead, we’re going to give you some hints and clues, and it will be a big fun mystery to figure it out.

To make things worse, these project reference paths are not stored in the Visual Studio project file, which gets checked into source control and shared with other team members.  Instead, they’re stored in a separate “user settings” file, which is more specific to the user, really, since it is also specific to the directory it’s in.  So every time a developer rebuilds their machine, or checks out a project to a different machine, or even branches a project in source control, they have to set up these project reference paths all over again.  I work on some fairly large solutions and use a healthy number of third party libraries, so setting up these paths can take 30–45 minutes or more.  I’ve been so annoyed by this that I’ve considered writing a Visual Studio extension that would allow you to store these paths in the project file itself, or failing that, some external file that would be checked in.  (Checking in the user settings file is undesirable, as there are many settings such as open windows and breakpoints that you would not want to share across a team.)

I’m sure there is a Big List of Good Reasons somewhere that describes why all of these convolutions were deemed necessary (such as allowing developers to check out dependent assemblies into different relative paths), but ultimately I believe this complexity is a weakness: it creates fragility.  Simplicity is wonderful for many reasons, and while sometimes we really do need to make things more complicated, we need to be careful to grow in complexity lest we also grow in chaos.

[What’s the difference?  Chaos is disorganized complexity, or dynamic instability.  Organized complexity is the result of structures in equilibrium.]

If a more complicated scenario is supported, then some product and tool maturity really begs to be included to provide discoverability, to make it obvious what is going on and why.  For a good discussion on discoverability, check out this .NET Rocks podcast where Mark Miller explains:

Or better yet, download a trial of CodeRush and open the discoverability window, which is an excellent example of a great user interface that makes a complicated set of functionality simple and obvious to use.  The designers of the Visual Studio IDE could learn a lot from these folks.

Reference Flavors & Properties

Visual Studio supports project references in addition to references to specific binaries.  When we have a static DLL, such as a third party library, we reference that DLL specifically.  Project references are nice because we can reference one of our own projects that hasn’t been built yet, and therefore has no DLL.  When we build project A that depends on project B, it will force project B to build as well.

In each reference, we can also specify whether to Copy Local (copy files locally), and this can be a point of confusion.  Local to what?  Here we’re talking about copying the assemblies that project B produces into project A’s output folder.  Why would we want to do this?  I’m not entirely certain, though I could make a guess that it’s easier to cascade these copy operations from one project to the next, to the next, and so on, until they wind up at their final location.  But considering how the performance of builds is tied so closely to disk access, as thousands of files often need to be processed, it begs the question of whether the locations of these files couldn’t just be tracked in memory, and only output in their final destinations at the very end of the build.  Furthermore, I ran across MSDN documentation a few months ago that recommended not changing the default Copy Local behavior, as it will likely cause problems.  No explanation was offered, as if to suggest “yes we know it’s complicated, so much that we’re not going to tell you why you should just leave it alone”.  Or such was my interpretation.

Specific Version is Too Specific

Another consideration to make is whether to target the Specific Version of the assembly or not.  This is the point of my most recent painful experience with references.  Typically, when I add a new reference to a third party library, Specific Version defaults to true.  When the default works, I leave it alone.

Yesterday I installed a new version of a third party library.  I maintain my own directory structure of third party libraries, copying them from their default install directories to organize them better, and so they stick around if I need to run their uninstaller (when supporting tools go bad).  I also check this directory into source control so they can all be easily shared across the development team.

The first thing I did was to update my reference paths.  Everyone on the team has to, which is a big repeated waste of time.  Then I noticed that my references were splatted.  That is, there were little yellow yield signs indicating that references were missing or otherwise confused.  It didn’t take long to figure out that the reference was pointing to version 7.3.3 with Specific Version set to true, and the new assemblies were version 7.3.4.  It had the name of the assembly, and the path to point right to it, but because I had originally told it to point to that specific version, that’s what it did.  That part makes sense.

I changed the Specific Version setting to false, on all of the references (separately) across all projects (another annoyance), and the splat signs went away.  I rebuilt the solution and it took about 20 minutes, compared to the minute or two it normally takes.  Not acceptable!  What was going on?

Upon looking at the build output window, and turning verbosity to Detailed, I noticed many lines that said “Considering …”.  In the process of looking for these new assemblies, it was considering just about every path it was aware of that had anything to do with development, Visual Studio, libraries, etc., to find these assemblies.  Why?  After all, I had given it valid reference hint paths.  I told it exactly where those assemblies were stored.  The names of the assemblies hadn’t changed.  I verified everything three, four, five times.  Yet it was ignoring all of that because of some set of complicated search rules that I am ignorant of, defined and buried deep in the bowels of MSBuild.  It told me where it was looking, but didn’t tell me why, so all of that output seems useless.

Where is this documented, you ask?  I don’t know.  I’ve asked on the forums and have gotten a bunch of “I don’t knows” for answers.  If you’re an MSBuild guru, please drop me a note and help me out.  It will be greatly appreciated.

I decided to turn Specific Version back on, thinking that it would see the new assembly the reference was pointing to, but doing so gave me the splat icons again.  Did the assemblies disappear off the disk?  No.  But for some reason, those references remember the version number of the old assemblies that I originally added the references to.  I expected that it would update the version, and cause the Specific Version references to point to the new 7.3.4 DLLs.  This didn’t happen, to my frustration.  I had to remove all of those references (keeping track of which ones were needed in which projects), and then re-add them all from scratch.  Specific Version was set to true by default, and when I compiled, MSBuild didn’t spend half the day searching for them.  Half a day later, after another half day of meetings, I was able to test the 10 lines of code I had written earlier.

It’s issues like this that you have to consider when someone suggests, “we should just upgrade our control library, it will be real quick”.  You never know.

Deployment Output – Where are we going, exactly?

I work primarily in Compact Framework for mobile devices, so part of the develop-test cycle involves deploying after building.  Just as there are complications in the .NET Framework’s build system, deployment can also be tricky.  If your solutions contain multiple projects, you need to be careful what platform you’re targeting, you have to be careful what version of the framework you’re targeting, etc.

When building and deploying, we noticed that all of our add-in DLLs weren’t being deployed.  We got no build or deployment errors in the output window.  Everything was successful, it said.  How could this be?

Apparently, you have to be really careful about where you’re deploying to.  In our system, we have an entry point EXE project and a bunch of add-in DLL projects.  The entry point application deploys into one directory, and add-ins get deployed into a subfolder called AddIns.  This has been working since Visual Studio 2005.  Now that we’ve switched to VS2008, have updated third party libraries, and have made some other changes, something went wrong.  At some point, our deployment folders changed on us. 

Counting on the output window for troubleshooting information once again, I noticed that there isn’t a lot of information regarding deployments.  Builds can fill thousands of lines very quickly, but deployment output just tells you what files it’s deploying—that is, where they’re coming from.  Unfortunately, it doesn’t tell you where it’s deploying them to.  What piece of information could be more critical to report during deployment?!  I wanted to know where my files were supposedly going, but no luck.

After several hours of scrutinizing our settings, I noticed that some of the add-in projects had a root deployment path of %CSIDL_PROGRAM_FILES% (\Program Files), where we were used to deploying, and others had a path of %CSIDL_PROGRAMS% (\Windows\Start Menu\Programs).  With the properties window being set as narrow as it is typically, this fact was obscured, but it makes a world of difference.  Looking in that new location, I found the files, and after updating the project properties, the problem was solved.

A Plea for Simplicity or Discoverability

Dependencies of any type are difficult to track.  Assembly dependencies can certainly get complicated, and that complexity can spiral out of control (into chaos) if we don’t have the right tools to understand and manage them.  When we add new ways of configuring, searching, upgrading, and using dependencies, we’re going to need much better tool support.  Above all, there should be some way to override all of the complications and tell our build system, “forget all that other crap and JUST GO HERE to find this”.

As far as we’ve come technologically with Visual Studio 2008, .NET Framework 3.5, C# 3.0, and all of the other goodness that I use and love (and we have come so far!), we still have a long way to go when it comes to developer experience.  We spend more time these days talking about improving user experience and interaction design in the applications we build for our clients, but the Visual Studio shell hasn’t shown any significant strides forward for many years.  I notice slight updates in linear gradients and the rendering of window borders and tabs, but there are no sweeping, dramatic changes.  There are many ways of making great improvements while still maintaining familiarity in usability, and in future posts, I’ll be talking about many of the ideas that I have to demonstrate.

This was continued with a discussion on deployment for mobile applications here, and a solution to some of these issues has been proposed in this article.


Posted in Compact Framework, Visual Studio | Tagged: | 11 Comments »

Visual Studio 2008 – Issues with Compact Framework

Posted by Dan Vanderboom on December 9, 2007

I’ve been developing for the Compact Framework for over three years now, and the road has been long.  So much is missing from CF that I have waited eagerly for v3.5 to arrive.  So, in typical fashion, I downloaded Visual Studio 2008 on November 20th and was up and running with it soon after.  The system I develop is distributed and highly complex, incorporating add-in modules, dependency injection, custom controls, MVC GUI patterns, SQL Server CE/Mobile, custom data synchronization, occassionally-offline behaviors, shared ORM data model across platforms, etc. At first glance, VS2008 looks good.  It looks like VS2005 with a few subtle changes, but it’s familiar, faster, and seems (so far) more stable.

Windows Mobile 5 R2 SDK – Failed Install (Quietly)

Converting from the VS2005 to the VS2008 project file format was at first problematic.  Though the installation didn’t error out, I got errors converting projects targeting Windows Mobile 5.  When I switched them to Pocket PC 2003, they worked, but this wasn’t going to be an acceptable solution.  I ended up uninstalling and then reinstalling the Windows Mobile 5 R2 SDK, and now I can successfully convert those projects. The worst part about this was that, although the projects seemed to upgrade, they couldn’t be read by Visual Studio, so they were unavailable in Solution Explorer.  So it’s impossible through Visual Studio to change them back.

SQL Server Mobile – References Updated Automatically

Annoyingly, the conversion from VS2005 to VS2008 projects assumed that I wanted to use SQL Server Compact 3.5 instead of the SQL Server Mobile 3.0 database I was referencing previously.  As much as I’d love to upgrade to the new database, and gain from improved performance as well as the ability to use the TOP keyword and use nested subqueries, our ORM tool is bound to v3.0.  Not a big deal to find and correct that problem.

Confused Path to SQL Server Mobile

Adding a reference to SQL Mobile 3.0 on the .NET tab of the Add References dialog points to VS2005 directory by default… not a good idea.  What happens when I uninstall VS2005?  This assembly goes away and now my references are broken.


So I copied this directory:

C:\Program Files\Microsoft Visual Studio 8\SmartDevices\SDK\SQL Server\Mobile

… to a “ThirdParty” folder structure under “SQL Server Mobile” that we can reference instead.  This folder gets shared and synchronized among developers.


I hoped it would have been easier to find, located in some shared place that makes sense.  There is no “Mobile” folder here:

C:\Program Files\Microsoft Visual Studio 9.0\SmartDevices\SDK\SQL Server

… which is too bad.  Though you’d think they’d put it somewhere like here instead, so it’s not VS folder dependent:

C:\Program Files\Microsoft.NET\SDK\CompactFramework\v2.0

… or even in one of these places:

C:\Program Files\Microsoft SQL Server 2005 Mobile Edition

C:\Program Files\Microsoft SQL Server Compact Edition

C:\Program Files\Microsoft SDKs

… but it’s not.  I think there is some great collective confusion at MS when it comes to organizing folder structures.  How many SDK directories do you need, anyway?


Build Times – Scary at First, Now Fantastic


After one of the development machines kept getting 20 minute build times on a 20 project solution (when we had 4–7 minute build times in VS2005), we were on the gut-wrenching verge of reverting back to VS2005.  But after some random tinkering, removing and re-adding references, and by deploying all add-in projects instead of referencing them from our entry point application, we were able to bring this down to about one minute. My laptop is a single core machine, and someone I work with has a dual core.  Because of the hype I’ve read about 2008’s support for dual core builds, I expected his machine to beat the pants off of mine during builds.  Such is not the case.  I still build a bit faster, all other aspects of the machines being pretty comparable. One thing MS definitely got right was the F5 behavior.  In VS2005, hitting F5 would rebuild our projects every time.  Even if a project would occassionally not get built, MSBuild would still “process” the project for way too long.  Now what we see is MSBuild activity bypassed altogether when a previous build succeeded and no changes have been made subsequently.  This, along with the other performance improvements (and some unexplained voodoo), is saving us somewhere around an hour per day per developer. Note that the build behavior is not perfect yet.  I still notice that projects that haven’t changed, and where no projects they depend on have changed, are still occassionally getting rebuilt for some reason.  But it’s good enough, and certainly an improvement over what we had, and there’s a possibility that our projects have some buried dependency (I don’t pretend to be an expert in MSBuild).


Not Yet Compact Framework 3.5


Now that our system has been converted to VS2008 and we’ve been working in that environment for a few weeks, the next step is to upgrade from Compact Framework 2.0 to 3.5.


LINQ Headaches

LINQ support in Compact Framework 3.5 deserves it’s own blog post.

Posted in Compact Framework, SQL Server Compact, Visual Studio | Tagged: , , , | Leave a Comment »