Critical Development

Language design, framework development, UI design, robotics and more.

Archive for August, 2008

Bad Ass Development Rig

Posted by Dan Vanderboom on August 23, 2008

[The powerful workstation described in this article is now for sale on eBay! Click here to see!]

The Need For Speed

I’m not a gamer, I’m a developer.  When I’m on my computer for eight to ten hours a day, I’m typically not rendering graphics, but rather writing, compiling, and testing code.  The writing part hardly requires any resources, but compiling code completely pegs out one of the processors on my dual core laptop (a 2.4 GHz Dell Latitude D830).  Parallel compilers exist, but C# in Visual Studio is not one of them, and by the sound of things, won’t be for quite some time.  This means that if I’m going to see a significant performance increase of this critical task, I’m going to need the fastest processor I can get (and overclock).

Compiling code is also disk intensive, especially toward the end of a build when output files are written to disk.  I ran some benchmarks of C# builds (in Visual Studio) of SharpDevelop.  I chose this code base because it’s fairly large, similar to my own solutions, and it’s open source so others can repeat our tests.  We tracked utilization of individual processors, disk I/O, etc.

Why am I so hell bent on compiling code as fast as possible?  Good question.


Micro Development Cycles

Software development consists of nested cycles.  There are organizational cycles that envelop project cycles that envelop several-week development sprints, and at the smallest level, it really all boils down to executing many micro development cycles every day.  You start with a goal such as fixing a bug or implementing a feature, do some generally-informal design in your head, plan out your work (again, typically in your head), write code for a few minutes, compile and fix until the build succeeds, deploy if necessary, test the changes, and repeat this sequence anywhere from 20 to 50 or more times in a productive day.  If you do test driven development, you write your tests before the functional code itself, but the cycle is otherwise essentially the same.

Develoment Cycle

Some of these steps take longer than others, and some of them, like designing or thinking about what code to write and where that logic belongs, are creative in nature and can’t be rushed.  But when we work on larger solutions in Visual Studio (and other tools), the time for tools to perform critical processing (compiling code in this case) can lead to Twiddling Thumb Syndrome (TTS).  This is not only an unfortunate affliction, it’s also one that can cause Turret-like symptoms, including swearing at one’s computer, at one’s software tools, and banging on things to entertain oneself while waiting for things to finish.  Sometimes, depending on your projects’ interdependencies and other details, build times can shoot up to several minutes (or worse) per build.  For a long time, I was getting build times in the 5-7 minute range, and it grows as solutions become larger.  Repeat this just 20 times (during which your computer is totally unresponsive) and you’ll quickly get the idea that you’re wasting lots of valuable time (two hours a day!).

Clearly this is unacceptable.  Even if my builds only took a minute, all of the aggregated time spent waiting for progress bars of all kinds (not just compiling) can add up to a significant chunk of wasted time.  In Scott Hanselman’s Ultimate Developer Rig article, which played a part in motivating me to build my own Ultimate Developer Rig, Scott hits the nail on the head:

I don’t want to have time to THINK about what it’s doing while I wait. I wait, in aggregate, at least 15 minutes a day, in a thousand tiny cuts of 10 seconds each, for my computer to finish doing something. Not compile-somethings, but I clicked-a-button-and-nothing-happened-oh-it-was-hung-somethings. Unacceptable. 15 minutes a day is 21.6 hours a year – or three full days – wasted.

I think Scott is being too conservative in his estimate.  It’s easy to waste at least 20-30 minutes a day waiting for general sluggishness, and considerably more when waiting for builds of large solutions.  If you have a computer that’s a few years old, it’s probably worse.  Thirty minutes a day is about 125 hours per year (over 3 weeks), and an hour a day is 6 weeks per year.

Flow = Mental Continuity

Look at it from another perspective.  Even if wasted time isn’t an issue, there’s still a matter of maintaining continuity of thought (and execution).  When we have a plan and are ready to act on it, but are held back behind some bottleneck in the process, we risk losing the fluid flow or mental momentum that we’ve built up.  Often, I have a sequence of pretty good ideas or things I’d like to try, but I end up waiting so long for the first step to finish, that by the time the computer is ready, I’ve lost track of my direction or next step.  This isn’t as much of a problem with long-term planning because those goals and steps tend to be written down, perhaps tracked in some kind of Scrum tool.  But when we’re talking about micro development cycles, a post-it note can be too formal (though when I’m waiting a lot, I do use these).  If we could get near-immediate feedback on our coding choices and reduce the wait time between the execution of tasks, we could maintain this flow better and longer, and our work would benefit from the increased mental continuity.

One analogy is that of reading a programming book.  Some of them are 800-1000 pages or more.  When you read one slowly, say a chapter every other week, it takes so long to read that by the time you finish chapter 10, you have a really hard time remembering what chapter 2 was all about.  But if you focus more and read through the same book in a week, then chapter 2 will still be fresh in your mind when you get to chapter 10, and you’ll be much better able to relate and connect ideas between them.  The whole book sticks in your memory better because all of its content is more cohesive in your mind.

Cost Justification

Scott created a nice computer for the price range he was shooting for, but for my own purposes, I wanted to go with something more extreme.  When I started playing with the numbers, I asked myself what the monthly cost would be for a top-of-the-line, $5,000 to $6,000 power machine.  Spread over 3 years, it comes to only $166 per month.  If you consider the proportion of this cost to the salary of a developer, figure out how much all of our unnecessary wasted time is worth, and realize that this is the primary and constantly-used hardware tool of an engineer, I think it’s very easy to justify.  This isn’t some elliptical trainer that’ll get used for two weeks and then spend the next five years in the garage or the storage shed.  This beast of burden will be put to serious work every day, and will make it easier and more pleasant to get work done.  In an age where we don’t even blink an eye at spending $1,000 on comfortable and ergonomic Herman Miller chairs, I think we’re long overdue for software engineers to start equipping themselves with appropriately-powerful computer hardware.

Compare the cost of a great workstation with the tools of other trades (carpentry, plumbing, automotive repair, etc.) and you’ll find that software development shops like to cut corners and go cheap on hardware because it’s possible to do so, not because it makes the most sense and delivers the greatest possible value.  If you’re in a warehouse and need a forklift, there’s no two ways about it.  But computers are commodities, and though they come in all shapes, sizes, and levels of power, the software you need will normally run on the slowest and most sluggish among them.

Welcome to My Home Office

Welcome to my office.  Since it’s going to appear in the background of many pictures, I thought I’d give a quick tour.  This is my brain dump wall, where many of my ideas first take form.


And around the corner from this room is the greatest Jack Daniel’s bar in the world, built by Christian Trauth (with a little bit of help from myself).


Bad Ass Components

I decided to take a field trip one day, and drove from the Milwaukee area where we live down to Fry’s in Chicago.  This was my first time to a Fry’s.  If you’ve never been to one, just imagine Disney World for computer geeks.  They’re absolutely huge (about 70,000 square feet of computer parts and other electronics).  I bought almost everything I needed there, having ordered a few parts online before this field trip took place.

Here’s what I picked up:

Intel D5400XS “SkullTrail” Motherboard – $575
Intel Core 2 Extreme Processor (QX97750) – $1510 (Tom’s Hardware Review)
  • 3.20 GHz (without overclocking)
  • 1600 MHz FSB
  • 12 MB L2 Cache
ThermalTake Bigwater 760is – $170
  • 2U Bay Drives Liquid Cooling System
Adaptec 5805 RAID Controller – $550
  • 8-Lane PCI Express
  • 512 MB DDR2 Cache
  • Battery Backup
3 Western Digital Velociraptor Hard Drives – $875
  • 900 GB Total
  • 10,000 rpm
  • SATA
8 GB (4 x 2 GB) of PC2-6400 RAM – $400
  • 800 MHz
  • ECC
  • Fully Buffered
GeForce 9800 GTX Video Card – $250
  • PCI Express 2.0
  • SLI Ready
  • 512 MB DDR3
Coolermaster Case – CMStacker 830 SE – $350
  • 1000 Watt Power Supply
  • Lots of Fan Slots
  • Very Modular

Total Damage – $4720

This doesn’t include extra fans (still need to purchase about 11 of them), and the things I already have: a pair of 24 inch monitors, Logitech G15 gaming keyboard (nice for the extra programmable keys), mouse, CD/DVD burner, media card reader, etc.  (When I calculate the cost at $166 per month over 3 years, it’s based on a total price tag of $6,000.)

Building a Bad Ass Development Rig

In the first picture are boxes for the case (and included 1000 Watt power supply), motherboard, video card, memory, and liquid cooling system.  The next two pictures show the motherboard mounted on the motherboard tray, which slides easily into the back of the case.  Notice how huge the video card is (on the right).  It takes up two slots for width, though it doesn’t plug into two slots (I’m not really much of a gamer, so no SLI for me).  The smaller card in the picture on the right is the Adaptec RAID controller.  I chose the slots that I did to maximize airflow; when I first put the graphics card in, it was partially obstructing a fan on the motherboard, so I moved it to the edge.  This blocked a connector on the motherboard, so I ended up moving it again.  Finding the right setup was a matter of trial and error.


Below you can see all the power cables hanging from the case.  They’re wrapped in a strong mesh that keeps the cables bundled together for improved airflow.  On the right, you can see a swinging door with dust filters and empty spaces for four fans (up to 150mm, not included with the case).  Notice the fan on the motherboard tray, and there’s a slot for another one in the roof of the case that you can’t see.  In addition to the fans, the sides, bottom, top, and front all let air pass through for maximum airflow.  The drives on the right are Western Digital Velociraptors: 300 GB and 10,000 rpm.  When set up in a RAID 0 (striping) configuration, they should provide wicked fast disk access, which is important because I’ll be running multiple virtual machines at a time.


Next you can see the modular drive cage, which is easier to install the drives in when it’s removed from the case (a couple screws on each side).  It’s nice that it has a fan on the front.  Overall, I’m very impressed with the case and all the attention to detail that’s gone into its design.  It was extremely easy to work with and reach everything.  It’s been several years since I’ve built a desktop computer, and I remember a lot more frustration when trying to reach inside of cases.  Notice that when I put the drive cage back in, I installed it higher up.  I couldn’t put it any higher because of power cables getting in the way, but I need a place for a CD/DVD burner and maybe a media reader anyway.  I moved it up because I’ll be installing a liquid cooling unit, and the radiator takes up two drive height units (2U).  If there’s any chance of that thing leaking (it better not!), I certainly don’t want water dripping down onto my hard drives.


Now I’m starting to wire everything up: power and reset switches, power LEDs, USB and Firewire ports, power to the motherboard and video card, power to the hard drives, and the interface cables from the RAID controller to the hard drives.  I start twist-tying the slack on cables and stuff the unused power cables into the ceiling of the case, where they stow away nicely (with more twist ties).  The right-most picture shows some of the other stuff included with the case: an IDE cable bound inside a tubular plastic sheath (for better airflow), SATA cables that I didn’t need because I used the ones that came with the RAID controller, a fan mount for one of the 11 heat sinks on the motherboard (fan not included), and a fun Do-Not-Disturb-style doorknob sign (included with the motherboard) that says “Warning: Noobs Beware.  You will be Pwned.”  And indeed you will be!


It’s finally time for some liquid cooling action.  With a motherboard called SkullTrail that was designed for overclocking two 771 processing chips, and a single QX9775 Core 2 Extreme quad core 3.2 GHz processor (to start with), you better believe I’ll be overclocking this bad ass machine to its limit!  I’ve heard rumors that 4.5 GHz is very manageable, and am hoping to be able to pull off upwards of 5 GHz, but we’ll see how it goes (with another processor, this would total 40 GHz across 8 parallel cores).  So far, the liquid cooling tops all other components in documentation: one user guide and one maintenance guide.  And don’t forget that you can’t take a sip of the cooling fluid, or eat any of the rock candy packets that come with the hard drives.  I know it’s tempting.


I Hit a Snag

The liquid cooling unit doesn’t fit.  It’s close, and I debated whether to let it hang out of the front of the case an inch and a half because of the motherboard being too close.  Not good enough.  Back to the drawing board!


The only way the cooling unit would fit flush in the case was if we cut out one of the aluminum support beams along the top of the case, and inserted the cooling unit in the top drive bays.  This would put the liquid above my hard drives, which I was trying to avoid, but I didn’t have any choice at this point.  So we jumped in his car and stopped at his place to pick up a dremmel.  Ten minutes later we were in the garage, case stripped down and upside down, cutting away.  You can see the end result in the photo on the right, which turned out very nice.


Finally, the liquid cooling system fits flush in the case.  We noticed that the cooling system had a fan speed control rheostat connected to it on a wire, and thought it would be nice if we could expose that through the case somehow, so we drilled a hole and fed it through the top (near the power and reset buttons).  I found a knob that fit on it from a robotics kit I purchased a few months ago, and it even matched the color of the case.  Bonus!  You can see the new knob in the picture below on the right.


Almost ready to boot up!  I’m waiting for the processor to arrive, and expect it any day now.  As soon as that comes in, I’ll be writing the next article in this Bad Ass Development Rig series, and we’ll see how much we can get this bad mamma jamma overclocked (without making it unstable, of course).  After that, I’ll be setting up the virtual machine system, all of my development environments, and then we’ll do some serious benchmarking.


Posted in Development Environment, Hardware, Personal | 16 Comments »

Small Gems in Visual Studio 2008 SP1

Posted by Dan Vanderboom on August 20, 2008

The first service pack for VS2008 is packed full of development goodness.  The Entity Framework has finally shipped, and despite some ungracious criticism (that nevertheless addresses some real concerns), a good foundation has been laid for us to start reaping the benefits of using business objects (entities), and to provide a compelling alternative to those of us who have been using ORM for years already and are sick of our current ORM vendor.  Other major deliverables include ADO.NET Data Services, support for SQL Server 2008, improvements in both WPF designers and WPF application performance, greater flexibility in WCF, and more.  You can find details here.

What you won’t find listed among the updates are two gems explained in this video, in which Charlie Calvert interviews DJ Park, the Visual Studio IDE PM.  The primary topic of the video is Live Semantic Errors, manifested as “squiggles” in the text editor.  Until now, the only errors that were recognized and indicated by squiggles have been syntax errors: that is, errors discoverable by the compiler’s parser.  Now that the Language Analysis Framework (LAF) is in place, which is the bridge between Visual Studio and the compiler, the IDE can obtain much more information about the semantics of that code.  This enables error checking of complex expressions, field initializers, and so on, and the detected errors show up in the task list tool window for any error detected in open files.

Live Semantic Analysis is neat, but even better is the potential that the Language Analysis Framework has to eventually provide C# developers with something VB has had for a while, which is background compilation.  Whether the C# team is heading down this road or not is unclear.  I’ve made some inquiries, so we’ll see who knows what and is willing to talk about it.

Update: It appears there’s a fair amount of interest at Microsoft to provide incremental compilation in C# to allow edit-and-continue functionality, though this is perceived as a long-term trend and not something we might expect in C# 4.0.  Incremental compilation adds a lot of complexity to a compiler, and past research showed that the performance benefits weren’t worth the effort.  The C# team actually started building an incremental compiler years ago and eventually gave up on it.  With parallel computing within reach now, the gains could be much more substantial as long as sophisticated task dependencies can be coordinated and the complexity managed.  The future of compiler technology, programming languages, and development tools should be pretty exciting!

At the end of this video, DJ mentioned another new hidden feature in SP1.  Up until now, if you added a comment with a special tag such as //TODO: or //HACK:, they would show up in the Task List tool window, but only for open windows.  I spent a few hours (years ago) coming up with a organization scheme for using these, creating my own tags (under Tools—>Options—>Environment—>Task List) and placed them all over my code as reminders.  It’s a much more natural place to put extremely detailed tasks (as opposed to product feature requests) because comments are attached to the very bits of code that require changing, and all comments get checked in and versioned with the code.  So I was quite dismayed when they started disappearing as I closed one code window after another.  With SP1, these comments remain in the task list!  Hoo rah!  I think I’m more excited about this feature than the steroid-pumped squiggles.

Posted in Visual Studio | Leave a Comment »

iPhone 3G: Amazing Stupidity in an Awesome Product

Posted by Dan Vanderboom on August 19, 2008

I just picked up my iPhone 3G yesterday.  After a couple years of suffering with Windows Mobile phones (and four years developing software for them), I decided that I’d had enough pain for one lifetime.  I want a device that does what I need it to do, and does it well, and I refuse to pull out a stylus or peck at impossibly small desktop-like drop-down menus any more.  Yes, I know that HTC has some amazing new touch-friendly interfaces (the Diamond, for example), but for one, I can’t wait any more; and two, it’s likely to be some kind of GUI polish stuck on top of the same ugly technology stack that’s always been there.  What happens when you want to install third party applications?  Are they going to magically fall into the same touch-friendly UI patterns?  Is HTC going to provide a controls library to support transparency, slick animations, and similar visual candy for developers of third-party applications?  I think not.  Windows Mobile is a dead end.

My first impressions of the iPhone?  It’s beautiful.  It’s elegant.  Google maps and navigation are brilliant.  Visual voice mail is a dream.  Skimming quickly through large lists of data is a breeze.  I love it.  Almost everything about it.

There’s one problem, and it irritates me… even infuriates me.  I’m referring to the lack of support for A2DP (stereo Bluetooth).  Aside from a bug in v1.1.1 of the firmware that inadvertently allowed music to stream to Bluetooth headsets, this feature—though possible to support in software—has been crippled.  After searching online last night for a few hours, I couldn’t find any statements from Apple or confirmed reasons elsewhere, but I speculate it has something to do with protecting music by not allowing it to be broadcast to other devices.

If this is the reason (and I’m not saying it is), it’s incredibly stupid.  It won’t stop people from using A2DP.  However, it will force people to spend lots of money ($60-70) for a bulky adapter to plug into the iPhone (in addition to $100 or more for the headset), or to hack the device to enable the feature.  I’ll be looking for option B, as I run a great deal and don’t want anything to increase the size to make the thing stick out of a pocket, or risk having it fall.

Apple has had a lot of time to support stereo Bluetooth, plenty of time to figure out the technical details.  They just don’t seem interested in giving in to their customers and users on this issue, regardless of the public outcry and obvious demand.  That’s amazing stupidity for an otherwise awesome product.  And that’s why we hack their devices.

Posted in iPhone, Mobile Devices | Leave a Comment »