The AwkEng Avoids Integration Risk by Integrating

Hi all,

Today's post shares some of my mental models for systems development. These models apply to software; they're harder to apply to hardware, but the mindset will serve hardware engineers well. Readers beware, many of my favorite quotes and phrases to follow.

So, I have a core belief that integration risk is one of the most underestimated risks in any project, and that mitigating that risk is the key to success. In other words, it's not making the pieces, it's getting them to work together that's truly hard. Projects that seem to be on track, but have have delays that appear towards towards the end of the schedule, are a classic sign of the realization of integration risk.

The easiest way to drive out integration risk is to actually integrate. The sooner you do it, the better. To quote an old mentor of mine, "how late do you want find out you're late?" There's nothing like the hard truth of transmitting a message between systems, or running a mechanical fit check, or a full up demo, to know where you really stand. And "passing" the integration test is usually pretty clear - it works or it doesn't - which tends to head off questions of "done" as in "code complete done" vs "done" as in "done-done."

Another way to phrase this is to say "you can't have integration risk if you're always integrated."

 In many ways, this is just anther take on one of the principles from the Agile manifesto, that "progress is measured in terms of working software."

Applying this in practice

Making all this work in practice requires a shift in mental models for what defines "working software" from a local view to a systems view. It also helps to stretch the definition of "working" to a place where someone asks "is it working?" and the answer is "well... technically yes, but we can only do x, y, and z."

The litmus test for "working" is whether or not all the existing code paths are being used. Any unused code paths are a form of work in progress, or WIP. The lessons from lean manufacturing tell us to minimize WIP as much as possible.

(A classic example of why reducing WIP is important: It may seem locally optimal to set up a stamping machine once and crank out a million parts to amortize the cost of machine setup, but the risk of generating scrap goes up. You may be making bad or unneeded parts, and you're certainly tying up money and working capital in inventory.)

In a similar way, writing unused code paths may seem locally optimal (i.e. one engineer can proceed unblocked and build out the back end half of an API without stopping), but the more unintegrated code paths that are written, the greater the risk of an integration defect, and the longer the delay at the system level for delivering working software.

An example to illustrate

This example, may be reductive, but take a simple task list feature within a React application. There's a front end to serve the app and a back end API connected to a database. It's possible to read the task list, add new tasks, or delete tasks.

Breaking down the work into a set of front end components like

1. list display
2. create button

3. delete button and

   back end APIs like

4. get list
5. create item

6. delete item

   results in exactly the sort of "unused code paths" that I advocate against.

In my mental model of technically working but only xyz systems development, I'd break the work down like this:

1. Can we pretty print a JSON blob in the browser of a list? (means front end, back end, and connection to database all work. It may be ugly, but it's technically working.)
2. Add a nicer front end component, so we're not looking at JSON
3. Can we demonstrate the "create feature?" (front and back both need to work)
4. Can we demonstrate the "delete feature?"
5. Do we have alarms and monitoring systems in place for all of the above?

(I just noticed that my mental model includes asking pointed questions, which is probably a pro tip.)

On paper, it's the same amount of work either way. In practice, I've found that the integrated model is faster and more fun. It creates a palpable sense of collaboration and creates a sense of team progress, rather than individual accomplishment, to check an item off the list. And integration bugs are caught early, which is the best time to find them.

So, ship early, ship often, ship full integrated systems!

cheers!
Sam
aka THE Awkward Engineer