The Only Valid Architecture is Validated Architecture

I have worked in the IT field for over 3 decades now, and I’ve seen a lot of effort expended in design, architecture and infrastructure areas to build out some incredible and fantastic platforms and underlying systems.  However, I’ve never seen a 100% utilization of that effort.  In general a large part of the work is never used or, even worse, becomes a blocker to future agility.  Why do we do that?  Why do major components of these architectural designs, many built by some of the most intelligent people I know, go unused?  From my experience, it is always due to our tendency to separate architecture usage from business feature usage.  Architecture alone does not add to our bottom line or overall success.  It is only when that architecture or design enables us to solve customer problems that business value is achieved.

Architecture, design, UX, and any other similar effort should only be expended in the pursuit of supporting business value.  Yes, Architects, Designers, etc, I do mean that without the business value you support you have no reason to do the work.  And, even more importantly, unless you are building architecture to directly support currently needed business value, you have no way of validating if you have designed the right architecture!  Only after validating your design by seeing it enable business value do you know if you have built a valid architecture.

From an Agile perspective, this makes sense.  Agile Principle # 10, the art of maximizing the amount of work not done, stresses that the simplest solution is often the best solution.  From a Lean perspective we are pushing to eliminate waste in the system; unused architecture is a huge source of waste (as well as quality issues).  Add in the Lean Startup perspective, which brings to the table that sense of experimentation to gain knowledge quickly, and we gain the perspective that true validation only comes from the customer.  Then apply all 9 SAFe® principles (yes, Intrinsic Motivation counts) and you start to see the picture that we should only consider architectural effort well spent and validated once we see it supporting business value and solving customer problems.

“But, wait!” you say, “if we delay creating architecture/infrastructure/design, we end up with a fragile mess!”  True enough, we definitely need to have intentional architecture so that we have a consistent and supportable direction with our designs.  We absolutely need to be looking ahead for the architecture, designs, patterns, infrastructure, and all the other needed components to support business value.  Enter the SAFe® Architectural Runway.  This Runway combines Intentional Architecture and Emergent Design to ensure that we are building the right amount of architecture up front, but ensuring that all of our efforts can be quickly validated by supporting actual business value.

Intentional Architecture

Where are we going with this solution?  What framework, capability, etc will be needed to support future business value?  What do we need in place to avoid future performance issues?  Are we headed in a direction that supports future security concerns?  Those are all things that need to be discussed and planned for.  However, each discussion we have should be accompanied by “what business value is upcoming that can validate this is the right direction?”

Emergent Design

Emergent Design is a core component of validating architecture incrementally.  The ability to create a ‘walking skeleton’ of the intent and then allowing the details of the design to emerge from the teams each increment allows us to validate the value of each architectural component as we progress.  The key is to ensure that we are establishing our measurements and leading indicators of the viability of the architecture and design we are pursuing.  Having a direction is the first step, but then we need the teams to build to that intention, and then stack business value on top to validate the intention. 

For example, let’s assume you are trying to move your application base to the cloud.  Your assumption is that moving your entire infrastructure to the cloud will result in cost savings and the ability to scale capacity much more quickly.  However, you have a massive amount of applications to manage, that all seem to be inter-connected, and you cannot interrupt operations.  In addition, most of your footprint is legacy apps that are client-server or mainframe based design.  A traditional mindset would state that you need to design a cloud capability to support all of these apps and their connectivity (which means migrating a number of mainframe apps) which will require a massive application and workflow design.  And you are correct, you do need a plan, but as von Moltke stated “No Battle Plan Survives Contact With the Enemy”.  Applying Intentional Architecture along with Emergent Design is the way to survive this massive effort.

Instead of pursuing a big bang approach to this effort, pursue an incremental, learning based approach to the architecture and design.  What early indicators would help prove we have the right concept of how to move to the cloud?  What early value can we pursue that will not only help us determine the right direction, but also confirm the perceived value?  The first step is to have a firm plan that is easily changed.  Create a clear vision of where you want to go, and your approach to getting there, but create the plan with a high level of abstraction.  Avoid the locked in design that can result from going too deep too quickly.  For each component of the design ask yourself “How does this support the business outcome we are looking for?”

Next, determine the architectural/design areas that are the most mission critical or present the most risk.  It is important to isolate these areas to be targeted for early learning.  Look for areas that will not only gain knowledge on the architectural direction, but also support the most business value, both of which will provide faster feedback on future direction.  Avoid the ‘sacred cows’, the areas that you want to put in, but don’t really need.  Add in the understanding of how you will measure this progress, looking for leading indicators that will help to provide faster pivot or pursue moments.  For example, if part of your reasoning for moving to the cloud is cost savings, determine how you can measure cost savings with each increment.  Sometimes you have to extrapolate or use non-monetary indicators early on, but get as quickly as you can to real savings measurements.

Now, build the bare minimum architecture you can to gain the knowledge you need, e.g. to move your metric needle forward.  If your assumption is message based connectivity in the cloud, but you are using file based communication in many areas, can you first get these apps to talk via a simple message queue or service bus?  Do you really need to move to the cloud before you have built the first step of communication?  As you build these incremental steps you start to play leapfrog: a little architecture, a little business, rinse and repeat, all while keeping an eye on the end target.

Both Emergent Design and Intentional Architecture require Validation to be successful.  The Architectural Runway is there (in part) to ensure that we are recognizing that need for validation.  The next time you are thinking of this cool new platform concept, wanting to implement the latest My/No/Yours SQL, remember to think “what business value needs this?  What business value can we build on this capability to validate we are going the right direction with our intent?  How can we quickly measure the success of the design?”  Then, build that business value on the early iterations of that architectural work, and look for customer validation for course validation or correction.

Solution Accuracy over Quality

In the last few years (maybe even decade) there has been a strong surge towards quality in our product development.  Not just limited to software, this focus has led to much stronger practices and tools leading to much more robust, scalable, and resilient products.  While this is in general a highly needed change from the somewhat sloppy work from previous generations of developers, but I believe we are missing the point in not applying the same, or more, attention to the accuracy of what we are building. 

Think about it.  If you are delivering a solution that is near 100% quality, what good does it do if it is not the optimal solution?  Regardless of your definition of quality, If I create a product that meets 100% of those levels, but does not solve the customer problem or advance the ability to use the product, did I do any good?  As an extreme example, if I write the highest quality game application that could exist, but what the customer needed was banking functionality, did I really add any value?  Don’t get me wrong, I am a huge advocate of quality in all products, and teach and coach software craftsmanship as a part of my work with my customers.  However, we have to balance that out with the accuracy of the solution.  Our attention to ‘Zero Defects’ has lead us away from ‘Solved the Problem’

Solution Accuracy, on the other hand, has a strong focus on learning, analyzing, measuring, validating and adjusting to deliver what our customers need.  Not what they tell us they need, but what they really need.  The adage “the customer doesn’t know what they want, they only know what they don’t want when they see it” is painfully true.  (Remember Henry Ford’s statement of “If I had asked customers what they want, they would have said faster horses”).  Accuracy is about incrementally working towards an optimal solution by applying the concepts of established Principle of Mission, iterating towards an optimal solution, and applying leading indicators that help us course correct or pivot as needed as we continue to study the impact on our customers of our emerging capability(ies).  From my years of working with Fortune 100 companies I have experienced time and again the lack of attention to accuracy and optimal solutions, instead following a Crystal Ball plan of predicting/forecasting what the customer needs, and most often arriving well short of the target.

Why is this a problem?

Assume Variability, Preserve Options

As an Enterprise Transformation coach and SAFe SPCT, I rely heavily on two core aspects to raise awareness to the solution accuracy problem.  The first is SAFe’s Principle #3 “Assume Variability, Preserve options”.  In general, this principle shows us the benefit of not relying on predictive ‘Point Based’ solutions that rely on far too little knowledge and information to state that we know what the customer needs/wants.  The typical 3-9 month corporate project that relies heavily on not only a stated outcome, but also a direct statement on how to achieve that outcome, usually accompanied by a project plan with everything detailed out to the low-level task.  If we were to honestly reflect on the results of this predictive planning we would quickly see that these efforts rarely are on time and within budge, but even worse, rarely actually solve the problem at hand. 

SAFe Principle #3 Assume Variability, Preserve Options.  Copyright Scaled Agile Inc

When we “Assume Variability” we understand that we don’t have the full level of knowledge yet (that will be gained as we iterate towards a solution) and that we should assume that our current assumptions are invalid because of the lack of that yet to be gained knowledge.  It is important to have assumptions of what the best path forward is, but when we assume variability exists we accept that we are most likely wrong (sometimes very wrong).  Assuming Variability says that we know we are going to learn more, so let’s take advantage of that variability and iterate towards the solution with a plan and process that inherently incorporates that new knowledge as we progress.

To ‘Preserve Options’ means that we never lock ourselves into a corner, never stating “this is the only way to solve this problem”, until we have gathered the most knowledge we can.  It’s great to have an assumed best path, but we also need to acknowledge, and sometimes pursue, other options that may turn out to be the better path.  Yes, that does mean sometimes you will pursue a solution that will be deprecated or dropped based on knowledge that indicates it is no longer a viable path.  If that sounds like wasted work, I would gladly trade that level of ‘waste’ for the waste we find when we pursue a point based solution and have the resulting adjustments, changes, and sometimes project cancellations due to chasing the wrong path. 

Tying these two together means we start out by allowing and encouraging more than one possible solution direction, pursue one or more to gain knowledge as fast as we can, and pivoting or dropping assumed possible solutions when the knowledge shows we should.  The end result is an optimal solution with less effort because we were not forced into retrofitting the wrong solution like a square peg in a round hole.

Example of “Assume Variability, Preserve Options”

I was working with a client on this concept, and as they began to understand this principle they shared a story with me that is a great example.  This client is highly regulated, and needs to process documents for regulatory and audit reasons on a regular basis.  When this new requirement came to light, the business asked the IT group how they could solve it.  Being the technology savvy group they were, they stated they could create an automated system that would process the documents as needed, at a cost of $8 million and about 6 months of effort.  They completed the project a little over budget and a little late, but since this was the norm they considered the project a great success.  Until they turned it on.

The first month the system only processed 3 documents based on the needed updates.  The second month?  2 documents.  The third?  3 documents.  The result after one year was that each document cost about $67k to process.  They then realized that if they had “Preserved Options” they would have realized that an alternative option would have been to hire a temp to come in one Saturday a month to process the documents, at a cost of approximately $75 per document.  Because they had not recognized the need to “Assume Variability” in that they did not yet know the volume of documents, they went to a point based solution that made sense.  Bear in mind that the automated system was still a viable option, but since the manual was such an easy approach it most likely would have been the first option to pursue.    The ‘quality’ of the effort was high based on the low number of defects in production, but the accuracy was 180 degrees from the optimum solution.

Focus on Solution Accuracy

Focusing on solution accuracy does not mean being predictive, it does not mean having to know all the answers at the start.  In fact, it assumes we don’t have all the answers, and incorporates learning early and adjusting based on gained knowledge throughout the effort.  Another coach just sent me an email with this quote on her signature line:

“Progress means getting nearer to the place you want to be. And if you have taken a wrong turning, then to go forward does not get you any nearer.If you are on the wrong road, progress means doing an about-turn and walking back to the right road; and in that case the man who turns back soonest is the most progressive man.”

C.S Lewis

Our goal in focusing on solution accuracy is trying to get to the most valuable and informative pivot or pursue moments as soon and as often as we can, and making those adjustments as needed in pursuit of the optimum solution.