Special Forces Architecture

Architects scanning for serious design flaws

 

I’ve been spending some very enjoyable time recently with our architecture team working through some of the complexities that we’ll be facing in our next planning iteration.  Many of those topics make for interesting posts in their own right, but what I want to discuss in this post is the architecture team itself.  

Why?  Because I’m pretty happy with how we’ve evolved to “do” architecture here.  

And why is that noteworthy?  Because too many of the software architecture teams I’ve worked in, with or around have had operating models that sucked.  In the worst cases, the teams have been “ivory tower” prima donna chokepoints creating pretty diagrams with methodological purity and bestowing them upon the engineering teams. At the other end of the scale, I’ve seen agile organizations run in a purely organic more with little or no architectural influence until they ran up against a tangled knot of incompatible systems and technologies burdened with massive architectural debt.  And everything in between.

So, how do we “do” architecture at Bonial?  I think it helps to start with the big picture, so I brainstormed with Al Vilegas (our chief architect) and we came up with the following ten principals that we think clearly and concisely articulate what’s important when it comes to architecture teams:

  1. Architects/teams should think strategically but operate tactically.  They should think about future requirements and ensure there is a reasonable path to get there.  On the flip side, only just enough should be developed iteratively to meet the current requirements while leaving a reasonable runway. 
  2. Architects/teams should have deep domain expertise, deep technical expertise, and deep business context.   Yes, that’s a lot, but without all three it’s difficult to give smart guidance in the face of ambiguity – which is where architects need to shine.  It takes time to earn this experience and the battle scars that come with it; as such, I generally call BS when I hear about “architects” with only a few years of experience.
  3. Architects must be team players.  They should be confident but humble.  Arrogance has no place in architecture.  They should recognize that the engineering teams are their customers, not their servants and approach problems with a service-oriented mindset.  They should listen a lot.  
  4. Architects/teams should be flexible.  Because of their skills and potential for impact, they’ll be assigned to the most important and toughest projects, and those change on a regular basis.  
  5. Architects/teams should be independent and entrepreneurial.  They should always be on the lookout for and seize opportunities to add value.  They shouldn’t need much daily or weekly guidance outside of the mission goals and the existing/target enterprise architecture.  They should ask lots of questions and keep their finger on the pulse of the flow of technical information.
  6. Architects must practice Extreme OwnershipThe should embrace accountability for the end result and expect to be involved in projects from start to finish. This means more often than not that they will operate as part of the specific team for the duration of the project.  They may also assist with the implementation, especially the most complex or most strategic elements.  “You architect it, you own it.”
  7. Architects/teams should be solid communicators.  They need to be able, through words, pictures and sometimes code, to communicate complex concepts in a manner that is understood by technical and non-technical people alike.  
  8. Architects/teams should be practical.  They need to be pragmatic and put the needs of the business above technical elegance or individual taste.  “Done is better than perfect.”
  9. Architects/teams should be mentors.  They should embrace the fact that they are not only building systems but also the next generation of senior engineers and architects.  
  10. Architects/teams must earn credibility and demonstrate influence.  An architect that has no impact is in the wrong role.  By doing the above this should come naturally.

If you take the principles above and squint a little bit, you’ll see more than a few analogs to how military special forces teams structure themselves and operate, as illustrated below: 

High-performing Architecture Teams Military Special Forces Teams
Small Small
Technical experts and domain specialists Military experts and domain specialists
Extensive experience, gained by years of practice implementing, fixing and maintaining complex systems Extensive experience, gained by years of learning through intense training and combat
Flexibly re-structure according to the mission Flexibly re-structure according to the mission
High degree of autonomy under the canopy of the business goals and enterprise architecture High degree of autonomy under the canopy of the mission and rules of engagement
Often join other teams to lead, support, mentor and/or be a force multiplier Often embed with other units to lead, support, mentor and/or be a force multiplier
Accountable for the end results Accountable for the mission success

Hence the nickname I coined for this model (and the title of this post): “Special Forces Architecture.”  

How does this work in practice?  

At Bonial, our 120 person engineering team has two people with “architect” titles, but another half dozen or so that are working in architecture roles and are considered part of the “architecture team.“  An even broader set of people, primarily senior engineers, regularly attend a weekly “architecture board” where we share plans and communicate changes to the architecture, generally on a weekly basis.  We recognize that almost everyone has a hand in developing and executing our architectural runway, so context is critical.  To paraphrase Forest Gump: “Architect is as architect does,” so we try to be as expansive and inclusive as possible in communicating.

The members of the architecture team are usually attached to other teams to support major initiatives, but we re-assess this on a quarterly basis to make sure the right support is provided in the right areas.  In some cases, the architecture team itself self-organizes into a project team to develop a complex framework or evaluate a critical new capability.  

Obviously there’s a lot going on – we typically have 8-10 primary work streams each with multiple projects – so the core architecture team can’t be closely involved with every project.  To manage the focus, we use a scoring system from 1-5 (1 = aware, 3 = consulting, 5 = leading) for what level of support or involvement is provided to each team or initiative.  In all cases, the architects need to ensure there’s a “big picture” available (including runway) and communicated to all of the engineers responsible for implementing.

For example, right now we have team members embedded in our critical new content management and publishing platform and our Kraken data platform.  We have one person working on a design and PoC updating the core user model.  Several members of the team are also designing and prototyping a new framework for managing machine learning algorithm lifecycles and testing.  And a few people have individual assignments to research or prepare for future runway topics.  In this way we expect to stay just far enough in front of the rest of the team to create runway without wasting time on phantom requirements.

Is this model perfect?  No.  But perfection isn’t our goal, so we optimize for the principles above – adaptability, autonomy, expertise, ownership, impact – and build around that.  Under this model, the Bonial platform has gone from a somewhat organic collection of monolithic apps running on a few dozen servers to a coherent set of domains consisting of APIs, micro-services, legacy systems and complex data stores running on hundreds of cloud instances across multiple continents.  I have my doubts that this would have happened in some of the more traditional models. 

I’m happy to answer questions about this model and talk about the good and the bad.  I’d also love to hear from you – what models have worked well in your experience?

The Micro-service Conundrum

 

Micro-services have been the rage in software circles over the past couple of years.  A natural evolution of service oriented architectures (SOA), and popularized by successful implementations at companies like Spotify, Soundcloud and many others, micro-services have become the “must have gadget this holiday season”: if you aren’t doing them, you must be doing something wrong.  

But is that true?  As much as people (and especially engineers) love black and white, the answer here is a firm “maybe.”  Here are some of the positives and negatives from one CTO’s perspective.

On the plus side, micro-service architectures provide an excellent canvas for rapid development and continuous integration.  Hard dependencies are minimized, business logic is localized, and the resulting services are typically cloud ready.  Developers tend to like micro-services because it allows for a great deal of independence.  It’s hard to understate the potential pain savings and optimizations – people, process and technology – that can be driven by moving to this type of architecture.

But it doesn’t come for free.  For starters, you’ll likely have a lot more moving pieces in terms of individual components and running executables.  A few weeks ago I wrote a post on the architectural heuristic: Simplify Simplify Simplify in which I posited that simple is better when it comes to minimizing TCO.  In that vein, one must ask if micro-services follow the rule.  Yes, each individual service itself is simpler than a bloated monolith as a result of the small size and tight boundaries.  But the total business logic in your enterprise hasn’t changed, and now you may have hundreds or thousands of additional code modules to manage and executables to orchestrate.  The good news is that cloud hosting providers like AWS provide an ever increasing set of tools to help with managing micro-service architectures (e.g. Lambda, Container Services), but it still requires a good deal of cultural and process change.

Another side effect of the proliferation of executables is potential increase in cost – many hosting providers and software vendors (e.g. APM providers) still price based on number of processes or agents.  If you take the same processing load and 10X the number of running processes, you might find yourself in a world of hurt pretty quickly.

Finally, in moving to micro-services, you’ll find yourself needing to address a host of new challenges that you may not have had to previously – service discovery, versioning, transactions and eventual consistency, event tracing, security, etc.  At a minimum, the upside benefits you’ll realize will be offset by developing competency and code to solve those new challenges.

So, what does this mean for the typical company.  If you have applications that are bloated monoliths, those are fantastic candidates for breaking down into smaller components or micro-services.  On the other hand, if you have a reasonably well architected system with decent boundaries in place already, I’d carefully weight the cost-benefits – maybe run a few trials projects to get a better sense of how it would fit into your platform.  Just realize that in many ways you’re “squeezing the balloon” – trading one set of problems for another.  So long as you’re happier with the new problems (and the corresponding benefits), you win.

In closing, whether you move to micro-services or not, I do think there are great lessons to be learned from applying the discipline required by micro-services – namely, enforcing clear boundaries around business logic and using “API thinking” to service a variety of clients.  I wonder if there isn’t a compromise to be had in which one uses the principles for developing and organizing the code, but you still deploy in a more constrained manner – “Code Micro, Deploy Macro.”  But that’s a discussion for another time. 

Simplify Simplify Simplify

Near the end of my first year in my first development job at Andersen Consulting (now Accenture) I was handed an exciting new project: to develop a Monitoring And Reporting Server for a major bank’s financial systems.  And to do it in a few weeks.

This was the mid-1990’s – in the very early days of the web and long before the ELK stack and other tools that would have made this pretty straightforward.  In addition, we were developing this with C/C++ and had to write it all from scratch as open source was still in its infancy and Java and C# were still to come.  

But what software engineer doesn’t like these kinds of challenges?  I was feeling pretty good about myself – I’d had a couple of solid wins up to this point, so in some ways this project was a nod to my success so far and a challenge to step up from pure coding to design and architecture.

Under the tutelage of my supervisor at the time I sank my teeth into it.  Given the challenges – no open source, unforgiving programming environments, limited compute and memory, limited time, etc. – you’d think I’d try to keep it simple.  Right.  On paper (yes, we still used paper) I created an incredibly sophisticated design with input channels (“Translators” in design pattern parlance, but even design patterns were in their infancy), normalization layers, storage mechanisms, and “bots” to look for anomalies.  It had self monitoring and auditing and adapters for new pluggable streams.  For good measure I designed it all to interoperate with other systems and technologies using a (complex and slow) CORBA ORB.  I abstracted everything just in case some future unknown requirement might require extensions or adaptions.  I was very proud of it.

It was never completed.  

Thank goodness.  While I was disappointed at the time, I realize now that this creation was designed to be a total monstrosity and likely a failure.  Soon into the coding I was already bogged down with massive issues in keeping track of the (hand written) thread pools as well as challenges managing the complexity of the modular system builds.  Had it been completed it would have taken a team of rocket scientists to understand and maintain what was, in the end, a pretty simple concept.  Fortunately another project came along with higher priority and I was put on that.  (Plus, though it was never said, I think my supervisor knew deep down that my approach was leading us to a bad place.)

What went wrong?  Easy – I’d violated the first rule of software architecture: SIMPLIFY SIMPLIFY SIMPLIFY. 

Duh!  Right?  Anyone in the field for more than a few seasons knows that more complexity leads to higher risk, more bugs, longer development times and greater total cost of ownership (TCO).  But we do it anyway.  The wisdom gets lost in the excitement of creating something big and beautiful.

It happens all the time. Even with all of the “bumpers” to keep us honest these days – solid design patterns built into many of the open source frameworks, Lean software practices that naturally select for simplicity – I still see it violated on a regular basis.  I spent several years consulting for large corporations, largely fixing violations to this rule.  Even today some of the biggest challenges my team faces at Bonial come from systems and projects that were simply too complex for the work they did.  

So, how do you implement this principle in practice?  There’s no magic – you just challenge yourself or, better yet, have the team challenge you.  When you’ve come up with your approach or design, ask yourself, “Ok, now how can I make this 30% simpler?”  Look for over-design especially abstractions and facades that aren’t really needed or could be added later.  Challenge the need for all concurrency implementations especially when app servers are in play.  Look hard at the sheer number of moving pieces and ask whether one object / module / server will suffice?  And always look for software, best practices and frameworks that already do what you’re trying to do – the simplest thing is not having to do it at all.