Note: this article is part 6 of a series called Accelerated Velocity. This part can be read stand-alone, but I recommend that you read the earlier parts so as to have the overall context.
Inefficiency drives me crazy. Its like fingernails on a chalkboard. When I’m the victim of an inefficient process, I can’t help but stew on the opportunity costs and become increasingly annoyed. This sadly means I’m quite often annoyed since inefficiency seems to be the natural rest state for most processes.
There are lots of reasons why inefficiency is the norm, but in general they fall into one of the following categories:
1) Poor process design
2) Poor process execution
3) Entropy and chance
4) External dependencies
The good news in software development is that Lean/agile best practices and reference implementations cover process design (#1). Process execution (#2) can likewise be helped by hiring great people and following agile best practices. Entropy (#3) can’t, by definition, be eliminated but the effects can be mitigated by addressing the others effectively.
Which leaves us with the bane of efficient processes and operations: dependencies (#4).
Simply put, a dependency is anything that needs to happen outside of the process/project in question in order for the process/project to proceed or complete. For example, a software project team may require an API from another team before it can finish its feature. Likewise a release may require certification by an external QA team before going to production. In both cases, the external dependency is the point where the process will likely get stuck or become a bottleneck, often with ripple effects extending throughout the system. The more dependencies, the more chances for disruption and delay.
So how does one reduce the impact of dependencies?
The simplest way is to remove the dependencies altogether. Start by forming teams that are self-contained, aligned behind the same mission, and ideally report to the same overall boss. Take, for example, the age-old divisions between product, development, QA, and operations. If these four groups report to different managers with different agendas, then the only reasonable outcome will be pain. So make it go away! Put them all on the same team. Get them focussed on the same goals. Give them all a stake in the overall success.
Second, distribute decision making and control. Any central governance committee will be a chokepoint, and should only exist when (a) having a chokepoint is the goal, or (b) when the stakes are so high that there are literally no other options. Otherwise push decision-making into the teams so that there is no wait time for decisions. Senior management should provide overall strategic guidance and the teams should make tactical decisions. (SAFe describes it well here.)
In 2014, Bonial carried a heavy burden of technical and organization dependencies and the result was near gridlock.
At the time, engineering was divided into five teams (four development teams and one ops team), and each team had integrated QA and supporting ops. So far, so good. Unfortunately, the chokepoints in governance and the technical restrictions imposed by a shared, monolithic code-base effectively minimized independent action for most of the teams, resulting in one, large, inter-connected mega-team.
There was a mechanism known as “the roadmap committee” which was nominally responsible for product governance, but in practice it had little to do with roadmap and more to do with selective project oversight. One of the roadmap committee policies held that nothing larger than a couple of days was technically allowed to be done without a blessing from this committee, so even relatively minor items languished in queues waiting for upcoming committee meetings.
What little did make it through the committee ran directly into the buzzsaw of the monolith. Nearly all Bonial software logic was embedded in a single large executable called “Portal3”. Every change to the monolith had to be coordinated with every other team to ensure no breakage. Every release required a full regression test of every enterprise system, even for small changes was on isolated components. This resulted in a 3-4 day “release war-room” every two weeks that tied down both ops and the team unfortunate enough to be on duty.
It was painful. It was slow. Everyone hated it.
We started where we had to – on the monolith. Efforts had been underway for a year or more to gradually move functionality off of the beast, but it became increasingly clear with each passing quarter that the “slow and steady” approach was not going to bear fruit in a timeframe relevant to mere mortals. So our lead architect, Al, and I decided on a brute force approach: we assembled a crack team which took a chainsaw to the codebase, broke it up into reasonably sized components, and then put each component back together. Hats off to the team that executed this project – wading through a spaghetti of code dependencies with the added burden of Grails was no pleasant task. But in a few months they were done and the benefits were felt immediately.
The breakup of the monolith enabled the different teams to release independently, so we dropped the “integrated release” process and each team tested and released on their own. The first couple of rounds were rough but we quickly hit our stride. Overall velocity immediately improved upon removing the massive waste of the dependent codebase and labor-intensive releases.
The breakup of the monolith also untethered the various team roadmaps, so around this time we aligned teams fully behind discreet areas of the business (“value streams” in SAFe parlance). We pushed decision making into the teams/streams, which became largely responsible for the execution of their roadmap with guidance from the executive team. The “roadmap committee” was disbanded and strategic planning was intensified around the quarterly planning cycle. It was, and still is, during the planning days each quarter that we identify, review and try to mitigate the major dependencies between teams. This visibility and awareness across all teams of the dependency risk is critical to managing the roadmap effectively.
Eventually we tried to take it to the next level – integrating online marketing and other go-to-market functions into vertically aligned product teams – but that didn’t go so well. I’ll save that story for another day.
The breakup of the monolith and distribution of control probably had the biggest positive impact in unleashing the latent velocity of the teams. The progress was visible. As each quarter went by, I marveled at how much initiative the teams were showing and how this translated into increased motivation and velocity.
To be sure, there were bumps and bruises along the way. Some product and engineering leaders stepped up and some struggled. Some teams adapted quickly and some resisted. Several people left the team in part because this setup required far more initiative and ownership than they were comfortable with. But in fairly short order this became the norm and our teams and leaders today would probably riot if I suggested going back to the old way of doing things.
Some closing thoughts:
- Organize teams for self-sufficiency and minimal skill dependencies
- Minimize or eliminate monoliths and shared ownership
- Keep the interface as simple, generic and flexible as possible when implementing shared systems (e.g. APIs or backend business systems)
- Build transparent about dependencies and manage them closely