Tag: software development

Accelerated Velocity: Getting Uncomfortable

Note: this article is part 10 of a series called Accelerated Velocity.  This part can be read stand-alone, but I recommend that you read the earlier parts so as to have the overall context.

“Confident.  Cocky.  Lazy.  Dead.” This admonishment against complacency was the mantra of Johnny “Dread” Wulguru, the villain in Tad William’s Otherland saga.  As true as it is for assassins like Dread, it’s also true (though perhaps not as literally lethal) for teams and companies that choose to rest on their laurels and stop challenging themselves.

Complacency is the enemy of innovation.  This has been proven over and over throughout history in every domain as once successful or dominant players suddenly found themselves lagging behind.  This is also a leadership failure.  Good leaders strive to prevent comfort from becoming complacency. 

Jeff Bezos at Amazon has baked this into the DNA of the company as enshrined in his “Day 1” message to shareholders.  Of note is what Mr. Bezos says happens when companies get comfortable on Day 2:

“Day 2 is stasis. Followed by irrelevance. Followed by excruciating, painful decline. Followed by death. And that is why it is always Day 1.”

Confident.  Cocky.  Lazy.  Dead.

It’s not easy, though.  Comfort is the reward for success after all.  “Don’t fix something that ain’t broke.”  Right?

Wrong.  The reward for success is being in position to hustle and build on that success.  Period. 

Getting Uncomfortable

It’s no different in software development teams. 

By 2016 we’d made significant strides in velocity and efficiency in Bonial product development.  Processes were in place.  An architecture roadmap existed.  Teams were healthy.  The monolith was (mostly) broken.  AWS was being adopted.  All signs pointed to very successful changes having taken place. 

At the same time I felt a certain sense of complacency settling in.  The dramatic improvements over the past couple of years had some thinking that it was now “good enough.”  Yet we still had projects that ran off the rails and took far longer than they should have.  We still couldn’t embrace the idea of an MVP.  We still had mindsets that change was dangerous and scary.  And, perhaps most important, many had a belief that we were as fast as we could or needed to be. 

Yes, we were better and faster, but I knew we had only begun to tap our potential.  It was time to get uncomfortable.

Engineering Change

Changing a deep-seated mindset in a large organization using a head-on approach is tough.  An easier and often more effective approach is to engineer and successfully demonstrate change in smaller sub-groups and spread out from there.  Once people see what is possible or, better yet, experience it themselves, they tend to be quite open to change. 

So we looked for opportunities to challenge individual teams to “think different”.  For example, on several occasions the company needed an important feature insanely fast.  Rather than say no, we asked teams to work in “hackathon mode” – essentially, to do whatever it took to get something to market in a few days even if the final solution was wrapped in duct tape and hooked to life support.  Not surprisingly we usually delivered and the business benefitted massively.  Yes, we then had to spend time refactoring and hardening to make the solution really stable, but the feature was in the market, business was reaping the benefits and the teams were proud of delivering fast.

On another occasion we had a team that struggled with velocity due, in part, to lack of test automation and an over-reliance on manual testing.  So I challenged the team to deploy the next big feature with zero manual testing; they had to go to production with only automated tests.  This made them very uncomfortable.  I told them I had their backs if it didn’t work out – they only needed to give it their best shot.  To their credit, the team signed up for the challenge and the release went out on time and had no production bugs.  This dramatic success made a strong statement to the rest of the organization.

Paradigm Shift

We also took advantage of our new app ecosystem.  Over the past few years the company has started several new “incubation” initiatives to explore new possibilities and expand our product portfolio.  We didn’t want to do these initiatives with our core product development teams because (a) we didn’t want to be continually wrestling with questions as to whether to focus on the new or old products, and (b) we feared that doing things like the “core” teams would be too slow. 

So we spun out standalone teams with all of the resources needed to operate independently.  Not surprisingly these “startup” teams moved much faster than any of our core teams.  In part this was because they were not burdened by legacy systems, technical debt, and risk/exposure of making mistakes that affect millions of users. But I think the bigger part was sheer necessity.  We ran our incubation projects like mini startups – they received funding, a target and a timeline and they had to hit those targets (or at least show significant progress) in order to receive more funding.  As a result, the teams were intensely focussed on delivering MVPs as quickly as possible, measuring the results in the market, and pivoting if needed. 

Between 2015 and 2017 we ran three major incubation projects and each one was faster than the last.  The most recent, Fashional, went from funding to launch in less than 12 weeks, which included ramping up development teams in two countries, building web and native mobile apps and lining up initial partners and marketing launch events.  This proven ability to move fast made a strong statement to our other teams. 

We soon had “core” teams making adjustments and shifting their thinking.  Over the next few quarters, every team had embraced a highly iterative, minimalistic approach to delivery that enabled us to try more things more quickly and, when needed, take more aggressive risks.   Now each team strives to deliver demonstrable, user-facing value every sprint.  Real value, not abstract progress.  Just like the agile book says.  This isn’t easy but is fundamentally required to drive minimalistic, iterative thinking.  The result is dramatic improvement in velocity while having more fun (success is fun). 

For sure this hasn’t been perfect.  Even today we still have teams that struggle to plan and deliver iteratively and we still have projects that take way too long.  On the flip side we have a much deeper culture of challenging ourselves, getting uncomfortable and continually improving. 

Confident.

Closing Thoughts

  • It’s easy to become complacent, especially after a period of success.  This is deadly.
  • Leaders must act to remove complacency and force themselves and their teams to “get uncomfortable” and push their own limits.
  • Break the problem into smaller chunks.  Work with entrepreneurial teams on initiatives that challenge the status quo.  Have them show the way.
  • Reward and celebrate success and make sure you have the team’s back.  Honor your commitments.

Special Forces Architecture

Architects scanning for serious design flaws

 

I’ve been spending some very enjoyable time recently with our architecture team working through some of the complexities that we’ll be facing in our next planning iteration.  Many of those topics make for interesting posts in their own right, but what I want to discuss in this post is the architecture team itself.  

Why?  Because I’m pretty happy with how we’ve evolved to “do” architecture here.  

And why is that noteworthy?  Because too many of the software architecture teams I’ve worked in, with or around have had operating models that sucked.  In the worst cases, the teams have been “ivory tower” prima donna chokepoints creating pretty diagrams with methodological purity and bestowing them upon the engineering teams. At the other end of the scale, I’ve seen agile organizations run in a purely organic more with little or no architectural influence until they ran up against a tangled knot of incompatible systems and technologies burdened with massive architectural debt.  And everything in between.

So, how do we “do” architecture at Bonial?  I think it helps to start with the big picture, so I brainstormed with Al Vilegas (our chief architect) and we came up with the following ten principals that we think clearly and concisely articulate what’s important when it comes to architecture teams:

  1. Architects/teams should think strategically but operate tactically.  They should think about future requirements and ensure there is a reasonable path to get there.  On the flip side, only just enough should be developed iteratively to meet the current requirements while leaving a reasonable runway. 
  2. Architects/teams should have deep domain expertise, deep technical expertise, and deep business context.   Yes, that’s a lot, but without all three it’s difficult to give smart guidance in the face of ambiguity – which is where architects need to shine.  It takes time to earn this experience and the battle scars that come with it; as such, I generally call BS when I hear about “architects” with only a few years of experience.
  3. Architects must be team players.  They should be confident but humble.  Arrogance has no place in architecture.  They should recognize that the engineering teams are their customers, not their servants and approach problems with a service-oriented mindset.  They should listen a lot.  
  4. Architects/teams should be flexible.  Because of their skills and potential for impact, they’ll be assigned to the most important and toughest projects, and those change on a regular basis.  
  5. Architects/teams should be independent and entrepreneurial.  They should always be on the lookout for and seize opportunities to add value.  They shouldn’t need much daily or weekly guidance outside of the mission goals and the existing/target enterprise architecture.  They should ask lots of questions and keep their finger on the pulse of the flow of technical information.
  6. Architects must practice Extreme Ownership.  The should embrace accountability for the end result and expect to be involved in projects from start to finish. This means more often than not that they will operate as part of the specific team for the duration of the project.  They may also assist with the implementation, especially the most complex or most strategic elements.  “You architect it, you own it.”
  7. Architects/teams should be solid communicators.  They need to be able, through words, pictures and sometimes code, to communicate complex concepts in a manner that is understood by technical and non-technical people alike.  
  8. Architects/teams should be practical.  They need to be pragmatic and put the needs of the business above technical elegance or individual taste.  “Done is better than perfect.”
  9. Architects/teams should be mentors.  They should embrace the fact that they are not only building systems but also the next generation of senior engineers and architects.  
  10. Architects/teams must earn credibility and demonstrate influence.  An architect that has no impact is in the wrong role.  By doing the above this should come naturally.

If you take the principles above and squint a little bit, you’ll see more than a few analogs to how military special forces teams structure themselves and operate, as illustrated below: 

High-performing Architecture Teams Military Special Forces Teams
Small Small
Technical experts and domain specialists Military experts and domain specialists
Extensive experience, gained by years of practice implementing, fixing and maintaining complex systems Extensive experience, gained by years of learning through intense training and combat
Flexibly re-structure according to the mission Flexibly re-structure according to the mission
High degree of autonomy under the canopy of the business goals and enterprise architecture High degree of autonomy under the canopy of the mission and rules of engagement
Often join other teams to lead, support, mentor and/or be a force multiplier Often embed with other units to lead, support, mentor and/or be a force multiplier
Accountable for the end results Accountable for the mission success

Hence the nickname I coined for this model (and the title of this post): “Special Forces Architecture.”  

How does this work in practice?  

At Bonial, our 120 person engineering team has two people with “architect” titles, but another half dozen or so that are working in architecture roles and are considered part of the “architecture team.“  An even broader set of people, primarily senior engineers, regularly attend a weekly “architecture board” where we share plans and communicate changes to the architecture, generally on a weekly basis.  We recognize that almost everyone has a hand in developing and executing our architectural runway, so context is critical.  To paraphrase Forest Gump: “Architect is as architect does,” so we try to be as expansive and inclusive as possible in communicating.

The members of the architecture team are usually attached to other teams to support major initiatives, but we re-assess this on a quarterly basis to make sure the right support is provided in the right areas.  In some cases, the architecture team itself self-organizes into a project team to develop a complex framework or evaluate a critical new capability.  

Obviously there’s a lot going on – we typically have 8-10 primary work streams each with multiple projects – so the core architecture team can’t be closely involved with every project.  To manage the focus, we use a scoring system from 1-5 (1 = aware, 3 = consulting, 5 = leading) for what level of support or involvement is provided to each team or initiative.  In all cases, the architects need to ensure there’s a “big picture” available (including runway) and communicated to all of the engineers responsible for implementing.

For example, right now we have team members embedded in our critical new content management and publishing platform and our Kraken data platform.  We have one person working on a design and PoC updating the core user model.  Several members of the team are also designing and prototyping a new framework for managing machine learning algorithm lifecycles and testing.  And a few people have individual assignments to research or prepare for future runway topics.  In this way we expect to stay just far enough in front of the rest of the team to create runway without wasting time on phantom requirements.

Is this model perfect?  No.  But perfection isn’t our goal, so we optimize for the principles above – adaptability, autonomy, expertise, ownership, impact – and build around that.  Under this model, the Bonial platform has gone from a somewhat organic collection of monolithic apps running on a few dozen servers to a coherent set of domains consisting of APIs, micro-services, legacy systems and complex data stores running on hundreds of cloud instances across multiple continents.  I have my doubts that this would have happened in some of the more traditional models. 

I’m happy to answer questions about this model and talk about the good and the bad.  I’d also love to hear from you – what models have worked well in your experience?

How we Plan at Bonial (part 3)

Collaborative digital stickies board that we use for planning.

Ok, after all that, how do we actually plan at Bonial?

The heart of our planning activities is the Quarterly Planning which is loosely modeled on Program Implement (PI) Planning from SAFe.  During quarterly planning / PI planning, everyone in the product development organization – developers, designers, architects, testers, product managers, operations specialists, designers, etc. – get together for a couple of days to map out their next phase.  We do our planning during the previous quarter’s HIP (Hardening Innovation and Planning) sprint, which is sprint 6 of each quarter.

Before I dive into the actual planning days, I should point out that the preparations start several weeks before when the product teams actively work with stakeholders, customer facing teams and the executive team to validates the backlogs against the current company priorities and business realities.  The prep phase looks something like this:

  • The senior management team and product strategy board review the overall strategy and primary business goals to assess if any change in focus is needed.  
  • Next we make sure that product and delivery management has the same level of clarity. We get the delivery leads and product owners together and communicate the company goals for the upcoming quarter to them, taking the time to answer questions about strategy, challenges, current market trends etc. Our goal here is to make sure that all our leaders are able to bring clarity to their teams so that local decisions are made with the right context.
  • 3-4 weeks before the planning event, the product management team starts curating the backlogs for the different product and system streams.  They create a “long list” of major features and work items and meet with stakeholders, customers and Bonial management to validate priorities. 
  • A week before planning the “long lists” are reduced to “short lists” of the highest priority items. This is probably the hardest part of the process and it requires saying “no” to things… we find that our stakeholders and customers all agree that discipline is needed so long as it mostly impacts other stakeholders and customers.  Over the years we’ve tried various formal mechanisms to prioritization – Weighted Short Job First, Feature Bucks, etc. – but in the end we find that different tools are needed for different situations and that, with experience, people often people intuitively know the order.
  • Over the next week the product team spends time working through open questions and details while architects and engineers do the same on the technical side.  There’s also generally some intense discussions about “bubble” items – features that are right on the cusp of making the list – as well as hot items that didn’t make the list.

I wish I could say that this process was easy.  The truth is that a great deal changes in three months – new opportunities and challenges, unexpected curveballs – so we’re constantly challenged to re-assess our priorities with each planning cycle.  On top of that there’s a lot we want to do, so we find ourselves often having hard discussions up until the planning day, especially around the “bubble items”.  It’s not clear to me that there’s a much easier way – we’re in a fast industry and a complex business – but we try to get better each quarter.

So the primary inputs to planning are a short, discreet, prioritized set of epic-sized initiatives for each team.  Most of these are functional but there are usually some architectural or operational topics as well.  That brings us now to the actual planning days (typically a Th/F):

  • On planning day 1, we start with a team breakfast at 0900 and then a kickoff presentation at 0930.  The kickoff presentation covers the big picture goals for the quarter and a quick review of each team’s focus and top items so everyone has context.  We also cover logistics – where they can find flip-charts and stickies, who’s in which rooms, etc.
  • Following the kickoff (and the kitchen cleanup), the teams go to their planning spaces and get started.  Basically, they start with the top priority item, plan it through to completion, and then repeat with the next item.  Once they get to the allocated capacity they stop planning.  The remaining items simply don’t get done.

Teams plan with flip charts for each sprint and colored stickies for tasks, milestones, etc.

  • “Full capacity” is an interesting and oft debated question.  We have a loose agreement that teams should reserve ~20% for bugs and team discretion and should reserve another ~20% for refactoring and architecture work. 
  • As the teams are planning they’re also working with other teams on inbound and outbound dependencies.  We’ve organized the teams to minimize dependencies but they’re still a fact of life.  The teams negotiate how to support each other based on overall priorities and goal (ref. the “context” from the breakfast).  Any un-resolved conflicts are escalated or raised at the review meeting (below).
  • At 4PM on the first day the scrum masters and other delivery managers get together to share their current plans with the group.  We use a web-based collaboration tool that allows each team to put virtual stickies on their assigned row with different colors illustrating milestones, spikes, tasks, releases, etc.  Dependencies are made visible by connecting two stickies with a line.  

Teams gather to review the day 1 draft plan.

  • Putting everything together allows us to visualize the major streams, see what made the cut and what didn’t, and address any dependency challenges or conflicts.  Generally there are several to-dos coming out of the review, primarily around working through dependencies or going to business stakeholders for clarification.
  • The morning of day 2 is primarily for making adjustments from the previous day, collaborating with other teams where combined efforts are needed and tying up loose ends.  Most teams wrap this up pretty early and then get back to their HIP sprint, others need most or all of the day.  
  • At 4PM on day two we grab a beer and get back together in front of the stickies board to review any changes from the previous day and discuss any unresolved conflicts.  This exercise typically goes much faster than the day 1 review.  At the end we check confidence and then head home for a much needed break.

Here’s the final plan from last quarter.  

Q2 final plan

It looks complex and it is complex.  Without developing our process, our teams and ourselves over the last couple of years we’d be hard pressed to effectively manage this complexity.

Following the planning we package up the plan and communicate a high level, consumable version for to the business and stakeholders.  We emphasize that these are our current targets and best estimates – this isn’t a contract.  We’ll do everything we can to stick to it but we may be surprised or, in good agile fashion, we may decide to make changes as the situation evolves.

So that brings us full nearly full circle.  I started this series during our last planning days and expected it to be a quick post.  As I pulled the thread, however, I realized how much work had gone into our evolution in this area.  I could also see that a high-level flyover would leave huge gaps in the journey, so I decided to fly lower.   

You can see by now that undertaking a journey like this takes a fair amount of time, experience and honest self-evaluation, regardless of the specific methodology you choose.  That said, the investment is worth it, and a great deal of value can be realized even early in the process.

In Bonial’s case, we had a few advantages as we set off on the journey.  First, everyone was open to change, even when the change made them nervous.  The importance of this can’t be overstated.  I’ve lost count of the organizations I’ve worked with in which the teams had no motivation to improve (though paradoxically most of them complained constantly about the status quo).  In the end the team has to want or at least be willing give it shot.  Which brings us to point two…

Second, we had good people and a healthy culture.  Where we lacked in experience and skills, we more than compensated by having a team of smart, energetic professionals.  With good people, you can generally solve any problem. 

Last, but not least, we have a skilled, SAFe-trained Release Train Manager to drive the process (though her role has evolved).  Even the finest orchestras of the world don’t play on they own- they have a conductor.  In our case the conductor/RTE ensures:

  • The stage is set. Everybody knows the timing, their roles and the rules of the game and All the needed supplies are in place and easily accessible to everybody.
  • Short (really short!) list of candidates for planning is finalized before we start.  The RTE ensure we’re observing Work in Progress (WIP) constraints, which are critical to maximizing throughput.  As she often says, “Let’s stop starting things and start finishing things instead.”  
  • People know who to go to regarding priorities and impediments during planning.
  • The planning is properly wrapped up, all roadmaps and agreements put together, and outcomes are properly communicated to all key stakeholders.
  • Solid retrospectives are done both on the quarter itself as well as the planning process so we can continue improving.

Whew!  That was a lot of writing for me and reading for you.  Kudos if you made it this far – I hope it was worth it.  So now you know how we do it – feel free to share your own stories about how you and your teams plan.  Best of luck in your own journey!

(Special thanks to Irina Zhovtobrukh (the mysterious RTE) for her contributions to this post as well as teaching us how to “conduct” better planning evolutions.)

How we Plan at Bonial (part 2: control)

blue angels - extreme control
Blue Angels – extreme control

As you read in the previous post, we shed some light on what we were (and weren’t) doing with some simple Clarity mechanisms with regards to planning our software development.  Now we needed to make sure everyone knew who should be doing what – a.k.a. Control. 

We started with a new roadmap governance process.  We knew that if we wanted to scale the organization we had to fundamentally rationalize the “roadmap committee”.  To that end we developed the following decision flow chart:

Bonial’s first update to roadmap governance

Though it appears complex, it’s built around a single principle: push as many decisions to the teams as possible.  The “roadmap committee” would be responsible for major strategy and funding decisions and for monitoring progress; the teams would execute under the broad guidance from the committee.  

This shift to distributed control was fundamental to our later growth and success but the truth is that it took the better part of a year until we “got it right-ish”.  It was an iterative process of building trust on all sides – management had to trust the teams to make good decisions, the teams had to trust management to provide clear guidance and hold to it, and the stakeholders had to trust both.  But it was worth it.  

Most importantly, the teams began to “own” their mission which changed everything. 

The Roadmap Committee has long since been replaced with other more focussed and lighter-weight mechanisms, but the principles still hold true – executive management sets the goals, allocates resources and provides experience and mentoring; the teams decide how to achieve the goals and execute.  We continue to explore different organizations and alignments to optimize our software development and delivery and we assume we’ll continue to experiment as we grow and our missions changes.

Another major step we took that impacted both control and clarity was to align our teams into Value Streams.  In our effort to improve how we applied Lean and Agile principles at the team and group levels, we decided to adopt best principles from the Scaled Agile Framework (SAFe) for software development at the enterprise level.  SAFe teams are built around “Programs” or “Value Streams” that allowed teams to focus on a specific portion of the mission and operate as independently as possible.  We deviated quite a bit from pure SAFe and formed three streams around our user facing efforts, our business systems and our operations initiatives.  Never-the-less the benefits were immediate as we reduced “prioritization hell” which is what I call the often fruitless act of trying to compare a revenue generating topic with, for example, a cost savings or security topic. 

Key takeaway: it’s impossible to both scale and maintain central control.  Effective scaling requires creating semi-autonomous, fully-capable teams organized to be relatively independent and provided with the clarity needed to tackle their mission.  This can be a tough step, especially in organizations with a long history of central control, but it’s a step that must be taken.  (FWIW I’ve seen the opposite and it’s not pretty.)

So now we knew what we were doing and who should be doing what.  We were getting a lot closer, but we had one more big step…

How we Plan at Bonial (part 2: clarity)

Clarity

How do you go about fixing something that requires you to change almost everything you do?  As described in part 1, this was the situation we faced at Bonial in late 2014 when it came to the governance and execution of our product development roadmap.  

Rather than re-inventing the wheel, we took advantage of proven play books – one for organization change and one for enterprise agile.  On the organizational side, we knew that the “top down, centralized control” model was already strained and would not scale.  So we leveraged elements from the (fantastic) book “Turn this Ship Around!” by Capt. David Marquett, which describes one organization’s journey from a top-down leadership structure to a “leader-leader” structure with distributed ownership and control.  Bonial would have to undergo a similar transformation – we needed everyone to be engaged and feeling ownership if we were to realize rapid transformation and scale. 

In the book, the author presents a couple dozen excellent leadership mechanisms and groups them under three high-level categories – Clarity, Control, and Competence.  In the interest of brevity, I’ll describe just a few of the things we did to improve in these categories.  (I’ll also break them up over several posts.)

Starting with clarity, we began with the simplest exercise possible: we documented all of the work-in-progress on one list.  Absurdly basic yet profound.  We created the first draft by literally going from team to team and asking them what projects were in progress and putting them in a Google Sheet.  (Why this format?  Because normalizing and adapting the existing tracking tools (Jira, Trello) would have taken far too long and wasted the team’s time and energy.  Also, Google Sheets allow for simultaneous editing which is critical for collaboration.)  To make this relevant for business stakeholders, we then dropped the small “story” and “task” level items and broke down the “saga” level items so that the resulting list was at a meaningful “epic” or “project” level.

Here’s a snap of an archived copy of the first version:

Screenshot of first Bonial Roadmap on Google Sheets

This exercise had several immediate impacts.  First, it showed our stakeholders that the engineering team was actually working on a quite a few projects and began to restore some confidence in the product development function.  Second, it shed light on all the projects and prompted a number of valid and constructive questions as to priorities and business justifications for the projects.  This in turn led directly to our decision to do more formal and intentional planning: we wanted to ensure that our engineering resources were “doing the right things,” not just “doing things right.”

Over time this simple Google Sheet has grown to be the primary tool for viewing and communicating the current quarter’s roadmap development.  We populate the sheet with the output of each quarter’s planning exercise (more on that to follow).  Twice a week we review the status of all items (red, yellow, green) and discuss as a team what we can do to adjust if needed.  The same spreadsheet is publicly available to all stakeholders for full transparency.  We’ve considered several times moving to more sophisticated (and expensive) tools but each time we decided that the Google Sheets does everything we need.

Key takeaway: it’s hard to plan if you don’t know what you’re already doing.  Take the time to get clarity on what’s happening, tune it to the right granularity, and ensure there’s full transparency.

In the next post I’ll talk about how we approach mechanisms for control.