Accelerated Velocity: Situational Awareness

“If a product or system chokes and it’s not being monitored, will anyone notice?”  Unlike the classic thought experiment, this tech version has a clear answer: yes.  Users will notice, customers will notice, and eventually your whole business will notice. 

No-one wants their first sign of trouble to be customer complaints or a downturn in the business, so smart teams invest in developing “situational awareness.” What’s that?  Simple – situation awareness is the result of having access to the tools, data and information needed to understand and act on all of the moving factors relating to the “situation.”  This term is often used in the context of crisis situations or other fast-paced, high-risk endeavors, but it applies to business and network operations as well.

Product development teams most definitely need situational awareness.  The product managers and development leads need to know what their users are doing and how their systems are performing in order to make wise decisions – for example, should the next iteration focus on features, scale or stability.  Sadly, these same product teams often see the tracking and monitoring that is needed for developing situational awareness as “nice-to-have’s” or something to be added when the mythical “someday” arrives. 

The result?  Users having good or bad experiences and no-one knowing either way.  Product strategy decisions being made on individual bias, intuition and incomplete snippets of information.  Not good.

Sun Tzu put it succinctly:

“If you know neither the enemy nor yourself, you will succumb in every battle.”

Situational awareness is a huge topic, so in this series I’m going to limit my focus to data collection (tracking and monitoring) and insights (analytics and visualization) at the product team level.  For the purposes of this series I’ll define ”tracking” as the data and tools that show what users/customers are doing and “monitoring” as the data and tools that focus on systems stability are performance.  Likewise I’ll use “analytics” to refer to tools that facilitate the conversion of data into usable intelligence and “visualization” as the tools for making that intelligence available to the right people at the right time.  I’ll cover monitoring in this article and tracking in a later article.

At Bonial in 2014 there was a feeling that things were fine – the software systems seemed to be reasonably stable and the users appeared happy.  Revenue was strong and the few broad indicators viewed by management seemed healthy.  Why worry?   

From a system stability and product evolution perspective it turns out there was plenty of reason to worry.  While some system-level monitoring was in place, there was little visibility into application performance, product availability or user experience.  Likewise our behavioral tracking was essentially limited to billing events and aggregated results in Google Analytics.  Perhaps most concerning: one of the primary metrics we had for feature success or failure was app store ratings.  Hmmm.

I wasn’t comfortable with this state of affairs.  I decided to start improving situational awareness around system health so I worked with Julius, our head of operations, to lay out a plan of attack.  We already had Icinga running at the system level as well as DataDog and Site24x7 running on a few applications – but they didn’t consistently answer the most fundamental question: “are our users having a good experience?” 

So we took some simple steps like adding new data collectors at critical points in the application stack.  Since full situation awareness requires that the insights be available to the right people at the right time, we also installed large screens around the office that showed a realtime stream of the most important metrics.  And then we looked at them (a surprisingly challenging final step). 

The Bonial NOC Monitor Wall
One of my “go to” overviews of critical APIs, showing two significant problems during the previous day.

The initial results weren’t pretty.  With additional visibility we discovered that the system was experiencing frequent degradations and outages.  In addition, we were regularly killing our own systems by overloading them with massive online marketing campaigns (for which we coined the term: “Self Denial of Service” or SDoS).  Our users were definitely not having the experience we wanted to provide.

(A funny side note: with the advent of monitoring and transparency, people started to ask: “why has the system become so unstable?”)

We had no choice but to respond aggressively.  We set up more effective alerting schemes as well as processes for handling alerts and dealing with outages.  Over time, we essentially set up a network operations center (NOC) with the primary responsibility of monitoring the systems and responding immediately to issues.  Though exhausting for those in the NOC (thank you), it was incredibly effective.  Eventually we transferred responsibility for incident detection and response to the teams (“you build it you run it”) who then carried the torch forward.

Over the better part of the next year we invested enormous effort into triaging the immediate issues and then making design and architecture changes to fix the underlying problems.  This was very expensive as we tapped our best engineers for this mission.  But over time daily issues became weekly became monthly.  Disruptions became less frequent and planning could be done with reasonable confidence as to the availability of engineers.  Monitoring shifted from being an early warning system to a tool for continuous improvement. 

As the year went on the stable system freed up our engineers to work on new capabilities instead of responding to outages.  This in turn became a massive contributor to our accelerated velocity.  Subsequent years were much the same – with continued investment in both awareness and tools for response, we confidently set and measure aggressive SLAs.  Our regular investment in this area massively reduced disruption.  We would never have been able to get as fast as we are today had we not made this investment.

We’ve made a lot of progress in situational awareness around our systems, but we still have a long way to go.  Despite the painful journey we’ve taken, it boggles my mind that some of our teams still push monitoring and tracking down the priority list in favor of “going fast”.  And we still have blind spots in our monitoring and alerting that allow edge-case issues – some very painful – to remain undetected.  But we learn and get better every time.

Some closing thoughts:

  • Ensuring sufficient situational awareness must be your top priority.  Teams can’t fix problems that they don’t know about.
  • Monitoring is not an afterthought.  SLAs and associated monitoring should be a required non-functional requirement (NFR) for every feature and project.
  • Don’t allow pain to persist – if there’s a big problem, invest aggressively in fixing it now.  If you don’t you’ll just compound the problem and demoralize your team.
  • Lead by example.  Know the system better than anyone else on the team.

 

In case you’re interested, here are some of the workhorses of our monitoring suite:

 

Accelerated Velocity: Building Great Teams

Note: this article is part 3 of a series called Accelerated Velocity.  This part can be read stand-alone, but I recommend that you read the earlier parts so as to have the overall context.

People working in teams are at the heart of every company.  Great companies have great people working in high performing teams.  Companies without great people will find it very difficult to get exceptional results. 

The harsh reality is that there aren’t that many great people to go around.  This results in competition for top talent, which is especially true in tech.  Companies and organizations use diverse strategies in addressing this challenge.  Some use their considerable resources (e.g. cash) to buy top talent though with dubious results – think big corporations and Wall Street banks.  Some create environments that are very attractive to the type of people they’re looking for – think Google and Amazon.  Some purposely start with inexperienced but promising people and develop their own talent – a strategy used by the big consulting companies.  Many drop out of the race altogether and settle for average or worse (and then hire the consulting companies to try to solve their challenges with processes and technology – which is great for the consulting companies).

But attracting talent is only half the battle.  Companies that succeed in hiring solid performers then have to ensure their people are in a position to perform, and this brings us to their teams.  Teams have a massive amplifying affect on the quantity and quality of each individual’s output.  My gut tells me that the same person working on two different teams may be 2-3X as productive depending on the quality of the team. 

So no matter how good a company is at attracting top talent, it then needs to ensure that the talent operates in healthy teams. 

What is a healthy team?  From my experience it looks something like this:

  • Competent, motivated people who are…
  • Equipped to succeed and operate with…
  • High integrity and professionalism…
  • Aligned behind a mission / vision

That doesn’t seem too hard.  So why aren’t healthy teams the norm?  Simple: because they’re fragile.  If any of the above pieces are missing, the integrity of the team is at risk.  Throw in tolerance for low performers, arrogant assholes, and whiners, mix in some disrespect and fear, and the team is broken.

(Note that the negatives influences outweigh the positives – as the proverb says: “One bad apple spoils the whole bushel.”  If you play sports you know this phenomenon well – a team full of solid players can easily be undone by a single weak link that disrupts the integrity of the team.)

This leads me to a few basic rules I follow when developing teams:

  1. Provide solid leadership
  2. Recruit selectively
  3. Invest in growth and development
  4. Break down barriers to getting and keeping good people
  5. Aggressively address low-performance and disruption

Bonial had a young team with a wide range of skill and experience in 2014.  Fortunately many of the team members had a bounty of raw talent and were motivated (or desired to be motivated).  Unfortunately there were also quite a few under-performers as well as some highly negative and disruptive personalities in the mix.  The combination of inexperience, underperformance and disruption had an amplifying downward effect on the teams.

To build confidence and start accelerating performance we needed to turn this situation around.  We started by counseling and, if behavior didn’t change, letting go the most egregiously low performers and disruptive people – not an easy thing to do and somewhat frowned upon in both the company and in German culture.   But the cost of keeping them on the team, thereby neutralizing and demoralizing the high performers, was far higher than the pain and cost of letting them go. 

(A quick side note: there were concerns among the management that letting low-performers go would demoralize the rest of the team.  Not surprisingly, quite the opposite happened – the teams were relieved to have the burdens lifted and were encouraged to know that their leads were committed to building high performing teams.)

We started doing a better job of mentoring people and setting clear performance goals.  Many thrived with guidance and coaching; some didn’t and we often mutually decided to part ways.  Over time the culture changed to where low performance and negativity were no longer tolerated.

At the same time we invested heavily in recruiting.  We hired dedicated internal recruiters specifically focussed on tech recruits.  We overhauled our recruiting and interview process to better screen for the talent, mentality and personality we needed.  We added rigor to our senior hiring practices, focussing more on assessing what the person can do vs what they say they can do.  And we added structure to the six month “probation” period, placing and enforcing gates throughout the process to ensure we’d hired the right people.  Finally, we learned the hard way that settling for mediocre candidates was not the path to success; it was far better to leave a position unfilled than to fill it with the wrong person.

How did we attract great candidates?  We focussed on our strengths and on attracting people who valued those attributes: opportunities for growth, freedom to make a substantial impact, competent team-mates, camaraderie, a culture of respect, and exposure to cutting-edge technologies.  Why these?  Because year over year, though employee satisfaction survey and direct feedback, we find these elements correlate very strongly with employee satisfaction, even more so than compensation and other benefits.  In short, we’ve worked hard to create an environment where our team-mates are excited to come to work every day.

(This is not to say we ignored competitive compensation; as I’ll describe in a later post, we also worked to ensure we paid a fair market salary and then provided a path for increasing compensation over time with experience.)

Over time, as our people became more experienced, our processes matured and our technology set became more advanced, Bonial became a great place for tech professionals to sharpen their skills and hone their craft.  New team members brought fresh ideas and at the same time had an opportunity to learn both by what we already had as well as what they helped create.  The result is what we have today: a team of teams full of capable professionals who are together performing at a level many times higher than in 2014

Some closing thoughts:

  • You’re only as good as the people on the teams.
  • Nurture and grow talented people. Help under-performers to perform. Let people go when necessary.
  • Get really good at recruiting.  Focus on what the candidate will do for you vs what they claim to have done in the past.
  • Don’t fall into the trap of believing process and tools are a substitute for good people.

Footnote: If you haven’t yet, I suggest your read about Google’s insightful research on team performance and how  ”psychological safety” is critical to developing high performing teams. 

Accelerated Velocity: Building Leaders

Note: this article is part 2 of a series called Accelerated Velocity.  This part can be read stand-alone, but I recommend that you read the earlier parts so as to have the overall context.

Positive changes require a guiding hand.  Sometimes this arises organically from a group of like-minded people, but far more often there’s a motivated individual driving the change.  In short – a leader.

Here’s the rub: the tech industry is notoriously deficient in developing leaders. Too often the first step in a leader’s journey starts when their manager leaves and they’re blessed with a dubious promotion… “Congratulations, you’rein charge now.  Good luck.”  If they’re fortunate their new boss is an experienced leader and has time to mentor them.  In a larger organization they may have access to some bland corporate training on how to host a meeting or write a project plan.  But the vast majority of people thrust into leadership and management roles in tech are largely left to their own devices to succeed.

Let me pause for a moment and highlight a subtle but important point: leadership and management are different skills.  Leadership is creating a vision and inspiring a group of people to go after the vision; management is organizing, equipping, and caring for people as well as taking care of the myriad details needed for the group to be successful.  There’s some overlap, and an effective leader or manager has competence in both areas, but they require different tools and a different mindset. This article is focussed on the leadership component.

So what does it take to develop competent and confident leaders?  When I look at some of the best-in-class “leadership-centric organizations” – militaries and large consulting companies for example – I see the following common elements:

  1. Heavy up-front investment in training
  2. High expectations of the leaders
  3. An reasonably structured environment in which to learn and grow
  4. A continuous cycle in which role models will coach and mentor the next generation

How did this look at Bonial?

Upon arriving I inherited a 40-ish strong engineering organization broken up into five teams, each headed by a “team lead.” The problem was that these team leads had no clear mandate or role, little or no leadership and management training, and essentially no power to carry out a mandate even if they’d had one.

This setup was intended to keep the organization flat and centralize the administrative burden of managing people so as to allow the team leads to focus on delivery. Unfortunately this put the leads in a largely figurehead role – they represented their teams and were somehow responsible for performance but had few tools to employ and little experience with which to effectively deploy them. They didn’t hire their people, administer compensation or manage any budgets. In fact, they couldn’t even approve vacations.  To this day it’s not clear to me, or them, what authority or responsibility they had.

This arrangement also created a massive chokepoint at the center of the organization – no major decisions were made without approval from “above”. The results were demoralized leads and frustrated teams.

Changing this dynamic was my first priority.  To scale our organization we’d need to operate as a federation of semi-autonomous teams, not as a traditional hierarchical organization.  For this we needed leads who could drive the changes we’d make over the coming years, but this would require a major shift in mindset.  After all, if I couldn’t trust them to approve vacations, why should I trust them with driving ROI from the millions of euros we’d be investing in their teams?  Engineers have the potential to produce incredibly valuable solutions; ensuring they have solid leadership is the first and most important responsibility of senior management.

We started with establishing a clear scope of responsibility and building our toolbox of skills.  I asked the leads if they were willing to “own” the team results and, though a little nervous, most were willing.  This meant they would now make the calls as to who was on the team and how those people were managed. They took over recruiting and compensation administration. They played a much stronger role in ensuring the teams had clarity on their mission and how the teams executed the mission. They received budgets for training, team events and discretionary purchases. And, yes, they even took responsibility for approving vacations.

We agreed to align around the leader-leader model espoused by David Marquet (https://www.davidmarquet.com/) in his book “Turn the Ship Around!  We read the book together and discussed the principles and how to apply them in daily practice.  The phrase “I intend to…” was baked into our vocabulary and mentality.  We eliminated top-down systems and learned to specify goals, not methods.  We focussed on achieving excellence, not just avoiding errors.  The list goes on.

I also started a “leadership roundtable” – 30 minutes each week where we’d meet in a small group and discuss experiences and best practices around core leadership and management topics: motivating people and teams, being effective, basic psychology, communicating, coaching and mentoring, discipline, recruiting, personal organizational skills, etc.  Over time, dozens of people – ranging from prospective team leads to product managers to people simply interested in leadership and management – participated in the roundtables, giving us a common foundation from which to work.

As I’ll share in a future article, we also created a career growth model that fully supported a management track as well as a technical track and, most importantly, the possibility to move back and forth freely between the two.  We encouraged people to give management a try and offered mentoring and support plus the risk-free option of being able to switch back to their former role if they preferred.  In the early days this was a tough sell – “team lead” had the reputation of being mostly pain with little upside.  Never-the-less a few brave souls gave it a shot and, to their surprise, found it rewarding (and have since grown into fantastic leads).

It wasn’t easy – we had a fair share of mistakes, failures and redos – but the positive effects were felt almost immediately. Over time this first generation of leads grew their teams and created cultures of continuous improvement. As the teams grew, the original leads mentored new leaders to take over the new teams and the cycle continued. As it stands today we have a dozen or so teams/squads led by capable leaders that started as software engineers, quality assurance pros, system engineers, etc. 

For what it’s worth, I believe the number one factor driving Bonial’s accelerated velocity was growth in leadership maturity. If you’re looking to engineer positive change, start here.

Some closing thoughts:

  • Positive change requires strong leadership.
  • A single leader can start the change process, but large-scale and enduring change requires distributed leadership (e.g. ”leader-leader”).
  • Formal training can be a great source of leadership and management tools, but mastering those tools requires time, a safe and constructive environment and active coaching and mentoring.
  • Growing a leadership team is not a linear or a smooth process.  The person driving and guiding the development must commit to the long game and must be willing to accept accountability for inconsistent results from first generation leads as they learn their trade.

Read part 3: Building Great Teams

Accelerated Velocity: How Bonial Got Really Fast at Building Software

My boss, Max (Bonial Group’s CEO), and I sat down recently for a “year-in-review” during which we discussed the ups and downs of 2017 as well as goals for the new year.  In wrapping up the conversation, I shared with him my gut feeling that velocity and productivity had improved over the past couple of years and were higher than they’d ever been at Bonial – perhaps as much as double when compared to 2014.  

He asked if I could quantify the change, so on a frigid Sunday a couple of weeks ago I sat down with a mug of hot tea and our development records to see what I could do. We’ve used the same “product roadmap” format since 1Q14 (described here), which meant I could use a “points” type approach to quantify business value delivered during each quarter.  As I was looking for relative change over time and I was consistent in the application, I felt this was a decent proxy for velocity.  

It took me a couple of hours but was well worth the effort.  Once I’d finished scoring and tabulating, I was pleasantly surprised to find that I’d significantly underestimated the improvements we’d made.  Here’s a high level overview of the results:

7X Velocity! Bonial team size, value delivered and productivity over time.

The net-net is that in 1Q 2018 we’ll be delivering ~630% more business value than we delivered in the first quarter of 2014, largely driven by the fact that each person on the team is ~250% more productive.  

Sweet.

The obvious next question: how did we do this?

The short answer is that there is no short answer.  There was no single magic button that we pushed to set us on this path to accelerated velocity; this was a long campaign that started small and grew, eventually spanning people, process, technology and culture.  Over time these learnings, improvements, changes and experiments – some large, some small, some successful, some not – built on each other and eventually created an environment in which the momentum sustained itself.  

Over the next few weeks I’ll summarize the major themes here in this blog for both myself as well as anyone who’s interested.  Along this journey I plan to cover (and will link when available):

  1. Building Leaders
  2. Building Great Teams
  3. Creating Situational Awareness
  4. Providing a Growth Path
  5. Clarifying Processes and Key Roles
  6. Enabling Independent Action
  7. Creating an Architecture Runway
  8. Optimizing the SDLC with DevOps
  9. Getting Uncomfortable
  10. Doing the Right Things
  11. Taking Ownership
  12. Building on What You’ve Got

Each of those topics could alone make for a small book, but I’ll try to keep the articles short and informative by focussing only on the most important elements.  If there’s an area in which you’d like me to dig deeper, let me know and I’ll see what I can do.  Assuming I get through all of those topics I’ll wrap things up with some final thoughts.

So let’s get started with part 2: Building Leaders

How we Plan at Bonial (part 3)

Collaborative digital stickies board that we use for planning.

Ok, after all that, how do we actually plan at Bonial?

The heart of our planning activities is the Quarterly Planning which is loosely modeled on Program Implement (PI) Planning from SAFe.  During quarterly planning / PI planning, everyone in the product development organization – developers, designers, architects, testers, product managers, operations specialists, designers, etc. – get together for a couple of days to map out their next phase.  We do our planning during the previous quarter’s HIP (Hardening Innovation and Planning) sprint, which is sprint 6 of each quarter.

Before I dive into the actual planning days, I should point out that the preparations start several weeks before when the product teams actively work with stakeholders, customer facing teams and the executive team to validates the backlogs against the current company priorities and business realities.  The prep phase looks something like this:

  • The senior management team and product strategy board review the overall strategy and primary business goals to assess if any change in focus is needed.  
  • Next we make sure that product and delivery management has the same level of clarity. We get the delivery leads and product owners together and communicate the company goals for the upcoming quarter to them, taking the time to answer questions about strategy, challenges, current market trends etc. Our goal here is to make sure that all our leaders are able to bring clarity to their teams so that local decisions are made with the right context.
  • 3-4 weeks before the planning event, the product management team starts curating the backlogs for the different product and system streams.  They create a “long list” of major features and work items and meet with stakeholders, customers and Bonial management to validate priorities. 
  • A week before planning the “long lists” are reduced to “short lists” of the highest priority items. This is probably the hardest part of the process and it requires saying “no” to things… we find that our stakeholders and customers all agree that discipline is needed so long as it mostly impacts other stakeholders and customers.  Over the years we’ve tried various formal mechanisms to prioritization – Weighted Short Job First, Feature Bucks, etc. – but in the end we find that different tools are needed for different situations and that, with experience, people often people intuitively know the order.
  • Over the next week the product team spends time working through open questions and details while architects and engineers do the same on the technical side.  There’s also generally some intense discussions about “bubble” items – features that are right on the cusp of making the list – as well as hot items that didn’t make the list.

I wish I could say that this process was easy.  The truth is that a great deal changes in three months – new opportunities and challenges, unexpected curveballs – so we’re constantly challenged to re-assess our priorities with each planning cycle.  On top of that there’s a lot we want to do, so we find ourselves often having hard discussions up until the planning day, especially around the “bubble items”.  It’s not clear to me that there’s a much easier way – we’re in a fast industry and a complex business – but we try to get better each quarter.

So the primary inputs to planning are a short, discreet, prioritized set of epic-sized initiatives for each team.  Most of these are functional but there are usually some architectural or operational topics as well.  That brings us now to the actual planning days (typically a Th/F):

  • On planning day 1, we start with a team breakfast at 0900 and then a kickoff presentation at 0930.  The kickoff presentation covers the big picture goals for the quarter and a quick review of each team’s focus and top items so everyone has context.  We also cover logistics – where they can find flip-charts and stickies, who’s in which rooms, etc.
  • Following the kickoff (and the kitchen cleanup), the teams go to their planning spaces and get started.  Basically, they start with the top priority item, plan it through to completion, and then repeat with the next item.  Once they get to the allocated capacity they stop planning.  The remaining items simply don’t get done.
Teams plan with flip charts for each sprint and colored stickies for tasks, milestones, etc.
  • “Full capacity” is an interesting and oft debated question.  We have a loose agreement that teams should reserve ~20% for bugs and team discretion and should reserve another ~20% for refactoring and architecture work. 
  • As the teams are planning they’re also working with other teams on inbound and outbound dependencies.  We’ve organized the teams to minimize dependencies but they’re still a fact of life.  The teams negotiate how to support each other based on overall priorities and goal (ref. the “context” from the breakfast).  Any un-resolved conflicts are escalated or raised at the review meeting (below).
  • At 4PM on the first day the scrum masters and other delivery managers get together to share their current plans with the group.  We use a web-based collaboration tool that allows each team to put virtual stickies on their assigned row with different colors illustrating milestones, spikes, tasks, releases, etc.  Dependencies are made visible by connecting two stickies with a line.  
Teams gather to review the day 1 draft plan.
  • Putting everything together allows us to visualize the major streams, see what made the cut and what didn’t, and address any dependency challenges or conflicts.  Generally there are several to-dos coming out of the review, primarily around working through dependencies or going to business stakeholders for clarification.
  • The morning of day 2 is primarily for making adjustments from the previous day, collaborating with other teams where combined efforts are needed and tying up loose ends.  Most teams wrap this up pretty early and then get back to their HIP sprint, others need most or all of the day.  
  • At 4PM on day two we grab a beer and get back together in front of the stickies board to review any changes from the previous day and discuss any unresolved conflicts.  This exercise typically goes much faster than the day 1 review.  At the end we check confidence and then head home for a much needed break.

Here’s the final plan from last quarter.  

Q2 final plan

It looks complex and it is complex.  Without developing our process, our teams and ourselves over the last couple of years we’d be hard pressed to effectively manage this complexity.

Following the planning we package up the plan and communicate a high level, consumable version for to the business and stakeholders.  We emphasize that these are our current targets and best estimates – this isn’t a contract.  We’ll do everything we can to stick to it but we may be surprised or, in good agile fashion, we may decide to make changes as the situation evolves.

So that brings us full nearly full circle.  I started this series during our last planning days and expected it to be a quick post.  As I pulled the thread, however, I realized how much work had gone into our evolution in this area.  I could also see that a high-level flyover would leave huge gaps in the journey, so I decided to fly lower.   

You can see by now that undertaking a journey like this takes a fair amount of time, experience and honest self-evaluation, regardless of the specific methodology you choose.  That said, the investment is worth it, and a great deal of value can be realized even early in the process.

In Bonial’s case, we had a few advantages as we set off on the journey.  First, everyone was open to change, even when the change made them nervous.  The importance of this can’t be overstated.  I’ve lost count of the organizations I’ve worked with in which the teams had no motivation to improve (though paradoxically most of them complained constantly about the status quo).  In the end the team has to want or at least be willing give it shot.  Which brings us to point two…

Second, we had good people and a healthy culture.  Where we lacked in experience and skills, we more than compensated by having a team of smart, energetic professionals.  With good people, you can generally solve any problem. 

Last, but not least, we have a skilled, SAFe-trained Release Train Manager to drive the process (though her role has evolved).  Even the finest orchestras of the world don’t play on they own- they have a conductor.  In our case the conductor/RTE ensures:

  • The stage is set. Everybody knows the timing, their roles and the rules of the game and All the needed supplies are in place and easily accessible to everybody.
  • Short (really short!) list of candidates for planning is finalized before we start.  The RTE ensure we’re observing Work in Progress (WIP) constraints, which are critical to maximizing throughput.  As she often says, “Let’s stop starting things and start finishing things instead.”  
  • People know who to go to regarding priorities and impediments during planning.
  • The planning is properly wrapped up, all roadmaps and agreements put together, and outcomes are properly communicated to all key stakeholders.
  • Solid retrospectives are done both on the quarter itself as well as the planning process so we can continue improving.

Whew!  That was a lot of writing for me and reading for you.  Kudos if you made it this far – I hope it was worth it.  So now you know how we do it – feel free to share your own stories about how you and your teams plan.  Best of luck in your own journey!

(Special thanks to Irina Zhovtobrukh (the mysterious RTE) for her contributions to this post as well as teaching us how to “conduct” better planning evolutions.)

How we Plan at Bonial (part 2: competence)

In the previous two posts I talked about the importance of clarity and control, but even perfect clarity and unlimited control will likely still lead to failure and frustration if the team isn’t ready to take on these new responsibilities. That’s where Competence comes in.

To build competence across the team we invested in experienced practitioners as well as training and mentoring. We hired a talented SAFe-trained development manager (“Release Train Engineer” in SAFe parlance) to both lead our transformation as well as provide training and mentoring.  We brought in agile and SAFe trainers for multi-day training sessions on team and enterprise agile (more on SAFe in later posts).  We started leadership and management training for our product owners, new team leads and lead developers. The more experienced members of the team actively coached others in best practices.

Why go through all this trouble?  Simple – a common source of failure I’ve seen over the years is this: the fantasy that calling something ‘agile’ somehow makes it agile.  Too often I’ve seen organizations slap on the label of “scrum teams,” appoint a newly hired Scrum Master or Agile Coach, tell them to have stand-ups and sprints, and then hope that “agile happens”… a.k.a. “fake it until you make it”.  Good luck.  Like it or not, you have to invest in training, excellent people and experienced leadership.

A word of advice: don’t skimp on the training. Our first training session involved a half-day session for only key leaders. As we quickly learned, that’s not training – that’s just a teaser.  Frankly I was part of the problem – I needed to shift my attitude and accept that, unless the whole team is on-board and up-to-speed, we’d never be able to run a full speed.  Yes, it was expensive in both time and money, but necessary.  We’ve since opened up both the breadth and depth of the training.

We also learned by doing. We built on a strong culture of open and honest retrospectives and we actively shared the learnings between teams. We experimented with new techniques and, when they worked, spread them throughout the organization. We actively cultured an environment of “low fear” so that people had space to learn and grow.

As a management team, we also worked hard to “specify goals, not methods” as part of the shift away from the Roadmap Committee described in the previous post. Why is this a competence topic? Because by forcing ourselves to stay out of the details we provided space for the teams to learn and grow. This also opened up room for lots of great ideas that may never have been voiced in a top-down approach.

Key takeaway: invest in training and regular, iterative experiential learning. Put your teams in positions where they need to stretch their knowledge and experience so that they have the context and confidence going forward to execute the mission (but actively support them as they learn).  And, as always, hire and retain great people.

One thing before we get back to the original topic – as I re-read these last three posts I can see how a reader might get the sense that we executed smoothly via a carefully orchestrated plan.  Not so.  There was trial-and-error, plenty of course adjustments and a mix of successes and failures.  That’s ok – it takes time.  What’s important is keeping your eye on the ultimate goal, being realistic and working together as a team to make it happen.

Ok, after a long detour through the background, back to the original topic…

How we Plan at Bonial (part 2: control)

blue angels - extreme control
Blue Angels – extreme control

As you read in the previous post, we shed some light on what we were (and weren’t) doing with some simple Clarity mechanisms with regards to planning our software development.  Now we needed to make sure everyone knew who should be doing what – a.k.a. Control. 

We started with a new roadmap governance process.  We knew that if we wanted to scale the organization we had to fundamentally rationalize the “roadmap committee”.  To that end we developed the following decision flow chart:

Bonial’s first update to roadmap governance

Though it appears complex, it’s built around a single principle: push as many decisions to the teams as possible.  The “roadmap committee” would be responsible for major strategy and funding decisions and for monitoring progress; the teams would execute under the broad guidance from the committee.  

This shift to distributed control was fundamental to our later growth and success but the truth is that it took the better part of a year until we “got it right-ish”.  It was an iterative process of building trust on all sides – management had to trust the teams to make good decisions, the teams had to trust management to provide clear guidance and hold to it, and the stakeholders had to trust both.  But it was worth it.  

Most importantly, the teams began to “own” their mission which changed everything. 

The Roadmap Committee has long since been replaced with other more focussed and lighter-weight mechanisms, but the principles still hold true – executive management sets the goals, allocates resources and provides experience and mentoring; the teams decide how to achieve the goals and execute.  We continue to explore different organizations and alignments to optimize our software development and delivery and we assume we’ll continue to experiment as we grow and our missions changes.

Another major step we took that impacted both control and clarity was to align our teams into Value Streams.  In our effort to improve how we applied Lean and Agile principles at the team and group levels, we decided to adopt best principles from the Scaled Agile Framework (SAFe) for software development at the enterprise level.  SAFe teams are built around “Programs” or “Value Streams” that allowed teams to focus on a specific portion of the mission and operate as independently as possible.  We deviated quite a bit from pure SAFe and formed three streams around our user facing efforts, our business systems and our operations initiatives.  Never-the-less the benefits were immediate as we reduced “prioritization hell” which is what I call the often fruitless act of trying to compare a revenue generating topic with, for example, a cost savings or security topic. 

Key takeaway: it’s impossible to both scale and maintain central control.  Effective scaling requires creating semi-autonomous, fully-capable teams organized to be relatively independent and provided with the clarity needed to tackle their mission.  This can be a tough step, especially in organizations with a long history of central control, but it’s a step that must be taken.  (FWIW I’ve seen the opposite and it’s not pretty.)

So now we knew what we were doing and who should be doing what.  We were getting a lot closer, but we had one more big step…

How we Plan at Bonial (part 2: clarity)

Clarity

How do you go about fixing something that requires you to change almost everything you do?  As described in part 1, this was the situation we faced at Bonial in late 2014 when it came to the governance and execution of our product development roadmap.  

Rather than re-inventing the wheel, we took advantage of proven play books – one for organization change and one for enterprise agile.  On the organizational side, we knew that the “top down, centralized control” model was already strained and would not scale.  So we leveraged elements from the (fantastic) book “Turn this Ship Around!” by Capt. David Marquett, which describes one organization’s journey from a top-down leadership structure to a “leader-leader” structure with distributed ownership and control.  Bonial would have to undergo a similar transformation – we needed everyone to be engaged and feeling ownership if we were to realize rapid transformation and scale. 

In the book, the author presents a couple dozen excellent leadership mechanisms and groups them under three high-level categories – Clarity, Control, and Competence.  In the interest of brevity, I’ll describe just a few of the things we did to improve in these categories.  (I’ll also break them up over several posts.)

Starting with clarity, we began with the simplest exercise possible: we documented all of the work-in-progress on one list.  Absurdly basic yet profound.  We created the first draft by literally going from team to team and asking them what projects were in progress and putting them in a Google Sheet.  (Why this format?  Because normalizing and adapting the existing tracking tools (Jira, Trello) would have taken far too long and wasted the team’s time and energy.  Also, Google Sheets allow for simultaneous editing which is critical for collaboration.)  To make this relevant for business stakeholders, we then dropped the small “story” and “task” level items and broke down the “saga” level items so that the resulting list was at a meaningful “epic” or “project” level.

Here’s a snap of an archived copy of the first version:

Screenshot of first Bonial Roadmap on Google Sheets

This exercise had several immediate impacts.  First, it showed our stakeholders that the engineering team was actually working on a quite a few projects and began to restore some confidence in the product development function.  Second, it shed light on all the projects and prompted a number of valid and constructive questions as to priorities and business justifications for the projects.  This in turn led directly to our decision to do more formal and intentional planning: we wanted to ensure that our engineering resources were “doing the right things,” not just “doing things right.”

Over time this simple Google Sheet has grown to be the primary tool for viewing and communicating the current quarter’s roadmap development.  We populate the sheet with the output of each quarter’s planning exercise (more on that to follow).  Twice a week we review the status of all items (red, yellow, green) and discuss as a team what we can do to adjust if needed.  The same spreadsheet is publicly available to all stakeholders for full transparency.  We’ve considered several times moving to more sophisticated (and expensive) tools but each time we decided that the Google Sheets does everything we need.

Key takeaway: it’s hard to plan if you don’t know what you’re already doing.  Take the time to get clarity on what’s happening, tune it to the right granularity, and ensure there’s full transparency.

In the next post I’ll talk about how we approach mechanisms for control. 

How We Plan at Bonial (Part 1: the early days)

Today is the first day of our quarterly planning ritual here at Bonial.  As I write this the teams are huddled away passionately discussing, digesting, challenging, and estimating their candidate work items.  We have over a hundred people from 25 different countries and multiple offices working through dozens of epics.  By tomorrow we’ll have a solid plan agreed upon by the engineers, designers, testers, data scientists, operations specialists and product managers as well as their stakeholders.  

It wasn’t always like this.

When I arrived at Bonial a couple of years ago, there was no documented roadmap or cohesive prioritization process.  The planning horizon ranged from intra-day for emergencies to a couple of weeks for most other items.  No-one had a clear understanding of what we were working on and why.  The stakeholders didn’t trust engineering and everyone was unhappy.

Getting from there to here hasn’t been easy.  Over the next few posts I’ll walk you through how we got to where we are today.  

But to understand the journey we have to start at the beginning-ish…

In 2014, Bonial was a mature startup with seven or eight years under its belt.  We had a very successful mobile and web app being used in a dozen or so countries.  The product development crew was organized into four scrum teams, an ops team and a design team and was responsible for developing all of the user facing systems as well as the critical business systems.  All-in-all, there were 40-50 people working together in product development.

Unfortunately the team was less effective than it could and should have been, in large part due to lack of clarity and governance.  For starters, not only was there a lack of a coherent roadmap, there wasn’t even any clear record of what work was currently being executed.  We had tickets in Jira scattered across a dozen or more “projects,” Trello boards, stickies on blackboards, and whole lot of ideas in people’s heads, but there was no one place a stakeholder could go and get a simple answer to the question: “what is the status of my project?”

What roadmap planning was done happened in a bi-weekly session called the “roadmap committee.”  This was a group of senior managers from the extended product development organization and stakeholders who reviewed development progress and made decisions on new initiatives.  I’m being nice when I say that it wasn’t much fun.  The selection of initiatives being governed was somewhat arbitrary and the value provided by the committee was questionable.  We often hashed over the same questions over and over again.  Unfortunately it was the only vehicle in place to provide some level of two-way communications regarding roadmap and status.

The end result was that no-one was happy.  The stakeholders and customers felt like their needs were ignored and that, when their projects were accepted, delivery was too slow.  The engineers felt like they were in a blender of arbitrary and incoherent requirements over which they felt no sense of ownership.  And the product management team was stuck in the middle, working to adjust to the latest change and managing both unhappy stakeholders and engineers both.  The end result was perceived and real low performance and sense that we were set up to fail.

So we decided to change this; the solution would require a great deal of work in many areas across the people/process/technology spectrums.  It all came together, though, in planning.  Stay tuned for part 2. 

Getting Extreme

 

In my previous post on Extreme Ownership I shared that I wished more technology companies would take the principles more seriously.  Over the last month my wish was granted right here at my company.  

Our executive team had an offsite strategy meeting last week, and one of the coolest things we did was take a deep dive into Extreme Ownership.  In the weeks leading up to the summit each member of the team – managing directors, senior execs and CxOs – read Extreme Ownership and prepared homework consisting of an introspective look into how they’d individually violated or been challenged by the principles as well as which principles we wanted to focus on bringing more into the company.

We discussed our experiences over dinner in a very candid fashion.  Each person shared one or two “fails” that tracked back to the principles, or challenges that could better have been solved by better applying the principles.  It’s not often that very skilled and accomplished senior executives are willing to admit to failures in front of their peers, so I think that says a lot about the character of those around the table as well as their commitment to Extreme Ownership.

Some of the maxims that resonated strongly and were repeatedly mentioned:

  • “There are no bad boats, only bad leaders” – the core idea here is that you have to look first at the immediate leader before blaming the team itself for underperformance.
  • “It’s not what you preach, it’s what you tolerate” – how true.  How brutal true.
  • “Check the ego” – as the authors note, “egos cloud and disrupt everything.”  If you don’t have the discipline to keep your ego in check you don’t deserve the trust and confidence of the people you lead.
  • “They don’t want me to fail” – how many times do we assume that a boss or outside organization is to purposely make our lives harder when they put an obstacle in our way?  Probably quite a bit.  And how often is that true?  Likely very seldom.  If we’d drop the assumption of hostile intent and the resulting “us vs them” attitude, business and life would be a lot easier.

One of the longer and more challenging discussions was around how to move to “Decentralized Command” – let’s face it, it’s not easy to step back and let others take charge of executing a mission that you’re accountable for.  But it must be done to scale the organization and to develop the next generation of leaders.  And guess what – sometime they will fail, and you’ll still own the result.  Our COO made a key point here – while failure in the SEALs often results in injury or death, a business fail will have much, much lighter consequences, so we need to take an objective look at the real risk and balance with the cost of not decentralizing. 

As a team we decided on three of the principles we’d like to focus on for the entire organization and each of us was assigned a buddy from within the group to challenge us grow in these areas.  

I was really energized by this process and I’d recommend it to any team that wants to move in this direction.  In hindsight, I recognize that our company already has a pretty solid accountable culture and a general lack of fear, which probably made this a lot easier; some teams will have to overcome much bigger culture and ego challenges.  Which, in the end, means it’s even more vital.