Revamping SAFe's Program Level PI Metrics Part 2/6: Business Impact

“Managers shape networks’ behavior by emphasizing indicators that they believe will ultimately lead to long term profitability” – Philip Anderson, Seven Levers for Guiding the Evolving Enterprise

Introduction

In Part 1 of this series, we introduced the Agile Release Train (ART) PI metrics dashboard and gave an overview of the 4 quadrants of measurement. This post explores the first and arguably most important quadrant – Business Impact.

As you may have guessed from the rather short list, there can be no useful generic set of metrics. They must be context driven for each ART based on both the mission of the ART and the organisational strategic themes that gave birth to it. As Mark Schwartz put it so elegantly in The Art of Business Value, “Business value is a hypothesis held by the organization’s leadership as to what will best accomplish the organization’s ultimate goals or desired outcomes”.

Definitions

Rationale

Fitness Function

When reading The Everything Store: Jeff Bezos and the Age of Amazon, I was particularly taken by the concept of the fitness function. Each team was required to propose ".. A linear equation that it could use to measure its own impact without ambiguity. … A group writing software code for the fulfillment centers might home in on decreasing the cost of shipping each type of product and reducing the time that elapsed between a customer's making a purchase and the item leaving the FC in a truck". Amazon has since moved to more discrete measures rather than equations (I suspect in large part due to the bottlenecks caused by Bezos' insistence on personally signing off each team's fitness function equation), but I believe the “fitness function mindset” has great merit in identifying the set of business impact metrics which best measure the performance of an ART.

To illustrate based on three ART's I work with:

• An ART at an organisation which ships parcels uses "First time delivery %". They implement numerous digital features enabling pre-communication with customers to avoid delivery vans arriving at empty houses. Moving this a percentage point has easily quantifiable bottom-line ROI impacts.
• An ART focused on Payment Assurance at an organisation which leverages "Delivery Partners" to execute field installation and service work. Claims for payment submitted by these partners are complex and require payment within tight SLA's. A fitness function based on Payment lead time and cost savings based on successful claim disputes would again easily be mapped to quantifiable ROI.
• A telco ART focused on self-service diagnostics for customers. The fitness function in this case would reference “reduced quantity of fault-related calls to call centers” (due to the customer having self-diagnosed and used the tool to make their own service booking if required), “reduced quantity of no-fault-found truck rolls” (due to the tool having aided the customer in identifying ‘user error’), “increased first call resolution rates for truck rolls” (due to the detailed diagnostic information available to service technicians).

Considerations when selecting fitness function components
Obviously, the foremost consideration is identifying a number of components from which one can model a monetary impact. However, I believe two other factors should be considered in the identification process:

• Impact on the Customer
• Ensuring a mix of both Leading and Lagging Measures

Net Promoter Score (NPS) is rapidly becoming the default customer loyalty metric, and whilst Reichheld argues in The Ultimate Question 2.0 that mature NPS implementations gain the ability to quantify the value of a movement in a specific NPS demographic I have yet to actually work with an organization that has reached this maturity. However, most have access to reasonably granular NPS metrics. The trick is identifying the NPS segments impacted by the customer interactions associated with the ART’s mission and incorporating those measures.

When it comes to identifying useful leading metrics, there can be no better inspiration than the Innovation Accounting concepts explained by Eric Ries in The Lean Startup. In some cases (particularly Digital), it can also be as simple as taking the popular Pirate Metrics for inspiration. For many trains with digital products, I also believe abandonment rate is an extremely valuable metric given that an abandoned transaction tends to equate to either a lost sale or added load on a call center.

Program Predictability

This is the standard proxy result measure defined in SAFe. It is a great way of ensuring focus on objectives whilst leaving space for Product Owners and Managers to make trade-off calls during PI execution. In short, I paraphrase it as "a measure of how closely the outcomes from PI execution live up to the expectations established with Business Owners at PI planning and how clear those expectations were".

But wait, there's more!

A good train will use far more granular results metrics than those listed above. Each feature should come with specific success measures that teams, product owners and managers should be using to steer their strategy and tactics (fuel for another post), but I am seeking here a PI level snapshot that can be utilized consistently at portfolio levels to understand the success or otherwise of investment strategy.

A closing note on the Fitness Function

I believe the fitness function definition should be identified and agreed at the ART Launch Workshop. Well-launched ARTs will have all the key Business Owners present at this workshop, and I strongly believe that agreement on how the business impact of the ART will be measured is a critical component of mission alignment.

Series Context

• Part 1 – Introduction and Overview
• Part 2 – Business Impact Metrics (You are here)
• Part 3 – Culture Metrics
• Part 4 – Quality Metrics
• Part 5 – Speed Metrics 
• Part 6 – Conclusion and Implementation

“The gold standard of metrics: Actionable, Accessible and Auditable ... For a report to be considered actionable it must demonstrate clear cause and effect … Make the reports as simple as possible, so everyone understands them … We must ensure that the data is credible” – Eric Ries, The Lean Startup

Revamping SAFe's Program Level PI Metrics Part 1/6 - Overview

"Performance of management should be measured by potential to stay in business, to protect investment, to ensure future dividends and jobs through improvement of product and service for the future, not by the quarterly dividend" - Deming

Whilst the Scaled Agile Framework (SAFe) has evolved significantly over the years since inception, one area that has lagged is that of metrics. Since the Agile Release Train (ART) is the key value-producing vehicle in SAFe, I have a particular interest in Program Metrics - especially those produced on the PI boundaries.

In tackling this topic, I have numerous motivations. Firstly, the desire to acknowledge that it is easier to critique than create. I have often harassed +Dean Leffingwell  over the need to revamp the PI metrics, but not until recently have I developed a set of thoughts which I believe meaningfully contribute to progress. Further, I wish to help organisations avoid falling into the all-too-common traps of mistaking velocity for productivity or simply adopting the default “on time, on budget, on scope” and phase gate inheritance. It is one thing to tout Principle 5 – Base milestones on objective evaluation of working systems, and quite another to provide a sample set of measures which provide a convincing alternative to traditional milestones and measures.

Scorecard Design

It is not enough to look at value alone. One must take a balanced view not just of the results being achieved but of the sustainability of those results. In defining the PI scorecard represented here, I was in pursuit of a set of metrics which answered the following question:

"Is the ART sustainably improving in its ability to generate value through the creation of a passionate, results-oriented culture relentlessly improving both its engineering and product management capabilities?"

After significant debate, I settled on 4 quadrants, each focused on a specific aspect of the question above:

For each quadrant, I have defined both a basic and advanced set of metrics.  The basics represent “the essentials”, the bare minimum that should be measured for a train.  However, if one desires to truly use metrics to both measure and identify opportunities for improvement some additional granularity is vital – and this is the focus of the additional advanced metrics.

Business Impact

Whilst at first glance this quadrant might look sparse, the trick is in the “Fitness Function”. Wikipedia defines it as “a particular type of objective function that is used to summarise, as a single figure of merit, how close a given design solution is to achieving the set aims”. Jeff Bezos at Amazon quite famously applied it, insisting that every team in the organization developed a fitness function to measure how effectively they were impacting the customer. It will be different for every ART, but should at minimum identify the key business performance measures that will be impacted as the ART fulfils its mission.

Culture

The focus in culture is advocacy. Do our people advocate working here? Do our stakeholders advocate our services? Are we managing to maintain a stable ART?

Quality

For quality, our primary question is “are we building quality in?” Unit Test coverage demonstrate progress with unit test automation, while “Mean time between Green Builds” and “MTTR from Red Build” provide good clues as to the establishment of an effective Continuous Integration mindset. From there we look at late phase defect counts and validation capacity to understand the extent to which our quality practices are “backloaded” – in short, how much is deferred to “end-to-end” feature validation and pre-release validation activities. And finally, we are looking to see incidents associated with deployments dropping.

Speed

This quadrant is focused on responsiveness - how rapidly can our ART respond to a newly identified opportunity or threat?  Thus, we start with Feature Lead Time - "how fast can we realise value after identifying a priority feature?". Additionally, we are looking for downward trends in time spent “on the path to production”, mean time to recover from incidents and frequency of deployments as our Devops work pays dividends.

Conclusion

In parts 2 through 5 of this series, I will delve into each quadrant in turn, exploring the definitions of and rationale for each measure and in part 6 wrap it all up with a look at usage of the complete dashboard.

Series Context

• Part 1 – Introduction and Overview (You are here)
• Part 2 – Business Impact Metrics
• Part 3 – Culture Metrics
• Part 4 – Quality Metrics
• Part 5 – Speed Metrics 
• Part 6 – Conclusion and Implementation 

"Short term profits are not a reliable indicator of performance of management. Anybody can pay dividends by deferring maintenance, cutting out research, or acquiring another company" – Deming

Tips for Designing and Leveraging Great Kanban Boards

Introduction

I’ve been working on an article about the SAFe Program Kanban, and found myself mixing in a number of basic Kanban techniques. As I read through the (overly lengthy) first draft and realised the fuzzy focus being caused by a mix of “Kanban 101” and “Program Kanban”, I found myself reflecting on the fact that a lot of people kind of “fall into Kanban”. The two most common cases I encounter are the dev team that evolves their Scrum implementation to the point that the arbitrary batching mechanism of the Sprint Boundary seems suboptimal and the Agile Release Train (ART) Product Management team taking their first crack at a Program Kanban. For whatever reason, many start to use it without ever understanding any of the basic tools available other than “use WIP limits”.

In this article, I’m going to cover two of the basic Kanban concepts every team should take advantage of and a third which tends to be more applicable for strategic Kanban systems than those operated at the dev team level.

Doing and Done

One of the simplest improvements you can make to a Kanban is the separation of states into “Doing” and “Done”. This separation enables far more accurate visualization of the true state of a piece of work, adds immensely to the granularity of the metrics that can be derived, and most importantly is critical to enabling the move from "push" to "pull" mindset.
Consider the simple yet very common 2-state Team Kanban below:

When the developer completes story S1, they will signal this by pushing it into Test. However, the system is now lying. The fact that the developer has placed it in test does not mean testing has commenced (the clue lay in the use of the word "pushed").

Consider an alternative:

Now, when the developer completes story S1, they place it in "Dev Done". This sends a signal to the testers that when they are ready, they can "pull" story S1 into Test. If we see a big queue building in Dev Done, we can see a bottleneck emerging in front of Test. If (over time), we discover that stories are spending significant amounts of time in "Dev Done" it should trigger some root cause analysis.

You could also achieve the same effect by making a 4 state Kanban as follows:

• Dev
• Ready for Test
• Test
• Ready for Acceptance

To be brutally honest, the difference is intellectual. Aesthetically, I tend to prefer the “Doing|Done” approach, partially because it leaves me with less apparent states in the Kanban and mainly because I tend to assign WIP limits spanning “Doing and Done”. In fact, when designing complex Kanbans I will often use a mix of “Single State” and “Multi-State (Doing|Done)” columns from a clarity perspective. The “Single State” columns tend to be those in which no activity is occurring – they’re just a queue (eg “Backlog”).

Exit Policies

The creator of the Kanban Method (David Anderson) identified 5 core properties of successful Kanban implementations, one of which was “Make Process Policies Explicit”. In designing the states of your Kanban system, you are beginning to fulfill this need by making the key stages of your workflow explicit (and supporting another of the key properties – “Manage Flow”). For the evolving Scrum Team, this is often sufficient as it will be supported by their “Definition of Done” (another explicit policy).

However, at the strategic level (or for a Kanban system that crosses team boundaries) we benefit by taking it to another level and defining “Exit Policies”. An Exit Policy is effectively the “Definition of Done” for a single state. Whilst it is up to the team member(s) (or Teams) exactly how they do the work for that state, it is not considered “Done” until it meets the exit policies for the state. These policies should be made visible as part of the Kanban design, and should be subject to review and evolution as part of continuous improvement activities. In the words of Taichi Ohno – “Without standards there can be no Kaizen”.

Note the explicit exit policies below each state heading in this Portfolio Kanban

Avatars

Every piece of information you can add to a Kanban board is valuable in conveying extra information to support conversation and insight. Most teams are familiar with the practice of creating a set of laminated “avatars” for each team member. When a team member is participating in the work on a card, they add their avatar to the card as a signal. Thus, anyone observing a Kanban and wanting to know who is working on a card gets instant gratification. Incidentally, this is for me one of the biggest failure areas of digital Kanban boards. To my knowledge, the only digital Kanban tool that supports multiple avatars on a single card is LeanKit – a very strange condition in a world centred on collaboration :)

Now to extend the concept. There is no reason to restrict avatars to the representation of individuals. If we create avatars for Dev Teams, we can (for example) understand which dev teams are involved in the implementation of a feature on a feature Kanban. Take it up a layer, and we can create avatars for ARTs and other delivery organizations. Suddenly, we can look at a portfolio Kanban and understand which delivery organizations are involved in the implementation of an Epic.

The cards above are Epics on a Portfolio Kanban.  The "Standard Avatars" (with pictures) represent individual BA's, whilst the smaller solid color avatars represent the impacted delivery organisations (an ART in one case, Business Units in others)

Conclusion

There are many more tips and tricks to creating powerful Kanban visualisations, but these are the three I find myself introducing again and again as I help Scrum teams leverage Kanban techniques and ART Leadership teams implement strategic flow Kanban systems.

Always remember, as +Inbar Oren put it so well, a good Kanban board TALKS:

• Tells
• Always Visible
• Lives
• Keeps it Simple
• Self-explanatory

Bringing your SAFe PI Plan to life during execution

Introduction

The SAFe PI planning event leaves every team on the Release Train with a beautifully organized backlog identifying everything they need to do together to succeed in meeting their PI objectives.

However, as we’ve learnt from military strategist Helmuth von Moltke, “no plan survives contact with the enemy!” The scale and energy of the event makes it easy to fall into the trap of believing we’ve created a plan that will survive and inherited mindset drives the belief we “now have a plan to manage compliance to”. It’s easy to miss the fact that what teams are committing to is their objectives. The carefully identified backlogs and dependencies are simply things we have produced to help us understand the outcomes we believe are possible and a point-in-time representation of the best approach to achieving them.

Successful execution relies on us remembering that backlogs must live. Estimates will be wrong, stories will have been missed, we will discover more innovative ways of achieving the objectives.  In short, circumstances will change. PI planning provides a great start by launching the PI with 5 to 10 teams having beautifully coordinated and well-groomed backlogs but success rests on what we do with them once we leave the room.

All too often, I see new teams and trains treat the PI plan as a static artifact. Your plan should always reflect your current best understanding of the situation, and the primary representation of your plan in SAFe is your backlog. Having previously covered the ongoing lifecycle of the individual backlog item, I will focus here on the of principles and practices I have found most useful in helping SAFe implementations manage the plan as a whole.

The importance of visual management

Any scaled agile implementation will inevitably wind up using a digital tool. But PI planning itself gives us the clue – it’s an amazing collaborative planning event enabled by physical visualization of the plan. If you want your plan to live, physical visualization is essential. By all means, be responsible and keep your digital tool up-to-date – but recognize it for the information refrigerator it is. I’ve seen all the tools used both well and poorly, but I’ve never seen a tool enable effective ongoing collaborative planning to anywhere near the level and efficiency possible with physical visualization

The Team PI Plan – from Planning Room to Team Room (PI level visualisation)

I have a starter template for team level plan visualisation that I suggest to all the teams I coach. It works best when a large horizontal space is available and it can be organised in one continuous run, but most teams can find space for it with a little ingenuity. As a template, it looks a little like the following:

Every area except the Inbox and Overflow (covered later in the article) will contain an indication of planned velocity and load (as represented by current contents). Additionally, PI objectives should be somewhere prominent (preferably next to current sprint).

Team Master Builders: Mobile wall, top row is current sprint, middle is next spring, bottom future sprints

Team Olympus: Current to future from right to left

Team Nintendo: Current sprint on right window, next sprint on middle, future on left

Additional details for each area follow:

Current Sprint 

This is the visualization every team should have for their current sprint – often known as the Scrum board. Not the focus of this article, but it’s always good to bring it to life.

Olympus Current Sprint: Take every opportunity to build team identity through visuals

Next Sprint

The upcoming sprint gets slightly more focus than other future sprints, as this is where we are getting our stories to “Definition of Ready”. Typical swimlanes might be “Prioritised”, “Specifying” and “Ready. Operation of this wall is covered in my last post on story lifecycle

Team Pixar: Next sprint helping manage movement of stories to definition of ready

Future Sprints

These areas basically mirror the content from the PI plan. The backlog items and their intended sprints are carried out of PI planning and placed straight into their spot in the team area. As each sprint concludes, the cards move to the right to indicate the shrinking number of future sprints left in the PI.

Extended Backlog Tools

Given that the backlog represents “all the work” for the team, it is a great place to capture other activities which are not necessarily either classical “user stories” or “tech stories”. Working with ARTs, we regularly devise “specialty card types” to help them articulate and manage their plans. The two I use most frequently are “Commitment Cards” to help with team-level dependency visualisation and “Capacity Cards” to represent “work that’s not necessarily very agile but definitely necessary and often responsible for killing a team if they fail to plan for it”.

Commitment Cards

I was introduced to this concept by Craig Larman and Bas Vodde’s Scaling Lean and Agile Development, and have found it invaluable in SAFe. The identification of dependencies on the Program Board during PI planning (and fun with red wool) is one of the most visually powerful tools in SAFe, but little formal coverage is given to reflecting them at the team level.

The notion of a commitment card is that when one team makes a commitment to another they place a (0 point) story in their backlog describing the commitment made. The recipient also records the commitment in their own backlog as a 0 point story.

This assists on a number of fronts. Firstly, a dependency is rarely on a single story – it typically operates at a higher level of abstraction. Secondly, it becomes a first class citizen in the team’s backlog refinement activities.

Finally, it assists teams populating the program board with achieving an appropriate level of granularity when reflecting their dependencies. The commitment should already be written at the appropriate summary level of granularity, and can simply be duplicated onto the program board. If the commitment card shifts in the course of planning, its twin on the program board needs to shift.

Capacity Reservation Cards

Many teams have work which cannot specifically be planned for but is nonetheless reasonably predictable in size and timing and must be catered to. Whilst team-level capacity allocation is one option for dealing with this, it’s a reasonably blunt instrument which does not cater to situations where the scale of the reservation required varies from sprint to sprint.

The most common cause of this comes in the form of hardening activities. We all know good agile doesn’t rely on hardening, but many enterprises face the harsh reality that the elimination of extended hardening periods will require dedicated work over the course of years. For example, I routinely work with organisations which have some form of enterprise-wide release process that spans 6-12 weeks. We tend to find that the capacity of a train is divided between implementing the features for the current PI, supporting the progress of features from the previous PI through the enterprise release process, and providing warranty support for the previous release. This is particularly prevalent where mainframes, ERP systems and other major COTS implementations form a major component of the technology stack.

The bulk of work for teams arising out of these processes will be unplanned (eg defect fixes, incident resolution), but the intensity of activity and likely volume of work is more predictable. In these situations, we will place “Capacity Reservation Cards” in the backlog setting aside a certain number of points in the team’s capacity to be available.

An additional use we make of this is setting aside capacity for the team to participate in ideation or “discovery workshops” on upcoming features as they are prepared for the next PI. Whilst the IP sprint in theory provides capacity for this work, it has the effect of creating a late-stage batching effect on team involvement in feature preparation. Inserting “Feature Discovery” items in the backlog allows time to be set aside for the team sprint by sprint and spread these activities over the whole PI.

On a closing note, the usage of this technique enables one of my favourite PI metrics – “% of capacity required for hardening support”. Downward trends in this metric indicate the effectiveness (or otherwise) of efforts to bring forward quality and minimise the need for downstream hardening.

Managing Change

Whilst as agilists we “welcome changing requirements”, there is a reality that at scale these changes will have impacts and often those impacts will extend beyond the team directly dealing with the change. A chaotic world where the backlog is constantly changing and the impact of these changes is not discovered until late in the PI is not a great thing to have.

As such, I have found two tools to be very useful – the “Inbox” and the “Overflow”. The Inbox provides us with a mechanism for managing the introduction of new items to the backlog, and the Overflow helps provide visibility of backlog items at risk of not being completed within the PI.

Inbox 

As new stories are identified by the product owner (or others), they are placed in the Inbox as a holding area. The team can then establish ceremonies with the Product Owner for incorporation of new items into the backlog. Common agreements would include the team (and PO) achieving shared understanding of the backlog item, converting it to user voice form, estimating it and deliberately considering the prioritisation as it is inserted into the backlog.

Team Pixar Inbox and Overflow

A little humor from Team Aliens: Guess which is the overflow :)

Overflow

As reality occurs, new backlog items are discovered, actual velocity is realised and planned velocity is updated this area comes into play. Any backlog items for the PI which will not fit based on the planned velocity are moved to the overflow to highlight the priority decisions being made by the Product Owner.

Backlog Refinement

Whilst supporting the sensitivity that led to this practice acquiring its new name, the unfortunate reality is that it most commonly seems to lead to people missing the key activity. Knowing that story elaboration is taken care of through specification workshops, the primary focus for refinement becomes the management of content and priority of the PI backlog for the team.

I generally place this on cadence either once per week or once per sprint as a one hour session. It is the responsibility of the product owner, but best performed as a shared activity with the scrum master. As this is an alignment activity, I also look for all teams on the train to schedule their backlog refinement session on the same day (leaving it open to the team what time of day). Motivations for this will become clear when we proceed to Program Backlog Refinement.

Key activities will include:

• Updating velocity forecasts: This is obviously data provided by the scrum-master, but may shift either based on yesterday’s weather or as team leave comes into focus. 
• Process the inbox: All stories from the inbox which have passed through the established entry conventions should be inserted into the backlog in the relevant sprint based on the product owner’s priorities. 
• Adjust scheduling to match capacity: Based on the updated velocity forecasts (and any newly incorporated stories), any overloaded sprints should have stories shuffled back down to future sprints until the load level falls below forecast velocity (or other agreed load constraints). This will often result in stories from the final sprint in the PI being moved into the overflow. 
• Validate commitment cards: Cast an eye over all the commitment cards in the backlog, asking the following questions: 
• “Will we still be able to honour this commitment?” 
• “Will this reprioritisation affect a commitment we’ve made?” 
• “Do our changed circumstances mean we no longer require another team to meet the commitment they have made to us?” 
• Adjust priorities: Now that the backlog has incorporated the inbox and been adjusted based on capacity and commitments, review for priority updates. At a bare minimum, take a good hard look at the overflow area and see whether any of those cards need to be traded back in and others demoted. 

On a final note, I suggest using a flagging system during refinement. Place a flag (and accompanying note) on any card that moves. This supports the following:

• Updating the digital tool to reflect any changes made to the physical backlog. 
• Updating the team at standup the next day on the outcomes of the refinement session. 
• Informing program backlog refinement. 

Program Backlog Refinement

Whilst not formally specified in SAFe, I have found this to be an essential extension as a synchronization event. You will recall that all teams put their team level backlog refinement session on cadence to occur on the same day. We leverage this by placing the program backlog refinement ceremony on the following day. Quite often, we settle on Thursdays for teams and Fridays for the program. This is based on the fact that teams commonly adopt “Wednesday to Tuesday” sprints. You’re then refining the backlog the day after Sprint Planning (to reflect any updates coming out of the planning session) and on day 7 of the sprint (by which time you’ve discovered some new surprises and have a good clue of any stories which might slip the sprint boundary).

The ceremony has two phases – often with slightly different participation:

Team Updates (<30 minutes)

Attended by the RTE, Product Manager, ScrumMasters and Product Owners.

All teams provide an update of any outcomes from their backlog refinement session which may affect other teams. The focus is on moving or at-risk commitments. Typically held at the program level visualization (whether it be program board or something else).

Overflow Review (30-45 minutes)

Attended by the Product Manager, Product Owners, RTE and potentially other stakeholders.

This is a “management by walking around” exercise. The group moves from team area to team area, with a particular focus on the contents of the overflow areas. Using the flags applied during team backlog refinement, the product owner walks through the outcomes of the previous day’s refinement and validates their prioritization calls.

It provides the group with insight into areas where the overflow is mounting or critical commitments are entering high-risk status. This information can then be fed into the release management meeting to support program level decision making on moving scope to other teams or establishing effective mitigation strategies.

Conclusion

An agile plan lives. In the spirit of transparency, it always reflects the current reality and beliefs of the train. It is critical on many fronts:

• Supporting a team maintaining ownership of their larger commitments, avoiding the trap of living only for the current sprint and losing sight of their commitment to a broader mission. 
• Supporting ongoing synchronisation of plans across all the teams in a train 
• Supporting effective trade-off decision making by the Release Management Team 
• Providing early warning to support effective mitigation activities as the plan fails to survive contact with the enemy. 

The PI planning event provides an amazing springboard, but effective planning must be continuous and based on a transparent reflection of reality. Utilising cadence and synchronisation as enablers, the practices described above are simple to implement and will provide astonishing contributions to maintaining the alignment, transparency and focus of a Release Train.

Team Backlog Evolution in SAFe - from 3 words on a sticky to a Ready to Play story

Introduction

The PI planning event provides a great pattern for filling a backlog, but its all too easy for teams to fail to capitalise on their great beginning with effective ongoing backlog refinement.

How do we get from those hastily scrawled post-it’s to fully articulated stories in user-voice form with comprehensive acceptance criteria? Given that many Release Trains move from “all-waterfall” to “all-agile” via a one-week quick-start, a solid set of starter pattern practices is a useful tool to help them accelerate their journey. This article focuses on the evolution of an individual backlog item. A follow-up article will cover techniques for managing the overall backlog.

Whose job is it?

In my early agile days, I learnt that a story was a token for an ongoing conversation between the team and their customer. The important thing was the conversation, some details of which needed to be preserved by writing them down.

Unfortunately, all too many people interpret user stories as “the new form of requirements”. They want to know who’s responsible. Does the Product Owner write the detailed stories and hand them to the team? Does the BA work with the Product Owner to do so? These are depressingly common anti-patterns, and our first mission is to avoid them. Story evolution is a collaborative activity regardless of the stage of maturity of the story, and I always start teams off with the mantra that nobody writes alone.

PI planning sets the scene perfectly. Each team starts with an empty backlog, and over the course of the breakouts works with their product owner, product manager and subject matter experts to explore their features – recording the results of the exploration as a set of backlog items.

So we’ve started well by having a conversation and recording a set of tokens for the details of the conversation on post-its – how do we keep that good pattern going after we leave PI planning?

Does it always have to be the whole team?

Having the whole team involved in an activity is an expensive (and not always productive) exercise. While the fully shared context is desirable, it’s not always necessary or valuable. Involving the whole team in workshops in their early life for the shared learning is great, but most evolve reasonably rapidly to some variation of Jeff Patton’s “3 Amigos” or triad approach. Jeff describes a triad as “Product Owner, BA and developer” (the 3 amigos). I don’t tend to be quite as prescriptive – what I’m looking for is diversity of input, so I tend to recommend “one of each discipline”. The most common format for this is “Product Owner, BA, Dev, Tester” or “Product Owner, UX, Dev, Tester”.

One caveat to this is to make sure triad memberships rotate to spread knowledge. The Product Owner will be a constant, and if the team has a BA they’ll also usually be relatively constant but what we don’t want is a pattern where it’s always the “dev lead and lead tester”. The more the understanding of the problem domain is spread around the team, the better the outcome!

Moving to User Voice Form

Teams rarely generate great user voice form stories during PI planning. There just isn’t time. If we’re lucky, they might have done so for their first sprint but most of the backlog will consist of items like “pay by the month”.

It’s highly tempting to detail someone to go off and “upgrade the stories”, but we miss something. The point of user-voice-form is to help us to look through the eyes of our users. The discussion as we identify roles and “so that” reasons enriches our understanding of what we’re trying to accomplish. The last thing we want is for the team to miss out on that conversation.

So, the team should set a goal of having a “rolling 3 sprints” of backlog that’s been refined to user voice form and moving as rapidly as possible to achieving this after PI planning. It’s a pretty rapid activity, so usually not all that hard to achieve with a set of well-facilitated one hour workshops. There’s a good reason for this. As we move to user voice form, we flush new insights – often in the form of missed stories or missed complexity. It’s good to learn about this early in the PI.

Establishing a Definition of Ready

Nothing destroys a sprint like a poorly defined set of stories entering the gate. All teams are taught to establish a definition of done, but I’m a huge personal fan of a “Definition of Ready”. This defines the set of conditions a story must meet in order to be accepted into Sprint Planning.

Exactly how much or how little is required to “be ready” is highly dependent on context. In highly regulated environments this often includes signoffs from legal and risk on acceptance tests. These are folks not renowned for moving fast - discovering you need an approval from legal in the middle of the sprint is a recipe for an unfinished story and missed sprint goals.

A good question to seed the definition is “what are you likely to discover in the middle of the story that might take more than a couple of days to get an answer to?”. This then forms the guidelines for key things to answer when preparing the story. The rule of thumb I’ve found quite useful is getting the acceptance tests to “80% lockdown”. There are always some questions that will be revealed once developers crack the story open, but the “80% rule” enables an approach where “if the expectation just changed by more than the 20% we allowed for the new detail its fuel for another story and a future sprint”.

My strong recommendation to all teams is “provide all the detail in your acceptance tests”. The temptation to elaborate the story by filling in tables of business rules and bullet point acceptance criteria before developing acceptance tests just leads to waste and redundancy. Using techniques like Specification by Example or BDD workshops take you straight to acceptance tests and provide very useful ways of exploring the problem domain collaboratively.

Employing a definition of ready as well as a definition of done establishes a two-way contract between the team and product owner:

• The product owner take accountability (with the support of the team) for ensuring that stories meet the Definition of Ready before being brought to sprint planning.
• The team takes accountability (with the support of the product owner) for ensuring that stories meet the Definition of Done before being reported as complete.

The instant benefit most teams will observe is both shortening the sprint planning meeting and enabling it to move focus away from “what will be done” to “how they will work together to do it”, and their ability to hit sprint goals will also increase significantly.

The “Next Sprint” Kanban

Story detail should leverage the law of the last responsible moment. We want it neither too early nor too late. My encouragement is to go no deeper than a user-voice form version of the story until the sprint before it is due to play.

Each sprint thus has a dual purpose. Complete the current sprint’s committed stories and flesh out the next sprint’s stories to meet the definition of ready. The first ingredient in doing so is to introduce the “Next Sprint Kanban” to visually manage this journey.

The board illustrated below was from a high assurance environment, where acceptance tests were detailed in the “Specifying” state, then stories were sent off to legal and risk in the “sign-off” state prior to hitting “Ready”.

Figure 1 - Team Aliens: A slightly different approach to visualizing next sprint preparation and a helping hand from Dilbert

Specification Workshops

“Specification by Example” is a great way to flesh out a story. Whilst there are whole books written on the concept (my favourite is definitely Gojko Adzic’s), I have found 3 keys to success:

1. Put them on cadence. Two-three 1-hour sessions per week will usually feed a team pretty well. Scheduling them straight after standup is great as you can use the standup to organise who (whole team or designated amigos) will participate.
2. Don’t produce polished detail, your focus is shared context. Use a whiteboard (preferably in the team area to allow for osmotic communication) and lots of photos as you explore to keep the pace moving and energy up. Word-smithing can be done either solo or by a pair (Product Owner/Tester works well) as followup.
3. The Product Owner needs to order and do some pre-thinking on the stories both to help them identify any subject matter experts or other stakeholders who might need to be involved and to assist the team in identifying appropriate amigos to participate. 

Story Splitting

Good backlogs are, of course, iceberg shaped. As Jeff Patton puts it so eloquently “stories are like cheese, best sliced fresh”. Story splitting will tend to occur naturally as a by-product of user-voice-form workshops and specification workshops. Whatever you do, don’t bury yourself in deep story hierarchies. One layer is good. “Here is my feature, and here are the stories that represent it”. When you split a story, tear up the original and replace it with the newly identified split stories.

What about Kanban?

Everything in this article works as well for a Kanban team as a Scrum team. In fact, I find that the more mature a team becomes in applying these practices the more it moves towards operating in flow with Kanban and leaving Scrum behind. Adaptation is simple – instead of thinking about “next 3 sprints” or “next Sprint”, apply WIP limits to identify the optimal amount of detailed story inventory to hold.

Want a supercharged ART? Don't settle for a Proxy Product Manager!

In a recent SPC class, we wound up in a very interesting discussion about the role of the RTE and the potential need for multiple RTE’s. Given that in our very first Release Train we had employed a “paired RTE” approach (described here by my colleague Em Campbell-Pretty), we had a fairly public opening position on the discussion. I have a client who has a rotational RTE system where each train has 2 RTE’s. One leads execution of the current PI, and the other focuses on preparation for the next. We’ve also seen and heard of numerous other variations. As the discussion progressed, something crystallized for me. In every single case where it’s been necessary, the Product Management function has either been weak/uncommitted or fulfilled in proxy by the IT organisation!

One of the great strengths of +Dean Leffingwell’s approach to defining SAFe has been his pragmatism. One example of this has been his historical approach to defining the type of people (traditional role descriptions) who might fulfill a SAFe role. As with the entire framework, he distills and documents ‘observed patterns’ rather than theory. I vividly remember a moment of shock and horror when I saw that he had listed “BA’s” as potential product managers. Whilst having acknowledged the occasional need to utilise “IT proxy” product owners, the notion of a BA as Product Manager triggered the following email:

Hi Dean,

I'm teaching an SPC and just had the product manager slide onscreen and someone in the audience pulled up the guidance article and said this could be a BA.

After I fell off my chair I suggested someone must have applied the content to the wrong article (ie meant product owner). Are you able to confirm I was right and someone hasn't started smoking strange drugs?

In typical Dean style he reminded me that many organisations have extremely senior strategic BA’s who could well fulfill the role – which got me thinking about a couple of BCG BA’s I had worked with and resulted in a humble apology.

The drawback, however, is that where Dean describes something that “can work in particular contexts”, it is all too common for organisations to interpret the guidance in unfortunate fashions – in this case implicit license to put an IT person in the role. This likelihood is inflated given that most large IT organisations have formed a “wedge division” of “business-facing IT” who act as interpreters and demand managers. The value proposition of such an organisation is threatened by the direct connection between business and developers underpinning agile and it desperately seeks survival .. “surely we are the logical product managers and owners!”

What Dean is looking for is someone who can fulfill the responsibilities and accountabilities of the role. So, what are they? A Product Manager is a person entrusted by the organisation with identifying the best mix of features and steering the optimal economic trade-offs between them to maximise the ROI on an investment of somewhere between $10M and $30M per year.

In recent months I’ve been fortunate enough to launch two Release Trains with not just exceptionally strong co-located business Product Managers but also dedicated co-located business Product Owners for every team. In both cases, they have matured further in a single PI than I have seen in many others across the course of 18 months. Following are some of my thoughts on the key differences:

• Agile is about eliminating the concept of “business and IT”. It’s about working together on a shared mission. It’s not about something “IT does to the business”. Proxies simply maintain the status quo wearing a slightly different disguise.
• The container of the system shapes the system being optimised. As long as the Release Train contains only IT, it will suffer from local optimisation with the relevant area of the business forming a separate system also locally optimising.
• Business people come with business networks. Whilst a short-lived team can source a product owner with deep domain knowledge, a long-lived team will over time work on features covering many domains – to survive the product manager and owners need to know how to bring the right subject matter experts to the table to provide the deep domain knowledge. This is far easier when you’re accessing your mobile phone contact list than researching an org chart!
• Business Product Managers/Owners know how to get the right people to the System Demos and demonstrate in business language about business outcomes being achieved.
• Business leaders bring more to the table than just content authority. They’ve also often received far more development in people leadership than their IT counterparts. Watching somebody bring their call centre team management experience to the table with dev teams is simply amazing. 
• Proxies will be proxies. They will naturally preserve their value proposition by acting as translators rather than connectors. 
• It’s very different when a business leader elects to defer something than when an IT leader asks a business leader to defer something!

Chatting to the executive sponsor during the second PI planning event for one of these trains recently, he shared that one of the things that particularly excited him was watching his business teams reach insights and form alignment at a business process level as they collaborated with the dev teams and their product owners in breaking down their features. He had a grin from ear to ear, but it wasn’t quite as big as the grin I was wearing the night before when I was forwarded an email from another train. It contained the following story:

Once upon a time there was a new Agile Release Train (ART) that had just finished its third sprint. Everyone was amazed at the velocity the teams had achieved. But at the retrospective 2 things became clear:

(1) the teams had worked long hours and many nights to achieve a higher velocity to overcome what they saw as a hump necessary to meet the desired business date for production after 2 more sprints.

(2) the teams were doing "wagile", coding in week 1, then testing in week 2.
The Product Manager thanked the team for their extra efforts, but told them, and - importantly - all the attendees at the showcase, that the velocity achieved was not sustainable, and long days and nights were not to be repeated.

The ART then reviewed the way the teams were working, and decided that extra attention needed to be paid to the adoption of Test Driven Development (TDD). The coach also refreshed the teams on how to conduct sprint planning for sustainable pace.

The Product Manager then told the teams to take as long as necessary to plan for the next sprint, including how to adopt TDD. With some further guidance from the Release Train Engineer (RTE) and Systems Team, the teams re-planned how they would work, as well as setting a more sustainable velocity target for sprint 4.

After I shot an email to the Product Manager saying “hey, can I please quote your story”, she regretfully informed me that she hadn’t been the author but had the following to add:

It was a tough decision knowing that maximum capability in the short term is what senior management are looking for, but also being aware the greater value lay in ensuring the team had the time and space to embed the practices and achieve a sustainable long term velocity and you know the saying, short term pain, long term gain.

On a closing note, my few weeks of pondering the topic have left me with a realization. We obsess about the critical role of the Release Train Engineer. At every gathering of SAFe people, discussions center around how to find, retain and avoid burning out great RTE’s. I think perhaps we’re guilty of spending too much effort on finding and growing RTE’s and not enough on Product Managers. In fact, I’d go so far as to say that in pursuit of great business outcomes and a thriving ART a great Product Manager can cover for a weak RTE far better than a great RTE can compensate for a weak Product Manager!

Getting the most from the Management Problem Solving session at SAFe PI Planning

The longer I have been facilitating PI planning, the more I have come to believe that the primary purpose of Day 1 is to set the scene for the Management Problem Solving session. The energy of the visioning, team breakouts and plan reviews are well-known, but the management problem solving at the end of Day 1 is a crucial part of the event often glossed over.

Let’s start with a simplified version of the PI Planning agenda:

Day 1:

• Present the vision (Leadership)
• Create a draft plan revealing the flaws and challenges in achieving the vision (Teams)
• Face into the insights generated by the teams (Leadership)

Day 2:

• Communicate the response to the team insights (Leadership)
• Breathe a sigh of relief and craft an achievable set of objectives with a supporting plan (Teams)

My goal in this post is to share thoughts on the best way to facilitate the session, but before getting into details I’d like to dwell for a moment on why it’s so important. In the early days, this event is often the only real time senior leaders spend at the gemba. Instilling a culture of lean leadership and management by walking around takes time – particularly with busy executives. They’re not only getting a taste of agile collaboration, but they get to listen to teams wrestle with the system of work as they try to find a way to achieve the desired outcomes. There is no “please explain” on the problems. No need to prepare briefing decks. They’re all writ-large in the transparency of the planning room discussions. They cross silos and boundaries, there is no hiding from reality. And the problems that would normally be quietly hidden within a given organisational silo are out in the open. We’ve preached about Leadership’s duty to fix the system of work in the training room, and now they’re at the coalface with the opportunity to mirror the collaboration the teams have been displaying all day. 

As a coach or facilitator, you have a golden opportunity – you need to make the most of it. The first hint is that if you’re the RTE, try and get an external facilitator (be it a coach or somebody else). You want a voice in the discussions.

Expectation Setting

The “official agenda” shows management problem solving occurring from 5-6pm. The reality is most sessions will run 2-3 hours – I’ve run as late as 11pm, and I’ve heard stories from Dean of post-midnight finishes. Participants need to know you’ll be asking them to work late and understand why. It gives them the chance to fly in the night before rather than trying to stay awake after an early morning flight, and to warn their families.

When you’re scheduling, provide a break (it’s been a busy day) and preferably some food (brains need food to face into tough discussions). I typically suggest scheduling the session for 6-9pm to set appropriate expectations, then work towards finishing between 8 and 8:30. Also, work with your scrum-masters. Help them understand that their primary goal at the end of day 1 is to provide clarity on the problems that need solving. Be ready to remind them of this at the final scrum of scrums.

Structuring the facilitation

I don’t tend to start with a fixed approach to facilitating the session. I’m observing during the day, looking at the types of problems emerging and thinking about the best way to structure the session to help achieve the optimal outcomes. While the leaders are having a break, I take the opportunity to create a working space and setup my visualisations. If at all possible, do it in the PI planning room so you’re surrounded by the day’s work.

At a minimum, you need a visual space for actions and decisions. Further visualisations will depend on approach. After much experimentation, I have settled on 3 primary variations.

The Retrospective Option

For this option, I structure virtually the entire conversation around a 4 L’s activity. 4 feedback areas are established (Loved, Learned, Loathed and Longed For), and participants are invited to respond with post-it’s which are then affinity grouped. 

Open with “Loved” – it’s good to get some positive acknowledgements flowing but also opens the conversation flow and tends to provide a little more safety for when the discussion gets tough. Then, open each affinity group in turn by asking for participants to elaborate. Virtually every affinity group will hold the seeds of 1-3 actions and/or decisions. Allow the conversation to flow, probing with open questions if necessary but be listening for the seed of an action.

It’s not uncommon to have 2 flip-charts full of recorded actions and decisions by the time you’ve processed your feedback.

The Simple Study Option

Open with a reflection on what went well. This should be short, but helps prepare to enter quality discussions about challenges.

Next, send your group out to study the work. Ask them to tour the room (as individuals) and study each team’s plan visualisation. At its simplest, you might arm them with a pad of post-it’s and a sharpie and ask them to note down the problems they believe need to be discussed for each team. You may also ask something more specific. On one occasion, I created a grid with a row for each team and a request that participants rate (on a scale of 0 to 10) both the achievability and the clarity of each team’s plan. A timebox of 15 minutes generally suffices for this.

Upon their return, collate the results. This might involve the creation of an affinity grouped set of post-its around discussion topics or problems to solve, or some other form of visualisation. The photo below shows the whiteboard where I collated the results of the “achievability and clarity” rating mission.

When collating, you might also choose to help them set some priorities. One technique is to provide 3 separate areas – “Must discuss tonight”, “Should discuss tonight”, “Nice to discuss tonight”. Whatever you don’t make it to becomes a takeaway list for the RTE to facilitate follow-up discussions around in Release Management Meetings.

Now you need to timebox the discussions. I believe Lean Coffee provides a very handy control mechanism. The topic timebox will vary depending on your approach – “team by team” will need longer timeboxes than “problem by problem”. Your goal with each timebox is to arrive at one or more actions or decisions in response to the prompt.

The "Hello Game" Study Option

This is a slightly more structured study option based on Thiagi’s “Hello Game” frame game.  The approach involves dividing participants into 3-4 “study groups”. As with the Simple Study Option, participants are given time to roam the room and study the plans. However, they are given a more structured mission. As the facilitator, you need to come up with one “framing question” per group. As members roam the room, they are asked to take notes in response to all 4 framing questions. However, you then enter a second phase. Each study group is assigned one of the framing questions to focus on, and receives an additional 15-20 minutes to survey the room and determine an approach to aggregating, visualising and playing back the key insights in response to their framing question. 

This is currently my favourite approach, as I find it yields deep engagement and more challenging discussions. There’s a little mayhem as people survey each other, but the one-on-one discussions held to gather their data and the effort the study groups put into visualising it (eg graphs, diagrams) is brilliant. The job of the facilitator, of course, is to spend all day pondering the 4 framing questions that will yield the best insights. As a coach, there are usually some key themes I can see but I’d like to give my participants the opportunity to generate their own learnings rather than hand them out myself. Good question selection facilitates this. I almost always create one or two questions based on the day’s events, but I also have some stock questions up my sleeve such as:

• How have we improved since our last PI planning event? 
• How could we have improved our preparation activities to enable a more effective PI Planning? 
• What systemic issues are evident in the teams planning that we do not have improvement plans in train to address? 
• What are the top issues most likely to derail our PI execution? 
• What issues are most likely to prevent the teams reaching a committed plan tomorrow if not addressed? 
• In which areas are we facing scope trade-offs, and what is the extent and contributing factors of those causes? 

Following is an example of a framing question I used recently to draw attention to something the leadership group really needed to ponder:

• What are the key factors contributing to the overabunce of tech stories and widget based objective statements?

Once the study groups have created their visualisations, ask each group in turn to present their outputs. Harvest actions and decisions as these conclusions are presented, explored and debated.

Closing

Once you’ve exhausted the topics opened up by your information gathering approach, there’s a final question to ask:

“What have we not yet discussed that we should have?”

It’s amazing how often the toughest topic or two of the night open up at this point. Facilitated well, the group should have reached the safety needed to delve into something that felt too controversial in the beginning.

Preparing for Playback

At this point, you can release all the attendees except the RTE and the Product Manager. Your final bit of work is to agree their playback approach for the morning. It’s fairly straightforward.

• Agree who will present the “positive feedback” 
• For each identified action and decision, identify whether it needs to be fed back to the team in the morning and who (RTE or Product Manager) will speak to it. 
• Agree on any pre-briefing that will be given to the Scrum Masters and Product Owners prior to the general Playback session.

Selecting a facilitation approach

I circulated the draft of this post with my colleagues to test it prior to publishing. Much of the resulting discussion involved “when to pick which option”, with particular focus on “how mature does a train have to be before you take the “hello game” study option. My view is that it’s not so much a matter of train maturity as facilitator bravery. For the first PI planning of any train, I usually start with the retrospective option. You are looking for obvious, surface level challenges. On all other occasions, I use the “hello game” study option. It provides the deepest discussions and insights, but as the facilitator you need the courage and conviction to ask your participants to do real work rather than sit around a table responding while the facilitator does most of the work :)

Final Note

On average, I find 30-40% of the actions and decisions to emerge from a good Problem Solving session affect Day 2. The remainder are for action by the leadership team during the PI. An executive in one of these sessions last year commented that the leadership team needed to hold themselves accountable to the teams for following through on these. The result was the inclusion of a section in the System Demo agenda for leadership to report back to teams and stakeholders on their progress with the actions emerging from Management Problem Solving. I loved it, as it demonstrates to the team that their hard work in planning drives leadership focus as well as their own work.