Achieving organisational Agility at Scale

The Agile movement was born from the poor consequences of techniques from the industrial revolution being inappropriately applied to complex adaptive systems. Simply put; when faced with an uncertain world, an approach that optimises for feedback and learning has proven to be more successful than one based on upfront knowledge. As a result Agile is all the rage these days and the focus within meetups, conferences, and propositions of top tier consulting firms is no longer whether companies should be Agile, it is how to achieve it.

I’d like to think that the debates between bottom up versus top down for Agile Transformations are largely over, clearly the answer is a mixture of both. But what does this actually mean in practice, and how can we hope to get ahead and learn from others?

In this piece I walk though that while Business Agility occurs at the bottom of an organisation, it needs to start at the top and is sustained only though the relentless support of its leadership.

Bottom up transformations create bubbles within the organisation which are extremely difficult to grow and very fragile – you are forever fighting the organisational culture. Top down transformations are either insubstantial or so slow as to fizzle out. Those that are pushed hard run the risk of upsetting the very people they were designed to support.

I advocate that the best approach for transformation is to encourage the organisation to instigate a rapid cycle of purpose and escalation, and I propose the following approach to achieve it.

1. Start at the top
2. Align on Direction
3. Establish Metrics
4. Pass to the teams to define solutions
5. Address constraints
6. Improve to shorten cycle time

This brings about a virtuous cycle of small changes towards a common ideal, raising organisational constraints that leadership continuously resolve.

The challenges of a bottom up approach

This is how most team facing Agile coaches have experienced the world. Assigned to one or more teams, they have worked to instil the correct values and culture in teams to assist them in becoming more self-sufficient and focused. The Agile coach brings a mirror to the team, enabling the team to introspect and improve for the better.

This usually brings great results for the team, the challenge is how to sustain these ways of working now that the team is functioning differently to the rest of the organisation.

The team probably has the following differences:

• Their demands on the time of the business stakeholders is higher
• Their ability, and likely willingness, to provide long term estimates and date commitments is lower
• The team will want to focus their efforts on a pipeline of their own creation in pursuit of a goal rather than taking orders to deliver tasks
• The team will want to retain a stable structure and working relationships

These differences build a bubble within the organisation. A section of the company that not just follows slightly different processes and governance, but has a different value set and culture.

As far as the rest of the organisation is concerned, this is something alien; and once identified the host organisation behaves akin to a living system and the antibodies come out to remove the “infection”. For the bubble to persist it requires a senior leader to be able to protect those around them from pressure to revert to the status quo; and this goes one of two ways:

Bottom up result one – the Bubble wins

The leader pushes on and instigates new bubbles with fellow like-minded employees and then moves up, widening their bubble until more of the organisation is inside these bubbles than outside. Now the status quo are these new ways of working and the pressure switches to be on everyone else to fall in line with the new majority. This can happen in small organisations, where the distance from top to bottom in the organisation is relatively small, and moving up to influence higher is possible quickly. In my experience this is a rare occurrence.

Bottom up result two – Stifle and pop

Sadly this is the more common situation. The leader grows the bubble as much as possible but the greater the size, the greater the pressure and eventually the bubble stabilises. The bubble remains under repeated attack and the areas on the edge start to be forced to compromise their new operations. The frustrated leader finds their options limited and to personally progress either moves sideways or leaves it all together. At this point, without the support of their advocate, the bubble bursts. New leadership comes in and in a very short space of time, the previous ways of working are restored.

This is commonly the scenario associated with the introduction of a consultancy that focuses on team delivery and team level ways of working. The Consultancy is able to create and protect a bubble, usually within IT, but can’t get over the organisational silos into the business portfolio management and finance. Success is declared in the bubble they have created and they then leave the organisation whereupon the immune system invades and reverts much of the operations back to how they were.

Craig Larman of LeSS fame sums this up beautifully in his first law of organisational behaviour:

“Organizations are implicitly optimized to avoid changing the status quo middle- and first-level manager and “specialist” positions & power structures.”

In other words, those people that have built their careers on the existing ways of working will work to preserve the current situation. These people are the antibodies of the organisation rooting out the Agile infection.

The challenges of a Top down approach

The top down approach unsurprisingly removes most of the issues with the bottom up approach but introduces a collection of new ones. Here you clearly have the buy-in of the senior leaders and is usually found where either you have a new senior leader or the existing leadership has had a push (poor results, recent experiences or relationships with consultancy partners). From first impressions this should be more successful, you have the people who have the greatest influence pushing for something, so in theory everyone below should just fall into line and the world becomes a better place…

What commonly happens is that the senior leaders agree on ways of working that they have heard or seen work elsewhere, and look to emulate those within their own organisation. On first impression, this seems really sensible, and people are really working for the best intentions, without subtext or political manoeuvring. However in implementation this approach runs into three common issues.

Top down result one: Agile becomes the goal

The understanding of the Agile values and principles are not truly appreciated and the more practical, tangible processes and tools are what Agile is identified with. These processes and tools are typically things that are used by people at the coal face, and importantly are used in places other than “here” (where the leaders are sitting). In other words this starts to be less about an organisation BEING Agile, and more about specific departments DOING Agile. Whether you are being Agile or not is hard to measure, that is all about opportunity, speed to adapt, workforce engagement etc, all hugely critical but hard to run reports on. Execution of Scrum practices and implementation of aligned tooling though, that is much easier to monitor and track. Slowly the purpose of the transformation slips, from feedback driven customer centricity, to just being able to demonstrate Agile associated practices. The Agile transformation starts to become a project, it becomes the goal rather than a means to an end.

“All of our teams are now using Scrum, every work item in IT is now in Jira. We are Agile, Hurrah!”,

But of course you aren’t. Success is declared relatively quickly, which should be suspicious in itself, because really, nothing has actually changed very much.

Top Down result two: Imposing someone else’s solution

“A friend of mine has a great new electric tin-opener, I saw them use it, was really impressed, so I am going to buy one.” This situation seems fairly reasonable and we have all done something similar at some point in the past. However this only works because the tins being opened are standardised. My friend’s tins are the same design as mine and therefore I will be able to use the device as well as they can. The association between problem and solution is predictable and repeatable. In complexity theory terms this is a mechanical deterministic system, with a good answer that can be repeated anywhere, any time and by anybody.

The issue here is that the ways of working in complex organisations can’t be just lifted and dumped to deliver the same results.

As Dan North put it: Behaviours = (Practices x Context)

The point here is that context is king, and yours will be different from theirs. You can’t take a solution from one complex adaptive system and attempt to replicate it. The ways of working and even organisational structure that exists in one organisation has been optimised from base values and principles to create something unique. If you are looking to emulate their success then it is that emergent continual improvement and their base values and principles that need to be copied, not their end result. You need to find what will work for you, not take the answer to someone else’s problem.

Typically the people on the delivery end of operations can point out the short comings in the suggested ways of working, as to why something might not work well here. The challenge is that the leadership are not looking to encourage values and principles, but more insist on predetermined practices. This is actually deeply anti Agile. These are no longer self-organised teams, but victims of an enterprise wide imposition, now commonly referred to as the Agile Industrial Complex. These enforced new ways of working upset the people doing the work, who are much closer to the issues. Their suggestions are dismissed in favour of this single set of operating processes often collated into an off the shelf standardised playbook of sorts. Ultimately this is likely to lead to a rejection of the new ways of working, reverting back to previous models with the added tag of, “Agile doesn’t work here!”

Top Down result three: Boiling the ocean

There are situations where the leadership is on board and have an understanding that we are looking for a mindset change, something deep and significant that affects the organisation holistically including the leaders themselves. They have made the case that you can’t lift and shift from somewhere else and so what is needed is something carefully crafted, that considers the subtle nuances of their current operations, market and people. In making the case that this is to be holistic… you’ve widened the scope of ambition. The reality of the consequences of the transformation starts to bite. This is going to affect processes, governance and funding, expressions of purpose, organisational design, performance management, technical practices, culture, tooling. Everything is in scope, everything is going to change.

This is now a huge undertaking and a company that is used to traditional waterfall ways of working will see this as the largest project ever attempted, which naturally needs extensive analysis to establish what will work. Before long for the complex nature of the challenge becomes evident, the multifaceted interdependencies between all the constituent elements become problematic. This is the most complex of complex projects; ultimately you end up trying to design an entire organisation from the ground up, starting with organisational structure, which instigates a round of upskilling and recruitment to bridge the gaps between the current and desired states.

This is enormous, and like any complex project will confound the most ardent of project managers whose tools and techniques are designed for deterministic predictable mechanical environments. Ultimately you end up instigating your company’s most critical waterfall project to implement Agile, which is as ludicrous as it sounds. The interdependencies of all the moving parts yields slow theoretical conclusions which are in turn fed into decisions on organisational design which due to their importance will be slow to make. The end result is something that is so large and so slow that it is highly likely to be abandoned.

The Holistic Transformation

What is needed is something that takes the best of both worlds, something that enables that Bottom up “bubble” to grow and be protected, but also something that enables changes to begin so that momentum, rather than inertia, builds.

A holistic transformation is where the Bottom Up and Top Down approaches are mutually supportive bringing a virtual circle of continuous improvement.

To help to understand this path, consider a much simpler situation than your current one, consider just one team, a one team company, just seven people in a room, and they want to improve things. Chances are they will all get together, discuss their challenges, and describe what better will feel like, look like. Then come up with some suggestions, pick one and see how that works out, and then repeat. Pretty simple really.

The challenge organisations have is fundamentally scaling this. The objective is to be able to apply the principles and practices of Kanban at an organisational level. Starting where you are, look to make continuous evolutionary improvements across the board, constantly learning from feedback to find the next issue to resolve. The optimum ideal is to build an adaptive learning organisation.

In my personal experience the best approach is to start at the top, then quickly loop down to try something and pass the feedback up to the top again. And I describe this journey in the following steps:

1. Start at the top
2. Align on Direction
3. Establish Metrics
4. Pass to the teams to define solutions
5. Address Constraints
6. Improve to shorten cycle time

Start at the top

 Ideally you want to start at the very top of the organisation, the Exec board. Where you start will determine the size of the bubble of Agility that you create. If you start solely with the IT director, expanding later to make a genuinely Agile organisation, encompassing Product, finance and HR will be very difficult. Starting is the hardest step, starting with all the necessary people and respective departments aligned is extremely difficult as the status quo is highly likely to see these people being incentivised to compete against each other, protecting their siloed departments, their seats of power.

Align on Direction

Before you can help an organisation to realise an Agile transformation, the leaders need to be clear about exactly what this really means. This needs to be clear from a conceptual values based perspective but critically also from a blunt practical perspective. The hidden powerful question “what is in it for me” needs to be answered.

It is important that Agile is not the goal, ensure we are always focusing on “What are we trying to improve”. When a leader states they want the company to be more Agile, ask “Why?”. What is the problem you are trying to solve, and then add to that, “And how are you measuring the problem so we know if what we are doing, is helping?” You can’t really start until you have these pieces in place. You are looking for both autonomy and alignment, unless you have a clear guiding vision from the top, then the activities of the teams underneath will go in all directions and not deliver the desired results. A common problem though is when the senior exec team doesn’t operate as a team. If they are not aligned themselves, how can you expect the rest of the organisation to fall in line? If you suspect that this scenario faces you, then it needs to be addressed immediately. For guidance; Patrick Lenconi’s five dysfunctions of a team deals directly with this topic.

Establish metrics

The measures of “Betterment” expressed as Metrics to the organisation are a corner stone of setting a vision everyone aligns against. It gives a point of unity between all employees in all departments. This is step one in reducing internal rivalry and increasing cooperation between siloes to delivery customer value. These metrics exist at a company level, but when applied throughout the organisation should manifest themselves at every level, at the portfolio, at the team and ultimately to the individual. The question to all employees: “Are YOU living these values?”

To get agreement on what better feels like, start with values, behaviours and culture. It is all too easy to fall into the trap of thinking through solution lenses; just do this or change that. The role of the leaders is to give purpose and inspiration, let the teams or teams of teams solve the problems – they are much closer to the situation. Helping the exec level team to reach this point and acting like a single collective intelligence united in purpose and resolve is a difficult challenge and is why introducing an enterprise level Agile coach is usually a good option. Someone skilled in both group facilitation and personal professional coaching.

Pass to the teams to define solutions

Once you have this guiding direction then the focus starts to switch to the teams, the Top Down intent is passed to the teams below for their Bottom Up solutions.

The challenge posed is this: “Given this is where we are, and those are the values, behaviours, principles that we are looking for, how can we change our activities, our processes and practices?”

The important point here is that the solutions come from the teams, they are empowered to improve their own circumstances, don’t fall into the mistake of trying to solve the problems at the top, that is to impose and disempower.

And just as some enterprise level Agile coaches are a good catalyst for the exec, so experienced team Agile coaches can really help make the difference for teams in figuring out what are some sensible options.

Some teams may want to consider Scrum, others may want to just start with improving their deployment pipeline, or putting some more rigor around prototyping and testing product ideas before delivery. Many of these ideas will be within the control of the team. The easiest adjustments to make will change the way team members work between themselves, not impacting anyone else, and these are probably where they will start – following the path of least resistance.

Escalate constraints

Ideally within just a few weeks those teams should start to notice that their instance of the company’s metrics have ticked up, probably not a lot, but it is a start. Even the longest hikes are still taken one step at a time.

These small improvements are fed back up to senior leaders, more focus from a team, better user testing, incorporating more sustainability in the work, less work in progress. These first pieces of feedback are crucial in keeping the momentum; providing the leaders with the reassurance that they are onto a good thing and in turn, their clear and widespread response to this information emboldens the teams to greater efforts.

The low hanging fruit are soon picked however, and the path to greatness is rarely the easy one. Teams will start to identify constraints to their self improvement that lie outside of their team. Reporting requirements from a central PMO, conflicting demands from multiple departments, team members split between multiple teams. This is the point where the bottom up really meets the top down. These issues can’t be resolved by the team so they are escalated up where systemic solutions are sought. Experiments are discussed with teams and instigated from the exec level to address these issues. The fact that the teams have requested something to be done and are involved in the solutions will give a much higher level of engagement and positivity than would otherwise be the case had management just implemented these changes without consultation.

Improve to shorten cycle time

If you can achieve this, you now have a situation where the entire organisation is proactively moving in the same direction. The first changes will likely be around development practices and processes. Before long attention will turn to the elephant in the room that is the interface between engineering and product and about how we define value; this increases pressure on integrating Product management and design thinking. Ultimately these changes will trigger discussions about organisational design, governance and then financing and performance management.

In summary

Stepping back you will see a cycle of small changes and escalations from the bottom driving a clear purpose and systemic changes from the top, round and round and round again, better and more assured with each cycle. As with all improvement cycles the smaller the batch size the faster the feedback and the easier it is too keep things going, don’t attempt to change too much at once, but what you do change, do it quickly.

If you are serious about enabling business agility, then it isn’t where you are going that is the challenge, it is the path that you take. That journey is long hard and full of unknowns, and the most reliable approach to take in such situations is to instigate a cycle of continuous improvement where both the top and the bottom of the organisation work in a reinforcing harmony towards a clear and common goal.

Comments are welcomed – even criticism. It is only through feedback that we learn.

Follow me on twitter: @philagiledesign 

or reach me on LinkedIn here