Agile Manifesto – the Dark Side – Part 1 of 4

The Agile Manifesto has been around a while now, although looking at some Agile governed projects you’d think nobody had read it. However there are plenty of fallacies in the industry stemming from misapplication of the manifesto.

The Manifesto has statements that recommend focus on one side of four paradigms, however it does not state that the “other end” of the scale is not necessary or intrinsically bad; the term used is “ while there is value in X…” Too often that has been misinterpreted as “Agile doesn’t include X”.

I’d like to take each one in turn and discuss just how much of the Dark Side is appropriate. So starting at the top:

Individuals and interactions over processes and tools

Just how much process and tooling is suitable?

This is a very hot topic on the forums currently as Agile becomes increasingly mainstream and more companies want to get the benefits. Like an onion, the outer layer of terminology, governance and processes are the most visible, and hence can be the easiest to adopt, like wearing a costume. That typically leads to a focus on processes and tools over individuals and interactions and is the single most common failing in Agile projects. However – assuming you have avoided this pitfall, what processes and tools are sensible? (I am only going to refer to those that can be used instead of human interactions as per the manifesto, so Development tools / frameworks, most of which are very beneficial in the right context, are not discussed)

It is fairly well agreed that for effective Agile delivery then you are going to need some form of continuous deployment and that is only sustainable with some manner of supporting automated test suite. Beyond that I’d argue that all tooling should be brought in deliberately due to a defined need, rather than an assumed starting tool kit.

So do we need one of those shiny Agile tracking tools, like JIRA or Version One etc? Instead of assuming you do, try assuming you don’t and then establish why you do, and then specifically what elements of the tool you really need, rather than adopting it blindly wholesale. Distributed teams are usually one of the main drivers – also one of the biggest challenges to team interactions.

Scrum, and even Kanban, come with their own processes, do we need them? I’d argue that Scrum without underlying XP practices is likely to be a lot less successful than the other way round but the original concept in Scrum, for example, three roles, four meetings is a sensible starting point.

Scrum (and Kanban) is a framework, so customising it to your needs is expected. Many teams will also have extra meetings, backlog refinement is a good case, or their own way to run retrospectives, or organise sprint planning. These are not to be shot down in flames with the cry of “Processes are Evil”, because in the majority of cases these are ways of working that the team has consciously adopted to simplify their activities, they might have even made them up. There is a significant difference between an externally defined process applied to a team and an internally customised one adopted from within. Rules (and by extension processes) enable action without full understanding, and consequently stifle innovation. Encourage teams to regularly challenge their processes to ensure that they still support activities rather than define them. If you are going to have a process, then make sure the team owns it, and therefore is free to change it.

Agreeing on some visual representation of the current state of work is nearly always a necessary step – so a task board or a burn down, some manner of backlog etc. It is often helpful to agree some mutual standards if working in a multi team environment – but each team should be free to customise their ways of working independently to a degree.

Processes are often the consequence of a lack of relationships, as the personal relationships between people become strained then processes are often brought in to compensate. The relationships can be strained either through failing trust or often due to scale. The growth pain organisations feel is usually a result of the shift from relationship and trust based operations to one of processes. With good relationships within your team and with people around them, then the need for processes will be reduced, so to identify where process is necessary I’d approach from the other side and ask where and why the relationships are poor?

Unless the whole organisation is Agile minded and relatively small, you are going to need to apply processes at the interfaces into poor relationships. That could be some structured status report into a nameless faceless (at least from the perspective of a lowly team developer) steering committee, or maybe an agreed set of steps into an inflexible established deployment team.

A consequence of the association between relationships and processes is that those team members that are not fully embedded in teams usually have more processes around their work than those in teams. So Architects might have review sessions, UX might need their own schedule of work and SLAs to deliver against etc. Again these aren’t intrinsically bad, but need to be agreed as a compensation for the lack of relationship rather than an opportunity to reinforce it.

In summary:

• Assume no processes and then adopt them as and when the team needs them, not by default
• Own your processes, rather than have them applied by central command and control
• A basic framework to show work and to harmonise group time is usually necessary
• Accept that for teams short on skills or experience, some guideline rules are going to be beneficial, but encourage the teams to challenge them frequently as their skills and experience improve
• Consider all the relationships inside and outside the team. Wherever there are issues – either due to building trust or maintain good communications then consider adding a process
• Processes compensate for lack of relationship, only adopt to fill the gap and resist going further as you may reinforce the lack of relationship

Next up – just how much Documentation is sensible in Agile delivery….

Comments welcome

#philagiledesign

Agile needs to be Business Strategy not IT Strategy

In my last posting I wrote about my frustrations with large projects dressed up to look Agile but without delivering WORKING SOFTWARE, so just compiling risk.

It is very easy to point and throw stones, but what can we actually do about this – and why do we often see Scrum governance at the build phase working within a waterfall implementation phase? Why do we get Water-Scrum-fall?

A place to start would be to observe where this behaviour is most likely to manifest. From my own person experience – and I’d love to get comments from others with differing experience – I see this most frequently in large companies whose main business has IT as an enabler, not as a main product or sales channel. Or basically, IT is not central to business strategy.

So I would suggest to address the issue of failing Agile initiatives / poor adoption of Agile, is to delve deeper into that business strategy, to understand how Agile principles can be incorporated into business leadership.

A common theme at Agile conferences at the moment is corporate culture and examples of Agile principles being applied outside of software engineering, and both of these are tapping away for the same reason, successful Agile adoption requires organisational cultural change and adoption of the principles throughout all departments – not just IT.

I would suggest that the first step would be to express the corporate Target Operating Model in Agile terms – explain how their roadmap can be iteratively achieved considering all elements, people, process and technology. Instead of feature driven end state targets, we should switch to Goal Orientated roadmaps * showing incremental value not a fixation on solutions we can’t be confident in.

* see http://www.romanpichler.com/tools/product-roadmap/ for details

People and Process will need to come before technology in any cultural change. People that value small steps, empowered teams and are customer centric. Processes that enable a safe to fail environment – and don’t mandate stage gate delivery. These will be seismic changes to many organisations – and would require exec level support to take root.

But how is this going to happen:

I would suggest that the Agile movement stops preaching to the choir and starts talking to the real power breakers. Getting project managers, architects and developers together is reinforcing something which is widely supported, but where it is really needed is at the executive level – and there is far too little genuine understanding of the cultural (rather than process) side of Agile behaviours with that group. We should start inviting these people to our arena and tailing our content accordingly. Produce collateral and case studies for IT departments to submit to their senior leadership.

The other route, probably more realistic, would be to focus not on the existing knowledge leaders in the Agile movement, but to those that already have the networks and trust with the necessary audience – the top management consultancies. I am pleased to see that this message is starting to get through as Delotte, McKinseys and the Boston Consulting Group have all set up Digital agency wings, mostly relatively recently, but I can’t help thinking that slightly misses the point – it once again puts Agile Principles in an IT box.

Agile shouldn’t be IT strategy, it needs to be Business Strategy.

Big used to eat small, now, fast eats slow, and Agile is fast. If we can bring the partners of the top consultancies to this line of thinking – then with top down influence we might start seeing some changes.

Comments welcome

#philagiledesign

Scrum governance at the build phase – that is NOT Agile

Agile delivery is very much in vogue, referred to as the latest must-have fashion accessory for your IT department, but far too many organisations are treating just like that, as if it is something that can be bought and installed, like a new shiny server with flashing lights that everyone can look at and say “Oooooh”. Having seen it, the business then drifts away and leave the infrastructure dept to do their thing and we all go back to doing what we did before that moment of excitement.

If you have a small project then it is reasonably credible to have a crack team work with the business to deliver something quickly, try it out and then improve on it. The delivery is small, the budget is probably similarly small and the whole initiative flies pretty low on the governance radar. This is where Agile started and excels.

Now think about a larger project, a more significant technological improvement or something with significant business impact. Now before this gets to any of those techie code writers, this initiative will be thought about… Target operating models refined, strategies penned, there might be process flows, visions, objectives and benefit realisation plans created. This work is vital, most of it falls under the Management Consulting area and for most companies this is taking place outside of the IT arena and therefore away from the shiny new Agile toy.

The business – having done all their thinking then speaks to the IT department and explains the wonderful plan, “we’ve done all the strategy work, you build it and then we’ll implement it”.

“But we’re Agile” the IT department responds, “In Agile we work out the requirements as we go and”….

“Yes but we’ve done most of that already – we’re not going to ask you to define all the work up front because we now know you don’t do that any more with your new shiny Agile, but we go live at the end of the year so work towards that, now off you go”.

At this point there is either a big fight and the company rethinks and there is a fundamental culture change to take Agile principles into the heart of business strategy or, which is far more likely, you have IT setting up an ‘Agile’ project with pretty fixed deliverables and a defined implementation date a long way in the future.

Because the scope of the work is probably significant, and the deliverables reasonably well understood there is a temptation to start large, multiple teams, maybe straight into one of those much misunderstood scaled frameworks. Teams are working away delivering chunks of software under Scrum governance which are connected together in some ever growing test environment, desperately trying to get through enough functionality before the predefined implementation date.

Because this is so realistic for large organisations it is very easy to see this as normal and hence perfectly acceptable, we are lulled along the path comfortable in what is familiar.

THIS IS NOT AGILE. THIS IS WATERFALL

• The requirements have been roughly designed up front
• You are now building it against a deadline
• Then you are going to perform some kind of holistic testing at the end (unit testing continually but end to end user journey / business process lifecycle testing and performance testing often at the end)
• A pile of tests build up for the end, because with no users there is no urgency to fix them
• Then you are going to implement it at the end– with associated Change Management

What in that is Agile? – well you have what looks like Scrum governance (or Kanban or similar) during that build phase. So there might be sprints and burndowns and backlogs, but take a step back. Go back to first principles:

DO YOU HAVE WORKING SOFTWARE? – NO.

This is what I refer to as AGILE AT THE BUILD PHASE.

This is a very dangerous place to be and the issue is one of risk. IT projects are de-risked in one of two ways.

• Work out everything up front, consider all of the options and then don’t change anything
• Only deliver a little bit, check it works in the real world and then improve it

‘Agile at the build phase’ skips the detailed requirements gathering but also has a traditional waterfall implementation big bang end so you don’t get the continual real work assurance of the iterative implementation either – basically you carry all the risk of the project from start to finish with little mitigation.

Not all risks become issues, maybe the software does work – hurrah – and you get away with it, but sometimes it doesn’t, some integration fails, some data migration is not as expected or the system does what it was designed to do but not what is really needed – and now you have a massive problem – missed dates, huge loss of face and the possibility of a complete write off. It happens, I have witnessed it, and if you haven’t just Google Agile failures and look to see the number of agile builds with few deployments.

Agile isn’t something you do, it isn’t a process and it isn’t limited to the IT department. Agile is a culture, a principle, a set of behaviours. It is something that an organisation IS.

For an organisation to embrace the benefits of Agile then it needs to manifest those principles inside the earliest discussions and right through an iterative delivery – not just leave it as a shiny toy for IT.

Credit to Dan North for the underlying theme of Principles over Process.

Comments welcome, #philagiledesign on twitter.

At the very least, if you start down an Agile delivery then at least change the release plan to deliver something early, and by deliver I mean into production to a real user, it is the wait to the end implementation plan that will kill you.

Offshoring is only cheaper if you can afford the mitigation

It is increasingly common to have software teams outsourced to various parts of the globe, could be to find the skills or more often based on the principle of exploiting the differing costs of living. Aside from the moral implications there, how effective is this really, especially in an agile context?

Firstly I suggest splitting down the concept of Distributed, I suggest three categories:

• Just round the corner – same culture, very close to the rest of the programme team but not co-located. Could be East London to West London, I have actually seen this being separate floors in the same building!
• Nearshore – similar culture, far enough away to make visits awkward but not prohibitive
• Offshore – Different Culture, typically a long haul flight away.

Agile is light on defined algorithm style instructions, it is, by definition, an emergent philosophy and hence requires continual communication. If you have a distributed operation then steps need to be taken to compensate for the communication issues – and those steps can be expensive.

There are two barriers to communication relevant in this context:

• Distance
• Culture

Regarding Distance, the best communication is face to face – co-located, because of the ability to pick up on the tone, body language and facial expressions that moderate and give subconscious context to the actual words. Put people over video conference they fail to see the distractions of the others and when teams talk it is as if we are all standing 5m away from each other. People also pay less attention to the speaker when on VC. Remove the screen so it is just a voice call and you lose huge amounts of information, take it one step further and have instant messenger and the value is considerably lower still. The least effective form of communication is obviously the written word, the only advantage it has is permanence, which in turn, for Agile, is dangerous as the content is static in a contextually changing world.

Communication has a long history of cultural related issues, it is very easy to misconstrue, incorrectly assume meaning and even cause offence due to cultural misalignment. There is no right or wrong, there is just similar and different.

The Dutch psychologist Geert Hofstede categorised national cultures according to 5 dimensions: Power Distance, Individualism, Uncertainty Avoidance, Masculinity and Long Term Orientation. Of these, in my own experience it is the difference in Power Distance that has generated the greatest issues in terms of Agile software development. Many countries popular for offshoring in Asia have very high Power Distance, which manifests as deference to hierarchy. The Agile approach encourages a safe trusting collaborative environment where hierarchy is typically flat.

So if you are considering (or already have) a distributed team consider distance and culture and what steps you will take to mitigate.

For team that are close consider arranging face to face sessions rather than video conferences, and be aware of how much this costs. When working with teams in London, Zurich and Kiev we’d have collocated release planning each three sprints (6 wks), that is moving 20 or so people a few thousand miles for a few days, 9 times a year – those costs rack up fast.

Teams that are very far away – I would suggest you’ll have to take a hit on productivity – just accept that it isn’t going to work as well, more time though means unhappy stakeholders and of course higher costs.

Gaining cultural understanding is far harder, can’t be fixed with an airline ticket. Regular contact will help, consider having people of that culture in your local team and visa versa – I have seen projects with senior local staff seconded over to the offshore location as a cultural liaison role. Arrange cultural awareness sessions with your teams to try to make the teams more aware.

They key to this is awareness. Be aware that your communication and hence your effectiveness is going to be significantly impacted. Awareness brings mitigation strategies and hopefully budgetary / timeline impacts.

Looking back on the Projects I have worked on, if you are just around the corner – then move to be collocated. The team are being forced to accept significant constraints usually because of local politics or logistical issues because those with the power to fix the situation refuse to believe the issues or just don’t care.

If you are near shoring, then you need a very large travel budget and plenty of good quality video conferencing facilities with dedicated rooms and good connectivity at both ends.

Finally if you choose to go down the offshoring route, then in addition to the above you’ll probably need permanent secondees in both directions and a continual cultural awareness programme running in the background- and you’ll probably still need to accept a slower delivery rate.

If you can do all that and still save money then maybe it is worth it after all.