Agile Development is definitely nothing new at OTTO – the term has long since reached Peak Hype. The first generation of Developers for whom 'Agile' is ancient history (or simply their regular working routine) are now almost Senior Devs. In brief, the term has lost its gloss and what we might now call 'Agile' is often a mix of Agile, Lean, XP and other working principles.
Quite a bit has already been written about our working approaches at http://otto.tech – but comparatively little about product development in cross-functional teams, and absolutely nada about 'triads'. So it's high time we changed that!
At the beginning of OTTO's Agile journey from 2011 onwards, all the 'Feature teams' (FTs) involved in the development of the Webshop were agile in the sense that they worked based on Scrum. Maybe not exactly according to textbook Scrum, but quite close. For a company that had previously driven traditional project management as defined by ITIL, with a pool of developers from which the project teams were recruited and with project durations of 18 months, this was already major progress.
To accompany the process we hired a consultancy that specialized in the introduction of Scrum. All teams were required to stick to THE PROCESS – in fact, they all had the exact same meeting sequence and schedule, from sprint planning right through to the retrospective.
This changed rapidly, as the teams were granted very far-reaching competencies at an early stage. In the target 'Lhotse' architecture, from the word go we distinguished between microarchitecture (for which the teams had full responsibility) and macroarchitecture (for which we established few fundamental rules).
After some discussion, the teams had adapted the processes to their own requirements latest by 2012.
These days we've long taken it for granted that methods like Scrum are tools, and that teams should be allowed to select the tools for their daily work themselves – and adapt these if required. Nevertheless, looking back I think it was a smart move that all of us, coming from a classic project organisation, shared a uniform working approach to begin with and only made adjustments when we'd understood well enough what Agile Development is all about.
“We are now in the “post-Agile” age. ‘Post’-something means that we are in an in-between period, where one thing has been incorporated into our culture and the next thing has not yet formed to the point of naming.“
-- Alistair Cockburn
Most teams today tend to work more in a Kanban system with the best of Scrum, i.e. daily standups and retrospectives.
What remain are the core values from the Agile world, enriched by certain elements from Lean, DevOps and XP:
Some teams naturally are further down the line in their journey than others and in some places you can certainly still see some cargo-cult Agile.
In a complex system, nobody can predict how a specific change will be received by the customer, so only constant testing on the customer will help. Of course, this entails that deviations from whatever the plan is become normal – and that far-reaching solution-level plans which stretch too far into the future make very little sense. Staying aware of the objectives is all the more important in preserving clarity on which parameters should be measured.
“Without data you are just another person with an opinion.“ -- W. E. Deming
In order to develop software further in a working approach like this, you need a diverse range of know-how. To be able to work at high speed in a perpetual cycle, you definitely need these skills in your team – otherwise you'll constantly be chewing your nails during the frequent waiting times in your feedback cycle.
Incidentally, the teams must also be permanently able to go into production independently of other teams in a largely automated manner, which is why many teams at OTTO now practice Continuous Deployment: practically every single 'git push' triggers a deployment pipeline and therefore leads to a go-live. Thousands of deployments per day are now complete normality.
In our company, this means that all Feature teams are cross-functional. Whether it's Frontend and Backend Development, Operations, Quality Assurance, Web Analysts, or User Experience (UX), all the expertise needed to develop and operate software is available within the team itself as often as possible.
Some roles are also shared. For example, Quality Assurance could be handled by 'the Developers' instead of by a single, dedicated Tester. In other cases (e.g. Analysts, UX), one expert is assigned to several teams. However, cross-team roles like this are often an unfortunate compromise: either they only participate in every second routine meeting, or they have two at the same time – and somehow they only partly 'belong' to the team. A clear 1:1 assignment would be better but sometimes it's impossible to implement this sensibly and we have to live with a compromise.
Work in a Feature team is therefore very diverse – which implies that every experienced person will come across things they still need to learn. Paradise for curious spirits!
Long gone are the 'bad old days' when a pure Development team threw a new chunk of software developed under time-pressure over the fence to an understandably risk-averse Operations team... or even argued with them about whether it's actually necessary to migrate to a newer version of the JDK again.
Instead, today all Feature teams also hold responsibility for operating 'their' software. Obviously, this expands the number of roles that a cross-functional team has to take on. Some teams have one or more 'Ops Specialists', whose main task is the operational topics, while in other teams the Developers share these tasks.
Since our entire infrastructure has now moved to the cloud, the boundaries between Dev and Ops are becoming increasingly blurred anyway. Infrastructure as Code sounds like a good thing (it is), but it still means that a team has to master an additional tools 'zoo'.
A common pattern is the daily assignment of operational topics to a 'pair' – pair programming is very common with us and pair operations, pair testing or simply 'pairing' is therefore normal in most teams. The so-called "Daily Action Pair" is responsible for everything that's difficult to plan, including critical bugs, analysing log files, looking at the metrics, improving the deployment pipelines, updating service dependencies, bringing the next Java version into production, improving the automation of standard tasks, being the person in charge for other teams' ad hoc requests – whatever's needed!
Nobody expects everyone in the team to be able to handle everything. It's enough if the overall team can complete all tasks, which means that the individual (even if this hurts) somehow becomes dispensable. DevOps is not a role, DevOps is a mindset.
The Feature teams at OTTO are therefore Agile, cross-functional teams that are end-to-end responsible to a very large degree for their product (see also Products over Projects by Siriam Narajan). Unlike in a project organisation, at OTTO we develop software intended to be used by and for our business for several years. Software like this is never 'finished' and the initial development is not sufficient. Someone has to take responsibility for operation, bug-fixing, monitoring and of course further development.
We try to define and delineate the responsibility of the teams by business criteria as far as possible. For instance, Team X is responsible for the shopping cart, Team Y for the product recommendations. The goal is to define the teams so that they can deliver changes to the customer in most situations without needing to involve another team. With more than 40 teams currently working in parallel, this is our biggest challenge right now.
Wherever possible we try to avoid technical delineations (e.g. Team X does the frontend, Team Y the backend) or delineations by roles (e.g. Team X develops, Team Y runs the software). This is because delineations like this almost always lead in practice to tightly coupled teams which are hardly able to take action without involving other teams.
A team's product responsibility needs to be extensive because the team will gather both technical and business expert knowledge that can't be gathered to the same extent outside the team.
At OTTO, among other things 'product responsibility in the team' means that product management – and therefore technical responsibility – is part of the team's mandate. For us, Feature teams are not pure Development teams made up of a bunch of techies. In each team, at least one person is responsible for the business roadmap and, as we will see in a moment, not in a secondary role, but in a decisive one.
In Scrum there's the outstandingly important role of the Product Owner. This role is generally placed outside the Development teams – so to speak as a proxy of the departments interfacing with the Development team – and it is in the hands of one person.
We found this very impractical.
If you're dealing with numerous teams and have to minimize their external dependencies (to avoid waiting times in Development – e.g. Team A can only continue with project X when Team B has completed something), the Product Owner role belongs in the teams. Only then can we avoid:
A further problem is that there are always different drivers for the required development efforts, at least business and technical factors. Differing competencies are also called for to assess the urgency of the specific development case: is it more important to build the next feature, or do we need to improve monitoring first? Both are important – so we must not only make the decision from a business perspective, and neither purely from a technical one.
Almost nobody is able to assess a fairly complex product from both a technical and a business perspective equally competently. It's also tremendously difficult to assign this role competently – so we decided not to attempt it! We split Product Ownership in each team into a business and a technical facet, with a Business Designer responsible for the business side and a Technical Designer for the technical side (cool labels are unfortunately not our strong point). Both have to agree on what the team will tackle next and neither can simply outvote the other – at least theoretically.
When we then add the Project Lead, Business Designers and Technical Designers form a 'triad'. This term is a little bizarre, as the Project Lead is neither responsible for a project (rather for a Product team or Feature team), nor are they the 'Lead'. They are simply responsible for processes and methods in the team – the Agile Coach where necessary, the team assistant, the communicator, the good soul of the team. We could also say that this role is the 'centre of lateral leadership responsibility'.
We would be smart to rename these roles, perhaps like this:
For us in the teams this no longer makes a difference, as every company has its own internal nomenclature. After a few years it's tough to get roles like this renamed anyhow – we might succeed in external communication at most.
No, this is not the path we're pursuing – some things you just have to leave behind! Cross-functional teams with product responsibility are large, and to match our teams' appetite we'd need XXL American pizzas and a team coming straight from the 'Elbe' (not the river but the OTTO 'Elbe' canteen).
However, our admittedly large teams do work in two groups: one group in the team fills the backlog pipeline, so to say, while the other group processes it. Smooth coordination is only possible because it is one team – but that still makes a team pretty big.
We've grown from 4 to well over 40 teams in around 10 years and yet we still work in the same team structure. You could call that a success story – and it is!
Nevertheless, we are slowly reaching the limits of growth. It's quite easy to imagine that overarching topics (business topics as well as technical ones) can be discussed and decided on quickly and unbureaucratically in a small group, while with 40 autonomous teams involved, discussing some questions takes a little longer. The transformation of otto.de into a marketplace also makes the situation trickier: there are now over 100 teams to coordinate, ranging from the Webshop to Logistics. Balancing alignment and autonomy is no small feat.
So it's quite possible that we will soon combine several teams from a single business domain into an overarching structure. This will let us work (both professionally and technically) to achieve greater decoupling between the business domains so that more items can be decided in a small group.
But it's also fully conceivable that we'll find a completely different solution. This journey never gets boring!