Successful Mountaineering – Reaching the Summit of LHOTSE with Agile Software Development

LHOTSE is the internal code name for our project to re-develop www.otto.de in-house, based on open source software and using agile methodologies such as Scrum and XP. LHOTSE went live on 24th October 2013 – three months before schedule. Although it took us roughly two years to develop, with a team of more than 100 experts, getting LHOTSE live was practically a „non-event“. So, why did everything work out so well? I believe a lot of it has to do with how we do agile software development. Let’s dig into some of the Principles behind the Agile Manifesto to find answers:

"Working software is the primary measure of progress."

We delivered working software right from the beginning. We had a basic working version of our online shop very early in the project. In 2012 this version was available to all employees on campus to test and use. In February 2013 we decided to release an „alpha“ version for selected customers (see this article on etailment.de and our first blog post on LHOTSE In May 2013 we released our „beta“ version to a wider range of customers. From then on we gradually increased the automatic distribution of customers from our old platform to our new platform, always measuring and testing if the new platform is keeping its promises concerning our main KPI s. For example, to test the performance and stability of LHOTSE, we diverted 80% of our real users on a high traffic day to the new platform and added another 40% of artificial traffic from load test generators. We made sure under way that our requirements were met. Finally, on 24th October 2013 we switched all our customers to LHOTSE. Unsurprisingly, this was only a small step.

"Business people and developers must work together daily throughout the project."

One of the major organisational changes we introduced with LHOTSE are our „functional teams“, aka feature teams. Each functional team covers a certain domain, e.g. product or order management, and has all skills needed to deliver working software: project leads, product managers, lead engineers, usability experts, designers, devs, ops, QA, etc. All these people work and sit together. Questions, e.g. between developers and business or QA can and will be answered in seconds. Our functional teams are also considerably stable. We do rotate and change team members, but we also try to keep teams stable so that people get to know each other and learn to work really well together.

"The most efficient and effective method of conveying information to and within a development team is face-to-face conversation."

Here’s a quote from one of my favourite songs:

„Technology made it easy for us to stay in touch while keeping a distance, 'til we just stayed distant and never touched. Now all we do is text too much.“ (Sage Francis – The Best Of Times)

Different context but certainly transferable. In LHOTSE the whole team of roughly 100 people sits together, in one building on one floor, reducing personal distance to a bare minimum of a few metres. Our internal developers as well as freelancers and other companies we work with, are required to be physically on campus, at OTTO, in Hamburg. Working from home is possible but it’s the exception not the rule. Having everybody in talking or walking distance makes decision making much easier, increases reaction times and team spirit. When talking about conveying information face-to-face, we also pay attention to team sizes. The ideal number of software developers in a team is 6 – my personal opinion. Less can make things tough if people are on holiday or on sick leave. More team members exponentially increases the communication path between team members and inverts the advantage to a disadvantage.

"Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter time scale."

Our sprint length is two weeks. That’s the maximum time frame before we deliver a new product increment. And that’s where we started two years ago. Since then, we've considerably increased our ability to deliver software into production. On 24th October 2013, besides going live, we subsequently pushed 7 deployments live. That’s roughly one live deployment per hour. Quite a step, considering the monthly „feature releases“ we were used to and the two-week increments we started with. So, what in terms of software development did we do for that? Here’s ourrecipe for velocity:

• Make sure you pay attention to quality. In the project management triangle of time, scope and budget, do not adjust quality, make quality a requirement.
• Use the patterns and methods summed up under the term Continuous Delivery. Check our blog posts on Continuous Delivery with Feature Toggles and Blue-Green-Deployment for details on how we did it.
• Encourage pair programming to spread knowledge and find bugs when you make them, not days or weeks later. Read our blog post on  14 friends of pair programming to find out more about how we learn and promote pair programming.
• Introduce TDD/BDD to make sure what you've built is right - and that it stays like that! Our code base is almost equally distributed between (unit-) test code and production code.
• Have dedicated QA experts in every team: to work with business and development, to get acceptance criteria and tests as early as story-writing, find bugs and ensure the definition of done is achieved.
• Introduce a zero-bug policy to keep the software clean. In the short run, this will make you slower, because, a minor bug will be treated with higher priority than a major user story. In the long run, this makes you quicker as you don’t build up bug-backlogs with extra overhead such as „bug-fixing-teams“. You can concentrate on permanent feature development.

All these measures give us a lot of security that we can go fast without making a mess!

"Agile processes promote sustainable developent. The sponsors, developers, and users should be able to maintain a constant pace indefinitely."

Did I mention that we've just finished sprint no. 56? Did I mention that going live was practically a “non-event”? Can you imagine what we are doing now. Correct: sprint no. 57. Constant pace is about two things. First, follow our recipe for velocity mentioned above. Second, don't try to reach your sprint goal by staying at work until 3am every two weeks before the end of the sprint. Constantly measure your average velocity of the past 5 sprints. Use the average velocity to forecast what you will be able to deliver next sprint and to plan ahead. Once the team is up and running, keep the velocity constant. If that's not enough, consider adding people to the team, but keep Brooks's Law in mind. If that's still not enough, consider setting up a further team, but make sure to do that early enough in the process to actually make that effective. And, last but not least, find those pain-points in your software and your process that make you slow. A good Scrum team will be quiet at the end of the sprint, working normally – not frantically bug fixing. And that will make people enjoy their work for a long time without needing a long recovery from a chaotic project. And that's what I would call constant pace.

BTW: I was wondering if “sprint” is the right word to use in this context?

"The best architectures, requirements, and designs emerge from self-organizing teams."

If you want to find out more about the architecture of LHOTSE, check out our previous blog post or our article published in OBJEKTspektrum In a nutshell, the main concept behind the architecture of LHOTSE and how we put business domains, corporate organisation and software architecture together, is “divide and conquer”. A very old and very new and very “en-vogue” concept – and a very powerful one. So how does that fit the principle heading this paragraph? Well, sort of semi. On the one hand we have a macro architecture designed up-front and imposed on all teams. Does not sound agile? OK, here's the pros: The macro architecture is very basic, very short and mainly focusses on loose coupling to make software modules, business domains and teams independent from each other. That's the law. That's not discussable. For each independent business domain and software module, however, there's maximum freedom. That's our micro architecture: Teams can chose their programming language, their data base system, their frameworks, their coding-style and design patterns. And why? Because the teams know best how to fit business and software together. OK, to be honest, we do have a rule that requires common base technologies, e.g. Java as programming language, git for version control, MongoDB as database, etc. We don't deliberatly make things as diverse and complicated as possible just for the sake of it. However, the point here is, that this is discussable.

Is that it?

That's it... at least from me in this post. Perhaps you've noticed that I only covered half of the Principles behind the Agile Manifesto. That's deliberate. It leaves room for further improvement. It leaves room for a sequel to this blog post. And it also symbolizes that it takes much more to make a project like LHOTSE successful.