We all have to start
Every high-profile software house started somewhere, in the middle of nothing (in a garage?) and with a code-base not so good.
Amazon was a monolithic system that in 2002 started a very smart process of re-engineering that lasted years. LinkedIn underwent a 2-monts full stop of feature development in 2011 to restructure the code-base. Now they are the example to follow. How is it possible? Is it all because they have money? Maybe but that’s not the entire truth.
The unicorns among the software companies fought their way to the excellence they are. Nothing was easy or free. It’s sheer hard work and application of good engineering practices every single day and a lot of courage and commitment.
If you work in small software company, like me, and you’re stuck with bad habits and lots of the typical pains of the software world, the good news is that everything can change. Your team, your company can change.
DevOps is a set of practices (not only tools!) that aims to apply in the software world the lean concepts of the manufacturing world.
At the heart of DevOps there are the Three Ways:
- The principles of Flow, which accelerate the delivery of work from Development to Operations to our customers;
- The principles of Feedback, which enable us to create ever safer system of work
- The principles of Continual Learning and experimentation, which foster a high-trust culture and a scientific approach to organizational improvement as part of our daily work.
If you follow these Ways change will happen. I’m not saying that’s easy. You’re going to face every kind of difficulty: from the technical to the psychological point of view. Your team has to strive every day to make this change happen. What you have to do is to commit to the Three Ways with small changes that provide a feedback quickly (think about days or weeks, not months) over and over again. This process will never stop, your team will be always improving in some aspects of the daily work.
DevOps @ MadLab
At MadLab we are a small team of 6 developers. At the beginning of 2016 we were 2. This big change exposed all our flaws with no mercy: from how we accepted and started work to the delivery of the applications to the customers. But this was a great opportunity to understand that we had to change. In that period I was reading the book The Phoenix Project and that was the key. I urge you to buy that book right now.
How the three ways are helpful to the MadLab team?
- The first way
1.1 Visualize the work
The first principle says we need to accelerate the work from development to operation and then to our customers. But what is work in the abstract work of the software development? Work is every activity we can think of: bug fix, development of new feature, index tuning of database, refactoring,… To accelerate work we need to see the work and to see to work we need to transform something invisible/abstract into something to visualize.
The first thing we did was to make the work visible. We configured our issue tracking software (JIRA) with a kanban board. Every unit of work is represented as a card in a virtual board. This enable us to see the amount of work we have to do in a visual way.
The kanban board enabled us to realize how much work there was in the system and the first reaction was: “Woah! All this work to do? How can we manage all this!?”.
This was the first big-little change that happened in our team.
The WIP limit
To accomplish the goals of the first way we also started to apply the golden rule of a kanban board: the WIP limit. To accelerate the flow from left to right we have to limit the work in progress. If you have less WIP the quality will go up, if the quality is higher the work will have less defects and work with less defect is likely to not come back (a feature poorly implemented comes back with a bug, for example). If work doesn’t come back the flow is improved.
You can observe how we did not make changes to the codebase, to our development tools or our roles in the team. Jobs descriptions and responsibilities are the same, too.
Tech/automation tools and practices
Other practices to improve the flow need tools and automation. We over time implemented continuous build, integration and then deployment practices. These implementations happened in the span of about 18 months and we’re still working on it to constantly improve.
The second way
The second way enables the constant flow of feedback from right to left in the stages of our value stream. It requires that we amplify feedback to prevent problems from happening again, or enable faster detection.
At MadLab we follow the second way trying to keep the quality closer to the source. After that a developer completes the assigned work he/she creates a pull-request. This unit of work has to be reviewed from someone else and be approved before it goes back to the master/trunk stream of development. This is a peer-review because the work is validated by a member of the team and not someone higher in the company hierarchy. With this particular activity we have lots of room for improvement but we’re working on it. DevOps is a journey that will never ends and we’ll always find something to improve. The important thing is to realize that you cannot achieve all at once but improvement will come over time.
To keep the quality close to the source we also apply the rule that whoever breaks a build is responsible to fix it as soon as possible. This way we prevent bad code to be pulled from someone else or to spread in other branches of development. Sometime we also do pair-programming that is an extreme implementation of the peer-review.
The advantage of pair programming is its gripping immediacy: it is impossible to ignore the reviewer when he or she is sitting right next to you. Most people will passively opt out if given the choice. With pair programming, that’s not possible. Each half of the pair has to understand the code, right then and there, as it’s being written. Pairing may be invasive, but it can also force a level of communication that you’d otherwise never achieve. – Jeff “Coding Horror” Atwood
The third way
The third way eanbles the creation of a high-trust culture that supports a dynamic, disciplined, and scientific approach to experimentation and risk-taking, facilitating the creation of organizational learning, both form our success and failers.
At MadLab we follow the third way trying to create a positive environment where errors are not occasions to blame someone but occasions to understand what happened and how. We try to be transparent with each other and to trust that each member of the team is working as best as he/she can. We encourage creativity: everyone can suggest ideas to improve our way of work and every idea is discussed together. We reserve time every Monday, about two hours, to decide what we need to improve, how, and to review previous decisions to eventually stop changes that weren’t so good. We strive to improve something every two weeks: it can be a small thing (better comments on commits) or something bigger (let’s start with mandatory peer-review).
Change can happen and the best way to do it is with evolutionary steps: some big, and some small. DevOps is spreading fast because it is based on the successful principles and lessons learned from manufacturing, high-reliability organization, high-trust management models, and other that have brought us to the state of the art we know today. DevOps is the outcome of applying the most trusted principles from the domain of physical manufacturing and leadership to the IT value stream (Lean, Theory Of Constraint, Toyota Production System and many others).
Based on my little experience with DevOps I can only encourage you to all try to apply the Three Ways with your team.
The Phoenix Project