TECH BLOG - Introduction Part 2: CI/CD Patterns and Best Practices

Continuous Integration is not a tool

I found the next quote somewhere on the internet and I think it's spot on:

Here is some bad news:

• Jenkins is not CI.
• JUnit is not unit testing.
• Jira is not user story management.

With this in mind, Continuous Integration is a practice, not a tool. There is this really good blog post from 2005 which is still relevant and it underlines the message perfectly: http://www.jamesshore.com/Blog/Continuous-Integration-on-a-Dollar-a-Day.html.

If you want to do CI right, the developers need to understand that CI is all about integrating and testing code regularly. There are a few best practices that can help to start:

• With a few exceptions, all developers should work directly on the master branch. By doing so, every check-in leads to immediate integration and testing. Developers used to work on feature branches - but then the integration doesn't happen on check-in anymore, but only when merging the changes to the master. Feature branches of course can be used, but then they should be merged to the master on daily bases. Here is a nice post regarding branches and CI: https://martinfowler.com/bliki/FeatureBranch.html

• There should be a clear and simple CI process for the developers to follow: The process describes the steps that need to be done when pushing code to the master. For instance you should make an update first and check, whether a build is running already. After the check-in, you should watch the build and first deployment and take care that you haven't destroyed something. A 7-steps process is described in the continuous delivery book: https://martinfowler.com/books/continuousDelivery...

• Be sure that the build and deployments don't take too long - this is a show stopper for CI because if something is taking a lot of time and is tedious, people tend to avoid it. And it would waste the time of the developers. This can be very complicated to achieve, but there are various posts that cover this topic. Overall you have to track each step, pick the ones that take the most time and continuously improve them.

• Take care of code quality checks and automated tests early on. When starting with a new project, it can be persuasive to simply start to develop features as fast as possible. It will impress the stakeholders and it is a fun thing to do. But ignoring tests in the beginning will soon lead to problems. Once the application has a certain size and complexity, adding test automation to the mix is inevitable. You'll have to do the work anyways, but if you are starting late it only gets more painful. So create tests from day 1 and add them to the CI pipeline. Same thing is true for code quality checks - the sooner you start the better.

Quick Feedback vs. Production-like Environment

We'll cover the deployment pipeline in greater detail later on - but for now it is important to understand what it does and what it is good for. Here is a suitable definition from devops.com:

The (deployment) pipeline breaks down the software delivery process into stages. Each stage is aimed at verifying the quality of new features from a different angle, to validate the new functionality and to prevent errors from affecting your users.

Here is a chart showing a basic deployment pipeline, which can be a good starting point for medium complex projects:

As you can see, a software package will pass through multiple stages before it arrives in production. Each stage has a particular reason, which we will discuss in one of my future blog posts. Without discussing in detail what happens at the stages, there are two important things to know and to consider:

1. Speed is crucial at the beginning of the pipeline. So a build on the local development environment should be finished within 5 minutes. The commit stage which builds the deliverable and runs the unit tests again on an independent machine should be finished in under 10 minutes while 5 minutes or less are preferable. This is important because the developer needs to get immediate feedback here. He should even wait to see whether his check-in works or broke something. If the first stages take too long, it endangers the whole process. Later on, the deployment and tests will take longer since the tests become more and more complex and the integration increases.
2. The closer the pipeline is to the production environment, the more production-like the environments need to be. This is important since it reduces the risk to fail when deploying to production. Starting with individual local development environments, each stage will increase the integration, add more components and will run more complex tests (see the testing pyramid in the previous post). In best case, the last stage before production is (almost) similar to the production environment. Because if the deployments and tests work there, then you'll have a very high probability that the deployment on production environment will be fine as well.

Automate as much as possible

There is a direct relation between the level of automation (or the amount of manual tasks) and business success. It is astonishing. The Puppet DevOps report describes the relation in more detail here: https://puppet.com/resources/whitepaper/state-of-devops-report. Companies that are high-performers when it comes to automation will twice as likely reach their goals than average performers or low performers. High performers automate more than 75% of all build and deployment tasks and reduce the number of manual tasks significantly.

There is a basic rule for automating tasks in the IT: When you have to do it twice, start thinking about automating it. When you need to do it a third time, automate it.

Automation should be introduced at least in these areas:

• Build and Deployment (obviously)
• Testing (of course), Code Quality Checks and Reporting
• Setup of Build Tools
• Setup of Environments to run the application
• Setup of Development environment

The difficulties when implementing automation is that it always competes with implementing new features. Therefore it is not advisable to spent too much time at once to automate manual tasks since it will reduce the team capacity to implement features with business value. It is better to continuously increase the level of automation, starting with the most important. So usually we try to add CI/CD tasks or general automation tasks to each sprint.

Something is broken? Fix it! Now!

This is an obvious thing but it is also the one rule that gets ignored far too often. The most important thing when it comes to CI/CD is that it's always in a working state. So whenever a part of the process fails, someone (the one that produced the failure would be best) needs to take care of it immediately. When your check-in breaks the build, fix it or revert it. If test execution fails, immediately review the failure and fix either the test or the tested code.

Try to run all your CI/CD steps as often as possible (multiple times a day would be best). First, if you do something often and regularly, it becomes easier. Second, if you integrate on a daily base, the changes that might cause errors become small and easy to understand which makes it easier to fix issues. On the other hand, if you wait too long and check-in a huge amount of code at once, it will become very hard to locate issues if errors arise.

Start measuring your delivery processes

You can't control what you can't measure - Tom DeMarcos famous quote is true for your delivery process as it is true for everything else. If you start to introduce CI/CD to your delivery process, how do you know if it is successful and whether it is a good investment?

So before you start, define some KPIs, measure them and define target values for them as a result of the improvements due to CI/CD. Doing so gives you the possiblity to measure your success and to prove the importance of your work to the senior management. Let me give you an quick overview of the most important KPIs for Software Development (we'll skip Quality Metrics here since it is a huge topic on its own):

Lead and Cycle Time

Lead and Cycle Time are the most imporant KPIs when it comes to measure the delivery process - it is all about how long it takes to deliver a feature to the customers. Here is the definition we use (there are many different available...):

Lead Time: The Lead time clock starts when the request is made and ends at delivery.

Cycle Time: The Cycle time clock starts when work begins on a feature request and ends when the item is ready for delivery.

When it comes to CI/CD only, the cycle time is pretty interesting. Also if your company has already established some kind of Delivery Pipeline, it can be used to measure the cycle time pretty good. Aside from that, tools like Jira can help to measure the time it takes for a story from the first definition to release in production. Most likely you already have this information hidden somewhere in the tools you use.

Team Velocity

A classic agile metric - the velocity of a team. Let's start with the obvious: you cannot compare the velocity of one team with another. It really doesn't work so don't try it. But what is possible is to measure the trend of a teams velocity. At first, while introducing CI/CD, it will have a negative impact on the velocity since developers have to work on automating manual tasks. But in total, CI/CD will increase the speed of your team significantly and Velocity is a nice KPI to show this increase.

Build and Deployment Metrics

When introducing CI/CD, these metrics are important to measure the success of the initiative itself. Because CI/CD will have an immediate and big impact on build and deployment related metrics. So measure them when you start, set a goal and try to move towards it step-by-step. Interesting KPIs are:

• Number of builds per day
• Number of build failures per day
• Duration of build, including automated tests
• Duration per deployment stage

Certainly, there is much more to discuss here, for instance whether it is possible to save money by introducing CI/CD. I might write a separate blog post about Software Delivery KPIs and Metrics, but for now, if you are interested to talk more about this topic, just let me know.

About the author:

Walter Pindhofer

Chief Solution Architect / Delivery Manager

In his role as CSA, Walter is responsible for designing and implementing fitting solutions for well-known enterprise companies. As a huge fan of Continuous Improvement, Walter is currently focusing his research on DevOps, CI/CD and agile methods.

Make an appointment for a consultation!