The next generation of continuous integration -




Main menu Articles




The Open Org

The next generation of continuous integration (/users/flepied) 16 Dec 2015 | Frédéric Lepied (Red Hat) (/users/flepied)

| 201


Image by :

Traditional continuous integration (CI) systems are designed as a pipeline of jobs. You have a peer review, then the build job, then the unit tests job, then the integration tests job, then the performance tests jobs, and so on. Each job is triggered by the successful completion of the previous job, and the first job is triggered by a change in the version control source code files. Of course, it can be more complex if you are targeting multiple binary platforms or if you are building a set of components to be able to test your full application.

But what happens when a job fails? According to Jez Humble and David Farley in the reference book Continuous Delivery (, you have to first follow this rule: "Don't Check in on a Broken Build." In other words, don't make it worse by pushing other changes. If you do, you won't be able to know the cause of the breakage. Humble and Farley suggest using one of these two strategies to handle breakages: 1. "Never go home on a broken build," meaning everyone on the team needs to stop working on their current tasks and switch to fixing the problem. 2. "Always be prepared to revert to the previous revision." Reverting can also be a strategy to avoid blocking the whole team. Of course, it could also be a mix: timebox the attempt to fix and then revert if outside of the allowed time. Another way to mitigate this issue is to use an integration branch to merge the modifications only once the integration branch is green (all the tests are OK). With this tactic you have the same problem on the integration branch, but your master branch is always in a usable state. This could work in small team when you can stop all the teammates from committing but even in small teams, this process often results in a CI that is red for long periods of time. You need to enforce a good discipline to succeed with this kind of CI or you can switch to the new way to do CI.

Next-generation CI Current-generation CI is centered around the CI server. Changes are detected by the CI server and triggering jobs. In next-generation CI, the system is centered around the code review system to be able to do actions before a modification is merged into the versioning system.

The decision of whether or not to implement a teammate code review process is up to you. I really recommend doing it to improve the quality of your code, but that's orthogonal to the CI system. Anyway, the important thing to understand is that the triggering of builds and tests are done by a new submission to the code review system. And once all the tests are green, the merge of the code to the master branch happens. This way, your master branch is always green and your developers can work in parallel on their modifications. This new CI system allows you to scale your automation effort and be fluid, as there are no longer blocking issues.

OpenStack workflow This new approach to CI has been implemented at scale in the OpenStack project to manage the CI of all the different sub-projects. To give you an idea of the scale, every day OpenStack handles 1,000 proposed patch sets, 7,500 posted comments and votes on Gerrit (, 16,000 test environments spawned, and 250 changes merged (source ( To implement this next-generation CI system, the OpenStack project uses the following components: Gerrit (, the code review and git repository manager. Zuul (, the git source repositories gating system. Jenkins (, the continuous integration server. Nodepool (, intelligent Jenkins slave provisioning on OpenStack clouds. These tools allow for parallel tests of the same project by using speculative merging strategies. If multiple reviews for the same project arise at the same time, Zuul is able to to test them in parallel by stacking them in a speculative manner. For example, if the reviews were named A, B, and C, Zuul will test A, A+B, and A+B+C in parallel. If they all succeed, it will have been the same as if A was tested and merged, then B was tested on top of the branch (A) and then merged, and the same for C on top of A+B. This speeds up the process a lot when you have multiple contributors to the same project. Zuul is also able to manage cross-project dependencies, allowing the merging of reviews between repositories. That's key in a git world where your components live in different git repositories.

Try it yourself For big teams or even for small teams, you can benefit from this workflow for your project by configuring the previously described components. There are puppet modules that can be used to configure these services easily. Another way is to use our own integration of these services called Software Factory ( You will get the following features: A nice integration under a single web menu. A single sign-on between all the services easily and external authentication on LDAP, GitHub or launchpad (cauth). A bug tracking system (Redmine ( Collaboration tools: Paste for sharing output or code extracts. Etherpad for collaborative editing. Managed upgrades from previous versions in an easy way. Because Software Factory is self-hosted (we use Software Factory to develop Software Factory), you can see it in action at ( If you want to test drive Software Factory, just follow the documentation at ( Keep us posted on what you have achieved using this new way of doing CI!

Topics : Business (/tags/business)

Continuous integration (/tags/continuous-integration)

Programming (/tags/programming)


OpenStack (/tags/openstack)

Frédéric Lepied - Frédéric Lepied is a Senior Manager at Red Hat. He has been involved with the Open Source movement since 1996. Contributing to Debian and XFree86, then becoming CTO for MandrakeSoft/Mandriva. He has always written and coordinated projects using Python (rpmlint, lads, msec, pxemngr, edeploy). At Red Hat, his team is contributing to OpenStack and building advanced CI/CD solutions. • More about me (/users/flepied)

Learn how you can contribute (/participate)

A guide to Git branching (/article/18/5/git-branching? utm_campaign=intrel)

How to retrieve source code of Python functions (/article/18/5/howretrieve-source-code-python-functions? utm_campaign=intrel)

Top 8 open source AI technologies in machine learning (/article/18/5/top-8open-source-ai-technologies-machinelearning?utm_campaign=intrel)

Edge computing and the importance of open infrastructure (/article/18/5/edge-computing? utm_campaign=intrel)

An introduction to the Pyramid web framework for Python (/article/18/5/pyramid-framework? utm_campaign=intrel)

Creating small containers with Buildah (/article/18/5/containersbuildah?utm_campaign=intrel)

Jody Hamilton on 16 Dec 2015


Great post, Frédéric. The traditional method is really more of a continuous nightmare. So much better to merge commits to master *after* they are tested, so that master is always deployable. My complaint with most CI approaches is the lack of inclusion of any manual QA process, UAT or other human review of the changes. Most focus only on automated tests and you can also do code review as you say. But in the real world you need to get feedback on work from stakeholders, and you don't want that testing and feedback to happen after merging to a staging branch or master for the same reasons that you don't want to merge the code before you do automated tests. To support that workflow we built ( which is now in Beta. We'd love to get your feedback.

Frédéric Lepied (/users/flepied) on 18 Dec 2015


Hi Jody, Your project seems interesting. Is it Open Source ?

Howard Tyson on 18 Dec 2015


I'm the Probo.CI project lead - Yep! The project is open source and then we offer a hosted version too. Open source: ( Hosted: (

thebadmonkeydev on 17 Dec 2015


I completely agree with the sentiment here. We've seen a huge adoption of the so-called 'Github Workflow' in the startup community. This workflow is essentially your proposed code review-centric approach and it's something I've written about before

( though I've never encountered speculative merging like you describe with Zuul

Frédéric Lepied (/users/flepied) on 18 Dec 2015


thebadmonkeydev: the big difference between the github way and the gerrit way is that you can manage the history of proposed patchset, test results and comments and go back in the history if you need using the gerrit way. More details at (


Jonathan Jenkins on 26 Dec 2015


Thanks for the article Frederic. Rare, I find myself clicking through so many links. Zuul was the one that really caught my attention. Do you experience many false positives when using this tool? Thanks in advance.


Sign up for news


Find us:

Privacy Policy | Terms of Use | Contact | Meet the Team | Visit


The next generation of continuous integration -

LOG IN SIGN UP Main menu Articles Resources About Community The Open Org The next generation of continuous integration (/users/flepied) 16 Dec ...

374KB Sizes 2 Downloads 5 Views

Recommend Documents

Educating the Next Generation
Contents. Educating the Next Generation • How the Teacher Training System

ReflexCash | The Next Generation of Affiliate Marketing
Take advantage of our cpa network ! We are working with the best products and services to boost publishers ROI. Maximize

Hacking: The Next Generation - WikiLeaks
O'Reilly Media, Inc. Hacking: The Next Generation, the image of a pirate ship on the cover, and related trade dress are

:: Next GENeration Republic Of Indonesian .::
Jan 22, 2009 - Kedudukan Presiden Soekarno menurut UUD 1945 adalah kedudukan Presiden selaku kepala pemerintahan dan kep

next generation vrv - Daikin

Next-generation sequencing - Scolaris
Dec 18, 2009 - Next-generation sequencing (also known as massively parallel sequencing) technologies are revolutionising

Next Generation Encryption - Cisco
Elliptic Curve Hash Security Levels Cryptographic Algorithm Configuration Guidelines IPsec VPN with Encapsulating Securi

Amazoncom: Azusa The Next Generation: Hezekiah Walker
Sep 23, 2017 - Azusa The Next Generation - Hezekiah Walker - Lastfm. Play fulllength songs from Azusa The Next Generatio

Social entrepreneurship and the next generation of giving - The
Nov 8, 2011 - Social entrepreneurship, as a field, focuses more on people and their new ideas than traditional philanthr

Comparison of continuous integration software - Wikipedia
The following table compares the features of some of the most popular continuous integration software; it is not compreh