The continuous integration pipeline evolves over time as architecture and design changes are made.   Manually configuring the pipeline can be a costly activity if the project contains a large number of modules.   Setting up a ‘Continuous Pipeline Configuration’ process may become a necessity for reducing this time-consuming effort, as well as for reducing the risk of important bug fixes or other changes being left out of the final product build due to an improper pipeline configuration.

Developing a CPC process to regularly validate and adjust the pipeline is something I highly recommend for any size project, but especially for large, complex modular projects. CPC is actually quite easy once you know what kind of source code changes will require corresponding changes in the pipeline configuration.

The CPC process works much the same as a CI server watching a version control system for change.   The pipeline configurator watches for changes to the build dependency graph for the full product build and makes the appropriate pipeline adjustments to the build and test triggers in the CI server. When a build or test dependency is removed from the graph, the corresponding CI trigger is removed and when a dependency is added, a trigger is added.

Continuous Pipeline Configuration

1. Capture the product build and test dependency graph
2. Capture the existing Continuous Integration pipeline configuration
3. Compare the dependency graph to the pipeline configuration
4. Update the CI triggers as needed.

The product build and test dependency graph is the complete list of ‘A depends on B’ relationships.   I like to think of this as the producer-consumer relationship where B is the producer and A is the consumer if A depends on B. Both A and B can be any type of process, activity or event, such as build, test, bundle, or deploy. It is not mandatory for these producers to actually publish binary artifacts. In some cases, the only output is a pass or fail result – enough to determine if a corresponding CI trigger should or should not execute the next event in the pipeline.  If you are using the Artifactory for your artifact repository, you use it to store and retrieve the dependency graph via the Artifactory Query Language.

If the consumer module’s dependency is a static constraint, the dependency is a ‘pull’ connection in the pipeline and if the dependency is a dynamic constraint, the dependency is a ‘push’ connection. Pull connection don’t need a CI trigger since the change to pick up new versions will be a code change that triggers the ‘consumer’ build or test. Push connections require finished build triggers in the CI server so new ‘producer’ builds will trigger ‘consumer builds if they succeed.

Continuous Pipeline Configuration is not limited to Continuous Integration.  It can be used for configuring of all of the pipelines in the Continuous Delivery process.  My thoughts on CD are very similar to those of Jame Betteley as he describes his views here: Methods and Tools.  Continuous Delivery extends the core principles of CI all the way through to the end of the software development life cycle.

We’re automation experts.  Let’s take Continuous Integration – and ultimately Continuous Delivery – to the next level with Continuous Pipeline Configuration.