How can I store data throughout multiple builds in a continuous integration pipeline?

Question

Our team is using an Azure DevOps pipeline for our continuous integration tasks (build, run unit test, determine code coverage, etc.).

Now we would like to track various measures (like, say, the number of compiler warnings) throughout multiple builds to dashboard them (use case 1), but also to prevent regression (use case 2). Both use cases require storing build results persistently - even after their retention time expires.

How can I store data (like a JSON or YAML string holding pairs of commit hash to multiple data fields) persistently from within an Azure DevOps pipeline throughout the pipeline's lifetime?

I'm looking for a technical solution, for sure, but conceptual answers with the DevOps-specific wording would help as well as I have a hard time even searching for solutions as I feel I'm not using the right wording.

Technically, I could imagine setting up an SQL database (on Azure?) that my CI stages communicate with to fill with persistent data. But that seems heavyweight. I could imagine storing the data in the repo itself but that seems weird and would cause additional commits that don't match actual version management patterns.

Ideally, the solution wouldn't involve any Azure DevOps pipeline plug-in (neither self-developed, nor from the marketplace) as they are hard to install inside our organization, but if excellent solutions for the question exist as plug-ins, hints are still welcome.

Answer 1

No better way within DevOps. Repositories are used to store source files that are under source control.

In DevOps, it stores the build results as Artifacts. By default, the artifacts are published to container with retention policies set. That means the results will be deleted one day per your retention policies set. Obviously, it doesn't meet your need of storing build results persistently.

If you just want to store the results somewhere and not be affected by the retention policy, then you can try to copy/publish the results to other storage service when running the pipeline. For example Azure Blob Storage, or other directories which the build service account can be accessed. Of course, adjacent git repo you did is also an option.

Answer 2

What you're looking for is a code quality metric for your development processes. You could roll your own, but SonarQube is purpose-built for this task and is used extensively in the industry.

Conceptually, you extend your build pipeline to include SonarQube analysis tasks that look for potential security issues, code smells, compiler warnings, potential bugs and stylistic issues. At the end of the pipeline, it publishes analysis findings, unit test results and code-coverage to the SonarQube database.

The web interface for SonarQube provides historical tracking for all these metrics and has APIs that you can use to extract data if you want to build custom reports that aren't part of the offering. For each code-smell or issue, it provides documentation on what's wrong, how to fix and the estimated resolution time. Great way to report back to management on how long it'll take to fix technical debt (the tool says 23 days!)

There's a paid cloud-offering called "SonarCloud". Alternatively, you can stand up "SonarQube" on-prem or in a container (Kubernetes, Docker, Azure Container Instance, Web App). My team uses a SQL database + Web App.

The Community Edition is free, but doesn't have all the bells and whistles. The paid Developer and Enterprise versions support branch-level analysis and can integrate directly into your pull-requests. There's an out-of-box integration where SonarQube will "decorate" your pull-requests by add comments to changed files highlighting the issues that are being introduced.

It can be finicky getting your pipeline properly configured, but once it's setup it's a great value add to your delivery process.