TG
All posts
2 min read

Kernel Tuning & CI/CD: Shrinking Release Cycles from 45 to 12 Minutes

Where 45 minutes actually went: a coverage gate that runs before the build, two sysctl values, and Restic replacing SQL dumps.

A 45-minute build does not just cost 45 minutes. It changes how people work: developers stop committing often, batch their changes into large pull requests, and hand you the merge conflicts and broken builds that follow.

Auditing the Jenkins and GitHub Actions pipelines across 6 production applications, the time was not lost in one place. It was spread across three layers — code quality, pipeline structure, and the operating system underneath the build agents.

WHERE THE 45 MINUTES WENTSHIFT-LEFT SECURITYPIPELINE ABSTRACTIONOS-LEVEL OPTIMISATIONephemeral agentsSTATE PRESERVATIONDeveloper pushcode changesGitHub pull request6 production appsSonarQubecoverage ≥ 80%, CVE scanJenkins master nodeGroovy shared librariesLinux build agenttcp_tw_reuse=1 · swappiness=10Docker layer cachemounted, not rebuiltResticincremental · encryptedTarget databasesnapshot pre-deployBackup storageS3 · Azure Blob1. open PR2. quality gate3. gate passedcoverage < 80%, abort4. spin up agent5. reuse layers6. build complete7. snapshot8. push encryptednumbered happy pathabort path
The gate runs before the build, so a pull request that fails coverage never costs agent time at all.

How a change now reaches deployment

  1. A developer pushes and opens a GitHub pull request.
  2. SonarQube scans it for code coverage, vulnerabilities, quality gates and security hotspots.
  3. Passing work moves on to the Jenkins master node.
  4. Jenkins spins up an ephemeral Linux build agent on demand.
  5. The build runs against a mounted Docker layer cache rather than rebuilding from scratch.
  6. On success, Restic takes over before anything deploys.
  7. It snapshots the target database — incremental, deduplicated, encrypted.
  8. That snapshot is pushed to object storage, S3 or Azure Blob.

The ordering in step 2 is the point. The gate used to run after the build, which meant a pull request could fail quality checks having already spent 45 minutes of agent time. Now the pipeline fails instantly if coverage drops below 80% or a vulnerability is found, and a bad change costs nothing.

The kernel was the bottleneck nobody looked at

Self-hosted agents running parallel Docker builds were hitting two limits that never show up in pipeline logs. Outbound connections were exhausting SNAT ports, and memory pressure was pushing the agents into aggressive swapping — so builds sat in I/O wait while appearing, from the outside, to simply be slow.

Both are settings, not hardware. Two lines in /etc/sysctl.conf removed the I/O wait entirely:

net.ipv4.tcp_tw_reuse = 1 lets the kernel reuse sockets sitting in TIME_WAIT, which is what the SNAT exhaustion actually was. vm.swappiness = 10 tells it to hold pages in memory rather than swapping under build pressure.

Pipelines as a library, not a file per project

The Jenkins master orchestrates but does not define. Pipeline logic lives in modular, declarative Groovy shared libraries — reusable and versioned — so six applications share one definition instead of drifting into six. Agents are ephemeral and disposable, which means every build starts from a clean environment and nothing accumulates between runs.

Backups that do not stall the pipeline

Pre-deployment database backups were traditional SQL dumps, and they were slow enough to be a meaningful share of the window. Restic replaced them with incremental, deduplicated, encrypted snapshots written straight to object storage — the same safety property, without the full-dump cost on every release.

Together these compressed release cycles by 73%, from 45 minutes to 12.

GitHub
LinkedIn