CI/CD Pipeline Standardization

Shared Templates vs Per-Team Customization

The goal is not one pipeline for all teams. The goal is a set of composable building blocks that cover 80% of use cases, with clear extension points for the other 20%. GitHub Actions reusable workflows are the cleanest implementation of this pattern. You define workflow templates in a central repository and teams reference them with uses: org/platform-pipelines/.github/workflows/build.yml@v2.

Jenkins shared libraries follow the same idea but with Groovy. Buildkite uses plugins. The principle is identical across tools: platform teams own the core pipeline logic, application teams own their configuration.

At Spotify, the CI platform team maintains about 15 shared workflow templates that cover all standard languages and deployment targets. Teams can override specific steps but the structure remains consistent. This consistency is what makes it possible to reason about build times and security compliance across 2,000+ repositories.

Build Time Optimization

Slow CI kills developer productivity. The data from DX and Sleuth shows that teams with CI times under 10 minutes deploy 3x more frequently than teams with 30+ minute pipelines. Here's where the time goes and how to fix it.

Dependency installation is the first target. Cache your node_modules, pip packages, and Go modules between runs. GitHub Actions caching and Buildkite's artifact system handle this natively. A cached npm install takes 5 seconds instead of 45.

Docker builds benefit enormously from layer caching. Use BuildKit with --cache-from pointing to a registry cache. A well-structured Dockerfile with proper layer ordering goes from 8 minutes to 90 seconds on cache hit.

Test parallelization is the next lever. Split your test suite across multiple runners. CircleCI and Buildkite have native test splitting. For GitHub Actions, use matrix strategies. A 20-minute test suite split across 4 runners finishes in 5 minutes plus overhead.

Security Scanning Integration

Every pipeline should include four scanning stages: SAST (static analysis of your code), SCA (dependency vulnerability scanning), container image scanning, and secret detection. Run them in parallel to avoid adding serial time to the pipeline.

Semgrep handles SAST with custom rules. Snyk or Dependabot covers SCA. Trivy scans container images and is free. GitGuardian or TruffleHog catches committed secrets. The key is making results actionable. Block merges only for critical and high vulnerabilities. Medium and low findings go to a tracking system for later remediation.

Pipeline as Code

Your pipeline definition lives in the repository alongside the application code. This is non-negotiable. No clicking through Jenkins UI to configure jobs. The pipeline file (.github/workflows/, Jenkinsfile, .buildkite/pipeline.yml) is code reviewed, version controlled, and auditable.

Platform teams provide starter templates that teams copy into their repos. When the platform team updates the shared template, teams get the update automatically on their next run (for referenced workflows) or through a renovate-style PR (for copied templates).

Deployment Approval Workflows

Automated deployments to development and staging environments on every merge to main. No human gates. Production deployments need a lightweight approval: a Slack notification with an approve button, or a GitHub deployment environment with required reviewers. The approval should take seconds, not hours. If your production approval process involves a change advisory board meeting, you've built a bottleneck that will push teams to batch changes, making each deployment riskier.