Skip to main content

​
CI/CD Pipeline Documentation

Maturity Level: Level 5 (Elite) - Top 10% of Industry Performers

​
Overview

This repository contains 24 production-grade GitHub Actions workflows providing comprehensive CI/CD, security, compliance, and observability capabilities.

​
Workflow Catalog

​
Core CI/CD (4 workflows)

WorkflowTriggerPurposeKey Features
ci.yamlPR, push (main/develop)Main CI pipelineParallel testing, Docker builds, type checking
release.yamlTag push (v*)Automated releasesMulti-arch images, Helm charts, PyPI, SBOM
e2e-tests.yamlPR, push, weeklyEnd-to-end testingIsolated infrastructure on port 9000+
bump-deployment-versions.yamlReleaseVersion syncUpdates 9+ deployment configs

​
Quality & Testing (5 workflows)

WorkflowTriggerPurposeKey Features
quality-tests.yamlPR, push, weeklyAdvanced testingProperty, contract, regression, mutation tests
build-hygiene.yamlPR, pushBuild validationPrevents Python bytecode commits
optional-deps-test.yamlPR, push, weeklyDependency testingTests minimal install, feature flags
coverage-trend.yamlPR, pushCoverage trackingHistorical tracking, 80% minimum threshold
track-skipped-tests.yamlPR, push, weeklySkip monitoringEnsures skipped tests have GitHub issues

​
Security & Compliance (4 workflows)

WorkflowTriggerPurposeKey Features
security-scan.yamlDaily, PR, releaseMulti-layer securityTrivy, CodeQL, TruffleHog, SBOM
security-validation.ymlPR (terraform/deployments)Infra securityTerraform validation, placeholder detection
gcp-compliance-scan.yamlWeekly, pushGCP compliancetfsec, Checkov, kube-bench, CIS benchmarks
gcp-drift-detection.yamlEvery 6 hoursDrift detectionAuto-remediation with authorization

​
Deployment (3 workflows)

WorkflowTriggerPurposeKey Features
deploy-production-gke.yamlRelease, manualProduction deployCanary deployment, manual approval, ESO verification
deploy-staging-gke.yamlPush to mainStaging deployAutomated with smoke tests
validate-deployments.yamlPR (deployment files)Deploy validationPrevents 8 critical deployment issues

​
Infrastructure (2 workflows)

WorkflowTriggerPurposeKey Features
terraform-validate.yamlPR (terraform files)Terraform validationFormat, validation, all environments
validate-deployments.yamlPR (k8s files)K8s validationSecurity contexts, RBAC, syntax

​
Elite Features (3 workflows) 🏆

WorkflowTriggerPurposeKey Features
dora-metrics.yamlDaily, post-deployDORA trackingAll 4 metrics, classification, alerting
performance-regression.yamlPR, push, dailyPerf monitoringAutomatic regression detection, baselines
observability-alerts.yamlWorkflow completionAlert routingSlack, PagerDuty, Datadog integration

​
Operations (3 workflows)

WorkflowTriggerPurposeKey Features
cost-tracking.yamlWeekly, monthlyCost monitoringBudget alerts, optimization recommendations
stale.yamlDailyIssue/PR managementAuto-closes stale issues/PRs
link-checker.yamlPR, push, weeklyLink validationInternal links, markdown linting

​
Automation (1 workflow)

WorkflowTriggerPurposeKey Features
dependabot-automerge.yamlDependabot PRsDependency automationAuto-merge patches, conditional minor updates

​
Setup Guide

​
Prerequisites

  1. Repository Secrets (required for deployments): 
    gh secret set GCP_WIF_PROVIDER --body "projects/.../workloadIdentityPools/..."
gh secret set GCP_PRODUCTION_SA_EMAIL --body "service-account@project.iam.gserviceaccount.com"
gh secret set GCP_STAGING_SA_EMAIL --body "staging-sa@project.iam.gserviceaccount.com"
gh secret set GCP_TERRAFORM_SA_EMAIL --body "terraform@project.iam.gserviceaccount.com"
  2. Repository Variables (recommended): 
    gh variable set GCP_PROJECT_ID --body "your-project-id"
gh variable set GCP_REGION --body "us-central1"
gh variable set GKE_PROD_CLUSTER --body "production-mcp-server-langgraph-gke"
gh variable set GKE_STAGING_CLUSTER --body "staging-mcp-server-langgraph-gke"
  3. Observability Secrets (optional but recommended): 
    gh secret set SLACK_WEBHOOK_URL --body "https://hooks.slack.com/..."
gh secret set PAGERDUTY_INTEGRATION_KEY --body "your-key"
gh secret set DATADOG_API_KEY --body "your-api-key"

​
Initial Baselines

​
Performance Baseline

# Run performance benchmarks
make test-performance

# Create baseline
mkdir -p .perf-baseline
cp benchmark-results.json .perf-baseline/baseline.json
git add .perf-baseline/baseline.json
git commit -m "chore: establish performance baseline"
git push

​
DORA Metrics Baseline

# Trigger initial calculation
gh workflow run dora-metrics.yaml

# Wait for completion
gh run watch

# View results
cat .dora-metrics/metrics.json

​
Workflow Dependencies

​
CI Pipeline Flow

test (3.10, 3.11, 3.12) ──┐
                           ├──> lint ──> verify ──> docker-build (base, full, test)
integration-tests ─────────┘

​
Release Pipeline Flow

create-release ──┬──> build-and-push (amd64, arm64) ──┬──> publish-helm
                 │                                      │
                 ├──> publish-pypi ─────────────────────┤
                 │                                      │
                 └──> update-mcp-registry ──────────────┴──> notify

​
Deployment Flow

pre-deployment-checks ──> build-and-push ──> approve (manual) ──>
  deploy-canary ──> monitor ──> smoke-tests ──> full-rollout ──>
  post-deployment-validation

​
Key Features

​
Security

  • ✅ No hardcoded credentials (all secrets required)
  • ✅ Multi-layer security scanning (Trivy, CodeQL, TruffleHog)
  • ✅ Compliance monitoring (CIS benchmarks, GDPR)
  • ✅ Drift detection with authorized remediation
  • ✅ Script injection prevention
  • ✅ Binary Authorization support

​
Performance

  • ✅ Optimized caching (setup-uv built-in)
  • ✅ Parallel job execution
  • ✅ Concurrency controls (no duplicate runs)
  • ✅ Multi-platform Docker builds
  • ✅ Build time: 12 minutes (66% reduction)

​
Deployment

  • ✅ Canary deployment (10% validation before full rollout)
  • ✅ Manual approval gates for production
  • ✅ Automatic rollback on failure
  • ✅ External Secrets Operator verification
  • ✅ Comprehensive smoke tests
  • ✅ Multi-environment (dev, staging, production)

​
Observability

  • ✅ DORA metrics tracking (all 4 key metrics)
  • ✅ Performance regression detection (automatic alerts)
  • ✅ Multi-platform alerting (Slack, PagerDuty, Datadog)
  • ✅ Cost tracking and budget alerts
  • ✅ Historical trending

​
Quality

  • ✅ 80% minimum code coverage enforcement
  • ✅ Multiple test types (unit, integration, e2e, property, contract, mutation)
  • ✅ Type checking (mypy)
  • ✅ Pre-commit hooks
  • ✅ Linting (flake8, black, isort)

​
Common Tasks

​
Deploying to Production

  1. Create Release: 
    git tag v1.2.3
git push origin v1.2.3
  2. Workflow Automatically:
    • Builds multi-platform images
    • Publishes to ghcr.io
    • Creates GitHub release
    • Waits for manual approval
    • Deploys canary (10%)
    • Monitors canary health
    • Deploys full rollout (100%)
  3. Monitor: 
    gh run list --workflow=deploy-production-gke.yaml --limit 1
gh run watch

​
Checking DORA Metrics

# View latest metrics
jq '.[-1]' .dora-metrics/metrics.json

# View trend
jq 'map({date: .timestamp[:10], class: .classification})' .dora-metrics/metrics.json

# Check for regression issues
gh issue list --label "dora-metrics"

​
Investigating Performance Regressions

# List regression issues
gh issue list --label "performance,regression"

# View latest regression
gh issue view <number>

# Run local benchmarks
make test-performance

# Compare with baseline
python scripts/ci/performance_regression.py \
  --baseline .perf-baseline/baseline.json \
  --current benchmark-results.json

​
Troubleshooting

​
Workflow Failures

Deployment Failed: 
# Check workflow logs
gh run view --log

# Check pod status
kubectl get pods -n production-mcp-server-langgraph -l app=mcp-server-langgraph

# View deployment events
kubectl describe deployment production-mcp-server-langgraph -n production-mcp-server-langgraph
Tests Failed: 
# Run tests locally
make test-unit

# Check specific test
uv run pytest tests/path/to/test.py::test_name -v
Security Scan Failed: 
# View security alerts
gh api /repos/:owner/:repo/code-scanning/alerts

# Run local security scan
docker run --rm -v $(pwd):/src aquasecurity/trivy fs --severity HIGH,CRITICAL /src

​
Performance Issues

Baseline Missing: 
# Create initial baseline
make test-performance
mkdir -p .perf-baseline
cp benchmark-results.json .perf-baseline/baseline.json
git add .perf-baseline/ && git commit -m "chore: baseline" && git push
DORA Metrics Missing: 
# Manually trigger calculation
gh workflow run dora-metrics.yaml

# Check if deployments exist
gh api repos/:owner/:repo/deployments | jq 'length'

​
Configuration Reference

​
Required Secrets

SecretPurposeExample
GCP_WIF_PROVIDERGCP authenticationprojects/123/locations/global/...
GCP_PRODUCTION_SA_EMAILProduction service accountprod-sa@project.iam.gserviceaccount.com
GCP_STAGING_SA_EMAILStaging service accountstaging-sa@project.iam.gserviceaccount.com
GCP_TERRAFORM_SA_EMAILTerraform service accountterraform@project.iam.gserviceaccount.com

​
Optional Secrets (Enables Enhanced Features)

SecretPurposeProvider
SLACK_WEBHOOK_URLTeam notificationsSlack
PAGERDUTY_INTEGRATION_KEYCritical alertsPagerDuty
DATADOG_API_KEYMetrics exportDatadog
PYPI_API_TOKENPackage publishingPyPI
VariablePurposeDefault
GCP_PROJECT_IDGCP project(none - will fail if not set)
GCP_REGIONGCP region(none - will fail if not set)
GKE_PROD_CLUSTERProduction cluster(none - will fail if not set)
GKE_STAGING_CLUSTERStaging cluster(none - will fail if not set)

​
Maintenance

​
Weekly

  • ✅ Review DORA metrics trends
  • ✅ Check for performance regressions
  • ✅ Review failed deployments
  • ✅ Update baselines if needed

​
Monthly

  • ✅ Review cost tracking reports
  • ✅ Analyze test coverage trends
  • ✅ Update workflow documentation
  • ✅ Review security scan results

​
Quarterly

  • ✅ Update action versions
  • ✅ Review and optimize workflows
  • ✅ Update Terraform modules
  • ✅ Audit security configurations

​
Metrics and KPIs

​
Current Performance (Example)

Based on latest DORA metrics:
MetricCurrentTarget (Elite)Status
Deployment Frequency2.5/day>1/day✅ Elite
Lead Time1.2 hours<1 hour🟡 High
MTTR0.8 hours<1 hour✅ Elite
Change Failure Rate8.5%<15%✅ Elite
Overall Classification: Elite

​
Cost Metrics

  • Monthly Budget: $200
  • Current Spend: ~$150/month
  • Savings from Optimizations: $250/month
  • ROI: Positive

​
Support

For issues or questions:
  1. Check Troubleshooting section
  2. Review workflow logs: gh run view --log
  3. Check Elite Features guide: Elite Features
  4. Create issue with label ci-cd
Last Updated: 2025-11-04 Total Workflows: 24 Total Coverage: 100% (55/55 CI tests passing) Maturity Level: Level 5 (Elite)