Domain 3.6 · part of 50–55%

Maintain pipelines

Keeping the delivery system itself healthy: measuring pipeline health, hunting flaky tests, making runs cheaper and faster, controlling concurrency, retaining the right artifacts for the right time, and moving off classic pipelines.

Monitoring pipeline health

Metric	Signals	Healthy target
Pass / success rate	Pipeline reliability; a broken main build blocks everyone	> 95%
Duration	Feedback speed; long runs discourage frequent integration	Trending down
Failure rate	How often runs fail — and whether it is code or infrastructure	Low and stable
Flaky-test rate	Tests that pass/fail without code change; erode trust	< 1%
Queue / wait time	Agent starvation — a concurrency problem, not a speed problem	< 5 min
MTTR for a red build	How fast the team unblocks a broken pipeline	< 30 min
Agent utilisation	Whether the pool is over/under-provisioned	Busy but with headroom

Where to see it: Azure DevOps Pipelines → Analytics (pass rate, duration, test failures over time), the Pipeline pass rate and Test failures reports, and dashboard widgets. GitHub surfaces workflow run history and, with the Insights/metrics, job-level timing; failures can fan out to alerts (below).

Flaky tests

A flaky test produces different results on the same code — the most corrosive pipeline problem, because it trains people to ignore red.

Common cause	Fix
Race conditions / unawaited async	Proper `await`; deterministic synchronisation
Order dependency between tests	Isolate state; no shared mutable fixtures
Real clock / time zones	Inject a clock; freeze time in tests
Unseeded randomness	Fix the random seed
External service latency	Mock it, or add health-checked service containers

Azure DevOps flaky-test management

Configure under Project Settings → Pipelines → Test management → Flaky tests. Detection can be via on-rerun (a test that passes on retry after failing) or via a system/custom flaky-bug workflow. You can choose whether flaky results count against the pipeline pass/fail — quarantining them keeps CI green while you fix the root cause.

azure-pipelines.yml auto-rerun failed tests (VSTest)

- task: VSTest@3
  inputs:
    rerunFailedTests: true
    rerunMaxAttempts: 3
    rerunFailedThreshold: 30     # give up if >30% failed (likely a real break, not flake)

Caveat — retry masks, it doesn't fix

Auto-rerun is a stopgap that keeps the pipeline moving, but the AZ-400 "best practice" answer is to quarantine the flaky test and fix the root cause. Retrying everything hides real regressions and inflates duration. Order of preference: fix the cause → quarantine while fixing → retry as a last resort.

Optimizing pipelines

For time

Technique	How it helps
Parallel jobs / matrix	Run independent work concurrently — reduce wall-clock time
Dependency caching	Skip re-downloading packages every run
Shallow clone (`fetchDepth: 1`)	Fetch only the tip commit
Path filters	Skip runs for irrelevant changes (docs-only)
Test Impact Analysis	Run only tests affected by the change
Job/artifact reuse	Build once, reuse the artifact downstream (don't rebuild per stage)
Self-hosted agents	Warm caches and local Docker layers persist between runs

Caching

azure-pipelines.yml Cache@2

- task: Cache@2
  displayName: Cache NuGet
  inputs:
    # key changes → cache miss → repopulate; restoreKeys give a fallback
    key: 'nuget | "$(Agent.OS)" | **/packages.lock.json'
    restoreKeys: |
      nuget | "$(Agent.OS)"
    path: '$(NUGET_PACKAGES)'

GitHub Actions actions/cache

- uses: actions/cache@v4
  with:
    path: ~/.npm
    key: ${{ runner.os }}-npm-${{ hashFiles('**/package-lock.json') }}
    restore-keys: |
      ${{ runner.os }}-npm-

# Many setup-* actions cache automatically:
# - uses: actions/setup-node@v4   with: { cache: npm }

Caveat — cache key hygiene

The cache key must include a hash of the lock file so a dependency change invalidates the cache; otherwise you serve stale packages and get non-reproducible builds. restoreKeys provide a partial-match fallback so a near-miss still avoids a full cold restore. Never cache build outputs you intend to ship — cache inputs (packages), publish outputs as artifacts (3.1).

Caveat — fetchDepth: 1 breaks some tools

A shallow clone speeds checkout but hides history. GitVersion, SonarQube, git blame and anything that walks commits need full history (fetchDepth: 0 in Azure Pipelines, fetch-depth: 0 in actions/checkout). This trade-off is directly examinable.

For cost

Lever	Saving
Self-hosted agents for long/frequent builds	No per-minute charge; only infrastructure cost
Right-size the OS image	Linux is cheapest; avoid macOS unless building for Apple
Path/branch filters	Don't spend minutes on changes that don't need CI
Caching + incremental builds	Fewer minutes per run
Tighter retention	Lower storage cost for runs and artifacts
Concurrency limits	Fewer parallel jobs to purchase

For reliability

azure-pipelines.yml timeouts and step retry

jobs:
  - job: Build
    timeoutInMinutes: 30           # fail fast instead of hanging
    cancelTimeoutInMinutes: 5
    steps:
      - script: ./flaky-network-step.sh
        retryCountOnTaskFailure: 3  # retry transient infra failures

Also: idempotent scripts (safe to re-run after a partial failure), health gates before production (3.2), and pinning tool/action versions so a third-party change can't break your build unexpectedly.

Pipeline concurrency

	Azure Pipelines	GitHub Actions
Unit of concurrency	Parallel jobs purchased per org	Concurrent jobs per plan; concurrency groups in YAML
Free tier (private)	1 hosted (limited minutes) or 1 self-hosted	Plan-dependent concurrent job limit
Serialise a critical path	`concurrency` in YAML / exclusive-lock check	`concurrency:` group

azure-pipelines.yml serialise prod deploys

lockBehavior: sequential   # with an exclusive-lock check on the environment

# or a pipeline-level group
- stage: DeployProd
  # the exclusive lock check on 'production' environment
  # ensures only one run deploys at a time

GitHub Actions concurrency group

# PRs: cancel the previous run when a new commit arrives
concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

# Production: let the current deploy FINISH, queue the next
concurrency:
  group: production-deploy
  cancel-in-progress: false

Caveat — cancel-in-progress by context

cancel-in-progress: true for PR/CI runs (a newer commit makes the old run pointless — saves minutes). cancel-in-progress: false for production deployments (never abort a half-finished deploy — queue the next one). Confusing the two either wastes agent time or corrupts a deployment.

Queue time is a concurrency problem

If runs sit in a queue, the fix is more parallel jobs / more agents (or a self-hosted pool), not a faster build. If each run is slow but starts immediately, the fix is the time optimisations above. Diagnose which before answering.

Retention strategy

What	Default	Where to configure
Pipeline runs & artifacts	30 days	Org/Project Settings → Pipelines → Settings (min/max days)
Classic release retention	~30–60 days	Release definition → Retention
A specific important run	—	Retention lease — "retain" a run to exempt it from cleanup
Azure Artifacts packages	Keep all	Feed → Settings → Retention (keep N latest / delete after N days)
GitHub Actions artifacts/logs	90 days (configurable, up to 400 enterprise)	`retention-days` per upload; repo/org settings

Artifact	Sensible retention
Feature-branch builds	7–14 days
Main-branch builds	30–90 days
Released / tagged builds	1 year or permanent (retention lease)
Production deployment records	1 year+ (audit)

Caveat — retention leases keep a build forever

"Keep this specific release build indefinitely for audit/compliance while everything else expires" → set a retention lease (retain the run), not a change to the global policy. Retained runs are exempt from the automatic cleanup. Deleting a run also deletes its artifacts and test results, so lease before you need it.

azure cli add a retention lease

az pipelines runs update \
  --id $(Build.BuildId) \
  --org https://dev.azure.com/MyOrg --project MyProject \
  --retention-lease-days 365

Migrating classic pipelines to YAML

	Classic (UI)	YAML
Definition	Clicked together in the portal	Code in the repo, versioned with the app
Review / audit	Hard — no diff, no PR	Full PR review and history
Reuse	Task groups	Templates, reusable workflows
Branching	One definition for all branches	Each branch can carry its own pipeline

Audit the classic pipeline — tasks, variables, triggers, service connections, variable groups, environments and gates.
Export what you can: a classic build pipeline has a View YAML option. Classic release pipelines cannot be exported — you recreate them as YAML stages.
Recreate structure: build → deployment stages; classic gates and manual interventions become environment approvals and checks.
Move variables: classic variables → variables: and variable groups; secrets → variable groups linked to Key Vault.
Map deployment groups → YAML environments with VM resources.
Run in parallel: keep the classic pipeline live and compare outputs until the YAML pipeline is trusted.
Cut over: point branch policies at the YAML pipeline, then disable/delete the classic one.

Caveat — classic release gates become environment checks

The single hardest part of migration: classic release pipelines cannot be exported to YAML, and their pre/post-deployment gates and manual interventions have to be re-implemented as approvals and checks on YAML environments (see 3.2). If a question asks how to migrate a classic release with an Azure Monitor gate, the answer is a Query Azure Monitor alerts check on the environment — not a YAML task.

Failure alerting (bridge to Domain 5)

azure-pipelines.yml notify Teams on failure

- script: |
    curl -X POST "$TEAMS_WEBHOOK" -H 'Content-Type: application/json' \
      -d "{\"text\":\"❌ $(Build.DefinitionName) failed on $(Build.SourceBranch)\"}"
  condition: failed()
  env:
    TEAMS_WEBHOOK: $(TeamsWebhookUrl)

# Better: Project Settings → Notifications, or Service hooks → Teams,
# filtered to failed builds only.

GitHub Actions notify on failure

- name: Notify Slack on failure
  if: failure()
  uses: slackapi/slack-github-action@v2
  with:
    payload: |
      { "text": "❌ ${{ github.workflow }} failed on ${{ github.ref }}" }
  env:
    SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}

Alerts on pipeline health belong to the wider instrumentation story in Domain 5 — including wiring an Azure Monitor alert back to trigger a pipeline for auto-remediation.

3.6 rapid recap

The requirement says…	Answer
Build takes 45 minutes — speed it up	Cache dependencies, parallelise, shallow clone, TIA
Builds queue for 20 minutes before starting	Buy more parallel jobs / add agents (concurrency, not speed)
Test fails randomly with no code change	Flaky test — quarantine, then fix the root cause
Cancel the previous PR build on a new push	`cancel-in-progress: true` / `pr: autoCancel`
Never abort an in-flight production deploy	`cancel-in-progress: false` / exclusive lock
Keep one release build forever for audit	Retention lease (retain the run)
Delete old feature-branch artifacts automatically	Retention policy at org/pipeline level
Cache keeps serving stale packages	Include the lock-file hash in the cache key
GitVersion/SonarQube fails after adding a cache/shallow clone	Use full history (`fetchDepth: 0`)
Migrate a classic release with gates to YAML	Recreate as YAML stages; gates → environment checks
Alert the team when the pipeline fails	Notifications / service hooks filtered to `failed()`