Implement guardrails and accountability

6.1 Define autonomy levels

Classify agent actions by risk

Risk-based autonomy means an agent may act within boundaries that match the impact and reversibility of what it is doing. Risk is not only a property of the action; it depends on where the change lands, how easily it can be reversed, and how quickly it reaches users or systems. Classify along three lenses:

Assign autonomy levels

Autonomy is never one setting — it is the combination of PR rules, environments, workflow permissions and tool access. And it should be routed by a machine-readable signal, not by someone reading a narrative:

flowchart LR
  PLAN["Agent plan<br/>plan.json with risk rating"] --> ROUTE{"risk"}
  ROUTE -->|"low"| AUTO["apply_auto<br/>runs after checks"]
  ROUTE -->|"medium · high"| GATED["apply<br/>environment: approval-required"]
  GATED --> APPROVER["Required reviewer<br/>approves or rejects"]

name: agent-plan-apply
on:
  workflow_dispatch:

permissions:
  contents: read

jobs:
  plan:
    runs-on: ubuntu-latest
    outputs:
      risk: ${{ steps.read.outputs.risk }}
    steps:
      - uses: actions/download-artifact@v8
        with:
          name: plan
          path: out
      - id: read
        run: echo "risk=$(jq -r .risk out/plan.json)" >> "$GITHUB_OUTPUT"

  apply_auto:
    needs: plan
    if: needs.plan.outputs.risk == 'low'
    runs-on: ubuntu-latest
    steps:
      - run: ./scripts/apply.sh out/plan.json

  apply:
    needs: plan
    if: needs.plan.outputs.risk != 'low'
    runs-on: ubuntu-latest
    environment: approval-required     # pauses for a required reviewer
    steps:
      - run: ./scripts/apply.sh out/plan.json

Grow autonomy progressively

1. The agent can only open PRs; it cannot merge.
2. It may auto-merge low-risk PRs once checks pass.
3. It may deploy to staging automatically.
4. It reaches production only through an environment approval.

Each step is earned with evidence from the previous one. The same logic applies to issue triage automations, whose automation level ranges from Full control (every change held for review) through Cautious (the default: only high-confidence changes apply) and Balanced to Full automation.

Stay compliant with security and Responsible AI standards

Organizational security standards and Responsible AI principles meet in the same practical controls: people remain accountable for what agents do (co-authored commits, requesters cannot self-approve); work stays transparent (session logs, plans, audit events); humans keep oversight of consequential decisions (reviews, environments); and exposure is minimized (least privilege, firewall, secrets kept out of the agent's reach). An autonomy level that cannot be audited is not compliant, however fast it is.

6.2 Implement guardrails and human-in-the-loop workflows

Identify the actions that require human judgement

Put people at decision points, not at every step:

• Merging into a protected branch.
• Deploying to production, or any irreversible data operation.
• Accessing secrets or changing permissions and tokens.
• Changing workflows, agent configuration or security settings.
• Accepting a security exception, a licence change or anything with legal weight.
• Choosing between conflicting agent outputs, or resolving ambiguity the task did not settle.

flowchart LR
  A["Agent opens PR<br/>with plan and risk level"] --> C["Checks validate"]
  C --> O["CODEOWNERS<br/>auto-requested"]
  O --> M{"Merge<br/>decision point"}
  M --> D{"Deploy<br/>decision point"}
  D --> P["Production<br/>environment approval"]

Block actions that violate policy

Blocking must be enforced by the platform. "Controls in docs only" is an anti-pattern: if the platform cannot enforce it, it will eventually be bypassed. Layer the controls so each catches what the previous one missed.

flowchart LR
  PRE["Before the action<br/>permissions · tool allowlists · firewall"] --> DURING["During the action<br/>preToolUse hooks · required checks · push protection"]
  DURING --> AFTER["After the action<br/>reviews · environments · audit log · artifacts"]

{
  "version": 1,
  "hooks": {
    "preToolUse": [
      {
        "type": "command",
        "bash": "./.github/hooks/scripts/block-dangerous.sh",
        "timeoutSec": 10
      }
    ],
    "postToolUse": [
      {
        "type": "http",
        "url": "https://audit.contoso.com/copilot",
        "timeoutSec": 5
      }
    ]
  }
}

#!/bin/bash
# Reads the preToolUse payload on stdin and denies destructive shell commands.
input=$(cat)
tool=$(jq -r '.toolName' <<< "$input")
args=$(jq -r '.toolArgs' <<< "$input")

if [[ "$tool" == "bash" ]] && grep -Eq 'rm -rf /|git push --force|curl [^|]*\| *(ba)?sh|sudo ' <<< "$args"; then
  jq -cn --arg r "Blocked by policy: destructive or remote-execution command" \
    '{permissionDecision: "deny", permissionDecisionReason: $r}'
  exit 0
fi

echo '{"permissionDecision": "allow"}'

Some rules cannot be expressed as settings alone. A defence-in-depth guardrail workflow, made a required check, can insist that at least one approval came from a human account rather than an automation identity:

name: agent-guardrails
on:
  pull_request:
    branches: [main]

permissions:
  contents: read
  pull-requests: read

jobs:
  require-human-approval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/github-script@v9
        with:
          script: |
            const { data: reviews } = await github.rest.pulls.listReviews({
              owner: context.repo.owner,
              repo: context.repo.repo,
              pull_number: context.payload.pull_request.number,
            });
            const human = reviews.some(r => r.state === 'APPROVED' && r.user?.type === 'User');
            if (!human) core.setFailed('A human approval is required before merge.');

Scope permissions and execution contexts for least privilege

Permissions are the true boundary of an agent's power — they decide what it can do regardless of its instructions or intent. Broad tokens turn small mistakes into large incidents.

permissions:
  contents: read            # workflow default: read-only

jobs:
  analyze:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - run: npm run analyze

  open_pr:
    needs: analyze
    runs-on: ubuntu-latest
    permissions:            # elevated only for the job that writes
      contents: write
      pull-requests: write
    steps:
      - uses: actions/checkout@v7
      - run: ./scripts/open-pr.sh

# Read-only analysis: only git commands may run without a prompt
copilot -p "Summarize the last 10 commits and flag breaking changes" -s \
  --allow-tool='shell(git:*)' --deny-tool='shell(git push)' --deny-tool=write

Require explicit authorization for irreversible or compliance-sensitive changes

GitHub builds several controlled paths into Copilot cloud agent, and you add the rest:

• Only users with write access can trigger cloud agent; comments from others are never shown to it.
• The agent pushes only to its own branch, and only through simple push operations — it cannot run git push or other Git commands directly.
• Its PRs are drafts; it cannot mark them ready, approve or merge them.
• The person who asked for the PR cannot approve it, preserving "required approvals".
• When a PR is opened under Copilot's own app identity (not attributed to a person), one additional approval is required if the repository already requires at least one — on by default in rulesets, always on for branch protection.
• Actions workflows do not run on its pushes until someone with write access clicks Approve and run workflows — disabling that is a repository-admin decision, and it can expose secrets to unreviewed code.
• Hidden characters and HTML comments are filtered from issues and comments before they reach the agent, to reduce prompt injection.

For the change itself, use environments with required reviewers for deploys and secret access, plan-first for irreversible work, and in GitHub Agentic Workflows, safe outputs so the agent only proposes writes that a separate, gated job performs after threat detection.

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment:
      name: production          # required reviewers · production-only secrets
    concurrency:
      group: production         # serialize production deploys
      cancel-in-progress: false
    steps:
      - uses: actions/checkout@v7
        with:
          ref: ${{ github.sha }}  # deploy an explicit commit, not "latest"
      - run: ./deploy.sh

# CODEOWNERS — only enforced when "Require review from Code Owners" is on
*                                     @org/core-team
/security/                            @org/security-team
/infra/                               @org/platform-team
/.github/workflows/                   @org/platform-team
/.github/agents/                      @org/ai-governance
/.github/hooks/                       @org/ai-governance
/.github/copilot-instructions.md      @org/ai-governance
/.github/workflows/copilot-setup-steps.yml  @org/ai-governance

Preserve execution velocity

An approval is only worth its delay if it materially reduces risk. Approvals everywhere produce governance fatigue and rubber-stamping — which is worse than no approval, because it looks like control.

Keep governance working over time

Controls drift: checks get renamed or removed, review requirements are relaxed, CODEOWNERS goes stale, permissions expand, secrets move to broader scopes. Treat governance as an operational responsibility with a cadence: