Perform evaluation, error analysis, and tuning

4.1 Define success criteria and evaluation signals

Specify expected outcomes and operational constraints

Evaluation starts before the agent runs. Write the criteria into the issue, reference them from the pull request, and validate against them. Alongside what must be true, state the operational constraints the work must respect:

Identify qualitative and quantitative signals

Quantitative signals scale and can be enforced as gates; qualitative ones catch what no check was written to detect. A healthy evaluation design uses both, and lets the quantitative gates remove the obvious failures so human reviewers spend their attention on the qualitative questions.

flowchart LR
  ISSUE["Issue<br/>task · acceptance criteria"] --> PR["Agent opens PR<br/>references the criteria"]
  PR --> RUNS["Workflows run<br/>build · test · lint · scans"]
  RUNS --> GATE["Required checks<br/>enforced by ruleset"]
  GATE --> REVIEW["Reviewer approves<br/>qualitative judgement"]
  REVIEW --> MERGE["Merge"]

Align evaluation criteria with development intent

An evaluation that measures the wrong thing rewards the wrong behaviour. Criteria must describe the intent of the task, and each should map to a signal that can actually observe it.

Generate evaluation signals with automated scanning

GitHub's security tools produce objective signals that fit straight into a required-check model.

name: Security gates
on:
  pull_request:
    branches: [main]

permissions:
  contents: read

jobs:
  dependency-review:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/dependency-review-action@v5
        with:
          fail-on-severity: high

  codeql:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      security-events: write
    steps:
      - uses: actions/checkout@v7
      - uses: github/codeql-action/init@v4
        with:
          languages: javascript-typescript
      - uses: github/codeql-action/analyze@v4

Cloud agent's built-in validation

Copilot cloud agent evaluates its own work before finishing, and tries to fix what it finds:

• CodeQL checks the code it generated for security issues.
• Newly added dependencies are checked against the GitHub Advisory Database for malware and for High or Critical CVSS vulnerabilities.
• Secret scanning looks for API keys, tokens and other secrets.
• Copilot code review gives it a second opinion.

This does not require a GitHub Secret Protection, Code Security or Advanced Security licence. Details appear in the session log, and a repository administrator can switch individual tools off under Settings → Copilot → Cloud agent → Validation tools — for example to avoid duplicating another security product. It is a first line of defence, not a replacement for your own required checks.

4.2 Analyze agent failures and identify root causes

Identify failures with logs, plans, traces, outputs and artifacts

The first step is always to gather the evidence the platform already produced, then compare intent with results: intent lives in the issue and the plan; results live in the commits, outputs and logs.

Classify root causes

flowchart TB
  START["Agent output is wrong or incomplete"] --> Q1{"Did it understand<br/>the task correctly?"}
  Q1 -->|"no — wrong goal, ignored criteria"| REASON["Reasoning error<br/>fix: instructions · criteria · plan-first"]
  Q1 -->|"yes"| Q2{"Did its tool calls<br/>succeed as intended?"}
  Q2 -->|"no — wrong command, misused workflow"| TOOL["Tool misuse<br/>fix: tool allowlist · skills · hooks"]
  Q2 -->|"yes"| Q3{"Was its information<br/>current and consistent?"}
  Q3 -->|"no — stale PR, conflicting docs"| CONTEXT["Context issue<br/>fix: single source of truth · update branch"]
  Q3 -->|"yes"| ENV["Environment issue<br/>fix: setup steps · firewall · secrets · runner"]

4.3 Tune agent behaviour based on evaluation results

flowchart LR
  RUN["Run the agent"] --> OBSERVE["Observe<br/>logs · traces · artifacts"]
  OBSERVE --> CLASSIFY["Classify<br/>the root cause"]
  CLASSIFY --> FIX["Fix one thing<br/>instructions · memory · tools"]
  FIX --> RERUN["Re-run and compare"]
  RERUN --> OBSERVE

Revise instructions, workflows or constraints

• Put build, test and validation commands in .github/copilot-instructions.md — an agent that can validate its own changes produces PRs that merge faster.
• Add path-specific instructions where one area keeps going wrong, and say what to do when requirements are ambiguous.
• Tighten a custom agent's prompt and description; split an over-broad agent into narrower ones.
• Make tasks smaller, with explicit acceptance criteria and file directions; move risky work to plan-first.
• Fix the workflow — triggers, required checks, a missing evidence upload.

# .github/copilot-instructions.md
This is a TypeScript monorepo using pnpm workspaces.

## Build and validate before every commit
- Install: pnpm install --frozen-lockfile
- Build:   pnpm -r build
- Test:    pnpm -r test   (all tests must pass; do not skip or delete tests)
- Lint:    pnpm lint

## Boundaries
- Never edit files under .github/workflows/ or infra/.
- Database migrations live in packages/db/migrations and are append-only.
- If a requirement is ambiguous, stop and ask in the PR instead of guessing.

Refine memory usage

• Delete wrong or misleading Copilot Memory facts (repository owners, Settings → Copilot → Memory).
• Move repeated corrections into instructions so they are authoritative and reviewed.
• Remove conflicting instruction files and keep one source of truth per topic.
• Update the issue or PR description when decisions change, so resumed sessions re-anchor correctly.

Refine tool usage and tool access

- name: Summarize the agent run for reviewers
  if: always()
  run: |
    {
      echo "## Agent evaluation"
      echo "- Commit: ${{ github.sha }}"
      echo "- Tests: $(jq -r '.numPassedTests' results.json) passed, $(jq -r '.numFailedTests' results.json) failed"
      echo "- Coverage delta: $(cat coverage-delta.txt)"
    } >> "$GITHUB_STEP_SUMMARY"

- uses: actions/upload-artifact@v7
  if: always()
  with:
    name: evaluation-${{ github.run_id }}
    path: |
      results.json
      coverage/