Plan and manage an Azure AI solution

1.1 Choose the appropriate Foundry services

This objective is pure decision-making, and it is the highest-density source of "which service?" questions in the whole exam. The diagram below is the map; the tables underneath are the tie-breakers.

flowchart TD
  Q["What does the workload need to do?"]
  Q --> G["Generate or reason<br/>over free text"] --> GM["Foundry Models<br/>LLM / SLM / reasoning"]
  Q --> R["Answer from<br/>private content"] --> RA{"How much content,<br/>how complex?"}
  RA -- "A handful of files,<br/>Foundry owns the lifecycle" --> FS["File search tool<br/>+ vector store"]
  RA -- "Enterprise corpus,<br/>filters, permissions, scale" --> AS["Azure AI Search index"]
  RA -- "Many sources, one reusable,<br/>permission-aware layer" --> IQ["Foundry IQ<br/>knowledge base"]
  Q --> P["Turn messy files into<br/>clean structured data"] --> CU["Content Understanding"]
  Q --> T["Bounded NLP / speech<br/>/ vision task"] --> FT["Foundry Tools<br/>Language · Speech · Translator · Vision"]
  Q --> A["Take actions,<br/>multi-step"] --> AG["Foundry Agent Service<br/>+ Toolbox"]
  Q --> W["Public web facts"] --> WS["Web search tool /<br/>Grounding with Bing"]

Choosing a model for each task

The trade-off table lives on the Foundations page; what follows is the part the exam actually phrases as scenarios.

Choosing a retrieval and indexing method

"Choose an appropriate method for retrieval and indexing" is its own bullet in the outline. Four options, and they are not interchangeable:

Memory, tools and knowledge for agents

Three different things that scenarios deliberately blur:

Conversation state

The transcript of the current interaction, held server-side as a conversation of items. Pass the conversation id on each call and history is maintained for you. You can bring your own Azure Cosmos DB for conversation state when compliance requires you to own the store.

Semantic memory

Durable facts and summaries extracted from conversations and stored by scope, then retrieved in later sessions. This is what "remembers the user across sessions" means — it is deliberately not the full transcript. Extraction is asynchronous, so a write is not immediately readable.

Knowledge

Grounding content the agent retrieves from: file search, Azure AI Search, Foundry IQ, SharePoint, Fabric, Work IQ, or the public web. Knowledge is read; memory is learned.

Tools

Actions the agent can take: functions, OpenAPI endpoints, MCP servers, code interpreter, browser automation, agent-to-agent calls. Curated into a toolbox for reuse and central governance.

1.2 Set up AI solutions in Foundry

Designing the Azure infrastructure

A defensible default enterprise topology, and the reasoning behind each choice:

1. One Foundry resource per security/governance boundary — typically per environment (dev, test, prod) or per business unit with genuinely different compliance needs. Not one per team: that multiplies networking and policy work for no benefit.
2. One project per team or application inside that resource. Projects inherit the parent's security and connections and can carry their own RBAC.
3. Model deployments at resource scope, shared by the projects. Size them with quota in mind — see quota and rate limits.
4. Connections for everything external: Azure AI Search for retrieval, Storage for documents, Application Insights for traces, Cosmos DB if you must own conversation state. Authenticate with managed identity, and remember the role assignment on the target.
5. Private networking where required: private endpoints on the Foundry resource and every connected service, public network access disabled. Note that key-based auth is not supported with private networking for the Azure AI Search tool — use the project managed identity.
6. Guardrails and Azure Policy applied at resource scope so every project inherits them.
7. Observability wired up from day one: Application Insights connected, tracing on, an evaluation dataset in source control.

Configuring model and agent deployments

A deployment binds a model, a version, a deployment type (the SKU) and a capacity allocation. Capacity is the lever that sets your rate limits.

# Deploy a model with the Azure CLI. The SKU name is the deployment TYPE.
az cognitiveservices account deployment create \
  --name my-foundry \
  --resource-group rg-ai103 \
  --deployment-name gpt-5-mini \
  --model-name gpt-5-mini \
  --model-version "2026-04-01" \
  --model-format OpenAI \
  --sku-name GlobalStandard \
  --sku-capacity 50          # capacity units -> TPM and, proportionally, RPM

Other settings the exam expects you to know exist:

• Version upgrade policy — auto-update to default, upgrade when expired, or pin. Models retire; pinning without a plan means an outage on retirement day.
• Guardrail assignment — every deployment gets one; the default is Microsoft.DefaultV2.
• Dynamic quota preview — lets a standard deployment opportunistically exceed its TPM when spare capacity exists. It can only ever raise the effective limit, never lower it, and the extra tokens bill at normal rates.
• Agent deployments — prompt agents are created as agent versions through the portal or SDK; hosted agents ship as a container image or a .zip that Foundry builds. Publishing promotes an agent to a managed resource with a stable endpoint.

CI/CD for Foundry projects

flowchart LR
  DEV["Developer<br/>branch"] --> PR["Pull request"]
  PR --> CI["CI: lint, unit tests,<br/>evaluate agent against<br/>a golden dataset"]
  CI -->|"quality gate passes"| BICEP["Deploy infra<br/>Bicep / Terraform:<br/>resource, project,<br/>deployments, connections"]
  BICEP --> AGENT["Create or update<br/>agent version via SDK"]
  AGENT --> TEST["Test project<br/>smoke + safety evals"]
  TEST -->|"promote"| PROD["Prod project<br/>publish agent,<br/>stable endpoint"]
  PROD --> MON["Continuous evaluation<br/>+ Azure Monitor alerts"]
  MON -.->|"regression"| DEV

The pieces the outline expects:

• Infrastructure as code — Bicep, ARM or Terraform for the resource, projects, deployments and connections. Deployments can be created with the Azure CLI or Bicep.
• Agent definitions in code — define prompt agents with the SDK or REST in the pipeline rather than clicking in the portal, so they get version control, code review and automated rollout.
• Automated evaluation as a gate — run evaluators against a golden dataset on every PR and fail the build on regression. This is the distinctive part.
• Agent versioning and rollback — versions are snapshotted automatically; you can roll back or compare.
• Environment promotion — separate dev/test/prod projects, promote by publishing the tested agent version.
• azd for scaffolding and deploying hosted agents from the command line.

1.3 Manage, monitor and secure AI systems

Quota, rate limits, scaling and cost

Quota is assigned to your subscription, per region, per model, per deployment type, in units of tokens per minute (TPM). You spend that quota by assigning capacity to deployments; a proportional requests per minute (RPM) limit comes with it (the ratio varies by model).

import time
import random
from openai import RateLimitError

def call_with_backoff(openai, *, max_attempts=5, **kwargs):
    """Exponential backoff with jitter.

    A 429 is not always a quota problem: it can be capacity throttling or a
    temporary rate-limit adjustment. Honour retry-after-ms when the service
    sends it, and only request more quota when the message really says the
    deployment's rate limit was exceeded.
    """
    for attempt in range(max_attempts):
        try:
            return openai.responses.create(**kwargs)
        except RateLimitError as err:
            if attempt == max_attempts - 1:
                raise
            retry_after_ms = getattr(err, "response", None) and err.response.headers.get("retry-after-ms")
            delay = int(retry_after_ms) / 1000 if retry_after_ms else (2**attempt) + random.random()
            time.sleep(delay)

Cost levers, cheapest first

• Right-size the model — an SLM or a mini tier for routing, extraction and classification; the big model only where it earns its cost.
• Batch deployment types for asynchronous work — roughly 50% cheaper.
• Prompt caching and shorter system prompts — input tokens are the silent majority of spend in RAG.
• Trim retrieval — fewer, better chunks beats stuffing the context window; it also improves groundedness.
• Tool search in a toolbox — stops every tool definition being sent on every request once you have dozens of tools.
• Provisioned throughput — for steady high volume it is cheaper and more predictable than standard at scale.
• Token limits via the AI Gateway — per-project, per-deployment TPM caps and total token quotas to stop one team consuming everything.

Monitoring performance, drift, safety and grounding

Foundry observability has three legs, and the exam expects you to name the right one: tracing (what happened), evaluation (was it any good), monitoring (is it still good in production).

Ingestion quality, index health and relevance

A separate bullet in the outline, and easy to overlook because it is search operations rather than model operations:

• Indexer execution history in Azure AI Search — success/failure counts, per-document errors and warnings, and the item-level failure thresholds that decide whether an indexer run fails.
• Index size, document count and storage/vector quota against the service tier — a silently full index stops accepting documents.
• Freshness — change-detection policy and indexer schedule; stale grounding data produces confidently wrong answers.
• Relevance — measure it, do not assume it. Evaluate retrieval separately from generation with document retrieval and relevance evaluators; hybrid + semantic ranking is the benchmark-backed baseline.
• Grounding quality — the groundedness evaluator tells you whether answers are supported by the retrieved context. A groundedness drop with a healthy model usually means the retrieval broke, not the model.

Security: identity, networking and credentials

1.4 Implement responsible AI

Guardrails and controls

A guardrail is a named collection of controls. Each control pairs three things: a risk to detect, an intervention point to scan, and an action to take. The classification models come from Azure AI Content Safety.

flowchart LR
  U["User input"] -->|"Control: user prompt attacks,<br/>hate, violence, PII…"| M["Model / agent"]
  M -->|"Control: tool call<br/>(agents, preview)"| TOOL["Tool"]
  TOOL -->|"Control: tool response<br/>(agents, preview)"| M
  M -->|"Control: output<br/>groundedness, protected material…"| OUT["Response to user"]

Responsible AI instrumentation

Guardrails are a runtime control. Evaluators are the measurement layer, and the outline names them explicitly under responsible AI.

"Detecting fabrications" in the outline maps to groundedness. Evaluations run at three lifecycle stages: base model selection (benchmarks and leaderboards), pre-production (your dataset, plus red teaming), and post-production (continuous, scheduled and alerted). Cluster analysis groups evaluation failures so you can see the pattern instead of reading individual rows.

Auditing, provenance and approvals

• Trace logging — OpenTelemetry traces in Application Insights are the audit trail of what the agent decided and which tools it called. Retain them to match your compliance requirement.
• Provenance metadata — citations from retrieval tools, source grounding and confidence scores from Content Understanding, and content credentials (C2PA) on generated media. "Where did this answer come from?" must have an answer.
• Approval workflows — human-in-the-loop gates before sensitive tool calls. In Agent Framework this is an approval-required tool that pauses the workflow and emits a request for a decision; in Foundry workflows it is the human in the loop pattern.
• Azure Policy at resource scope for organisation-wide rules, and Azure Monitor alerts on safety signals.

Governing agent behaviour