- 4.1 Apply language model text analysis — entities, topics, summaries, structured JSON, sentiment and tone, safety and sensitive content, translation, domain customisation
- 4.2 Implement speech solutions — speech to text and text to speech for agents, speech as an agent modality, custom speech, audio reasoning, speech translation
4.1 Apply language model text analysis
flowchart TD
Q["A text task arrives"] --> A{"Is it a solved,<br/>bounded NLP task?"}
A -- "Yes: PII, sentiment, language,<br/>key phrases, health, NER" --> L["Azure Language<br/>stable API · cheap · auditable<br/>containers available"]
A -- "No: bespoke schema, domain<br/>reasoning, house style" --> G["Generative prompting<br/>+ structured outputs"]
A -- "Translation" --> T{"What matters most?"}
T -- "Coverage, cost, documents,<br/>glossary control" --> TR["Azure Translator<br/>(+ Custom Translator)"]
T -- "Tone, context, mixed<br/>instructions in one pass" --> LLM["LLM-powered<br/>translation flow"]
L -.->|"gaps"| G
G -.->|"needs determinism"| L Two ways to do everything, and how to choose
| Azure Language (Foundry Tool) | Generative prompting | |
|---|---|---|
| Output | Fixed schema, category labels, confidence scores, character offsets | Whatever you define — free text or a JSON Schema |
| Determinism | High and versionable (you can pin the model version) | Variable; constrain with temperature 0 and structured outputs |
| Cost at volume | Low and predictable per document | Token-based; grows with prompt and content length |
| Auditability | Strong — offsets and scores per entity | Needs you to build it |
| Flexibility | Only what the service supports | Anything you can describe |
| Data residency / offline | Containers, including disconnected environments | Cloud model call |
| Best for | PII redaction, language detection, sentiment at scale, health text, compliance pipelines | Bespoke schemas, domain reasoning, summarising to a house style, mixed tasks in one call |
The deciding words
"Redact before storage", "must run on-premises", "confidence score per entity", "regulatory evidence", "millions of documents, cost per document matters" → Azure Language. "Summarise in our compliance department's format", "extract these 9 bespoke fields", "decide whether to escalate and explain why" → generative prompting with structured outputs. When a scenario needs both determinism and a bespoke schema over documents, the answer is usually Content Understanding (see Skill 5).
Azure Language capabilities
| Capability | Returns | Notes |
|---|---|---|
| PII detection | Detected entities with categories, offsets and confidence, plus a redacted string | Text, conversation and document variants. Synthetic-replacement anonymisation is in preview. Recent additions include date of birth, licence plate, sort code and VIN |
| Named entity recognition | People, places, organisations, quantities, dates and more | Prebuilt NER for standard categories; custom NER to train your own categories from labelled text |
| Sentiment analysis & opinion mining | Positive / neutral / negative with scores, per document and per sentence; opinion mining links opinions to targets | "The food was great but the service was slow" needs opinion mining to separate the two targets |
| Key phrase extraction | The main talking points as phrases | The lightweight "what is this about?" — closest thing to topic extraction in the service |
| Language detection | Language, ISO code, confidence | Route multilingual input before anything else |
| Summarisation | Extractive (selected sentences) and abstractive (newly written), plus conversation summarisation with issue/resolution aspects | Abstractive summarisation is powered by fine-tuned Phi models |
| Text analytics for health | Clinical entities, relations, assertions and UMLS links | Available as a container |
| Custom text classification | Your own single- or multi-label categories | Train from labelled documents |
| Conversational language understanding (CLU) | Intents and entities from utterances | The successor to LUIS |
| Question answering | Answers from a curated knowledge base | Custom question answering over FAQs and documents |
from azure.core.credentials import AzureKeyCredential
from azure.ai.textanalytics import TextAnalyticsClient
client = TextAnalyticsClient(endpoint=LANGUAGE_ENDPOINT, credential=AzureKeyCredential(KEY))
# PII detection returns a redacted string plus per-entity detail and a
# confidence score — deterministic and auditable, which a prompt is not.
result = client.recognize_pii_entities(
documents=[transcript],
language="en",
categories_filter=["Person", "PhoneNumber", "Email", "CreditCardNumber"],
)[0]
print(result.redacted_text)
for entity in result.entities:
print(entity.category, entity.text, round(entity.confidence_score, 2)) Azure Language as an agent tool
Azure Language exposes an MCP server preview so agents can call
NER, text analytics for health, CLU, custom question answering, language detection, sentiment, summarisation, key
phrase extraction and PII redaction as tools. It is reachable at
https://<resource>.cognitiveservices.azure.com/language/mcp, and there is a local MCP server you
can host yourself. Adding it needs a connection in the Foundry project so the agent can
authenticate. Note that network-secured Foundry projects can require publicly accessible MCP servers.
Entities, topics, summaries and structured JSON from prompts
When the shape is bespoke, one prompt with a schema replaces several service calls — and is usually what the exam means by "extract entities, topics, summaries, and structured JSON outputs by using generative prompting".
from pydantic import BaseModel
from typing import Literal
class ComplaintAnalysis(BaseModel):
summary: str
product: str
sentiment: Literal["positive", "neutral", "negative"]
topics: list[str]
contains_personal_data: bool
escalate: bool
# One call replaces separate entity, topic, sentiment and summarisation calls —
# and the schema guarantees the shape for downstream code.
result = openai.responses.parse(
model="gpt-5-mini",
input=[
{"role": "system", "content":
"Analyse the customer complaint. Escalate only when there is a "
"regulatory, safety or contractual breach. Topics must come from: "
"billing, delivery, quality, support, privacy."},
{"role": "user", "content": complaint_text},
],
text_format=ComplaintAnalysis,
)
analysis = result.output_parsed Why structured outputs beat asking for JSON
"Respond in JSON" produces JSON most of the time; structured outputs constrain generation to the schema, so types, enums and required fields are guaranteed and your code never has to strip a markdown fence or retry a parse failure. If an exam option offers "instruct the model to return JSON" against "use structured outputs / a response schema", the schema wins.
Sentiment, tone, safety and sensitive content
| Requirement | Use |
|---|---|
| Positive / neutral / negative at document and sentence level, with scores | Azure Language sentiment analysis |
| Which aspect the opinion is about ("the food" vs "the service") | Opinion mining |
| Tone, emotion, frustration, urgency, politeness — nuance beyond three labels | Generative prompting, ideally with an enum in a schema |
| Hate, sexual, self-harm, violence in content | Content Safety / guardrails — a safety control, not a sentiment task |
| Personal or sensitive data in the text | PII detection for redaction; the PII guardrail control preview to block at the model boundary |
| Jailbreak attempts in user input | User prompt attack control (Prompt Shields) |
Sentiment is not safety
A classic distractor: a scenario about "detecting abusive messages in a community forum" offers sentiment analysis. Negative sentiment is not harm. Abuse, hate and threats are Content Safety harm categories. Conversely, "how do customers feel about the new pricing?" is sentiment, not a safety control.
Translation
| Azure Translator | LLM-powered translation flow | |
|---|---|---|
| Coverage | 100+ languages, mature and consistent | Strong on major languages, weaker on low-resource ones |
| Cost at volume | Per character, low and predictable | Per token, higher |
| Documents | Document Translation preserves layout and formatting | You rebuild the document yourself |
| Terminology control | Custom Translator (trained on your bilingual data, referenced by category ID) and glossaries | Instructions and few-shot examples in the prompt |
| Context and tone | Sentence-level; limited awareness of surrounding context | Excellent — can preserve register, adapt tone, and translate plus summarise in one pass |
| Choose it when | High volume, documents, regulated terminology, predictable cost | Context and tone matter, or translation is one step in a larger reasoning task |
Customising outputs for domain tasks
Escalating in cost and effort — the exam expects you to reach for the cheapest one that works:
- Instructions and few-shot examples. Define the format, vocabulary and rules in the system prompt. Solves most "in our house style" requirements.
- Structured outputs. Lock the schema so downstream systems can rely on it.
- Grounding. Supply the domain's policies, glossary or reference documents through retrieval so the model uses your definitions rather than its priors.
- Custom models in Azure Language. Custom NER and custom text classification, trained on labelled data, when you need deterministic domain categories at scale. (Custom sentiment, custom text analytics for health and custom summarisation were retired in January 2025 in favour of generative approaches.)
- Custom Translator for domain terminology in translation.
- Fine-tuning a model. Last resort: highest cost and effort, needs both control-plane and data-plane permissions (only Foundry Owner has both), and can be evaluated on a Developer deployment type.
4.2 Implement speech solutions
flowchart LR MIC["Microphone"] --> STT["Speech to text<br/>(custom speech for<br/>jargon and accents)"] STT --> AG["Agent / model<br/>reasoning + tools"] AG --> TTS["Text to speech<br/>standard · HD · MAI · custom voice"] TTS --> SPK["Speaker"] MIC -.->|"or: one low-latency,<br/>speech-native pipeline"| VL["Voice Live API<br/>barge-in · avatars ·<br/>bring your own agent"] VL -.-> SPK
Speech to text
| Mode | Use for |
|---|---|
| Real-time | Live conversation, dictation, agent voice input |
| Batch transcription | Large volumes of recorded audio in storage, asynchronous |
| Fast transcription | Quick synchronous transcription of a whole file |
| Custom speech | A model adapted to your domain vocabulary, product names, acronyms, accents or noisy audio. Deployed to its own endpoint; point endpoint_id at it |
| Diarisation | Who spoke when, in multi-speaker audio |
| Language identification | Detect the spoken language, including continuous switching |
import azure.cognitiveservices.speech as speechsdk
speech_config = speechsdk.SpeechConfig(endpoint=SPEECH_ENDPOINT)
speech_config.speech_recognition_language = "en-GB"
# A custom speech model handles product names, acronyms and accents that the
# base model mis-transcribes. Point the config at your deployed endpoint id.
speech_config.endpoint_id = "your-custom-speech-endpoint-id"
recognizer = speechsdk.SpeechRecognizer(speech_config=speech_config)
result = recognizer.recognize_once_async().get()
# Synthesis: pick a voice by name; SSML gives control over prosody and style.
speech_config.speech_synthesis_voice_name = "en-GB-SoniaNeural"
synthesizer = speechsdk.SpeechSynthesizer(speech_config=speech_config)
synthesizer.speak_text_async("Your order has shipped.").get() Text to speech
| Voice type | What it is | Access |
|---|---|---|
| Standard neural voices | Prebuilt, 100+ languages and locales; all neural voices are multilingual and fluent in their own language and English | Generally available |
HD voices (DragonHDLatestNeural) | Higher fidelity, accept a temperature for prosodic variation | Generally available |
| MAI voices preview | Ultra-fast, low-latency expressive TTS optimised for real-time Voice Live | Preview |
| OpenAI voices | OpenAI TTS voices available through Azure Speech | Generally available |
| Custom neural voice | A brand voice trained from your voice talent's recordings. Lite for proofs of concept; pro for studio-quality, multi-emotion, cross-lingual | Limited Access — application plus recorded consent from the voice talent |
| Personal voice | A voice model from a short sample, in seconds, speaking close to 100 languages | Limited Access. Outputs are watermarked for synthetic-speech detection |
| TTS avatar | Synthetic video of an avatar speaking; prebuilt or custom, real-time or batch | Custom avatar is Limited Access |
SSML is how you control delivery: voice selection, speaking styles, prosody, pauses, and how numbers, dates, currency and character strings are spoken. Batch synthesis handles long-form text.
<speak version="1.0" xmlns="http://www.w3.org/2001/10/synthesis"
xmlns:mstts="https://www.w3.org/2001/mstts" xml:lang="en-GB">
<voice name="en-GB-SoniaNeural">
<mstts:express-as style="empathetic" styledegree="1.4">
I'm sorry your delivery was late.
</mstts:express-as>
<break time="400ms"/>
Your refund of <say-as interpret-as="currency">GBP 24.99</say-as>
has been issued. Reference
<say-as interpret-as="characters">AB1274</say-as>.
<prosody rate="-8%">Thank you for your patience.</prosody>
</voice>
</speak> Speech as an agent modality: the Voice Live API
The chained architecture (STT → model → TTS) is easy to reason about but accumulates latency at every hop and loses prosody. Voice Live is the speech-native alternative: one low-latency API for real-time voice conversation, with interruption handling, optional avatars, function calling, and the ability to bring your own agent from Agent Service or your own deployed model from Foundry Models.
| Chained STT → model → TTS | Voice Live API | |
|---|---|---|
| Latency | Sum of every hop | Optimised end to end for real-time conversation |
| Interruption / barge-in | You implement it | Built in |
| Prosody and emotion | Lost at the text boundary | Preserved through the pipeline |
| Control | Full — swap any component | Configured through the API |
| Choose when | Asynchronous or batch audio, or you need a specific component | Live voice agents, phone-style experiences, avatars |
Voice scenarios and their answers
- "Customers must be able to interrupt the assistant mid-sentence" → Voice Live.
- "The assistant mis-hears our product names" → custom speech model.
- "The assistant must sound like our brand ambassador" → custom neural voice (Limited Access, consent required).
- "Give each user their own synthesised voice from a short sample" → personal voice (Limited Access, watermarked).
- "Transcribe ten thousand archived call recordings overnight" → batch transcription, not real-time.
- "Identify who said what in a meeting recording" → diarisation.
Multimodal reasoning from audio
Distinct from transcription: a multimodal model can take audio directly as input and reason over it — answering questions about a recording, judging tone, or summarising a call without a separate transcription step. Content Understanding also has an audio analyzer that transcribes and then applies your schema, and its post-call analytics scenario is exactly this: transcribe, extract fields, generate a summary and a sentiment classification in one pass.
Audio models and guardrails
The guardrail system applies to Foundry Models sold by Azure except prompts and completions processed by audio transcription models. If a scenario requires content moderation over voice input, apply the guardrail to the downstream model or agent that consumes the transcript, not to transcription itself.
Speech translation
| Approach | What it does |
|---|---|
| Speech translation (Speech SDK) | Speech to translated text, to one or many target languages; also direct speech-to-speech |
| Live Interpreter | Continuously identifies the spoken language with no input language set, and delivers low-latency speech-to-speech translation in a natural voice that preserves the speaker's style. Needs personal voice access and a supporting region |
| Custom translation in speech | Integrates Custom Translator via a category ID; requires a multi-service resource for the full feature set |
| LLM-powered flow | Transcribe, then translate with a model when tone, context or a combined task matters |
Skill 4 rapid-fire review
- Deterministic, auditable, containerisable, cheap at volume → Azure Language. Bespoke schema or reasoning → prompting + structured outputs.
- Sentence-level sentiment plus the target of the opinion → opinion mining.
- Summarisation is extractive (selected sentences) or abstractive (newly written, Phi-powered).
- Custom sentiment, custom text analytics for health and custom summarisation were retired in January 2025.
- Containers exist for sentiment, NER, PII, health, custom NER and summarisation; some run disconnected.
- Translator for volume, documents and glossaries; Custom Translator for domain terminology via a category ID.
- Custom speech fixes recognition; custom neural voice changes how it sounds — both are different problems.
- Custom neural voice, personal voice and custom avatar are Limited Access; personal voice output is watermarked.
- Voice Live for low-latency, interruptible voice agents with optional avatars.
- Guardrails do not apply to audio transcription models.