01 — Implement & Manage an Analytics Solution
Official Exam Weight: 30–35% 📁 ← Back to Home
🗺 Domain Overview
mindmap
root((Implement & Manage))
1.1 Workspace Settings
Spark Settings
Domain Settings
OneLake Settings
Apache Airflow Settings
1.2 Lifecycle Management
Version Control
Database Projects
Deployment Pipelines
Variable Libraries
1.3 Security & Governance
Workspace Access Controls
Item-Level Access
Row-Level Security
Column-Level Security
Object-Level Security
Folder/File-Level Security
Dynamic Data Masking
Sensitivity Labels
Endorsement
Audit Logs
OneLake Security
1.4 Orchestration
Dataflow Gen2 vs Pipeline vs Notebook
Schedules & Event Triggers
Orchestration Patterns
Parameters & Dynamic Expressions
⚙️ 1.1 Configure Microsoft Fabric Workspace Settings
Spark Workspace Settings
| Setting | Description | Impact |
|---|---|---|
| Default runtime | Spark runtime version (e.g., Runtime 1.2, 1.3) | Determines Spark, Python, R, Delta Lake versions |
| Starter pools | Pre-warmed Spark pools for fast start (best-effort) | Sessions typically start in seconds instead of ~3 min |
| Custom pools | Configure node size, autoscale, dynamic allocation | Control compute for specific workloads |
| Custom live pools | Custom pools kept hydrated during a scheduled window, with environment libraries pre-installed | Predictable ~5 sec session start for scheduled / latency-sensitive jobs |
| Environment | Attach libraries, Spark properties, resource files | Consistent dependencies across notebooks |
| Resource profile | Predefined Spark/Delta config set (spark.fabric.resourceProfile) |
Tunes write vs read behaviour, incl. whether V-Order is on |
| Native execution engine | Hardware-accelerated Spark execution | Up to 4x faster for supported operations |
| High concurrency mode | Share Spark sessions across notebooks | Saves CU consumption when multiple users work simultaneously |
| Automatic log retention | Spark logs and event log retention | Debugging and monitoring |
Resource profiles:
| Profile | Optimized for | V-Order |
|---|---|---|
writeHeavy (default for new workspaces) |
Ingestion, ETL, streaming writes | ❌ Disabled |
readHeavyForSpark |
Frequent Spark reads, interactive queries | ❌ Disabled (Optimize Write on) |
readHeavyForPBI |
Power BI Direct Lake queries on Delta tables | ✅ Enabled |
custom |
Fully user-defined Spark configs | User-defined |
1
2
# Override the workspace/environment profile for this session
spark.conf.set("spark.fabric.resourceProfile", "readHeavyForPBI")
flowchart TD
WS["📁 Fabric Workspace"] --> SPARK["⚙️ Spark Settings"]
SPARK --> POOL["🏊 Pool Configuration"]
POOL --> STARTER["Starter Pool\n(Pre-warmed, fast start)"]
POOL --> CUSTOM["Custom Pool\n(User-defined size)"]
SPARK --> ENV["📦 Environment"]
ENV --> LIBS["Libraries\n(PyPI, Conda, .whl, .jar)"]
ENV --> PROPS["Spark Properties\n(spark.conf settings)"]
SPARK --> RUNTIME["🔧 Runtime Version"]
Exam Caveat ⚠️:
- Starter pools and custom pools are billed only while a Spark session is active — idle pool time is not billed
- Starter pools are best-effort — for guaranteed fast starts on scheduled jobs, use a custom live pool
- Creating custom pools requires the workspace Admin role (plus capacity admin permission to size them)
- New workspaces default to the
writeHeavyresource profile, so V-Order is off by default — switch toreadHeavyForPBIfor Direct Lake–heavy tables- Environments can be shared across multiple notebooks within a workspace
- The native execution engine does not support all Spark operations — unsupported ops fall back to standard Spark
Domain Workspace Settings
Domains in Fabric are a way to group workspaces by business area (e.g., Finance, Marketing, Engineering).
| Feature | Description |
|---|---|
| Domain assignment | Assign a workspace to a specific domain |
| Domain roles | Domain admins, domain contributors |
| Data governance | Policies and endorsement can be applied at domain level |
| Discovery | Users can discover workspaces within their domain in the data hub |
OneLake Workspace Settings
| Setting | Scope | Description |
|---|---|---|
| Shortcut cache | Workspace (OneLake tab) | Caches files read through external shortcuts to cut egress cost and latency. Retention 1–28 days (reset on each access); files > 1 GB aren’t cached; Reset cache clears it |
| Users can access data stored in OneLake with apps external to Fabric | Tenant (admin portal) | When off, external tools using ADLS Gen2 APIs / OneLake file explorer are blocked; Fabric engines (Spark, SQL) still work |
Exam Caveat ⚠️:
- Shortcut caching supports GCS, Amazon S3, S3-compatible, and on-premises data gateway shortcuts — not ADLS Gen2
- Blocking external apps is a tenant setting, not a workspace one — and it must stay on to use shortcuts with OneLake security
Apache Airflow Workspace Settings
Apache Airflow jobs run Python DAGs on a managed Airflow runtime inside Fabric. Configure the default runtime in Workspace settings → Data Factory → Apache Airflow runtime settings.
| Property | Starter Pool (default) | Custom Pool |
|---|---|---|
| Startup | Instant runtime | Created in a stopped state, then always on |
| Idle behaviour | Auto-shutdown after 20 min of inactivity — no idle cost | Always on until manually paused — constant base cost |
| Size | Fixed (Large) | Small or Large compute nodes, extra nodes (≈3 more workers each), autoscale |
| Base CU | 10 CU (Large) | Small 5 CU / Large 10 CU (recommended minimum F8 / F16) |
| Best for | Development, testing, infrequent jobs | Production — Airflow scheduler runs 24/7 |
Exam Caveat ⚠️:
- Each Apache Airflow job gets its own isolated pool — pools aren’t shared between jobs
- If the workspace setting “Customize compute configurations for items” is off, every Airflow job uses the starter pool
- Need DAGs to run on an Airflow schedule unattended? → Custom pool (a starter pool shuts down when idle)
🔄 1.2 Implement Lifecycle Management in Fabric
Version Control (Git Integration)
Fabric supports Git integration for version-controlling workspace items.
flowchart LR
WS["📁 Fabric Workspace"] <-->|"Sync"| REPO["🔧 Git Repository\n(Azure DevOps or GitHub)"]
REPO --> BRANCH["🌿 Branches\n(Feature branches,\nPR workflow)"]
BRANCH --> MERGE["🔀 Merge / PR\n(Code review)"]
MERGE --> MAIN["✅ Main Branch\n(Production-ready)"]
Supported Git providers:
| Provider | Support |
|---|---|
| Azure DevOps | Supported (cloud only) — Microsoft Entra ID credentials |
| GitHub | Supported (cloud only) — personal access token |
| GitHub Enterprise | Supported (cloud only) |
| Other Git / on-prem servers | Not supported natively |
Items that support Git integration (data engineering highlights):
| Area | Items |
|---|---|
| Data Engineering | Notebooks, Spark Job Definitions, Lakehouses (metadata), Environments, User Data Functions, GraphQL |
| Data Factory | Pipelines, Dataflow Gen2, Copy job, Mirrored databases, Apache Airflow jobs (preview) |
| Data Warehouse / Databases | Warehouses, SQL databases |
| Real-Time Intelligence | Eventhouses, KQL databases, KQL querysets, Eventstreams, Real-Time Dashboards, Activator |
| Power BI | Semantic models, Reports (preview) |
| CI/CD | Variable libraries |
Full, current list: Git integration — supported items.
Exam Caveat ⚠️:
- Git integration syncs item definitions (metadata), not the data itself
- Each workspace connects to one branch at a time — you switch branches at workspace level, or branch out to a new feature workspace
- Unsupported items in a connected workspace are ignored, not deleted
- Conflicts must be resolved before syncing
Database Projects
Database projects provide a code-first, schema-as-code approach for managing Warehouse schemas.
| Feature | Description |
|---|---|
| SQL project | Version-controlled set of SQL scripts defining schema |
| Schema compare | Compare project schema vs deployed warehouse |
| Publish | Deploy schema changes from project to warehouse |
| IDE support | VS Code with SQL Database Projects extension |
Deployment Pipelines
Deployment pipelines enable CI/CD for Fabric — promoting items across environments.
flowchart LR
DEV["🟢 Development\n(Workspace)"] -->|Deploy| TEST["🟡 Test\n(Workspace)"]
TEST -->|Deploy| PROD["🔴 Production\n(Workspace)"]
| Feature | Description |
|---|---|
| Stages | Up to 10 stages (typically: Dev → Test → Prod) |
| Deployment rules | Override data sources, parameters per stage |
| Selective deployment | Deploy individual items or all items |
| Auto-binding | Deployed items maintain relationships |
| Comparison view | See differences between stages before deploying |
Exam Caveat ⚠️:
- Deployment pipelines deploy item definitions, not data
- You need at least Contributor on both source and target workspaces
- Deployment pipelines and Git integration can be used together — Git for code versioning, pipelines for environment promotion
Variable Libraries
A variable library is a workspace item that holds environment-specific values (connection strings, lakehouse IDs, file paths) outside item definitions, with one value set per stage.
| Feature | Description |
|---|---|
| Value sets | Dev / Test / Prod values for the same variables; exactly one active value set per workspace |
| Consumers | Pipelines, Lakehouse shortcuts, Notebooks (notebookutils.variableLibrary), Dataflow Gen2 (CI/CD), Copy job, User data functions, Eventstream |
| CI/CD | Git-tracked and deployable; deployment pipelines can activate the right value set per stage — the active set itself isn’t overwritten by deploys |
Exam Caveat ⚠️: Prefer variable libraries over hardcoded values or per-item deployment rules when the same items must point at different sources in each stage.
🔒 1.3 Configure Security & Governance
Security Layers in Fabric
graph TD
TENANT["🏢 Tenant-Level\n(Admin portal settings)"] --> WS["📁 Workspace-Level\n(Roles: Admin, Member,\nContributor, Viewer)"]
WS --> ITEM["📄 Item-Level\n(Share / Permissions per item)"]
ITEM --> DATA["📊 Data-Level Security"]
DATA --> RLS["🔐 Row-Level Security (RLS)"]
DATA --> CLS["🔐 Column-Level Security (CLS)"]
DATA --> OLS["🔐 Object-Level Security (OLS)"]
DATA --> FLS["📁 Folder/File-Level Security"]
DATA --> DDM["🎭 Dynamic Data Masking"]
Workspace-Level Access Controls
| Role | Permissions |
|---|---|
| Admin | Full control — manage settings, members, delete workspace |
| Member | Create, edit, delete items; share items; cannot manage workspace settings |
| Contributor | Create, edit, delete items; cannot share or manage members |
| Viewer | View items only; cannot edit, create, or delete |
Exam Caveat ⚠️: Workspace Viewer role does not automatically grant access to the underlying data — additional item-level or data-level permissions may be needed.
Item-Level Access Controls
You can share individual Fabric items (Lakehouse, Warehouse, Report, etc.) with specific users, granting:
| Permission | What It Grants |
|---|---|
| Read | View item metadata and content |
| ReadAll | Read all data in the item (bypasses data-level security for some items) |
| ReadData | Read data via SQL (Warehouse, SQL Analytics Endpoint) |
| Write | Edit the item |
| Reshare | Share the item with others |
Row-Level Security (RLS)
RLS restricts which rows a user can see based on filters.
Where RLS is supported:
| Item | RLS Method |
|---|---|
| Warehouse | T-SQL CREATE SECURITY POLICY + FUNCTION |
| SQL Analytics Endpoint | Same as Warehouse |
| Semantic Model | DAX filter expressions in Power BI |
| Lakehouse (via SQL) | Via SQL Analytics Endpoint |
| Lakehouse (all engines) | OneLake security role with a row filter (SQL predicate) on the Delta table |
1
2
3
4
5
6
7
8
9
10
11
-- Example: RLS in a Fabric Warehouse
CREATE FUNCTION dbo.fn_SecurityPredicate(@Region AS NVARCHAR(50))
RETURNS TABLE
WITH SCHEMABINDING
AS
RETURN SELECT 1 AS result
WHERE @Region = USER_NAME()
OR USER_NAME() = 'admin@contoso.com';
CREATE SECURITY POLICY RegionFilter
ADD FILTER PREDICATE dbo.fn_SecurityPredicate(Region) ON dbo.Sales;
Column-Level Security (CLS)
CLS restricts which columns a user can see.
1
2
3
-- Example: Grant SELECT on specific columns only
GRANT SELECT ON dbo.Employees(Name, Department, Title) TO [analyst@contoso.com];
-- The Salary column is NOT included — the user cannot see it
Exam Caveat ⚠️: CLS is supported in Warehouse and SQL Analytics Endpoint via T-SQL
GRANT/DENYon specific columns, and on Lakehouse tables via OneLake security (set columns to Hide in a role) — the latter applies across Spark, SQL, and Direct Lake.
Object-Level Security (OLS)
OLS controls access to entire objects (tables, views, columns, folders).
| Where | How |
|---|---|
| Warehouse / SQL Analytics Endpoint | T-SQL GRANT / DENY / REVOKE on tables, views, schemas |
| OneLake security (Lakehouse) | Role grants on specific tables, schemas, or folders |
| Semantic Model (Power BI) | OLS roles with none permission — objects become completely invisible |
Folder/File-Level Security (OneLake)
Folder/file-level access in a Lakehouse is implemented with OneLake security roles (see OneLake Security below).
| Feature | Description |
|---|---|
| Scope | Folders and files (Files section), tables and schemas (Tables section — tables are folders too) |
| Inheritance | Permissions granted on a folder apply to all its files and subfolders |
| Traversal | Access to a child lets the user list/traverse its parents — but not siblings |
Dynamic Data Masking (DDM)
DDM masks data at query time without changing stored data.
| Mask Function | Effect | Example |
|---|---|---|
| Default | Full mask based on data type | XXXX for strings, 0 for numbers |
Shows first char + XXX@XXX.com |
mXXX@XXXX.com |
|
| Partial | Shows prefix + padding + suffix | Cus-XXXX-er |
| Random | Random number in specified range | 42 (random) |
1
2
3
4
5
6
-- Example: Apply dynamic data masking in Warehouse
ALTER TABLE dbo.Customers
ALTER COLUMN Email ADD MASKED WITH (FUNCTION = 'email()');
ALTER TABLE dbo.Customers
ALTER COLUMN CreditCard ADD MASKED WITH (FUNCTION = 'partial(0,"XXXX-XXXX-XXXX-",4)');
Exam Caveat ⚠️: Users with
UNMASKpermission can see the original data. DDM is not a security boundary for users with elevated database permissions.
Sensitivity Labels
| Feature | Description |
|---|---|
| Source | Microsoft Purview Information Protection |
| Apply to | Any Fabric item (Lakehouse, Warehouse, Report, etc.) |
| Labels | E.g., Public, General, Confidential, Highly Confidential |
| Inheritance | Labels propagate downstream (e.g., Lakehouse → Report) |
| Encryption | Can enforce encryption on exported data |
Item Endorsement
| Endorsement Level | Meaning | Who Can Apply |
|---|---|---|
| Promoted | Recommended for use | Item owner, workspace member |
| Certified | Meets organization’s quality standards | Designated certifiers (admin-defined) |
| Master data | Authoritative source of truth | Designated certifiers |
Audit Logs
| Feature | Description |
|---|---|
| Source | Microsoft 365 Unified Audit Log |
| Access | Microsoft Purview compliance portal |
| Events | Item creation, deletion, access, sharing, data access |
| Retention | Audit (Standard): 180 days; longer with Audit (Premium) retention policies (up to 10 years) |
| Fabric activity log | Fabric/Power BI-only log for Fabric admins via REST API — 30 days |
| API | Office 365 Management Activity API for programmatic access / SIEM export |
Exam Caveat ⚠️: Accessing the unified audit log requires an Audit Logs role (Purview / Exchange Online) — a Fabric workspace role is not enough.
OneLake Security
OneLake security is the data-plane role model for data in OneLake, enforced consistently across Spark, the SQL analytics endpoint (user’s identity mode), Direct Lake on OneLake semantic models, and other engines.
| Component | Detail |
|---|---|
| Who creates roles | Workspace Admin or Member (via Manage OneLake security on the item) |
| Permissions | Read or ReadWrite (ReadWrite = create/edit/delete folders, tables, files, shortcuts via Spark / OneLake APIs) |
| Data in role | Tables, schemas, or folders — plus row-level and column-level security on Delta tables |
| Members | Entra users, groups, service principals / managed identities |
| Default role | DefaultReader gives anyone with ReadAll access to all lakehouse data — edit or delete it to lock down |
| Supported items | Lakehouse (Read, ReadWrite), mirrored databases & mirrored catalogs (Read) |
| Encryption | At rest with Microsoft-managed keys by default; customer-managed keys optional per workspace |
Exam Caveat ⚠️:
- OneLake security roles only restrict Viewers / users with item Read — workspace Admin, Member, Contributor always have full read/write on the data
- Roles are grant-only (no deny). Multiple roles combine as a union; within one role, OLS ∩ RLS ∩ CLS
- A role with ReadWrite can’t contain RLS or CLS
- RLS and CLS for the same table must live in the same role — combining an RLS role with a separate CLS role causes query errors
- Shortcut security: OneLake-to-OneLake shortcuts use the calling user’s identity; external shortcuts use the connection credential, with OneLake security applied on top
🔀 1.4 Orchestrate Processes
Choosing the Right Orchestration Tool
flowchart TD
START([Orchestration Need]) --> Q1{What type of\ntransformation?}
Q1 -->|"No-code / low-code\nvisual transforms"| DF["🔄 Dataflow Gen2\n(Power Query M)"]
Q1 -->|"Code-first\ncomplex logic"| NB["📓 Notebook\n(PySpark / Spark SQL)"]
Q1 -->|"Orchestrate multiple\nactivities / steps"| PIPE["🔧 Pipeline\n(ADF-like orchestration)"]
Q1 -->|"Python DAGs /\nexisting Airflow code"| AIR["🌬️ Apache Airflow job"]
PIPE --> PIPE_ACT["Activities:\n• Copy data / Copy job\n• Notebook\n• Dataflow\n• Stored Procedure\n• ForEach / If / Until\n• Web / Webhook"]
Comparison Table
| Feature | Dataflow Gen2 | Notebook | Pipeline |
|---|---|---|---|
| Persona | Citizen developer, analyst | Data engineer, data scientist | Data engineer, orchestrator |
| Language | M (Power Query) | PySpark, Spark SQL, R | Visual designer + expressions |
| Code required | No (visual UI) | Yes | Minimal (visual + expressions) |
| Best for | Simple transforms, mashups | Complex transforms, ML, large data | Orchestrating multi-step workflows |
| Data movement | Built-in connectors (150+) | Read/write via Spark | Copy activity (90+ connectors) |
| Scalability | Medium | High (Spark cluster) | Depends on activity type |
| Triggers | Schedule, pipeline | Schedule, pipeline, on-demand | Schedule (incl. interval), event-based |
Exam Caveat ⚠️:
- Dataflow Gen2 is the replacement for Dataflow Gen1 (Power BI Dataflows) — Gen2 can write to Lakehouse, Warehouse, KQL Database
- Pipelines can call Notebooks, Dataflow Gen2, and Copy jobs as activities — use pipelines as the orchestration layer
- Pure data movement (bulk, incremental, or CDC) with no orchestration logic? → Copy job is simpler than a pipeline with a Copy activity
- A Notebook can also orchestrate by calling other notebooks via
%run,notebookutils.notebook.run(), ornotebookutils.notebook.runMultiple()(parallel / DAG) —mssparkutilsis the legacy name ofnotebookutils
Schedules and Event-Based Triggers
| Trigger Type | Description | Use Case |
|---|---|---|
| Schedule | Run at specific times or on a fixed interval (every n minutes/hours, daily, weekly…) | Daily/hourly ETL jobs; ADF “tumbling window” workloads map to interval schedules |
| Event-based (storage) | React to OneLake or Azure Blob Storage file events (created, deleted…) | Process files as they land |
| Event-based (job / workspace) | React to Fabric job events (item run succeeded/failed) or workspace events | Chain items across workspaces, react to failures |
| On-demand | Manual run or REST API | Ad-hoc runs, testing |
Exam Caveat ⚠️:
- Event triggers are built on Activator (plus an Eventstream for the event source) — the trigger is saved as a Reflex item in the workspace
- Storage-event triggers expose the file name and folder path as trigger parameters you can use in pipeline expressions
- Schedules can send failure notifications to users/groups (Schedule → Failure notifications)
Orchestration Patterns with Notebooks and Pipelines
Parameters and Dynamic Expressions
Pipeline parameters:
1
2
3
4
{
"sourcePath": "@pipeline().parameters.InputFolder",
"fileName": "@concat(pipeline().parameters.Prefix, '_', formatDateTime(utcnow(), 'yyyyMMdd'), '.parquet')"
}
Notebook parameters (parameter cell, overridden by the pipeline Notebook activity or notebookutils.notebook.run):
1
2
3
4
# Notebook parameter cell (toggle "Mark cell as parameters")
input_path = "abfss://workspace@onelake.dfs.fabric.microsoft.com/lakehouse.Lakehouse/Files/raw/"
output_table = "cleaned_sales"
load_date = "2026-04-08"
Notebook orchestration from code:
1
2
3
4
5
6
7
8
9
10
11
12
13
# Run one child notebook (timeout in seconds, parameters dict) and capture its exit value
result = notebookutils.notebook.run("Bronze_to_Silver", 600, {"load_date": "2026-10-13"})
# Run several notebooks in parallel / as a DAG inside one Spark session
dag = {
"activities": [
{"name": "silver_sales", "path": "Bronze_to_Silver_Sales"},
{"name": "silver_customers", "path": "Bronze_to_Silver_Customers"},
{"name": "gold", "path": "Silver_to_Gold", "dependencies": ["silver_sales", "silver_customers"]}
]
}
notebookutils.notebook.validateDAG(dag)
notebookutils.notebook.runMultiple(dag)
Exam Caveat ⚠️:
runMultiple()child notebooks share the parent’s Spark session compute — concurrency is bounded by driver cores. Use a child’snotebookutils.notebook.exit(value)to return a (string) result.
Common orchestration patterns:
flowchart TD
PIPE["🔧 Pipeline"] --> COPY["📋 Copy Activity\n(Ingest raw data)"]
COPY --> NB1["📓 Notebook 1\n(Bronze → Silver)"]
NB1 --> NB2["📓 Notebook 2\n(Silver → Gold)"]
NB2 --> DF["🔄 Dataflow Gen2\n(Final transforms for BI)"]
DF --> REFRESH["🔄 Semantic Model\nRefresh"]
📊 Quick-Reference Scenario Table
| Scenario | Requirement | Answer |
|---|---|---|
| Promote items from Dev → Test → Prod | Environment promotion | Deployment Pipelines |
| Version-control notebook code | Source control | Git Integration (Azure DevOps / GitHub) |
| Restrict users to see only their region’s data | Data filtering | Row-Level Security (RLS) |
| Hide salary column from analysts | Column restriction | Column-Level Security (CLS) |
| Mask email addresses in query results | Data masking | Dynamic Data Masking (DDM) |
| Orchestrate Copy → Notebook → Dataflow sequence | Multi-step workflow | Pipeline |
| No-code transform CSV to Lakehouse table | Visual ETL | Dataflow Gen2 |
| Complex PySpark transform on large dataset | Code-first ETL | Notebook |
| Run several notebooks in parallel with dependencies | Code-first orchestration | notebookutils.notebook.runMultiple() |
| Trigger pipeline when file lands in OneLake | Event-driven | Storage event trigger (Activator) |
| Track who accessed what data | Audit trail | Microsoft Purview unified audit log |
| Mark a dataset as trusted and organization-approved | Data quality signal | Certified endorsement |
| Control Spark cluster size for a workspace | Compute management | Custom pool in Spark settings |
| Guaranteed ~5 sec Spark start for a scheduled job | Predictable startup | Custom live pool |
| Faster Direct Lake reads on new Delta tables | V-Order needed | readHeavyForPBI resource profile |
| Airflow DAGs must run 24/7 on schedule in production | Always-on Airflow | Apache Airflow custom pool |
| Viewer must see only some lakehouse tables/rows in Spark + SQL | Data-plane security | OneLake security role (OLS + RLS) |
| Same items need different source IDs in Dev/Test/Prod | Stage-specific config | Variable library (value sets) |
| Schema-as-code for Warehouse | Version-controlled schema | Database Projects |