Q: Your AKS cluster generates 800 GB of stdout/stderr logs daily, resulting in an astronomical Azure Log Analytics ingestion invoice ($2.30/GB). Developers mostly generate noisy debug logs that are never queried. How do you re-architect Container Insights using Data Collection Rules (DCR) and ContainerLogV2 to slash ingestion volume by over 60%?
Step-by-step engineering runbook for reducing Azure Log Analytics ingestion costs by 65% using Data Collection Rules (DCR), custom log filtering, and ContainerLogV2 schema transformation.
#Azure #Azure Monitor #Container Insights #Data Collection Rules #FinOps #Log Analytics
🎙️ Candidate Opening & Architectural Context
"By default, Azure Monitor Container Insights ingests all stdout/stderr logs from all containers into the legacy ContainerLog table. Our monthly ingestion bill exceeded $55,000. We implemented Data Collection Rules with KQL transformations to filter noise at the ingestion pipe."
Advertisement
⚡ Recommended Practice Lab
Want to master this scenario in a live sandbox? Stephane Maarek's AWS Certified DevOps Engineer Professional Masterclass on Udemy covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
1️⃣
Enable ContainerLogV2 High-Performance Schema
Switch from legacy ContainerLog schema to the modern, compact ContainerLogV2 format:
- ConfigMap Update: Deployed
container-azm-ms-agentconfigConfigMap inkube-systemsetting[log_collection_settings.schema] containerlog_schema_version = 'v2'. - Storage Efficiency: ContainerLogV2 consolidates pod and container metadata into a single structured record, reducing ingestion bytes by approximately 15% out of the box.
Pro Tip: ContainerLogV2 replaces verbose individual log entry rows with structured JSON columns, speeding up KQL query speeds by up to 3x.
2️⃣
Create Data Collection Rule (DCR) with Ingestion-Time KQL Transformations
Filter out irrelevant namespaces and discard low-value debug log entries before they hit billing storage:
- Create DCR: Defined an Azure Monitor Data Collection Rule with an ingestion-time KQL stream transformation.
- Transformation KQL:
source | where PodNamespace !in ('kube-system', 'gatekeeper-system', 'datadog') | where LogSeverity !in ('DEBUG', 'TRACE'). - Link DCR to AKS: Associated DCR with cluster:
az monitor data-collection-rule association create --name aks-dcr-assoc --rule-id $DCR_ID --resource $AKS_CLUSTER_ID.
Pro Tip: Ingestion-time transformations evaluate log records at the Azure Monitor ingestion pipeline. Discarded rows are not counted toward billing.
3️⃣
Configure Table Plan: Analytics vs Basic Log Tier
Shift high-volume, low-criticality logs to the Basic Logs pricing tier ($0.50/GB vs $2.30/GB):
- Basic Plan Migration: Configured
ContainerLogV2table plan toBasicfor non-production environments viaaz monitor log-analytics workspace table update -g rg-ops -w prod-laws -n ContainerLogV2 --plan Basic. - Retention Policy: Set interactive retention to 30 days and archived historical logs to Azure Storage Archive tier for compliance.
Pro Tip: Basic Logs provide ingestion and simple KQL search at more than a 75% discount compared to full Analytics plan tables.
4️⃣
Audit Ingestion Volume Reductions & Query Latency
Measure and verify the exact reduction in daily GB ingestion:
- Audit Query: Executed KQL query on
Usage | where DataType == 'ContainerLogV2' | summarize sum(Quantity) by bin(TimeGenerated, 1d). - Results: Daily ingestion dropped from 820 GB/day to 275 GB/day, reducing monthly monitoring expenditure by over $37,000.
Pro Tip: Continuous monitoring of the Usage table ensures that newly deployed microservices with accidental debug logging are promptly detected.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Upgrading to ContainerLogV2 and implementing Data Collection Rules with KQL transformations filters out noisy namespaces and debug logs before billing, slashing Azure Monitor costs by over 65%."
⚡ 60-Second Elevator Pitch Talking Points
- Migrate from legacy ContainerLog to the optimized ContainerLogV2 schema.
- Deploy Data Collection Rules (DCR) with KQL transformations to filter out system and debug logs at ingestion.
- Transition non-production log workspaces to the cost-effective Basic Logs plan.
- Achieve an immediate 65% reduction in monthly Log Analytics ingestion expenses.
Advertisement