⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All FinOps & System Design Interview Questions Scenario 6 of 44 in FinOps & System Design
Staff SRE / Distributed Systems Architect System Design Observability & Big Data Pipelines System Design
🎯 Target Role / Context: Staff SRE / Principal Big Data Architect System Design Interview

Q: Your microservices fleet generates 100 Terabytes of unstructured and JSON logs every single day (~1.2 million events/sec). Commercial SaaS tools (Datadog, Splunk) would cost over $250,000/month. How do you architect an in-house, distributed, highly available log aggregation and indexing platform that provides sub-2-second search queries while keeping infrastructure costs under $15,000/month?

Architecting an ultra-high-throughput, cost-efficient observability logging pipeline capable of ingesting 100 TB/day (1.2 million log events/sec) with sub-second query latency using Vector, Kafka, ClickHouse, and object storage tiering.

#System Design #Kafka #Vector #ClickHouse #Observability #Loki #Big Data
🎙️ Candidate Opening & Architectural Context
"At 100 TB/day, traditional Elasticsearch/OpenSearch clusters collapse under Lucene inverted-index memory pressure and multi-petabyte disk I/O requirements. We designed a modern columnar logging pipeline using Vector, Apache Kafka, and ClickHouse."
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? The Linux Foundation's FinOps Certified Practitioner (FOCP) Program covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: At 100 TB/day, traditional Elasticsearch/OpenSearch clusters collapse under Lucene inverted-index memory pressure and multi-petabyte disk I/O requirements. We designed a modern columnar logging pipeline using Vector, Apache Kafka, and ClickHouse.

1️⃣

Deploy High-Performance Edge Log Collectors with Vector DaemonSets

Collect, parse, and compress log streams at the node layer without taxing host CPU:

  • Vector DaemonSets: Deployed Vector (written in Rust) on every Kubernetes worker node, consuming stdout logs from container runtimes via /var/log/pods.
  • Schema Normalization: Vector parses JSON payloads, drops useless health check logs, enriches logs with Kubernetes metadata (pod, namespace, git commit), and compresses payloads using Zstandard (zstd level 3).
Pro Tip: Vector consumes 75% less memory and CPU than Fluentbit/Logstash, preventing log collection daemons from starving application containers of compute.
2️⃣

Buffer Ingestion Bursts with Apache Kafka / Redpanda Streaming Tier

Decouple high-frequency ingest spikes from downstream indexing pipelines:

  • Partitioning Strategy: Provisioned a 12-broker Kafka cluster with 96 partitions distributed across 3 Availability Zones, partitioned by tenant_id or service_name.
  • Retention & Throughput: Configured 6-hour disk retention buffer, absorbing massive downstream ClickHouse maintenance restarts or query spikes without dropping a single log entry.
Pro Tip: Buffering with Kafka guarantees at-least-once log delivery even during severe database compaction or network brownouts.
Advertisement
3️⃣

Store & Index Columnar Logs with ClickHouse Distributed Engine

Ingest structured events into ClickHouse for instantaneous SQL querying:

  • Kafka Engine Table: ClickHouse consumer tables consume directly from Kafka partitions in micro-batches (50,000 rows or 2 seconds).
  • MergeTree Partitioning: Data is written to ReplacingMergeTree partitioned by toYYYYMMDD(timestamp) and ordered by (tenant_id, service_name, level, timestamp).
  • Compression: ClickHouse columnar encoding achieves 8:1 to 10:1 data compression ratios on log data.
Pro Tip: ClickHouse processes aggregate queries and regex searches across billions of rows in milliseconds by scanning only relevant compressed columns.
4️⃣

Implement Tiered Cold Storage on Cloud Object Storage (S3 / GCS)

Optimize multi-petabyte storage expenses by offloading historical data to object storage:

  • Hot Tier (NVMe SSD): Retains 3 days of raw logs on local high-speed NVMe disks for instant incident triaging.
  • Cold Tier (S3 / GCS): ClickHouse Storage Policy automatically moves parts older than 3 days to AWS S3 / Google Cloud Storage, retaining 90 days of searchable historical records.
  • Total Cost: S3 storage + 12 ClickHouse compute nodes cost $11,200/month, slashing monitoring expenses by 95% compared to commercial SaaS.
Pro Tip: ClickHouse queries against S3 object storage remain fully interactive, transparently caching metadata and index marks locally.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Ingesting 100 TB/day sustainably requires replacing inverted-index search with a columnar database (ClickHouse), buffered by Kafka, collected by Rust-based Vector, and tiered to cloud object storage."
⚡ 60-Second Elevator Pitch Talking Points
  • Collect and enrich logs at the node layer using high-performance Vector DaemonSets.
  • Buffer bursty traffic through a multi-partition Kafka cluster to guarantee zero data loss.
  • Ingest micro-batches into ClickHouse columnar storage for sub-second analytical search.
  • Tier logs older than 3 days to S3/GCS object storage to maintain 90-day retention under budget.
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Engineering bespoke in-house developer tools instead of adopting standardized platform Golden Paths when configuring System Design TB/Day Bespoke internal tools create long-term maintenance debt and fragmented workflows, whereas standardized Golden Paths accelerate delivery.
  2. Overlooking infrastructure cost visibility and FinOps allocation tagging across multi-tenant environments when configuring System Design TB/Day Without granular resource tagging, engineering organizations cannot attribute cloud infrastructure costs to specific teams or business units.
  3. Failing to define explicit Service Level Objectives (SLOs) and Error Budgets to balance feature velocity with reliability when configuring System Design TB/Day Without agreed SLOs, engineering and product teams lack an objective data-driven framework to prioritize technical debt over new features.
  4. Neglecting automated continuous drift detection between declarative configuration and deployed cloud state when configuring System Design TB/Day Unmanaged drift accumulates over time, causing future automated deployments to fail or overwrite emergency manual adjustments.
Advertisement
Want more FinOps & System Design scenarios?
Explore our complete collection of scenario-based FinOps & System Design interview runbooks.
Browse All FinOps & System Design Questions →