Q: How do you design a centralized multi-tenant log aggregation architecture using OpenTelemetry Collector and Grafana Loki that guarantees namespace isolation, prevents log ingestion runaway costs, and scales across multiple Kubernetes clusters?

Deploy OpenTelemetry Collector as a DaemonSet with filelog receiver. Extract Kubernetes metadata (namespace, pod, container, git_commit) using the k8sattributes processor. Sanitize logs to mask PII (credit cards, JWTs, API tokens) using transformprocessor before logs ever leave the node.

processors:
  k8sattributes:
    auth_type: 'serviceAccount'
    extract:
      metadata: [k8s.namespace.name, k8s.pod.name, k8s.container.name]
  transform:
    log_statements:
      - context: log
        statements:
          - replace_pattern(body, '(?i)(bearer\s+)[a-zA-Z0-9_\-\.]+', '$$1[REDACTED]')

Dynamically route logs into Grafana Loki using the Loki exporter, injecting X-Scope-OrgID headers mapped to Kubernetes namespaces or business units. This enables Loki's native microservices-mode multi-tenancy, isolating log search queries and retention policies between teams.

exporters:
  loki:
    endpoint: 'http://loki-gateway.logging:80/loki/api/v1/push'
    headers:
      'X-Scope-OrgID': 'tenant-team-a'

Configure Loki overrides and tenant limits (ingestion_rate_mb, max_streams_per_user, burst_size_mb). Implement loki-canary and Prometheus alerting on tenant throughput to automatically drop debug-level chatter or throttle noisy microservices before network and storage degradation occurs.

limits_config:
  per_tenant_override_config: /etc/loki/overrides.yaml
  ingestion_rate_mb: 10
  ingestion_burst_size_mb: 20
  max_streams_per_user: 10000