From DevOps Engineer to AI Engineer: You Don’t Have to Start from Zero

01 The Baseline Foundation

02 The Evolution Matrix

💡 "The fundamentals don’t disappear. The engineering surface area grows."

A machine learning engineer might write a python script that calls model.generate(). But when that script must handle 5,000 queries per second with a 99.9% uptime SLA, sub-second latency, zero PII exfiltration, and cost-controlled GPU autoscaling, it is a DevOps problem.

03 Deep Dive · Real World Production

The model integration is only one part of putting it into production.

You still need to handle:

In tutorials, building a RAG app is 15 lines of Python using LangChain. In an enterprise handling confidential enterprise data, customer PII, and financial records, you still need to handle these nine mission-critical operational challenges:

🎯 "That’s where cloud and DevOps experience becomes directly useful."

A pure software engineer or data scientist will stumble when configuring secure VPC endpoints for S3 document ingestion, isolating vector database nodes, or setting up Karpenter node termination handlers for spot GPU instances. You already know how to build this.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-llama3-serving
  namespace: ai-platform
  labels:
    app.kubernetes.io/name: vllm-llama3
spec:
  replicas: 2
  selector:
    matchLabels:
      app: vllm-llama3
  template:
    metadata:
      labels:
        app: vllm-llama3
    spec:
      nodeSelector:
        node.kubernetes.io/instance-type: "g5.2xlarge" # NVIDIA A10G (24GB VRAM)
      tolerations:
        - key: "nvidia.com/gpu"
          operator: "Exists"
          effect: "NoSchedule"
      containers:
        - name: vllm-engine
          image: vllm/vllm-openai:v0.5.4
          command: ["python3", "-m", "vllm.entrypoints.openai.api_server"]
          args:
            - "--model=meta-llama/Meta-Llama-3-8B-Instruct"
            - "--tensor-parallel-size=1"
            - "--gpu-memory-utilization=0.90"
            - "--max-model-len=8192"
            - "--port=8000"
          ports:
            - containerPort: 8000
              name: http
          resources:
            limits:
              nvidia.com/gpu: "1"
              memory: "32Gi"
              cpu: "8"
            requests:
              nvidia.com/gpu: "1"
              memory: "16Gi"
              cpu: "4"
          readinessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 90
            periodSeconds: 10
          volumeMounts:
            - mountPath: /dev/shm
              name: dshm
      volumes:
        - name: dshm
          emptyDir:
            medium: Memory
            sizeLimit: 8Gi

04 The Gap Analysis

05 The Strategic Synthesis

“AI engineers build intelligent systems. DevOps engineers bring the operational discipline to make systems reliable, secure, scalable, and supportable.”

The opportunity isn’t DevOps versus AI engineering.

It’s DevOps applied to AI—with security built in.

There is no war between DevOps and AI Engineering. The industry does not need fewer infrastructure engineers; it desperately needs infrastructure engineers who understand token latency, vector similarity search, GPU allocation, and AI security boundaries.

06 Interview Scenarios · Senior AI Platform Tier

“Design an enterprise RAG architecture handling 100,000 corporate documents with strict multi-tenant access control and sub-second p99 latency.”

The Senior Production Architecture Runbook:

1. Hard Filter at the Vector Layer (Pre-Filtering): Never rely on LLM prompts for access control. Store tenant IDs, user ACL groups, and document classification tags as vector payload metadata. Apply strict payload pre-filtering (e.g. Qdrant filter condition tenant_id = user.org AND allowed_groups INTERSECT user.roles) before ANN graph traversal.
2. Two-Stage Retrieval (Dense + Lexical + Rerank): Perform hybrid search: dense vector search (semantic intent) + BM25 keyword search (acronyms, employee IDs, part numbers). Pass top 50 candidates through a lightweight cross-encoder reranker (Cohere or BGE-Reranker) to select the top 5 highest-relevance passages.
3. Embedding & LLM Caching: Implement semantic caching with Redis. If a user asks a question semantically identical (cosine similarity > 0.96) to a recent query with identical ACLs, return cached completions, saving 90% latency and token cost.
4. Streaming TTFT Optimization: Return responses via HTTP chunked transfer / Server-Sent Events (SSE). While total completion takes 2.5s, Time-To-First-Token (TTFT) drops to < 350ms, delivering instant perceived responsiveness.

“Your vLLM inference cluster on AWS EKS is experiencing sudden spikes in Time-To-First-Token (TTFT) and intermittent CUDA Out-of-Memory (OOM) crashes under peak load. How do you troubleshoot and resolve it?”

The Senior Production Architecture Runbook:

1. Inspect vLLM Metrics & DCGM Telemetry: Export NVIDIA DCGM (Data Center GPU Manager) metrics to Prometheus. Check vllm:num_requests_waiting and vllm:gpu_cache_usage_factor. If cache usage is 100% and waiting requests spike, the KV cache is thrashing, forcing continuous request queuing.
2. Tune GPU Memory Utilization & Max Model Length: Adjust --gpu-memory-utilization (typically set to 0.90 or 0.92, never 1.0 to allow headroom for activation memory during long context sequences). Bound --max-model-len to the actual enterprise requirement (e.g. 8192 rather than default 32k) to reclaim gigabytes of KV cache memory.
3. Mount Shared Memory Volume: PyTorch multi-process tensor parallelism relies on /dev/shm. If missing, CUDA crashes with bus errors under load. Ensure an emptyDir volume with medium: Memory (at least 8Gi) is mounted to /dev/shm.
4. Autoscale on Queue Depth, Not CPU: Traditional CPU/memory HPA fails for LLM inference. Configure KEDA (Kubernetes Event-driven Autoscaling) using Prometheus metrics to scale pods when vllm:num_requests_waiting > 5 for 30 consecutive seconds.

“An AI agent equipped with internal database and email tools receives an indirect prompt injection via an uploaded supplier invoice. How do you prevent data exfiltration and unauthorized actions?”

The Senior Production Architecture Runbook:

1. Dual-LLM Isolation Architecture: Untrusted content (uploaded invoices, scraped web pages) is processed by a quarantined reader LLM with zero tool access. It generates a sanitized, structured summary that is passed to the primary agent as pure data.
2. Strict Principle of Least Agency (OWASP LLM06): Never grant an agent direct SQL execution or write access to production databases. Tools must be restricted to parameterized API endpoints with read-only scopes.
3. Human-in-the-Loop (HITL) for Destructive Actions: Any state-mutating tool (sending emails to external recipients, issuing payments, modifying database records) requires an asynchronous approval token before execution.
4. Semantic Guardrails (NeMo / Llama Guard): Deploy an input/output guardrail layer that inspects tool arguments for prompt injection patterns and regex-scans completions for credit cards, API keys, and internal IP addresses before delivery.

“How do you build an automated CI/CD evaluation gate for an enterprise LLM application before promoting prompt or model weight updates to production?”

The Senior Production Architecture Runbook:

1. Version-Controlled Prompt Artifacts: Store prompt templates as versioned code files (YAML/JSON) in Git with strict schema contracts, not hardcoded strings inside Python modules.
2. Automated Evaluation Test Suite (Golden Dataset): Maintain an immutable golden dataset of 500 edge-case production queries with ground truth annotations. Run automated evaluation using Ragas and DeepEval in GitHub Actions.
3. Hard Numerical CI Gates: Enforce blocking CI rules: PR fails if Faithfulness < 0.92, Context Recall < 0.88, or Hallucination Rate > 2%.
4. Canary Rollout with Langfuse Telemetry: Promote accepted changes through a 10% canary traffic split. Monitor production user feedback (thumbs up/down) and latency in Langfuse for 2 hours before full 100% rollout.

“Your vector database query latency degraded 10x (from 40ms to 450ms) as the vector index grew from 50,000 to 5 million embeddings. How do you isolate and resolve the bottleneck?”

The Senior Production Architecture Runbook:

1. Diagnose RAM vs Disk Spilling (Working Set Sizing): Calculate raw index footprint: 5M vectors × 1536 dimensions × 4 bytes (float32) ≈ 30.7 GB raw vectors + ~20GB HNSW graph overhead ≈ 50GB RAM. If your instance only had 32GB RAM, the database began swapping to disk, causing the 10x latency collapse.
2. Enable Scalar Quantization (SQ) / Product Quantization (PQ): Compress vector dimensions from float32 to int8 (scalar quantization). This reduces memory footprint by 75% (from 50GB to ~13GB) with < 1% degradation in recall, allowing the entire graph to fit resident in RAM.
3. Tune HNSW Search Parameters (ef_search): High ef values increase recall at the cost of exponential search comparisons. Tune ef_search down to balance recall (e.g. 0.98 recall at ef=64 vs ef=256).
4. Partition Sharding & Read Replicas: Horizontally shard the collection across multiple vector nodes by tenant or hash ID, and deploy read replicas behind an internal load balancer to distribute query throughput.

07 The Action Roadmap

🧠 Practice AI/ML Infrastructure Interview Questions →

08 Interactive Problem Scenarios