⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All Kubernetes Interview Questions Scenario 95 of 207 in Kubernetes
Senior DevOps / SRE [L2] Kubernetes Advanced Scenarios Production Scenario [L2]
🎯 Target Role / Context: Kubernetes Scenario #85 (L2) · Advanced Scenarios

Q: A CronJob job ran but the pod isn't showing in `kubectl get jobs`.

successfulJobsHistoryLimit may be 0 or 1 and old jobs were cleaned. Adjust to keep history.

#Kubernetes #Advanced Scenarios #L2 #Container Orchestration #K8s
🎙️ Candidate Opening & Architectural Context
"<strong>When a CronJob job ran but the pod isn't showing in kubectl get jobs, the first correction is that <code>kubectl get jobs</code> lists Job objects, not pods.</strong> A missing entry usually means the Job was garbage-collected (history limits or TTL), you are in the wrong namespace, or the CronJob never created a Job at all. Interviewers use this to test whether you know the CronJob to Job to Pod ownership chain and where each layer is cleaned up. The answer is a short diagnostic runbook: confirm the CronJob fired, find the Job, find the pod by label, then read events and logs."
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: <strong>When a CronJob job ran but the pod isn't showing in kubectl get jobs, the first correction is that <code>kubectl get jobs</code> lists Job objects, not pods.</strong> A missing entry usually means the Job was garbage-collected (history limits or TTL), you are in the wrong namespace, or the CronJob never created a Job at all. Interviewers use this to test whether you know the CronJob to Job to Pod ownership chain and where each layer is cleaned up. The answer is a short diagnostic runbook: confirm the CronJob fired, find the Job, find the pod by label, then read events and logs.

1️⃣

Mental model: CronJob creates Job creates Pod

A CronJob does not run containers. It creates a Job on schedule, and the Job controller creates pods. Each layer has its own retention rules.

  • CronJob: records lastScheduleTime and lastSuccessfulTime in status. If these are empty or old, no Job was created and you debug the schedule, not the pod.
  • Job: kept according to successfulJobsHistoryLimit (default 3) and failedJobsHistoryLimit (default 1). Older Jobs and their pods are deleted by the CronJob controller.
  • Pod: belongs to the Job through ownerReferences and carries the label job-name=<job> (newer clusters also set batch.kubernetes.io/job-name). Pods vanish when the Job is deleted or when ttlSecondsAfterFinished expires.
  • Different commands: kubectl get jobs shows Jobs, kubectl get pods shows pods. Completed pods show Completed status and are hidden by some UIs and by --field-selector=status.phase=Running.
  • Ownership check: run kubectl get job <name> -o jsonpath='{.metadata.ownerReferences[0].name}' to prove which CronJob created a Job, which helps when several CronJobs share similar name prefixes in one namespace.
2️⃣

Hands-on: pod isn't showing in kubectl get jobs, find the run

Work down the chain in the right namespace and widen the search before concluding the job did not run.

kubectl -n prod get cronjob my-report -o wide
kubectl -n prod describe cronjob my-report | sed -n '/Events/,$p'

# Jobs created by this CronJob (newest last)
kubectl -n prod get jobs --sort-by=.metadata.creationTimestamp | grep my-report

# Pods for a specific Job, including Completed ones
kubectl -n prod get pods -l job-name=my-report-28774000 -o wide

# Events survive a little longer than the objects (default ~1h)
kubectl -n prod get events --sort-by=.lastTimestamp | grep my-report
  • Namespace first: use -n or -A. CronJob objects and their Jobs always live in the same namespace.
  • CronJob status: kubectl describe cronjob shows last schedule time, active Jobs and Events such as SuccessfulCreate or MissSchedule.
  • All Jobs for it: Jobs are named <cronjob>-<timestamp>, so grep by prefix, and sort by creation time.
  • Pods by label: if the Job exists, list pods through the job-name label rather than guessing names.
  • Time window: compare lastScheduleTime with the current time and the cron expression. Cron schedules use the controller-manager's time zone unless spec.timeZone is set, so a job may have run at a different hour than you expected.
Advertisement
3️⃣

Root causes when the Job or pod is gone

Match what you see to a cause. Each one has a different fix.

  • History limit cleanup: with successfulJobsHistoryLimit: 0 the Job is deleted as soon as it finishes, so it can never be listed afterwards.
  • TTL after finished: spec.jobTemplate.spec.ttlSecondsAfterFinished removes the Job and its pods after the set seconds, which makes logs unrecoverable unless shipped elsewhere.
  • Never created: suspend: true, a wrong schedule or timeZone, concurrencyPolicy: Forbid while a previous run is stuck, or startingDeadlineSeconds exceeded after controller downtime.
  • Created but no pod: a ResourceQuota, LimitRange or admission webhook rejecting the pod. The Job events then show FailedCreate with the reason, even though no pod exists.
  • Concurrency skips: with concurrencyPolicy: Forbid the controller records a skipped run in events rather than creating a Job, and with Replace it deletes the still-running Job and its pods, which also explains pods that disappear mid-run.
Pro Tip: If the Job exists with <code>COMPLETIONS 0/1</code> and no pods, read <code>kubectl describe job</code> Events first. <code>FailedCreate</code> messages name the quota or webhook that blocked it.
4️⃣

Reproduce and verify

Don't wait for the next schedule. Create a Job on demand from the CronJob template and watch it.

kubectl -n prod create job --from=cronjob/my-report my-report-manual-1
kubectl -n prod wait --for=condition=complete job/my-report-manual-1 --timeout=300s
kubectl -n prod logs job/my-report-manual-1
kubectl -n prod get cronjob my-report -o jsonpath='{.status.lastSuccessfulTime}'
  • Manual trigger: kubectl create job --from=cronjob/my-report my-report-manual-1 uses the same pod template, so quota or RBAC problems reproduce immediately.
  • Watch: kubectl get pods -w -l job-name=my-report-manual-1 and then kubectl logs job/my-report-manual-1.
  • Failure states: ImagePullBackOff, CrashLoopBackOff and exceeded backoffLimit all leave a failed Job whose pods remain until the failed history limit removes them.
  • Confirm the fix: after correcting the cause, check that lastSuccessfulTime advances on the next scheduled run.
  • Evidence after failure: if the run failed and you only need to see why, kubectl describe job and kubectl logs job/<name> --all-containers still work while the Job object exists, so capture output before retention expires.
5️⃣

Production practice

Make CronJob runs observable so you are never debugging after the evidence is deleted.

  • Keep enough history: use successfulJobsHistoryLimit: 3 and failedJobsHistoryLimit: 3 or more for jobs you debug often, and avoid TTL values shorter than your alert response time.
  • Ship logs: send stdout to a log backend so output outlives the pod.
  • Alert on absence: with kube-state-metrics, alert when kube_cronjob_status_last_successful_time is older than the expected interval, not only on failures.
  • Set deadlines: define startingDeadlineSeconds, activeDeadlineSeconds and an explicit timeZone so skips and hangs are predictable.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Pro-Tip: Remember that jobs and pods are separate objects. Find the Job through the CronJob, then the pod via the job-name label, and check history limits and TTL before assuming it never ran."
⚡ 60-Second Elevator Pitch Talking Points
  • kubectl get jobs lists Jobs, not pods, and a CronJob creates a Job which creates pods, each with its own cleanup rules.
  • I check the CronJob status and events, look in the right namespace, find the Job and its pods with the job-name label, and read FailedCreate events for quota or webhook blocks.
  • I automate it with generous history limits, shipped logs and an alert on stale lastSuccessfulTime.
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Expecting pods in the output of kubectl get jobs The command lists only Job objects, so you must query pods separately by label.
  2. Setting successfulJobsHistoryLimit to 0 or a tiny TTL, then debugging later The Job and its pods are deleted immediately, so logs and events are gone.
  3. Ignoring concurrencyPolicy: Forbid A hung previous run silently makes the controller skip new schedules.
  4. Looking only at pod status for missing runs If quota or a webhook blocked creation, no pod ever existed; the evidence is in Job events.
Advertisement
📥 FREE DOWNLOAD · 101-PAGE COMPANION HANDBOOK
Studying for Kubernetes & SRE Technical Rounds?
Download the complete 100-question PDF field guide covering all 11 core modules with offline diagnostic runbooks.
📥 Download PDF (Free) Read Online Guide →
Want more Kubernetes scenarios?
Explore our complete collection of scenario-based Kubernetes interview runbooks.
Browse All Kubernetes Questions →