⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All Platform Engineering & IDP Interview Questions Scenario 1 of 50 in Platform Engineering & IDP
Staff Platform Engineer Platform Engineering Internal Developer Platforms & Catalogs Enterprise IDP
🎯 Target Role / Context: Enterprise Platform Engineering Loop · Developer Infrastructure Track

Q: Your enterprise Backstage software catalog suddenly stops updating new services and entities. Logs reveal '403 API rate limit exceeded' from GitHub. How do you triage this immediately and redesign ingestion for scale?

Root cause isolation and architectural remediation when Spotify Backstage crashes or ceases catalog ingestion due to GitHub API rate-limiting across 2,000+ microservice repositories.

#Platform Engineering #Backstage #GitHub #Rate Limiting #Software Catalog #IDP #DevEx
🎙️ Candidate Opening & Architectural Context
"When Backstage ingestion hits GitHub's 5,000 requests-per-hour rate limit, catalog discovery halts across the organization. The root cause is almost always aggressive polling frequencies in catalog-info.yaml providers, lack of GitHub App token multiplexing, or missing event-driven webhook ingestion."
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: When Backstage ingestion hits GitHub's 5,000 requests-per-hour rate limit, catalog discovery halts across the organization. The root cause is almost always aggressive polling frequencies in catalog-info.yaml providers, lack of GitHub App token multiplexing, or missing event-driven webhook ingestion.

1

Immediate Mitigation: Transition from Personal Access Token to GitHub App

Switch Backstage's GitHub authentication from a Personal Access Token (PAT) (5,000 req/hr) to a GitHub App installation. GitHub Apps provide 5,000 requests per hour per installation, scaling with organization repositories (up to 12,500 req/hr for enterprise organizations).

# app-config.yaml
integrations:
  github:
    - host: github.com
      apps:
        - appId: ${GITHUB_APP_ID}
          clientId: ${GITHUB_CLIENT_ID}
          clientSecret: ${GITHUB_CLIENT_SECRET}
          webhookSecret: ${GITHUB_WEBHOOK_SECRET}
          privateKey: ${GITHUB_PRIVATE_KEY}
2

Shift from Full-Tree Polling to Event-Driven Webhook Ingestion

Disable recursive catalog scanning on 5-minute cron schedules. Configure GitHub Webhooks targeting Backstage's /api/catalog/github/webhook endpoint on push and repository events to trigger incremental entity refreshes only when catalog-info.yaml files change.

Pro Tip: Scale Reality: Polling 3,000 repositories every 10 minutes generates 18,000 API calls/hr, instantly exhausting API limits. Webhooks reduce steady-state API consumption by over 96%.
Advertisement
3

Tune Catalog Provider Batch Sizing & Schedule Frequency

In catalog.providers.github, adjust schedule.frequency from 5m to 120m for background safety scans, and set batch size to prevent burst concurrency.

catalog:
  providers:
    github:
      organizationProvider:
        organization: 'acme-enterprise'
        schedule:
          frequency: { minutes: 120 }
          timeout: { minutes: 30 }
          initialDelay: { seconds: 15 }
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Never rely on raw REST polling for large enterprise catalogs. Multiplex GitHub Apps and configure event-driven webhooks for catalog-info.yaml updates."
⚡ 60-Second Elevator Pitch Talking Points
  • Switch authentication immediately from static PATs to GitHub Apps to gain dynamic enterprise rate limits.
  • Deploy GitHub webhooks for push events so catalog refreshes trigger only on catalog-info.yaml commits.
  • Increase background discovery poll interval from 5 minutes to 2 hours for reconciliation fallbacks.
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Engineering bespoke in-house developer tools instead of adopting standardized platform Golden Paths when configuring Backstage Software Catalog Bespoke internal tools create long-term maintenance debt and fragmented workflows, whereas standardized Golden Paths accelerate delivery.
  2. Overlooking infrastructure cost visibility and FinOps allocation tagging across multi-tenant environments when configuring Backstage Software Catalog Without granular resource tagging, engineering organizations cannot attribute cloud infrastructure costs to specific teams or business units.
  3. Failing to define explicit Service Level Objectives (SLOs) and Error Budgets to balance feature velocity with reliability when configuring Backstage Software Catalog Without agreed SLOs, engineering and product teams lack an objective data-driven framework to prioritize technical debt over new features.
  4. Neglecting automated continuous drift detection between declarative configuration and deployed cloud state when configuring Backstage Software Catalog Unmanaged drift accumulates over time, causing future automated deployments to fail or overwrite emergency manual adjustments.
Advertisement
Want more Platform Engineering & IDP scenarios?
Explore our complete collection of scenario-based Platform Engineering & IDP interview runbooks.
Browse All Platform Engineering & IDP Questions →