Q: How do you architect and scale the Backstage Software Catalog across an enterprise with thousands of services split between hundreds of individual repositories and giant monorepos, while overcoming GitHub API rate limits and stale catalog entities?

Replace polling-based catalog processors (catalog-backend-module-github) with event-driven Entity Providers (GithubEntityProvider) paired with GitHub Webhooks. Changes to catalog-info.yaml trigger immediate delta ingestion rather than periodic full-tree scans.

// catalog.ts
const githubProvider = GithubEntityProvider.fromConfig(env.config, {
  logger: env.logger,
  schedule: env.scheduler.createScheduledTaskRunner({
    frequency: { minutes: 120 },
    timeout: { minutes: 15 },
  }),
});
builder.addEntityProvider(githubProvider);

For large monorepos containing hundreds of services, avoid single giant catalog files. Structure nested directories with individual catalog-info.yaml files and register wildcards (target: 'https://github.com/org/monorepo/blob/main/**/catalog-info.yaml'), leveraging monorepo sparse checkouts or tree API caches in the backend.

# app-config.yaml
catalog:
  locations:
    - type: url
      target: https://github.com/org/monorepo/blob/main/**/catalog-info.yaml
      rules:
        - allow: [Component, System, API]

Replace default SQLite with a high-availability PostgreSQL cluster with read-replicas. Configure catalog processing queues and batching, caching GitHub API responses in Redis to withstand burst rate limits during enterprise-wide pipeline runs.

database:
  client: pg
  connection:
    host: postgres-cluster.internal
    port: 5432
    user: backstage
    password: ${POSTGRES_PASSWORD}
    ssl: { rejectUnauthorized: true }