Pull Request: Dynamic System Configuration & Multi-Tier Feature Flags

Summary

Implements a centralized System Configuration & Feature Flags Management system across both Backend and Frontend. This eliminates hardcoded business constants, allows runtime feature toggling, enables zero-downtime Firecrawl API key rotation, and dynamically feeds business quotas/rate-limits directly into active middleware and background workers.

Backend Changes (data-crawler-be)

1. Database & Schema

• Added SystemConfig Prisma model with composite indexing (category, isPublic).
• Generated and applied Supabase migration: 20260908093359_add_system_config.
• Seeded and synchronized exactly 43 production-grade configurations:
  • FEATURE_FLAG (8): feature.registration.enabled, feature.maintenance_mode.enabled, feature.change_detection.enabled, feature.webhook.deliveries.enabled, etc.
  • INTEGRATION (8): Preserved dynamic integration.firecrawl.api_key for zero-downtime key rotation, alongside Webhook/SMTP metadata and Firecrawl base URL.
  • GENERAL (16): Crawler limits (max_pages, timeout_ms, concurrency), storage paths, and app metadata.
  • SECURITY (11): User quotas (max_pages, jobs_per_day), Rate Limits (rate_limit.max_requests), and auth session timeouts.
• Cleaned up 16 static/infrastructure secrets (IAM keys, Redis bootstrap, AES master KEK) from DB to preserve Zero-Trust principles.

2. Architecture & Modules (src/modules/system-config/)

• Followed 5-layer architecture: DTOs, Zod Validation, Repository (exclusive Prisma), Service, Controller, and Routes.
• Two-Level Cache:
  • L1 (In-Memory Map): Sub-millisecond lookup latency (< 0.1ms), 5-minute TTL.
  • L2 (Redis Pub/Sub): Listens to system_config:events to invalidate caches across cluster instances instantly upon mutation.
• RBAC & Audit:
  • Added permissions: SYSTEM_CONFIG_READ, SYSTEM_CONFIG_MANAGE (assigned to ADMIN).
  • Audit logging on create, update, toggle, and delete events.
• Application Wiring:
  • auth.service.ts: Checks feature.registration.enabled before user signup.
  • maintenance.middleware.ts: Intercepts non-admin traffic with HTTP 503 when maintenance mode is active.
  • rate-limit.middleware.ts: Reads rate_limit.max_requests dynamically.
  • crawl.worker.processor.ts: Checks change-detection and webhook flags during crawl execution.
  • firecrawl.client.ts: Resolves integration.firecrawl.api_key dynamically from cache with .env fallback.