feat(system-config): sync configs with .env and add 7-day retention cron jobs (!19) · Merge Requests · ThinhNC / upgrade-data-crawler-be

feat(system-config): sync configs with .env and add 7-day retention cron jobs

Summary of Changes

1. 7-Day Retention Cleanup Cron Jobs

BullMQ Cron Implementation: Added cleanup-exports scheduled job running daily at 03:00 AM (0 3 * * *), complementing the existing cleanup-audit-logs job at 02:00 AM (0 2 * * *).
Retention Period: Updated default retention policy from 30 days to 7 days (DEFAULT_AUDIT_LOG_RETENTION_DAYS = 7, DEFAULT_EXPORT_RETENTION_DAYS = 7).
Storage Cleanup: Automatically deletes both database records and physical export files via StorageService.deleteFile() (supporting AWS S3, Supabase Storage, and local disk).
Dynamic Configuration & Feature Flags: Integrated SystemConfigService dynamic checks for retention.audit_logs_days, retention.exports_days, feature.cron.cleanup_audit_logs.enabled, and feature.cron.cleanup_exports.enabled.

2. Synchronization of System Configs & Feature Flags with `.env`

Environment Driven Defaults: Updated DEFAULT_SYSTEM_CONFIGS in system-config.constant.ts to dynamically resolve values from process.env.*:
- Supabase Storage / S3 configs: S3_ENDPOINT, S3_REGION, S3_BUCKET, S3_FORCE_PATH_STYLE, STORAGE_DRIVER, STORAGE_EXPORT_DIR.
- Crawler, JWT, Quota, Rate Limit, SMTP, and Firecrawl configurations.
Upsert Value Sync: Updated ensureDefault in system-config.repository.ts and ensureDefaultConfigs(syncValues = true) in system-config.service.ts to update existing database values during initialization or synchronization.
New API Endpoint: Added POST /api/v1/system/configs/sync-env protected by system_configs.manage permission to trigger immediate sync and flush Redis cache on demand.
Migration Script: Enhanced scripts/sync-system-configs.ts (pnpm config:sync) to upsert values into Postgres.

## Summary of Changes

### 1. 7-Day Retention Cleanup Cron Jobs
- **BullMQ Cron Implementation**: Added `cleanup-exports` scheduled job running daily at 03:00 AM (`0 3 * * *`), complementing the existing `cleanup-audit-logs` job at 02:00 AM (`0 2 * * *`).
- **Retention Period**: Updated default retention policy from 30 days to 7 days (`DEFAULT_AUDIT_LOG_RETENTION_DAYS = 7`, `DEFAULT_EXPORT_RETENTION_DAYS = 7`).
- **Storage Cleanup**: Automatically deletes both database records and physical export files via `StorageService.deleteFile()` (supporting AWS S3, Supabase Storage, and local disk).
- **Dynamic Configuration & Feature Flags**: Integrated `SystemConfigService` dynamic checks for `retention.audit_logs_days`, `retention.exports_days`, `feature.cron.cleanup_audit_logs.enabled`, and `feature.cron.cleanup_exports.enabled`.

### 2. Synchronization of System Configs & Feature Flags with `.env`
- **Environment Driven Defaults**: Updated `DEFAULT_SYSTEM_CONFIGS` in `system-config.constant.ts` to dynamically resolve values from `process.env.*`:
  - Supabase Storage / S3 configs: `S3_ENDPOINT`, `S3_REGION`, `S3_BUCKET`, `S3_FORCE_PATH_STYLE`, `STORAGE_DRIVER`, `STORAGE_EXPORT_DIR`.
  - Crawler, JWT, Quota, Rate Limit, SMTP, and Firecrawl configurations.
- **Upsert Value Sync**: Updated `ensureDefault` in `system-config.repository.ts` and `ensureDefaultConfigs(syncValues = true)` in `system-config.service.ts` to update existing database values during initialization or synchronization.
- **New API Endpoint**: Added `POST /api/v1/system/configs/sync-env` protected by `system_configs.manage` permission to trigger immediate sync and flush Redis cache on demand.
- **Migration Script**: Enhanced `scripts/sync-system-configs.ts` (`pnpm config:sync`) to upsert values into Postgres.