feat(core): implement essential saas and crawler lifecycle features (!3) · Merge Requests · ThinhNC / upgrade-data-crawler-be

feat(core): implement essential saas and crawler lifecycle features

Implement Core Missing SaaS & Crawler Lifecycle Features

Overview

This pull request implements 5 essential features required for a production-grade Data Crawler / SaaS backend, following the 5-layer architectural pattern (Route -> Controller -> Service -> Repository -> Prisma) defined in AGENTS.md.

Key Features Added

1. User Profile & Quota Usage Monitoring

Avatar Support: Added nullable avatarUrl to model User in prisma/schema.prisma (migration: 20260903000000_add_user_avatar_url) and supported profile updates via PUT /api/v1/auth/me.
Quota & Usage Endpoint (GET /api/v1/auth/me/usage): Computes real-time daily quota consumption based on business timezone Asia/Ho_Chi_Minh (UTC+7), tracking today's executed jobs, remaining jobs, active concurrent crawls, and total pages extracted.

2. Crawl Job Lifecycle (Delete & Re-run)

Job Deletion (DELETE /api/v1/crawl-jobs/:id): Enforces tenant ownership checks to prevent IDOR, guards against deleting active RUNNING jobs, cleans up associated export and diff files from storage (Local / S3), and cascades record deletion atomically in a database transaction.
Job Re-run / Retry (POST /api/v1/crawl-jobs/:id/rerun): Clones existing job configuration (startUrl, mode, maxPages, maxDepth, urls), validates quota limits, and enqueues a new job into BullMQ.

3. Job Execution Logging (`CrawlJobLog`)

Worker Step-by-Step Logging: The BullMQ worker processor (crawl.worker.processor.ts) records lifecycle checkpoints (INITIALIZE, FETCH, PROCESS, EXPORT, DIFF, COMPLETED, FAILED) into crawl_job_logs.
Logs Inspection (GET /api/v1/crawl-jobs/:id/logs): Paginated endpoint for frontend clients to inspect granular execution logs chronologically.

4. Webhook Management & Testing

Ping Test (POST /api/v1/webhooks/configs/:id/test): Sends a mock test.ping payload with an HMAC-SHA256 signature (X-Webhook-Signature) to the user's destination URL and records execution metrics in webhook_deliveries.
Config Update (PATCH /api/v1/webhooks/configs/:id): Allows modifying URL, subscribed events, active state, or secrets (automatically re-encrypted with AES-256-GCM) without losing past delivery history.

5. Dashboard Analytics & Export Storage Management

Dashboard Stats (GET /api/v1/dashboard/stats): Aggregates platform metrics including job counts by status, pages scraped (success vs. failure), active scheduled crawls, and quota balances.
Export Management: Added GET /api/v1/exports (paginated listing of user export archives) and DELETE /api/v1/exports/:id (removes archive file from storage and purges database record).

## Implement Core Missing SaaS & Crawler Lifecycle Features

### Overview
This pull request implements **5 essential features** required for a production-grade Data Crawler / SaaS backend, following the 5-layer architectural pattern (`Route -> Controller -> Service -> Repository -> Prisma`) defined in `AGENTS.md`.

---

### Key Features Added

#### 1. User Profile & Quota Usage Monitoring
- **Avatar Support**: Added nullable `avatarUrl` to `model User` in `prisma/schema.prisma` (migration: `20260903000000_add_user_avatar_url`) and supported profile updates via `PUT /api/v1/auth/me`.
- **Quota & Usage Endpoint (`GET /api/v1/auth/me/usage`)**: Computes real-time daily quota consumption based on business timezone `Asia/Ho_Chi_Minh` (UTC+7), tracking today's executed jobs, remaining jobs, active concurrent crawls, and total pages extracted.

#### 2. Crawl Job Lifecycle (Delete & Re-run)
- **Job Deletion (`DELETE /api/v1/crawl-jobs/:id`)**: Enforces tenant ownership checks to prevent IDOR, guards against deleting active `RUNNING` jobs, cleans up associated export and diff files from storage (Local / S3), and cascades record deletion atomically in a database transaction.
- **Job Re-run / Retry (`POST /api/v1/crawl-jobs/:id/rerun`)**: Clones existing job configuration (`startUrl`, `mode`, `maxPages`, `maxDepth`, `urls`), validates quota limits, and enqueues a new job into BullMQ.

#### 3. Job Execution Logging (`CrawlJobLog`)
- **Worker Step-by-Step Logging**: The BullMQ worker processor (`crawl.worker.processor.ts`) records lifecycle checkpoints (`INITIALIZE`, `FETCH`, `PROCESS`, `EXPORT`, `DIFF`, `COMPLETED`, `FAILED`) into `crawl_job_logs`.
- **Logs Inspection (`GET /api/v1/crawl-jobs/:id/logs`)**: Paginated endpoint for frontend clients to inspect granular execution logs chronologically.

#### 4. Webhook Management & Testing
- **Ping Test (`POST /api/v1/webhooks/configs/:id/test`)**: Sends a mock `test.ping` payload with an HMAC-SHA256 signature (`X-Webhook-Signature`) to the user's destination URL and records execution metrics in `webhook_deliveries`.
- **Config Update (`PATCH /api/v1/webhooks/configs/:id`)**: Allows modifying URL, subscribed events, active state, or secrets (automatically re-encrypted with AES-256-GCM) without losing past delivery history.

#### 5. Dashboard Analytics & Export Storage Management
- **Dashboard Stats (`GET /api/v1/dashboard/stats`)**: Aggregates platform metrics including job counts by status, pages scraped (success vs. failure), active scheduled crawls, and quota balances.
- **Export Management**: Added `GET /api/v1/exports` (paginated listing of user export archives) and `DELETE /api/v1/exports/:id` (removes archive file from storage and purges database record).