feat(crawler): support real-time page persistence, partial export and prevent duplicate rerun (!12) · Merge Requests · ThinhNC / upgrade-data-crawler-be

feat(crawler): support real-time page persistence, partial export and prevent duplicate rerun

Overview

This pull request introduces real-time incremental page persistence, partial data export capability, safe cancellation handling, and duplicate rerun prevention to enhance reliability and data safety during crawl jobs.

Key Changes

1. Real-time Incremental Page Persistence

Firecrawl Polling (firecrawl.service.ts): Updated onProgress callback to stream currently scraped pages (currentPages) during active polling iterations for both full crawls and batch scrapes.
Worker Processor (crawl.worker.processor.ts):
- Implemented persistIncrementalPages and persistSinglePage.
- Crawled pages are now immediately normalized and upserted to the database (crawl_pages) during execution instead of waiting for the job to complete.
- Added URL deduplication tracking (persistedUrls) to ensure idempotency.

2. Job Cancellation & State Preservation

Enhanced cancel flow in crawl.worker.processor.ts so when a job is canceled mid-execution:
- All previously persisted pages are safely preserved.
- successPages correctly reflects actual pages saved.
- Job status remains CANCELED instead of being erroneously overridden to FAILED.

3. Partial Data Export Support

Export Service (crawl-export.service.ts):
- Relaxed export eligibility to allow exports for jobs with status === "COMPLETED" as well as status IN ("CANCELED", "FAILED") as long as successPages > 0.
- Appended warnings and metadata indicating partial dataset export.

4. Duplicate Rerun Prevention (Concurrency Guard)

Job Service & Repository (crawl-job.service.ts, crawl-job.repository.ts):
- Introduced static in-memory concurrency mutex (rerunLocks) to prevent rapid consecutive clicks from creating multiple duplicate pending jobs.
- Added findRecentActiveJob to deduplicate jobs requested with the same target URL and config within a 5-second window.

## Overview
This pull request introduces **real-time incremental page persistence**, **partial data export capability**, **safe cancellation handling**, and **duplicate rerun prevention** to enhance reliability and data safety during crawl jobs.

---

## Key Changes

### 1. Real-time Incremental Page Persistence
- **Firecrawl Polling (`firecrawl.service.ts`)**: Updated `onProgress` callback to stream currently scraped pages (`currentPages`) during active polling iterations for both full crawls and batch scrapes.
- **Worker Processor (`crawl.worker.processor.ts`)**:
  - Implemented `persistIncrementalPages` and `persistSinglePage`.
  - Crawled pages are now immediately normalized and upserted to the database (`crawl_pages`) during execution instead of waiting for the job to complete.
  - Added URL deduplication tracking (`persistedUrls`) to ensure idempotency.

### 2. Job Cancellation & State Preservation
- Enhanced cancel flow in `crawl.worker.processor.ts` so when a job is canceled mid-execution:
  - All previously persisted pages are safely preserved.
  - `successPages` correctly reflects actual pages saved.
  - Job status remains `CANCELED` instead of being erroneously overridden to `FAILED`.

### 3. Partial Data Export Support
- **Export Service (`crawl-export.service.ts`)**:
  - Relaxed export eligibility to allow exports for jobs with `status === "COMPLETED"` as well as `status IN ("CANCELED", "FAILED")` as long as `successPages > 0`.
  - Appended warnings and metadata indicating partial dataset export.

### 4. Duplicate Rerun Prevention (Concurrency Guard)
- **Job Service & Repository (`crawl-job.service.ts`, `crawl-job.repository.ts`)**:
  - Introduced static in-memory concurrency mutex (`rerunLocks`) to prevent rapid consecutive clicks from creating multiple duplicate pending jobs.
  - Added `findRecentActiveJob` to deduplicate jobs requested with the same target URL and config within a 5-second window.