fix(crawl-jobs): implement soft delete for quota retention and auto-complete finished jobs (!13) · Merge Requests · ThinhNC / upgrade-data-crawler-be

fix(crawl-jobs): implement soft delete for quota retention and auto-complete finished jobs

Summary of Changes

1. Daily Quota Retention via Soft Delete

Problem: Previously, deleting a crawl job triggered a hard delete (DELETE FROM crawl_jobs), causing countJobsSince and sumPagesCrawledByUser to decrease and prematurely resetting the user's daily quota (maxJobsPerDayLimit).
Solution:
- Added deletedAt (DateTime?) and deletedBy (String?) to the CrawlJob model in schema.prisma with corresponding indexes (deleted_at, [user_id, deleted_at]).
- Generated and applied migration 20260907123000_add_crawl_job_soft_delete.
- Updated CrawlJobRepository.delete() to soft-delete the job record while cleaning up child tables (crawlAsset, crawlJobLog, crawlExport, crawlPage) and storage files.
- Excluded soft-deleted jobs from findById, find, findByScheduleId, countConcurrentJobs, and dashboard stats queries.
- Kept soft-deleted jobs included in countJobsSince and sumPagesCrawledByUser to preserve daily usage integrity.

2. Auto-completion for Stalled / Orphaned Finished Jobs

Problem: Seed job 088f635c-9c3a-4467-93bb-e58f001bf001 (and potential stalled jobs) remained in RUNNING status indefinitely even though all 52 target pages were already processed (49 success, 3 failed), preventing data export.
Solution:
- In CrawlJobService.findById and findAllByUser, automatically detect jobs in RUNNING status where processedPages >= targetPages and update them to COMPLETED with finishedAt.
- In CrawlExportService.createExport, allow export generation by auto-completing finished jobs before validating export readiness.
- Updated prisma/seed.ts to set sample Job 1 status to COMPLETED.

## Summary of Changes

### 1. Daily Quota Retention via Soft Delete
- **Problem**: Previously, deleting a crawl job triggered a hard delete (`DELETE FROM crawl_jobs`), causing `countJobsSince` and `sumPagesCrawledByUser` to decrease and prematurely resetting the user's daily quota (`maxJobsPerDayLimit`).
- **Solution**:
  - Added `deletedAt` (`DateTime?`) and `deletedBy` (`String?`) to the `CrawlJob` model in `schema.prisma` with corresponding indexes (`deleted_at`, `[user_id, deleted_at]`).
  - Generated and applied migration `20260907123000_add_crawl_job_soft_delete`.
  - Updated `CrawlJobRepository.delete()` to soft-delete the job record while cleaning up child tables (`crawlAsset`, `crawlJobLog`, `crawlExport`, `crawlPage`) and storage files.
  - Excluded soft-deleted jobs from `findById`, `find`, `findByScheduleId`, `countConcurrentJobs`, and dashboard stats queries.
  - Kept soft-deleted jobs included in `countJobsSince` and `sumPagesCrawledByUser` to preserve daily usage integrity.

### 2. Auto-completion for Stalled / Orphaned Finished Jobs
- **Problem**: Seed job `088f635c-9c3a-4467-93bb-e58f001bf001` (and potential stalled jobs) remained in `RUNNING` status indefinitely even though all 52 target pages were already processed (49 success, 3 failed), preventing data export.
- **Solution**:
  - In `CrawlJobService.findById` and `findAllByUser`, automatically detect jobs in `RUNNING` status where `processedPages >= targetPages` and update them to `COMPLETED` with `finishedAt`.
  - In `CrawlExportService.createExport`, allow export generation by auto-completing finished jobs before validating export readiness.
  - Updated `prisma/seed.ts` to set sample Job 1 status to `COMPLETED`.