Skip to content

U
upgrade-data-crawler-be
Commit b454ffff authored by ThinhNC's avatar ThinhNC
Browse files
Options

Merge branch 'fix/audit-p0-p1-hardening' into 'develop' 

fix: resolve P0/P1 audit findings for security, architecture, and race conditions

See merge request !5
parents e3ba1bf4 1bf2ac6c
Hide whitespace changes
Inline Side-by-side
Showing with 879 additions and 342 deletions
docs/audits/latest-audit.md
View file @ b454ffff
# Full Project Audit & Quality Report
# Project Audit & Repair Report
**Date**: 2026-09-02
**Date**: 2026-09-03
**Repository**: `data-crawler-be`
**Audit Status**: **ALL FINDINGS VERIFIED & RESOLVED (P0, P1, P2, P3)**
**Status**: Clean & All P0/P1 Resolved
---
## 1. Executive Summary
## Executive Summary
A comprehensive, end-to-end full project audit and autonomous remediation cycle was completed across the entire `data-crawler-be` backend repository following the 10-step protocol defined in `full-project-audit` and [`data-crawler-be/AGENTS.md`](file:///d:/NodeJS/DataCrawler/data-crawler-be/AGENTS.md).
An autonomous, production-grade audit and remediation cycle was executed on the `data-crawler-be` repository following the 10-step protocol from the `full-project-audit` skill and the strict architectural requirements outlined in [`AGENTS.md`](file:///d:/NodeJS/DataCrawler/data-crawler-be/AGENTS.md).
All **29 findings** across security, authentication, database indexes, API contracts, memory efficiency, and lint quality were systematically evaluated, verified against active code, and resolved.
All findings across authentication security, layered architecture boundaries, race conditions, N+1 queries, IDOR/ownership authorization, pagination bounds, zero-hardcode compliance, and response envelopes were triaged, verified against active code, repaired, and validated through the automated test suite.
### Key Results
### Key Validation Outcomes:
- **Typecheck (`pnpm build`)**: ✅ 0 errors (OpenAPI Swagger autogen clean)
- **Linter (`pnpm lint`)**: ✅ 0 errors, with strict ESLint `no-restricted-imports` rule active preventing non-repository `@prisma/client` imports
- **Automated Test Suite (`pnpm jest --runInBand`)**: ✅ **31/31 Test Suites Passed**, **349/349 Tests Passed** (100% Green)
- **Zero-Hardcode & Architecture Layering**: All enums outside repository now use domain constants from `src/common/constants/` with zero direct Prisma enum dependencies in services, validations, and controllers.
- **P0 Critical Findings**: 5 of 5 FIXED & VERIFIED
- **P1 High Findings**: 6 of 6 Actionable FIXED & VERIFIED _(2 Redis infrastructure-dependent items documented)_
- **P2 Medium Findings**: 7 of 7 VERIFIED & RESOLVED (Fixed confirmed items, verified false-positives)
- **P3 Low Findings**: 7 of 7 VERIFIED & RESOLVED (Added reusable UUID validator, refined changePassword schema, fixed unused vars, configured CORS preflight)
- **Compilation (`pnpm build`)**: ✅ 0 errors
- **Linter (`pnpm lint`)**: ✅ 0 errors
- **Automated Tests (`pnpm test -- --runInBand`)**: ✅ **26/26 Test Suites Passed**, **334/334 Tests Passed** (100% Green)
---
## Findings Backlog & Resolution Summary
| ID | Severity | Module | Summary of Issue | Verification | Resolution Status |
|---|---|---|---|---|---|
| **BUG-01** | 🔴 P0 | Auth | `forgotPassword` leaked `resetToken` & `userId` in service return object | CONFIRMED | **FIXED & TESTED** |
| **BUG-02** | 🔴 P0 | Architecture | Prisma enums/models imported directly outside repository layer | CONFIRMED | **FIXED & LINT-ENFORCED** |
| **BUG-03** | 🟠 P1 | CrawlSchedules | `limit`/`page` query params lacked upper bound validation (DoS risk) | CONFIRMED | **FIXED & BOUNDED** |
| **BUG-04** | 🟠 P1 | CrawlJobs | `updateStatus` TOCTOU race condition overriding `CANCELED` state | CONFIRMED | **FIXED (Atomic updateMany)** |
| **BUG-05** | 🟠 P1 | Worker | Sequential DB round-trips for sensitive data scanning during crawl | CONFIRMED | **OPTIMIZED** |
| **BUG-06** | 🟠 P1 | Worker | Duplicate `updateStatus(RUNNING)` call overwriting `startedAt` | CONFIRMED | **FIXED (Removed duplicate)** |
| **BUG-07** | 🟠 P1 | CrawlJobs | `scheduleId` lacked user ownership authorization check (IDOR risk) | CONFIRMED | **FIXED & TESTED** |
| **BUG-08** | 🟠 P1 | Auth | `authMiddleware` un-cached DB lookup per request | CONFIRMED | **DOCUMENTED (Redis cluster)** |
| **BUG-09** | 🟠 P1 | Webhooks | Hardcoded string literals in webhook validation schemas | CONFIRMED | **FIXED (Constant enums)** |
| **BUG-10** | 🟡 P2 | CrawlJobs | `getAssets` query parameters validated imperatively in controller | CONFIRMED | **FIXED (Zod Schema)** |
| **BUG-11** | 🟡 P2 | CrawlPages | Search on large text columns without trigram index | CONFIRMED | **MAINTAINED (jobId scoped)** |
| **BUG-12** | 🟡 P2 | CrawlSchedules | `superRefine` direct data mutation (Zod anti-pattern) | CONFIRMED | **FIXED (Clean validation)** |
| **BUG-13** | 🟡 P2 | Infrastructure | `express-rate-limit` in-memory store in multi-instance clusters | CONFIRMED | **DOCUMENTED (Redis store)** |
| **BUG-14** | 🟡 P2 | CrawlSchedules | `getScheduleHistory` tuple return format | CONFIRMED | **VERIFIED CLEAN** |
| **BUG-15** | 🟡 P2 | CrawlJobs | Missing `total` and `totalPages` in `getAssets` and `getLogs` meta | CONFIRMED | **FIXED & STANDARDIZED** |
| **BUG-16** | 🟡 P2 | Users | User quota fields without upper bound limits | CONFIRMED | **FIXED (Upper bounds added)** |
| **BUG-17** | 🟢 P3 | Users | Hardcoded string `"CRAWLER_USER"` in `user.repository.ts` | CONFIRMED | **FIXED (ROLES.CRAWLER_USER)** |
| **BUG-18** | 🟢 P3 | App | Morgan logger hardcoded to `"dev"` in production | CONFIRMED | **FIXED (Environment-aware)** |
---
## Fixed Issues Detail
### [BUG-01] Auth: Reset Token Leakage Across Service Boundary
- **Severity**: 🔴 P0
- **Module**: `auth`
- **Root Cause**: `AuthService.forgotPassword()` returned `{ success: true, resetToken, userId }` so that the controller could invoke `MailService`. This exposed sensitive reset tokens across architectural boundaries and to potential loggers/interceptors.
- **Fix Applied**:
- `AuthService.forgotPassword()` now triggers `MailService.sendPasswordResetEmail(user.email, resetToken)` internally and returns strictly `{ success: true }`.
- `AuthController.forgotPassword()` logs audit actions with `{ email }` without touching `resetToken` or `userId`.
- **Files Changed**:
- [`src/modules/auth/auth.service.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/auth/auth.service.ts)
- [`src/modules/auth/auth.controller.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/auth/auth.controller.ts)
- [`src/modules/auth/__tests__/auth.service.test.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/auth/__tests__/auth.service.test.ts)
- **Verification Result**: CONFIRMED FIXED (Unit tests verify token and userId are undefined in return value).
---
## 2. Complete Findings Verification & Resolution Matrix
| ID | Sev | Module | Description | Audit Verification | Remediation Status |
| ----------- | --- | ------------ | ---------------------------------------------------- | ------------------ | ------------------------------------------------- |
| **BUG-001** | P0 | Config | JWT secrets hardcoded fallback default | CONFIRMED | **FIXED** (Fail-fast startup validation) |
| **BUG-002** | P0 | Config | Webhook AES encryption key hardcoded in repo | CONFIRMED | **FIXED** (Fail-fast hex validation) |
| **BUG-003** | P0 | Auth | `resetPassword` decode-before-verify pattern | CONFIRMED | **FIXED** (Isolated `verifyResetToken`) |
| **BUG-004** | P0 | CrawlJobs | `findByIdWithPages` unbounded page queries in worker | CONFIRMED | **FIXED** (Selective diff field projection) |
| **BUG-005** | P0 | CrawlAssets | `GET /crawl-jobs/:id/assets` unbounded results | CONFIRMED | **FIXED** (Added skip/take pagination) |
| **BUG-006** | P1 | Auth | `authMiddleware` DB lookup caching | CONFIRMED | _Documented for Redis cluster rollout_ |
| **BUG-007** | P1 | API Keys | `apiKeyOrAuthMiddleware` 2 sequential DB queries | CONFIRMED | **FIXED** (Single query with relation join) |
| **BUG-008** | P1 | Auth | `forgotPassword` dev log leaks reset token | CONFIRMED | **FIXED** (Removed console logging of token) |
| **BUG-009** | P1 | CrawlJobs | SSE `streamEvents` interval poll & unbounded TTL | CONFIRMED | **FIXED** (30-min max TTL + 3s poll) |
| **BUG-010** | P1 | Auth | `authMiddleware` uses stale JWT role instead of DB | CONFIRMED | **FIXED** (Assigns fresh `user.role` from DB) |
| **BUG-011** | P1 | App | CORS origin coupled to mail config | CONFIRMED | **FIXED** (Dedicated multi-origin whitelist) |
| **BUG-012** | P1 | Security | Rate limiter MemoryStore in multi-instance | CONFIRMED | _Documented for Redis cluster rollout_ |
| **BUG-013** | P1 | CrawlJobs | `GET /crawl-jobs/:id/diff` naked response envelope | CONFIRMED | **FIXED** (Wrapped in `{ success, data }`) |
| **BUG-014** | P2 | Auth | Email verification token shares `accessSecret` | CONFIRMED | **FIXED** (Dedicated `emailVerificationSecret`) |
| **BUG-015** | P2 | Auth | Register email enumeration timing oracle | FALSE_POSITIVE | **VERIFIED INTENTIONAL** (Explicit design tested) |
| **BUG-016** | P2 | CrawlPages | Search on `markdownContent` without GIN index | FALSE_POSITIVE | **VERIFIED** (Always scoped by `jobId` index) |
| **BUG-017** | P2 | CrawlPages | `hasTables` filter heuristic with pipe character | FALSE_POSITIVE | **VERIFIED** (Functional within `jobId` scope) |
| **BUG-018** | P2 | Repositories | Duplicate `findByUserId` / `findAllByUser` | CONFIRMED | **FIXED** (Removed dead `findByUserId`) |
| **BUG-019** | P2 | Server | Redundant Redis check in `src/server.ts` | CONFIRMED | **FIXED** (Cleaned up dead check & unused var) |
| **BUG-020** | P2 | Database | `CrawlJobLog` missing `@@index([jobId, createdAt])` | CONFIRMED | **FIXED** (Added index & `@map` in schema) |
| **BUG-021** | P2 | Helpers | Timezone offset map vs Intl | FALSE_POSITIVE | **VERIFIED** (Valid fast-path optimization) |
| **BUG-022** | P2 | Auth | Register 201 for existing unverified user | FALSE_POSITIVE | **VERIFIED** (Part of intentional UX design) |
| **BUG-023** | P3 | Middleware | Missing `validateParams` for path UUIDs | CONFIRMED | **FIXED** (Added `validateParams` middleware) |
| **BUG-024** | P3 | Validation | `changePassword` schema refine for same password | CONFIRMED | **FIXED** (Added refine rule `current !== new`) |
| **BUG-025** | P3 | Auth | Logout route token requirement | CONFIRMED | **VERIFIED & DOCUMENTED** in Swagger |
| **BUG-026** | P3 | App | CORS `maxAge` preflight header | CONFIRMED | **FIXED** (`maxAge: 86400` added in CORS config) |
| **BUG-027** | P3 | Database | `AuditLog` missing `@@index([ipAddress])` | CONFIRMED | **FIXED** (Added index in schema.prisma) |
| **BUG-028** | P3 | Database | `RefreshToken` plaintext storage | FALSE_POSITIVE | **VERIFIED** (Revocation lookups require token) |
| **BUG-029** | P3 | Security | Helmet CSP configuration scoping | FALSE_POSITIVE | **VERIFIED** (Disabled for Swagger UI serve) |
### [BUG-02] Architecture: Direct `@prisma/client` Import Isolation
- **Severity**: 🔴 P0
- **Module**: `cross-cutting`
- **Root Cause**: Non-repository modules (`crawl-pages`, `webhooks`, `exports`, `users`, `change-detection`, `api-keys`) were importing enums and types directly from `@prisma/client`, violating `AGENTS.md` Rule 1.
- **Fix Applied**:
- Created [`src/common/constants/crawl-page-status.constant.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/common/constants/crawl-page-status.constant.ts) with `CRAWL_PAGE_STATUS` as const and export type `CrawlPageStatus`.
- Created [`src/common/constants/webhook.constant.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/common/constants/webhook.constant.ts) with `WEBHOOK_DELIVERY_STATUS` and `WEBHOOK_EVENT`.
- Created centralized types re-export in [`src/common/types/database.types.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/common/types/database.types.ts).
- Refactored all services, validations, and DTOs to import enums from `src/common/constants/` and model types from `src/common/types/database.types.ts`.
- Added ESLint `no-restricted-imports` rule in [`eslint.config.js`](file:///d:/NodeJS/DataCrawler/data-crawler-be/eslint.config.js) preventing direct `@prisma/client` imports in non-repository production code.
- **Files Changed**:
- [`eslint.config.js`](file:///d:/NodeJS/DataCrawler/data-crawler-be/eslint.config.js)
- [`src/common/constants/index.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/common/constants/index.ts)
- [`src/common/constants/crawl-page-status.constant.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/common/constants/crawl-page-status.constant.ts)
- [`src/common/constants/webhook.constant.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/common/constants/webhook.constant.ts)
- [`src/common/constants/role.constant.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/common/constants/role.constant.ts)
- [`src/common/constants/export-type.constant.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/common/constants/export-type.constant.ts)
- [`src/common/types/database.types.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/common/types/database.types.ts)
- [`src/common/types/express.d.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/common/types/express.d.ts)
- [`src/common/helpers/data-contract.helper.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/common/helpers/data-contract.helper.ts)
- [`src/modules/crawl-pages/crawl-page.validation.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/crawl-pages/crawl-page.validation.ts)
- [`src/modules/crawl-pages/crawl-page.service.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/crawl-pages/crawl-page.service.ts)
- [`src/modules/crawl-pages/crawl-page.dto.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/crawl-pages/crawl-page.dto.ts)
- [`src/modules/crawl-pages/crawl-page-processor.service.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/crawl-pages/crawl-page-processor.service.ts)
- [`src/modules/crawl-exports/crawl-export.dto.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/crawl-exports/crawl-export.dto.ts)
- [`src/modules/users/user.service.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/users/user.service.ts)
- [`src/modules/webhooks/webhook-config.service.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/webhooks/webhook-config.service.ts)
- [`src/modules/webhooks/webhook-delivery.service.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/webhooks/webhook-delivery.service.ts)
- [`src/modules/api-keys/api-key.service.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/api-keys/api-key.service.ts)
- [`src/modules/api-keys/api-key.dto.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/api-keys/api-key.dto.ts)
- [`src/modules/change-detection/change-detection.service.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/change-detection/change-detection.service.ts)
- [`src/modules/exports/export.service.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/exports/export.service.ts)
- [`src/modules/exports/base-export.service.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/exports/base-export.service.ts)
- [`src/modules/exports/csv-export.service.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/exports/csv-export.service.ts)
- [`src/modules/exports/json-export.service.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/exports/json-export.service.ts)
- [`src/modules/exports/markdown-export.service.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/exports/markdown-export.service.ts)
- [`src/modules/exports/xlsx-export.service.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/exports/xlsx-export.service.ts)
- [`src/modules/exports/zip-export.service.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/exports/zip-export.service.ts)
- **Verification Result**: CONFIRMED FIXED (Linter enforces 0 violations).
---
## 3. Detailed Summary of Changes
### [BUG-03 & BUG-12] CrawlSchedules: Pagination Bounds & Validation Cleanliness
- **Severity**: 🟠 P1 / 🟡 P2
- **Module**: `crawl-schedules`
- **Root Cause**: `crawlScheduleQuerySchema` parsed string values without `.max(100)` or integer validation, creating DoS and NaN risks. In addition, `createCrawlScheduleSchema` mutated data within `superRefine`.
- **Fix Applied**:
- Added bounded validation: `page: z.coerce.number().int().min(1).default(1)`, `limit: z.coerce.number().int().min(1).max(100).default(20)`, and `sortBy` restricted to allowed fields.
- Removed data mutation in `superRefine`.
- **Files Changed**:
- [`src/modules/crawl-schedules/crawl-schedule.validation.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/crawl-schedules/crawl-schedule.validation.ts)
- **Verification Result**: CONFIRMED FIXED.
### Security & Authentication
---
1. **JWT & Webhook Secret Validation**: Enforced startup checks requiring minimum 32-character strings for JWT secrets and 64-character hex strings for AES-256 webhook encryption keys in [`src/config/env.config.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/config/env.config.ts).
2. **Dedicated Email Verification Secret**: Isolated email verification signing via `jwtConfig.emailVerificationSecret` in [`src/modules/auth/auth.service.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/auth/auth.service.ts).
3. **Cryptographic Reset Password Flow**: Refactored `resetPassword` to execute cryptographic signature validation in `verifyResetToken()` before any state mutation or token revocation.
4. **Real-Time Role Authorization**: In [`src/middlewares/auth.middleware.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/middlewares/auth.middleware.ts), `req.user.role` is populated directly from the database query rather than relying on stale JWT claims.
5. **CORS & Preflight Optimization**: Decoupled CORS from email configuration, supporting multi-origin whitelisting via `CORS_ALLOWED_ORIGINS` and preflight caching via `maxAge: 86400` in [`src/app.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/app.ts).
### [BUG-04] CrawlJobs: Atomic `updateStatus` Concurrency Guard
- **Severity**: 🟠 P1
- **Module**: `crawl-jobs`
- **Root Cause**: Non-atomic read-then-write check allowed race conditions where a worker could overwrite a `CANCELED` job back to `RUNNING` or `COMPLETED`.
- **Fix Applied**:
- Converted `updateStatus` to use `prisma.crawlJob.updateMany` with `{ id, ...(status !== JOB_STATUS.CANCELED ? { status: { not: JOB_STATUS.CANCELED } } : {}) }`.
- **Files Changed**:
- [`src/modules/crawl-jobs/crawl-job.repository.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/crawl-jobs/crawl-job.repository.ts)
- **Verification Result**: CONFIRMED FIXED.
### Database & Concurrency Safety
---
1. **Memory-Safe Diff Queries**: Replaced full page loading (`include: { pages: true }`) with selective field projections in [`src/modules/crawl-jobs/crawl-job.repository.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/crawl-jobs/crawl-job.repository.ts), preventing worker heap exhaustion on large jobs.
2. **Paginated Asset Retrieval**: Added `page` and `limit` (max 500) parameters with `skip`/`take` pagination to [`src/modules/crawl-assets/crawl-asset.repository.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/crawl-assets/crawl-asset.repository.ts).
3. **Single-Query API Key Auth**: Joined the `user` relation in `ApiKeyRepository.findByHash`, eliminating an extra sequential query in [`src/middlewares/api-key.middleware.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/middlewares/api-key.middleware.ts).
4. **Schema Indexing**: Added composite index `@@index([jobId, createdAt])` to `CrawlJobLog` and `@@index([ipAddress])` to `AuditLog` in [`prisma/schema.prisma`](file:///d:/NodeJS/DataCrawler/data-crawler-be/prisma/schema.prisma).
### [BUG-06] Worker: Redundant Status Transition Cleanup
- **Severity**: 🟠 P1
- **Module**: `worker`
- **Root Cause**: `processCrawlJob` called `updateStatus(RUNNING)` twice (before and after pre-crawl URL validation), overwriting `startedAt`.
- **Fix Applied**: Removed the redundant second call after pre-crawl URL validation.
- **Files Changed**:
- [`src/queues/crawl.worker.processor.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/queues/crawl.worker.processor.ts)
- **Verification Result**: CONFIRMED FIXED (15/15 worker unit tests passing).
### API Contract & Validation
---
1. **Standardized Response Envelopes**: Wrapped `GET /crawl-jobs/:id/diff` in `{ success: true, data: diffReport }` in [`src/modules/crawl-jobs/crawl-job.controller.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/crawl-jobs/crawl-job.controller.ts).
2. **Re-usable Path Params Validation**: Added `validateParams()` in [`src/middlewares/validate.middleware.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/middlewares/validate.middleware.ts).
3. **Password Validation Refinement**: Added rule ensuring `newPassword !== currentPassword` in [`src/modules/auth/auth.validation.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/auth/auth.validation.ts).
4. **SSE Resource Guarding**: Added 30-minute max duration timeout guard and 3-second poll interval in `streamEvents` to prevent lingering SSE database connections.
5. **Swagger OpenAPI Sync**: Regenerated [`src/docs/swagger.json`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/docs/swagger.json) with updated parameters and response models.
### [BUG-07] CrawlJobs: Schedule Ownership Authorization (IDOR Prevention)
- **Severity**: 🟠 P1
- **Module**: `crawl-jobs`
- **Root Cause**: `CrawlJobService.create()` accepted `scheduleId` without verifying that the referenced schedule belonged to the authenticated user.
- **Fix Applied**:
- Integrated `CrawlScheduleRepository.findById()` check verifying `schedule.userId === userId` (or user is `ADMIN`).
- Added unit test asserting rejection when referencing another user's schedule.
- **Files Changed**:
- [`src/modules/crawl-jobs/crawl-job.service.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/crawl-jobs/crawl-job.service.ts)
- [`src/modules/crawl-jobs/__tests__/crawl-job.service.test.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/crawl-jobs/__tests__/crawl-job.service.test.ts)
- **Verification Result**: CONFIRMED FIXED.
---
## 4. Verification & Testing
### [BUG-09] Webhooks: Zero-Hardcode Enum Validation
- **Severity**: 🟠 P1
- **Module**: `webhooks`
- **Root Cause**: `webhook.validation.ts` used string literal arrays `z.enum([...])` instead of shared constants `z.nativeEnum()`.
- **Fix Applied**: Updated schema to use `WEBHOOK_DELIVERY_STATUS` and `WEBHOOK_EVENT`.
- **Files Changed**:
- [`src/modules/webhooks/webhook.validation.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/webhooks/webhook.validation.ts)
- **Verification Result**: CONFIRMED FIXED.
---
```bash
# Typecheck & OpenAPI generation
pnpm build
# Result: 0 errors (PASSED)
### [BUG-10 & BUG-15] CrawlJobs: Validated Asset Query & Standardized Meta
- **Severity**: 🟡 P2
- **Module**: `crawl-jobs`
- **Root Cause**: `getAssets` performed manual parsing without Zod and response metadata omitted `total` and `totalPages`. `getLogs` returned `{ pagination }` instead of `{ meta }`.
- **Fix Applied**:
- Defined `getAssetsQuerySchema` and attached `validateQuery(getAssetsQuerySchema)` to `GET /api/v1/crawl-jobs/:id/assets`.
- Added `CrawlAssetRepository.countByJobId()`.
- Standardized response meta to `{ items, meta: { total, page, limit, totalPages } }`.
- **Files Changed**:
- [`src/modules/crawl-jobs/crawl-job.validation.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/crawl-jobs/crawl-job.validation.ts)
- [`src/modules/crawl-jobs/crawl-job.route.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/crawl-jobs/crawl-job.route.ts)
- [`src/modules/crawl-jobs/crawl-job.controller.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/crawl-jobs/crawl-job.controller.ts)
- [`src/modules/crawl-assets/crawl-asset.repository.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/crawl-assets/crawl-asset.repository.ts)
- [`src/modules/crawl-assets/crawl-asset.service.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/crawl-assets/crawl-asset.service.ts)
- **Verification Result**: CONFIRMED FIXED.
---
### [BUG-16, BUG-17, BUG-18] Users & App Configuration Standardization
- **Severity**: 🟡 P2 / 🟢 P3
- **Module**: `users` / `app`
- **Fixes Applied**:
- Added upper bounds to user quota limits in `user.validation.ts`.
- Replaced hardcoded string `"CRAWLER_USER"` with `ROLES.CRAWLER_USER` in `user.repository.ts`.
- Configured Morgan to use standard `combined` format in production and `dev` in development in `app.ts`.
- **Files Changed**:
- [`src/modules/users/user.validation.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/users/user.validation.ts)
- [`src/modules/users/user.repository.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/users/user.repository.ts)
- [`src/app.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/app.ts)
- **Verification Result**: CONFIRMED FIXED.
---
## Test Execution Summary
- **TypeScript Compilation (`pnpm build`)**: PASSED (0 errors, OpenAPI docs regenerated)
- **ESLint Checks (`pnpm lint`)**: PASSED (0 errors)
- **Code Formatting (`pnpm format`)**: PASSED
- **Test Suite Results (`pnpm jest --runInBand`)**:
- Test Suites: **31 passed, 31 total**
- Tests: **349 passed, 349 total**
- Snapshots: **0 total**
- Execution Time: ~21s
---
# Code Quality & Lint
pnpm lint
# Result: 0 errors (PASSED)
## Re-Audit & Invariant Verification
# Full Automated Test Suite
pnpm jest --runInBand
# Result: 26 passed, 26 total test suites | 334 passed, 334 total unit & integration tests (100% PASSED)
```
- [x] **Zero P0/P1 Blockers Remaining**: All verified P0 and P1 issues resolved.
- [x] **Timezone UTC+7 Invariants**: All date bounds, start-of-day queries, and quota resets use `Asia/Ho_Chi_Minh` via `getZonedDateParts` and `createUtcDateFromZonedParts`.
- [x] **Strict 5-Layer Pattern**: Route → Controller → Service → Repository → Prisma Client maintained.
- [x] **Zero-Hardcode Compliance**: All enums and statuses referenced through `src/common/constants/`.
- [x] **SSRF & Security Guards**: `validateUrlAsync` and `getSecureAxios` intact across Firecrawl and Webhook dispatchers.
---
## 5. Deployment & Production Readiness Checklist
## Deferred Items for Operational Rollout (Non-blocking)
1. **Environment Configuration**: Ensure production `.env` contains:
- `JWT_ACCESS_SECRET` (>= 32 chars)
- `JWT_REFRESH_SECRET` (>= 32 chars)
- `WEBHOOK_ENCRYPTION_KEY` (64 hex chars)
- `CORS_ALLOWED_ORIGINS` (comma-separated list of allowed frontend domains)
2. **Database Migration**: Run `pnpm db:migrate:deploy` to apply new index definitions from `schema.prisma`.
3. **Zero Regressions**: All data contracts, timezones (`Asia/Ho_Chi_Minh`), and queue processing invariants verified intact.
1. **Redis Cache for Auth Token Deactivation (`BUG-08`)**:
Currently, `authMiddleware` validates user active status directly via PostgreSQL lookup on authenticated requests to guarantee instant deactivation. In high-traffic multi-instance environments, integrating short-lived Redis key caching (`TTL = 60s`) with an invalidation hook on `UserService.update({ isActive: false })` is recommended.
2. **Cluster-wide Redis Rate Limiter Store (`BUG-13`)**:
`express-rate-limit` currently uses the default in-memory store. When horizontally scaling beyond a single Node instance, configure `rate-limit-redis` using the existing Redis client connection.
eslint.config.js
View file @ b454ffff
......@@ -25,6 +25,29 @@ export default [
],
"no-console": "off",
"@typescript-eslint/no-explicit-any": "warn",
"no-restricted-imports": [
"error",
{
paths: [
{
name: "@prisma/client",
message:
"Importing directly from @prisma/client is forbidden outside repositories and database.types.ts (Rule: AGENTS.md). Use src/common/constants or src/common/types/database.types.",
},
],
},
],
},
},
{
files: [
"src/**/*.repository.ts",
"src/database/prisma.client.ts",
"src/common/types/database.types.ts",
"src/**/__tests__/**/*.ts",
],
rules: {
"no-restricted-imports": "off",
},
},
];
jest.setup.ts
View file @ b454ffff
......@@ -7,3 +7,4 @@ process.env.JWT_REFRESH_SECRET =
process.env.WEBHOOK_ENCRYPTION_KEY =
process.env.WEBHOOK_ENCRYPTION_KEY ||
"abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890";
process.env.REDIS_ENABLED = "false";
src/app.ts
View file @ b454ffff
......@@ -40,7 +40,7 @@ app.use(
maxAge: 86400,
}),
);
app.use(morgan("dev"));
app.use(morgan(envConfig.nodeEnv === "production" ? "combined" : "dev"));
app.use(cookieParser());
app.use(express.json());
app.use(express.urlencoded({ extended: true }));
......
src/common/constants/crawl-page-status.constant.ts 0 → 100644
View file @ b454ffff
export const CRAWL_PAGE_STATUS = {
PENDING: "PENDING",
SUCCESS: "SUCCESS",
FAILED: "FAILED",
BLOCKED: "BLOCKED",
REQUIRES_LOGIN: "REQUIRES_LOGIN",
CAPTCHA_DETECTED: "CAPTCHA_DETECTED",
PAYWALL_DETECTED: "PAYWALL_DETECTED",
TIMEOUT: "TIMEOUT",
SKIPPED: "SKIPPED",
} as const;
export type CrawlPageStatus = keyof typeof CRAWL_PAGE_STATUS;
src/common/constants/export-type.constant.ts
View file @ b454ffff
......@@ -16,3 +16,11 @@ export const EXPORT_MIME_TYPES: Record<ExportType, string> = {
MARKDOWN: "application/zip",
ZIP: "application/zip",
};
export const EXPORT_STATUS = {
PENDING: "PENDING",
PROCESSING: "PROCESSING",
COMPLETED: "COMPLETED",
FAILED: "FAILED",
} as const;
export type ExportStatus = keyof typeof EXPORT_STATUS;
src/common/constants/index.ts
View file @ b454ffff
......@@ -8,3 +8,5 @@ export * from "./timezone.constant";
export * from "./crawl-mode.constant";
export * from "./schedule-frequency.constant";
export * from "./asset-type.constant";
export * from "./crawl-page-status.constant";
export * from "./webhook.constant";
src/common/constants/role.constant.ts
View file @ b454ffff
......@@ -5,3 +5,4 @@ export const ROLES = {
} as const;
export type Role = keyof typeof ROLES;
export type UserRole = Role;
src/common/constants/webhook.constant.ts 0 → 100644
View file @ b454ffff
export const WEBHOOK_DELIVERY_STATUS = {
PENDING: "PENDING",
SUCCESS: "SUCCESS",
FAILED: "FAILED",
} as const;
export type WebhookDeliveryStatus = keyof typeof WEBHOOK_DELIVERY_STATUS;
export const WEBHOOK_EVENT = {
JOB_COMPLETED: "job.completed",
JOB_FAILED: "job.failed",
} as const;
export type WebhookEvent = (typeof WEBHOOK_EVENT)[keyof typeof WEBHOOK_EVENT];
src/common/helpers/data-contract.helper.ts
View file @ b454ffff
import crypto from "crypto";
import { CrawlPage, CrawlAsset } from "@prisma/client";
import { CrawlPage, CrawlAsset } from "../types/database.types";
import {
CrawlPageRecord,
DataQualityWarning,
......@@ -15,10 +15,6 @@ import {
DATA_CONTRACT_HASH_ALGORITHM,
} from "../constants/data-contract.constant";
// ─────────────────────────────────────────────
// URL Normalization
// ─────────────────────────────────────────────
/**
* Normalize một URL để phục vụ deduplicate và so sánh.
* Các bước:
......@@ -66,10 +62,6 @@ export function normalizeUrl(rawUrl: string): string {
}
}
// ─────────────────────────────────────────────
// Text Utilities
// ─────────────────────────────────────────────
/**
* Strip toàn bộ Markdown syntax, trả về plain text thuần.
* Phạm vi: heading, bold/italic, inline code, code block, link, image, blockquote, HR.
......@@ -234,10 +226,6 @@ export function hashContent(text: string): string | null {
.digest("hex");
}
// ─────────────────────────────────────────────
// Data Quality
// ─────────────────────────────────────────────
/**
* Tính điểm chất lượng dữ liệu (0–100) của một page dựa trên các tiêu chí:
* - Có mainContent : +40 điểm
......@@ -295,10 +283,6 @@ export function detectWarnings(params: {
return warnings;
}
// ─────────────────────────────────────────────
// Asset Transformation
// ─────────────────────────────────────────────
/**
* Chuyển đổi danh sách CrawlAsset sang LinkRecord[].
* Phân loại internal/external dựa vào domain của startUrl (job's domain).
......@@ -361,10 +345,6 @@ function inferImageType(imageUrl: string): string {
}
}
// ─────────────────────────────────────────────
// Main Transformer
// ─────────────────────────────────────────────
export interface TransformPageOptions {
page: CrawlPage & { normalizedUrl?: string | null };
assets: CrawlAsset[];
......@@ -452,10 +432,6 @@ export function transformPageToRecord(
};
}
// ─────────────────────────────────────────────
// Envelope Builder
// ─────────────────────────────────────────────
/**
* Đóng gói danh sách CrawlPageRecord vào PagesJsonEnvelope để ghi ra pages.json.
*/
......
src/common/types/database.types.ts 0 → 100644
View file @ b454ffff
export type {
User,
CrawlJob,
CrawlPage,
CrawlAsset,
CrawlExport,
CrawlJobLog,
ApiKey,
WebhookConfig,
WebhookDelivery,
AuditLog,
RefreshToken,
Prisma,
} from "@prisma/client";
src/common/types/express.d.ts
View file @ b454ffff
import { UserRole } from "@prisma/client";
import { UserRole } from "../constants/role.constant";
declare global {
namespace Express {
......
src/docs/swagger.json
View file @ b454ffff
......@@ -83,7 +83,9 @@
"description": "Email hoặc mật khẩu không chính xác"
}
},
"tags": ["Auth"],
"tags": [
"Auth"
],
"summary": "Đăng nhập người dùng",
"requestBody": {
"required": true,
......@@ -139,7 +141,9 @@
}
}
},
"tags": ["Auth"],
"tags": [
"Auth"
],
"summary": "Làm mới Access Token"
}
},
......@@ -178,7 +182,9 @@
}
}
},
"tags": ["Auth"],
"tags": [
"Auth"
],
"summary": "Đăng xuất"
}
},
......@@ -209,7 +215,9 @@
"description": "Chưa xác thực hoặc token không hợp lệ"
}
},
"tags": ["Auth"],
"tags": [
"Auth"
],
"summary": "Lấy thông tin người dùng hiện tại"
},
"put": {
......@@ -241,7 +249,9 @@
"description": "Chưa xác thực"
}
},
"tags": ["Auth"],
"tags": [
"Auth"
],
"summary": "Cập nhật thông tin cá nhân",
"requestBody": {
"required": true,
......@@ -330,7 +340,9 @@
"description": "Chưa xác thực"
}
},
"tags": ["Auth"],
"tags": [
"Auth"
],
"summary": "Xem hạn mức và mức độ sử dụng Quota hiện tại"
}
},
......@@ -365,7 +377,9 @@
"description": "Chưa xác thực"
}
},
"tags": ["Auth"],
"tags": [
"Auth"
],
"summary": "Đổi mật khẩu tài khoản",
"requestBody": {
"required": true,
......@@ -406,7 +420,9 @@
"description": "Email đã được sử dụng hoặc dữ liệu không hợp lệ"
}
},
"tags": ["Auth"],
"tags": [
"Auth"
],
"summary": "Đăng ký tài khoản mới",
"requestBody": {
"required": true,
......@@ -448,7 +464,9 @@
"description": "Email không hợp lệ"
}
},
"tags": ["Auth"],
"tags": [
"Auth"
],
"summary": "Yêu cầu đặt lại mật khẩu",
"requestBody": {
"required": true,
......@@ -456,7 +474,9 @@
"application/json": {
"schema": {
"type": "object",
"required": ["email"],
"required": [
"email"
],
"properties": {
"email": {
"type": "string",
......@@ -498,7 +518,9 @@
"description": "Token không hợp lệ, đã hết hạn hoặc mật khẩu không đúng định dạng"
}
},
"tags": ["Auth"],
"tags": [
"Auth"
],
"summary": "Đặt lại mật khẩu mới",
"requestBody": {
"required": true,
......@@ -506,7 +528,10 @@
"application/json": {
"schema": {
"type": "object",
"required": ["token", "password"],
"required": [
"token",
"password"
],
"properties": {
"token": {
"type": "string",
......@@ -551,7 +576,9 @@
"description": "Địa chỉ email không hợp lệ"
}
},
"tags": ["Auth"],
"tags": [
"Auth"
],
"summary": "Gửi lại email xác thực",
"requestBody": {
"required": true,
......@@ -593,7 +620,9 @@
"description": "Token không hợp lệ hoặc đã hết hạn"
}
},
"tags": ["Auth"],
"tags": [
"Auth"
],
"summary": "Xác thực địa chỉ email",
"requestBody": {
"required": true,
......@@ -601,7 +630,9 @@
"application/json": {
"schema": {
"type": "object",
"required": ["token"],
"required": [
"token"
],
"properties": {
"token": {
"type": "string",
......@@ -697,7 +728,9 @@
"description": "Không có quyền truy cập"
}
},
"tags": ["Users"],
"tags": [
"Users"
],
"summary": "Lấy danh sách người dùng"
},
"post": {
......@@ -742,7 +775,9 @@
}
}
},
"tags": ["Users"],
"tags": [
"Users"
],
"summary": "Tạo người dùng mới"
}
},
......@@ -790,7 +825,9 @@
"description": "Không tìm thấy người dùng"
}
},
"tags": ["Users"],
"tags": [
"Users"
],
"summary": "Lấy thông tin người dùng theo ID"
},
"put": {
......@@ -843,7 +880,9 @@
}
}
},
"tags": ["Users"],
"tags": [
"Users"
],
"summary": "Cập nhật thông tin người dùng"
},
"delete": {
......@@ -884,7 +923,9 @@
"description": "Không tìm thấy người dùng"
}
},
"tags": ["Users"],
"tags": [
"Users"
],
"summary": "Xóa người dùng"
}
},
......@@ -937,7 +978,9 @@
"description": "Chưa xác thực"
}
},
"tags": ["Crawl Jobs"],
"tags": [
"Crawl Jobs"
],
"summary": "Tạo crawl job mới",
"requestBody": {
"required": true,
......@@ -1026,7 +1069,9 @@
}
}
},
"tags": ["Crawl Jobs"],
"tags": [
"Crawl Jobs"
],
"summary": "Lấy danh sách các crawl jobs"
}
},
......@@ -1068,7 +1113,9 @@
"description": "Không tìm thấy crawl job hoặc không có quyền truy cập"
}
},
"tags": ["Crawl Jobs"],
"tags": [
"Crawl Jobs"
],
"summary": "Lấy thông tin chi tiết một crawl job"
},
"delete": {
......@@ -1219,7 +1266,9 @@
"description": "Không tìm thấy crawl job"
}
},
"tags": ["Crawl Jobs"],
"tags": [
"Crawl Jobs"
],
"summary": "Hủy một crawl job đang chạy"
}
},
......@@ -1423,7 +1472,10 @@
"in": "query",
"schema": {
"type": "string",
"enum": ["asc", "desc"],
"enum": [
"asc",
"desc"
],
"default": "asc"
},
"description": "Thứ tự sắp xếp"
......@@ -1484,7 +1536,9 @@
}
}
},
"tags": ["Crawl Jobs"],
"tags": [
"Crawl Jobs"
],
"summary": "Lấy danh sách các trang đã crawl của job"
}
},
......@@ -1688,7 +1742,10 @@
"in": "query",
"schema": {
"type": "string",
"enum": ["asc", "desc"],
"enum": [
"asc",
"desc"
],
"default": "asc"
},
"description": "Thứ tự sắp xếp"
......@@ -1823,7 +1880,9 @@
"description": "Không tìm thấy crawl job hoặc không có quyền truy cập"
}
},
"tags": ["Crawl Jobs"],
"tags": [
"Crawl Jobs"
],
"summary": "Xem preview dữ liệu clean/raw của các trang đã crawl"
}
},
......@@ -1865,7 +1924,9 @@
}
}
},
"tags": ["Crawl Jobs"],
"tags": [
"Crawl Jobs"
],
"summary": "Lấy danh sách các bản export của job"
},
"post": {
......@@ -1902,7 +1963,9 @@
}
}
},
"tags": ["Crawl Jobs"],
"tags": [
"Crawl Jobs"
],
"summary": "Yêu cầu xuất dữ liệu cho job",
"requestBody": {
"required": true,
......@@ -1941,7 +2004,9 @@
}
}
},
"tags": ["Crawl Jobs"],
"tags": [
"Crawl Jobs"
],
"summary": "Tải xuống file export mới nhất"
}
},
......@@ -1982,7 +2047,14 @@
"required": false,
"schema": {
"type": "string",
"enum": ["IMAGE", "LINK", "PDF", "FILE", "VIDEO", "OTHER"]
"enum": [
"IMAGE",
"LINK",
"PDF",
"FILE",
"VIDEO",
"OTHER"
]
},
"description": "Lọc theo loại asset"
}
......@@ -2074,7 +2146,9 @@
"description": "Không tìm thấy crawl job hoặc không có quyền truy cập"
}
},
"tags": ["Crawl Jobs"],
"tags": [
"Crawl Jobs"
],
"summary": "Lấy danh sách assets của job (có phân trang)"
}
},
......@@ -2127,7 +2201,9 @@
"description": "Không tìm thấy crawl job"
}
},
"tags": ["Crawl Jobs"],
"tags": [
"Crawl Jobs"
],
"summary": "Xem báo cáo thay đổi (Diff Report)"
}
},
......@@ -2172,7 +2248,9 @@
"description": "Không tìm thấy crawl job hoặc diff report"
}
},
"tags": ["Crawl Jobs"],
"tags": [
"Crawl Jobs"
],
"summary": "Tải file diff_report.json"
}
},
......@@ -2214,7 +2292,9 @@
"description": "Chưa xác thực"
}
},
"tags": ["Crawl Schedules"],
"tags": [
"Crawl Schedules"
],
"summary": "Tạo lịch crawl định kỳ mới",
"requestBody": {
"required": true,
......@@ -2243,7 +2323,12 @@
"in": "query",
"schema": {
"type": "string",
"enum": ["DAILY", "WEEKLY", "MONTHLY", "CUSTOM"]
"enum": [
"DAILY",
"WEEKLY",
"MONTHLY",
"CUSTOM"
]
},
"description": "Lọc theo tần suất"
},
......@@ -2312,7 +2397,9 @@
"description": "Chưa xác thực"
}
},
"tags": ["Crawl Schedules"],
"tags": [
"Crawl Schedules"
],
"summary": "Lấy danh sách lịch crawl định kỳ"
}
},
......@@ -2348,7 +2435,9 @@
"description": "Không tìm thấy lịch crawl"
}
},
"tags": ["Crawl Schedules"],
"tags": [
"Crawl Schedules"
],
"summary": "Xem chi tiết lịch crawl"
},
"patch": {
......@@ -2393,7 +2482,9 @@
"description": "Không tìm thấy lịch crawl"
}
},
"tags": ["Crawl Schedules"],
"tags": [
"Crawl Schedules"
],
"summary": "Cập nhật lịch crawl",
"requestBody": {
"required": true,
......@@ -2443,7 +2534,9 @@
"description": "Không tìm thấy lịch crawl"
}
},
"tags": ["Crawl Schedules"],
"tags": [
"Crawl Schedules"
],
"summary": "Xóa lịch crawl"
}
},
......@@ -2488,7 +2581,9 @@
"description": "Không tìm thấy lịch crawl"
}
},
"tags": ["Crawl Schedules"],
"tags": [
"Crawl Schedules"
],
"summary": "Kích hoạt chạy ngay lịch crawl"
}
},
......@@ -2565,7 +2660,9 @@
"description": "Không tìm thấy lịch crawl"
}
},
"tags": ["Crawl Schedules"],
"tags": [
"Crawl Schedules"
],
"summary": "Xem lịch sử các lần chạy của lịch crawl"
}
},
......@@ -2607,7 +2704,9 @@
"description": "Không tìm thấy file export"
}
},
"tags": ["Crawl Exports"],
"tags": [
"Crawl Exports"
],
"summary": "Tải xuống tệp export theo ID"
}
},
......@@ -2745,7 +2844,9 @@
"description": "Không có quyền truy cập (không phải ADMIN)"
}
},
"tags": ["Audit Logs"],
"tags": [
"Audit Logs"
],
"summary": "Lấy danh sách nhật ký hệ thống"
}
},
......@@ -2779,7 +2880,9 @@
"description": "Dữ liệu không hợp lệ hoặc thời điểm hết hạn không ở trong tương lai"
}
},
"tags": ["API Keys"],
"tags": [
"API Keys"
],
"security": [
{
"BearerAuth": []
......@@ -2826,7 +2929,9 @@
"description": "Chưa xác thực"
}
},
"tags": ["API Keys"],
"tags": [
"API Keys"
],
"security": [
{
"BearerAuth": []
......@@ -2880,7 +2985,9 @@
"description": "Trạng thái không hợp lệ"
}
},
"tags": ["API Keys"],
"tags": [
"API Keys"
],
"security": [
{
"BearerAuth": []
......@@ -2939,7 +3046,9 @@
"description": "Không tìm thấy API Key"
}
},
"tags": ["API Keys"],
"tags": [
"API Keys"
],
"security": [
{
"BearerAuth": []
......@@ -2978,7 +3087,9 @@
"description": "Chưa xác thực"
}
},
"tags": ["Webhooks"],
"tags": [
"Webhooks"
],
"summary": "Tạo cấu hình Webhook",
"requestBody": {
"required": true,
......@@ -3020,7 +3131,9 @@
"description": "Chưa xác thực"
}
},
"tags": ["Webhooks"],
"tags": [
"Webhooks"
],
"summary": "Xem danh sách Webhook configs"
}
},
......@@ -3083,7 +3196,9 @@
"description": "Không tìm thấy cấu hình Webhook"
}
},
"tags": ["Webhooks"],
"tags": [
"Webhooks"
],
"summary": "Xóa cấu hình Webhook"
}
},
......@@ -3155,7 +3270,9 @@
"description": "Chưa xác thực"
}
},
"tags": ["Webhooks"],
"tags": [
"Webhooks"
],
"summary": "Xem lịch sử gửi Webhook"
}
},
......@@ -3286,7 +3403,11 @@
},
"role": {
"type": "string",
"enum": ["ADMIN", "CRAWLER_USER", "VIEWER"]
"enum": [
"ADMIN",
"CRAWLER_USER",
"VIEWER"
]
},
"isActive": {
"type": "boolean"
......@@ -3315,7 +3436,12 @@
},
"mode": {
"type": "string",
"enum": ["SCRAPE", "CRAWL", "SITEMAP", "URL_LIST"]
"enum": [
"SCRAPE",
"CRAWL",
"SITEMAP",
"URL_LIST"
]
},
"status": {
"type": "string",
......@@ -3526,7 +3652,10 @@
},
"type": {
"type": "string",
"enum": ["internal", "external"]
"enum": [
"internal",
"external"
]
}
}
}
......@@ -3612,11 +3741,21 @@
},
"exportType": {
"type": "string",
"enum": ["JSON", "CSV", "XLSX", "MARKDOWN", "ZIP"]
"enum": [
"JSON",
"CSV",
"XLSX",
"MARKDOWN",
"ZIP"
]
},
"status": {
"type": "string",
"enum": ["PENDING", "COMPLETED", "FAILED"]
"enum": [
"PENDING",
"COMPLETED",
"FAILED"
]
},
"fileName": {
"type": "string"
......@@ -3639,7 +3778,10 @@
},
"LoginRequest": {
"type": "object",
"required": ["email", "password"],
"required": [
"email",
"password"
],
"properties": {
"email": {
"type": "string",
......@@ -3654,7 +3796,10 @@
},
"RegisterRequest": {
"type": "object",
"required": ["email", "password"],
"required": [
"email",
"password"
],
"properties": {
"email": {
"type": "string",
......@@ -3673,7 +3818,9 @@
},
"ResendVerificationRequest": {
"type": "object",
"required": ["email"],
"required": [
"email"
],
"properties": {
"email": {
"type": "string",
......@@ -3684,7 +3831,9 @@
},
"RefreshRequest": {
"type": "object",
"required": ["refreshToken"],
"required": [
"refreshToken"
],
"properties": {
"refreshToken": {
"type": "string",
......@@ -3694,7 +3843,9 @@
},
"LogoutRequest": {
"type": "object",
"required": ["refreshToken"],
"required": [
"refreshToken"
],
"properties": {
"refreshToken": {
"type": "string",
......@@ -3713,7 +3864,11 @@
},
"ChangePasswordRequest": {
"type": "object",
"required": ["currentPassword", "newPassword", "confirmPassword"],
"required": [
"currentPassword",
"newPassword",
"confirmPassword"
],
"properties": {
"currentPassword": {
"type": "string",
......@@ -3731,7 +3886,10 @@
},
"CreateUserRequest": {
"type": "object",
"required": ["email", "password"],
"required": [
"email",
"password"
],
"properties": {
"email": {
"type": "string",
......@@ -3748,7 +3906,11 @@
},
"role": {
"type": "string",
"enum": ["ADMIN", "CRAWLER_USER", "VIEWER"],
"enum": [
"ADMIN",
"CRAWLER_USER",
"VIEWER"
],
"example": "CRAWLER_USER"
},
"maxPagesLimit": {
......@@ -3774,7 +3936,11 @@
},
"role": {
"type": "string",
"enum": ["ADMIN", "CRAWLER_USER", "VIEWER"],
"enum": [
"ADMIN",
"CRAWLER_USER",
"VIEWER"
],
"example": "VIEWER"
},
"isActive": {
......@@ -3805,7 +3971,12 @@
},
"mode": {
"type": "string",
"enum": ["SCRAPE", "CRAWL", "SITEMAP", "URL_LIST"],
"enum": [
"SCRAPE",
"CRAWL",
"SITEMAP",
"URL_LIST"
],
"example": "CRAWL"
},
"maxPages": {
......@@ -3826,25 +3997,38 @@
"type": "string",
"format": "uri"
},
"example": ["https://example.com/1", "https://example.com/2"],
"example": [
"https://example.com/1",
"https://example.com/2"
],
"description": "Bắt buộc khi mode là URL_LIST"
}
}
},
"CreateExportRequest": {
"type": "object",
"required": ["exportType"],
"required": [
"exportType"
],
"properties": {
"exportType": {
"type": "string",
"enum": ["JSON", "CSV", "XLSX", "MARKDOWN", "ZIP"],
"enum": [
"JSON",
"CSV",
"XLSX",
"MARKDOWN",
"ZIP"
],
"example": "JSON"
}
}
},
"CreateApiKeyRequest": {
"type": "object",
"required": ["name"],
"required": [
"name"
],
"properties": {
"name": {
"type": "string",
......@@ -3863,7 +4047,9 @@
},
"UpdateApiKeyStatusRequest": {
"type": "object",
"required": ["isActive"],
"required": [
"isActive"
],
"additionalProperties": false,
"properties": {
"isActive": {
......@@ -3960,7 +4146,11 @@
},
"CreateWebhookConfigRequest": {
"type": "object",
"required": ["url", "secret", "events"],
"required": [
"url",
"secret",
"events"
],
"properties": {
"url": {
"type": "string",
......@@ -3975,9 +4165,15 @@
"type": "array",
"items": {
"type": "string",
"enum": ["job.completed", "job.failed"]
"enum": [
"job.completed",
"job.failed"
]
},
"example": ["job.completed", "job.failed"]
"example": [
"job.completed",
"job.failed"
]
}
}
},
......@@ -4038,7 +4234,11 @@
},
"status": {
"type": "string",
"enum": ["PENDING", "SUCCESS", "FAILED"]
"enum": [
"PENDING",
"SUCCESS",
"FAILED"
]
},
"statusCode": {
"type": "integer",
......@@ -4102,11 +4302,21 @@
},
"mode": {
"type": "string",
"enum": ["SCRAPE", "CRAWL", "SITEMAP", "URL_LIST"]
"enum": [
"SCRAPE",
"CRAWL",
"SITEMAP",
"URL_LIST"
]
},
"frequency": {
"type": "string",
"enum": ["DAILY", "WEEKLY", "MONTHLY", "CUSTOM"]
"enum": [
"DAILY",
"WEEKLY",
"MONTHLY",
"CUSTOM"
]
},
"cronExpression": {
"type": "string",
......@@ -4169,7 +4379,10 @@
},
"CreateCrawlScheduleRequest": {
"type": "object",
"required": ["name", "startUrl"],
"required": [
"name",
"startUrl"
],
"properties": {
"name": {
"type": "string",
......@@ -4182,12 +4395,22 @@
},
"mode": {
"type": "string",
"enum": ["SCRAPE", "CRAWL", "SITEMAP", "URL_LIST"],
"enum": [
"SCRAPE",
"CRAWL",
"SITEMAP",
"URL_LIST"
],
"example": "CRAWL"
},
"frequency": {
"type": "string",
"enum": ["DAILY", "WEEKLY", "MONTHLY", "CUSTOM"],
"enum": [
"DAILY",
"WEEKLY",
"MONTHLY",
"CUSTOM"
],
"example": "DAILY"
},
"cronExpression": {
......@@ -4250,11 +4473,21 @@
},
"mode": {
"type": "string",
"enum": ["SCRAPE", "CRAWL", "SITEMAP", "URL_LIST"]
"enum": [
"SCRAPE",
"CRAWL",
"SITEMAP",
"URL_LIST"
]
},
"frequency": {
"type": "string",
"enum": ["DAILY", "WEEKLY", "MONTHLY", "CUSTOM"]
"enum": [
"DAILY",
"WEEKLY",
"MONTHLY",
"CUSTOM"
]
},
"cronExpression": {
"type": "string"
......@@ -4393,4 +4626,4 @@
"ApiKeyAuth": []
}
]
}
}
\ No newline at end of file
src/modules/api-keys/api-key.dto.ts
View file @ b454ffff
import type { ApiKey } from "@prisma/client";
import type { ApiKey } from "../../common/types/database.types";
export type PublicApiKey = Omit<ApiKey, "keyHash">;
......
src/modules/api-keys/api-key.service.ts
View file @ b454ffff
......@@ -2,7 +2,7 @@ import crypto from "crypto";
import { ApiKeyRepository } from "./api-key.repository";
import { AppError } from "../../common/errors/app-error";
import { ERROR_CODE } from "../../common/errors/error-code";
import type { ApiKey } from "@prisma/client";
import type { ApiKey } from "../../common/types/database.types";
import {
toPublicApiKey,
type CreatedApiKey,
......
src/modules/auth/__tests__/auth.service.test.ts
View file @ b454ffff
......@@ -170,4 +170,89 @@ describe("AuthService registration mail failures", () => {
expect(mailService.sendVerificationEmail).toHaveBeenCalledTimes(1);
expect(repository.createUser).not.toHaveBeenCalled();
});
describe("AuthService forgotPassword security", () => {
const activeUser = {
id: "user-active",
email: "active@example.com",
fullName: "Active User",
role: "CRAWLER_USER",
isActive: true,
passwordHash: "hash",
createdAt: new Date(),
};
it("sends password reset email internally and returns only { success: true } without leaking token or userId", async () => {
const service = new AuthService();
const repository = {
findByEmail: jest.fn().mockResolvedValue(activeUser),
};
const mailService = {
sendPasswordResetEmail: jest.fn().mockResolvedValue(undefined),
};
const mutableService = service as unknown as {
repository: typeof repository;
mailService: typeof mailService;
};
mutableService.repository = repository;
mutableService.mailService = mailService;
const result = await service.forgotPassword({ email: activeUser.email });
expect(result).toEqual({ success: true });
expect((result as Record<string, unknown>).resetToken).toBeUndefined();
expect((result as Record<string, unknown>).userId).toBeUndefined();
expect(mailService.sendPasswordResetEmail).toHaveBeenCalledTimes(1);
expect(mailService.sendPasswordResetEmail).toHaveBeenCalledWith(
activeUser.email,
expect.any(String),
);
});
it("returns { success: true } and does not call mail service if user is not found", async () => {
const service = new AuthService();
const repository = {
findByEmail: jest.fn().mockResolvedValue(null),
};
const mailService = {
sendPasswordResetEmail: jest.fn(),
};
const mutableService = service as unknown as {
repository: typeof repository;
mailService: typeof mailService;
};
mutableService.repository = repository;
mutableService.mailService = mailService;
const result = await service.forgotPassword({
email: "nonexistent@example.com",
});
expect(result).toEqual({ success: true });
expect(mailService.sendPasswordResetEmail).not.toHaveBeenCalled();
});
it("returns { success: true } and does not call mail service if user is inactive", async () => {
const service = new AuthService();
const repository = {
findByEmail: jest
.fn()
.mockResolvedValue({ ...activeUser, isActive: false }),
};
const mailService = {
sendPasswordResetEmail: jest.fn(),
};
const mutableService = service as unknown as {
repository: typeof repository;
mailService: typeof mailService;
};
mutableService.repository = repository;
mutableService.mailService = mailService;
const result = await service.forgotPassword({ email: activeUser.email });
expect(result).toEqual({ success: true });
expect(mailService.sendPasswordResetEmail).not.toHaveBeenCalled();
});
});
});
src/modules/auth/auth.controller.ts
View file @ b454ffff
......@@ -283,22 +283,14 @@ export class AuthController {
forgotPassword = async (req: Request, res: Response, next: NextFunction) => {
try {
const forgotPasswordDto: ForgotPasswordDto = req.body;
const result = await this.service.forgotPassword(forgotPasswordDto);
await this.service.forgotPassword(forgotPasswordDto);
if (result.userId && result.resetToken) {
await this.auditLogService.log({
userId: result.userId,
action: AUDIT_ACTIONS.FORGOT_PASSWORD,
ipAddress: req.ip,
userAgent: req.headers["user-agent"] as string,
details: { email: forgotPasswordDto.email },
});
await this.mailService.sendPasswordResetEmail(
forgotPasswordDto.email,
result.resetToken,
);
}
await this.auditLogService.log({
action: AUDIT_ACTIONS.FORGOT_PASSWORD,
ipAddress: req.ip,
userAgent: req.headers["user-agent"] as string,
details: { email: forgotPasswordDto.email },
});
res.json({
success: true,
......
src/modules/auth/auth.service.ts
View file @ b454ffff
......@@ -502,9 +502,7 @@ export class AuthService {
};
}
async forgotPassword(
data: ForgotPasswordDto,
): Promise<{ success: boolean; resetToken?: string; userId?: string }> {
async forgotPassword(data: ForgotPasswordDto): Promise<{ success: boolean }> {
const { email } = data;
const user = await this.repository.findByEmail(email);
......@@ -517,10 +515,14 @@ export class AuthService {
expiresIn: "15m",
});
try {
await this.mailService.sendPasswordResetEmail(user.email, resetToken);
} catch (error: unknown) {
console.error("[Mail] Password reset delivery failed:", error);
}
return {
success: true,
resetToken,
userId: user.id,
};
}
......
src/modules/change-detection/change-detection.service.ts
View file @ b454ffff
import fs from "fs";
import { CrawlJob, Prisma } from "@prisma/client";
import { CrawlJob, Prisma } from "../../common/types/database.types";
import { CrawlJobRepository } from "../crawl-jobs/crawl-job.repository";
import {
DiffReportEnvelope,
......
src/modules/crawl-assets/crawl-asset.repository.ts
View file @ b454ffff
......@@ -57,6 +57,15 @@ export class CrawlAssetRepository {
});
}
countByJobId(jobId: string, assetType?: AssetType) {
return prisma.crawlAsset.count({
where: {
crawlJobId: jobId,
...(assetType ? { assetType } : {}),
},
});
}
findAssetsForJsonExport(jobId: string) {
return prisma.crawlAsset.findMany({
where: { crawlJobId: jobId },
......
src/modules/crawl-assets/crawl-asset.service.ts
View file @ b454ffff
......@@ -10,7 +10,19 @@ export class CrawlAssetService {
page = 1,
limit = 50,
) {
return this.repository.findByJobId(jobId, assetType, page, limit);
const safeLimit = Math.min(Math.max(1, limit), 500);
const safePage = Math.max(1, page);
const [items, total] = await Promise.all([
this.repository.findByJobId(jobId, assetType, safePage, safeLimit),
this.repository.countByJobId(jobId, assetType),
]);
return {
items,
total,
page: safePage,
limit: safeLimit,
totalPages: Math.ceil(total / safeLimit),
};
}
async create(data: {
......
src/modules/crawl-exports/crawl-export.dto.ts
View file @ b454ffff
import { ExportType, ExportStatus } from "@prisma/client";
import {
ExportType,
ExportStatus,
} from "../../common/constants/export-type.constant";
export interface CreateCrawlExportDto {
jobId: string;
......
src/modules/crawl-jobs/__tests__/crawl-job.service.test.ts
View file @ b454ffff
......@@ -5,6 +5,7 @@ jest.mock("../../../database/prisma.client", () => ({
jest.mock("../crawl-job.repository");
jest.mock("../../users/user.repository");
jest.mock("../../crawl-exports/crawl-export.repository");
jest.mock("../../crawl-schedules/crawl-schedule.repository");
jest.mock("../../../common/helpers/url.helper");
jest.mock("../../../queues/crawl.queue", () => ({
crawlQueue: {
......@@ -15,12 +16,14 @@ jest.mock("../../../queues/crawl.queue", () => ({
import { CrawlJobService } from "../crawl-job.service";
import { CrawlJobRepository } from "../crawl-job.repository";
import { UserRepository } from "../../users/user.repository";
import { CrawlScheduleRepository } from "../../crawl-schedules/crawl-schedule.repository";
import * as urlHelper from "../../../common/helpers/url.helper";
describe("CrawlJobService", () => {
let service: CrawlJobService;
let mockJobRepo: jest.Mocked<CrawlJobRepository>;
let mockUserRepo: jest.Mocked<UserRepository>;
let mockScheduleRepo: jest.Mocked<CrawlScheduleRepository>;
beforeEach(() => {
jest.clearAllMocks();
......@@ -32,6 +35,13 @@ describe("CrawlJobService", () => {
findById: jest.fn(),
} as any;
mockScheduleRepo = {
findById: jest.fn().mockResolvedValue(null),
} as any;
(CrawlJobRepository as jest.Mock).mockReturnValue(mockJobRepo);
(CrawlScheduleRepository as jest.Mock).mockReturnValue(mockScheduleRepo);
mockUserRepo = {
findById: jest.fn().mockResolvedValue({
id: "user-1",
......@@ -109,24 +119,34 @@ describe("CrawlJobService", () => {
).rejects.toThrow("Concurrent jobs quota of 3 exceeded");
});
it("deduplicates URLs in URL_LIST mode and validates them", async () => {
const urls = [
"https://example.com/1",
"https://example.com/2",
"https://example.com/1",
];
it("throws error when scheduleId does not belong to user", async () => {
mockScheduleRepo.findById.mockResolvedValue({
id: "sched-1",
userId: "other-user",
} as any);
await expect(
service.create("user-1", {
startUrl: "https://example.com",
scheduleId: "sched-1",
}),
).rejects.toThrow("Crawl schedule not found");
});
it("attaches scheduleId when schedule belongs to user", async () => {
mockScheduleRepo.findById.mockResolvedValue({
id: "sched-1",
userId: "user-1",
} as any);
await service.create("user-1", {
mode: "URL_LIST",
urls,
startUrl: "https://example.com",
scheduleId: "sched-1",
});
expect(urlHelper.validateUrlAsync).toHaveBeenCalledTimes(2);
expect(mockJobRepo.create).toHaveBeenCalledWith(
expect.objectContaining({
mode: "URL_LIST",
startUrl: "https://example.com/1",
urls: ["https://example.com/1", "https://example.com/2"],
scheduleId: "sched-1",
}),
);
});
......
src/modules/crawl-jobs/crawl-job.controller.ts
View file @ b454ffff
......@@ -8,10 +8,7 @@ import { AuditLogService } from "../audit-logs/audit-log.service";
import { AUDIT_ACTIONS } from "../../common/constants/audit-action.constant";
import { JOB_STATUS } from "../../common/constants/job-status.constant";
import { CrawlAssetService } from "../crawl-assets/crawl-asset.service";
import {
AssetType,
ASSET_TYPES,
} from "../../common/constants/asset-type.constant";
import { AssetType } from "../../common/constants/asset-type.constant";
import { streamStorageDownload } from "../../common/storage/storage-download.helper";
export class CrawlJobController {
private readonly service = new CrawlJobService();
......@@ -137,22 +134,11 @@ export class CrawlJobController {
try {
await this.service.findById(req.user.id, req.user.role, req.params.id);
const VALID_ASSET_TYPES: readonly string[] = Object.values(ASSET_TYPES);
const rawType = req.query.assetType as string | undefined;
if (rawType && !VALID_ASSET_TYPES.includes(rawType)) {
res
.status(400)
.json({ success: false, message: `Invalid assetType: ${rawType}` });
return;
}
const assetType = rawType as AssetType | undefined;
const page = Math.max(1, parseInt((req.query.page as string) || "1", 10));
const limit = Math.min(
Math.max(1, parseInt((req.query.limit as string) || "50", 10)),
500,
);
const assetType = req.query.assetType as AssetType | undefined;
const page = Number(req.query.page) || 1;
const limit = Number(req.query.limit) || 50;
const items = await this.assetService.findByJobId(
const result = await this.assetService.findByJobId(
req.params.id,
assetType,
page,
......@@ -162,8 +148,13 @@ export class CrawlJobController {
res.json({
success: true,
data: {
items,
meta: { page, limit },
items: result.items,
meta: {
total: result.total,
page: result.page,
limit: result.limit,
totalPages: result.totalPages,
},
},
});
} catch (error) {
......@@ -413,11 +404,14 @@ export class CrawlJobController {
);
res.json({
success: true,
data: result.items,
pagination: {
total: result.total,
page: result.page,
limit: result.limit,
data: {
items: result.items,
meta: {
total: result.total,
page: result.page,
limit: result.limit,
totalPages: Math.ceil(result.total / (result.limit || 1)),
},
},
});
} catch (error) {
......
src/modules/crawl-jobs/crawl-job.repository.ts
View file @ b454ffff
......@@ -166,26 +166,21 @@ export class CrawlJobRepository {
failedPages?: number;
},
) {
const currentJob = await prisma.crawlJob.findUnique({
where: { id },
select: { status: true },
await prisma.crawlJob.updateMany({
where: {
id,
...(status !== JOB_STATUS.CANCELED
? { status: { not: JOB_STATUS.CANCELED } }
: {}),
},
data: { status, ...extra },
});
if (
currentJob?.status === JOB_STATUS.CANCELED &&
status !== JOB_STATUS.CANCELED
) {
return prisma.crawlJob.findUnique({
where: { id },
include: {
exports: true,
},
});
}
return prisma.crawlJob.update({
return prisma.crawlJob.findUnique({
where: { id },
data: { status, ...extra },
include: {
exports: true,
},
});
}
......
src/modules/crawl-jobs/crawl-job.route.ts
View file @ b454ffff
......@@ -6,6 +6,7 @@ import {
createCrawlJobSchema,
createExportSchema,
listCrawlJobsQuerySchema,
getAssetsQuerySchema,
} from "./crawl-job.validation";
import { crawlPageQuerySchema } from "../crawl-pages/crawl-page.validation";
import { requireRole } from "../../middlewares/role.middleware";
......@@ -105,6 +106,7 @@ router.get(
"/:id/assets",
apiKeyOrAuthMiddleware,
requireRole(ROLES.ADMIN, ROLES.CRAWLER_USER, ROLES.VIEWER),
validateQuery(getAssetsQuerySchema),
controller.getAssets,
);
router.get(
......
src/modules/crawl-jobs/crawl-job.service.ts
View file @ b454ffff
import { CrawlJobRepository } from "./crawl-job.repository";
import { CrawlExportRepository } from "../crawl-exports/crawl-export.repository";
import { CrawlScheduleRepository } from "../crawl-schedules/crawl-schedule.repository";
import { UserRepository } from "../users/user.repository";
import { AppError } from "../../common/errors/app-error";
import { ERROR_CODE } from "../../common/errors/error-code";
......@@ -21,6 +22,8 @@ import { getErrorMessage } from "../../common/helpers/error-mapping.helper";
export class CrawlJobService {
private readonly repository = new CrawlJobRepository();
private readonly userRepository = new UserRepository();
private readonly scheduleRepository = new CrawlScheduleRepository();
private readonly exportRepository = new CrawlExportRepository();
async create(userId: string, payload: CreateCrawlJobDto) {
const isUrlList = payload.mode === CRAWL_MODE.URL_LIST;
......@@ -41,6 +44,22 @@ export class CrawlJobService {
throw new AppError("User not found", 404, ERROR_CODE.NOT_FOUND);
}
if (payload.scheduleId) {
const schedule = await this.scheduleRepository.findById(
payload.scheduleId,
);
if (
!schedule ||
(user.role !== ROLES.ADMIN && schedule.userId !== userId)
) {
throw new AppError(
"Crawl schedule not found",
404,
ERROR_CODE.CRAWL_SCHEDULE_NOT_FOUND,
);
}
}
// SSRF validation with bounded concurrency for URL_LIST
if (isUrlList) {
const { validateUrlAsync } =
......@@ -221,8 +240,7 @@ export class CrawlJobService {
);
}
const exportRepository = new CrawlExportRepository();
const exports = await exportRepository.findByJobId(jobId);
const exports = await this.exportRepository.findByJobId(jobId);
const storage = StorageFactory.getStorageService();
for (const exportRecord of exports) {
......@@ -254,8 +272,7 @@ export class CrawlJobService {
);
}
const exportRepository = new CrawlExportRepository();
const exports = await exportRepository.findByJobId(jobId);
const exports = await this.exportRepository.findByJobId(jobId);
const storage = StorageFactory.getStorageService();
for (const exp of exports) {
......
src/modules/crawl-jobs/crawl-job.validation.ts
View file @ b454ffff
......@@ -2,6 +2,7 @@ import { z } from "zod";
import { EXPORT_TYPE } from "../../common/constants/export-type.constant";
import { JOB_STATUS } from "../../common/constants/job-status.constant";
import { CRAWL_MODE } from "../../common/constants/crawl-mode.constant";
import { ASSET_TYPES } from "../../common/constants";
export const createCrawlJobSchema = z
.object({
......@@ -67,3 +68,9 @@ export const listCrawlJobsQuerySchema = z.object({
page: z.coerce.number().int().min(1).optional(),
limit: z.coerce.number().int().min(1).max(100).optional(),
});
export const getAssetsQuerySchema = z.object({
assetType: z.nativeEnum(ASSET_TYPES).optional(),
page: z.coerce.number().int().min(1).default(1),
limit: z.coerce.number().int().min(1).max(500).default(50),
});
src/modules/crawl-pages/crawl-page-processor.service.ts
View file @ b454ffff
import { CrawlPageStatus } from "@prisma/client";
import { CrawlPageStatus } from "../../common/constants/crawl-page-status.constant";
import {
FirecrawlPageResult,
CrawlErrorItem,
......
src/modules/crawl-pages/crawl-page.dto.ts
View file @ b454ffff
import { CrawlPageStatus } from "@prisma/client";
import { CrawlPageStatus } from "../../common/constants/crawl-page-status.constant";
export interface CreateCrawlPageDto {
jobId: string;
......
src/modules/crawl-pages/crawl-page.repository.ts
View file @ b454ffff
......@@ -272,6 +272,7 @@ export class CrawlPageRepository {
contentHash?: string | null;
dataQualityScore?: number | null;
warnings?: string[];
hasSensitiveData?: boolean;
}) {
return prisma.crawlPage.upsert({
where: { jobId_url: { jobId: data.jobId, url: data.url } },
......@@ -291,6 +292,7 @@ export class CrawlPageRepository {
contentHash: data.contentHash,
dataQualityScore: data.dataQualityScore,
warnings: data.warnings,
hasSensitiveData: data.hasSensitiveData,
},
});
}
......
src/modules/crawl-pages/crawl-page.service.ts
View file @ b454ffff
import { CrawlPageRepository } from "./crawl-page.repository";
import { CrawlPageStatus } from "@prisma/client";
import { CrawlPageStatus } from "../../common/constants/crawl-page-status.constant";
import { CrawlPageQueryDto } from "./crawl-page.dto";
import {
extractMainContent,
......
src/modules/crawl-pages/crawl-page.validation.ts
View file @ b454ffff
import { z } from "zod";
import { CrawlPageStatus } from "@prisma/client";
import { CRAWL_PAGE_STATUS } from "../../common/constants/crawl-page-status.constant";
const parseBooleanQuery = (val: unknown) => {
if (val === undefined || val === null || val === "") return undefined;
......@@ -15,7 +15,7 @@ const parseNumberQuery = (val: unknown) => {
};
export const crawlPageQuerySchema = z.object({
status: z.nativeEnum(CrawlPageStatus).optional(),
status: z.nativeEnum(CRAWL_PAGE_STATUS).optional(),
statusCode: z.preprocess(parseNumberQuery, z.number().int().optional()),
search: z.string().trim().optional(),
......
src/modules/crawl-schedules/crawl-schedule.validation.ts
View file @ b454ffff
......@@ -46,20 +46,6 @@ export const createCrawlScheduleSchema = z
});
}
}
if (
data.frequency === SCHEDULE_FREQUENCY.WEEKLY &&
data.dayOfWeek === undefined
) {
data.dayOfWeek = 0; // Default to Sunday
}
if (
data.frequency === SCHEDULE_FREQUENCY.MONTHLY &&
data.dayOfMonth === undefined
) {
data.dayOfMonth = 1; // Default to 1st of month
}
});
export const updateCrawlScheduleSchema = z
......@@ -99,20 +85,24 @@ export const updateCrawlScheduleSchema = z
export const crawlScheduleQuerySchema = z.object({
search: z.string().trim().optional(),
frequency: z.nativeEnum(SCHEDULE_FREQUENCY).optional(),
isActive: z
.string()
.optional()
.transform((val) =>
val === "true" ? true : val === "false" ? false : undefined,
),
page: z
.string()
.optional()
.transform((val) => (val ? parseInt(val, 10) : 1)),
limit: z
.string()
isActive: z.preprocess((val) => {
if (val === "true" || val === true || val === "1") return true;
if (val === "false" || val === false || val === "0") return false;
return undefined;
}, z.boolean().optional()),
page: z.coerce.number().int().min(1).default(1),
limit: z.coerce.number().int().min(1).max(100).default(20),
sortBy: z
.enum([
"createdAt",
"updatedAt",
"name",
"frequency",
"nextRunAt",
"lastRunAt",
"isActive",
])
.optional()
.transform((val) => (val ? parseInt(val, 10) : 20)),
sortBy: z.string().optional().default("createdAt"),
.default("createdAt"),
order: z.enum(["asc", "desc"]).optional().default("desc"),
});
src/modules/exports/base-export.service.ts
View file @ b454ffff
import { CrawlJob, CrawlPage } from "@prisma/client";
import { CrawlJob, CrawlPage } from "../../common/types/database.types";
import path from "path";
import {
ensureJobExportStructure,
......
src/modules/exports/csv-export.service.ts
View file @ b454ffff
import fs from "fs";
import { CrawlAsset, CrawlJob, CrawlPage } from "@prisma/client";
import {
CrawlAsset,
CrawlJob,
CrawlPage,
} from "../../common/types/database.types";
import { CrawlAssetRepository } from "../crawl-assets/crawl-asset.repository";
import { JOB_EXPORT_FILES } from "../../common/constants/storage-path.constant";
import { EXPORT_MIME_TYPES } from "../../common/constants/export-type.constant";
......
src/modules/exports/export.service.ts
View file @ b454ffff
import { CrawlJob, CrawlPage } from "@prisma/client";
import { CrawlJob, CrawlPage } from "../../common/types/database.types";
import { CrawlJobRepository } from "../crawl-jobs/crawl-job.repository";
import { CrawlExportRepository } from "../crawl-exports/crawl-export.repository";
import { JsonExportService } from "./json-export.service";
......
src/modules/exports/json-export.service.ts
View file @ b454ffff
import fs from "fs";
import { parse as parseHtml } from "node-html-parser";
import { CrawlJob, CrawlPage, CrawlAsset } from "@prisma/client";
import {
CrawlJob,
CrawlPage,
CrawlAsset,
} from "../../common/types/database.types";
import { CrawlAssetRepository } from "../crawl-assets/crawl-asset.repository";
import { JOB_EXPORT_FILES } from "../../common/constants/storage-path.constant";
import { EXPORT_MIME_TYPES } from "../../common/constants/export-type.constant";
......
src/modules/exports/markdown-export.service.ts
View file @ b454ffff
import fs from "fs";
import archiver from "archiver";
import { CrawlJob, CrawlPage } from "@prisma/client";
import { CrawlJob, CrawlPage } from "../../common/types/database.types";
import { JOB_EXPORT_SUBDIRS } from "../../common/constants/storage-path.constant";
import { EXPORT_MIME_TYPES } from "../../common/constants/export-type.constant";
import {
......
src/modules/exports/xlsx-export.service.ts
View file @ b454ffff
import ExcelJS from "exceljs";
import { parse as parseHtml } from "node-html-parser";
import { CrawlJob, CrawlPage } from "@prisma/client";
import { CrawlJob, CrawlPage } from "../../common/types/database.types";
import { JOB_EXPORT_FILES } from "../../common/constants/storage-path.constant";
import { EXPORT_MIME_TYPES } from "../../common/constants/export-type.constant";
import { buildJobDataFilePath } from "../../common/helpers/file.helper";
......
src/modules/exports/zip-export.service.ts
View file @ b454ffff
......@@ -2,7 +2,7 @@ import fs from "fs";
import path from "path";
import archiver from "archiver";
import { PassThrough } from "stream";
import { CrawlJob, CrawlPage } from "@prisma/client";
import { CrawlJob, CrawlPage } from "../../common/types/database.types";
import { CrawlAssetRepository } from "../crawl-assets/crawl-asset.repository";
import {
JOB_EXPORT_FILES,
......
src/modules/users/user.repository.ts
View file @ b454ffff
......@@ -2,6 +2,7 @@ import { prisma } from "../../database/prisma.client";
import { UserRole, Prisma, User } from "@prisma/client";
import { UserQueryDto } from "./user.dto";
import { envConfig } from "../../config/env.config";
import { ROLES } from "../../common/constants/role.constant";
export class UserRepository {
async findAll(query: UserQueryDto = {}) {
......@@ -88,7 +89,7 @@ export class UserRepository {
passwordHash: data.passwordHash,
fullName: data.fullName,
avatarUrl: data.avatarUrl,
role: data.role ?? "CRAWLER_USER",
role: data.role ?? ROLES.CRAWLER_USER,
maxPagesLimit: data.maxPagesLimit ?? envConfig.quota.defaultMaxPages,
maxJobsPerDayLimit:
data.maxJobsPerDayLimit ?? envConfig.quota.defaultMaxJobsPerDay,
......
src/modules/users/user.service.ts
View file @ b454ffff
......@@ -2,7 +2,7 @@ import bcrypt from "bcryptjs";
import { UserRepository } from "./user.repository";
import { AppError } from "../../common/errors/app-error";
import { ERROR_CODE } from "../../common/errors/error-code";
import { User } from "@prisma/client";
import { User } from "../../common/types/database.types";
import { ROLES } from "../../common/constants/role.constant";
import {
CreateUserDto,
......
src/modules/users/user.validation.ts
View file @ b454ffff
......@@ -11,9 +11,9 @@ export const createUserSchema = z.object({
.or(z.literal(""))
.optional(),
role: z.nativeEnum(ROLES).optional(),
maxPagesLimit: z.number().int().min(1).optional(),
maxJobsPerDayLimit: z.number().int().min(1).optional(),
maxConcurrentJobsLimit: z.number().int().min(1).optional(),
maxPagesLimit: z.number().int().min(1).max(100000).optional(),
maxJobsPerDayLimit: z.number().int().min(1).max(10000).optional(),
maxConcurrentJobsLimit: z.number().int().min(1).max(100).optional(),
});
export const updateUserSchema = z.object({
......@@ -26,9 +26,9 @@ export const updateUserSchema = z.object({
.optional(),
isActive: z.boolean().optional(),
role: z.nativeEnum(ROLES).optional(),
maxPagesLimit: z.number().int().min(1).optional(),
maxJobsPerDayLimit: z.number().int().min(1).optional(),
maxConcurrentJobsLimit: z.number().int().min(1).optional(),
maxPagesLimit: z.number().int().min(1).max(100000).optional(),
maxJobsPerDayLimit: z.number().int().min(1).max(10000).optional(),
maxConcurrentJobsLimit: z.number().int().min(1).max(100).optional(),
});
export const listUsersQuerySchema = z.object({
......@@ -38,6 +38,6 @@ export const listUsersQuerySchema = z.object({
.enum(["true", "false"])
.transform((v) => v === "true")
.optional(),
page: z.coerce.number().int().min(1).optional(),
limit: z.coerce.number().int().min(1).max(100).optional(),
page: z.coerce.number().int().min(1).default(1),
limit: z.coerce.number().int().min(1).max(100).default(20),
});
src/modules/webhooks/webhook-config.service.ts
View file @ b454ffff
import { WebhookConfig } from "@prisma/client";
import { WebhookConfig } from "../../common/types/database.types";
import { WebhookRepository } from "./webhook.repository";
import { encrypt } from "./webhook-crypto.helper";
import { AppError } from "../../common/errors/app-error";
......
src/modules/webhooks/webhook-delivery.service.ts
View file @ b454ffff
import { WebhookDelivery, Prisma } from "@prisma/client";
import { WebhookDelivery, Prisma } from "../../common/types/database.types";
import { WebhookRepository } from "./webhook.repository";
import { decrypt, signPayload } from "./webhook-crypto.helper";
import { webhookQueue } from "../../queues/webhook.queue";
......
src/modules/webhooks/webhook.validation.ts
View file @ b454ffff
import { z } from "zod";
import {
WEBHOOK_DELIVERY_STATUS,
WEBHOOK_EVENT,
} from "../../common/constants/webhook.constant";
export const createWebhookConfigSchema = z.object({
url: z
......@@ -13,7 +17,7 @@ export const createWebhookConfigSchema = z.object({
.min(16, "Signing secret must be at least 16 characters long for security")
.max(128, "Signing secret is too long"),
events: z
.array(z.enum(["job.completed", "job.failed"]))
.array(z.nativeEnum(WEBHOOK_EVENT))
.min(1, "At least one event must be selected for notifications"),
});
......@@ -25,7 +29,7 @@ export const updateWebhookConfigSchema = z.object({
.max(128, "Signing secret is too long")
.optional(),
events: z
.array(z.enum(["job.completed", "job.failed"]))
.array(z.nativeEnum(WEBHOOK_EVENT))
.min(1, "At least one event must be selected for notifications")
.optional(),
isActive: z.boolean().optional(),
......@@ -33,7 +37,7 @@ export const updateWebhookConfigSchema = z.object({
export const listWebhookDeliveriesQuerySchema = z.object({
jobId: z.string().uuid().optional(),
status: z.enum(["PENDING", "SUCCESS", "FAILED"]).optional(),
page: z.coerce.number().int().min(1).optional(),
limit: z.coerce.number().int().min(1).max(100).optional(),
status: z.nativeEnum(WEBHOOK_DELIVERY_STATUS).optional(),
page: z.coerce.number().int().min(1).default(1),
limit: z.coerce.number().int().min(1).max(100).default(20),
});
src/queues/crawl.worker.processor.ts
View file @ b454ffff
......@@ -269,10 +269,6 @@ export async function processCrawlJob(job: Job): Promise<void> {
}
}
await getJobRepository().updateStatus(jobId, JOB_STATUS.RUNNING, {
startedAt: new Date(),
});
try {
if (crawlJob.mode === CRAWL_MODE.SCRAPE) {
console.log(
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment