Skip to content

U
upgrade-data-crawler-be
Commit c168a47c authored by ThinhNC's avatar ThinhNC
Browse files
Options

refactor(common): standardize hardcoded constants and enforce zero-hardcode architecture

parent 65fc55f1
Expand all Show whitespace changes
Inline Side-by-side
Showing with 13806 additions and 8747 deletions
.agents/rules/design.md
View file @ c168a47c
...@@ -22,10 +22,10 @@ Route → Controller → Service → Repository → Prisma → Postgre ...@@ -22,10 +22,10 @@ Route → Controller → Service → Repository → Prisma → Postgre
- Mọi endpoint nhận dữ liệu từ client (`body`, `query`, `params`) phải có schema kiểm thực tương ứng bằng Zod. - Mọi endpoint nhận dữ liệu từ client (`body`, `query`, `params`) phải có schema kiểm thực tương ứng bằng Zod.
- Sử dụng middleware dùng chung: - Sử dụng middleware dùng chung:
```typescript ```typescript
import { validate } from '../../middlewares/validate.middleware'; import { validate } from "../../middlewares/validate.middleware";
import { mySchema } from './my.validation'; import { mySchema } from "./my.validation";
router.post('/', validate(mySchema), myController.create); router.post("/", validate(mySchema), myController.create);
``` ```
- DTO type được suy diễn trực tiếp từ schema: `type MyDto = z.infer<typeof mySchema>;`. - DTO type được suy diễn trực tiếp từ schema: `type MyDto = z.infer<typeof mySchema>;`.
...@@ -35,10 +35,10 @@ Route → Controller → Service → Repository → Prisma → Postgre ...@@ -35,10 +35,10 @@ Route → Controller → Service → Repository → Prisma → Postgre
- Bắt buộc dùng `AppError` kèm HTTP status code và mã `ERROR_CODE`: - Bắt buộc dùng `AppError` kèm HTTP status code và mã `ERROR_CODE`:
```typescript ```typescript
import { AppError } from '../../common/errors/app-error'; import { AppError } from "../../common/errors/app-error";
import { ERROR_CODE } from '../../common/errors/error-code'; import { ERROR_CODE } from "../../common/errors/error-code";
throw new AppError('Resource not found', 404, ERROR_CODE.NOT_FOUND); throw new AppError("Resource not found", 404, ERROR_CODE.NOT_FOUND);
``` ```
- Định dạng response lỗi chuẩn: - Định dạng response lỗi chuẩn:
```json ```json
...@@ -55,5 +55,6 @@ Route → Controller → Service → Repository → Prisma → Postgre ...@@ -55,5 +55,6 @@ Route → Controller → Service → Repository → Prisma → Postgre
## 4. Hợp Đồng Dữ Liệu Cào (Data Contract V1) ## 4. Hợp Đồng Dữ Liệu Cào (Data Contract V1)
Bảo toàn hợp đồng dữ liệu quy định tại `docs/DATA_CONTRACT_V1.md`: Bảo toàn hợp đồng dữ liệu quy định tại `docs/DATA_CONTRACT_V1.md`:
- Dữ liệu thô (`raw`): Nguyên bản HTML từ Firecrawl. - Dữ liệu thô (`raw`): Nguyên bản HTML từ Firecrawl.
- Dữ liệu sạch (`clean`): Markdown chuẩn hóa qua Turndown, lọc bỏ script/ads/styles, tính toán `dataQualityScore` và `contentHash`. - Dữ liệu sạch (`clean`): Markdown chuẩn hóa qua Turndown, lọc bỏ script/ads/styles, tính toán `dataQualityScore` và `contentHash`.
.agents/rules/workflow.md
View file @ c168a47c
...@@ -20,6 +20,7 @@ Khi xây dựng hoặc sửa đổi tính năng trong `data-crawler-be`, thực ...@@ -20,6 +20,7 @@ Khi xây dựng hoặc sửa đổi tính năng trong `data-crawler-be`, thực
## 2. Quy Chuẩn Commit Git ## 2. Quy Chuẩn Commit Git
Áp dụng chuẩn Conventional Commits: Áp dụng chuẩn Conventional Commits:
- `feat(<module>):` Thêm chức năng mới - `feat(<module>):` Thêm chức năng mới
- `fix(<module>):` Sửa lỗi nghiệp vụ hoặc kỹ thuật - `fix(<module>):` Sửa lỗi nghiệp vụ hoặc kỹ thuật
- `refactor(<module>):` Tối ưu hóa code mà không thay đổi tính năng - `refactor(<module>):` Tối ưu hóa code mà không thay đổi tính năng
......
.agents/skills/code-review-and-quality/SKILL.md
View file @ c168a47c
...@@ -110,14 +110,14 @@ Small, focused changes are easier to review, faster to merge, and safer to deplo ...@@ -110,14 +110,14 @@ Small, focused changes are easier to review, faster to merge, and safer to deplo
~1000 lines changed → Too large. Split it. ~1000 lines changed → Too large. Split it.
``` ```
**Watch file size, not just diff size.** A small diff can still push a file past a healthy boundary — around 1000 *total* lines in a single file (distinct from the ~1000 *changed*-lines threshold above) is a common inspection signal, not a hard cap. When a change materially grows an already-large file, ask whether to extract helpers, subcomponents, or modules *first*, before piling more on. Decompose, then add. **Watch file size, not just diff size.** A small diff can still push a file past a healthy boundary — around 1000 _total_ lines in a single file (distinct from the ~1000 _changed_-lines threshold above) is a common inspection signal, not a hard cap. When a change materially grows an already-large file, ask whether to extract helpers, subcomponents, or modules _first_, before piling more on. Decompose, then add.
**What counts as "one change":** A single self-contained modification that addresses one thing, includes related tests, and keeps the system functional after submission. One part of a feature — not the whole feature. **What counts as "one change":** A single self-contained modification that addresses one thing, includes related tests, and keeps the system functional after submission. One part of a feature — not the whole feature.
**Splitting strategies when a change is too large:** **Splitting strategies when a change is too large:**
| Strategy | How | When | | Strategy | How | When |
|----------|-----|------| | ----------------- | ------------------------------------------------------- | ----------------------- |
| **Stack** | Submit a small change, start the next one based on it | Sequential dependencies | | **Stack** | Submit a small change, start the next one based on it | Sequential dependencies |
| **By file group** | Separate changes for groups needing different reviewers | Cross-cutting concerns | | **By file group** | Separate changes for groups needing different reviewers | Cross-cutting concerns |
| **Horizontal** | Create shared code/stubs first, then consumers | Layered architecture | | **Horizontal** | Create shared code/stubs first, then consumers | Layered architecture |
...@@ -179,8 +179,8 @@ For each file changed: ...@@ -179,8 +179,8 @@ For each file changed:
Label every comment with its severity so the author knows what's required vs optional: Label every comment with its severity so the author knows what's required vs optional:
| Prefix | Meaning | Author Action | | Prefix | Meaning | Author Action |
|--------|---------|---------------| | ----------------------------- | ------------------ | ------------------------------------------------------- |
| *(no prefix)* | Required change | Must address before merge | | _(no prefix)_ | Required change | Must address before merge |
| **Critical:** | Blocks merge | Security vulnerability, data loss, broken functionality | | **Critical:** | Blocks merge | Security vulnerability, data loss, broken functionality |
| **Nit:** | Minor, optional | Author may ignore — formatting, style preferences | | **Nit:** | Minor, optional | Author may ignore — formatting, style preferences |
| **Optional:** / **Consider:** | Suggestion | Worth considering but not required | | **Optional:** / **Consider:** | Suggestion | Worth considering but not required |
...@@ -188,7 +188,7 @@ Label every comment with its severity so the author knows what's required vs opt ...@@ -188,7 +188,7 @@ Label every comment with its severity so the author knows what's required vs opt
This prevents authors from treating all feedback as mandatory and wasting time on optional suggestions. This prevents authors from treating all feedback as mandatory and wasting time on optional suggestions.
**Lead with what matters.** Order findings by leverage: correctness and security first, then structural regressions and missed simplifications, then everything else. Don't bury a real issue under cosmetic nits — a few high-conviction comments beat a long list. If you have one structural problem and ten nits, the structural problem *is* the review. **Lead with what matters.** Order findings by leverage: correctness and security first, then structural regressions and missed simplifications, then everything else. Don't bury a real issue under cosmetic nits — a few high-conviction comments beat a long list. If you have one structural problem and ten nits, the structural problem _is_ the review.
### Step 5: Verify the Verification ### Step 5: Verify the Verification
...@@ -222,6 +222,7 @@ Human makes the final call ...@@ -222,6 +222,7 @@ Human makes the final call
This catches issues that a single model might miss — different models have different blind spots. This catches issues that a single model might miss — different models have different blind spots.
**Example prompt for a review agent:** **Example prompt for a review agent:**
``` ```
Review this code change for correctness, security, and adherence to Review this code change for correctness, security, and adherence to
our project conventions. The spec says [X]. The change should [Y]. our project conventions. The spec says [X]. The change should [Y].
...@@ -281,6 +282,7 @@ When reviewing code — whether written by you, another agent, or a human: ...@@ -281,6 +282,7 @@ When reviewing code — whether written by you, another agent, or a human:
Part of code review is dependency review: Part of code review is dependency review:
**Before adding any dependency:** **Before adding any dependency:**
1. Does the existing stack solve this? (Often it does.) 1. Does the existing stack solve this? (Often it does.)
2. How large is the dependency? (Check bundle impact.) 2. How large is the dependency? (Check bundle impact.)
3. Is it actively maintained? (Check last commit, open issues.) 3. Is it actively maintained? (Check last commit, open issues.)
...@@ -293,11 +295,11 @@ Part of code review is dependency review: ...@@ -293,11 +295,11 @@ Part of code review is dependency review:
1. **Read the changelog, not just the version number.** Semver is a promise the maintainer may not have kept — a "patch" can carry a behavioral change. For a major bump, read the migration notes and find what breaks. 1. **Read the changelog, not just the version number.** Semver is a promise the maintainer may not have kept — a "patch" can carry a behavioral change. For a major bump, read the migration notes and find what breaks.
2. **One dependency per change.** Upgrade and merge them individually (or in small related groups). When a bulk bump breaks the build, you've lost which package did it; a single-package change makes the cause obvious and the revert clean. 2. **One dependency per change.** Upgrade and merge them individually (or in small related groups). When a bulk bump breaks the build, you've lost which package did it; a single-package change makes the cause obvious and the revert clean.
3. **Let the tests decide.** The upgrade is verified by a green suite before *and* after, not by "it installed." If coverage around the dependency's behavior is thin, that gap is the real finding — add a test first. 3. **Let the tests decide.** The upgrade is verified by a green suite before _and_ after, not by "it installed." If coverage around the dependency's behavior is thin, that gap is the real finding — add a test first.
4. **Mind the transitive graph.** Most installed packages are ones nobody chose directly. Review the lockfile diff, not just `package.json`; a single direct bump can pull in dozens of indirect changes. 4. **Mind the transitive graph.** Most installed packages are ones nobody chose directly. Review the lockfile diff, not just `package.json`; a single direct bump can pull in dozens of indirect changes.
5. **Keep the lockfile honest.** Commit it, review its diff, and never hand-edit it. The lockfile is the thing that actually pins what ships. 5. **Keep the lockfile honest.** Commit it, review its diff, and never hand-edit it. The lockfile is the thing that actually pins what ships.
For triaging `npm audit` findings and supply-chain risk (typosquatting, compromised maintainers), follow the `security-and-hardening` skill — this section covers the upgrade *workflow*, that one covers the security verdict. For triaging `npm audit` findings and supply-chain risk (typosquatting, compromised maintainers), follow the `security-and-hardening` skill — this section covers the upgrade _workflow_, that one covers the security verdict.
## The Review Checklist ## The Review Checklist
...@@ -305,20 +307,24 @@ For triaging `npm audit` findings and supply-chain risk (typosquatting, compromi ...@@ -305,20 +307,24 @@ For triaging `npm audit` findings and supply-chain risk (typosquatting, compromi
## Review: [PR/Change title] ## Review: [PR/Change title]
### Context ### Context
- [ ] I understand what this change does and why - [ ] I understand what this change does and why
### Correctness ### Correctness
- [ ] Change matches spec/task requirements - [ ] Change matches spec/task requirements
- [ ] Edge cases handled - [ ] Edge cases handled
- [ ] Error paths handled - [ ] Error paths handled
- [ ] Tests cover the change adequately - [ ] Tests cover the change adequately
### Readability ### Readability
- [ ] Names are clear and consistent - [ ] Names are clear and consistent
- [ ] Logic is straightforward - [ ] Logic is straightforward
- [ ] No unnecessary complexity - [ ] No unnecessary complexity
### Architecture ### Architecture
- [ ] Follows existing patterns - [ ] Follows existing patterns
- [ ] No unnecessary coupling or dependencies - [ ] No unnecessary coupling or dependencies
- [ ] Appropriate abstraction level - [ ] Appropriate abstraction level
...@@ -326,6 +332,7 @@ For triaging `npm audit` findings and supply-chain risk (typosquatting, compromi ...@@ -326,6 +332,7 @@ For triaging `npm audit` findings and supply-chain risk (typosquatting, compromi
- [ ] No feature logic in shared modules; file stays within a healthy size - [ ] No feature logic in shared modules; file stays within a healthy size
### Security ### Security
- [ ] No secrets in code - [ ] No secrets in code
- [ ] Input validated at boundaries - [ ] Input validated at boundaries
- [ ] No injection vulnerabilities - [ ] No injection vulnerabilities
...@@ -333,19 +340,23 @@ For triaging `npm audit` findings and supply-chain risk (typosquatting, compromi ...@@ -333,19 +340,23 @@ For triaging `npm audit` findings and supply-chain risk (typosquatting, compromi
- [ ] External data sources treated as untrusted - [ ] External data sources treated as untrusted
### Performance ### Performance
- [ ] No N+1 patterns - [ ] No N+1 patterns
- [ ] No unbounded operations - [ ] No unbounded operations
- [ ] Pagination on list endpoints - [ ] Pagination on list endpoints
### Verification ### Verification
- [ ] Tests pass - [ ] Tests pass
- [ ] Build succeeds - [ ] Build succeeds
- [ ] Manual verification done (if applicable) - [ ] Manual verification done (if applicable)
### Verdict ### Verdict
- [ ] **Approve** — Ready to merge - [ ] **Approve** — Ready to merge
- [ ] **Request changes** — Issues must be addressed - [ ] **Request changes** — Issues must be addressed
``` ```
## See Also ## See Also
- For detailed security review guidance, see `../../references/security-checklist.md` - For detailed security review guidance, see `../../references/security-checklist.md`
...@@ -354,7 +365,7 @@ For triaging `npm audit` findings and supply-chain risk (typosquatting, compromi ...@@ -354,7 +365,7 @@ For triaging `npm audit` findings and supply-chain risk (typosquatting, compromi
## Common Rationalizations ## Common Rationalizations
| Rationalization | Reality | | Rationalization | Reality |
|---|---| | ------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| "It works, that's good enough" | Working code that's unreadable, insecure, or architecturally wrong creates debt that compounds. | | "It works, that's good enough" | Working code that's unreadable, insecure, or architecturally wrong creates debt that compounds. |
| "I wrote it, so I know it's correct" | Authors are blind to their own assumptions. Every change benefits from another set of eyes. | | "I wrote it, so I know it's correct" | Authors are blind to their own assumptions. Every change benefits from another set of eyes. |
| "We'll clean it up later" | Later never comes. The review is the quality gate — use it. Require cleanup before merge, not after. | | "We'll clean it up later" | Later never comes. The review is the quality gate — use it. Require cleanup before merge, not after. |
......
.agents/skills/full-project-audit/SKILL.md
View file @ c168a47c
...@@ -31,6 +31,7 @@ The workflow operates **autonomously** without requiring manual user prompt paci ...@@ -31,6 +31,7 @@ The workflow operates **autonomously** without requiring manual user prompt paci
## Core Invariants & Safety Guardrails ## Core Invariants & Safety Guardrails
### 1. General Safety Guardrails ### 1. General Safety Guardrails
- **DO NOT** delete data or drop database tables/schemas. - **DO NOT** delete data or drop database tables/schemas.
- **DO NOT** execute destructive migrations (`pnpm db:migrate:reset` or manual `DROP TABLE`). - **DO NOT** execute destructive migrations (`pnpm db:migrate:reset` or manual `DROP TABLE`).
- **DO NOT** remove authentication checks, disable authorization middleware, or weaken Zod validation schemas. - **DO NOT** remove authentication checks, disable authorization middleware, or weaken Zod validation schemas.
...@@ -38,7 +39,9 @@ The workflow operates **autonomously** without requiring manual user prompt paci ...@@ -38,7 +39,9 @@ The workflow operates **autonomously** without requiring manual user prompt paci
- **DO NOT** weaken security controls or mock out security middleware merely to make test suites pass. - **DO NOT** weaken security controls or mock out security middleware merely to make test suites pass.
### 2. Financial Logic Invariants ### 2. Financial Logic Invariants
When auditing or repairing applications handling wallets, transactions, budgets, or accounting: When auditing or repairing applications handling wallets, transactions, budgets, or accounting:
- **Income**: Increases target wallet balance. - **Income**: Increases target wallet balance.
- **Expense**: Decreases target wallet balance. - **Expense**: Decreases target wallet balance.
- **Transfer**: Decreases source wallet balance, increases destination wallet balance. A transfer **must never** be counted as income or expense in revenue/spending analytics. - **Transfer**: Decreases source wallet balance, increases destination wallet balance. A transfer **must never** be counted as income or expense in revenue/spending analytics.
...@@ -49,6 +52,7 @@ When auditing or repairing applications handling wallets, transactions, budgets, ...@@ -49,6 +52,7 @@ When auditing or repairing applications handling wallets, transactions, budgets,
- **Ownership & Tenant Isolation**: Never trust `userId` or `walletId` from client body or query params. Always verify that the authenticated user owns the resource being accessed or modified. - **Ownership & Tenant Isolation**: Never trust `userId` or `walletId` from client body or query params. Always verify that the authenticated user owns the resource being accessed or modified.
### 3. Timezone Invariants (Asia/Ho_Chi_Minh — UTC+7) ### 3. Timezone Invariants (Asia/Ho_Chi_Minh — UTC+7)
- The official business timezone is **`Asia/Ho_Chi_Minh` (UTC+7, +07:00)**. - The official business timezone is **`Asia/Ho_Chi_Minh` (UTC+7, +07:00)**.
- **Boundary Auditing**: All date filters, `startOfDay`, `endOfDay`, monthly aggregations, budget periods, reports, reminders, cron schedules, and daily quotas must be calculated in `Asia/Ho_Chi_Minh`. - **Boundary Auditing**: All date filters, `startOfDay`, `endOfDay`, monthly aggregations, budget periods, reports, reminders, cron schedules, and daily quotas must be calculated in `Asia/Ho_Chi_Minh`.
- **Near-Midnight Invariant**: A transaction occurring at `23:59:59` or `00:00:01` Vietnam time must strictly belong to the correct Vietnam business calendar date, regardless of whether the server or database runs in UTC (`+00:00`). - **Near-Midnight Invariant**: A transaction occurring at `23:59:59` or `00:00:01` Vietnam time must strictly belong to the correct Vietnam business calendar date, regardless of whether the server or database runs in UTC (`+00:00`).
...@@ -123,7 +127,9 @@ Convert all audit findings into a structured, prioritized backlog using the foll ...@@ -123,7 +127,9 @@ Convert all audit findings into a structured, prioritized backlog using the foll
- **P3 — Low**: Code quality, architectural convention drift, minor CPU/memory optimizations, documentation inaccuracies, or cosmetic formatting. - **P3 — Low**: Code quality, architectural convention drift, minor CPU/memory optimizations, documentation inaccuracies, or cosmetic formatting.
#### Finding Entry Structure #### Finding Entry Structure
For every finding recorded in the backlog: For every finding recorded in the backlog:
- **ID**: e.g., `BUG-P0-01`, `BUG-P1-02` - **ID**: e.g., `BUG-P0-01`, `BUG-P1-02`
- **Severity**: `P0` / `P1` / `P2` / `P3` - **Severity**: `P0` / `P1` / `P2` / `P3`
- **Module**: Feature/module directory name - **Module**: Feature/module directory name
...@@ -136,6 +142,7 @@ For every finding recorded in the backlog: ...@@ -136,6 +142,7 @@ For every finding recorded in the backlog:
- **Required Tests**: Specific test cases to prove the bug is resolved and prevent regressions - **Required Tests**: Specific test cases to prove the bug is resolved and prevent regressions
#### Priority Order for Triage #### Priority Order for Triage
1. Data corruption & data loss 1. Data corruption & data loss
2. Financial calculation and balance errors 2. Financial calculation and balance errors
3. Security vulnerabilities (SSRF, Auth/IDOR, Injection) 3. Security vulnerabilities (SSRF, Auth/IDOR, Injection)
...@@ -164,6 +171,7 @@ Before applying any code changes, rigorously verify every **P0** and **P1** find ...@@ -164,6 +171,7 @@ Before applying any code changes, rigorously verify every **P0** and **P1** find
### Step 4 — Fix P0 Issues ### Step 4 — Fix P0 Issues
Implement fixes for all `CONFIRMED` P0 findings adhering to these rules: Implement fixes for all `CONFIRMED` P0 findings adhering to these rules:
- **Smallest Safe Change**: Make the minimal diff necessary to fix the root cause. - **Smallest Safe Change**: Make the minimal diff necessary to fix the root cause.
- **Preserve Architecture**: Follow existing repository patterns and layered architecture. - **Preserve Architecture**: Follow existing repository patterns and layered architecture.
- **No Unrelated Refactors**: Do not reformat or clean up unrelated code in the same change. - **No Unrelated Refactors**: Do not reformat or clean up unrelated code in the same change.
...@@ -197,6 +205,7 @@ Validate that all P0 fixes are working and introduce no regressions: ...@@ -197,6 +205,7 @@ Validate that all P0 fixes are working and introduce no regressions:
### Step 6 — Fix P1 Issues ### Step 6 — Fix P1 Issues
Once P0 fixes are verified and green: Once P0 fixes are verified and green:
- Apply targeted, minimal fixes for all `CONFIRMED` P1 findings. - Apply targeted, minimal fixes for all `CONFIRMED` P1 findings.
- Maintain the same strict standards: no architectural disruption, no breaking contract changes, minimal clean diff. - Maintain the same strict standards: no architectural disruption, no breaking contract changes, minimal clean diff.
...@@ -226,13 +235,14 @@ Perform a second full audit pass over the entire codebase to verify resolution a ...@@ -226,13 +235,14 @@ Perform a second full audit pass over the entire codebase to verify resolution a
### Step 9 — Secondary Fixes (Convergence Loop) ### Step 9 — Secondary Fixes (Convergence Loop)
If the re-audit uncovers new `CONFIRMED` P0 or P1 issues caused by recent edits: If the re-audit uncovers new `CONFIRMED` P0 or P1 issues caused by recent edits:
1. Re-enter the loop: `VERIFY -> FIX -> TEST -> RE-AUDIT`. 1. Re-enter the loop: `VERIFY -> FIX -> TEST -> RE-AUDIT`.
2. Iterate until: 2. Iterate until:
- Zero confirmed P0 issues remain. - Zero confirmed P0 issues remain.
- Zero confirmed P1 issues remain. - Zero confirmed P1 issues remain.
- All tests pass cleanly. - All tests pass cleanly.
- Lint and typecheck pass with zero errors. - Lint and typecheck pass with zero errors.
3. *Convergence limit*: If an issue cannot be resolved within 3 iterations without major architectural redesign, document it clearly in the report as `Deferred` and stop the loop. 3. _Convergence limit_: If an issue cannot be resolved within 3 iterations without major architectural redesign, document it clearly in the report as `Deferred` and stop the loop.
--- ---
...@@ -251,14 +261,17 @@ Create directory `docs/audits/` (if it does not exist) and write the final repor ...@@ -251,14 +261,17 @@ Create directory `docs/audits/` (if it does not exist) and write the final repor
**Status**: [Clean / Action Required / Converged] **Status**: [Clean / Action Required / Converged]
## Executive Summary ## Executive Summary
Concise 2–3 paragraph summary of the audit scope, critical issues discovered, fixes applied, test outcomes, and current repository health. Concise 2–3 paragraph summary of the audit scope, critical issues discovered, fixes applied, test outcomes, and current repository health.
## Initial Findings Backlog ## Initial Findings Backlog
Summary table of all findings from Step 2 with Severity (P0, P1, P2, P3), Module, and Status (Fixed / Verified / Deferred). Summary table of all findings from Step 2 with Severity (P0, P1, P2, P3), Module, and Status (Fixed / Verified / Deferred).
## Fixed Issues Detail ## Fixed Issues Detail
### [BUG-P0-01] [Issue Title] ### [BUG-P0-01] [Issue Title]
- **Severity**: P0 - **Severity**: P0
- **Module**: [module] - **Module**: [module]
- **Root Cause**: [explanation] - **Root Cause**: [explanation]
...@@ -270,18 +283,22 @@ Summary table of all findings from Step 2 with Severity (P0, P1, P2, P3), Module ...@@ -270,18 +283,22 @@ Summary table of all findings from Step 2 with Severity (P0, P1, P2, P3), Module
[... repeat for all fixed P0 and P1 issues ...] [... repeat for all fixed P0 and P1 issues ...]
## Test Execution Summary ## Test Execution Summary
- **Typecheck**: PASSED - **Typecheck**: PASSED
- **Lint**: PASSED - **Lint**: PASSED
- **Unit & Integration Tests**: [X] passed, 0 failed - **Unit & Integration Tests**: [X] passed, 0 failed
- **New Tests Added**: [list of new test suites] - **New Tests Added**: [list of new test suites]
## Re-Audit Results ## Re-Audit Results
Detailed checklist proving no secondary regressions, contract breakages, or timezone errors remain. Detailed checklist proving no secondary regressions, contract breakages, or timezone errors remain.
## Remaining & Deferred Issues (P2 / P3) ## Remaining & Deferred Issues (P2 / P3)
List of non-blocking P2 and P3 issues scheduled for future maintenance cycles with recommended remediation. List of non-blocking P2 and P3 issues scheduled for future maintenance cycles with recommended remediation.
## Risk Assessment & Next Steps ## Risk Assessment & Next Steps
- Remaining operational or infrastructure risks. - Remaining operational or infrastructure risks.
- Actionable recommendations for the development team. - Actionable recommendations for the development team.
``` ```
...@@ -291,6 +308,7 @@ List of non-blocking P2 and P3 issues scheduled for future maintenance cycles wi ...@@ -291,6 +308,7 @@ List of non-blocking P2 and P3 issues scheduled for future maintenance cycles wi
## Final Output Summary ## Final Output Summary
Upon completion of the workflow, output a clear, concise terminal summary: Upon completion of the workflow, output a clear, concise terminal summary:
- **P0 Fixed**: Total count and IDs - **P0 Fixed**: Total count and IDs
- **P1 Fixed**: Total count and IDs - **P1 Fixed**: Total count and IDs
- **P2 / P3 Remaining**: Total count and IDs - **P2 / P3 Remaining**: Total count and IDs
......
.agents/skills/shop-amazon/SKILL.md
View file @ c168a47c
...@@ -35,6 +35,7 @@ Gửi yêu cầu tới `POST /api/v1/crawl-jobs`: ...@@ -35,6 +35,7 @@ Gửi yêu cầu tới `POST /api/v1/crawl-jobs`:
``` ```
> **Quy tắc chống chặn (Anti-bot):** > **Quy tắc chống chặn (Anti-bot):**
>
> - Amazon áp dụng cơ chế phát hiện bot rất mạnh. Luôn đặt `delayMs >= 2000`ms. > - Amazon áp dụng cơ chế phát hiện bot rất mạnh. Luôn đặt `delayMs >= 2000`ms.
> - Đảm bảo worker bắt cờ `CAPTCHA_DETECTED` hoặc `BLOCKED` trong bảng `crawl_pages` để cảnh báo kịp thời. > - Đảm bảo worker bắt cờ `CAPTCHA_DETECTED` hoặc `BLOCKED` trong bảng `crawl_pages` để cảnh báo kịp thời.
...@@ -49,12 +50,43 @@ Gửi yêu cầu tới `POST /api/v1/crawl-jobs`: ...@@ -49,12 +50,43 @@ Gửi yêu cầu tới `POST /api/v1/crawl-jobs`:
"domain": "amazon.com", "domain": "amazon.com",
"name": "Amazon Product Standard Template", "name": "Amazon Product Standard Template",
"fields": [ "fields": [
{ "name": "title", "selector": "#productTitle, h2 a.a-link-normal span", "type": "text", "required": true }, {
{ "name": "price", "selector": ".a-price .a-offscreen, span.a-price-whole", "type": "text", "required": false }, "name": "title",
{ "name": "rating", "selector": "span[data-hook='rating-out-of-text'], span.a-icon-alt", "type": "text", "required": false }, "selector": "#productTitle, h2 a.a-link-normal span",
{ "name": "reviewCount", "selector": "#acrCustomerReviewText, span[data-hook='total-review-count']", "type": "number", "required": false }, "type": "text",
{ "name": "mainImage", "selector": "#landingImage, .s-image", "type": "attribute", "attributeName": "src", "required": false }, "required": true
{ "name": "availability", "selector": "#availability span", "type": "text", "required": false } },
{
"name": "price",
"selector": ".a-price .a-offscreen, span.a-price-whole",
"type": "text",
"required": false
},
{
"name": "rating",
"selector": "span[data-hook='rating-out-of-text'], span.a-icon-alt",
"type": "text",
"required": false
},
{
"name": "reviewCount",
"selector": "#acrCustomerReviewText, span[data-hook='total-review-count']",
"type": "number",
"required": false
},
{
"name": "mainImage",
"selector": "#landingImage, .s-image",
"type": "attribute",
"attributeName": "src",
"required": false
},
{
"name": "availability",
"selector": "#availability span",
"type": "text",
"required": false
}
] ]
} }
``` ```
...@@ -64,5 +96,6 @@ Gửi yêu cầu tới `POST /api/v1/crawl-jobs`: ...@@ -64,5 +96,6 @@ Gửi yêu cầu tới `POST /api/v1/crawl-jobs`:
## 4. Xuất Dữ Liệu Sau Khi Cào ## 4. Xuất Dữ Liệu Sau Khi Cào
Sau khi Job đạt trạng thái `COMPLETED`: Sau khi Job đạt trạng thái `COMPLETED`:
- Kích hoạt export sang Excel qua `POST /api/v1/exports` với `exportType: "XLSX"`. - Kích hoạt export sang Excel qua `POST /api/v1/exports` với `exportType: "XLSX"`.
- Báo cáo kết quả và đường dẫn tải file xuất cho người dùng. - Báo cáo kết quả và đường dẫn tải file xuất cho người dùng.
.claude/CLAUDE.md
View file @ c168a47c
...@@ -22,6 +22,7 @@ Route → Controller → Service → Repository → Prisma Client → ...@@ -22,6 +22,7 @@ Route → Controller → Service → Repository → Prisma Client →
``` ```
### Quy tắc bất di bất dịch: ### Quy tắc bất di bất dịch:
1. **Chỉ Repository được gọi Prisma:** Tuyệt đối **chỉ có** các file `*.repository.ts` được import `prisma` hoặc `PrismaClient`. Service, Controller, Worker hay Helper **không bao giờ** được gọi Prisma trực tiếp. 1. **Chỉ Repository được gọi Prisma:** Tuyệt đối **chỉ có** các file `*.repository.ts` được import `prisma` hoặc `PrismaClient`. Service, Controller, Worker hay Helper **không bao giờ** được gọi Prisma trực tiếp.
2. **Cấu trúc Module chuẩn:** Mọi tính năng nghiệp vụ đặt tại `src/modules/<feature>/` với đầy đủ các file quy chuẩn: 2. **Cấu trúc Module chuẩn:** Mọi tính năng nghiệp vụ đặt tại `src/modules/<feature>/` với đầy đủ các file quy chuẩn:
- `<feature>.route.ts`: Định nghĩa endpoint, gắn middleware (auth, validate, rate-limit). - `<feature>.route.ts`: Định nghĩa endpoint, gắn middleware (auth, validate, rate-limit).
......
.claude/agents/reviewer.md
View file @ c168a47c
...@@ -16,21 +16,25 @@ Bạn là **Reviewer Sub-Agent** chịu trách nhiệm kiểm duyệt mọi thay ...@@ -16,21 +16,25 @@ Bạn là **Reviewer Sub-Agent** chịu trách nhiệm kiểm duyệt mọi thay
## 1. Danh Sách Kiểm Tra Bắt Buộc (Review Checklist) ## 1. Danh Sách Kiểm Tra Bắt Buộc (Review Checklist)
### A. Tính Tuân Thủ Kiến Trúc (Architectural Compliance) ### A. Tính Tuân Thủ Kiến Trúc (Architectural Compliance)
- [ ] **Quy tắc Prisma độc quyền:** Chỉ duy nhất các file `*.repository.ts` được import `prisma` hoặc `PrismaClient`. Tuyệt đối không chấp nhận Prisma query trong Controller, Service, Worker hay Middleware. - [ ] **Quy tắc Prisma độc quyền:** Chỉ duy nhất các file `*.repository.ts` được import `prisma` hoặc `PrismaClient`. Tuyệt đối không chấp nhận Prisma query trong Controller, Service, Worker hay Middleware.
- [ ] **Phân tách tầng rõ ràng:** Controller không chứa logic tính toán nghiệp vụ; Service không can thiệp vào định dạng response HTTP (`res.status()`). - [ ] **Phân tách tầng rõ ràng:** Controller không chứa logic tính toán nghiệp vụ; Service không can thiệp vào định dạng response HTTP (`res.status()`).
- [ ] **Đăng ký Route:** Route mới đã được đăng ký vào `src/routes/index.ts` và có prefix hợp lệ `/api/v1/...`. - [ ] **Đăng ký Route:** Route mới đã được đăng ký vào `src/routes/index.ts` và có prefix hợp lệ `/api/v1/...`.
### B. Tính Toàn Vẹn Dữ Liệu & Xác Thực (Validation & Type Safety) ### B. Tính Toàn Vẹn Dữ Liệu & Xác Thực (Validation & Type Safety)
- [ ] **Xác thực Zod:** Toàn bộ Body, Query và Params phải đi qua `validate(schema)` middleware. - [ ] **Xác thực Zod:** Toàn bộ Body, Query và Params phải đi qua `validate(schema)` middleware.
- [ ] **Kiểu dữ liệu TypeScript:** Không dùng `any` bừa bãi. Sử dụng `z.infer<typeof schema>` cho các DTO. - [ ] **Kiểu dữ liệu TypeScript:** Không dùng `any` bừa bãi. Sử dụng `z.infer<typeof schema>` cho các DTO.
- [ ] **Xử lý lỗi:** Lỗi phải được ném ra qua `AppError` với `statusCode` và `ERROR_CODE` chuẩn mực, không dùng `throw new Error()`. - [ ] **Xử lý lỗi:** Lỗi phải được ném ra qua `AppError` với `statusCode` và `ERROR_CODE` chuẩn mực, không dùng `throw new Error()`.
### C. Hiệu Năng & Cơ Sở Dữ Liệu (Performance & Database) ### C. Hiệu Năng & Cơ Sở Dữ Liệu (Performance & Database)
- [ ] **Tránh N+1 Query:** Khi truy vấn dữ liệu liên kết, phải sử dụng `include` hoặc `select` hợp lý thay vì gọi lặp lại trong vòng lặp `for`/`forEach`. - [ ] **Tránh N+1 Query:** Khi truy vấn dữ liệu liên kết, phải sử dụng `include` hoặc `select` hợp lý thay vì gọi lặp lại trong vòng lặp `for`/`forEach`.
- [ ] **Đúng chỉ mục (Index):** Các trường thường xuyên filter, sort (`status`, `createdAt`, `userId`, `domain`) phải có index trong `schema.prisma`. - [ ] **Đúng chỉ mục (Index):** Các trường thường xuyên filter, sort (`status`, `createdAt`, `userId`, `domain`) phải có index trong `schema.prisma`.
- [ ] **Xử lý Stream:** Các tác vụ export file dung lượng lớn bắt buộc phải dùng luồng (Stream) thay vì dồn toàn bộ vào RAM. - [ ] **Xử lý Stream:** Các tác vụ export file dung lượng lớn bắt buộc phải dùng luồng (Stream) thay vì dồn toàn bộ vào RAM.
### D. Kiểm Thử & Kiểm Định (Testing & Verification) ### D. Kiểm Thử & Kiểm Định (Testing & Verification)
- [ ] Đã bổ sung unit test tương ứng trong thư mục `__tests__/` liền kề. - [ ] Đã bổ sung unit test tương ứng trong thư mục `__tests__/` liền kề.
- [ ] Lệnh `pnpm lint` chạy không có cảnh báo nghiêm trọng hoặc lỗi cú pháp. - [ ] Lệnh `pnpm lint` chạy không có cảnh báo nghiêm trọng hoặc lỗi cú pháp.
- [ ] Lệnh `pnpm test -- --runInBand` chạy thành công 100%. - [ ] Lệnh `pnpm test -- --runInBand` chạy thành công 100%.
...@@ -40,6 +44,7 @@ Bạn là **Reviewer Sub-Agent** chịu trách nhiệm kiểm duyệt mọi thay ...@@ -40,6 +44,7 @@ Bạn là **Reviewer Sub-Agent** chịu trách nhiệm kiểm duyệt mọi thay
## 2. Tiêu Chuẩn Phản Hồi Khi Review ## 2. Tiêu Chuẩn Phản Hồi Khi Review
Khi đưa ra nhận xét, Reviewer phải phân loại theo 3 mức độ: Khi đưa ra nhận xét, Reviewer phải phân loại theo 3 mức độ:
1. 🔴 **[BLOCKER]**: Vi phạm nghiêm trọng kiến trúc (ví dụ: Service gọi Prisma), lỗ hổng bảo mật, làm gãy test. Yêu cầu sửa ngay lập tức. 1. 🔴 **[BLOCKER]**: Vi phạm nghiêm trọng kiến trúc (ví dụ: Service gọi Prisma), lỗ hổng bảo mật, làm gãy test. Yêu cầu sửa ngay lập tức.
2. 🟡 **[WARNING]**: Chưa tối ưu hiệu năng, thiếu test case biên hoặc chưa cập nhật Swagger. Cần cân nhắc xử lý. 2. 🟡 **[WARNING]**: Chưa tối ưu hiệu năng, thiếu test case biên hoặc chưa cập nhật Swagger. Cần cân nhắc xử lý.
3. 🟢 **[SUGGESTION]**: Góp ý làm gọn code, đặt tên biến rõ nghĩa hơn hoặc cải thiện comment. 3. 🟢 **[SUGGESTION]**: Góp ý làm gọn code, đặt tên biến rõ nghĩa hơn hoặc cải thiện comment.
.claude/rules/design.md
View file @ c168a47c
...@@ -46,10 +46,10 @@ Hệ thống áp dụng kiến trúc phân lớp hướng dịch vụ (Layered C ...@@ -46,10 +46,10 @@ Hệ thống áp dụng kiến trúc phân lớp hướng dịch vụ (Layered C
- Mọi dữ liệu đầu vào từ người dùng (Body, Query, Params) **bắt buộc** phải được định nghĩa Schema bằng **Zod** trong `<feature>.validation.ts`. - Mọi dữ liệu đầu vào từ người dùng (Body, Query, Params) **bắt buộc** phải được định nghĩa Schema bằng **Zod** trong `<feature>.validation.ts`.
- Sử dụng middleware dùng chung `validateMiddleware`: - Sử dụng middleware dùng chung `validateMiddleware`:
```typescript ```typescript
import { validate } from '../../middlewares/validate.middleware'; import { validate } from "../../middlewares/validate.middleware";
import { createCrawlJobSchema } from './crawl-job.validation'; import { createCrawlJobSchema } from "./crawl-job.validation";
router.post('/', validate(createCrawlJobSchema), crawlJobController.create); router.post("/", validate(createCrawlJobSchema), crawlJobController.create);
``` ```
- Định nghĩa kiểu TypeScript tương ứng (`DTO`) bằng `z.infer<typeof schema>` trong `<feature>.dto.ts`. - Định nghĩa kiểu TypeScript tương ứng (`DTO`) bằng `z.infer<typeof schema>` trong `<feature>.dto.ts`.
...@@ -60,11 +60,11 @@ Hệ thống áp dụng kiến trúc phân lớp hướng dịch vụ (Layered C ...@@ -60,11 +60,11 @@ Hệ thống áp dụng kiến trúc phân lớp hướng dịch vụ (Layered C
- Không dùng `throw new Error("...")` một cách tùy tiện. - Không dùng `throw new Error("...")` một cách tùy tiện.
- Bắt buộc kế thừa từ `AppError`: - Bắt buộc kế thừa từ `AppError`:
```typescript ```typescript
import { AppError } from '../../common/errors/app-error'; import { AppError } from "../../common/errors/app-error";
import { ERROR_CODE } from '../../common/errors/error-code'; import { ERROR_CODE } from "../../common/errors/error-code";
if (!job) { if (!job) {
throw new AppError('Crawl job not found', 404, ERROR_CODE.NOT_FOUND); throw new AppError("Crawl job not found", 404, ERROR_CODE.NOT_FOUND);
} }
``` ```
- Cấu trúc response trả về cho client luôn thống nhất: - Cấu trúc response trả về cho client luôn thống nhất:
......
.claude/rules/tech-defaults.md
View file @ c168a47c
...@@ -7,7 +7,7 @@ ...@@ -7,7 +7,7 @@
## 1. Môi Trường & Phiên Bản Chuẩn (Core Runtime) ## 1. Môi Trường & Phiên Bản Chuẩn (Core Runtime)
| Thành phần | Phiên bản / Thư viện | Ghi chú quy ước | | Thành phần | Phiên bản / Thư viện | Ghi chú quy ước |
| :--- | :--- | :--- | | :------------------ | :------------------- | :---------------------------------------------- |
| **Node.js** | `>= 20.x` | Sử dụng cú pháp ES2022+ hiện đại | | **Node.js** | `>= 20.x` | Sử dụng cú pháp ES2022+ hiện đại |
| **Package Manager** | `pnpm@9.15.0` | Không dùng npm hay yarn để tránh lệch pnpm-lock | | **Package Manager** | `pnpm@9.15.0` | Không dùng npm hay yarn để tránh lệch pnpm-lock |
| **Ngôn ngữ** | `TypeScript 5.7` | `strict: true`, không dùng kiểu `any` vô căn cứ | | **Ngôn ngữ** | `TypeScript 5.7` | `strict: true`, không dùng kiểu `any` vô căn cứ |
......
.claude/rules/workflow.md
View file @ c168a47c
...@@ -82,6 +82,7 @@ pnpm build ...@@ -82,6 +82,7 @@ pnpm build
## 5. Quy Trình Bổ Sung Queue Worker ## 5. Quy Trình Bổ Sung Queue Worker
Khi tạo thêm Worker xử lý tác vụ nền: Khi tạo thêm Worker xử lý tác vụ nền:
1. Tạo Queue tại `src/queues/<job-name>.queue.ts`. 1. Tạo Queue tại `src/queues/<job-name>.queue.ts`.
2. Tạo Processor xử lý logic tại `src/queues/<job-name>.worker.processor.ts`. 2. Tạo Processor xử lý logic tại `src/queues/<job-name>.worker.processor.ts`.
3. Khởi tạo Worker file tại `src/queues/<job-name>.worker.ts`. 3. Khởi tạo Worker file tại `src/queues/<job-name>.worker.ts`.
......
.claude/skills/shop-amazon.md
View file @ c168a47c
...@@ -31,6 +31,7 @@ Khi khởi tạo `CrawlJob` qua API `POST /api/v1/crawl-jobs`, sử dụng paylo ...@@ -31,6 +31,7 @@ Khi khởi tạo `CrawlJob` qua API `POST /api/v1/crawl-jobs`, sử dụng paylo
``` ```
> **Lưu ý chống chặn (Anti-bot):** > **Lưu ý chống chặn (Anti-bot):**
>
> - Luôn thiết lập `delayMs` tối thiểu `2000`ms để tránh bị hệ thống Amazon chặn IP hoặc hiển thị CAPTCHA. > - Luôn thiết lập `delayMs` tối thiểu `2000`ms để tránh bị hệ thống Amazon chặn IP hoặc hiển thị CAPTCHA.
> - Đảm bảo worker bắt cờ `CAPTCHA_DETECTED` và `BLOCKED` trong bảng `crawl_pages` để cảnh báo người dùng. > - Đảm bảo worker bắt cờ `CAPTCHA_DETECTED` và `BLOCKED` trong bảng `crawl_pages` để cảnh báo người dùng.
......
AGENTS.md
View file @ c168a47c
...@@ -13,7 +13,8 @@ Route → Controller → Service → Repository → Prisma Client → ...@@ -13,7 +13,8 @@ Route → Controller → Service → Repository → Prisma Client →
``` ```
### Quy tắc bất biến: ### Quy tắc bất biến:
- **Độc quyền Prisma:** Chỉ duy nhất các file `*.repository.ts` được phép import và gọi `prisma` hoặc `PrismaClient`. Service, Controller, Worker, Helper và Middleware **tuyệt đối không** được gọi Prisma trực tiếp.
- **Độc quyền Prisma:** Chỉ duy nhất các file `*.repository.ts` được phép import và gọi `prisma` hoặc `PrismaClient`. Service, Controller, Worker, Helper và Middleware **tuyệt đối không** được gọi Prisma trực tiếp. Đồng thời, các tầng ngoài Repository (DTO, Service, Controller, Middleware) **không import enum từ `@prisma/client`** (ví dụ: `UserRole`, `CrawlJobStatus`, `CrawlMode`, `ExportType`, `AssetType`), mà phải sử dụng types từ `src/common/constants/`.
- **Tổ chức Module chuẩn (`src/modules/<feature>/`):** - **Tổ chức Module chuẩn (`src/modules/<feature>/`):**
- `<feature>.route.ts`: Khai báo endpoints, gắn middleware (`auth`, `role`, `validate`, `rateLimit`). - `<feature>.route.ts`: Khai báo endpoints, gắn middleware (`auth`, `role`, `validate`, `rateLimit`).
- `<feature>.controller.ts`: Nhận HTTP request, trích xuất parameters, gọi Service, trả response HTTP chuẩn. - `<feature>.controller.ts`: Nhận HTTP request, trích xuất parameters, gọi Service, trả response HTTP chuẩn.
...@@ -23,18 +24,41 @@ Route → Controller → Service → Repository → Prisma Client → ...@@ -23,18 +24,41 @@ Route → Controller → Service → Repository → Prisma Client →
- `__tests__/`: Chứa colocated unit/integration tests cho module. - `__tests__/`: Chứa colocated unit/integration tests cho module.
- **Routing:** Mọi router module mới phải được mount tập trung trong `src/routes/index.ts` với prefix `/api/v1/`. - **Routing:** Mọi router module mới phải được mount tập trung trong `src/routes/index.ts` với prefix `/api/v1/`.
- **Mã dùng chung (`src/common/`):** Chỉ đặt vào `src/common/` (errors, helpers, constants, types, storage) khi code thực sự được tái sử dụng qua ít nhất 2 modules. - **Mã dùng chung (`src/common/`):** Chỉ đặt vào `src/common/` (errors, helpers, constants, types, storage) khi code thực sự được tái sử dụng qua ít nhất 2 modules.
- **Chuẩn Hóa Constants & Cấm Tuyệt Đối Hardcode (Zero Hardcode Principle):**
- **Tập trung tại `src/common/constants/`:** Mọi chuỗi trạng thái (`JOB_STATUS`), vai trò người dùng (`ROLES`), chế độ thu thập (`CRAWL_MODE`), tần suất lịch (`SCHEDULE_FREQUENCY`), kiểu xuất dữ liệu (`EXPORT_TYPE`), loại tài nguyên (`ASSET_TYPE`), múi giờ (`DEFAULT_TIMEZONE`), v.v. **bắt buộc** phải được định nghĩa trong `src/common/constants/*.constant.ts` dưới dạng object `as const` và export type `keyof typeof CONSTANT`. Barrel export tập trung tại `src/common/constants/index.ts`.
- **Cấm hardcode chuỗi / mảng trong code:** Tuyệt đối không viết trực tiếp string literal (ví dụ: `'COMPLETED'`, `'SCRAPE'`, `'ADMIN'`, `'Asia/Ho_Chi_Minh'`) trong Controllers, Services, Workers, DTOs, Repositories, Helpers hoặc Schemas. Luôn dùng `CONSTANT.KEY` hoặc `Object.values(CONSTANT)`.
- **Sử dụng trong Zod Validation (`*.validation.ts`):** Luôn dùng `z.nativeEnum(CONSTANT)` thay vì khai báo mảng chuỗi `z.enum(['VAL1', 'VAL2'])`.
- **Pattern chuẩn:**
```typescript
// 1. Khai báo (src/common/constants/crawl-mode.constant.ts):
export const CRAWL_MODE = {
SCRAPE: 'SCRAPE',
CRAWL: 'CRAWL',
SITEMAP: 'SITEMAP',
URL_LIST: 'URL_LIST',
} as const;
export type CrawlMode = keyof typeof CRAWL_MODE;
// 2. Validation (crawl-job.validation.ts):
mode: z.nativeEnum(CRAWL_MODE).optional().default(CRAWL_MODE.SCRAPE)
// 3. Logic nghiệp vụ (crawl-job.service.ts / crawl.worker.processor.ts):
if (job.mode === CRAWL_MODE.URL_LIST) { ... }
```
--- ---
## 2. Dữ Liệu & Tích Hợp (Data, Prisma & Workers) ## 2. Dữ Liệu & Tích Hợp (Data, Prisma & Workers)
### A. Cơ sở dữ liệu & Prisma Migrations ### A. Cơ sở dữ liệu & Prisma Migrations
- `prisma/schema.prisma` là nguồn chân lý duy nhất (Single Source of Truth) của database schema. - `prisma/schema.prisma` là nguồn chân lý duy nhất (Single Source of Truth) của database schema.
- Thao tác Prisma thông qua script runner: `node scripts/prisma-run.js <cmd>`. Runner tự động tổng hợp `DATABASE_URL` từ các biến môi trường cấu hình (`DB_HOST`, `DB_PORT`, `DB_USER`, `DB_PASSWORD`, `DB_NAME`, `DB_SSL`). - Thao tác Prisma thông qua script runner: `node scripts/prisma-run.js <cmd>`. Runner tự động tổng hợp `DATABASE_URL` từ các biến môi trường cấu hình (`DB_HOST`, `DB_PORT`, `DB_USER`, `DB_PASSWORD`, `DB_NAME`, `DB_SSL`).
- Mọi thay đổi schema phải sinh migration tương ứng bằng `pnpm db:migrate` và commit đồng thời cả `schema.prisma` lẫn thư mục migration. - Mọi thay đổi schema phải sinh migration tương ứng bằng `pnpm db:migrate` và commit đồng thời cả `schema.prisma` lẫn thư mục migration.
- **CẤM:** Không bao giờ chạy `pnpm db:migrate:reset` trừ khi người dùng yêu cầu rõ ràng việc xóa trắng dữ liệu. - **CẤM:** Không bao giờ chạy `pnpm db:migrate:reset` trừ khi người dùng yêu cầu rõ ràng việc xóa trắng dữ liệu.
### B. Hàng đợi bất đồng bộ & Worker (BullMQ + Redis) ### B. Hàng đợi bất đồng bộ & Worker (BullMQ + Redis)
- Các tác vụ nặng (thu thập web, gửi webhook, chạy lịch cron) phải chuyển qua hàng đợi BullMQ: - Các tác vụ nặng (thu thập web, gửi webhook, chạy lịch cron) phải chuyển qua hàng đợi BullMQ:
- `crawl.queue.ts` / `crawl.worker.ts` / `crawl.worker.processor.ts` - `crawl.queue.ts` / `crawl.worker.ts` / `crawl.worker.processor.ts`
- `webhook.queue.ts` / `webhook.worker.ts` - `webhook.queue.ts` / `webhook.worker.ts`
......
CRAWL_DATA_EXPRESS_TYPESCRIPT_PRISMA_MIGRATION_INIT.md
View file @ c168a47c
...@@ -583,10 +583,10 @@ src/database/prisma.client.ts ...@@ -583,10 +583,10 @@ src/database/prisma.client.ts
``` ```
```ts ```ts
import { PrismaClient } from '@prisma/client'; import { PrismaClient } from "@prisma/client";
export const prisma = new PrismaClient({ export const prisma = new PrismaClient({
log: ['error', 'warn'], log: ["error", "warn"],
}); });
``` ```
...@@ -594,9 +594,10 @@ Nếu cần log query khi development: ...@@ -594,9 +594,10 @@ Nếu cần log query khi development:
```ts ```ts
export const prisma = new PrismaClient({ export const prisma = new PrismaClient({
log: process.env.NODE_ENV === 'development' log:
? ['query', 'error', 'warn'] process.env.NODE_ENV === "development"
: ['error', 'warn'], ? ["query", "error", "warn"]
: ["error", "warn"],
}); });
``` ```
...@@ -628,17 +629,17 @@ Không để repository xử lý nghiệp vụ. ...@@ -628,17 +629,17 @@ Không để repository xử lý nghiệp vụ.
### 11.1. Route ### 11.1. Route
```ts ```ts
import { Router } from 'express'; import { Router } from "express";
import { CrawlJobController } from './crawl-job.controller'; import { CrawlJobController } from "./crawl-job.controller";
import { authMiddleware } from '../../middlewares/auth.middleware'; import { authMiddleware } from "../../middlewares/auth.middleware";
const router = Router(); const router = Router();
const controller = new CrawlJobController(); const controller = new CrawlJobController();
router.post('/', authMiddleware, controller.create); router.post("/", authMiddleware, controller.create);
router.get('/', authMiddleware, controller.findAll); router.get("/", authMiddleware, controller.findAll);
router.get('/:id', authMiddleware, controller.findById); router.get("/:id", authMiddleware, controller.findById);
router.post('/:id/cancel', authMiddleware, controller.cancel); router.post("/:id/cancel", authMiddleware, controller.cancel);
export default router; export default router;
``` ```
...@@ -648,8 +649,8 @@ export default router; ...@@ -648,8 +649,8 @@ export default router;
### 11.2. Controller ### 11.2. Controller
```ts ```ts
import { Request, Response, NextFunction } from 'express'; import { Request, Response, NextFunction } from "express";
import { CrawlJobService } from './crawl-job.service'; import { CrawlJobService } from "./crawl-job.service";
export class CrawlJobController { export class CrawlJobController {
private readonly service = new CrawlJobService(); private readonly service = new CrawlJobService();
...@@ -717,9 +718,9 @@ export class CrawlJobController { ...@@ -717,9 +718,9 @@ export class CrawlJobController {
### 11.3. Service ### 11.3. Service
```ts ```ts
import { CrawlJobRepository } from './crawl-job.repository'; import { CrawlJobRepository } from "./crawl-job.repository";
import { AppError } from '../../common/errors/app-error'; import { AppError } from "../../common/errors/app-error";
import { crawlQueue } from '../../queues/crawl.queue'; import { crawlQueue } from "../../queues/crawl.queue";
export class CrawlJobService { export class CrawlJobService {
private readonly repository = new CrawlJobRepository(); private readonly repository = new CrawlJobRepository();
...@@ -738,7 +739,7 @@ export class CrawlJobService { ...@@ -738,7 +739,7 @@ export class CrawlJobService {
maxDepth: payload.maxDepth, maxDepth: payload.maxDepth,
}); });
await crawlQueue.add('crawl-job', { await crawlQueue.add("crawl-job", {
jobId: job.id, jobId: job.id,
}); });
...@@ -753,7 +754,7 @@ export class CrawlJobService { ...@@ -753,7 +754,7 @@ export class CrawlJobService {
const job = await this.repository.findById(jobId); const job = await this.repository.findById(jobId);
if (!job || job.userId !== userId) { if (!job || job.userId !== userId) {
throw new AppError('Crawl job not found', 404); throw new AppError("Crawl job not found", 404);
} }
return job; return job;
...@@ -762,11 +763,11 @@ export class CrawlJobService { ...@@ -762,11 +763,11 @@ export class CrawlJobService {
async cancel(userId: string, jobId: string) { async cancel(userId: string, jobId: string) {
const job = await this.findById(userId, jobId); const job = await this.findById(userId, jobId);
if (job.status === 'COMPLETED') { if (job.status === "COMPLETED") {
throw new AppError('Completed job cannot be canceled', 400); throw new AppError("Completed job cannot be canceled", 400);
} }
return this.repository.updateStatus(jobId, 'CANCELED'); return this.repository.updateStatus(jobId, "CANCELED");
} }
} }
``` ```
...@@ -776,8 +777,8 @@ export class CrawlJobService { ...@@ -776,8 +777,8 @@ export class CrawlJobService {
### 11.4. Repository ### 11.4. Repository
```ts ```ts
import { prisma } from '../../database/prisma.client'; import { prisma } from "../../database/prisma.client";
import { CrawlJobStatus } from '@prisma/client'; import { CrawlJobStatus } from "@prisma/client";
export class CrawlJobRepository { export class CrawlJobRepository {
create(data: { create(data: {
...@@ -801,7 +802,7 @@ export class CrawlJobRepository { ...@@ -801,7 +802,7 @@ export class CrawlJobRepository {
findAllByUser(userId: string, query: any) { findAllByUser(userId: string, query: any) {
return prisma.crawlJob.findMany({ return prisma.crawlJob.findMany({
where: { userId }, where: { userId },
orderBy: { createdAt: 'desc' }, orderBy: { createdAt: "desc" },
include: { include: {
exports: true, exports: true,
}, },
...@@ -839,27 +840,27 @@ prisma/seed.ts ...@@ -839,27 +840,27 @@ prisma/seed.ts
``` ```
```ts ```ts
import { PrismaClient, UserRole } from '@prisma/client'; import { PrismaClient, UserRole } from "@prisma/client";
import bcrypt from 'bcryptjs'; import bcrypt from "bcryptjs";
const prisma = new PrismaClient(); const prisma = new PrismaClient();
async function main() { async function main() {
const passwordHash = await bcrypt.hash('Admin@123456', 10); const passwordHash = await bcrypt.hash("Admin@123456", 10);
await prisma.user.upsert({ await prisma.user.upsert({
where: { email: 'admin@crawl.local' }, where: { email: "admin@crawl.local" },
update: {}, update: {},
create: { create: {
email: 'admin@crawl.local', email: "admin@crawl.local",
passwordHash, passwordHash,
fullName: 'System Admin', fullName: "System Admin",
role: UserRole.ADMIN, role: UserRole.ADMIN,
isActive: true, isActive: true,
}, },
}); });
console.log('Seed completed'); console.log("Seed completed");
} }
main() main()
......
README.md
View file @ c168a47c
...@@ -5,7 +5,7 @@ Backend API cho hệ thống crawl dữ liệu web. Người dùng dán link, h ...@@ -5,7 +5,7 @@ Backend API cho hệ thống crawl dữ liệu web. Người dùng dán link, h
## Tech Stack ## Tech Stack
| Thành phần | Công nghệ | | Thành phần | Công nghệ |
| --------------- | -------------------------------------- | | --------------- | ------------------------------------- |
| Runtime | Node.js + TypeScript | | Runtime | Node.js + TypeScript |
| Framework | Express.js | | Framework | Express.js |
| ORM | Prisma (Code First Migration) | | ORM | Prisma (Code First Migration) |
...@@ -112,8 +112,8 @@ docker ps ...@@ -112,8 +112,8 @@ docker ps
Hai container cần chạy: Hai container cần chạy:
| Container | Service | Port | | Container | Service | Port |
| -------------------- | ---------- | ------ | | --------------------- | ---------- | ------ |
| `crawl_data_postgres`| PostgreSQL | `5432` | | `crawl_data_postgres` | PostgreSQL | `5432` |
| `crawl_data_redis` | Redis | `6379` | | `crawl_data_redis` | Redis | `6379` |
> DB name mặc định trong Docker là `crawl_data_db` — đảm bảo `DB_NAME` trong `.env` khớp với giá trị này. > DB name mặc định trong Docker là `crawl_data_db` — đảm bảo `DB_NAME` trong `.env` khớp với giá trị này.
...@@ -129,6 +129,7 @@ pnpm db:migrate:init ...@@ -129,6 +129,7 @@ pnpm db:migrate:init
``` ```
Lệnh này sẽ: Lệnh này sẽ:
1. Đọc `prisma/schema.prisma` 1. Đọc `prisma/schema.prisma`
2. Tạo folder migration đầu tiên trong `prisma/migrations/` 2. Tạo folder migration đầu tiên trong `prisma/migrations/`
3. Apply migration xuống PostgreSQL 3. Apply migration xuống PostgreSQL
...@@ -154,7 +155,7 @@ pnpm db:seed ...@@ -154,7 +155,7 @@ pnpm db:seed
Seed tạo 3 tài khoản mặc định để test: Seed tạo 3 tài khoản mặc định để test:
| Email | Password | Role | | Email | Password | Role |
| --------------------- | ---------------- | ------------- | | -------------------- | ---------------- | ------------ |
| `admin@crawl.local` | `Admin@123456` | ADMIN | | `admin@crawl.local` | `Admin@123456` | ADMIN |
| `crawl@crawl.local` | `Crawler@123456` | CRAWLER_USER | | `crawl@crawl.local` | `Crawler@123456` | CRAWLER_USER |
| `viewer@crawl.local` | `Viewer@123456` | VIEWER | | `viewer@crawl.local` | `Viewer@123456` | VIEWER |
...@@ -213,7 +214,7 @@ pnpm worker ...@@ -213,7 +214,7 @@ pnpm worker
## Các lệnh hữu ích ## Các lệnh hữu ích
| Lệnh | Mô tả | | Lệnh | Mô tả |
| -------------------------- | ---------------------------------------------------------- | | ------------------------ | --------------------------------------------------------- |
| `pnpm db:migrate:init` | Tạo migration lần đầu (`--name init`) | | `pnpm db:migrate:init` | Tạo migration lần đầu (`--name init`) |
| `pnpm db:migrate` | Tạo migration mới sau khi sửa `schema.prisma` | | `pnpm db:migrate` | Tạo migration mới sau khi sửa `schema.prisma` |
| `pnpm db:migrate:deploy` | Apply migration lên staging/production (không dùng dev) | | `pnpm db:migrate:deploy` | Apply migration lên staging/production (không dùng dev) |
...@@ -301,7 +302,7 @@ Hệ thống hỗ trợ chuẩn hóa dữ liệu đầu ra **Data Contract v1**, ...@@ -301,7 +302,7 @@ Hệ thống hỗ trợ chuẩn hóa dữ liệu đầu ra **Data Contract v1**,
### 8.1 Mô hình Hai Lớp Output (Clean vs. Raw Output Model) ### 8.1 Mô hình Hai Lớp Output (Clean vs. Raw Output Model)
| Lớp Output | Trường Dữ Liệu | Đặc Điểm & Mô Tả | Đối Tượng Sử Dụng | | Lớp Output | Trường Dữ Liệu | Đặc Điểm & Mô Tả | Đối Tượng Sử Dụng |
| :--- | :--- | :--- | :--- | | :--------------- | :------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------------------------------------------ |
| **Raw Output** | `rawMarkdown` | Nội dung Markdown thô nguyên bản thu thập từ crawler, giữ nguyên menu, header, sidebar và footer. | FE Debug / Reconstruct trang gốc | | **Raw Output** | `rawMarkdown` | Nội dung Markdown thô nguyên bản thu thập từ crawler, giữ nguyên menu, header, sidebar và footer. | FE Debug / Reconstruct trang gốc |
| **Clean Output** | `mainContent` | Thân bài chính đã qua thuật toán lọc nhiễu tự động (`extractMainContent`), loại bỏ menu nav, liên kết mạng xã hội, bài viết liên quan và copyright footer. Vẫn giữ cú pháp Markdown. | **AI Agent / LLM Prompt Context / RAG** | | **Clean Output** | `mainContent` | Thân bài chính đã qua thuật toán lọc nhiễu tự động (`extractMainContent`), loại bỏ menu nav, liên kết mạng xã hội, bài viết liên quan và copyright footer. Vẫn giữ cú pháp Markdown. | **AI Agent / LLM Prompt Context / RAG** |
| **Clean Text** | `cleanText` | Plain text thuần túy đã xóa sạch toàn bộ ký tự định dạng Markdown (`stripMarkdown`). | Đếm từ (`wordCount`) & Hash (`contentHash`) | | **Clean Text** | `cleanText` | Plain text thuần túy đã xóa sạch toàn bộ ký tự định dạng Markdown (`stripMarkdown`). | Đếm từ (`wordCount`) & Hash (`contentHash`) |
...@@ -321,6 +322,7 @@ Hệ thống hỗ trợ chuẩn hóa dữ liệu đầu ra **Data Contract v1**, ...@@ -321,6 +322,7 @@ Hệ thống hỗ trợ chuẩn hóa dữ liệu đầu ra **Data Contract v1**,
### 8.3 Chỉ số Chất lượng Dữ liệu & Cảnh báo (Quality Metrics & Warnings) ### 8.3 Chỉ số Chất lượng Dữ liệu & Cảnh báo (Quality Metrics & Warnings)
Mỗi bản ghi trang đã crawl trả về đầy đủ các trường đo lường chất lượng: Mỗi bản ghi trang đã crawl trả về đầy đủ các trường đo lường chất lượng:
- **`normalizedUrl`**: URL đã loại bỏ các tham số tracking (`utm_*`, `fbclid`, `gclid`), loại bỏ fragment và chuẩn hóa host/scheme để tránh trùng lặp. - **`normalizedUrl`**: URL đã loại bỏ các tham số tracking (`utm_*`, `fbclid`, `gclid`), loại bỏ fragment và chuẩn hóa host/scheme để tránh trùng lặp.
- **`wordCount`**: Số từ tính trên `cleanText`. - **`wordCount`**: Số từ tính trên `cleanText`.
- **`contentHash`**: Mã SHA-256 tính từ `cleanText` phục vụ deduplication trên Vector DB. - **`contentHash`**: Mã SHA-256 tính từ `cleanText` phục vụ deduplication trên Vector DB.
...@@ -348,5 +350,3 @@ export-job-c4b8e21a.zip ...@@ -348,5 +350,3 @@ export-job-c4b8e21a.zip
``` ```
Xem chi tiết Data Contract đầy đủ tại [DATA_CONTRACT_V1.md](docs/DATA_CONTRACT_V1.md). Xem chi tiết Data Contract đầy đủ tại [DATA_CONTRACT_V1.md](docs/DATA_CONTRACT_V1.md).
docker-compose.yml
View file @ c168a47c
version: '3.8' version: "3.8"
services: services:
postgres: postgres:
......
docs/STORAGE_S3_MINIO.md
View file @ c168a47c
...@@ -5,7 +5,7 @@ ...@@ -5,7 +5,7 @@
Storage được chọn bằng `STORAGE_DRIVER`: Storage được chọn bằng `STORAGE_DRIVER`:
| Cấu hình | Nơi lưu file | | Cấu hình | Nơi lưu file |
| --- | --- | | ---------------------- | ---------------------------------------------- |
| `STORAGE_DRIVER=local` | Lưu trực tiếp trong `STORAGE_EXPORT_DIR` | | `STORAGE_DRIVER=local` | Lưu trực tiếp trong `STORAGE_EXPORT_DIR` |
| `STORAGE_DRIVER=s3` | Dùng AWS S3, MinIO hoặc dịch vụ tương thích S3 | | `STORAGE_DRIVER=s3` | Dùng AWS S3, MinIO hoặc dịch vụ tương thích S3 |
......
docs/audits/latest-audit.md
View file @ c168a47c
...@@ -13,8 +13,9 @@ A comprehensive, end-to-end full project audit and autonomous remediation cycle ...@@ -13,8 +13,9 @@ A comprehensive, end-to-end full project audit and autonomous remediation cycle
All **29 findings** across security, authentication, database indexes, API contracts, memory efficiency, and lint quality were systematically evaluated, verified against active code, and resolved. All **29 findings** across security, authentication, database indexes, API contracts, memory efficiency, and lint quality were systematically evaluated, verified against active code, and resolved.
### Key Results ### Key Results
- **P0 Critical Findings**: 5 of 5 FIXED & VERIFIED - **P0 Critical Findings**: 5 of 5 FIXED & VERIFIED
- **P1 High Findings**: 6 of 6 Actionable FIXED & VERIFIED *(2 Redis infrastructure-dependent items documented)* - **P1 High Findings**: 6 of 6 Actionable FIXED & VERIFIED _(2 Redis infrastructure-dependent items documented)_
- **P2 Medium Findings**: 7 of 7 VERIFIED & RESOLVED (Fixed confirmed items, verified false-positives) - **P2 Medium Findings**: 7 of 7 VERIFIED & RESOLVED (Fixed confirmed items, verified false-positives)
- **P3 Low Findings**: 7 of 7 VERIFIED & RESOLVED (Added reusable UUID validator, refined changePassword schema, fixed unused vars, configured CORS preflight) - **P3 Low Findings**: 7 of 7 VERIFIED & RESOLVED (Added reusable UUID validator, refined changePassword schema, fixed unused vars, configured CORS preflight)
- **Compilation (`pnpm build`)**: ✅ 0 errors - **Compilation (`pnpm build`)**: ✅ 0 errors
...@@ -26,19 +27,19 @@ All **29 findings** across security, authentication, database indexes, API contr ...@@ -26,19 +27,19 @@ All **29 findings** across security, authentication, database indexes, API contr
## 2. Complete Findings Verification & Resolution Matrix ## 2. Complete Findings Verification & Resolution Matrix
| ID | Sev | Module | Description | Audit Verification | Remediation Status | | ID | Sev | Module | Description | Audit Verification | Remediation Status |
|---|---|---|---|---|---| | ----------- | --- | ------------ | ---------------------------------------------------- | ------------------ | ------------------------------------------------- |
| **BUG-001** | P0 | Config | JWT secrets hardcoded fallback default | CONFIRMED | **FIXED** (Fail-fast startup validation) | | **BUG-001** | P0 | Config | JWT secrets hardcoded fallback default | CONFIRMED | **FIXED** (Fail-fast startup validation) |
| **BUG-002** | P0 | Config | Webhook AES encryption key hardcoded in repo | CONFIRMED | **FIXED** (Fail-fast hex validation) | | **BUG-002** | P0 | Config | Webhook AES encryption key hardcoded in repo | CONFIRMED | **FIXED** (Fail-fast hex validation) |
| **BUG-003** | P0 | Auth | `resetPassword` decode-before-verify pattern | CONFIRMED | **FIXED** (Isolated `verifyResetToken`) | | **BUG-003** | P0 | Auth | `resetPassword` decode-before-verify pattern | CONFIRMED | **FIXED** (Isolated `verifyResetToken`) |
| **BUG-004** | P0 | CrawlJobs | `findByIdWithPages` unbounded page queries in worker | CONFIRMED | **FIXED** (Selective diff field projection) | | **BUG-004** | P0 | CrawlJobs | `findByIdWithPages` unbounded page queries in worker | CONFIRMED | **FIXED** (Selective diff field projection) |
| **BUG-005** | P0 | CrawlAssets | `GET /crawl-jobs/:id/assets` unbounded results | CONFIRMED | **FIXED** (Added skip/take pagination) | | **BUG-005** | P0 | CrawlAssets | `GET /crawl-jobs/:id/assets` unbounded results | CONFIRMED | **FIXED** (Added skip/take pagination) |
| **BUG-006** | P1 | Auth | `authMiddleware` DB lookup caching | CONFIRMED | *Documented for Redis cluster rollout* | | **BUG-006** | P1 | Auth | `authMiddleware` DB lookup caching | CONFIRMED | _Documented for Redis cluster rollout_ |
| **BUG-007** | P1 | API Keys | `apiKeyOrAuthMiddleware` 2 sequential DB queries | CONFIRMED | **FIXED** (Single query with relation join) | | **BUG-007** | P1 | API Keys | `apiKeyOrAuthMiddleware` 2 sequential DB queries | CONFIRMED | **FIXED** (Single query with relation join) |
| **BUG-008** | P1 | Auth | `forgotPassword` dev log leaks reset token | CONFIRMED | **FIXED** (Removed console logging of token) | | **BUG-008** | P1 | Auth | `forgotPassword` dev log leaks reset token | CONFIRMED | **FIXED** (Removed console logging of token) |
| **BUG-009** | P1 | CrawlJobs | SSE `streamEvents` interval poll & unbounded TTL | CONFIRMED | **FIXED** (30-min max TTL + 3s poll) | | **BUG-009** | P1 | CrawlJobs | SSE `streamEvents` interval poll & unbounded TTL | CONFIRMED | **FIXED** (30-min max TTL + 3s poll) |
| **BUG-010** | P1 | Auth | `authMiddleware` uses stale JWT role instead of DB | CONFIRMED | **FIXED** (Assigns fresh `user.role` from DB) | | **BUG-010** | P1 | Auth | `authMiddleware` uses stale JWT role instead of DB | CONFIRMED | **FIXED** (Assigns fresh `user.role` from DB) |
| **BUG-011** | P1 | App | CORS origin coupled to mail config | CONFIRMED | **FIXED** (Dedicated multi-origin whitelist) | | **BUG-011** | P1 | App | CORS origin coupled to mail config | CONFIRMED | **FIXED** (Dedicated multi-origin whitelist) |
| **BUG-012** | P1 | Security | Rate limiter MemoryStore in multi-instance | CONFIRMED | *Documented for Redis cluster rollout* | | **BUG-012** | P1 | Security | Rate limiter MemoryStore in multi-instance | CONFIRMED | _Documented for Redis cluster rollout_ |
| **BUG-013** | P1 | CrawlJobs | `GET /crawl-jobs/:id/diff` naked response envelope | CONFIRMED | **FIXED** (Wrapped in `{ success, data }`) | | **BUG-013** | P1 | CrawlJobs | `GET /crawl-jobs/:id/diff` naked response envelope | CONFIRMED | **FIXED** (Wrapped in `{ success, data }`) |
| **BUG-014** | P2 | Auth | Email verification token shares `accessSecret` | CONFIRMED | **FIXED** (Dedicated `emailVerificationSecret`) | | **BUG-014** | P2 | Auth | Email verification token shares `accessSecret` | CONFIRMED | **FIXED** (Dedicated `emailVerificationSecret`) |
| **BUG-015** | P2 | Auth | Register email enumeration timing oracle | FALSE_POSITIVE | **VERIFIED INTENTIONAL** (Explicit design tested) | | **BUG-015** | P2 | Auth | Register email enumeration timing oracle | FALSE_POSITIVE | **VERIFIED INTENTIONAL** (Explicit design tested) |
...@@ -62,6 +63,7 @@ All **29 findings** across security, authentication, database indexes, API contr ...@@ -62,6 +63,7 @@ All **29 findings** across security, authentication, database indexes, API contr
## 3. Detailed Summary of Changes ## 3. Detailed Summary of Changes
### Security & Authentication ### Security & Authentication
1. **JWT & Webhook Secret Validation**: Enforced startup checks requiring minimum 32-character strings for JWT secrets and 64-character hex strings for AES-256 webhook encryption keys in [`src/config/env.config.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/config/env.config.ts). 1. **JWT & Webhook Secret Validation**: Enforced startup checks requiring minimum 32-character strings for JWT secrets and 64-character hex strings for AES-256 webhook encryption keys in [`src/config/env.config.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/config/env.config.ts).
2. **Dedicated Email Verification Secret**: Isolated email verification signing via `jwtConfig.emailVerificationSecret` in [`src/modules/auth/auth.service.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/auth/auth.service.ts). 2. **Dedicated Email Verification Secret**: Isolated email verification signing via `jwtConfig.emailVerificationSecret` in [`src/modules/auth/auth.service.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/auth/auth.service.ts).
3. **Cryptographic Reset Password Flow**: Refactored `resetPassword` to execute cryptographic signature validation in `verifyResetToken()` before any state mutation or token revocation. 3. **Cryptographic Reset Password Flow**: Refactored `resetPassword` to execute cryptographic signature validation in `verifyResetToken()` before any state mutation or token revocation.
...@@ -69,12 +71,14 @@ All **29 findings** across security, authentication, database indexes, API contr ...@@ -69,12 +71,14 @@ All **29 findings** across security, authentication, database indexes, API contr
5. **CORS & Preflight Optimization**: Decoupled CORS from email configuration, supporting multi-origin whitelisting via `CORS_ALLOWED_ORIGINS` and preflight caching via `maxAge: 86400` in [`src/app.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/app.ts). 5. **CORS & Preflight Optimization**: Decoupled CORS from email configuration, supporting multi-origin whitelisting via `CORS_ALLOWED_ORIGINS` and preflight caching via `maxAge: 86400` in [`src/app.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/app.ts).
### Database & Concurrency Safety ### Database & Concurrency Safety
1. **Memory-Safe Diff Queries**: Replaced full page loading (`include: { pages: true }`) with selective field projections in [`src/modules/crawl-jobs/crawl-job.repository.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/crawl-jobs/crawl-job.repository.ts), preventing worker heap exhaustion on large jobs. 1. **Memory-Safe Diff Queries**: Replaced full page loading (`include: { pages: true }`) with selective field projections in [`src/modules/crawl-jobs/crawl-job.repository.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/crawl-jobs/crawl-job.repository.ts), preventing worker heap exhaustion on large jobs.
2. **Paginated Asset Retrieval**: Added `page` and `limit` (max 500) parameters with `skip`/`take` pagination to [`src/modules/crawl-assets/crawl-asset.repository.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/crawl-assets/crawl-asset.repository.ts). 2. **Paginated Asset Retrieval**: Added `page` and `limit` (max 500) parameters with `skip`/`take` pagination to [`src/modules/crawl-assets/crawl-asset.repository.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/crawl-assets/crawl-asset.repository.ts).
3. **Single-Query API Key Auth**: Joined the `user` relation in `ApiKeyRepository.findByHash`, eliminating an extra sequential query in [`src/middlewares/api-key.middleware.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/middlewares/api-key.middleware.ts). 3. **Single-Query API Key Auth**: Joined the `user` relation in `ApiKeyRepository.findByHash`, eliminating an extra sequential query in [`src/middlewares/api-key.middleware.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/middlewares/api-key.middleware.ts).
4. **Schema Indexing**: Added composite index `@@index([jobId, createdAt])` to `CrawlJobLog` and `@@index([ipAddress])` to `AuditLog` in [`prisma/schema.prisma`](file:///d:/NodeJS/DataCrawler/data-crawler-be/prisma/schema.prisma). 4. **Schema Indexing**: Added composite index `@@index([jobId, createdAt])` to `CrawlJobLog` and `@@index([ipAddress])` to `AuditLog` in [`prisma/schema.prisma`](file:///d:/NodeJS/DataCrawler/data-crawler-be/prisma/schema.prisma).
### API Contract & Validation ### API Contract & Validation
1. **Standardized Response Envelopes**: Wrapped `GET /crawl-jobs/:id/diff` in `{ success: true, data: diffReport }` in [`src/modules/crawl-jobs/crawl-job.controller.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/crawl-jobs/crawl-job.controller.ts). 1. **Standardized Response Envelopes**: Wrapped `GET /crawl-jobs/:id/diff` in `{ success: true, data: diffReport }` in [`src/modules/crawl-jobs/crawl-job.controller.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/crawl-jobs/crawl-job.controller.ts).
2. **Re-usable Path Params Validation**: Added `validateParams()` in [`src/middlewares/validate.middleware.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/middlewares/validate.middleware.ts). 2. **Re-usable Path Params Validation**: Added `validateParams()` in [`src/middlewares/validate.middleware.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/middlewares/validate.middleware.ts).
3. **Password Validation Refinement**: Added rule ensuring `newPassword !== currentPassword` in [`src/modules/auth/auth.validation.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/auth/auth.validation.ts). 3. **Password Validation Refinement**: Added rule ensuring `newPassword !== currentPassword` in [`src/modules/auth/auth.validation.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/auth/auth.validation.ts).
......
docs/postman/README.md
View file @ c168a47c
...@@ -7,6 +7,7 @@ Tài liệu này hướng dẫn cách vận hành, cấu trúc và danh sách c ...@@ -7,6 +7,7 @@ Tài liệu này hướng dẫn cách vận hành, cấu trúc và danh sách c
## 🚀 1. Cách chạy kiểm thử ## 🚀 1. Cách chạy kiểm thử
### Cách 1: Chạy trực tiếp bằng Postman (GUI) ### Cách 1: Chạy trực tiếp bằng Postman (GUI)
1. **Import vào Postman**: 1. **Import vào Postman**:
- File Collection: [data-crawler.postman_collection.json](./data-crawler.postman_collection.json) - File Collection: [data-crawler.postman_collection.json](./data-crawler.postman_collection.json)
- File Environment: [data-crawler.postman_environment.json](./data-crawler.postman_environment.json) - File Environment: [data-crawler.postman_environment.json](./data-crawler.postman_environment.json)
...@@ -17,6 +18,7 @@ Tài liệu này hướng dẫn cách vận hành, cấu trúc và danh sách c ...@@ -17,6 +18,7 @@ Tài liệu này hướng dẫn cách vận hành, cấu trúc và danh sách c
- Nhấn **Run Data Crawler BE API**. - Nhấn **Run Data Crawler BE API**.
### Cách 2: Chạy tự động bằng Newman (CLI) ### Cách 2: Chạy tự động bằng Newman (CLI)
Yêu cầu đã khởi chạy Server Backend tại `http://localhost:3000`. Chạy lệnh sau tại thư mục gốc của dự án: Yêu cầu đã khởi chạy Server Backend tại `http://localhost:3000`. Chạy lệnh sau tại thư mục gốc của dự án:
```bash ```bash
...@@ -24,6 +26,7 @@ npx newman run "docs/postman/data-crawler.postman_collection.json" -e "docs/post ...@@ -24,6 +26,7 @@ npx newman run "docs/postman/data-crawler.postman_collection.json" -e "docs/post
``` ```
Chạy từng folder kịch bản cụ thể: Chạy từng folder kịch bản cụ thể:
```bash ```bash
npx newman run "docs/postman/data-crawler.postman_collection.json" -e "docs/postman/data-crawler.postman_environment.json" --folder "Crawl Jobs" --reporters cli npx newman run "docs/postman/data-crawler.postman_collection.json" -e "docs/postman/data-crawler.postman_environment.json" --folder "Crawl Jobs" --reporters cli
``` ```
...@@ -33,72 +36,75 @@ npx newman run "docs/postman/data-crawler.postman_collection.json" -e "docs/post ...@@ -33,72 +36,75 @@ npx newman run "docs/postman/data-crawler.postman_collection.json" -e "docs/post
## 📋 2. Chi tiết các Nhóm Kiểm thử (Test Suites) ## 📋 2. Chi tiết các Nhóm Kiểm thử (Test Suites)
### 🔐 A. Nhóm Auth (Xác thực & Ủy quyền) ### 🔐 A. Nhóm Auth (Xác thực & Ủy quyền)
Kiểm tra luồng đăng nhập, lấy thông tin cá nhân, cập nhật tài khoản và cơ chế refresh token. Kiểm tra luồng đăng nhập, lấy thông tin cá nhân, cập nhật tài khoản và cơ chế refresh token.
1. **Login**: 1. **Login**:
- *Endpoint*: `POST /auth/login` - _Endpoint_: `POST /auth/login`
- *Test Assertions*: Trả về HTTP 200, `success: true`, sinh ra `accessToken` và `refreshToken`, tự động lưu vào môi trường Postman (`token`, `refreshToken`). - _Test Assertions_: Trả về HTTP 200, `success: true`, sinh ra `accessToken` và `refreshToken`, tự động lưu vào môi trường Postman (`token`, `refreshToken`).
2. **Get Me**: 2. **Get Me**:
- *Endpoint*: `GET /auth/me` - _Endpoint_: `GET /auth/me`
- *Test Assertions*: Đính kèm Bearer token. Trả về chính xác thông tin User (`id`, `email`, `role`, `isActive`). - _Test Assertions_: Đính kèm Bearer token. Trả về chính xác thông tin User (`id`, `email`, `role`, `isActive`).
3. **Refresh Token**: 3. **Refresh Token**:
- *Endpoint*: `POST /auth/refresh` - _Endpoint_: `POST /auth/refresh`
- *Test Assertions*: Nhận `refreshToken`, cấp lại `accessToken` mới, cập nhật lại biến môi trường. - _Test Assertions_: Nhận `refreshToken`, cấp lại `accessToken` mới, cập nhật lại biến môi trường.
4. **Logout**: 4. **Logout**:
- *Endpoint*: `POST /auth/logout` - _Endpoint_: `POST /auth/logout`
- *Test Assertions*: Thu hồi token trong database, xóa khỏi biến môi trường Postman. - _Test Assertions_: Thu hồi token trong database, xóa khỏi biến môi trường Postman.
5. **Login with Invalid Credentials (Edge Case)**: 5. **Login with Invalid Credentials (Edge Case)**:
- *Test Assertions*: Trả về `401 Unauthorized`. - _Test Assertions_: Trả về `401 Unauthorized`.
6. **Get Me without Token (Edge Case)**: 6. **Get Me without Token (Edge Case)**:
- *Test Assertions*: Trả về `401 Unauthorized`. - _Test Assertions_: Trả về `401 Unauthorized`.
--- ---
### ⚙️ B. Nhóm Crawl Jobs & Clean/Raw Preview ### ⚙️ B. Nhóm Crawl Jobs & Clean/Raw Preview
Kiểm tra các hoạt động tạo tác vụ crawl, lấy danh sách trang, xem trước dữ liệu sạch/thô và lọc chất lượng. Kiểm tra các hoạt động tạo tác vụ crawl, lấy danh sách trang, xem trước dữ liệu sạch/thô và lọc chất lượng.
1. **Create Crawl Job**: 1. **Create Crawl Job**:
- *Endpoint*: `POST /crawl-jobs` - _Endpoint_: `POST /crawl-jobs`
- *Body Payload*: `{ "startUrl": "https://example.com", "mode": "CRAWL", "maxPages": 50, "maxDepth": 2 }` - _Body Payload_: `{ "startUrl": "https://example.com", "mode": "CRAWL", "maxPages": 50, "maxDepth": 2 }`
- *Test Assertions*: HTTP 201 Created, trả về job ID mới (`job_id`). - _Test Assertions_: HTTP 201 Created, trả về job ID mới (`job_id`).
2. **Get Crawl Jobs (Paginated & Filtered)**: 2. **Get Crawl Jobs (Paginated & Filtered)**:
- *Endpoint*: `GET /crawl-jobs?status=PENDING&page=1&limit=10` - _Endpoint_: `GET /crawl-jobs?status=PENDING&page=1&limit=10`
- *Test Assertions*: Trả về danh sách phân trang `{ items: Array, meta: { total, page, limit, totalPages } }`. - _Test Assertions_: Trả về danh sách phân trang `{ items: Array, meta: { total, page, limit, totalPages } }`.
3. **Get Crawl Job by ID**: 3. **Get Crawl Job by ID**:
- *Endpoint*: `GET /crawl-jobs/:id` - _Endpoint_: `GET /crawl-jobs/:id`
- *Test Assertions*: Trả về chi tiết các thông số của Job (`startUrl`, `mode`, `status`, `totalPages`, `successPages`, `failedPages`). - _Test Assertions_: Trả về chi tiết các thông số của Job (`startUrl`, `mode`, `status`, `totalPages`, `successPages`, `failedPages`).
4. **Get Crawled Pages (Metadata List)**: 4. **Get Crawled Pages (Metadata List)**:
- *Endpoint*: `GET /crawl-jobs/:id/pages` - _Endpoint_: `GET /crawl-jobs/:id/pages`
- *Query Params*: Supports `status`, `statusCode`, `search`, `dataQualityScore`, `hasTables`, `hasImages`, `hasLinks`, `wordCount`, `sortBy`, `order`. - _Query Params_: Supports `status`, `statusCode`, `search`, `dataQualityScore`, `hasTables`, `hasImages`, `hasLinks`, `wordCount`, `sortBy`, `order`.
- *Test Assertions*: Trả về mảng danh sách trang kèm theo `normalizedUrl`, `dataQualityScore`, `wordCount`, `warnings`, `hasSensitiveData` (không chứa payload markdown dài để tối ưu băng thông). - _Test Assertions_: Trả về mảng danh sách trang kèm theo `normalizedUrl`, `dataQualityScore`, `wordCount`, `warnings`, `hasSensitiveData` (không chứa payload markdown dài để tối ưu băng thông).
5. **Get Crawled Pages Preview (Clean vs. Raw Output)**: 5. **Get Crawled Pages Preview (Clean vs. Raw Output)**:
- *Endpoint*: `GET /crawl-jobs/:id/pages/preview?minQualityScore=50` (hoặc `GET /crawl-jobs/:id/pages?preview=true`) - _Endpoint_: `GET /crawl-jobs/:id/pages/preview?minQualityScore=50` (hoặc `GET /crawl-jobs/:id/pages?preview=true`)
- *Test Assertions*: - _Test Assertions_:
- Phải chứa đủ 3 trường nội dung đại diện cho hai lớp Output: - Phải chứa đủ 3 trường nội dung đại diện cho hai lớp Output:
- `rawMarkdown`: Markdown thô nguyên bản thu thập được. - `rawMarkdown`: Markdown thô nguyên bản thu thập được.
- `mainContent`: Thân bài chính đã qua lọc bỏ nhiễu nav/footer/sidebar **(Khuyến nghị cho AI Agents / LLM)**. - `mainContent`: Thân bài chính đã qua lọc bỏ nhiễu nav/footer/sidebar **(Khuyến nghị cho AI Agents / LLM)**.
- `cleanText`: Văn bản thuần túy đã xóa sạch ký tự định dạng Markdown. - `cleanText`: Văn bản thuần túy đã xóa sạch ký tự định dạng Markdown.
- Hỗ trợ kiểm tra các chỉ số chất lượng: `dataQualityScore` (0-100), `wordCount`, `contentHash` (SHA-256), `warnings` (`NAV_NOISE`, `TOO_SHORT`, `DUPLICATE_CONTENT`, ...). - Hỗ trợ kiểm tra các chỉ số chất lượng: `dataQualityScore` (0-100), `wordCount`, `contentHash` (SHA-256), `warnings` (`NAV_NOISE`, `TOO_SHORT`, `DUPLICATE_CONTENT`, ...).
6. **Get Job Assets**: 6. **Get Job Assets**:
- *Endpoint*: `GET /crawl-jobs/:id/assets?assetType=IMAGE` - _Endpoint_: `GET /crawl-jobs/:id/assets?assetType=IMAGE`
- *Query Params*: `assetType` (enum: `IMAGE`, `LINK`, `PDF`, `FILE`, `VIDEO`, `OTHER`). - _Query Params_: `assetType` (enum: `IMAGE`, `LINK`, `PDF`, `FILE`, `VIDEO`, `OTHER`).
- *Test Assertions*: Trả về danh sách tài nguyên hình ảnh/liên kết thu thập được từ các trang. - _Test Assertions_: Trả về danh sách tài nguyên hình ảnh/liên kết thu thập được từ các trang.
--- ---
### 💾 C. Nhóm Export & Download ### 💾 C. Nhóm Export & Download
Kiểm tra luồng khởi tạo và tải về các tập tin xuất bản cho Crawl Job đã hoàn thành (`COMPLETED`). Kiểm tra luồng khởi tạo và tải về các tập tin xuất bản cho Crawl Job đã hoàn thành (`COMPLETED`).
1. **Get Crawl Job Exports**: 1. **Get Crawl Job Exports**:
- *Endpoint*: `GET /crawl-jobs/:id/exports` - _Endpoint_: `GET /crawl-jobs/:id/exports`
- *Test Assertions*: Trả về danh sách các tệp tin xuất bản đã tạo của Job. - _Test Assertions_: Trả về danh sách các tệp tin xuất bản đã tạo của Job.
2. **Create Export for Job**: 2. **Create Export for Job**:
- *Endpoint*: `POST /crawl-jobs/:id/exports` - _Endpoint_: `POST /crawl-jobs/:id/exports`
- *Body Payload*: `{ "exportType": "ZIP" }` (Các định dạng hỗ trợ: `JSON`, `CSV`, `XLSX`, `MARKDOWN`, `ZIP`). - _Body Payload_: `{ "exportType": "ZIP" }` (Các định dạng hỗ trợ: `JSON`, `CSV`, `XLSX`, `MARKDOWN`, `ZIP`).
- *Test Assertions*: HTTP 201 Created, khởi tạo bản export thành công và lưu `export_id`. - _Test Assertions_: HTTP 201 Created, khởi tạo bản export thành công và lưu `export_id`.
3. **Download Export File**: 3. **Download Export File**:
- *Endpoint*: `GET /exports/:exportId/download` (hoặc `GET /crawl-jobs/:id/download`) - _Endpoint_: `GET /exports/:exportId/download` (hoặc `GET /crawl-jobs/:id/download`)
- *Test Assertions*: Trả về stream binary tệp tin kèm theo đúng Header `Content-Disposition`. - _Test Assertions_: Trả về stream binary tệp tin kèm theo đúng Header `Content-Disposition`.
- **Đặc quyền cấu trúc ZIP Output**: - **Đặc quyền cấu trúc ZIP Output**:
- Thư mục `/data/raw/pages.raw.json` & `/data/clean/pages.clean.json`. - Thư mục `/data/raw/pages.raw.json` & `/data/clean/pages.clean.json`.
- Thư mục `/markdown/raw/` & `/markdown/clean/`. - Thư mục `/markdown/raw/` & `/markdown/clean/`.
...@@ -107,6 +113,7 @@ Kiểm tra luồng khởi tạo và tải về các tập tin xuất bản cho C ...@@ -107,6 +113,7 @@ Kiểm tra luồng khởi tạo và tải về các tập tin xuất bản cho C
--- ---
### 🛡️ D. Nhóm Permission & Edge Case Tests (Phân quyền & Lỗi nghiệp vụ) ### 🛡️ D. Nhóm Permission & Edge Case Tests (Phân quyền & Lỗi nghiệp vụ)
1. **Get Non-Existent Job**: 1. **Get Non-Existent Job**:
- Gửi ID UUID không tồn tại -> Kiểm tra phản hồi `404 Not Found` và `code: "CRAWL_JOB_NOT_FOUND"`. - Gửi ID UUID không tồn tại -> Kiểm tra phản hồi `404 Not Found` và `code: "CRAWL_JOB_NOT_FOUND"`.
2. **Cancel Completed Job**: 2. **Cancel Completed Job**:
......
docs/postman/data-crawler.postman_collection.json
View file @ c168a47c
...@@ -769,15 +769,8 @@ ...@@ -769,15 +769,8 @@
], ],
"url": { "url": {
"raw": "{{base_url}}/crawl-jobs/:id/pages/preview?minQualityScore=50", "raw": "{{base_url}}/crawl-jobs/:id/pages/preview?minQualityScore=50",
"host": [ "host": ["{{base_url}}"],
"{{base_url}}" "path": ["crawl-jobs", ":id", "pages", "preview"],
],
"path": [
"crawl-jobs",
":id",
"pages",
"preview"
],
"query": [ "query": [
{ {
"key": "minQualityScore", "key": "minQualityScore",
...@@ -825,14 +818,8 @@ ...@@ -825,14 +818,8 @@
], ],
"url": { "url": {
"raw": "{{base_url}}/crawl-jobs/:id/assets?assetType=IMAGE", "raw": "{{base_url}}/crawl-jobs/:id/assets?assetType=IMAGE",
"host": [ "host": ["{{base_url}}"],
"{{base_url}}" "path": ["crawl-jobs", ":id", "assets"],
],
"path": [
"crawl-jobs",
":id",
"assets"
],
"query": [ "query": [
{ {
"key": "assetType", "key": "assetType",
...@@ -1311,10 +1298,21 @@ ...@@ -1311,10 +1298,21 @@
"request": { "request": {
"method": "POST", "method": "POST",
"header": [ "header": [
{ "key": "Authorization", "value": "Bearer {{token}}", "type": "text" }, {
{ "key": "Content-Type", "value": "application/json", "type": "text" } "key": "Authorization",
"value": "Bearer {{token}}",
"type": "text"
},
{
"key": "Content-Type",
"value": "application/json",
"type": "text"
}
], ],
"body": { "mode": "raw", "raw": "{\n \"exportType\": \"ZIP\"\n}" }, "body": {
"mode": "raw",
"raw": "{\n \"exportType\": \"ZIP\"\n}"
},
"url": { "url": {
"raw": "{{base_url}}/crawl-jobs/:id/exports", "raw": "{{base_url}}/crawl-jobs/:id/exports",
"host": ["{{base_url}}"], "host": ["{{base_url}}"],
...@@ -1341,7 +1339,13 @@ ...@@ -1341,7 +1339,13 @@
], ],
"request": { "request": {
"method": "GET", "method": "GET",
"header": [{ "key": "Authorization", "value": "Bearer {{token}}", "type": "text" }], "header": [
{
"key": "Authorization",
"value": "Bearer {{token}}",
"type": "text"
}
],
"url": { "url": {
"raw": "{{base_url}}/exports/:exportId/download", "raw": "{{base_url}}/exports/:exportId/download",
"host": ["{{base_url}}"], "host": ["{{base_url}}"],
...@@ -1370,10 +1374,21 @@ ...@@ -1370,10 +1374,21 @@
"request": { "request": {
"method": "POST", "method": "POST",
"header": [ "header": [
{ "key": "Authorization", "value": "Bearer {{token}}", "type": "text" }, {
{ "key": "Content-Type", "value": "application/json", "type": "text" } "key": "Authorization",
"value": "Bearer {{token}}",
"type": "text"
},
{
"key": "Content-Type",
"value": "application/json",
"type": "text"
}
], ],
"body": { "mode": "raw", "raw": "{\n \"exportType\": \"JSON\"\n}" }, "body": {
"mode": "raw",
"raw": "{\n \"exportType\": \"JSON\"\n}"
},
"url": { "url": {
"raw": "{{base_url}}/crawl-jobs/:id/exports", "raw": "{{base_url}}/crawl-jobs/:id/exports",
"host": ["{{base_url}}"], "host": ["{{base_url}}"],
...@@ -1400,7 +1415,13 @@ ...@@ -1400,7 +1415,13 @@
], ],
"request": { "request": {
"method": "GET", "method": "GET",
"header": [{ "key": "Authorization", "value": "Bearer {{token}}", "type": "text" }], "header": [
{
"key": "Authorization",
"value": "Bearer {{token}}",
"type": "text"
}
],
"url": { "url": {
"raw": "{{base_url}}/exports/:exportId/download", "raw": "{{base_url}}/exports/:exportId/download",
"host": ["{{base_url}}"], "host": ["{{base_url}}"],
...@@ -1429,10 +1450,21 @@ ...@@ -1429,10 +1450,21 @@
"request": { "request": {
"method": "POST", "method": "POST",
"header": [ "header": [
{ "key": "Authorization", "value": "Bearer {{token}}", "type": "text" }, {
{ "key": "Content-Type", "value": "application/json", "type": "text" } "key": "Authorization",
"value": "Bearer {{token}}",
"type": "text"
},
{
"key": "Content-Type",
"value": "application/json",
"type": "text"
}
], ],
"body": { "mode": "raw", "raw": "{\n \"exportType\": \"CSV\"\n}" }, "body": {
"mode": "raw",
"raw": "{\n \"exportType\": \"CSV\"\n}"
},
"url": { "url": {
"raw": "{{base_url}}/crawl-jobs/:id/exports", "raw": "{{base_url}}/crawl-jobs/:id/exports",
"host": ["{{base_url}}"], "host": ["{{base_url}}"],
...@@ -1459,7 +1491,13 @@ ...@@ -1459,7 +1491,13 @@
], ],
"request": { "request": {
"method": "GET", "method": "GET",
"header": [{ "key": "Authorization", "value": "Bearer {{token}}", "type": "text" }], "header": [
{
"key": "Authorization",
"value": "Bearer {{token}}",
"type": "text"
}
],
"url": { "url": {
"raw": "{{base_url}}/exports/:exportId/download", "raw": "{{base_url}}/exports/:exportId/download",
"host": ["{{base_url}}"], "host": ["{{base_url}}"],
...@@ -1488,10 +1526,21 @@ ...@@ -1488,10 +1526,21 @@
"request": { "request": {
"method": "POST", "method": "POST",
"header": [ "header": [
{ "key": "Authorization", "value": "Bearer {{token}}", "type": "text" }, {
{ "key": "Content-Type", "value": "application/json", "type": "text" } "key": "Authorization",
"value": "Bearer {{token}}",
"type": "text"
},
{
"key": "Content-Type",
"value": "application/json",
"type": "text"
}
], ],
"body": { "mode": "raw", "raw": "{\n \"exportType\": \"XLSX\"\n}" }, "body": {
"mode": "raw",
"raw": "{\n \"exportType\": \"XLSX\"\n}"
},
"url": { "url": {
"raw": "{{base_url}}/crawl-jobs/:id/exports", "raw": "{{base_url}}/crawl-jobs/:id/exports",
"host": ["{{base_url}}"], "host": ["{{base_url}}"],
...@@ -1518,7 +1567,13 @@ ...@@ -1518,7 +1567,13 @@
], ],
"request": { "request": {
"method": "GET", "method": "GET",
"header": [{ "key": "Authorization", "value": "Bearer {{token}}", "type": "text" }], "header": [
{
"key": "Authorization",
"value": "Bearer {{token}}",
"type": "text"
}
],
"url": { "url": {
"raw": "{{base_url}}/exports/:exportId/download", "raw": "{{base_url}}/exports/:exportId/download",
"host": ["{{base_url}}"], "host": ["{{base_url}}"],
...@@ -1547,10 +1602,21 @@ ...@@ -1547,10 +1602,21 @@
"request": { "request": {
"method": "POST", "method": "POST",
"header": [ "header": [
{ "key": "Authorization", "value": "Bearer {{token}}", "type": "text" }, {
{ "key": "Content-Type", "value": "application/json", "type": "text" } "key": "Authorization",
"value": "Bearer {{token}}",
"type": "text"
},
{
"key": "Content-Type",
"value": "application/json",
"type": "text"
}
], ],
"body": { "mode": "raw", "raw": "{\n \"exportType\": \"MARKDOWN\"\n}" }, "body": {
"mode": "raw",
"raw": "{\n \"exportType\": \"MARKDOWN\"\n}"
},
"url": { "url": {
"raw": "{{base_url}}/crawl-jobs/:id/exports", "raw": "{{base_url}}/crawl-jobs/:id/exports",
"host": ["{{base_url}}"], "host": ["{{base_url}}"],
...@@ -1576,7 +1642,13 @@ ...@@ -1576,7 +1642,13 @@
], ],
"request": { "request": {
"method": "GET", "method": "GET",
"header": [{ "key": "Authorization", "value": "Bearer {{token}}", "type": "text" }], "header": [
{
"key": "Authorization",
"value": "Bearer {{token}}",
"type": "text"
}
],
"url": { "url": {
"raw": "{{base_url}}/exports/:exportId/download", "raw": "{{base_url}}/exports/:exportId/download",
"host": ["{{base_url}}"], "host": ["{{base_url}}"],
......
eslint.config.js
View file @ c168a47c
import tsParser from '@typescript-eslint/parser'; import tsParser from "@typescript-eslint/parser";
import tsPlugin from '@typescript-eslint/eslint-plugin'; import tsPlugin from "@typescript-eslint/eslint-plugin";
export default [ export default [
{ {
ignores: ['dist/**', 'node_modules/**', 'eslint.config.js'], ignores: ["dist/**", "node_modules/**", "eslint.config.js"],
}, },
{ {
files: ['src/**/*.ts'], files: ["src/**/*.ts"],
languageOptions: { languageOptions: {
parser: tsParser, parser: tsParser,
parserOptions: { parserOptions: {
ecmaVersion: 'latest', ecmaVersion: "latest",
sourceType: 'module', sourceType: "module",
}, },
}, },
plugins: { plugins: {
'@typescript-eslint': tsPlugin, "@typescript-eslint": tsPlugin,
}, },
rules: { rules: {
'no-unused-vars': 'off', "no-unused-vars": "off",
'@typescript-eslint/no-unused-vars': [ "@typescript-eslint/no-unused-vars": [
'warn', "warn",
{ argsIgnorePattern: '^_', varsIgnorePattern: '^_' }, { argsIgnorePattern: "^_", varsIgnorePattern: "^_" },
], ],
'no-console': 'off', "no-console": "off",
'@typescript-eslint/no-explicit-any': 'warn', "@typescript-eslint/no-explicit-any": "warn",
}, },
}, },
]; ];
jest.config.js
View file @ c168a47c
module.exports = { module.exports = {
preset: 'ts-jest', preset: "ts-jest",
testEnvironment: 'node', testEnvironment: "node",
testMatch: ['**/*.test.ts'], testMatch: ["**/*.test.ts"],
moduleFileExtensions: ['ts', 'js', 'json'], moduleFileExtensions: ["ts", "js", "json"],
moduleNameMapper: { moduleNameMapper: {
'node-html-parser': '<rootDir>/src/__mocks__/node-html-parser.ts', "node-html-parser": "<rootDir>/src/__mocks__/node-html-parser.ts",
}, },
modulePathIgnorePatterns: ['<rootDir>/dist/'], modulePathIgnorePatterns: ["<rootDir>/dist/"],
setupFiles: ['<rootDir>/jest.setup.ts'], setupFiles: ["<rootDir>/jest.setup.ts"],
}; };
jest.setup.ts
View file @ c168a47c
process.env.JWT_ACCESS_SECRET = process.env.JWT_ACCESS_SECRET || 'test-access-secret-for-jest-must-be-32-chars-long'; process.env.JWT_ACCESS_SECRET =
process.env.JWT_REFRESH_SECRET = process.env.JWT_REFRESH_SECRET || 'test-refresh-secret-for-jest-must-be-32-chars-long'; process.env.JWT_ACCESS_SECRET ||
process.env.WEBHOOK_ENCRYPTION_KEY = process.env.WEBHOOK_ENCRYPTION_KEY || 'abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890'; "test-access-secret-for-jest-must-be-32-chars-long";
process.env.JWT_REFRESH_SECRET =
process.env.JWT_REFRESH_SECRET ||
"test-refresh-secret-for-jest-must-be-32-chars-long";
process.env.WEBHOOK_ENCRYPTION_KEY =
process.env.WEBHOOK_ENCRYPTION_KEY ||
"abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890";
pnpm-lock.yaml
View file @ c168a47c
This diff is collapsed.
prisma/seed.ts
View file @ c168a47c
import { PrismaClient, UserRole } from '@prisma/client'; import { PrismaClient, UserRole } from "@prisma/client";
import bcrypt from 'bcryptjs'; import bcrypt from "bcryptjs";
const prisma = new PrismaClient(); const prisma = new PrismaClient();
async function main() { async function main() {
const adminPasswordHash = await bcrypt.hash('Admin@123456', 10); const adminPasswordHash = await bcrypt.hash("Admin@123456", 10);
const crawlerPasswordHash = await bcrypt.hash('Crawler@123456', 10); const crawlerPasswordHash = await bcrypt.hash("Crawler@123456", 10);
const viewerPasswordHash = await bcrypt.hash('Viewer@123456', 10); const viewerPasswordHash = await bcrypt.hash("Viewer@123456", 10);
await prisma.user.upsert({ await prisma.user.upsert({
where: { email: 'admin@crawl.local' }, where: { email: "admin@crawl.local" },
update: {}, update: {},
create: { create: {
email: 'admin@crawl.local', email: "admin@crawl.local",
passwordHash: adminPasswordHash, passwordHash: adminPasswordHash,
fullName: 'System Admin', fullName: "System Admin",
role: UserRole.ADMIN, role: UserRole.ADMIN,
isActive: true, isActive: true,
}, },
}); });
await prisma.user.upsert({ await prisma.user.upsert({
where: { email: 'crawl@crawl.local' }, where: { email: "crawl@crawl.local" },
update: {}, update: {},
create: { create: {
email: 'crawl@crawl.local', email: "crawl@crawl.local",
passwordHash: crawlerPasswordHash, passwordHash: crawlerPasswordHash,
fullName: 'Crawl User', fullName: "Crawl User",
role: UserRole.CRAWLER_USER, role: UserRole.CRAWLER_USER,
isActive: true, isActive: true,
maxPagesLimit: 100, maxPagesLimit: 100,
...@@ -36,12 +36,12 @@ async function main() { ...@@ -36,12 +36,12 @@ async function main() {
}); });
await prisma.user.upsert({ await prisma.user.upsert({
where: { email: 'viewer@crawl.local' }, where: { email: "viewer@crawl.local" },
update: {}, update: {},
create: { create: {
email: 'viewer@crawl.local', email: "viewer@crawl.local",
passwordHash: viewerPasswordHash, passwordHash: viewerPasswordHash,
fullName: 'Viewer User', fullName: "Viewer User",
role: UserRole.VIEWER, role: UserRole.VIEWER,
isActive: true, isActive: true,
maxPagesLimit: 20, maxPagesLimit: 20,
...@@ -50,7 +50,7 @@ async function main() { ...@@ -50,7 +50,7 @@ async function main() {
}, },
}); });
console.log('Seed completed'); console.log("Seed completed");
} }
main() main()
......
scripts/prisma-run.js
View file @ c168a47c
require('dotenv').config(); require("dotenv").config();
const { spawn } = require('child_process'); const { spawn } = require("child_process");
if (!process.env.DATABASE_URL) { if (!process.env.DATABASE_URL) {
const password = encodeURIComponent(process.env.DB_PASSWORD || ''); const password = encodeURIComponent(process.env.DB_PASSWORD || "");
const user = encodeURIComponent(process.env.DB_USER || 'postgres'); const user = encodeURIComponent(process.env.DB_USER || "postgres");
const host = process.env.DB_HOST || 'localhost'; const host = process.env.DB_HOST || "localhost";
const port = process.env.DB_PORT || '5432'; const port = process.env.DB_PORT || "5432";
const name = process.env.DB_NAME || 'datacrawler'; const name = process.env.DB_NAME || "datacrawler";
const isSupabase = host.includes('supabase.co') || host.includes('pooler.supabase.com'); const isSupabase =
const ssl = process.env.DB_SSL === 'true' || isSupabase ? '&sslmode=require' : ''; host.includes("supabase.co") || host.includes("pooler.supabase.com");
const ssl =
process.env.DB_SSL === "true" || isSupabase ? "&sslmode=require" : "";
process.env.DATABASE_URL = `postgresql://${user}:${password}@${host}:${port}/${name}?schema=public${ssl}`; process.env.DATABASE_URL = `postgresql://${user}:${password}@${host}:${port}/${name}?schema=public${ssl}`;
} }
const args = process.argv.slice(2); const args = process.argv.slice(2);
const cmd = process.platform === 'win32' ? 'npx.cmd' : 'npx'; const cmd = process.platform === "win32" ? "npx.cmd" : "npx";
const child = spawn(cmd, ['prisma', ...args], { const child = spawn(cmd, ["prisma", ...args], {
stdio: 'inherit', stdio: "inherit",
env: process.env, env: process.env,
shell: true, shell: true,
}); });
child.on('exit', (code) => process.exit(code ?? 1)); child.on("exit", (code) => process.exit(code ?? 1));
src/app.ts
View file @ c168a47c
import express from 'express'; import express from "express";
import cors from 'cors'; import cors from "cors";
import helmet from 'helmet'; import helmet from "helmet";
import morgan from 'morgan'; import morgan from "morgan";
import cookieParser from 'cookie-parser'; import cookieParser from "cookie-parser";
import swaggerUi from 'swagger-ui-express'; import swaggerUi from "swagger-ui-express";
import { errorMiddleware, notFoundMiddleware } from './middlewares/error.middleware'; import {
import routes from './routes'; errorMiddleware,
import swaggerDocument from './docs/swagger.json'; notFoundMiddleware,
import healthRoute from './modules/health/health.route'; } from "./middlewares/error.middleware";
import { rateLimitMiddleware } from './middlewares/rate-limit.middleware'; import routes from "./routes";
import { envConfig } from './config/env.config'; import swaggerDocument from "./docs/swagger.json";
import { parseTrustProxy } from './common/helpers/proxy.helper'; import healthRoute from "./modules/health/health.route";
import { rateLimitMiddleware } from "./middlewares/rate-limit.middleware";
import { envConfig } from "./config/env.config";
import { parseTrustProxy } from "./common/helpers/proxy.helper";
const app = express(); const app = express();
app.set('trust proxy', parseTrustProxy(envConfig.trustProxy)); app.set("trust proxy", parseTrustProxy(envConfig.trustProxy));
app.use( app.use(
helmet({ helmet({
...@@ -27,7 +30,7 @@ app.use( ...@@ -27,7 +30,7 @@ app.use(
if (!origin) return callback(null, true); if (!origin) return callback(null, true);
if ( if (
envConfig.cors.allowedOrigins.includes(origin) || envConfig.cors.allowedOrigins.includes(origin) ||
envConfig.cors.allowedOrigins.includes('*') envConfig.cors.allowedOrigins.includes("*")
) { ) {
return callback(null, true); return callback(null, true);
} }
...@@ -37,17 +40,16 @@ app.use( ...@@ -37,17 +40,16 @@ app.use(
maxAge: 86400, maxAge: 86400,
}), }),
); );
app.use(morgan('dev')); app.use(morgan("dev"));
app.use(cookieParser()); app.use(cookieParser());
app.use(express.json()); app.use(express.json());
app.use(express.urlencoded({ extended: true })); app.use(express.urlencoded({ extended: true }));
app.use('/health', healthRoute); app.use("/health", healthRoute);
app.use('/api-docs', swaggerUi.serve, swaggerUi.setup(swaggerDocument)); app.use("/api-docs", swaggerUi.serve, swaggerUi.setup(swaggerDocument));
app.use('/api/v1', rateLimitMiddleware, routes); app.use("/api/v1", rateLimitMiddleware, routes);
app.use(notFoundMiddleware); app.use(notFoundMiddleware);
app.use(errorMiddleware); app.use(errorMiddleware);
export default app; export default app;
src/common/constants/asset-type.constant.ts 0 → 100644
View file @ c168a47c
export const ASSET_TYPE = {
IMAGE: "IMAGE",
LINK: "LINK",
PDF: "PDF",
FILE: "FILE",
VIDEO: "VIDEO",
OTHER: "OTHER",
} as const;
export const ASSET_TYPES = ASSET_TYPE;
export type AssetType = keyof typeof ASSET_TYPE;
src/common/constants/audit-action.constant.ts
View file @ c168a47c
export const AUDIT_ACTIONS = { export const AUDIT_ACTIONS = {
LOGIN: 'LOGIN', LOGIN: "LOGIN",
REGISTER: 'REGISTER', REGISTER: "REGISTER",
LOGOUT: 'LOGOUT', LOGOUT: "LOGOUT",
UPDATE_ME: 'UPDATE_ME', UPDATE_ME: "UPDATE_ME",
CHANGE_PASSWORD: 'CHANGE_PASSWORD', CHANGE_PASSWORD: "CHANGE_PASSWORD",
CREATE_JOB: 'CREATE_JOB', CREATE_JOB: "CREATE_JOB",
CANCEL_JOB: 'CANCEL_JOB', CANCEL_JOB: "CANCEL_JOB",
DOWNLOAD_EXPORT: 'DOWNLOAD_EXPORT', DOWNLOAD_EXPORT: "DOWNLOAD_EXPORT",
ADMIN_CREATE_USER: 'ADMIN_CREATE_USER', ADMIN_CREATE_USER: "ADMIN_CREATE_USER",
ADMIN_UPDATE_USER: 'ADMIN_UPDATE_USER', ADMIN_UPDATE_USER: "ADMIN_UPDATE_USER",
ADMIN_DELETE_USER: 'ADMIN_DELETE_USER', ADMIN_DELETE_USER: "ADMIN_DELETE_USER",
FORGOT_PASSWORD: 'FORGOT_PASSWORD', FORGOT_PASSWORD: "FORGOT_PASSWORD",
RESET_PASSWORD: 'RESET_PASSWORD', RESET_PASSWORD: "RESET_PASSWORD",
VERIFY_EMAIL: 'VERIFY_EMAIL', VERIFY_EMAIL: "VERIFY_EMAIL",
RESEND_VERIFICATION: 'RESEND_VERIFICATION', RESEND_VERIFICATION: "RESEND_VERIFICATION",
CREATE_API_KEY: 'CREATE_API_KEY', CREATE_API_KEY: "CREATE_API_KEY",
UPDATE_API_KEY_STATUS: 'UPDATE_API_KEY_STATUS', UPDATE_API_KEY_STATUS: "UPDATE_API_KEY_STATUS",
REVOKE_API_KEY: 'REVOKE_API_KEY', REVOKE_API_KEY: "REVOKE_API_KEY",
CREATE_WEBHOOK_CONFIG: 'CREATE_WEBHOOK_CONFIG', CREATE_WEBHOOK_CONFIG: "CREATE_WEBHOOK_CONFIG",
DELETE_WEBHOOK_CONFIG: 'DELETE_WEBHOOK_CONFIG', DELETE_WEBHOOK_CONFIG: "DELETE_WEBHOOK_CONFIG",
REDELIVER_WEBHOOK: 'REDELIVER_WEBHOOK', REDELIVER_WEBHOOK: "REDELIVER_WEBHOOK",
} as const; } as const;
export type AuditAction = typeof AUDIT_ACTIONS[keyof typeof AUDIT_ACTIONS]; export type AuditAction = (typeof AUDIT_ACTIONS)[keyof typeof AUDIT_ACTIONS];
src/common/constants/crawl-mode.constant.ts 0 → 100644
View file @ c168a47c
export const CRAWL_MODE = {
SCRAPE: "SCRAPE",
CRAWL: "CRAWL",
SITEMAP: "SITEMAP",
URL_LIST: "URL_LIST",
} as const;
export const CRAWL_MODES = CRAWL_MODE;
export type CrawlMode = keyof typeof CRAWL_MODE;
src/common/constants/data-contract.constant.ts
View file @ c168a47c
...@@ -4,7 +4,7 @@ ...@@ -4,7 +4,7 @@
*/ */
/** Phiên bản hiện tại của Data Contract schema */ /** Phiên bản hiện tại của Data Contract schema */
export const DATA_CONTRACT_SCHEMA_VERSION = '1.0.0'; export const DATA_CONTRACT_SCHEMA_VERSION = "1.0.0";
/** Ngưỡng số từ tối thiểu; dưới ngưỡng này page bị gắn cảnh báo TOO_SHORT */ /** Ngưỡng số từ tối thiểu; dưới ngưỡng này page bị gắn cảnh báo TOO_SHORT */
export const DATA_QUALITY_MIN_WORD_COUNT = 50; export const DATA_QUALITY_MIN_WORD_COUNT = 50;
...@@ -16,4 +16,4 @@ export const DATA_QUALITY_MIN_SCORE = 30; ...@@ -16,4 +16,4 @@ export const DATA_QUALITY_MIN_SCORE = 30;
* Thuật toán hash dùng để tạo contentHash cho nội dung page. * Thuật toán hash dùng để tạo contentHash cho nội dung page.
* Dùng 'sha256' để đảm bảo đủ entropy cho việc phát hiện duplicate. * Dùng 'sha256' để đảm bảo đủ entropy cho việc phát hiện duplicate.
*/ */
export const DATA_CONTRACT_HASH_ALGORITHM = 'sha256'; export const DATA_CONTRACT_HASH_ALGORITHM = "sha256";
src/common/constants/export-type.constant.ts
View file @ c168a47c
export const EXPORT_TYPE = { export const EXPORT_TYPE = {
JSON: 'JSON', JSON: "JSON",
CSV: 'CSV', CSV: "CSV",
XLSX: 'XLSX', XLSX: "XLSX",
MARKDOWN: 'MARKDOWN', MARKDOWN: "MARKDOWN",
ZIP: 'ZIP', ZIP: "ZIP",
} as const; } as const;
export type ExportType = keyof typeof EXPORT_TYPE; export type ExportType = keyof typeof EXPORT_TYPE;
export const EXPORT_MIME_TYPES: Record<ExportType, string> = { export const EXPORT_MIME_TYPES: Record<ExportType, string> = {
JSON: 'application/json', JSON: "application/json",
CSV: 'text/csv', CSV: "text/csv",
XLSX: 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet', XLSX: "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
// MARKDOWN export tạo file .zip chứa nhiều file markdown — nên dùng application/zip // MARKDOWN export tạo file .zip chứa nhiều file markdown — nên dùng application/zip
MARKDOWN: 'application/zip', MARKDOWN: "application/zip",
ZIP: 'application/zip', ZIP: "application/zip",
}; };
src/common/constants/index.ts 0 → 100644
View file @ c168a47c
export * from "./role.constant";
export * from "./job-status.constant";
export * from "./export-type.constant";
export * from "./audit-action.constant";
export * from "./storage-path.constant";
export * from "./data-contract.constant";
export * from "./timezone.constant";
export * from "./crawl-mode.constant";
export * from "./schedule-frequency.constant";
export * from "./asset-type.constant";
src/common/constants/job-status.constant.ts
View file @ c168a47c
export const JOB_STATUS = { export const JOB_STATUS = {
PENDING: 'PENDING', PENDING: "PENDING",
QUEUED: 'QUEUED', QUEUED: "QUEUED",
RUNNING: 'RUNNING', RUNNING: "RUNNING",
PROCESSING_EXPORT: 'PROCESSING_EXPORT', PROCESSING_EXPORT: "PROCESSING_EXPORT",
COMPLETED: 'COMPLETED', COMPLETED: "COMPLETED",
FAILED: 'FAILED', FAILED: "FAILED",
CANCELED: 'CANCELED', CANCELED: "CANCELED",
EXPIRED: 'EXPIRED', EXPIRED: "EXPIRED",
} as const; } as const;
export type JobStatus = keyof typeof JOB_STATUS; export type JobStatus = keyof typeof JOB_STATUS;
src/common/constants/role.constant.ts
View file @ c168a47c
export const ROLES = { export const ROLES = {
ADMIN: 'ADMIN', ADMIN: "ADMIN",
CRAWLER_USER: 'CRAWLER_USER', CRAWLER_USER: "CRAWLER_USER",
VIEWER: 'VIEWER', VIEWER: "VIEWER",
} as const; } as const;
export type Role = keyof typeof ROLES; export type Role = keyof typeof ROLES;
src/common/constants/schedule-frequency.constant.ts 0 → 100644
View file @ c168a47c
export const SCHEDULE_FREQUENCY = {
DAILY: "DAILY",
WEEKLY: "WEEKLY",
MONTHLY: "MONTHLY",
CUSTOM: "CUSTOM",
} as const;
export const SCHEDULE_FREQUENCIES = SCHEDULE_FREQUENCY;
export type ScheduleFrequency = keyof typeof SCHEDULE_FREQUENCY;
src/common/constants/storage-path.constant.ts
View file @ c168a47c
export const JOB_EXPORT_SUBDIRS = { export const JOB_EXPORT_SUBDIRS = {
DATA: 'data', DATA: "data",
DATA_RAW: 'data/raw', DATA_RAW: "data/raw",
DATA_CLEAN: 'data/clean', DATA_CLEAN: "data/clean",
MARKDOWN: 'markdown', MARKDOWN: "markdown",
MARKDOWN_RAW: 'markdown/raw', MARKDOWN_RAW: "markdown/raw",
MARKDOWN_CLEAN: 'markdown/clean', MARKDOWN_CLEAN: "markdown/clean",
RAW: 'raw', RAW: "raw",
LOGS: 'logs', LOGS: "logs",
} as const; } as const;
export const JOB_EXPORT_FILES = { export const JOB_EXPORT_FILES = {
METADATA: 'metadata.json', METADATA: "metadata.json",
SUMMARY: 'summary.json', SUMMARY: "summary.json",
DATA_QUALITY_JSON: 'data_quality.json', DATA_QUALITY_JSON: "data_quality.json",
PAGES_JSON: 'pages.json', PAGES_JSON: "pages.json",
PAGES_RAW_JSON: 'pages.raw.json', PAGES_RAW_JSON: "pages.raw.json",
PAGES_CLEAN_JSON: 'pages.clean.json', PAGES_CLEAN_JSON: "pages.clean.json",
PAGES_CSV: 'pages.csv', PAGES_CSV: "pages.csv",
LINKS_CSV: 'links.csv', LINKS_CSV: "links.csv",
IMAGES_CSV: 'images.csv', IMAGES_CSV: "images.csv",
PAGES_XLSX: 'pages.xlsx', PAGES_XLSX: "pages.xlsx",
TABLES_XLSX: 'tables.xlsx', TABLES_XLSX: "tables.xlsx",
ERRORS_JSON: 'errors.json', ERRORS_JSON: "errors.json",
CRAWL_LOG: 'crawl-log.txt', CRAWL_LOG: "crawl-log.txt",
STRUCTURED_JSON: 'structured.json', STRUCTURED_JSON: "structured.json",
DIFF_REPORT_JSON: 'diff_report.json', DIFF_REPORT_JSON: "diff_report.json",
} as const; } as const;
export function buildCrawlResultZipName(jobId: string): string { export function buildCrawlResultZipName(jobId: string): string {
...@@ -36,5 +36,5 @@ export function buildCrawlResultZipKey(jobId: string): string { ...@@ -36,5 +36,5 @@ export function buildCrawlResultZipKey(jobId: string): string {
} }
export function buildMarkdownZipName(jobId: string): string { export function buildMarkdownZipName(jobId: string): string {
return 'markdown.zip'; return "markdown.zip";
} }
src/common/constants/timezone.constant.ts 0 → 100644
View file @ c168a47c
export const DEFAULT_TIMEZONE = "Asia/Ho_Chi_Minh"; // Vietnam UTC+7
src/common/errors/app-error.ts
View file @ c168a47c
import { ErrorCode } from './error-code'; import { ErrorCode } from "./error-code";
export class AppError extends Error { export class AppError extends Error {
public readonly statusCode: number; public readonly statusCode: number;
...@@ -6,7 +6,12 @@ export class AppError extends Error { ...@@ -6,7 +6,12 @@ export class AppError extends Error {
public readonly details?: unknown; public readonly details?: unknown;
public readonly isOperational: boolean; public readonly isOperational: boolean;
constructor(message: string, statusCode: number = 500, code?: ErrorCode, details?: unknown) { constructor(
message: string,
statusCode: number = 500,
code?: ErrorCode,
details?: unknown,
) {
super(message); super(message);
this.statusCode = statusCode; this.statusCode = statusCode;
this.code = code; this.code = code;
...@@ -19,4 +24,3 @@ export class AppError extends Error { ...@@ -19,4 +24,3 @@ export class AppError extends Error {
} }
} }
} }
src/common/errors/error-code.ts
View file @ c168a47c
export const ERROR_CODE = { export const ERROR_CODE = {
UNAUTHORIZED: 'UNAUTHORIZED', UNAUTHORIZED: "UNAUTHORIZED",
FORBIDDEN: 'FORBIDDEN', FORBIDDEN: "FORBIDDEN",
NOT_FOUND: 'NOT_FOUND', NOT_FOUND: "NOT_FOUND",
VALIDATION_ERROR: 'VALIDATION_ERROR', VALIDATION_ERROR: "VALIDATION_ERROR",
INTERNAL_SERVER_ERROR: 'INTERNAL_SERVER_ERROR', INTERNAL_SERVER_ERROR: "INTERNAL_SERVER_ERROR",
INVALID_CREDENTIALS: 'INVALID_CREDENTIALS', INVALID_CREDENTIALS: "INVALID_CREDENTIALS",
USER_INACTIVE: 'USER_INACTIVE', USER_INACTIVE: "USER_INACTIVE",
TOKEN_EXPIRED: 'TOKEN_EXPIRED', TOKEN_EXPIRED: "TOKEN_EXPIRED",
TOKEN_INVALID: 'TOKEN_INVALID', TOKEN_INVALID: "TOKEN_INVALID",
DUPLICATE_ENTRY: 'DUPLICATE_ENTRY', DUPLICATE_ENTRY: "DUPLICATE_ENTRY",
MAIL_DELIVERY_FAILED: 'MAIL_DELIVERY_FAILED', MAIL_DELIVERY_FAILED: "MAIL_DELIVERY_FAILED",
CRAWL_JOB_NOT_FOUND: 'CRAWL_JOB_NOT_FOUND', CRAWL_JOB_NOT_FOUND: "CRAWL_JOB_NOT_FOUND",
CRAWL_JOB_ALREADY_COMPLETED: 'CRAWL_JOB_ALREADY_COMPLETED', CRAWL_JOB_ALREADY_COMPLETED: "CRAWL_JOB_ALREADY_COMPLETED",
CRAWL_JOB_NOT_COMPLETED: 'CRAWL_JOB_NOT_COMPLETED', CRAWL_JOB_NOT_COMPLETED: "CRAWL_JOB_NOT_COMPLETED",
PRIVATE_IP_BLOCKED: 'PRIVATE_IP_BLOCKED', PRIVATE_IP_BLOCKED: "PRIVATE_IP_BLOCKED",
INVALID_URL: 'INVALID_URL', INVALID_URL: "INVALID_URL",
QUOTA_MAX_PAGES_EXCEEDED: 'QUOTA_MAX_PAGES_EXCEEDED', QUOTA_MAX_PAGES_EXCEEDED: "QUOTA_MAX_PAGES_EXCEEDED",
QUOTA_JOBS_PER_DAY_EXCEEDED: 'QUOTA_JOBS_PER_DAY_EXCEEDED', QUOTA_JOBS_PER_DAY_EXCEEDED: "QUOTA_JOBS_PER_DAY_EXCEEDED",
QUOTA_CONCURRENT_JOBS_EXCEEDED: 'QUOTA_CONCURRENT_JOBS_EXCEEDED', QUOTA_CONCURRENT_JOBS_EXCEEDED: "QUOTA_CONCURRENT_JOBS_EXCEEDED",
EXPORT_NOT_FOUND: 'EXPORT_NOT_FOUND', EXPORT_NOT_FOUND: "EXPORT_NOT_FOUND",
EXPORT_FILE_MISSING: 'EXPORT_FILE_MISSING', EXPORT_FILE_MISSING: "EXPORT_FILE_MISSING",
UNSUPPORTED_EXPORT_TYPE: 'UNSUPPORTED_EXPORT_TYPE', UNSUPPORTED_EXPORT_TYPE: "UNSUPPORTED_EXPORT_TYPE",
API_KEY_INVALID: 'API_KEY_INVALID', API_KEY_INVALID: "API_KEY_INVALID",
API_KEY_EXPIRED: 'API_KEY_EXPIRED', API_KEY_EXPIRED: "API_KEY_EXPIRED",
WEBHOOK_CONFIG_NOT_FOUND: 'WEBHOOK_CONFIG_NOT_FOUND', WEBHOOK_CONFIG_NOT_FOUND: "WEBHOOK_CONFIG_NOT_FOUND",
CRAWL_SCHEDULE_NOT_FOUND: 'CRAWL_SCHEDULE_NOT_FOUND', CRAWL_SCHEDULE_NOT_FOUND: "CRAWL_SCHEDULE_NOT_FOUND",
DIFF_REPORT_NOT_FOUND: 'DIFF_REPORT_NOT_FOUND', DIFF_REPORT_NOT_FOUND: "DIFF_REPORT_NOT_FOUND",
} as const; } as const;
export type ErrorCode = keyof typeof ERROR_CODE; export type ErrorCode = keyof typeof ERROR_CODE;
src/common/helpers/__tests__/schedule-calculator.helper.test.ts
View file @ c168a47c
...@@ -4,41 +4,41 @@ import { ...@@ -4,41 +4,41 @@ import {
getTimezoneOffsetMinutes, getTimezoneOffsetMinutes,
getZonedDateParts, getZonedDateParts,
DEFAULT_TIMEZONE, DEFAULT_TIMEZONE,
} from '../schedule-calculator.helper'; } from "../schedule-calculator.helper";
describe('schedule-calculator.helper', () => { describe("schedule-calculator.helper", () => {
describe('isValidCronExpression', () => { describe("isValidCronExpression", () => {
it('returns true for valid standard cron expressions', () => { it("returns true for valid standard cron expressions", () => {
expect(isValidCronExpression('* * * * *')).toBe(true); expect(isValidCronExpression("* * * * *")).toBe(true);
expect(isValidCronExpression('0 0 * * *')).toBe(true); expect(isValidCronExpression("0 0 * * *")).toBe(true);
expect(isValidCronExpression('*/15 0-23 * * *')).toBe(true); expect(isValidCronExpression("*/15 0-23 * * *")).toBe(true);
expect(isValidCronExpression('0 9 1,15 * 1-5')).toBe(true); expect(isValidCronExpression("0 9 1,15 * 1-5")).toBe(true);
expect(isValidCronExpression('30 4 1 * 0')).toBe(true); expect(isValidCronExpression("30 4 1 * 0")).toBe(true);
}); });
it('returns false for invalid cron expressions', () => { it("returns false for invalid cron expressions", () => {
expect(isValidCronExpression('')).toBe(false); expect(isValidCronExpression("")).toBe(false);
expect(isValidCronExpression('invalid')).toBe(false); expect(isValidCronExpression("invalid")).toBe(false);
expect(isValidCronExpression('0 0 * *')).toBe(false); // 4 parts expect(isValidCronExpression("0 0 * *")).toBe(false); // 4 parts
expect(isValidCronExpression('0 0 * * * *')).toBe(false); // 6 parts expect(isValidCronExpression("0 0 * * * *")).toBe(false); // 6 parts
expect(isValidCronExpression('60 * * * *')).toBe(false); // invalid minute expect(isValidCronExpression("60 * * * *")).toBe(false); // invalid minute
expect(isValidCronExpression('* 25 * * *')).toBe(false); // invalid hour expect(isValidCronExpression("* 25 * * *")).toBe(false); // invalid hour
expect(isValidCronExpression('* * 32 * *')).toBe(false); // invalid dom expect(isValidCronExpression("* * 32 * *")).toBe(false); // invalid dom
expect(isValidCronExpression('* * * 13 *')).toBe(false); // invalid month expect(isValidCronExpression("* * * 13 *")).toBe(false); // invalid month
expect(isValidCronExpression('* * * * 8')).toBe(false); // invalid dow expect(isValidCronExpression("* * * * 8")).toBe(false); // invalid dow
}); });
}); });
describe('Vietnam timezone (UTC+7) calculations', () => { describe("Vietnam timezone (UTC+7) calculations", () => {
it('returns +420 minutes offset for Asia/Ho_Chi_Minh', () => { it("returns +420 minutes offset for Asia/Ho_Chi_Minh", () => {
expect(getTimezoneOffsetMinutes('Asia/Ho_Chi_Minh')).toBe(420); expect(getTimezoneOffsetMinutes("Asia/Ho_Chi_Minh")).toBe(420);
expect(DEFAULT_TIMEZONE).toBe('Asia/Ho_Chi_Minh'); expect(DEFAULT_TIMEZONE).toBe("Asia/Ho_Chi_Minh");
}); });
it('extracts Vietnam zoned date parts correctly', () => { it("extracts Vietnam zoned date parts correctly", () => {
// 12:15 UTC is 19:15 Vietnam time (+7 hours) // 12:15 UTC is 19:15 Vietnam time (+7 hours)
const utcDate = new Date('2026-09-02T12:15:00.000Z'); const utcDate = new Date("2026-09-02T12:15:00.000Z");
const parts = getZonedDateParts(utcDate, 'Asia/Ho_Chi_Minh'); const parts = getZonedDateParts(utcDate, "Asia/Ho_Chi_Minh");
expect(parts.year).toBe(2026); expect(parts.year).toBe(2026);
expect(parts.month).toBe(8); // Sept (0-indexed) expect(parts.month).toBe(8); // Sept (0-indexed)
...@@ -48,107 +48,107 @@ describe('schedule-calculator.helper', () => { ...@@ -48,107 +48,107 @@ describe('schedule-calculator.helper', () => {
expect(parts.dayOfWeek).toBe(3); // Wednesday expect(parts.dayOfWeek).toBe(3); // Wednesday
}); });
it('calculates DAILY crawl at 20:00 VN time (same day in VN)', () => { it("calculates DAILY crawl at 20:00 VN time (same day in VN)", () => {
// Current time: 19:15 VN time (12:15 UTC) // Current time: 19:15 VN time (12:15 UTC)
const from = new Date('2026-09-02T12:15:00.000Z'); const from = new Date("2026-09-02T12:15:00.000Z");
const next = calculateNextRun({ const next = calculateNextRun({
frequency: 'DAILY', frequency: "DAILY",
hour: 20, hour: 20,
minute: 0, minute: 0,
timezone: 'Asia/Ho_Chi_Minh', timezone: "Asia/Ho_Chi_Minh",
fromDate: from, fromDate: from,
}); });
// 20:00 VN time on Sept 2 is 13:00 UTC on Sept 2 // 20:00 VN time on Sept 2 is 13:00 UTC on Sept 2
expect(next.toISOString()).toBe('2026-09-02T13:00:00.000Z'); expect(next.toISOString()).toBe("2026-09-02T13:00:00.000Z");
}); });
it('calculates DAILY crawl at 02:00 VN time (next day in VN)', () => { it("calculates DAILY crawl at 02:00 VN time (next day in VN)", () => {
// Current time: 19:15 VN time on Sept 2 (12:15 UTC) // Current time: 19:15 VN time on Sept 2 (12:15 UTC)
const from = new Date('2026-09-02T12:15:00.000Z'); const from = new Date("2026-09-02T12:15:00.000Z");
const next = calculateNextRun({ const next = calculateNextRun({
frequency: 'DAILY', frequency: "DAILY",
hour: 2, hour: 2,
minute: 0, minute: 0,
timezone: 'Asia/Ho_Chi_Minh', timezone: "Asia/Ho_Chi_Minh",
fromDate: from, fromDate: from,
}); });
// 02:00 VN time on Sept 3 is 19:00 UTC on Sept 2 // 02:00 VN time on Sept 3 is 19:00 UTC on Sept 2
expect(next.toISOString()).toBe('2026-09-02T19:00:00.000Z'); expect(next.toISOString()).toBe("2026-09-02T19:00:00.000Z");
}); });
it('calculates WEEKLY crawl on Friday (day 5) at 08:00 VN time', () => { it("calculates WEEKLY crawl on Friday (day 5) at 08:00 VN time", () => {
// Current time: Wednesday Sept 2, 19:15 VN time // Current time: Wednesday Sept 2, 19:15 VN time
const from = new Date('2026-09-02T12:15:00.000Z'); const from = new Date("2026-09-02T12:15:00.000Z");
const next = calculateNextRun({ const next = calculateNextRun({
frequency: 'WEEKLY', frequency: "WEEKLY",
dayOfWeek: 5, dayOfWeek: 5,
hour: 8, hour: 8,
minute: 0, minute: 0,
timezone: 'Asia/Ho_Chi_Minh', timezone: "Asia/Ho_Chi_Minh",
fromDate: from, fromDate: from,
}); });
// Friday Sept 4, 08:00 VN time is Sept 4, 01:00 UTC // Friday Sept 4, 08:00 VN time is Sept 4, 01:00 UTC
expect(next.toISOString()).toBe('2026-09-04T01:00:00.000Z'); expect(next.toISOString()).toBe("2026-09-04T01:00:00.000Z");
}); });
it('calculates MONTHLY crawl on 15th at 09:30 VN time', () => { it("calculates MONTHLY crawl on 15th at 09:30 VN time", () => {
// Current time: Sept 2, 19:15 VN time // Current time: Sept 2, 19:15 VN time
const from = new Date('2026-09-02T12:15:00.000Z'); const from = new Date("2026-09-02T12:15:00.000Z");
const next = calculateNextRun({ const next = calculateNextRun({
frequency: 'MONTHLY', frequency: "MONTHLY",
dayOfMonth: 15, dayOfMonth: 15,
hour: 9, hour: 9,
minute: 30, minute: 30,
timezone: 'Asia/Ho_Chi_Minh', timezone: "Asia/Ho_Chi_Minh",
fromDate: from, fromDate: from,
}); });
// Sept 15, 09:30 VN time is Sept 15, 02:30 UTC // Sept 15, 09:30 VN time is Sept 15, 02:30 UTC
expect(next.toISOString()).toBe('2026-09-15T02:30:00.000Z'); expect(next.toISOString()).toBe("2026-09-15T02:30:00.000Z");
}); });
it('calculates MONTHLY crawl for next month if day has passed in VN time', () => { it("calculates MONTHLY crawl for next month if day has passed in VN time", () => {
// Current time: Sept 2, 19:15 VN time // Current time: Sept 2, 19:15 VN time
const from = new Date('2026-09-02T12:15:00.000Z'); const from = new Date("2026-09-02T12:15:00.000Z");
const next = calculateNextRun({ const next = calculateNextRun({
frequency: 'MONTHLY', frequency: "MONTHLY",
dayOfMonth: 1, dayOfMonth: 1,
hour: 9, hour: 9,
minute: 0, minute: 0,
timezone: 'Asia/Ho_Chi_Minh', timezone: "Asia/Ho_Chi_Minh",
fromDate: from, fromDate: from,
}); });
// Oct 1, 09:00 VN time is Oct 1, 02:00 UTC // Oct 1, 09:00 VN time is Oct 1, 02:00 UTC
expect(next.toISOString()).toBe('2026-10-01T02:00:00.000Z'); expect(next.toISOString()).toBe("2026-10-01T02:00:00.000Z");
}); });
it('calculates CUSTOM cron in Vietnam time', () => { it("calculates CUSTOM cron in Vietnam time", () => {
// Current time: Sept 2, 19:15 VN time (12:15 UTC) // Current time: Sept 2, 19:15 VN time (12:15 UTC)
// Cron: "0 22 * * *" (22:00 VN time every day) // Cron: "0 22 * * *" (22:00 VN time every day)
const from = new Date('2026-09-02T12:15:00.000Z'); const from = new Date("2026-09-02T12:15:00.000Z");
const next = calculateNextRun({ const next = calculateNextRun({
frequency: 'CUSTOM', frequency: "CUSTOM",
cronExpression: '0 22 * * *', cronExpression: "0 22 * * *",
timezone: 'Asia/Ho_Chi_Minh', timezone: "Asia/Ho_Chi_Minh",
fromDate: from, fromDate: from,
}); });
// 22:00 VN time on Sept 2 is 15:00 UTC on Sept 2 // 22:00 VN time on Sept 2 is 15:00 UTC on Sept 2
expect(next.toISOString()).toBe('2026-09-02T15:00:00.000Z'); expect(next.toISOString()).toBe("2026-09-02T15:00:00.000Z");
}); });
it('throws error on invalid cron expression', () => { it("throws error on invalid cron expression", () => {
expect(() => { expect(() => {
calculateNextRun({ calculateNextRun({
frequency: 'CUSTOM', frequency: "CUSTOM",
cronExpression: 'invalid cron', cronExpression: "invalid cron",
timezone: 'Asia/Ho_Chi_Minh', timezone: "Asia/Ho_Chi_Minh",
}); });
}).toThrow('Invalid cron expression'); }).toThrow("Invalid cron expression");
}); });
}); });
}); });
src/common/helpers/data-contract.helper.ts
View file @ c168a47c
This diff is collapsed.
src/common/helpers/error-mapping.helper.ts
View file @ c168a47c
...@@ -4,103 +4,103 @@ ...@@ -4,103 +4,103 @@
*/ */
export function mapCrawlError(rawError: string | null | undefined): string { export function mapCrawlError(rawError: string | null | undefined): string {
if (!rawError) { if (!rawError) {
return 'Lỗi không xác định trong quá trình cào dữ liệu.'; return "Lỗi không xác định trong quá trình cào dữ liệu.";
} }
const err = rawError.toLowerCase(); const err = rawError.toLowerCase();
// 1. Chặn bởi robots.txt // 1. Chặn bởi robots.txt
if ( if (
err.includes('robots.txt') || err.includes("robots.txt") ||
err.includes('blocked by robots') || err.includes("blocked by robots") ||
err.includes('robots blocked') err.includes("robots blocked")
) { ) {
return 'Trang web đích ngăn chặn việc cào dữ liệu thông qua tệp robots.txt.'; return "Trang web đích ngăn chặn việc cào dữ liệu thông qua tệp robots.txt.";
} }
// 2. CAPTCHA // 2. CAPTCHA
if (err.includes('captcha')) { if (err.includes("captcha")) {
return 'Trang web đích yêu cầu xác minh CAPTCHA. Hệ thống không thể bypass — trang được đánh dấu CAPTCHA_DETECTED.'; return "Trang web đích yêu cầu xác minh CAPTCHA. Hệ thống không thể bypass — trang được đánh dấu CAPTCHA_DETECTED.";
} }
// 3. Yêu cầu đăng nhập // 3. Yêu cầu đăng nhập
if (err.includes('requires login') || err.includes('login required')) { if (err.includes("requires login") || err.includes("login required")) {
return 'Trang web đích yêu cầu đăng nhập. Hệ thống không thể truy cập nội dung — trang được đánh dấu REQUIRES_LOGIN.'; return "Trang web đích yêu cầu đăng nhập. Hệ thống không thể truy cập nội dung — trang được đánh dấu REQUIRES_LOGIN.";
} }
// 4. Paywall // 4. Paywall
if (err.includes('paywall')) { if (err.includes("paywall")) {
return 'Trang web đích được bảo vệ bởi paywall (nội dung trả phí). Hệ thống không thể truy cập — trang được đánh dấu PAYWALL_DETECTED.'; return "Trang web đích được bảo vệ bởi paywall (nội dung trả phí). Hệ thống không thể truy cập — trang được đánh dấu PAYWALL_DETECTED.";
} }
// 5. Lỗi API Key / Xác thực // 5. Lỗi API Key / Xác thực
if ( if (
err.includes('unauthorized') || err.includes("unauthorized") ||
err.includes('api key') || err.includes("api key") ||
err.includes('apikey') || err.includes("apikey") ||
(err.includes('forbidden') && err.includes('key')) (err.includes("forbidden") && err.includes("key"))
) { ) {
return 'Lỗi xác thực hệ thống cào dữ liệu (API Key không hợp lệ, hết hạn hoặc vượt quá giới hạn gói dịch vụ).'; return "Lỗi xác thực hệ thống cào dữ liệu (API Key không hợp lệ, hết hạn hoặc vượt quá giới hạn gói dịch vụ).";
} }
// 6. Timeout // 6. Timeout
if (err.includes('timeout') || err.includes('timed out')) { if (err.includes("timeout") || err.includes("timed out")) {
return 'Kết nối đến trang web đích bị quá thời gian (Timeout). Trang web phản hồi quá chậm.'; return "Kết nối đến trang web đích bị quá thời gian (Timeout). Trang web phản hồi quá chậm.";
} }
// 7. Chặn bởi Cloudflare / Security / IP Blocked // 7. Chặn bởi Cloudflare / Security / IP Blocked
if ( if (
err.includes('cloudflare') || err.includes("cloudflare") ||
err.includes('403') || err.includes("403") ||
err.includes('forbidden') || err.includes("forbidden") ||
err.includes('access denied') || err.includes("access denied") ||
(err.includes('block') && (err.includes('ip') || err.includes('bot'))) (err.includes("block") && (err.includes("ip") || err.includes("bot")))
) { ) {
return 'Yêu cầu bị từ chối. Trang web đích chặn kết nối cào dữ liệu (chặn IP, phát hiện bot hoặc bảo vệ bởi Cloudflare/WAF).'; return "Yêu cầu bị từ chối. Trang web đích chặn kết nối cào dữ liệu (chặn IP, phát hiện bot hoặc bảo vệ bởi Cloudflare/WAF).";
} }
// 8. DNS / URL không đúng / Không tìm thấy host // 8. DNS / URL không đúng / Không tìm thấy host
if ( if (
err.includes('dns') || err.includes("dns") ||
err.includes('getaddrinfo') || err.includes("getaddrinfo") ||
err.includes('enotfound') || err.includes("enotfound") ||
err.includes('invalid url') || err.includes("invalid url") ||
err.includes('cannot parse url') err.includes("cannot parse url")
) { ) {
return 'Địa chỉ URL không hợp lệ hoặc không thể phân giải tên miền (trang web không tồn tại hoặc sai đường dẫn).'; return "Địa chỉ URL không hợp lệ hoặc không thể phân giải tên miền (trang web không tồn tại hoặc sai đường dẫn).";
} }
// 9. Rate limit / 429 // 9. Rate limit / 429
if ( if (
err.includes('rate limit') || err.includes("rate limit") ||
err.includes('429') || err.includes("429") ||
err.includes('too many requests') err.includes("too many requests")
) { ) {
return 'Yêu cầu bị từ chối do tần suất truy cập vượt quá giới hạn cho phép (Rate Limit). Vui lòng thử lại sau.'; return "Yêu cầu bị từ chối do tần suất truy cập vượt quá giới hạn cho phép (Rate Limit). Vui lòng thử lại sau.";
} }
// 10. Chặn IP Private (SSRF Protection) // 10. Chặn IP Private (SSRF Protection)
if (err.includes('private ip') || err.includes('private_ip_blocked')) { if (err.includes("private ip") || err.includes("private_ip_blocked")) {
return 'Không được phép cào dữ liệu từ địa chỉ IP nội bộ hoặc mạng nội bộ (SSRF Protection).'; return "Không được phép cào dữ liệu từ địa chỉ IP nội bộ hoặc mạng nội bộ (SSRF Protection).";
} }
// Fallback: không khớp bất kỳ pattern nào — trả về message tổng quát // Fallback: không khớp bất kỳ pattern nào — trả về message tổng quát
// Raw error được log ở tầng worker, không expose technical detail lên frontend // Raw error được log ở tầng worker, không expose technical detail lên frontend
return 'Đã xảy ra lỗi trong quá trình cào dữ liệu. Vui lòng thử lại hoặc kiểm tra URL đích.'; return "Đã xảy ra lỗi trong quá trình cào dữ liệu. Vui lòng thử lại hoặc kiểm tra URL đích.";
} }
export function getErrorMessage(error: unknown): string { export function getErrorMessage(error: unknown): string {
if (error instanceof Error) { if (error instanceof Error) {
return error.message; return error.message;
} }
if (typeof error === 'string') { if (typeof error === "string") {
return error; return error;
} }
if ( if (
error && error &&
typeof error === 'object' && typeof error === "object" &&
'message' in error && "message" in error &&
typeof (error as Record<string, unknown>).message === 'string' typeof (error as Record<string, unknown>).message === "string"
) { ) {
return (error as { message: string }).message; return (error as { message: string }).message;
} }
......
src/common/helpers/file.helper.ts
View file @ c168a47c
import path from 'path'; import path from "path";
import fs from 'fs'; import fs from "fs";
import { storageConfig } from '../../config/storage.config'; import { storageConfig } from "../../config/storage.config";
import { import {
JOB_EXPORT_SUBDIRS, JOB_EXPORT_SUBDIRS,
buildCrawlResultZipName, buildCrawlResultZipName,
buildMarkdownZipName, buildMarkdownZipName,
} from '../constants/storage-path.constant'; } from "../constants/storage-path.constant";
import { generatePageFileName } from './slug.helper'; import { generatePageFileName } from "./slug.helper";
export function ensureDirExists(dirPath: string): void { export function ensureDirExists(dirPath: string): void {
if (!fs.existsSync(dirPath)) { if (!fs.existsSync(dirPath)) {
...@@ -76,7 +76,7 @@ export function buildJobMarkdownFilePath( ...@@ -76,7 +76,7 @@ export function buildJobMarkdownFilePath(
jobId: string, jobId: string,
index: number, index: number,
url: string, url: string,
ext = 'md', ext = "md",
): { fileName: string; filePath: string } { ): { fileName: string; filePath: string } {
const fileName = generatePageFileName(index, url, ext); const fileName = generatePageFileName(index, url, ext);
const dirPath = buildJobSubDir(jobId, JOB_EXPORT_SUBDIRS.MARKDOWN); const dirPath = buildJobSubDir(jobId, JOB_EXPORT_SUBDIRS.MARKDOWN);
...@@ -87,7 +87,7 @@ export function buildJobMarkdownRawFilePath( ...@@ -87,7 +87,7 @@ export function buildJobMarkdownRawFilePath(
jobId: string, jobId: string,
index: number, index: number,
url: string, url: string,
ext = 'md', ext = "md",
): { fileName: string; filePath: string } { ): { fileName: string; filePath: string } {
const fileName = generatePageFileName(index, url, ext); const fileName = generatePageFileName(index, url, ext);
const dirPath = buildJobSubDir(jobId, JOB_EXPORT_SUBDIRS.MARKDOWN_RAW); const dirPath = buildJobSubDir(jobId, JOB_EXPORT_SUBDIRS.MARKDOWN_RAW);
...@@ -98,7 +98,7 @@ export function buildJobMarkdownCleanFilePath( ...@@ -98,7 +98,7 @@ export function buildJobMarkdownCleanFilePath(
jobId: string, jobId: string,
index: number, index: number,
url: string, url: string,
ext = 'md', ext = "md",
): { fileName: string; filePath: string } { ): { fileName: string; filePath: string } {
const fileName = generatePageFileName(index, url, ext); const fileName = generatePageFileName(index, url, ext);
const dirPath = buildJobSubDir(jobId, JOB_EXPORT_SUBDIRS.MARKDOWN_CLEAN); const dirPath = buildJobSubDir(jobId, JOB_EXPORT_SUBDIRS.MARKDOWN_CLEAN);
...@@ -109,7 +109,7 @@ export function buildJobRawFilePath( ...@@ -109,7 +109,7 @@ export function buildJobRawFilePath(
jobId: string, jobId: string,
index: number, index: number,
url: string, url: string,
ext = 'html', ext = "html",
): { fileName: string; filePath: string } { ): { fileName: string; filePath: string } {
const fileName = generatePageFileName(index, url, ext); const fileName = generatePageFileName(index, url, ext);
const dirPath = buildJobSubDir(jobId, JOB_EXPORT_SUBDIRS.RAW); const dirPath = buildJobSubDir(jobId, JOB_EXPORT_SUBDIRS.RAW);
......
src/common/helpers/proxy.helper.ts
View file @ c168a47c
export const parseTrustProxy = (value: string): boolean | number | string => { export const parseTrustProxy = (value: string): boolean | number | string => {
const lowercaseVal = value.trim().toLowerCase(); const lowercaseVal = value.trim().toLowerCase();
if (lowercaseVal === 'true') return true; if (lowercaseVal === "true") return true;
if (lowercaseVal === 'false') return false; if (lowercaseVal === "false") return false;
const parsed = parseInt(lowercaseVal, 10); const parsed = parseInt(lowercaseVal, 10);
if (!isNaN(parsed) && String(parsed) === value.trim()) { if (!isNaN(parsed) && String(parsed) === value.trim()) {
......
src/common/helpers/slug.helper.ts
View file @ c168a47c
...@@ -2,19 +2,19 @@ export function toSlug(text: string): string { ...@@ -2,19 +2,19 @@ export function toSlug(text: string): string {
return text return text
.toLowerCase() .toLowerCase()
.trim() .trim()
.replace(/[^\w\s-]/g, '') .replace(/[^\w\s-]/g, "")
.replace(/[\s_-]+/g, '-') .replace(/[\s_-]+/g, "-")
.replace(/^-+|-+$/g, ''); .replace(/^-+|-+$/g, "");
} }
export function urlToPageSlug(url: string): string { export function urlToPageSlug(url: string): string {
try { try {
const pathname = new URL(url).pathname; const pathname = new URL(url).pathname;
const segment = pathname.split('/').filter(Boolean).pop() || 'home'; const segment = pathname.split("/").filter(Boolean).pop() || "home";
const slug = toSlug(segment); const slug = toSlug(segment);
return slug || 'page'; return slug || "page";
} catch { } catch {
return 'page'; return "page";
} }
} }
...@@ -24,6 +24,6 @@ export function generatePageFileName( ...@@ -24,6 +24,6 @@ export function generatePageFileName(
ext: string, ext: string,
): string { ): string {
const slug = urlToPageSlug(url); const slug = urlToPageSlug(url);
const prefix = String(index + 1).padStart(3, '0'); const prefix = String(index + 1).padStart(3, "0");
return `${prefix}-${slug}.${ext}`; return `${prefix}-${slug}.${ext}`;
} }
src/common/helpers/url.helper.ts
View file @ c168a47c
This diff is collapsed.
src/common/storage/local-storage.service.ts
View file @ c168a47c
import fs from 'fs'; import fs from "fs";
import path from 'path'; import path from "path";
import { Readable } from 'stream'; import { Readable } from "stream";
import { pipeline } from 'stream/promises'; import { pipeline } from "stream/promises";
import { import {
IStorageService, IStorageService,
UploadResult, UploadResult,
UploadStreamOptions, UploadStreamOptions,
} from './storage.interface'; } from "./storage.interface";
import { storageConfig } from '../../config/storage.config'; import { storageConfig } from "../../config/storage.config";
import { ensureDirExists, getFileSizeBytes } from '../helpers/file.helper'; import { ensureDirExists, getFileSizeBytes } from "../helpers/file.helper";
export class LocalStorageService implements IStorageService { export class LocalStorageService implements IStorageService {
private getAbsolutePath(key: string): string { private getAbsolutePath(key: string): string {
......
src/common/storage/s3-storage.service.ts
View file @ c168a47c
This diff is collapsed.
src/common/storage/storage.factory.ts
View file @ c168a47c
import { IStorageService } from './storage.interface'; import { IStorageService } from "./storage.interface";
import { LocalStorageService } from './local-storage.service'; import { LocalStorageService } from "./local-storage.service";
import { S3StorageService } from './s3-storage.service'; import { S3StorageService } from "./s3-storage.service";
import { storageConfig } from '../../config/storage.config'; import { storageConfig } from "../../config/storage.config";
export class StorageFactory { export class StorageFactory {
private static instance: IStorageService; private static instance: IStorageService;
...@@ -12,9 +12,9 @@ export class StorageFactory { ...@@ -12,9 +12,9 @@ export class StorageFactory {
!StorageFactory.instance || !StorageFactory.instance ||
StorageFactory.activeDriver !== storageConfig.driver StorageFactory.activeDriver !== storageConfig.driver
) { ) {
if (storageConfig.driver === 's3') { if (storageConfig.driver === "s3") {
StorageFactory.instance = new S3StorageService(); StorageFactory.instance = new S3StorageService();
} else if (storageConfig.driver === 'local') { } else if (storageConfig.driver === "local") {
StorageFactory.instance = new LocalStorageService(); StorageFactory.instance = new LocalStorageService();
} else { } else {
throw new Error(`Unsupported storage driver: ${storageConfig.driver}`); throw new Error(`Unsupported storage driver: ${storageConfig.driver}`);
......
src/common/storage/storage.interface.ts
View file @ c168a47c
import { Readable } from 'stream'; import { Readable } from "stream";
export interface UploadResult { export interface UploadResult {
fileName: string; fileName: string;
......
src/common/types/data-contract.types.ts
View file @ c168a47c
This diff is collapsed.
src/common/types/express.d.ts
View file @ c168a47c
import { UserRole } from '@prisma/client'; import { UserRole } from "@prisma/client";
declare global { declare global {
namespace Express { namespace Express {
......
src/config/database.config.ts
View file @ c168a47c
import { envConfig } from './env.config'; import { envConfig } from "./env.config";
export const databaseConfig = { export const databaseConfig = {
url: envConfig.databaseUrl, url: envConfig.databaseUrl,
......
src/config/env.config.ts
View file @ c168a47c
This diff is collapsed.
src/config/firecrawl.config.ts
View file @ c168a47c
import { envConfig } from './env.config'; import { envConfig } from "./env.config";
export const firecrawlConfig = envConfig.firecrawl; export const firecrawlConfig = envConfig.firecrawl;
src/config/jwt.config.ts
View file @ c168a47c
This diff is collapsed.
src/config/mail.config.ts
View file @ c168a47c
This diff is collapsed.
src/config/storage.config.ts
View file @ c168a47c
This diff is collapsed.
src/database/prisma.client.ts
View file @ c168a47c
This diff is collapsed.
src/docs/swagger-paths.ts
View file @ c168a47c
This diff is collapsed.
src/docs/swagger.json
View file @ c168a47c
This diff is collapsed.
src/docs/swagger.ts
View file @ c168a47c
This diff is collapsed.
src/middlewares/api-key.middleware.ts
View file @ c168a47c
This diff is collapsed.
src/middlewares/auth.middleware.ts
View file @ c168a47c
This diff is collapsed.
src/middlewares/error.middleware.ts
View file @ c168a47c
This diff is collapsed.
src/middlewares/rate-limit.middleware.ts
View file @ c168a47c
This diff is collapsed.
src/middlewares/role.middleware.ts
View file @ c168a47c
This diff is collapsed.
src/middlewares/validate.middleware.ts
View file @ c168a47c
This diff is collapsed.
src/modules/api-keys/api-key.controller.ts
View file @ c168a47c
This diff is collapsed.
src/modules/api-keys/api-key.dto.ts
View file @ c168a47c
This diff is collapsed.
src/modules/api-keys/api-key.repository.ts
View file @ c168a47c
This diff is collapsed.
src/modules/api-keys/api-key.route.ts
View file @ c168a47c
This diff is collapsed.
src/modules/api-keys/api-key.service.ts
View file @ c168a47c
This diff is collapsed.
src/modules/api-keys/api-key.validation.ts
View file @ c168a47c
This diff is collapsed.
src/modules/audit-logs/audit-log.dto.ts
View file @ c168a47c
This diff is collapsed.
src/modules/audit-logs/audit-log.route.ts
View file @ c168a47c
This diff is collapsed.
src/modules/audit-logs/audit-log.service.ts
View file @ c168a47c
This diff is collapsed.
src/modules/auth/auth.controller.ts
View file @ c168a47c
This diff is collapsed.
src/modules/auth/auth.dto.ts
View file @ c168a47c
This diff is collapsed.
src/modules/auth/auth.repository.ts
View file @ c168a47c
This diff is collapsed.
src/modules/auth/auth.route.ts
View file @ c168a47c
This diff is collapsed.
src/modules/auth/auth.service.ts
View file @ c168a47c
This diff is collapsed.
src/modules/auth/auth.validation.ts
View file @ c168a47c
This diff is collapsed.
src/modules/crawl-assets/crawl-asset.dto.ts
View file @ c168a47c
This diff is collapsed.
src/modules/crawl-jobs/crawl-job.dto.ts
View file @ c168a47c
This diff is collapsed.
src/modules/crawl-jobs/crawl-job.route.ts
View file @ c168a47c
This diff is collapsed.
src/modules/crawl-jobs/crawl-job.service.ts
View file @ c168a47c
This diff is collapsed.
src/modules/crawl-pages/crawl-page.dto.ts
View file @ c168a47c
This diff is collapsed.
src/modules/dashboard/dashboard.route.ts
View file @ c168a47c
This diff is collapsed.
src/modules/dashboard/dashboard.service.ts
View file @ c168a47c
This diff is collapsed.
src/modules/exports/base-export.service.ts
View file @ c168a47c
This diff is collapsed.
src/modules/exports/csv-export.service.ts
View file @ c168a47c
This diff is collapsed.
src/modules/exports/export.service.ts
View file @ c168a47c
This diff is collapsed.
src/modules/exports/json-export.service.ts
View file @ c168a47c
This diff is collapsed.
src/modules/exports/xlsx-export.service.ts
View file @ c168a47c
This diff is collapsed.
src/modules/exports/zip-export.service.ts
View file @ c168a47c
This diff is collapsed.
src/modules/firecrawl/firecrawl.client.ts
View file @ c168a47c
This diff is collapsed.
src/modules/firecrawl/firecrawl.dto.ts
View file @ c168a47c
This diff is collapsed.
src/modules/firecrawl/firecrawl.service.ts
View file @ c168a47c
This diff is collapsed.
src/modules/health/health.controller.ts
View file @ c168a47c
This diff is collapsed.
src/modules/health/health.route.ts
View file @ c168a47c
This diff is collapsed.
src/modules/health/health.service.ts
View file @ c168a47c
This diff is collapsed.
src/modules/mail/mail.service.ts
View file @ c168a47c
This diff is collapsed.
src/modules/users/user.controller.ts
View file @ c168a47c
This diff is collapsed.
src/modules/users/user.dto.ts
View file @ c168a47c
This diff is collapsed.
src/modules/users/user.repository.ts
View file @ c168a47c
This diff is collapsed.
src/modules/users/user.route.ts
View file @ c168a47c
This diff is collapsed.
src/modules/users/user.service.ts
View file @ c168a47c
This diff is collapsed.
src/modules/users/user.validation.ts
View file @ c168a47c
This diff is collapsed.
src/modules/webhooks/webhook.controller.ts
View file @ c168a47c
This diff is collapsed.
src/modules/webhooks/webhook.repository.ts
View file @ c168a47c
This diff is collapsed.
src/modules/webhooks/webhook.route.ts
View file @ c168a47c
This diff is collapsed.
src/modules/webhooks/webhook.validation.ts
View file @ c168a47c
This diff is collapsed.
src/queues/__tests__/crawl.worker.test.ts
View file @ c168a47c
This diff is collapsed.
src/queues/crawl.queue.ts
View file @ c168a47c
This diff is collapsed.
src/queues/crawl.worker.processor.ts
View file @ c168a47c
This diff is collapsed.
src/queues/crawl.worker.ts
View file @ c168a47c
This diff is collapsed.
src/queues/schedule.worker.ts
View file @ c168a47c
This diff is collapsed.
src/queues/webhook.queue.ts
View file @ c168a47c
This diff is collapsed.
src/queues/webhook.worker.ts
View file @ c168a47c
This diff is collapsed.
src/routes/index.ts
View file @ c168a47c
This diff is collapsed.
src/server.ts
View file @ c168a47c
This diff is collapsed.
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment