Merge branch 'develop' into 'main'

Develop

See merge request !22

.agents/rules/design.md

+# Quy Tắc Thiết Kế Kiến Trúc (design.md)
+
+> Chuẩn mực kiến trúc phân lớp, nguyên tắc xác thực và xử lý lỗi cho Antigravity trong `data-crawler-be`.
+
+---
+
+## 1. Kiến Trúc 5 Lớp Đơn Hướng (Unidirectional 5-Layer Pattern)
+
+```
+Route  →  Controller  →  Service  →  Repository  →  Prisma  →  PostgreSQL
+```
+
+- **Route:** Chỉ làm nhiệm vụ cấu hình endpoint HTTP, middleware chuỗi (auth, role, validation, rateLimit). Không viết logic xử lý.
+- **Controller:** Nhận và chuyển đổi kiểu dữ liệu HTTP, gọi Service tương ứng, định dạng kết quả response `res.status(...).json(...)`.
+- **Service:** Xử lý nghiệp vụ chính (Business Logic), điều phối nhiều Repository, tích hợp hàng đợi BullMQ. Service không xử lý response HTTP và **không bao giờ gọi Prisma trực tiếp**.
+- **Repository:** Nơi **duy nhất** được phép import và gọi Prisma Client. Đảm bảo toàn bộ câu truy vấn (query, include, select, transaction) được cô lập tại đây.
+
+---
+
+## 2. Xác Thực Đầu Vào Tại Biên (Boundary Validation with Zod)
+
+- Mọi endpoint nhận dữ liệu từ client (`body`, `query`, `params`) phải có schema kiểm thực tương ứng bằng Zod.
+- Sử dụng middleware dùng chung:
+  ```typescript
+  import { validate } from "../../middlewares/validate.middleware";
+  import { mySchema } from "./my.validation";
+
+  router.post("/", validate(mySchema), myController.create);
+  ```
+- DTO type được suy diễn trực tiếp từ schema: `type MyDto = z.infer<typeof mySchema>;`.
+
+---
+
+## 3. Quản Lý Lỗi Tập Trung (Centralized Error Handling)
+
+- Bắt buộc dùng `AppError` kèm HTTP status code và mã `ERROR_CODE`:
+  ```typescript
+  import { AppError } from "../../common/errors/app-error";
+  import { ERROR_CODE } from "../../common/errors/error-code";
+
+  throw new AppError("Resource not found", 404, ERROR_CODE.NOT_FOUND);
+  ```
+- Định dạng response lỗi chuẩn:
+  ```json
+  {
+    "success": false,
+    "message": "Thông điệp lỗi",
+    "code": "ERROR_CODE",
+    "errors": []
+  }
+  ```
+
+---
+
+## 4. Hợp Đồng Dữ Liệu Cào (Data Contract V1)
+
+Bảo toàn hợp đồng dữ liệu quy định tại `docs/DATA_CONTRACT_V1.md`:
+
+- Dữ liệu thô (`raw`): Nguyên bản HTML từ Firecrawl.
+- Dữ liệu sạch (`clean`): Markdown chuẩn hóa qua Turndown, lọc bỏ script/ads/styles, tính toán `dataQualityScore` và `contentHash`.

.agents/rules/tech-defaults.md

+# Công Nghệ Mặc Định & Cấu Hình Chuẩn (tech-defaults.md)
+
+> Bảng quy ước phiên bản và danh mục công nghệ cốt lõi trong `data-crawler-be`.
+
+---
+
+## 1. Môi Trường Thực Thi & Ngôn Ngữ
+
+- **Node.js:** `>= 20.x`
+- **Package Manager:** `pnpm@9.15.0` (chỉ dùng `pnpm`, không dùng `npm` hay `yarn`)
+- **TypeScript:** `5.7` (`strict: true`, không dùng `any` bừa bãi)
+- **Web Framework:** `Express.js 4.21`
+
+---
+
+## 2. Lưu Trữ & Hàng Đợi
+
+- **Cơ sở dữ liệu:** PostgreSQL 15+ (chạy Docker local qua `docker-compose.yml`)
+- **ORM:** Prisma `5.22.x` (quản lý qua `scripts/prisma-run.js`)
+- **Queue / In-Memory Store:** Redis 7 + `bullmq ^5.34.0`
+- **Storage Driver:** Hỗ trợ song song Local (`storage/exports/`) và S3/MinIO (`@aws-sdk/client-s3`)
+
+---
+
+## 3. Crawl & Làm Sạch Dữ Liệu
+
+- **Crawl Engine:** `@mendable/firecrawl-js ^1.19.0` (SCRAPE, CRAWL, SITEMAP, URL_LIST)
+- **DOM Parser:** `cheerio ^1.2.0` & `node-html-parser ^8.0.4`
+- **Markdown Conversion:** `turndown ^7.2.0`
+- **Export formats:** `exceljs ^4.4.0` (XLSX), `json2csv ^6.0.0-alpha.2` (CSV), `archiver ^7.0.1` (ZIP)
+
+---
+
+## 4. Bảo Mật & Xác Thực
+
+- **Auth:** JWT (`jsonwebtoken ^9.0.2`, Access Token + Refresh Token trong database)
+- **Hash:** `bcryptjs ^2.4.3`
+- **Rate Limit:** `express-rate-limit ^8.5.2`
+- **HTTP Headers:** `helmet ^8.0.0` (tắt CSP cho Swagger UI)
+- **Phân quyền (RBAC):** `ADMIN`, `CRAWLER_USER`, `VIEWER`

.agents/rules/workflow.md

+# Quy Tắc Quy Trình Phát Triển (workflow.md)
+
+> Hướng dẫn quy trình phát triển tính năng, kiểm thử, migration cơ sở dữ liệu và quy chuẩn Git cho Antigravity.
+
+---
+
+## 1. Vòng Đời Triển Khai Tính Năng (Feature Lifecycle)
+
+Khi xây dựng hoặc sửa đổi tính năng trong `data-crawler-be`, thực hiện tuần tự:
+
+1. **Khảo sát & Thiết kế:** Kiểm tra quan hệ trong `prisma/schema.prisma` và tài liệu `docs/DATA_CONTRACT_V1.md`.
+2. **Tầng Repository:** Viết hàm truy vấn Prisma trong `src/modules/<feature>/<feature>.repository.ts`.
+3. **Tầng Service:** Viết business logic, kiểm tra quota, tích hợp BullMQ queue trong `<feature>.service.ts`.
+4. **Tầng Controller & DTO:** Định nghĩa Zod schema trong `<feature>.validation.ts`, kiểu DTO trong `<feature>.dto.ts`, xử lý HTTP request trong `<feature>.controller.ts`.
+5. **Gắn Route & Swagger:** Mount route tại `<feature>.route.ts`, đăng ký vào `src/routes/index.ts`, chạy `pnpm swagger`.
+6. **Kiểm thử & QA:** Thêm unit test vào `__tests__/`, chạy `pnpm lint` và `pnpm test -- --runInBand`.
+
+---
+
+## 2. Quy Chuẩn Commit Git
+
+Áp dụng chuẩn Conventional Commits:
+
+- `feat(<module>):` Thêm chức năng mới
+- `fix(<module>):` Sửa lỗi nghiệp vụ hoặc kỹ thuật
+- `refactor(<module>):` Tối ưu hóa code mà không thay đổi tính năng
+- `test(<module>):` Bổ sung hoặc sửa đổi unit test
+- `docs(<module>):` Cập nhật tài liệu, Swagger, README
+- `chore:` Nâng cấp gói, cấu hình môi trường, script
+
+---
+
+## 3. Quy Trình Làm Việc Với Prisma Database
+
+- Mọi thay đổi cấu trúc bảng thực hiện tại `prisma/schema.prisma`.
+- Luôn chạy migration thông qua script runner:
+  ```bash
+  # Tạo và áp dụng migration development
+  pnpm db:migrate
+  # Sinh lại Prisma Client
+  pnpm prisma:generate
+  # Kiểm tra trạng thái
+  pnpm db:migrate:status
+  ```
+- **Nghiêm cấm:** Không chạy `pnpm db:migrate:reset` khi chưa được sự xác nhận rõ ràng của người dùng.

.agents/skills/security-audit/CLIENT-SIDE.md

+# Client-Side and Browser Hunting
+
+#### When to use this file
+
+Reach for this file when meaningful trust decisions or untrusted rendering happen in the browser: single-page apps, browser extensions, embedded webviews, and anything that renders attacker-influenceable content into the DOM, receives cross-window messages, opens WebSockets, or serves credentialed cross-origin responses. These bugs live in code the server never executes — the fragment after `#`, `window.name`, a `postMessage` payload — so server-side escaping and the classes in `ATTACK-CLASSES.md` don't cover them.
+
+Use alongside `ATTACK-CLASSES.md`. The injection class there covers server-side sinks; this file covers the browser-side source→sink paths, cross-origin trust, and UI-redress classes that only exist client-side.
+
+Pick the relevant classes based on Phase 1. Split per surface (DOM rendering, message/WebSocket handlers, auth-carrying endpoints) for large front-ends.
+
+## Core discipline (include in every agent prompt for this domain)
+
+```
+- Client-side taint needs a controllable SOURCE and an executing SINK on the client path. A source with no sink, or a sink fed only server-rendered trusted data, is not a finding. Name both and show untrusted data reaching the sink unsanitized.
+- The impact must cross to a victim or cross an origin. XSS in the attacker's own DOM, or a "leak" of the attacker's own data, is not a finding. State whose session executes it or whose cross-origin data it steals.
+- Framework auto-escaping is a real mitigation. React/Vue/Angular escape interpolation by default — the finding is where the code opts OUT (`dangerouslySetInnerHTML`, `v-html`, `bypassSecurityTrust*`, `$sce.trustAs*`). Do not report escaped interpolation.
+- A missing header or attribute (X-Frame-Options, frame-ancestors, rel=noopener, SameSite) is only a finding with a concrete sensitive action behind it. A bare missing flag with no state-changing action or credentialed cross-origin read is a hardening note.
+```
+
+## DOM-based injection attack classes (subagent_type: `general`)
+
+**DOM-based XSS**
+Trace client-side sources — `location.hash`/`search`/`href`/`pathname`, `document.referrer`, `window.name`, `postMessage` data, `document.cookie` — into execution sinks: `innerHTML`/`outerHTML`, `document.write`, `eval`, `Function`, `setTimeout`/`setInterval` with a string argument, `element.src`/`href` set to a `javascript:` URI, jQuery `$(...)`/`.html()`, or framework escape hatches (`dangerouslySetInnerHTML`, `v-html`, `bypassSecurityTrustHtml`). The bug is source→sink with no sanitization _on the client path_; server-side escaping never sees fragment or `window.name` data.
+
+**DOM clobbering**
+Attacker-injected `id`/`name` attributes — surviving an HTML sanitizer that strips script but allows attributes — that shadow a global the script later reads (`window.config`, a `form.action`, a flag checked before initialization). Look for code reading `window.X`/`document.X` that an injected element named `X` can override. Requires a markup-injection sink that permits `id`/`name`.
+
+## Client-side trust and messaging attack classes (subagent_type: `general`)
+
+**postMessage origin trust**
+A `message` handler that acts on `event.data` (writes the DOM, calls a privileged function, stores a token) without checking `event.origin` against an allowlist, or with a weak check (`indexOf`, `startsWith`, unanchored regex, `endsWith` on the host). Also the send side: `postMessage(data, '*')` leaking data to any embedder. Confirm the handler does something security-relevant with the data.
+
+**Cross-site WebSocket hijacking (CSWSH)**
+A WebSocket handshake authenticated only by ambient cookies, with no `Origin` check and no per-session CSRF token — an attacker page opens a socket in the victim's authenticated context and reads/writes their data. Find the upgrade handler; check whether it validates `Origin` and binds to a token, not just the cookie.
+
+**CORS with credentials**
+A server that reflects the request `Origin` into `Access-Control-Allow-Origin` while sending `Access-Control-Allow-Credentials: true`, or allowlists `null` or a weak suffix match — any origin then reads authenticated responses. The finding is reflection or weak-match _with credentials_, not a wildcard alone (`*` with credentials is rejected by browsers).
+
+## UI-redress and navigation attack classes (subagent_type: `general`)
+
+**Clickjacking**
+A state-changing action (transfer, delete, grant, confirm) reachable in a framed page with no `X-Frame-Options: DENY`/`SAMEORIGIN` and no `frame-ancestors` CSP and no UI framebusting. A missing frame guard on a read-only page with no sensitive action is not a finding — require the action.
+
+**Reverse tabnabbing**
+A link whose target is attacker-influenceable, opened with `target="_blank"`, letting the opened page rewrite `window.opener.location` to a phishing origin. Modern browsers imply `noopener` for `target="_blank"`, so this is a finding only where the code sets `rel="opener"` explicitly, uses `window.open` without `noopener`, or the threat model includes older browsers — check before reporting.
+
+**Client-side open redirect / navigation**
+A navigation built from a client source (`location = params.get('next')`, `location.hash` fed into `location.href`, a router redirect) with no allowlist — including `javascript:`/`data:` schemes that promote the redirect into XSS. Distinct from a server open-redirect: the sink is in JS, so the server never sees it.
+
+## Prototype pollution attack classes (subagent_type: `general`)
+
+**Prototype pollution and gadget chain**
+An attacker-controlled key (`__proto__`, `constructor.prototype`) reaching a _nested/recursive_ write — a deep merge, `lodash.set`-style path assignment, `obj[a][b]=v` with an attacker-controlled segment, or a query-string parser that builds nested objects — that lands on `Object.prototype`. A plain `JSON.parse` or shallow `Object.assign` does NOT pollute. Require the recursive sink AND a gadget that reads the polluted property (an options object checked with `opts.isAdmin`, a template reading a config default, a sink that concatenates a polluted `src`). Pollution with no reachable gadget is not exploitable; the gadget is what turns it into XSS, auth bypass, or (in Node) RCE.
+
+## Universal moves (apply across the above)
+
+- **Start from the sink and walk back to a client source.** Grep the execution sinks (`innerHTML`, `eval`, `document.write`, `dangerouslySetInnerHTML`, `postMessage`, `new WebSocket`) and trace each argument back to `location`/`name`/`referrer`/message data. A sink fed only server-rendered trusted data is not a finding.
+- **Server escaping ends where the fragment begins.** Data after `#`, plus `window.name` and cross-window messages, never reaches the server — so server-side filters can't see it. That blind spot is the DOM-XSS goldmine.
+- **Enumerate the escape hatches.** In an auto-escaping framework, the candidate list _is_ every `dangerouslySetInnerHTML`/`v-html`/`bypassSecurityTrust*`/`$sce.trustAs*` call. Start there.
+
+## Validation rules (apply before reporting ANY finding here)
+
+1. **Confirm a controllable source AND an executing sink on the client path.** Cite the source (`location.hash`, `event.data`, `window.name`) and the sink (`innerHTML`, `eval`, navigation), and show untrusted data reaching the sink without sanitization. A source with no sink, or a sink fed only trusted data, is not a finding.
+2. **For prototype pollution, prove the recursive write AND a gadget.** Show the nested/recursive assignment that reaches `Object.prototype`, then the code that later reads the polluted property to a security-relevant effect. Pollution with no reachable gadget is not exploitable.
+3. **For messaging / CORS / WebSocket, show the origin check is absent or weak.** Cite the handler and the missing or `indexOf`/`startsWith`/unanchored-regex origin check, and that the data drives a security-relevant action or a credentialed cross-origin read. Reflection plus credentials, not a bare wildcard.
+4. **For UI-redress, require the sensitive action behind the missing guard.** Name the state-changing action that gets framed (clickjacking) or the attacker-controlled `_blank` link (tabnabbing). A missing `X-Frame-Options`/`rel=noopener` with nothing sensitive behind it is a hardening note — and framebusting, `frame-ancestors`, or the browser's `noopener` default may already defeat it. Check before reporting.
+5. **Return ONLY confirmed findings** with the client source→sink path and whose session it fires in — or "No exploitable client-side issues found" if that's honest.

.agents/skills/security-audit/HUNTING.md

+# Vulnerability Hunting
+
+### Phase 2: Hunt for vulnerabilities
+
+Launch **multiple `general` agents in parallel** via the Task tool. Use `general`, not `research` — general agents can spawn their own sub-agents via the Task tool, so when a hunter finds a rabbit hole that needs deeper investigation (e.g., tracing injection into an auth subsystem it doesn't fully understand), it can spin up a focused `research` sub-agent rather than trying to do everything in one context window.
+
+Each agent gets the architecture summary from Phase 1 injected into its prompt plus the hunting methodology and validation rules. Launch them in a single message so they run concurrently.
+
+**How many agents?** Use Phase 1 to decide. More focused agents produce better results than broad ones that run out of context. For a small library, 3-4 agents may suffice. For a large application with distinct subsystems, launch 8-12+ — split by attack class AND by subsystem. If Phase 1 revealed an auth system, a plugin system, a media pipeline, and a comment engine, each of those could warrant its own injection agent, its own logic agent, etc.
+
+Every agent prompt MUST include:
+
+1. The architecture summary from Phase 1 (copy it in verbatim)
+2. The specific attack class and scope to investigate
+3. Relevant file paths from Phase 1 as starting points
+4. The hunting methodology (below)
+5. The validation rules (below)
+
+#### Hunting methodology — include in every Phase 2 agent prompt
+
+Tell each agent to think like an attacker, not a code reviewer:
+
+```
+## How to hunt
+
+Don't just check if defenses exist. Try to break them.
+
+READ THE CODE AT DEPTH. Don't stop at the first function. Follow the data through
+every layer — from the entry point through validation, transformation, storage, retrieval,
+and output. Bugs live in the gaps between layers.
+
+Think about these angles:
+
+1. THE HAPPY PATH IS DEFENDED. ATTACK THE SAD PATH.
+   Error handlers, fallback branches, catch blocks, default cases, timeout paths,
+   retry logic, cleanup routines. What happens when things fail? Are errors handled
+   with the same rigor as success? Does a failed validation leave state half-modified?
+
+2. WHAT HAPPENS AT BOUNDARIES?
+   Empty input. Maximum-length input. Null vs undefined vs missing. Zero. Negative numbers.
+   Unicode edge cases. The first item and the last item. One more than the maximum. Exactly
+   at the rate limit. The moment a token expires.
+
+3. WHAT DO COMPONENTS ASSUME ABOUT EACH OTHER?
+   Does the database layer assume the API layer validated input? Does the renderer assume
+   content was sanitized on write? Does the auth middleware assume routes register themselves
+   correctly? Find where trust is implicit and test whether it's justified.
+
+4. WHAT IF OPERATIONS HAPPEN IN THE WRONG ORDER?
+   Call step 3 before step 1. Call delete during create. Send the callback before the request.
+   Hit the confirmation endpoint without starting the flow. Replay a completed flow.
+
+5. WHAT IF TWO THINGS HAPPEN AT ONCE?
+   Two requests to the same resource. Modify while reading. Delete while iterating.
+   Publish while someone else is editing. Two users claiming the same unique resource.
+
+6. WHERE DO TWO PARSERS OR VALIDATORS DISAGREE?
+   Input accepted by the schema but rejected by the database. URL parsed differently by
+   the router vs the application code. Content-type header says one thing, body is another.
+   Filename extension vs MIME type vs magic bytes.
+
+7. WHAT SURVIVES A ROUND TRIP?
+   Data stored then retrieved — is it the same? Does encoding change? Does escaping
+   double-up? Is a relative path resolved differently on read vs write? Does serialization
+   lose type information?
+
+8. WHAT DOES THE CONFIGURATION CONTROL?
+   What happens when config is missing or default? Can an environment variable override a
+   security control? Does a feature flag disable validation? What's the security posture
+   during setup/first-run before config is complete?
+
+9. FOLLOW THE MONEY (OR THE PRIVILEGE).
+   For every operation that changes state, ask: who authorized this? Trace back to the
+   permission check. Is it checking the right permission? Is it checking against the right
+   resource? Is there a parallel path to the same state change that checks differently
+   or not at all?
+
+10. LOOK FOR LEAKED CONTEXT.
+    Error messages that reveal internal paths. Stack traces in production. Timing differences
+    that reveal whether a record exists. Response size differences. HTTP headers that
+    disclose versions. Debug endpoints that survived into production.
+
+11. WHAT PARAMETERS OVERRIDE SECURITY-RELEVANT DEFAULTS?
+    Where a default is safe but a user-supplied parameter can change it. Look for
+    every input that overrides a security-relevant default and check if the override
+    is gated by appropriate permissions.
+
+12. WHERE DO UNVERIFIED CLAIMS DRIVE TRUST DECISIONS?
+    Anywhere self-declared identity, capability, or metadata influences an access
+    or trust decision without independent verification.
+
+GO DEEP, AND PROVE IT. You can spawn sub-agents: if evaluating a candidate finding needs
+deep understanding of a subsystem, use the Task tool to launch a research agent instead of
+holding everything in one context. And where the code is locally runnable, don't just reason
+about it — extract the suspect function into a minimal harness (or build and run the target)
+and test the hypothesis directly. A reproduced result beats an argued one.
+
+YOUR SCOPE IS YOUR PRIMARY FOCUS, NOT A BOUNDARY.
+If while investigating your assigned area you notice something wrong in a different
+category — a permission issue while tracing injection, a race condition while reviewing
+auth — report it. Don't ignore a bug because it's "not your area." Attackers don't
+respect category boundaries.
+
+## Validation rules — apply before reporting ANY finding
+1. You MUST construct a concrete attack (exact inputs, requests, or action sequence)
+2. The attack MUST achieve meaningful impact (not just "learn field names" or "cause an error")
+3. Check if another layer already prevents exploitation — if so, it's a hardening note, not a finding
+4. If the baseline comparable has the same pattern, note whether it's been exploited there
+5. If your exploit depends on parser/runtime behavior, verify against the relevant spec or implementation — do not reason from intuition.
+6. Return ONLY confirmed findings with concrete attacks, or "No exploitable vulnerabilities found" if that's honest.
+```

.agents/skills/security-audit/MEMORY-SAFETY-AND-BINARY.md

+# Memory Safety, Binary, and Kernel Hunting
+
+#### When to use this file
+
+The attack classes in `ATTACK-CLASSES.md` are tuned for web apps, APIs, and services. Reach for _this_ file when the target processes untrusted bytes in a memory-unsafe context: C/C++/Objective-C, Rust `unsafe`, kernel modules and drivers, parsers and decoders (image/video/font/archive/PDB), reverse-engineering and dev tooling, network daemons, firmware, and language runtimes/JITs. These targets fail differently from web apps — the bug is a memory corruption or a logic error in privileged code, not an injection or an access-control gap — so the hunt needs a different lens.
+
+Pick the relevant classes based on Phase 1. Split per subsystem for large targets.
+
+## Core discipline (include in every agent prompt for this domain)
+
+```
+- A buffer sized for the common case can still overflow on adversarial input. Verify every "this length is bounded" claim against the WORST case, not the happy path.
+- "Huge count = guaranteed crash" is FALSE. An oversized copy length is size- and libc-dependent: it often faults, but the copy primitive can also wrap or land a short, scattered write first. Determine the actual write behavior before downgrading to DoS-only.
+- Static offsets are a guess; the crash dump is truth. An unreproduced bug is not a bug — if you claim exploitability, say exactly which input reaches which sink and what the observable result is.
+- Sanitizer silence ≠ safety where the deref is outside instrumented code (hand-written asm, JIT-emitted, intra-allocation). Don't trust a clean ASan run for those.
+```
+
+## Memory-safety attack classes (subagent_type: `general`)
+
+**Spatial: out-of-bounds read/write**
+
+- **Length subtraction underflow** — a copy/loop bound is `a - b` (`uri.len - prefix`, `total - consumed`) where the attacker can make `b > a`. Negative → casts to ~SIZE_MAX. Map which bytes land where; don't assume "just a crash."
+- **Operator-precedence / multi-term length errors** — an unparenthesized `+`/`-` length chain (`endp - begin + consume`) that silently over-adds when one term is attacker-sized. Audit each CALLER's value of the variable term — the common caller is often correct-by-accident on the zero path and survives testing.
+- **`sizeof(*p)` vs `sizeof(element)` pointer-depth confusion** — an allocation/copy size computed one indirection too deep (`gid_t **` → `sizeof(*p)`=8 not 4). The bounds check passes because it uses the same wrong unit. Compiled tell: `shl $0x3` where `shl $0x2` was meant.
+- **Wire-length into fixed stack buffer** — a function rebuilds a network/user blob into a fixed array using an attacker length field, with the bounds check missing/late or computed on the wrong headroom (a header pre-written into the buffer). Re-derive true headroom (size minus fixed prefix); confirm no guard precedes the copy.
+
+**Temporal: use-after-free / lifetime**
+
+- **Embedded waiter-anchor freed without draining** — a struct embeds a list head (`selinfo`/`knlist`/timer/knote) reachable by unprivileged poll/select/kqueue, and a free path destroys it but skips the drain a wakeup path does. For every `selrecord(&obj->x)`, require a matching drain on EACH path that can free `obj`.
+- **Cached raw pointer + reallocating owner** — a view caches `base+offset`, a grow/realloc path moves the backing store, and the invalidation walks only the _current_ wrapper's view set while grow _replaces_ the wrapper. The original view dangles.
+
+**Type confusion**
+
+- **Read-and-write confusion → addrof/fakeobj** — a confusion that reads a pointer slot as a scalar (addrof) and writes a scalar into a pointer slot (fakeobj). The standard pivot of runtime/JIT exploitation; the prior art is about the PROBLEM CLASS (NaN-boxing, cached typed-array data pointer), not the specific target.
+- **Hierarchical-walker leaf check skipped** — a page-table / nested / B-tree / extent walker checks the valid bit but not the leaf/size bit at level N, then descends treating an attacker-owned leaf as an interior node.
+
+**Value: uninitialized & oracle**
+
+- **Uninitialized worst-case buffer + observable compare = read oracle** — a buffer sized to a MAX constant is partially written, then compared against attacker bytes with an attacker-controlled compare length where match/no-match is observable. No memory-disclosure bug needed; the gap between actual output and MAX-size is the leak window. Brute one byte/connection, hint the structural bits, parallelize.
+
+## Kernel & privileged-interface attack classes (subagent_type: `general`)
+
+- **User-copy bounds + double-fetch (TOCTOU)** — a syscall/ioctl/Mach-trap entry whose user-copy primitive (`copyin` / `copy_from_user`) brings attacker memory in, then re-reads the SAME user address after a check. Any fact derived from concurrently-mutable user memory and trusted on a later pass is a double-fetch even when each op is individually correct.
+- **Object lifecycle / UAF (IOKit/OSObject and friends)** — unbalanced retain/release on an externally-reachable object; a method that releases on one path but a sibling dispatch (compat/fallback/ptrace) forgot it. Diff the duplicated dispatch paths.
+- **Unchecked downcast / type confusion** — `OSDynamicCast` (or any tagged-union cast) whose result is used without a null check, or a selector/index into a dispatch table without a bounds check.
+- **World-writable / under-permissioned powerful interface** — a device node, admin socket, or mgmt API exposed more broadly than its power, that validates the request SHAPE (index in range) but never the requester's AUTHORITY over the named resource. Danger = power × reachability; enumerate the surface reachable from the _actual_ untrusted context first.
+- **Validate-then-act-on-stale-state** — a fast path and a compat/ptrace/fallback path to the same operation where one copy forgot a guard the other performs.
+
+## Universal moves (apply across the above)
+
+- **Audit the incomplete fix.** A targeted patch is a high-signal pointer to a dangerous sink with the analysis already done. Read the diff → find the exact sink it hardened → scan the same function, parallel paths, and alternate callers for the SAME tainted-data-to-sink shape the patch missed. Incomplete fixes are their own bug class.
+- **Trust asymmetry between two ends of a protocol.** A filter/verification/size-cap installed on one side of a connection but missing on the symmetric call on the other. Find the protective call → grep its mirror on the opposite role → if absent, the earliest unprotected pre-auth parse is the prize. A malicious server/MITM is a real attacker.
+- **Chain a weak primitive.** A blocked path means you haven't found the right pivot, not that it's unexploitable. Always ask "what does this actually let me do, and what runs automatically once I can put bytes on disk?" (plugin dirs, autoload, `.git/hooks`, `conftest.py`).
+- **Hunt where the crowd isn't.** The tools researchers themselves trust — debuggers, disassemblers, scanners, dev tooling — are under-audited and high-impact. Old code and obscure formats are gold.
+
+## Validation rules (apply before reporting ANY finding here)
+
+1. **Build a debuggable target first.** Wire in crash dumps + a debugger before you claim exploitability. You can't iterate on what you can't observe.
+2. **Read the offset from the crash, not the disassembly.** Send a cyclic (De Bruijn) pattern; the faulting register values give the exact offset. A variable-length prefix (handle, optional field, padding) shifts the geometry off the static prediction.
+3. **Prove a UAF by reclaim-and-compare** when the sanitizer is blind (asm/JIT/intra-allocation): trigger the dangling view, reclaim the freed region with a size-matched content-controlled allocation, write through the dangler, read the reclaimer back — aliasing either way proves it.
+4. **Distinguish crash from exploitable.** For an OOB write, map which bytes land where and whether a security-relevant field is reachable; for a "huge count," prove the bounded-write case before calling it DoS-only.
+5. **Return ONLY confirmed findings** with the exact input → sink path and the observable result, or "No exploitable memory-safety issues found" if that's honest.

.agents/skills/security-audit/RECONNAISSANCE.md

+# Reconnaissance
+
+### Phase 1: Understand the application
+
+Before looking for bugs, understand what you're auditing. This requires depth, not just a directory listing. Launch **multiple `research` agents in parallel** to map different aspects of the codebase:
+
+**Agent 1a: Overview, tech stack, and comparable baseline**
+
+```
+Explore the codebase at <path>. Answer:
+1. What is this application? What kind of software? (web app, API, CLI tool, library, daemon, desktop app, mobile backend, etc.)
+2. Who uses it and how? (end users, developers, operators, other services)
+3. What's the tech stack? (languages, frameworks, databases, runtime, deployment model)
+4. What comparable mainstream software exists? What security tradeoffs does the comparable accept?
+5. What's the high-level directory structure?
+Return specific file paths for key entry points.
+```
+
+**Agent 1b: Trust boundaries and access control**
+
+```
+Explore the codebase at <path>. Find and read ALL code related to:
+1. Trust boundaries — where does untrusted input enter the system? (HTTP requests, CLI args, file reads, IPC, message queues, environment variables, config files, etc.)
+2. Authentication — how do callers prove identity? (sessions, tokens, API keys, mTLS, Unix sockets, etc.) If there's no authentication, note that.
+3. Authorization — how are permissions enforced? (middleware, decorators, capability checks, file permissions, etc.) If there's no authorization model, note that.
+4. Privilege separation — does the code run as root? Drop privileges? Use sandboxing? Fork workers?
+5. Any bypass mechanisms (dev-only modes, test helpers, setup flows, debug flags)
+Return the trust model: who are the actors, what can each do by design, and which code enforces it. Include specific file paths and line numbers.
+```
+
+**Agent 1c: Input surface inventory**
+
+```
+Explore the codebase at <path>. Produce a complete inventory of where external input enters the system:
+1. Network-facing surfaces (HTTP endpoints, gRPC services, WebSocket handlers, TCP/UDP listeners, etc.) — list each with method/verb and purpose
+2. File-based input (file uploads, config file parsing, log ingestion, import/export, etc.)
+3. IPC and inter-service input (message queues, shared memory, Unix sockets, environment variables, CLI arguments)
+4. User-generated content surfaces (anywhere users provide content that is stored and later rendered, served, or processed)
+5. External integrations (OAuth, webhooks, third-party APIs, plugin loading, dynamic code execution)
+6. All places where input reaches dangerous sinks (SQL/query builders, HTML/template output, file paths, shell commands, deserialization, eval, dynamic imports)
+Return specific file paths. Be exhaustive.
+```
+
+Collect all three agents' outputs and synthesize them into `<output-dir>/architecture.md`:
+
+- 1-2 page structured summary covering application type, tech stack, trust model, input surfaces, and baseline comparable
+- Include the key file paths from all agents — these become the starting points for Phase 2
+- This document is injected verbatim into every Phase 2 agent prompt
+
+If Phase 1 agents reveal the codebase is larger or more complex than expected (e.g., plugin system, multi-tenant architecture, complex auth chains, multiple deployment targets), launch additional `research` agents to map those areas before proceeding. The quality of Phase 2 depends entirely on the quality of Phase 1.

.agents/skills/security-audit/VALIDATION-AND-REPORTING.md

+# Validation, Reporting, and Verification
+
+### Phase 3: Validate findings
+
+Collect all findings from Phase 2 agents and **consolidate duplicates first**. Phase 2 deliberately overlaps agent scopes, so the same issue is frequently reported by more than one hunter — merge findings that share a root cause before validating, or you'll validate and report the same bug multiple times. For each remaining finding, launch a **separate `research` validation agent** that tries to disprove it. The hunting agents are biased toward finding things; the validation agents are biased toward killing false positives. This adversarial step is critical.
+
+For findings from the same attack surface, batch them into one validation agent. Launch validation agents in parallel where they cover independent areas.
+
+Each validation agent prompt should:
+
+1. State the specific finding being validated (title, claimed attack, claimed impact)
+2. Ask the agent to read the exact code paths and verify each step of the trace
+3. Ask it to apply these tests (the adversarial, Phase 3 form of the canonical validation rules in [HUNTING.md](HUNTING.md) — here a separate agent tries to make each one fail):
+
+**Validation tests:**
+
+1. **Exploitation test**: Read the actual code at each step of the trace. Does the data flow work as claimed? Can you construct the exact input (HTTP request, CLI invocation, API call, crafted file, etc.) that triggers this?
+2. **Impact test**: What does the attacker actually get? If the answer is "they learn field names" or "they cause an error", that's not meaningful impact — not a finding on its own (at most a building block for a chain).
+3. **Baseline test**: Does the identified comparable have the same pattern? If yes, has it been exploited? If never exploited in years of production use, understand why before reporting.
+4. **Mitigation test**: Is there another layer that prevents exploitation? Check middleware, database constraints, framework defaults.
+5. **Parser/runtime behavior test**: If the exploit depends on how a parser or runtime handles specific input, verify against the actual spec or implementation — do not reason from intuition.
+
+Tell each validation agent:
+
+```
+Your job is to DISPROVE this finding. Read the actual source code at every step. If you cannot disprove it, confirm it with the exact code that makes it exploitable. Return one of:
+- "CONFIRMED: [explanation of why it's real, with code evidence]"
+- "REJECTED: [explanation of what the finding got wrong, with code evidence]"
+```
+
+**Kill false positives aggressively, but don't kill real findings.** A short report with 3 real findings is worth more than a long report with 30 theoretical ones. An honest "nothing found" is valid — but push hard before reaching that conclusion.
+
+### Phase 4: Report
+
+Write the report to the output directory established in Setup.
+
+**Output files:**
+
+1. `REPORT.md` -- Main report with:
+   - One-paragraph executive summary (honest assessment of security posture)
+   - Identified baseline and how this application compares
+   - Findings table (severity, title, one-line description)
+   - Each finding with: file path, concrete attack scenario, impact, recommended fix
+   - Hardening notes section (defense-in-depth suggestions, NOT findings)
+   - Positive patterns section (what the codebase does well -- this calibrates trust in the audit)
+
+2. `FINDINGS-DETAIL.md` -- For each finding rated MEDIUM or above:
+   - Complete data flow from input to sink with file:line references
+   - Exact HTTP request(s) to trigger
+   - What the attacker gets
+   - How the baseline comparable handles the same scenario
+
+Keep it short. If the report is longer than the codebase deserves, you're padding.
+
+### Phase 5: Structured output and schema check
+
+For every finding that survived Phase 3 validation, produce a structured JSON object conforming to the schema defined in `report-schema.json` (in the same directory as this skill file — read it via the Read tool before writing output). Write the result to `<output-dir>/findings.json`.
+
+The schema supports two verdict types via `oneOf`:
+
+- **`confirmed`** — a validated vulnerability with full trace, execution, and remediation
+- **`rejected`** — a finding that was investigated and determined to be factually incorrect
+
+**Before writing `findings.json`:**
+
+1. Read `report-schema.json` from this skill's directory. Follow it exactly — `additionalProperties: false` is enforced, so extra fields will make the output invalid.
+2. For each finding, populate every required field. If you cannot fill `trace` with real file paths and line numbers verified against the source, the finding is not sufficiently verified — go back and verify it or reject it. Mind the required fields that aren't self-evident: `intended_behavior` (what the code is _supposed_ to do, so the defect is legible), `confidence` (`low`/`medium`/`high`, with a reason), and the `severity` object (`likelihood`/`impact`/`overall_severity`). All `severity` scores use the schema's **lowercase** enum — `informational`/`low`/`medium`/`high`/`critical`; the UPPERCASE tiers in SKILL.md and REPORT.md are prose labels, not valid JSON values.
+3. Run `node <skill-dir>/validate-findings.cjs <output-dir>/findings.json` to validate. It checks required fields, enum values, structural constraints, and `additionalProperties`. This is a structural check only — it confirms the JSON conforms to the schema, not that the findings are correct. Factual verification is Phase 6's job. Fix any failures before proceeding.
+
+### Phase 6: Independent verification
+
+The structured output from Phase 5 forces self-validation, but the same agent that wrote the finding also wrote the JSON — it won't catch its own blind spots. This phase uses a fresh agent to independently verify every claim in `findings.json`.
+
+Launch **one `research` agent per confirmed finding** via the Task tool, all in parallel. Each agent gets exactly one finding from `findings.json` and verifies it independently. Give each agent the JSON object for its finding and this prompt:
+
+```
+You are an independent verifier. You did NOT write this finding. Your job is to read the actual source code and verify that every factual claim is correct.
+
+1. Read the file and line number cited in EVERY trace step. Verify:
+   - The file exists at that path
+   - The line number matches the described code
+   - The scope (function name) is correct
+   - The description accurately reflects what the code does
+
+2. Verify the root_cause statement by reading the cited file and confirming the described defect exists.
+
+3. Verify the execution payloads would actually work, in terms that fit the target:
+   - Does the entry point exist as claimed — the endpoint/URL, CLI command, exported function, syscall/ioctl, message handler, or tool the attacker invokes?
+   - Does the invocation match — HTTP method, argument shape, call signature, or message format?
+   - Would the input survive validation and parsing on the real code path?
+   - Would the relevant authentication, authorization, or ownership check pass as described?
+
+4. Verify conditions are complete — are there prerequisites the finding missed?
+
+5. Check the remediation code_changes — would the fix actually prevent the attack without breaking normal functionality?
+
+6. Verify `intended_behavior` accurately states what the code should do, and that `confidence` matches the strength of the evidence — don't leave `high` on a claim you couldn't fully trace.
+
+Return one of:
+- "VERIFIED" — all claims checked out against the source
+- "CORRECTED: [field]: [what was wrong] → [what it should be]" — factual error in a specific field
+- "REJECTED: [reason]" — the finding is fundamentally wrong
+```
+
+Apply the agent's corrections:
+
+- **VERIFIED** findings: no changes needed
+- **CORRECTED** findings: update the specific fields in `findings.json`, re-run the schema validation script
+- **REJECTED** findings: change their `verdict` to `"rejected"` with the agent's reason, or remove them entirely
+
+After applying corrections, reconcile the prose deliverables: update `REPORT.md` and `FINDINGS-DETAIL.md` so they match the final `findings.json`. Remove or amend any finding the verification gate rejected or corrected — the human-readable report and the machine-readable output must not disagree.
+
+This is the final quality gate. Do not skip it.

.agents/skills/security-audit/report-schema.json

+{
+  "$comment": "Single source of truth for findings.json structure (see SKILL.md Phase 5). validate-findings.cjs reads this file directly and interprets it — there is no second copy of these rules to keep in sync.",
+  "output_schema": {
+    "oneOf": [
+      {
+        "type": "object",
+        "description": "Confirmed vulnerability — provide the complete, independently verified report.",
+        "properties": {
+          "verdict": {
+            "type": "string",
+            "const": "confirmed"
+          },
+          "title": {
+            "type": "string",
+            "description": "A concise, standard title for the vulnerability."
+          },
+          "description": {
+            "type": "string",
+            "description": "Comprehensive explanation of the vulnerability. Include any reproduction details (proof-of-concept input, configuration, observed output or crash) here."
+          },
+          "root_cause": {
+            "type": "string",
+            "description": "One sentence using the template: '[function_or_component] in [file] does not [missing action], allowing [consequence]'. MUST include the function/component name and file name where the defect exists."
+          },
+          "intended_behavior": {
+            "type": "string",
+            "description": "What was the developer trying to build? Explain the intended, non-vulnerable business logic."
+          },
+          "trace": {
+            "type": "array",
+            "minItems": 2,
+            "items": {
+              "type": "object",
+              "properties": {
+                "kind": {
+                  "type": "string",
+                  "enum": ["entrypoint", "propagation", "sink"]
+                },
+                "file": {
+                  "type": "string",
+                  "description": "Exact file path relative to repository root."
+                },
+                "line": {
+                  "type": "integer"
+                },
+                "scope": {
+                  "type": "string",
+                  "description": "Bare function or method name. No parentheses, no arguments."
+                },
+                "description": {
+                  "type": "string",
+                  "description": "Factual description of the state change or data movement."
+                }
+              },
+              "required": ["kind", "file", "line", "scope", "description"],
+              "additionalProperties": false
+            },
+            "description": "Sequential code trace from entrypoint to sink, verified against actual source code. The first step must be kind 'entrypoint', the last must be kind 'sink', and any intermediate steps must be kind 'propagation' (enforced by the validator)."
+          },
+          "conditions": {
+            "type": "array",
+            "items": {
+              "type": "object",
+              "properties": {
+                "kind": {
+                  "type": "string",
+                  "enum": [
+                    "authentication_level",
+                    "authorization_role",
+                    "user_interaction",
+                    "system_configuration",
+                    "network_routing",
+                    "environmental_dependency",
+                    "data_state",
+                    "timing_dependency",
+                    "third_party_dependency"
+                  ]
+                },
+                "description": {
+                  "type": "string"
+                }
+              },
+              "required": ["kind", "description"],
+              "additionalProperties": false
+            },
+            "description": "Factual prerequisites for exploitation. Empty array if exploitable by default."
+          },
+          "execution": {
+            "type": "object",
+            "properties": {
+              "attacker_perspective": {
+                "type": "string",
+                "description": "Who is the attacker and their starting point."
+              },
+              "payloads": {
+                "type": "array",
+                "items": {
+                  "type": "string"
+                },
+                "description": "Specific malicious inputs, HTTP requests, or scripts."
+              },
+              "instructions": {
+                "type": "array",
+                "items": {
+                  "type": "string"
+                },
+                "description": "Linear array of all attacker actions from setup through exploitation."
+              },
+              "expected_result": {
+                "type": "string",
+                "description": "Observable outcome confirming successful exploitation."
+              }
+            },
+            "required": [
+              "attacker_perspective",
+              "payloads",
+              "instructions",
+              "expected_result"
+            ],
+            "additionalProperties": false
+          },
+          "remediation": {
+            "type": "object",
+            "properties": {
+              "strategy": {
+                "type": "string",
+                "description": "High-level explanation of the fix."
+              },
+              "code_changes": {
+                "type": "array",
+                "items": {
+                  "type": "object",
+                  "properties": {
+                    "file_name": {
+                      "type": "string"
+                    },
+                    "fixed_code": {
+                      "type": "string"
+                    }
+                  },
+                  "required": ["file_name", "fixed_code"],
+                  "additionalProperties": false
+                }
+              }
+            },
+            "required": ["strategy"],
+            "additionalProperties": false
+          },
+          "severity": {
+            "type": "object",
+            "properties": {
+              "likelihood": {
+                "type": "object",
+                "properties": {
+                  "score": {
+                    "type": "string",
+                    "enum": [
+                      "informational",
+                      "low",
+                      "medium",
+                      "high",
+                      "critical"
+                    ]
+                  },
+                  "reason": {
+                    "type": "string"
+                  }
+                },
+                "required": ["score", "reason"],
+                "additionalProperties": false
+              },
+              "impact": {
+                "type": "object",
+                "properties": {
+                  "score": {
+                    "type": "string",
+                    "enum": [
+                      "informational",
+                      "low",
+                      "medium",
+                      "high",
+                      "critical"
+                    ]
+                  },
+                  "reason": {
+                    "type": "string"
+                  }
+                },
+                "required": ["score", "reason"],
+                "additionalProperties": false
+              },
+              "overall_severity": {
+                "type": "string",
+                "enum": ["informational", "low", "medium", "high", "critical"]
+              }
+            },
+            "required": ["likelihood", "impact", "overall_severity"],
+            "additionalProperties": false
+          },
+          "confidence": {
+            "type": "object",
+            "properties": {
+              "score": {
+                "type": "string",
+                "enum": ["low", "medium", "high"]
+              },
+              "reason": {
+                "type": "string",
+                "description": "Why you scored the confidence this way. Mention any missing files, complex routing, or ambiguous data flows."
+              }
+            },
+            "required": ["score", "reason"],
+            "additionalProperties": false
+          }
+        },
+        "required": [
+          "verdict",
+          "title",
+          "description",
+          "root_cause",
+          "intended_behavior",
+          "trace",
+          "conditions",
+          "execution",
+          "remediation",
+          "severity",
+          "confidence"
+        ],
+        "additionalProperties": false
+      },
+      {
+        "type": "object",
+        "description": "Rejected finding — the described behavior is factually incorrect or the code path does not exist.",
+        "properties": {
+          "verdict": {
+            "type": "string",
+            "const": "rejected"
+          },
+          "reason": {
+            "type": "string",
+            "description": "Explain which specific claims in the finding are factually wrong (e.g., code path doesn't exist, mitigation prevents the described flow, trace is incorrect)."
+          }
+        },
+        "required": ["verdict", "reason"],
+        "additionalProperties": false
+      }
+    ]
+  }
+}

.agents/skills/security-audit/validate-findings.cjs

+#!/usr/bin/env node
+
+/**
+ * Validates findings.json against report-schema.json.
+ * Usage: node validate-findings.cjs <path-to-findings.json>
+ *
+ * The validation rules live in report-schema.json — the single source of truth.
+ * This script reads that schema at runtime and interprets the subset of JSON
+ * Schema it uses: type (object|array|string|integer), properties, required,
+ * additionalProperties:false, enum, const, items, minItems, and oneOf.
+ *
+ * Some constraints can't be expressed in that subset (a confirmed trace must
+ * start at an "entrypoint", end at a "sink", and only use "propagation" for
+ * intermediate steps). They're applied as an explicit, clearly-labelled
+ * semantic layer after schema validation.
+ *
+ * Zero dependencies. Exits 0 on success, 1 on validation failure.
+ */
+
+const fs = require("fs");
+const path = require("path");
+
+const file = process.argv[2];
+if (!file) {
+  console.error("Usage: node validate-findings.cjs <path-to-findings.json>");
+  process.exit(1);
+}
+
+const schemaPath = path.join(__dirname, "report-schema.json");
+let itemSchema;
+try {
+  const doc = JSON.parse(fs.readFileSync(schemaPath, "utf8"));
+  itemSchema = doc.output_schema;
+  if (!itemSchema)
+    throw new Error('report-schema.json is missing top-level "output_schema"');
+} catch (e) {
+  console.error(`Failed to load schema from ${schemaPath}:`, e.message);
+  process.exit(1);
+}
+
+let findings;
+try {
+  findings = JSON.parse(fs.readFileSync(file, "utf8"));
+} catch (e) {
+  console.error("Failed to parse JSON:", e.message);
+  process.exit(1);
+}
+
+if (!Array.isArray(findings)) {
+  console.error("findings.json must be an array");
+  process.exit(1);
+}
+
+// --- Generic JSON Schema interpreter (the subset used by report-schema.json) ---
+
+function typeOf(v) {
+  if (Array.isArray(v)) return "array";
+  if (v === null) return "null";
+  return typeof v; // "object" | "string" | "number" | "boolean"
+}
+
+// For oneOf: find a property defined with a `const` so error messages can point
+// at the intended branch (e.g. discriminate confirmed vs rejected by "verdict").
+function findDiscriminator(schema) {
+  if (!schema.properties) return null;
+  for (const [key, sub] of Object.entries(schema.properties)) {
+    if (sub && Object.prototype.hasOwnProperty.call(sub, "const")) {
+      return { key, value: sub.const };
+    }
+  }
+  return null;
+}
+
+function validate(value, schema, p, errors) {
+  if (schema.oneOf) {
+    // Prefer the branch whose const discriminator matches, so the caller sees
+    // detailed errors for the branch they clearly intended.
+    for (const branch of schema.oneOf) {
+      const disc = findDiscriminator(branch);
+      if (
+        disc &&
+        value &&
+        typeof value === "object" &&
+        value[disc.key] === disc.value
+      ) {
+        validate(value, branch, p, errors);
+        return;
+      }
+    }
+    // No discriminator matched. If every branch is discriminated by the same
+    // key, report the bad discriminator value clearly.
+    const discs = schema.oneOf.map(findDiscriminator).filter(Boolean);
+    if (
+      discs.length === schema.oneOf.length &&
+      value &&
+      typeof value === "object"
+    ) {
+      const key = discs[0].key;
+      const allowed = discs.map((d) => JSON.stringify(d.value)).join(", ");
+      errors.push(
+        `${p}: "${key}" must be one of ${allowed}, got ${JSON.stringify(value[key])}`,
+      );
+      return;
+    }
+    const passing = schema.oneOf.filter(
+      (b) => collect(value, b, p).length === 0,
+    );
+    if (passing.length !== 1) {
+      errors.push(`${p}: does not match exactly one of the allowed schemas`);
+    }
+    return;
+  }
+
+  if (
+    Object.prototype.hasOwnProperty.call(schema, "const") &&
+    value !== schema.const
+  ) {
+    errors.push(
+      `${p}: must equal ${JSON.stringify(schema.const)}, got ${JSON.stringify(value)}`,
+    );
+  }
+
+  if (schema.enum && !schema.enum.includes(value)) {
+    const allowed = schema.enum.map((v) => JSON.stringify(v)).join(", ");
+    errors.push(
+      `${p}: invalid value ${JSON.stringify(value)} (expected one of ${allowed})`,
+    );
+  }
+
+  switch (schema.type) {
+    case "object": {
+      if (typeOf(value) !== "object") {
+        errors.push(`${p}: expected object, got ${typeOf(value)}`);
+        return;
+      }
+      for (const req of schema.required || []) {
+        if (!(req in value))
+          errors.push(`${p}: missing required field "${req}"`);
+      }
+      for (const key of Object.keys(value)) {
+        if (schema.properties && key in schema.properties) {
+          validate(value[key], schema.properties[key], `${p}.${key}`, errors);
+        } else if (schema.additionalProperties === false) {
+          errors.push(`${p}: unexpected field "${key}"`);
+        }
+      }
+      break;
+    }
+    case "array": {
+      if (typeOf(value) !== "array") {
+        errors.push(`${p}: expected array, got ${typeOf(value)}`);
+        return;
+      }
+      if (
+        typeof schema.minItems === "number" &&
+        value.length < schema.minItems
+      ) {
+        errors.push(
+          `${p}: must have at least ${schema.minItems} item(s), got ${value.length}`,
+        );
+      }
+      if (schema.items) {
+        value.forEach((el, i) =>
+          validate(el, schema.items, `${p}[${i}]`, errors),
+        );
+      }
+      break;
+    }
+    case "integer": {
+      if (typeOf(value) !== "number" || !Number.isInteger(value)) {
+        errors.push(`${p}: expected integer, got ${typeOf(value)}`);
+      }
+      break;
+    }
+    case "string": {
+      if (typeOf(value) !== "string") {
+        errors.push(`${p}: expected string, got ${typeOf(value)}`);
+      }
+      break;
+    }
+    default:
+      break; // no type constraint at this node
+  }
+}
+
+function collect(value, schema, p) {
+  const errors = [];
+  validate(value, schema, p, errors);
+  return errors;
+}
+
+// --- Run ----------------------------------------------------------------------
+
+let errorCount = 0;
+
+findings.forEach((f, i) => {
+  const label = `[${i}] ${(f && (f.title || f.reason)) || "(untitled)"}`;
+  console.log(`Checking ${label}`);
+
+  const errs = collect(f, itemSchema, `[${i}]`);
+
+  // Semantic layer — constraints the schema subset can't express:
+  // a confirmed trace must be one entrypoint, zero or more propagation steps,
+  // then one sink.
+  if (
+    f &&
+    f.verdict === "confirmed" &&
+    Array.isArray(f.trace) &&
+    f.trace.length > 0
+  ) {
+    if (f.trace[0] && f.trace[0].kind !== "entrypoint") {
+      errs.push(
+        `[${i}].trace[0].kind must be "entrypoint", got ${JSON.stringify(f.trace[0].kind)}`,
+      );
+    }
+    const last = f.trace.length - 1;
+    if (f.trace[last] && f.trace[last].kind !== "sink") {
+      errs.push(
+        `[${i}].trace[${last}].kind must be "sink", got ${JSON.stringify(f.trace[last].kind)}`,
+      );
+    }
+    for (let j = 1; j < last; j++) {
+      if (f.trace[j] && f.trace[j].kind !== "propagation") {
+        errs.push(
+          `[${i}].trace[${j}].kind must be "propagation", got ${JSON.stringify(f.trace[j].kind)}`,
+        );
+      }
+    }
+  }
+
+  for (const msg of errs) console.error("  ERROR:", msg);
+  errorCount += errs.length;
+});
+
+console.log();
+if (errorCount === 0) {
+  console.log(`PASS: ${findings.length} findings valid`);
+} else {
+  console.error(
+    `FAIL: ${errorCount} error(s) across ${findings.length} findings`,
+  );
+  process.exit(1);
+}

.agents/skills/shop-amazon/SKILL.md

+---
+name: shop-amazon
+description: "Quy trình chuẩn để thu thập, bóc tách dữ liệu sản phẩm có cấu trúc từ sàn Amazon sử dụng Firecrawl và template extraction"
+---
+
+# Skill: Thu Thập & Bóc Tách Sản Phẩm Amazon (shop-amazon)
+
+Quy trình chuẩn dành cho Antigravity để tự động cấu hình và kích hoạt tác vụ cào dữ liệu sản phẩm từ sàn thương mại điện tử Amazon trong `data-crawler-be`.
+
+---
+
+## 1. Mục Đích & Phạm Vi Áp Dụng
+
+- Dùng khi cần cào danh sách sản phẩm hoặc thông tin chi tiết một sản phẩm trên Amazon (`amazon.com`, `amazon.co.jp`, v.v.).
+- Bóc tách các trường: Tiêu đề (`title`), Giá bán (`price`), Đánh giá (`rating`), Số lượng đánh giá (`reviewCount`), Ảnh sản phẩm (`mainImage`), Tình trạng (`availability`).
+
+---
+
+## 2. Cấu Hình Job Đề Xuất (Payload Mẫu)
+
+Gửi yêu cầu tới `POST /api/v1/crawl-jobs`:
+
+```json
+{
+  "startUrl": "https://www.amazon.com/s?k=laptop",
+  "mode": "CRAWL",
+  "maxPages": 25,
+  "maxDepth": 2,
+  "delayMs": 2500,
+  "timeoutMs": 45000,
+  "retryCount": 3,
+  "respectRobotsTxt": true,
+  "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
+}
+```
+
+> **Quy tắc chống chặn (Anti-bot):**
+>
+> - Amazon áp dụng cơ chế phát hiện bot rất mạnh. Luôn đặt `delayMs >= 2000`ms.
+> - Đảm bảo worker bắt cờ `CAPTCHA_DETECTED` hoặc `BLOCKED` trong bảng `crawl_pages` để cảnh báo kịp thời.
+
+---
+
+## 3. Extraction Template Cho Amazon
+
+Định nghĩa template trích xuất có cấu trúc cho domain `amazon.com`:
+
+```json
+{
+  "domain": "amazon.com",
+  "name": "Amazon Product Standard Template",
+  "fields": [
+    {
+      "name": "title",
+      "selector": "#productTitle, h2 a.a-link-normal span",
+      "type": "text",
+      "required": true
+    },
+    {
+      "name": "price",
+      "selector": ".a-price .a-offscreen, span.a-price-whole",
+      "type": "text",
+      "required": false
+    },
+    {
+      "name": "rating",
+      "selector": "span[data-hook='rating-out-of-text'], span.a-icon-alt",
+      "type": "text",
+      "required": false
+    },
+    {
+      "name": "reviewCount",
+      "selector": "#acrCustomerReviewText, span[data-hook='total-review-count']",
+      "type": "number",
+      "required": false
+    },
+    {
+      "name": "mainImage",
+      "selector": "#landingImage, .s-image",
+      "type": "attribute",
+      "attributeName": "src",
+      "required": false
+    },
+    {
+      "name": "availability",
+      "selector": "#availability span",
+      "type": "text",
+      "required": false
+    }
+  ]
+}
+```
+
+---
+
+## 4. Xuất Dữ Liệu Sau Khi Cào
+
+Sau khi Job đạt trạng thái `COMPLETED`:
+
+- Kích hoạt export sang Excel qua `POST /api/v1/exports` với `exportType: "XLSX"`.
+- Báo cáo kết quả và đường dẫn tải file xuất cho người dùng.

.claude/CLAUDE.md

+# CLAUDE.md - Bộ Não Dự Án (data-crawler-be)
+
+> Tài liệu hướng dẫn trung tâm dành cho Claude Agent khi làm việc trên mã nguồn **data-crawler-be**.
+
+---
+
+## 1. Tổng quan Dự Án (Project Overview)
+
+- **Tên dự án:** `data-crawler-be`
+- **Mục đích:** Dịch vụ Backend API cho hệ thống thu thập dữ liệu web (Data Crawler & Scraper). Hỗ trợ nhận URL, crawl website tự động (qua Firecrawl API hoặc scraper engine nội bộ), chuẩn hóa nội dung (HTML -> Markdown/Text), trích xuất structured data (sử dụng Extraction Templates), lưu trữ tài nguyên (assets) và xuất dữ liệu sang các định dạng `JSON`, `CSV`, `XLSX`, `MARKDOWN`, `ZIP`.
+- **Cơ chế xử lý:** Bất đồng bộ dựa trên hàng đợi **BullMQ + Redis**, worker phân luồng xử lý riêng biệt.
+- **Package Manager:** `pnpm@9.15.0` (tuân thủ nghiêm ngặt, không dùng `npm` hay `yarn`).
+
+---
+
+## 2. Kiến Trúc Cốt Lõi (Architectural Golden Rules)
+
+Mọi luồng xử lý dữ liệu nghiệp vụ bắt buộc phải tuân theo thứ tự phân tầng 5 lớp (Strict Layered Architecture):
+
+```
+Route  →  Controller  →  Service  →  Repository  →  Prisma Client  →  PostgreSQL
+```
+
+### Quy tắc bất di bất dịch:
+
+1. **Chỉ Repository được gọi Prisma:** Tuyệt đối **chỉ có** các file `*.repository.ts` được import `prisma` hoặc `PrismaClient`. Service, Controller, Worker hay Helper **không bao giờ** được gọi Prisma trực tiếp.
+2. **Cấu trúc Module chuẩn:** Mọi tính năng nghiệp vụ đặt tại `src/modules/<feature>/` với đầy đủ các file quy chuẩn:
+   - `<feature>.route.ts`: Định nghĩa endpoint, gắn middleware (auth, validate, rate-limit).
+   - `<feature>.controller.ts`: Xử lý HTTP request/response, bắt lỗi chuyển cho next hoặc dùng AppError.
+   - `<feature>.service.ts`: Xử lý logic nghiệp vụ thuần túy, gọi một hoặc nhiều repository.
+   - `<feature>.repository.ts`: Thao tác trực tiếp với cơ sở dữ liệu qua Prisma.
+   - `<feature>.dto.ts` & `<feature>.validation.ts`: Định nghĩa types và schema xác thực Zod.
+   - `__tests__/`: Unit/Integration tests sử dụng Jest.
+3. **Routing tập trung:** Tất cả route của module phải được đăng ký vào `src/routes/index.ts` và gắn prefix `/api/v1`.
+4. **Mã dùng chung:** Chỉ đưa vào `src/common/` khi code thực sự được tái sử dụng ở từ 2 module trở lên (errors, helpers, constants, types, storage drivers).
+
+---
+
+## 3. Quản Lý Cơ Sở Dữ Liệu & Prisma
+
+- **File nguồn chân lý:** `prisma/schema.prisma`.
+- **Cơ chế chạy Prisma:** Dự án dùng script bọc `scripts/prisma-run.js` để tự động tổng hợp `DATABASE_URL` từ các biến môi trường rời rạc (`DB_HOST`, `DB_PORT`, `DB_USER`, `DB_PASSWORD`, `DB_NAME`, `DB_SSL`).
+- **Lệnh migration an toàn:**
+  - `pnpm db:migrate`: Tạo và chạy migration trong môi trường development.
+  - `pnpm db:migrate:deploy`: Chạy migration trong môi trường production/CI.
+  - `pnpm prisma:generate`: Sinh lại Prisma Client sau khi sửa schema.
+  - `pnpm prisma:studio`: Mở giao diện xem dữ liệu Prisma.
+- **CẤM:** Tuyệt đối **không** tự ý chạy `pnpm db:migrate:reset` trừ khi người dùng yêu cầu rõ ràng việc xóa trắng dữ liệu local.
+
+---
+
+## 4. Hàng Đợi & Worker (BullMQ + Redis)
+
+- Các tác vụ nặng (crawl dữ liệu, chạy lịch biểu schedule, gửi webhook) đều đưa vào hàng đợi:
+  - `crawl.queue.ts` & `crawl.worker.processor.ts` (Worker: `crawl.worker.ts`)
+  - `webhook.queue.ts` & `webhook.worker.ts`
+  - `schedule.worker.ts`
+- **Quy tắc Worker:**
+  - Xử lý phải có tính **Idempotent** (chạy lại không sinh lỗi trùng lặp dữ liệu).
+  - Duy trì đúng vòng đời trạng thái của Job: `PENDING` -> `QUEUED` -> `RUNNING` -> `PROCESSING_EXPORT` -> `COMPLETED` / `FAILED` / `CANCELED`.
+  - Cập nhật log chi tiết vào bảng `crawl_job_logs` theo từng step.
+
+---
+
+## 5. Lưu Trữ & Hợp Đồng Dữ Liệu (Storage & Data Contract)
+
+- Hỗ trợ 2 driver lưu trữ linh hoạt qua `src/common/storage/`: `local` (thư mục `storage/exports/`) hoặc `s3` (AWS S3 / MinIO).
+- Tuân thủ hợp đồng dữ liệu chuẩn tại `docs/DATA_CONTRACT_V1.md`:
+  - Dữ liệu thô (raw): bảo toàn cấu trúc trả về từ Firecrawl / HTML ban đầu.
+  - Dữ liệu sạch (clean): Markdown chuẩn hóa, lọc thẻ rác, bóc tách metadata, tính toán `contentHash` và `dataQualityScore`.
+- **Không chỉnh sửa thủ công:**
+  - `dist/` (build output)
+  - `storage/exports/*` (runtime files)
+  - `src/docs/swagger.json` (sinh tự động qua `pnpm swagger`)
+
+---
+
+## 6. Lệnh Thường Dùng (Cheatsheet Commands)
+
+Chạy tất cả lệnh từ thư mục `data-crawler-be`:
+
+```bash
+# Development
+pnpm dev             # Chạy API server với ts-node-dev và tự động build swagger
+pnpm worker          # Chạy Crawl Worker riêng biệt
+pnpm swagger         # Sinh lại file docs Swagger từ route và controller
+
+# Quality & Testing
+pnpm lint            # Kiểm tra lỗi cú pháp ESLint
+pnpm format          # Format toàn bộ code bằng Prettier
+pnpm test            # Chạy toàn bộ test suite Jest (--runInBand)
+pnpm test -- <path>  # Chạy test cho một file/thư mục cụ thể
+
+# Build & Production
+pnpm build           # Build Swagger và compile TypeScript sang dist/
+pnpm start           # Chạy server production từ dist/server.js
+```
+
+---
+
+## 7. Quy Tắc Ứng Xử Của Agent (Do's & Don'ts)
+
+- **DO:** Luôn đọc kỹ code hiện có trước khi sửa đổi, kiểm tra tính tương thích type trong TypeScript.
+- **DO:** Viết Jest test tương ứng khi thêm logic mới vào Service, Worker, Repository hoặc Export pipeline.
+- **DO:** Dùng `AppError` kèm mã lỗi chuẩn từ `src/common/errors/error-code.ts`.
+- **DON'T:** Không bypass tầng Repository để query trực tiếp từ Service/Controller.
+- **DON'T:** Không commit file `.env`, `CLAUDE.local.md` hay `settings.local.json`.
+- **DON'T:** Không tự ý cài đặt thêm dependency nặng nếu thư viện sẵn có đã hỗ trợ.

.claude/agents/crawler-specialist.md

+---
+name: crawler-specialist
+description: "Chuyên gia kỹ thuật thu thập dữ liệu web, Firecrawl API, BullMQ worker và xử lý dữ liệu cào"
+tools:
+  - view_file
+  - replace_file_content
+  - multi_replace_file_content
+  - run_command
+---
+
+# Sub-Agent: Chuyên Gia Thu Thập Dữ Liệu Web (crawler-specialist.md)
+
+Bạn là **Crawler Specialist Sub-Agent** am hiểu sâu sắc về kiến trúc thu thập dữ liệu web, xử lý worker bất đồng bộ và chuẩn hóa dữ liệu trong **data-crawler-be**.
+
+---
+
+## 1. Phạm Vi Chuyên Môn
+
+1. **Firecrawl API Integration (`src/modules/firecrawl/`):**
+   - Cấu hình các mode: `SCRAPE`, `CRAWL`, `SITEMAP`, `URL_LIST`.
+   - Xử lý options: `includeTags`, `excludeTags`, `waitFor`, `mobile`, `skipTlsVerification`.
+   - Cơ chế phòng ngừa bị chặn (Anti-bot): thiết lập delay hợp lý giữa các request, tùy biến User-Agent.
+2. **Hàng Đợi & Phân Luồng Worker (`src/queues/`):**
+   - Tối ưu hóa xử lý đồng thời trong BullMQ worker (`concurrency`, `limiter`).
+   - Xử lý Retry với Exponential Backoff khi gặp sự cố mạng hoặc rate-limit từ website đích.
+   - Đảm bảo tính Idempotent của worker: nếu worker khởi động lại giữa chừng, không lưu trùng trang đã cào.
+3. **Chuẩn Hóa & Làm Sạch Dữ Liệu (`src/modules/crawl-pages/`):**
+   - Loại bỏ thẻ script, style, quảng cáo, iframe không cần thiết qua Cheerio / Node-HTML-Parser.
+   - Chuyển đổi mã nguồn HTML sang định dạng Markdown tối ưu cho LLM qua Turndown.
+   - Tính toán chỉ số chất lượng `dataQualityScore` và băm nội dung `contentHash` để phục vụ Change Detection.
+4. **Trích Xuất Định Dạng Cấu Trúc (`src/modules/extraction-templates/`):**
+   - Áp dụng các quy tắc CSS Selector và regex để bóc tách các trường cụ thể (tiêu đề, giá bán, mô tả, ảnh, thông số kỹ thuật).
+5. **Hệ Thống Xuất Dữ Liệu Đa Định Dạng (`src/modules/crawl-exports/`):**
+   - Quản lý pipeline xuất file `JSON`, `CSV`, `XLSX`, `MARKDOWN`, `ZIP`.
+   - Kết hợp lưu trữ cục bộ hoặc Cloud S3/MinIO qua Storage Driver.

.claude/agents/researcher.md

+---
+name: researcher
+description: "Chuyên gia nghiên cứu, phân tích kiến trúc và truy vết mã nguồn cho data-crawler-be"
+tools:
+  - view_file
+  - list_dir
+  - grep_search
+  - search_web
+---
+
+# Sub-Agent: Chuyên Gia Nghiên Cứu & Khảo Sát (researcher.md)
+
+Bạn là **Researcher Sub-Agent** chuyên trách việc điều tra, đọc hiểu kiến trúc và phân tích mã nguồn cho dự án **data-crawler-be**.
+
+---
+
+## 1. Mục Tiêu & Trách Nhiệm
+
+1. **Khảo sát hệ thống:** Đọc và hiểu cặn kẽ luồng dữ liệu hiện tại trước khi bất kỳ dòng code nào được thay đổi.
+2. **Truy vết luồng dữ liệu 5 lớp:**
+   - Theo dõi từ `Route` -> `Controller` -> `Service` -> `Repository` -> `Prisma Model`.
+   - Xác định rõ quan hệ cha-con, khóa ngoại, các index và rằng buộc trong `prisma/schema.prisma`.
+3. **Phân tích tác động (Impact Analysis):**
+   - Đánh giá xem việc thay đổi một bảng trong DB có ảnh hưởng đến các Worker nền (`crawl.worker.ts`, `schedule.worker.ts`, `webhook.worker.ts`) hay không.
+   - Kiểm tra tính tương thích của API đối với các client bên ngoài hoặc frontend (`data-crawler-fe`).
+4. **Tham chiếu hợp đồng dữ liệu:** Luôn đối chiếu với `docs/DATA_CONTRACT_V1.md` khi xem xét các thay đổi liên quan đến cấu trúc `CrawlPage` và `CrawlExport`.
+
+---
+
+## 2. Nguyên Tắc Hoạt Động (Rules of Engagement)
+
+- **Read-Only First:** Không thực hiện sửa đổi file hoặc chạy các lệnh làm thay đổi trạng thái hệ thống trong quá trình nghiên cứu.
+- **Dẫn chứng cụ thể:** Khi báo cáo phát hiện, luôn cung cấp đường dẫn file chính xác kèm số dòng liên quan.
+- **Đánh giá rủi ro:** Chỉ ra cụ thể các rủi ro tiềm ẩn (ví dụ: N+1 query trong Prisma, race condition trong worker concurrency, rò rỉ bộ nhớ khi export file lớn).
+
+---
+
+## 3. Mẫu Báo Cáo Phân Tích (Deliverable Template)
+
+Khi hoàn thành nghiên cứu, cung cấp kết quả theo định dạng:
+
+```markdown
+### Báo Cáo Khảo Sát Kỹ Thuật
+
+1. **Hiện trạng mã nguồn:** (Tóm tắt module và các file liên quan)
+2. **Luồng dữ liệu thực tế:** (Sơ đồ Route -> Service -> Repo -> DB)
+3. **Các điểm nghẽn / Rủi ro phát hiện:** (Chỉ rõ vị trí code cụ thể)
+4. **Đề xuất phương án thực hiện:** (Các bước cụ thể để triển khai an toàn)
+```

.claude/agents/reviewer.md

+---
+name: reviewer
+description: "Chuyên gia đánh giá chất lượng mã nguồn, kiến trúc và an toàn bảo mật cho data-crawler-be"
+tools:
+  - view_file
+  - grep_search
+  - run_command
+---
+
+# Sub-Agent: Chuyên Gia Đánh Giá Mã Nguồn (reviewer.md)
+
+Bạn là **Reviewer Sub-Agent** chịu trách nhiệm kiểm duyệt mọi thay đổi mã nguồn trước khi tích hợp vào nhánh chính của **data-crawler-be**.
+
+---
+
+## 1. Danh Sách Kiểm Tra Bắt Buộc (Review Checklist)
+
+### A. Tính Tuân Thủ Kiến Trúc (Architectural Compliance)
+
+- [ ] **Quy tắc Prisma độc quyền:** Chỉ duy nhất các file `*.repository.ts` được import `prisma` hoặc `PrismaClient`. Tuyệt đối không chấp nhận Prisma query trong Controller, Service, Worker hay Middleware.
+- [ ] **Phân tách tầng rõ ràng:** Controller không chứa logic tính toán nghiệp vụ; Service không can thiệp vào định dạng response HTTP (`res.status()`).
+- [ ] **Đăng ký Route:** Route mới đã được đăng ký vào `src/routes/index.ts` và có prefix hợp lệ `/api/v1/...`.
+
+### B. Tính Toàn Vẹn Dữ Liệu & Xác Thực (Validation & Type Safety)
+
+- [ ] **Xác thực Zod:** Toàn bộ Body, Query và Params phải đi qua `validate(schema)` middleware.
+- [ ] **Kiểu dữ liệu TypeScript:** Không dùng `any` bừa bãi. Sử dụng `z.infer<typeof schema>` cho các DTO.
+- [ ] **Xử lý lỗi:** Lỗi phải được ném ra qua `AppError` với `statusCode` và `ERROR_CODE` chuẩn mực, không dùng `throw new Error()`.
+
+### C. Hiệu Năng & Cơ Sở Dữ Liệu (Performance & Database)
+
+- [ ] **Tránh N+1 Query:** Khi truy vấn dữ liệu liên kết, phải sử dụng `include` hoặc `select` hợp lý thay vì gọi lặp lại trong vòng lặp `for`/`forEach`.
+- [ ] **Đúng chỉ mục (Index):** Các trường thường xuyên filter, sort (`status`, `createdAt`, `userId`, `domain`) phải có index trong `schema.prisma`.
+- [ ] **Xử lý Stream:** Các tác vụ export file dung lượng lớn bắt buộc phải dùng luồng (Stream) thay vì dồn toàn bộ vào RAM.
+
+### D. Kiểm Thử & Kiểm Định (Testing & Verification)
+
+- [ ] Đã bổ sung unit test tương ứng trong thư mục `__tests__/` liền kề.
+- [ ] Lệnh `pnpm lint` chạy không có cảnh báo nghiêm trọng hoặc lỗi cú pháp.
+- [ ] Lệnh `pnpm test -- --runInBand` chạy thành công 100%.
+
+---
+
+## 2. Tiêu Chuẩn Phản Hồi Khi Review
+
+Khi đưa ra nhận xét, Reviewer phải phân loại theo 3 mức độ:
+
+1. 🔴 **[BLOCKER]**: Vi phạm nghiêm trọng kiến trúc (ví dụ: Service gọi Prisma), lỗ hổng bảo mật, làm gãy test. Yêu cầu sửa ngay lập tức.
+2. 🟡 **[WARNING]**: Chưa tối ưu hiệu năng, thiếu test case biên hoặc chưa cập nhật Swagger. Cần cân nhắc xử lý.
+3. 🟢 **[SUGGESTION]**: Góp ý làm gọn code, đặt tên biến rõ nghĩa hơn hoặc cải thiện comment.

.claude/memory.md

+# memory.md - Bộ Nhớ Bền Vững Của Claude (Project Persistent Memory)
+
+> File lưu trữ ngữ cảnh kiến trúc, các quyết định kỹ thuật quan trọng và lưu ý đặc thù của dự án `data-crawler-be`.
+
+---
+
+## 1. Trạng Thái Hiện Tại Của Hệ Thống
+
+- **Các module đã hoàn thiện trong `src/modules/`:**
+  - `auth`: Đăng ký, đăng nhập JWT (access token trong cookie/header, refresh token lưu database), đổi mật khẩu.
+  - `users`: Quản lý người dùng và quota (giới hạn số trang, số job mỗi ngày, số job đồng thời).
+  - `crawl-jobs`: Tạo job, xem danh sách, chi tiết tiến độ, hủy job, retry job.
+  - `crawl-pages`: Lưu trữ các trang đã cào được (`CrawlPage`), kiểm tra chất lượng nội dung, bóc tách `structuredData`, cảnh báo dữ liệu nhạy cảm.
+  - `crawl-assets`: Lưu hình ảnh, liên kết, PDF, file đính kèm tìm thấy trong các trang.
+  - `crawl-schedules`: Lập lịch cào định kỳ với cron expressions và tần suất (DAILY, WEEKLY, MONTHLY, CUSTOM), hỗ trợ so sánh khác biệt (`autoDiff`).
+  - `change-detection`: So sánh diff giữa các phiên bản cào để phát hiện nội dung trang bị thay đổi.
+  - `extraction-templates`: Quản lý template trích xuất theo domain (CSS selector/JSON field mapping).
+  - `crawl-exports`: Xuất kết quả cào sang `JSON`, `CSV`, `XLSX`, `MARKDOWN`, `ZIP`.
+  - `firecrawl`: Wrapper tích hợp Firecrawl API (hỗ trợ scrape single page, crawl whole site, map sitemap, url list).
+  - `webhooks`: Đăng ký URL webhook và phân phối kết quả (HMAC signature, retry attempts).
+  - `audit-logs`: Ghi nhận lịch sử thao tác của người dùng.
+  - `api-keys`: Quản lý API Key cho client bên thứ 3.
+  - `health`: Kiểm tra sức khỏe dịch vụ, kết nối DB và Redis.
+
+---
+
+## 2. Các Quyết Định Kỹ Thuật Quan Trọng (Key Architectural Decisions)
+
+1. **Prisma Connection Pooling & DATABASE_URL:**
+   - Script `scripts/prisma-run.js` chịu trách nhiệm tạo chuỗi `DATABASE_URL` động nếu chưa có trong biến môi trường. Hỗ trợ tự động nhận diện Supabase pooler và SSL.
+2. **Strict Layering Constraint:**
+   - Controller chỉ nhận request, gọi Service.
+   - Service chỉ chứa business logic, điều phối các Repository.
+   - Chỉ `*.repository.ts` được dùng Prisma. Tuyệt đối không query Prisma trong Service/Controller.
+3. **Queue Processing Workflow:**
+   - Khi API nhận request cào -> Tạo record `CrawlJob` (status `PENDING` -> `QUEUED`) -> Đẩy vào `crawl.queue.ts`.
+   - `crawl.worker.processor.ts` lắng nghe:
+     - Chuyển status `RUNNING`.
+     - Gọi Firecrawl SDK để lấy nội dung.
+     - Phân tích HTML/Markdown, bóc tách link/assets, đánh giá quality score.
+     - Lưu các bản ghi `CrawlPage` và `CrawlAsset`.
+     - Tự động kích hoạt export nếu người dùng cấu hình export tự động.
+     - Chuyển status sang `COMPLETED` (hoặc `FAILED` nếu có lỗi không thể phục hồi).
+4. **Export Stream:**
+   - Dùng stream với `archiver` và `exceljs` để tránh tràn bộ nhớ Node.js (out-of-memory) khi dữ liệu cào lên tới hàng trăm nghìn trang.
+
+---
+
+## 3. Các "Bẫy Kỹ Thuật" Cần Tránh (Gotchas & Warnings)
+
+- **Helmet CSP và Swagger:** Helmet mặc định bật CSP khiến giao diện Swagger UI bị chặn load CSS. Trong `app.ts` đã tắt CSP riêng cho Swagger (`contentSecurityPolicy: false`).
+- **Prisma Cascade Delete:** Quan hệ giữa `CrawlJob` với `CrawlPage`, `CrawlExport`, `CrawlJobLog` là `Cascade`. Xóa job sẽ xóa hết các trang liên quan.
+- **Trust Proxy:** Cần gọi qua helper `parseTrustProxy` để rate limiting và lấy IP người dùng chính xác khi chạy sau Reverse Proxy/Nginx/Cloudflare.

.claude/rules/design.md

+# Quy Tắc Thiết Kế Kiến Trúc (design.md)
+
+> Quy chuẩn thiết kế phần mềm, cấu trúc các tầng và chuẩn mực mã nguồn của hệ thống `data-crawler-be`.
+
+---
+
+## 1. Kiến Trúc 5 Lớp (5-Layer Pattern)
+
+Hệ thống áp dụng kiến trúc phân lớp hướng dịch vụ (Layered Clean Architecture):
+
+```
+┌────────────────────────────────────────┐
+│ 1. Route (src/modules/<feature>/*.route.ts)
+│    - Gắn URI, HTTP Method, Middleware (Auth, Validate, Rate-limit)
+└───────────────────┬────────────────────┘
+                    │
+┌───────────────────▼────────────────────┐
+│ 2. Controller (src/modules/<feature>/*.controller.ts)
+│    - Tiếp nhận Request, trích xuất Params/Body/Query
+│    - Gọi Service tương ứng
+│    - Định dạng Response chuẩn HTTP (200, 201, 204...)
+└───────────────────┬────────────────────┘
+                    │
+┌───────────────────▼────────────────────┐
+│ 3. Service (src/modules/<feature>/*.service.ts)
+│    - Chứa Business Logic, kiểm tra Quota, nghiệp vụ cào dữ liệu
+│    - Gọi một hoặc nhiều Repository
+│    - Đẩy job vào BullMQ nếu là tác vụ bất đồng bộ
+└───────────────────┬────────────────────┘
+                    │
+┌───────────────────▼────────────────────┐
+│ 4. Repository (src/modules/<feature>/*.repository.ts)
+│    - Chịu trách nhiệm DUY NHẤT về việc truy vấn cơ sở dữ liệu
+│    - Định nghĩa Prisma select, include, pagination, filter
+└───────────────────┬────────────────────┘
+                    │
+┌───────────────────▼────────────────────┐
+│ 5. Database (PostgreSQL via Prisma Client)
+└────────────────────────────────────────┘
+```
+
+---
+
+## 2. Xác Thực Dữ Liệu Tại Biên (Boundary Validation with Zod)
+
+- Mọi dữ liệu đầu vào từ người dùng (Body, Query, Params) **bắt buộc** phải được định nghĩa Schema bằng **Zod** trong `<feature>.validation.ts`.
+- Sử dụng middleware dùng chung `validateMiddleware`:
+  ```typescript
+  import { validate } from "../../middlewares/validate.middleware";
+  import { createCrawlJobSchema } from "./crawl-job.validation";
+
+  router.post("/", validate(createCrawlJobSchema), crawlJobController.create);
+  ```
+- Định nghĩa kiểu TypeScript tương ứng (`DTO`) bằng `z.infer<typeof schema>` trong `<feature>.dto.ts`.
+
+---
+
+## 3. Xử Lý Lỗi Tập Trung (Centralized Error Handling)
+
+- Không dùng `throw new Error("...")` một cách tùy tiện.
+- Bắt buộc kế thừa từ `AppError`:
+  ```typescript
+  import { AppError } from "../../common/errors/app-error";
+  import { ERROR_CODE } from "../../common/errors/error-code";
+
+  if (!job) {
+    throw new AppError("Crawl job not found", 404, ERROR_CODE.NOT_FOUND);
+  }
+  ```
+- Cấu trúc response trả về cho client luôn thống nhất:
+  ```json
+  {
+    "success": false,
+    "message": "Chi tiết lỗi",
+    "code": "ERROR_CODE_ENUM",
+    "errors": [] // (nếu là lỗi validate form)
+  }
+  ```
+
+---
+
+## 4. Hợp Đồng Dữ Liệu Cào (Data Contract V1)
+
+Bảo toàn hợp đồng dữ liệu quy định tại `docs/DATA_CONTRACT_V1.md`:
+
+- **Trang Cào (`CrawlPage`):**
+  - `url`, `normalizedUrl`: URL chuẩn hóa (loại bỏ tracking query params không cần thiết).
+  - `markdownContent`: Nội dung chính sau khi dọn sạch thẻ rác HTML.
+  - `structuredData`: Dữ liệu bóc tách dựa trên Extraction Template.
+  - `dataQualityScore`: Điểm chất lượng nội dung (tính dựa trên tỷ lệ text/tag, mật độ từ).
+  - `hasSensitiveData`: Cờ phát hiện email/số điện thoại/khóa API nhạy cảm.
+
+---
+
+## 5. Trừu Tượng Hóa Tầng Lưu Trữ (Storage Abstraction)
+
+- Mọi thao tác ghi file kết quả export và lưu trữ asset phải đi qua interface lưu trữ chung tại `src/common/storage/`:
+  - `LocalStorageProvider`: Lưu file tại ổ đĩa server (`storage/exports/`).
+  - `S3StorageProvider`: Tải file trực tiếp lên AWS S3 hoặc MinIO tương thích S3.
+- Không hardcode đường dẫn file vật lý trong tầng Controller hoặc Service.

.claude/rules/tech-defaults.md

+# Công Nghệ Mặc Định & Cấu Hình Chuẩn (tech-defaults.md)
+
+> Danh sách công nghệ cốt lõi, phiên bản và các quy ước mặc định được áp dụng trong dự án `data-crawler-be`.
+
+---
+
+## 1. Môi Trường & Phiên Bản Chuẩn (Core Runtime)
+
+| Thành phần          | Phiên bản / Thư viện | Ghi chú quy ước                                 |
+| :------------------ | :------------------- | :---------------------------------------------- |
+| **Node.js**         | `>= 20.x`            | Sử dụng cú pháp ES2022+ hiện đại                |
+| **Package Manager** | `pnpm@9.15.0`        | Không dùng npm hay yarn để tránh lệch pnpm-lock |
+| **Ngôn ngữ**        | `TypeScript 5.7`     | `strict: true`, không dùng kiểu `any` vô căn cứ |
+| **Web Framework**   | `Express.js 4.21`    | Tách rời App configuration và Server listener   |
+
+---
+
+## 2. Cơ Sở Dữ Liệu & Bộ Nhớ Đệm (Database & Caching)
+
+- **Database:** PostgreSQL (chạy Docker local qua `docker-compose.yml` trên port `5432`).
+- **ORM:** Prisma `5.22.x`
+  - Đặt tên model dạng `PascalCase`, ánh xạ bảng dạng `snake_case` thông qua `@@map("table_name")`.
+  - Khóa chính luôn là UUID v4 (`@id @default(uuid()) @db.Uuid`).
+  - Luôn định nghĩa `createdAt` (`@default(now())`) và `updatedAt` (`@updatedAt`).
+- **Cache & Queue Broker:** Redis `7.x` (chạy trên port `6379`)
+  - Kết nối thông qua thư viện `ioredis` và `bullmq`.
+
+---
+
+## 3. Crawl Engine & Phân Tích Nội Dung (Crawl Stack)
+
+- **Engine thu thập:** `@mendable/firecrawl-js ^1.19.0`
+  - Chế độ cào linh hoạt: `SCRAPE` (trang đơn), `CRAWL` (toàn bộ website theo độ sâu `maxDepth`), `SITEMAP` (dựa trên sơ đồ trang), `URL_LIST` (danh sách URL rời).
+- **Phân tích DOM & Làm sạch nội dung:**
+  - `cheerio ^1.2.0` & `node-html-parser ^8.0.4`: Bóc tách DOM, tìm kiếm selector nhanh.
+  - `turndown ^7.2.0`: Chuyển đổi mã nguồn HTML đã làm sạch thành Markdown dễ đọc cho LLM.
+
+---
+
+## 4. Xuất Dữ Liệu & Lưu Trữ (Export & Storage)
+
+- **Các định dạng hỗ trợ:**
+  - `JSON`: JSON mảng các trang thu thập được.
+  - `CSV`: Tạo bằng `json2csv ^6.0.0-alpha.2`.
+  - `XLSX`: Tạo bằng `exceljs ^4.4.0` (hỗ trợ phân tab, tự động tính độ rộng cột).
+  - `MARKDOWN`: Xuất file `.md` từng trang.
+  - `ZIP`: Đóng gói toàn bộ file xuất kèm thư mục assets bằng `archiver ^7.0.1`.
+- **Cloud Storage:** `@aws-sdk/client-s3` và `@aws-sdk/lib-storage` (tương thích AWS S3 và MinIO).
+
+---
+
+## 5. Bảo Mật & Xác Thực (Security & Auth)
+
+- **Mã hóa mật khẩu:** `bcryptjs ^2.4.3` với salt round chuẩn = 10.
+- **Token:** `jsonwebtoken ^9.0.2` (Access token thời hạn 1 ngày, Refresh token 7 ngày lưu database).
+- **Bảo vệ HTTP:** `helmet ^8.0.0` (tắt CSP cho Swagger), `cors ^2.8.5`, `express-rate-limit ^8.5.2`.
+- **Phân quyền (RBAC):**
+  - `ADMIN`: Toàn quyền hệ thống, quản lý người dùng, xem toàn bộ log.
+  - `CRAWLER_USER`: Người dùng tiêu chuẩn, có thể tạo job cào và xuất file theo quota.
+  - `VIEWER`: Chỉ xem danh sách job và tải dữ liệu đã cào sẵn.

.claude/rules/workflow.md

+# Quy Tắc Quy Trình Phát Triển (workflow.md)
+
+> Quy định về các bước làm việc, quy chuẩn Git, cơ sở dữ liệu và kiểm thử chất lượng mã nguồn trong `data-crawler-be`.
+
+---
+
+## 1. Quy Trình Phát Triển Tính Năng Mới (Feature Lifecycle)
+
+Khi xây dựng một module hoặc tính năng mới, Agent cần thực hiện tuần tự theo 6 bước:
+
+```
+[1. Khảo sát Schema & Yêu Cầu]
+        ↓
+[2. Viết Repository (Prisma Query)]
+        ↓
+[3. Viết Service (Business Logic & Validation)]
+        ↓
+[4. Viết Controller & Định nghĩa Route]
+        ↓
+[5. Đăng ký Route vào index.ts & Viết Swagger]
+        ↓
+[6. Viết Jest Test & Chạy Lint]
+```
+
+---
+
+## 2. Quy Chuẩn Git & Commit Message
+
+Tuân thủ chuẩn **Conventional Commits**:
+
+- `feat(module):` Thêm tính năng mới (ví dụ: `feat(crawl-jobs): add priority queue option`)
+- `fix(module):` Sửa lỗi (ví dụ: `fix(auth): handle expired refresh token race condition`)
+- `refactor(module):` Tái cấu trúc mã mà không đổi hành vi nghiệp vụ
+- `test(module):` Thêm hoặc cập nhật unit tests
+- `docs(module):` Cập nhật tài liệu, Swagger, README
+- `chore:` Thay đổi cấu hình build, dependencies, tooling
+
+---
+
+## 3. Quy Trình Làm Việc Với Cơ Sở Dữ Liệu (Prisma Migration Workflow)
+
+Khi cần thay đổi cấu trúc bảng hoặc thêm trường mới:
+
+1. Chỉnh sửa schema tại `prisma/schema.prisma`.
+2. Tạo và áp dụng migration:
+   ```bash
+   pnpm db:migrate
+   # Hoặc nếu là lần đầu: pnpm db:migrate:init
+   ```
+3. Sinh lại Prisma Client để cập nhật kiểu dữ liệu TypeScript:
+   ```bash
+   pnpm prisma:generate
+   ```
+4. Kiểm tra trạng thái migration:
+   ```bash
+   pnpm db:migrate:status
+   ```
+5. **QUAN TRỌNG:** Commit cả file `schema.prisma` lẫn thư mục migration mới được sinh ra trong `prisma/migrations/`.
+
+---
+
+## 4. Quy Trình Kiểm Thử & Kiểm Tra Chất Lượng (QA & Validation)
+
+Trước khi coi một tác vụ là hoàn thành, Agent bắt buộc phải chạy các bước kiểm tra sau:
+
+```bash
+# 1. Kiểm tra định dạng và quy chuẩn cú pháp:
+pnpm lint
+
+# 2. Cập nhật tài liệu API:
+pnpm swagger
+
+# 3. Chạy test suite:
+pnpm test -- --runInBand
+
+# 4. Kiểm tra khả năng build production:
+pnpm build
+```
+
+---
+
+## 5. Quy Trình Bổ Sung Queue Worker
+
+Khi tạo thêm Worker xử lý tác vụ nền:
+
+1. Tạo Queue tại `src/queues/<job-name>.queue.ts`.
+2. Tạo Processor xử lý logic tại `src/queues/<job-name>.worker.processor.ts`.
+3. Khởi tạo Worker file tại `src/queues/<job-name>.worker.ts`.
+4. Đảm bảo cấu hình retry với exponential backoff và cơ chế log lỗi vào cơ sở dữ liệu.

.claude/settings.json

+{
+  "$schema": "https://json.schemastore.org/partial-claude-code-settings.json",
+  "version": "1.0",
+  "permissions": {
+    "allowAutoExecution": [
+      "pnpm lint",
+      "pnpm format",
+      "pnpm swagger",
+      "pnpm test",
+      "pnpm build",
+      "pnpm prisma:generate",
+      "node scripts/prisma-run.js migrate status"
+    ],
+    "denyExecution": [
+      "pnpm db:migrate:reset",
+      "rm -rf *",
+      "git push --force",
+      "git reset --hard"
+    ],
+    "fileProtection": {
+      "readOnly": [
+        "pnpm-lock.yaml",
+        "dist/**",
+        "storage/exports/**",
+        "src/docs/swagger.json"
+      ]
+    }
+  },
+  "hooks": {
+    "postEdit": [
+      {
+        "pattern": "src/modules/**/*.route.ts",
+        "command": "pnpm swagger",
+        "description": "Tự động sinh lại Swagger JSON khi có route thay đổi"
+      },
+      {
+        "pattern": "prisma/schema.prisma",
+        "command": "pnpm prisma:generate",
+        "description": "Tự động sinh lại Prisma Client khi schema thay đổi"
+      }
+    ],
+    "preCommit": [
+      {
+        "command": "pnpm lint",
+        "description": "Kiểm tra cú pháp và quy chuẩn code trước khi commit"
+      },
+      {
+        "command": "pnpm test -- --runInBand",
+        "description": "Chạy unit test để đảm bảo không làm gãy tính năng"
+      }
+    ]
+  },
+  "environment": {
+    "NODE_ENV": "development",
+    "PAGER": "cat"
+  }
+}

.claude/skills/security-audit/CLIENT-SIDE.md

+# Client-Side and Browser Hunting
+
+#### When to use this file
+
+Reach for this file when meaningful trust decisions or untrusted rendering happen in the browser: single-page apps, browser extensions, embedded webviews, and anything that renders attacker-influenceable content into the DOM, receives cross-window messages, opens WebSockets, or serves credentialed cross-origin responses. These bugs live in code the server never executes — the fragment after `#`, `window.name`, a `postMessage` payload — so server-side escaping and the classes in `ATTACK-CLASSES.md` don't cover them.
+
+Use alongside `ATTACK-CLASSES.md`. The injection class there covers server-side sinks; this file covers the browser-side source→sink paths, cross-origin trust, and UI-redress classes that only exist client-side.
+
+Pick the relevant classes based on Phase 1. Split per surface (DOM rendering, message/WebSocket handlers, auth-carrying endpoints) for large front-ends.
+
+## Core discipline (include in every agent prompt for this domain)
+
+```
+- Client-side taint needs a controllable SOURCE and an executing SINK on the client path. A source with no sink, or a sink fed only server-rendered trusted data, is not a finding. Name both and show untrusted data reaching the sink unsanitized.
+- The impact must cross to a victim or cross an origin. XSS in the attacker's own DOM, or a "leak" of the attacker's own data, is not a finding. State whose session executes it or whose cross-origin data it steals.
+- Framework auto-escaping is a real mitigation. React/Vue/Angular escape interpolation by default — the finding is where the code opts OUT (`dangerouslySetInnerHTML`, `v-html`, `bypassSecurityTrust*`, `$sce.trustAs*`). Do not report escaped interpolation.
+- A missing header or attribute (X-Frame-Options, frame-ancestors, rel=noopener, SameSite) is only a finding with a concrete sensitive action behind it. A bare missing flag with no state-changing action or credentialed cross-origin read is a hardening note.
+```
+
+## DOM-based injection attack classes (subagent_type: `general`)
+
+**DOM-based XSS**
+Trace client-side sources — `location.hash`/`search`/`href`/`pathname`, `document.referrer`, `window.name`, `postMessage` data, `document.cookie` — into execution sinks: `innerHTML`/`outerHTML`, `document.write`, `eval`, `Function`, `setTimeout`/`setInterval` with a string argument, `element.src`/`href` set to a `javascript:` URI, jQuery `$(...)`/`.html()`, or framework escape hatches (`dangerouslySetInnerHTML`, `v-html`, `bypassSecurityTrustHtml`). The bug is source→sink with no sanitization _on the client path_; server-side escaping never sees fragment or `window.name` data.
+
+**DOM clobbering**
+Attacker-injected `id`/`name` attributes — surviving an HTML sanitizer that strips script but allows attributes — that shadow a global the script later reads (`window.config`, a `form.action`, a flag checked before initialization). Look for code reading `window.X`/`document.X` that an injected element named `X` can override. Requires a markup-injection sink that permits `id`/`name`.
+
+## Client-side trust and messaging attack classes (subagent_type: `general`)
+
+**postMessage origin trust**
+A `message` handler that acts on `event.data` (writes the DOM, calls a privileged function, stores a token) without checking `event.origin` against an allowlist, or with a weak check (`indexOf`, `startsWith`, unanchored regex, `endsWith` on the host). Also the send side: `postMessage(data, '*')` leaking data to any embedder. Confirm the handler does something security-relevant with the data.
+
+**Cross-site WebSocket hijacking (CSWSH)**
+A WebSocket handshake authenticated only by ambient cookies, with no `Origin` check and no per-session CSRF token — an attacker page opens a socket in the victim's authenticated context and reads/writes their data. Find the upgrade handler; check whether it validates `Origin` and binds to a token, not just the cookie.
+
+**CORS with credentials**
+A server that reflects the request `Origin` into `Access-Control-Allow-Origin` while sending `Access-Control-Allow-Credentials: true`, or allowlists `null` or a weak suffix match — any origin then reads authenticated responses. The finding is reflection or weak-match _with credentials_, not a wildcard alone (`*` with credentials is rejected by browsers).
+
+## UI-redress and navigation attack classes (subagent_type: `general`)
+
+**Clickjacking**
+A state-changing action (transfer, delete, grant, confirm) reachable in a framed page with no `X-Frame-Options: DENY`/`SAMEORIGIN` and no `frame-ancestors` CSP and no UI framebusting. A missing frame guard on a read-only page with no sensitive action is not a finding — require the action.
+
+**Reverse tabnabbing**
+A link whose target is attacker-influenceable, opened with `target="_blank"`, letting the opened page rewrite `window.opener.location` to a phishing origin. Modern browsers imply `noopener` for `target="_blank"`, so this is a finding only where the code sets `rel="opener"` explicitly, uses `window.open` without `noopener`, or the threat model includes older browsers — check before reporting.
+
+**Client-side open redirect / navigation**
+A navigation built from a client source (`location = params.get('next')`, `location.hash` fed into `location.href`, a router redirect) with no allowlist — including `javascript:`/`data:` schemes that promote the redirect into XSS. Distinct from a server open-redirect: the sink is in JS, so the server never sees it.
+
+## Prototype pollution attack classes (subagent_type: `general`)
+
+**Prototype pollution and gadget chain**
+An attacker-controlled key (`__proto__`, `constructor.prototype`) reaching a _nested/recursive_ write — a deep merge, `lodash.set`-style path assignment, `obj[a][b]=v` with an attacker-controlled segment, or a query-string parser that builds nested objects — that lands on `Object.prototype`. A plain `JSON.parse` or shallow `Object.assign` does NOT pollute. Require the recursive sink AND a gadget that reads the polluted property (an options object checked with `opts.isAdmin`, a template reading a config default, a sink that concatenates a polluted `src`). Pollution with no reachable gadget is not exploitable; the gadget is what turns it into XSS, auth bypass, or (in Node) RCE.
+
+## Universal moves (apply across the above)
+
+- **Start from the sink and walk back to a client source.** Grep the execution sinks (`innerHTML`, `eval`, `document.write`, `dangerouslySetInnerHTML`, `postMessage`, `new WebSocket`) and trace each argument back to `location`/`name`/`referrer`/message data. A sink fed only server-rendered trusted data is not a finding.
+- **Server escaping ends where the fragment begins.** Data after `#`, plus `window.name` and cross-window messages, never reaches the server — so server-side filters can't see it. That blind spot is the DOM-XSS goldmine.
+- **Enumerate the escape hatches.** In an auto-escaping framework, the candidate list _is_ every `dangerouslySetInnerHTML`/`v-html`/`bypassSecurityTrust*`/`$sce.trustAs*` call. Start there.
+
+## Validation rules (apply before reporting ANY finding here)
+
+1. **Confirm a controllable source AND an executing sink on the client path.** Cite the source (`location.hash`, `event.data`, `window.name`) and the sink (`innerHTML`, `eval`, navigation), and show untrusted data reaching the sink without sanitization. A source with no sink, or a sink fed only trusted data, is not a finding.
+2. **For prototype pollution, prove the recursive write AND a gadget.** Show the nested/recursive assignment that reaches `Object.prototype`, then the code that later reads the polluted property to a security-relevant effect. Pollution with no reachable gadget is not exploitable.
+3. **For messaging / CORS / WebSocket, show the origin check is absent or weak.** Cite the handler and the missing or `indexOf`/`startsWith`/unanchored-regex origin check, and that the data drives a security-relevant action or a credentialed cross-origin read. Reflection plus credentials, not a bare wildcard.
+4. **For UI-redress, require the sensitive action behind the missing guard.** Name the state-changing action that gets framed (clickjacking) or the attacker-controlled `_blank` link (tabnabbing). A missing `X-Frame-Options`/`rel=noopener` with nothing sensitive behind it is a hardening note — and framebusting, `frame-ancestors`, or the browser's `noopener` default may already defeat it. Check before reporting.
+5. **Return ONLY confirmed findings** with the client source→sink path and whose session it fires in — or "No exploitable client-side issues found" if that's honest.

.claude/skills/security-audit/HUNTING.md

+# Vulnerability Hunting
+
+### Phase 2: Hunt for vulnerabilities
+
+Launch **multiple `general` agents in parallel** via the Task tool. Use `general`, not `research` — general agents can spawn their own sub-agents via the Task tool, so when a hunter finds a rabbit hole that needs deeper investigation (e.g., tracing injection into an auth subsystem it doesn't fully understand), it can spin up a focused `research` sub-agent rather than trying to do everything in one context window.
+
+Each agent gets the architecture summary from Phase 1 injected into its prompt plus the hunting methodology and validation rules. Launch them in a single message so they run concurrently.
+
+**How many agents?** Use Phase 1 to decide. More focused agents produce better results than broad ones that run out of context. For a small library, 3-4 agents may suffice. For a large application with distinct subsystems, launch 8-12+ — split by attack class AND by subsystem. If Phase 1 revealed an auth system, a plugin system, a media pipeline, and a comment engine, each of those could warrant its own injection agent, its own logic agent, etc.
+
+Every agent prompt MUST include:
+
+1. The architecture summary from Phase 1 (copy it in verbatim)
+2. The specific attack class and scope to investigate
+3. Relevant file paths from Phase 1 as starting points
+4. The hunting methodology (below)
+5. The validation rules (below)
+
+#### Hunting methodology — include in every Phase 2 agent prompt
+
+Tell each agent to think like an attacker, not a code reviewer:
+
+```
+## How to hunt
+
+Don't just check if defenses exist. Try to break them.
+
+READ THE CODE AT DEPTH. Don't stop at the first function. Follow the data through
+every layer — from the entry point through validation, transformation, storage, retrieval,
+and output. Bugs live in the gaps between layers.
+
+Think about these angles:
+
+1. THE HAPPY PATH IS DEFENDED. ATTACK THE SAD PATH.
+   Error handlers, fallback branches, catch blocks, default cases, timeout paths,
+   retry logic, cleanup routines. What happens when things fail? Are errors handled
+   with the same rigor as success? Does a failed validation leave state half-modified?
+
+2. WHAT HAPPENS AT BOUNDARIES?
+   Empty input. Maximum-length input. Null vs undefined vs missing. Zero. Negative numbers.
+   Unicode edge cases. The first item and the last item. One more than the maximum. Exactly
+   at the rate limit. The moment a token expires.
+
+3. WHAT DO COMPONENTS ASSUME ABOUT EACH OTHER?
+   Does the database layer assume the API layer validated input? Does the renderer assume
+   content was sanitized on write? Does the auth middleware assume routes register themselves
+   correctly? Find where trust is implicit and test whether it's justified.
+
+4. WHAT IF OPERATIONS HAPPEN IN THE WRONG ORDER?
+   Call step 3 before step 1. Call delete during create. Send the callback before the request.
+   Hit the confirmation endpoint without starting the flow. Replay a completed flow.
+
+5. WHAT IF TWO THINGS HAPPEN AT ONCE?
+   Two requests to the same resource. Modify while reading. Delete while iterating.
+   Publish while someone else is editing. Two users claiming the same unique resource.
+
+6. WHERE DO TWO PARSERS OR VALIDATORS DISAGREE?
+   Input accepted by the schema but rejected by the database. URL parsed differently by
+   the router vs the application code. Content-type header says one thing, body is another.
+   Filename extension vs MIME type vs magic bytes.
+
+7. WHAT SURVIVES A ROUND TRIP?
+   Data stored then retrieved — is it the same? Does encoding change? Does escaping
+   double-up? Is a relative path resolved differently on read vs write? Does serialization
+   lose type information?
+
+8. WHAT DOES THE CONFIGURATION CONTROL?
+   What happens when config is missing or default? Can an environment variable override a
+   security control? Does a feature flag disable validation? What's the security posture
+   during setup/first-run before config is complete?
+
+9. FOLLOW THE MONEY (OR THE PRIVILEGE).
+   For every operation that changes state, ask: who authorized this? Trace back to the
+   permission check. Is it checking the right permission? Is it checking against the right
+   resource? Is there a parallel path to the same state change that checks differently
+   or not at all?
+
+10. LOOK FOR LEAKED CONTEXT.
+    Error messages that reveal internal paths. Stack traces in production. Timing differences
+    that reveal whether a record exists. Response size differences. HTTP headers that
+    disclose versions. Debug endpoints that survived into production.
+
+11. WHAT PARAMETERS OVERRIDE SECURITY-RELEVANT DEFAULTS?
+    Where a default is safe but a user-supplied parameter can change it. Look for
+    every input that overrides a security-relevant default and check if the override
+    is gated by appropriate permissions.
+
+12. WHERE DO UNVERIFIED CLAIMS DRIVE TRUST DECISIONS?
+    Anywhere self-declared identity, capability, or metadata influences an access
+    or trust decision without independent verification.
+
+GO DEEP, AND PROVE IT. You can spawn sub-agents: if evaluating a candidate finding needs
+deep understanding of a subsystem, use the Task tool to launch a research agent instead of
+holding everything in one context. And where the code is locally runnable, don't just reason
+about it — extract the suspect function into a minimal harness (or build and run the target)
+and test the hypothesis directly. A reproduced result beats an argued one.
+
+YOUR SCOPE IS YOUR PRIMARY FOCUS, NOT A BOUNDARY.
+If while investigating your assigned area you notice something wrong in a different
+category — a permission issue while tracing injection, a race condition while reviewing
+auth — report it. Don't ignore a bug because it's "not your area." Attackers don't
+respect category boundaries.
+
+## Validation rules — apply before reporting ANY finding
+1. You MUST construct a concrete attack (exact inputs, requests, or action sequence)
+2. The attack MUST achieve meaningful impact (not just "learn field names" or "cause an error")
+3. Check if another layer already prevents exploitation — if so, it's a hardening note, not a finding
+4. If the baseline comparable has the same pattern, note whether it's been exploited there
+5. If your exploit depends on parser/runtime behavior, verify against the relevant spec or implementation — do not reason from intuition.
+6. Return ONLY confirmed findings with concrete attacks, or "No exploitable vulnerabilities found" if that's honest.
+```

.claude/skills/security-audit/MEMORY-SAFETY-AND-BINARY.md

+# Memory Safety, Binary, and Kernel Hunting
+
+#### When to use this file
+
+The attack classes in `ATTACK-CLASSES.md` are tuned for web apps, APIs, and services. Reach for _this_ file when the target processes untrusted bytes in a memory-unsafe context: C/C++/Objective-C, Rust `unsafe`, kernel modules and drivers, parsers and decoders (image/video/font/archive/PDB), reverse-engineering and dev tooling, network daemons, firmware, and language runtimes/JITs. These targets fail differently from web apps — the bug is a memory corruption or a logic error in privileged code, not an injection or an access-control gap — so the hunt needs a different lens.
+
+Pick the relevant classes based on Phase 1. Split per subsystem for large targets.
+
+## Core discipline (include in every agent prompt for this domain)
+
+```
+- A buffer sized for the common case can still overflow on adversarial input. Verify every "this length is bounded" claim against the WORST case, not the happy path.
+- "Huge count = guaranteed crash" is FALSE. An oversized copy length is size- and libc-dependent: it often faults, but the copy primitive can also wrap or land a short, scattered write first. Determine the actual write behavior before downgrading to DoS-only.
+- Static offsets are a guess; the crash dump is truth. An unreproduced bug is not a bug — if you claim exploitability, say exactly which input reaches which sink and what the observable result is.
+- Sanitizer silence ≠ safety where the deref is outside instrumented code (hand-written asm, JIT-emitted, intra-allocation). Don't trust a clean ASan run for those.
+```
+
+## Memory-safety attack classes (subagent_type: `general`)
+
+**Spatial: out-of-bounds read/write**
+
+- **Length subtraction underflow** — a copy/loop bound is `a - b` (`uri.len - prefix`, `total - consumed`) where the attacker can make `b > a`. Negative → casts to ~SIZE_MAX. Map which bytes land where; don't assume "just a crash."
+- **Operator-precedence / multi-term length errors** — an unparenthesized `+`/`-` length chain (`endp - begin + consume`) that silently over-adds when one term is attacker-sized. Audit each CALLER's value of the variable term — the common caller is often correct-by-accident on the zero path and survives testing.
+- **`sizeof(*p)` vs `sizeof(element)` pointer-depth confusion** — an allocation/copy size computed one indirection too deep (`gid_t **` → `sizeof(*p)`=8 not 4). The bounds check passes because it uses the same wrong unit. Compiled tell: `shl $0x3` where `shl $0x2` was meant.
+- **Wire-length into fixed stack buffer** — a function rebuilds a network/user blob into a fixed array using an attacker length field, with the bounds check missing/late or computed on the wrong headroom (a header pre-written into the buffer). Re-derive true headroom (size minus fixed prefix); confirm no guard precedes the copy.
+
+**Temporal: use-after-free / lifetime**
+
+- **Embedded waiter-anchor freed without draining** — a struct embeds a list head (`selinfo`/`knlist`/timer/knote) reachable by unprivileged poll/select/kqueue, and a free path destroys it but skips the drain a wakeup path does. For every `selrecord(&obj->x)`, require a matching drain on EACH path that can free `obj`.
+- **Cached raw pointer + reallocating owner** — a view caches `base+offset`, a grow/realloc path moves the backing store, and the invalidation walks only the _current_ wrapper's view set while grow _replaces_ the wrapper. The original view dangles.
+
+**Type confusion**
+
+- **Read-and-write confusion → addrof/fakeobj** — a confusion that reads a pointer slot as a scalar (addrof) and writes a scalar into a pointer slot (fakeobj). The standard pivot of runtime/JIT exploitation; the prior art is about the PROBLEM CLASS (NaN-boxing, cached typed-array data pointer), not the specific target.
+- **Hierarchical-walker leaf check skipped** — a page-table / nested / B-tree / extent walker checks the valid bit but not the leaf/size bit at level N, then descends treating an attacker-owned leaf as an interior node.
+
+**Value: uninitialized & oracle**
+
+- **Uninitialized worst-case buffer + observable compare = read oracle** — a buffer sized to a MAX constant is partially written, then compared against attacker bytes with an attacker-controlled compare length where match/no-match is observable. No memory-disclosure bug needed; the gap between actual output and MAX-size is the leak window. Brute one byte/connection, hint the structural bits, parallelize.
+
+## Kernel & privileged-interface attack classes (subagent_type: `general`)
+
+- **User-copy bounds + double-fetch (TOCTOU)** — a syscall/ioctl/Mach-trap entry whose user-copy primitive (`copyin` / `copy_from_user`) brings attacker memory in, then re-reads the SAME user address after a check. Any fact derived from concurrently-mutable user memory and trusted on a later pass is a double-fetch even when each op is individually correct.
+- **Object lifecycle / UAF (IOKit/OSObject and friends)** — unbalanced retain/release on an externally-reachable object; a method that releases on one path but a sibling dispatch (compat/fallback/ptrace) forgot it. Diff the duplicated dispatch paths.
+- **Unchecked downcast / type confusion** — `OSDynamicCast` (or any tagged-union cast) whose result is used without a null check, or a selector/index into a dispatch table without a bounds check.
+- **World-writable / under-permissioned powerful interface** — a device node, admin socket, or mgmt API exposed more broadly than its power, that validates the request SHAPE (index in range) but never the requester's AUTHORITY over the named resource. Danger = power × reachability; enumerate the surface reachable from the _actual_ untrusted context first.
+- **Validate-then-act-on-stale-state** — a fast path and a compat/ptrace/fallback path to the same operation where one copy forgot a guard the other performs.
+
+## Universal moves (apply across the above)
+
+- **Audit the incomplete fix.** A targeted patch is a high-signal pointer to a dangerous sink with the analysis already done. Read the diff → find the exact sink it hardened → scan the same function, parallel paths, and alternate callers for the SAME tainted-data-to-sink shape the patch missed. Incomplete fixes are their own bug class.
+- **Trust asymmetry between two ends of a protocol.** A filter/verification/size-cap installed on one side of a connection but missing on the symmetric call on the other. Find the protective call → grep its mirror on the opposite role → if absent, the earliest unprotected pre-auth parse is the prize. A malicious server/MITM is a real attacker.
+- **Chain a weak primitive.** A blocked path means you haven't found the right pivot, not that it's unexploitable. Always ask "what does this actually let me do, and what runs automatically once I can put bytes on disk?" (plugin dirs, autoload, `.git/hooks`, `conftest.py`).
+- **Hunt where the crowd isn't.** The tools researchers themselves trust — debuggers, disassemblers, scanners, dev tooling — are under-audited and high-impact. Old code and obscure formats are gold.
+
+## Validation rules (apply before reporting ANY finding here)
+
+1. **Build a debuggable target first.** Wire in crash dumps + a debugger before you claim exploitability. You can't iterate on what you can't observe.
+2. **Read the offset from the crash, not the disassembly.** Send a cyclic (De Bruijn) pattern; the faulting register values give the exact offset. A variable-length prefix (handle, optional field, padding) shifts the geometry off the static prediction.
+3. **Prove a UAF by reclaim-and-compare** when the sanitizer is blind (asm/JIT/intra-allocation): trigger the dangling view, reclaim the freed region with a size-matched content-controlled allocation, write through the dangler, read the reclaimer back — aliasing either way proves it.
+4. **Distinguish crash from exploitable.** For an OOB write, map which bytes land where and whether a security-relevant field is reachable; for a "huge count," prove the bounded-write case before calling it DoS-only.
+5. **Return ONLY confirmed findings** with the exact input → sink path and the observable result, or "No exploitable memory-safety issues found" if that's honest.

.claude/skills/security-audit/RECONNAISSANCE.md

+# Reconnaissance
+
+### Phase 1: Understand the application
+
+Before looking for bugs, understand what you're auditing. This requires depth, not just a directory listing. Launch **multiple `research` agents in parallel** to map different aspects of the codebase:
+
+**Agent 1a: Overview, tech stack, and comparable baseline**
+
+```
+Explore the codebase at <path>. Answer:
+1. What is this application? What kind of software? (web app, API, CLI tool, library, daemon, desktop app, mobile backend, etc.)
+2. Who uses it and how? (end users, developers, operators, other services)
+3. What's the tech stack? (languages, frameworks, databases, runtime, deployment model)
+4. What comparable mainstream software exists? What security tradeoffs does the comparable accept?
+5. What's the high-level directory structure?
+Return specific file paths for key entry points.
+```
+
+**Agent 1b: Trust boundaries and access control**
+
+```
+Explore the codebase at <path>. Find and read ALL code related to:
+1. Trust boundaries — where does untrusted input enter the system? (HTTP requests, CLI args, file reads, IPC, message queues, environment variables, config files, etc.)
+2. Authentication — how do callers prove identity? (sessions, tokens, API keys, mTLS, Unix sockets, etc.) If there's no authentication, note that.
+3. Authorization — how are permissions enforced? (middleware, decorators, capability checks, file permissions, etc.) If there's no authorization model, note that.
+4. Privilege separation — does the code run as root? Drop privileges? Use sandboxing? Fork workers?
+5. Any bypass mechanisms (dev-only modes, test helpers, setup flows, debug flags)
+Return the trust model: who are the actors, what can each do by design, and which code enforces it. Include specific file paths and line numbers.
+```
+
+**Agent 1c: Input surface inventory**
+
+```
+Explore the codebase at <path>. Produce a complete inventory of where external input enters the system:
+1. Network-facing surfaces (HTTP endpoints, gRPC services, WebSocket handlers, TCP/UDP listeners, etc.) — list each with method/verb and purpose
+2. File-based input (file uploads, config file parsing, log ingestion, import/export, etc.)
+3. IPC and inter-service input (message queues, shared memory, Unix sockets, environment variables, CLI arguments)
+4. User-generated content surfaces (anywhere users provide content that is stored and later rendered, served, or processed)
+5. External integrations (OAuth, webhooks, third-party APIs, plugin loading, dynamic code execution)
+6. All places where input reaches dangerous sinks (SQL/query builders, HTML/template output, file paths, shell commands, deserialization, eval, dynamic imports)
+Return specific file paths. Be exhaustive.
+```
+
+Collect all three agents' outputs and synthesize them into `<output-dir>/architecture.md`:
+
+- 1-2 page structured summary covering application type, tech stack, trust model, input surfaces, and baseline comparable
+- Include the key file paths from all agents — these become the starting points for Phase 2
+- This document is injected verbatim into every Phase 2 agent prompt
+
+If Phase 1 agents reveal the codebase is larger or more complex than expected (e.g., plugin system, multi-tenant architecture, complex auth chains, multiple deployment targets), launch additional `research` agents to map those areas before proceeding. The quality of Phase 2 depends entirely on the quality of Phase 1.

.claude/skills/security-audit/VALIDATION-AND-REPORTING.md

+# Validation, Reporting, and Verification
+
+### Phase 3: Validate findings
+
+Collect all findings from Phase 2 agents and **consolidate duplicates first**. Phase 2 deliberately overlaps agent scopes, so the same issue is frequently reported by more than one hunter — merge findings that share a root cause before validating, or you'll validate and report the same bug multiple times. For each remaining finding, launch a **separate `research` validation agent** that tries to disprove it. The hunting agents are biased toward finding things; the validation agents are biased toward killing false positives. This adversarial step is critical.
+
+For findings from the same attack surface, batch them into one validation agent. Launch validation agents in parallel where they cover independent areas.
+
+Each validation agent prompt should:
+
+1. State the specific finding being validated (title, claimed attack, claimed impact)
+2. Ask the agent to read the exact code paths and verify each step of the trace
+3. Ask it to apply these tests (the adversarial, Phase 3 form of the canonical validation rules in [HUNTING.md](HUNTING.md) — here a separate agent tries to make each one fail):
+
+**Validation tests:**
+
+1. **Exploitation test**: Read the actual code at each step of the trace. Does the data flow work as claimed? Can you construct the exact input (HTTP request, CLI invocation, API call, crafted file, etc.) that triggers this?
+2. **Impact test**: What does the attacker actually get? If the answer is "they learn field names" or "they cause an error", that's not meaningful impact — not a finding on its own (at most a building block for a chain).
+3. **Baseline test**: Does the identified comparable have the same pattern? If yes, has it been exploited? If never exploited in years of production use, understand why before reporting.
+4. **Mitigation test**: Is there another layer that prevents exploitation? Check middleware, database constraints, framework defaults.
+5. **Parser/runtime behavior test**: If the exploit depends on how a parser or runtime handles specific input, verify against the actual spec or implementation — do not reason from intuition.
+
+Tell each validation agent:
+
+```
+Your job is to DISPROVE this finding. Read the actual source code at every step. If you cannot disprove it, confirm it with the exact code that makes it exploitable. Return one of:
+- "CONFIRMED: [explanation of why it's real, with code evidence]"
+- "REJECTED: [explanation of what the finding got wrong, with code evidence]"
+```
+
+**Kill false positives aggressively, but don't kill real findings.** A short report with 3 real findings is worth more than a long report with 30 theoretical ones. An honest "nothing found" is valid — but push hard before reaching that conclusion.
+
+### Phase 4: Report
+
+Write the report to the output directory established in Setup.
+
+**Output files:**
+
+1. `REPORT.md` -- Main report with:
+   - One-paragraph executive summary (honest assessment of security posture)
+   - Identified baseline and how this application compares
+   - Findings table (severity, title, one-line description)
+   - Each finding with: file path, concrete attack scenario, impact, recommended fix
+   - Hardening notes section (defense-in-depth suggestions, NOT findings)
+   - Positive patterns section (what the codebase does well -- this calibrates trust in the audit)
+
+2. `FINDINGS-DETAIL.md` -- For each finding rated MEDIUM or above:
+   - Complete data flow from input to sink with file:line references
+   - Exact HTTP request(s) to trigger
+   - What the attacker gets
+   - How the baseline comparable handles the same scenario
+
+Keep it short. If the report is longer than the codebase deserves, you're padding.
+
+### Phase 5: Structured output and schema check
+
+For every finding that survived Phase 3 validation, produce a structured JSON object conforming to the schema defined in `report-schema.json` (in the same directory as this skill file — read it via the Read tool before writing output). Write the result to `<output-dir>/findings.json`.
+
+The schema supports two verdict types via `oneOf`:
+
+- **`confirmed`** — a validated vulnerability with full trace, execution, and remediation
+- **`rejected`** — a finding that was investigated and determined to be factually incorrect
+
+**Before writing `findings.json`:**
+
+1. Read `report-schema.json` from this skill's directory. Follow it exactly — `additionalProperties: false` is enforced, so extra fields will make the output invalid.
+2. For each finding, populate every required field. If you cannot fill `trace` with real file paths and line numbers verified against the source, the finding is not sufficiently verified — go back and verify it or reject it. Mind the required fields that aren't self-evident: `intended_behavior` (what the code is _supposed_ to do, so the defect is legible), `confidence` (`low`/`medium`/`high`, with a reason), and the `severity` object (`likelihood`/`impact`/`overall_severity`). All `severity` scores use the schema's **lowercase** enum — `informational`/`low`/`medium`/`high`/`critical`; the UPPERCASE tiers in SKILL.md and REPORT.md are prose labels, not valid JSON values.
+3. Run `node <skill-dir>/validate-findings.cjs <output-dir>/findings.json` to validate. It checks required fields, enum values, structural constraints, and `additionalProperties`. This is a structural check only — it confirms the JSON conforms to the schema, not that the findings are correct. Factual verification is Phase 6's job. Fix any failures before proceeding.
+
+### Phase 6: Independent verification
+
+The structured output from Phase 5 forces self-validation, but the same agent that wrote the finding also wrote the JSON — it won't catch its own blind spots. This phase uses a fresh agent to independently verify every claim in `findings.json`.
+
+Launch **one `research` agent per confirmed finding** via the Task tool, all in parallel. Each agent gets exactly one finding from `findings.json` and verifies it independently. Give each agent the JSON object for its finding and this prompt:
+
+```
+You are an independent verifier. You did NOT write this finding. Your job is to read the actual source code and verify that every factual claim is correct.
+
+1. Read the file and line number cited in EVERY trace step. Verify:
+   - The file exists at that path
+   - The line number matches the described code
+   - The scope (function name) is correct
+   - The description accurately reflects what the code does
+
+2. Verify the root_cause statement by reading the cited file and confirming the described defect exists.
+
+3. Verify the execution payloads would actually work, in terms that fit the target:
+   - Does the entry point exist as claimed — the endpoint/URL, CLI command, exported function, syscall/ioctl, message handler, or tool the attacker invokes?
+   - Does the invocation match — HTTP method, argument shape, call signature, or message format?
+   - Would the input survive validation and parsing on the real code path?
+   - Would the relevant authentication, authorization, or ownership check pass as described?
+
+4. Verify conditions are complete — are there prerequisites the finding missed?
+
+5. Check the remediation code_changes — would the fix actually prevent the attack without breaking normal functionality?
+
+6. Verify `intended_behavior` accurately states what the code should do, and that `confidence` matches the strength of the evidence — don't leave `high` on a claim you couldn't fully trace.
+
+Return one of:
+- "VERIFIED" — all claims checked out against the source
+- "CORRECTED: [field]: [what was wrong] → [what it should be]" — factual error in a specific field
+- "REJECTED: [reason]" — the finding is fundamentally wrong
+```
+
+Apply the agent's corrections:
+
+- **VERIFIED** findings: no changes needed
+- **CORRECTED** findings: update the specific fields in `findings.json`, re-run the schema validation script
+- **REJECTED** findings: change their `verdict` to `"rejected"` with the agent's reason, or remove them entirely
+
+After applying corrections, reconcile the prose deliverables: update `REPORT.md` and `FINDINGS-DETAIL.md` so they match the final `findings.json`. Remove or amend any finding the verification gate rejected or corrected — the human-readable report and the machine-readable output must not disagree.
+
+This is the final quality gate. Do not skip it.

.claude/skills/security-audit/report-schema.json

+{
+  "$comment": "Single source of truth for findings.json structure (see SKILL.md Phase 5). validate-findings.cjs reads this file directly and interprets it — there is no second copy of these rules to keep in sync.",
+  "output_schema": {
+    "oneOf": [
+      {
+        "type": "object",
+        "description": "Confirmed vulnerability — provide the complete, independently verified report.",
+        "properties": {
+          "verdict": {
+            "type": "string",
+            "const": "confirmed"
+          },
+          "title": {
+            "type": "string",
+            "description": "A concise, standard title for the vulnerability."
+          },
+          "description": {
+            "type": "string",
+            "description": "Comprehensive explanation of the vulnerability. Include any reproduction details (proof-of-concept input, configuration, observed output or crash) here."
+          },
+          "root_cause": {
+            "type": "string",
+            "description": "One sentence using the template: '[function_or_component] in [file] does not [missing action], allowing [consequence]'. MUST include the function/component name and file name where the defect exists."
+          },
+          "intended_behavior": {
+            "type": "string",
+            "description": "What was the developer trying to build? Explain the intended, non-vulnerable business logic."
+          },
+          "trace": {
+            "type": "array",
+            "minItems": 2,
+            "items": {
+              "type": "object",
+              "properties": {
+                "kind": {
+                  "type": "string",
+                  "enum": ["entrypoint", "propagation", "sink"]
+                },
+                "file": {
+                  "type": "string",
+                  "description": "Exact file path relative to repository root."
+                },
+                "line": {
+                  "type": "integer"
+                },
+                "scope": {
+                  "type": "string",
+                  "description": "Bare function or method name. No parentheses, no arguments."
+                },
+                "description": {
+                  "type": "string",
+                  "description": "Factual description of the state change or data movement."
+                }
+              },
+              "required": ["kind", "file", "line", "scope", "description"],
+              "additionalProperties": false
+            },
+            "description": "Sequential code trace from entrypoint to sink, verified against actual source code. The first step must be kind 'entrypoint', the last must be kind 'sink', and any intermediate steps must be kind 'propagation' (enforced by the validator)."
+          },
+          "conditions": {
+            "type": "array",
+            "items": {
+              "type": "object",
+              "properties": {
+                "kind": {
+                  "type": "string",
+                  "enum": [
+                    "authentication_level",
+                    "authorization_role",
+                    "user_interaction",
+                    "system_configuration",
+                    "network_routing",
+                    "environmental_dependency",
+                    "data_state",
+                    "timing_dependency",
+                    "third_party_dependency"
+                  ]
+                },
+                "description": {
+                  "type": "string"
+                }
+              },
+              "required": ["kind", "description"],
+              "additionalProperties": false
+            },
+            "description": "Factual prerequisites for exploitation. Empty array if exploitable by default."
+          },
+          "execution": {
+            "type": "object",
+            "properties": {
+              "attacker_perspective": {
+                "type": "string",
+                "description": "Who is the attacker and their starting point."
+              },
+              "payloads": {
+                "type": "array",
+                "items": {
+                  "type": "string"
+                },
+                "description": "Specific malicious inputs, HTTP requests, or scripts."
+              },
+              "instructions": {
+                "type": "array",
+                "items": {
+                  "type": "string"
+                },
+                "description": "Linear array of all attacker actions from setup through exploitation."
+              },
+              "expected_result": {
+                "type": "string",
+                "description": "Observable outcome confirming successful exploitation."
+              }
+            },
+            "required": [
+              "attacker_perspective",
+              "payloads",
+              "instructions",
+              "expected_result"
+            ],
+            "additionalProperties": false
+          },
+          "remediation": {
+            "type": "object",
+            "properties": {
+              "strategy": {
+                "type": "string",
+                "description": "High-level explanation of the fix."
+              },
+              "code_changes": {
+                "type": "array",
+                "items": {
+                  "type": "object",
+                  "properties": {
+                    "file_name": {
+                      "type": "string"
+                    },
+                    "fixed_code": {
+                      "type": "string"
+                    }
+                  },
+                  "required": ["file_name", "fixed_code"],
+                  "additionalProperties": false
+                }
+              }
+            },
+            "required": ["strategy"],
+            "additionalProperties": false
+          },
+          "severity": {
+            "type": "object",
+            "properties": {
+              "likelihood": {
+                "type": "object",
+                "properties": {
+                  "score": {
+                    "type": "string",
+                    "enum": [
+                      "informational",
+                      "low",
+                      "medium",
+                      "high",
+                      "critical"
+                    ]
+                  },
+                  "reason": {
+                    "type": "string"
+                  }
+                },
+                "required": ["score", "reason"],
+                "additionalProperties": false
+              },
+              "impact": {
+                "type": "object",
+                "properties": {
+                  "score": {
+                    "type": "string",
+                    "enum": [
+                      "informational",
+                      "low",
+                      "medium",
+                      "high",
+                      "critical"
+                    ]
+                  },
+                  "reason": {
+                    "type": "string"
+                  }
+                },
+                "required": ["score", "reason"],
+                "additionalProperties": false
+              },
+              "overall_severity": {
+                "type": "string",
+                "enum": ["informational", "low", "medium", "high", "critical"]
+              }
+            },
+            "required": ["likelihood", "impact", "overall_severity"],
+            "additionalProperties": false
+          },
+          "confidence": {
+            "type": "object",
+            "properties": {
+              "score": {
+                "type": "string",
+                "enum": ["low", "medium", "high"]
+              },
+              "reason": {
+                "type": "string",
+                "description": "Why you scored the confidence this way. Mention any missing files, complex routing, or ambiguous data flows."
+              }
+            },
+            "required": ["score", "reason"],
+            "additionalProperties": false
+          }
+        },
+        "required": [
+          "verdict",
+          "title",
+          "description",
+          "root_cause",
+          "intended_behavior",
+          "trace",
+          "conditions",
+          "execution",
+          "remediation",
+          "severity",
+          "confidence"
+        ],
+        "additionalProperties": false
+      },
+      {
+        "type": "object",
+        "description": "Rejected finding — the described behavior is factually incorrect or the code path does not exist.",
+        "properties": {
+          "verdict": {
+            "type": "string",
+            "const": "rejected"
+          },
+          "reason": {
+            "type": "string",
+            "description": "Explain which specific claims in the finding are factually wrong (e.g., code path doesn't exist, mitigation prevents the described flow, trace is incorrect)."
+          }
+        },
+        "required": ["verdict", "reason"],
+        "additionalProperties": false
+      }
+    ]
+  }
+}

.claude/skills/security-audit/validate-findings.cjs

+#!/usr/bin/env node
+
+/**
+ * Validates findings.json against report-schema.json.
+ * Usage: node validate-findings.cjs <path-to-findings.json>
+ *
+ * The validation rules live in report-schema.json — the single source of truth.
+ * This script reads that schema at runtime and interprets the subset of JSON
+ * Schema it uses: type (object|array|string|integer), properties, required,
+ * additionalProperties:false, enum, const, items, minItems, and oneOf.
+ *
+ * Some constraints can't be expressed in that subset (a confirmed trace must
+ * start at an "entrypoint", end at a "sink", and only use "propagation" for
+ * intermediate steps). They're applied as an explicit, clearly-labelled
+ * semantic layer after schema validation.
+ *
+ * Zero dependencies. Exits 0 on success, 1 on validation failure.
+ */
+
+const fs = require("fs");
+const path = require("path");
+
+const file = process.argv[2];
+if (!file) {
+  console.error("Usage: node validate-findings.cjs <path-to-findings.json>");
+  process.exit(1);
+}
+
+const schemaPath = path.join(__dirname, "report-schema.json");
+let itemSchema;
+try {
+  const doc = JSON.parse(fs.readFileSync(schemaPath, "utf8"));
+  itemSchema = doc.output_schema;
+  if (!itemSchema)
+    throw new Error('report-schema.json is missing top-level "output_schema"');
+} catch (e) {
+  console.error(`Failed to load schema from ${schemaPath}:`, e.message);
+  process.exit(1);
+}
+
+let findings;
+try {
+  findings = JSON.parse(fs.readFileSync(file, "utf8"));
+} catch (e) {
+  console.error("Failed to parse JSON:", e.message);
+  process.exit(1);
+}
+
+if (!Array.isArray(findings)) {
+  console.error("findings.json must be an array");
+  process.exit(1);
+}
+
+// --- Generic JSON Schema interpreter (the subset used by report-schema.json) ---
+
+function typeOf(v) {
+  if (Array.isArray(v)) return "array";
+  if (v === null) return "null";
+  return typeof v; // "object" | "string" | "number" | "boolean"
+}
+
+// For oneOf: find a property defined with a `const` so error messages can point
+// at the intended branch (e.g. discriminate confirmed vs rejected by "verdict").
+function findDiscriminator(schema) {
+  if (!schema.properties) return null;
+  for (const [key, sub] of Object.entries(schema.properties)) {
+    if (sub && Object.prototype.hasOwnProperty.call(sub, "const")) {
+      return { key, value: sub.const };
+    }
+  }
+  return null;
+}
+
+function validate(value, schema, p, errors) {
+  if (schema.oneOf) {
+    // Prefer the branch whose const discriminator matches, so the caller sees
+    // detailed errors for the branch they clearly intended.
+    for (const branch of schema.oneOf) {
+      const disc = findDiscriminator(branch);
+      if (
+        disc &&
+        value &&
+        typeof value === "object" &&
+        value[disc.key] === disc.value
+      ) {
+        validate(value, branch, p, errors);
+        return;
+      }
+    }
+    // No discriminator matched. If every branch is discriminated by the same
+    // key, report the bad discriminator value clearly.
+    const discs = schema.oneOf.map(findDiscriminator).filter(Boolean);
+    if (
+      discs.length === schema.oneOf.length &&
+      value &&
+      typeof value === "object"
+    ) {
+      const key = discs[0].key;
+      const allowed = discs.map((d) => JSON.stringify(d.value)).join(", ");
+      errors.push(
+        `${p}: "${key}" must be one of ${allowed}, got ${JSON.stringify(value[key])}`,
+      );
+      return;
+    }
+    const passing = schema.oneOf.filter(
+      (b) => collect(value, b, p).length === 0,
+    );
+    if (passing.length !== 1) {
+      errors.push(`${p}: does not match exactly one of the allowed schemas`);
+    }
+    return;
+  }
+
+  if (
+    Object.prototype.hasOwnProperty.call(schema, "const") &&
+    value !== schema.const
+  ) {
+    errors.push(
+      `${p}: must equal ${JSON.stringify(schema.const)}, got ${JSON.stringify(value)}`,
+    );
+  }
+
+  if (schema.enum && !schema.enum.includes(value)) {
+    const allowed = schema.enum.map((v) => JSON.stringify(v)).join(", ");
+    errors.push(
+      `${p}: invalid value ${JSON.stringify(value)} (expected one of ${allowed})`,
+    );
+  }
+
+  switch (schema.type) {
+    case "object": {
+      if (typeOf(value) !== "object") {
+        errors.push(`${p}: expected object, got ${typeOf(value)}`);
+        return;
+      }
+      for (const req of schema.required || []) {
+        if (!(req in value))
+          errors.push(`${p}: missing required field "${req}"`);
+      }
+      for (const key of Object.keys(value)) {
+        if (schema.properties && key in schema.properties) {
+          validate(value[key], schema.properties[key], `${p}.${key}`, errors);
+        } else if (schema.additionalProperties === false) {
+          errors.push(`${p}: unexpected field "${key}"`);
+        }
+      }
+      break;
+    }
+    case "array": {
+      if (typeOf(value) !== "array") {
+        errors.push(`${p}: expected array, got ${typeOf(value)}`);
+        return;
+      }
+      if (
+        typeof schema.minItems === "number" &&
+        value.length < schema.minItems
+      ) {
+        errors.push(
+          `${p}: must have at least ${schema.minItems} item(s), got ${value.length}`,
+        );
+      }
+      if (schema.items) {
+        value.forEach((el, i) =>
+          validate(el, schema.items, `${p}[${i}]`, errors),
+        );
+      }
+      break;
+    }
+    case "integer": {
+      if (typeOf(value) !== "number" || !Number.isInteger(value)) {
+        errors.push(`${p}: expected integer, got ${typeOf(value)}`);
+      }
+      break;
+    }
+    case "string": {
+      if (typeOf(value) !== "string") {
+        errors.push(`${p}: expected string, got ${typeOf(value)}`);
+      }
+      break;
+    }
+    default:
+      break; // no type constraint at this node
+  }
+}
+
+function collect(value, schema, p) {
+  const errors = [];
+  validate(value, schema, p, errors);
+  return errors;
+}
+
+// --- Run ----------------------------------------------------------------------
+
+let errorCount = 0;
+
+findings.forEach((f, i) => {
+  const label = `[${i}] ${(f && (f.title || f.reason)) || "(untitled)"}`;
+  console.log(`Checking ${label}`);
+
+  const errs = collect(f, itemSchema, `[${i}]`);
+
+  // Semantic layer — constraints the schema subset can't express:
+  // a confirmed trace must be one entrypoint, zero or more propagation steps,
+  // then one sink.
+  if (
+    f &&
+    f.verdict === "confirmed" &&
+    Array.isArray(f.trace) &&
+    f.trace.length > 0
+  ) {
+    if (f.trace[0] && f.trace[0].kind !== "entrypoint") {
+      errs.push(
+        `[${i}].trace[0].kind must be "entrypoint", got ${JSON.stringify(f.trace[0].kind)}`,
+      );
+    }
+    const last = f.trace.length - 1;
+    if (f.trace[last] && f.trace[last].kind !== "sink") {
+      errs.push(
+        `[${i}].trace[${last}].kind must be "sink", got ${JSON.stringify(f.trace[last].kind)}`,
+      );
+    }
+    for (let j = 1; j < last; j++) {
+      if (f.trace[j] && f.trace[j].kind !== "propagation") {
+        errs.push(
+          `[${i}].trace[${j}].kind must be "propagation", got ${JSON.stringify(f.trace[j].kind)}`,
+        );
+      }
+    }
+  }
+
+  for (const msg of errs) console.error("  ERROR:", msg);
+  errorCount += errs.length;
+});
+
+console.log();
+if (errorCount === 0) {
+  console.log(`PASS: ${findings.length} findings valid`);
+} else {
+  console.error(
+    `FAIL: ${errorCount} error(s) across ${findings.length} findings`,
+  );
+  process.exit(1);
+}

.claude/skills/shop-amazon.md

+# Tác Vụ Tái Sử Dụng: Cào & Bóc Tách Dữ Liệu Sản Phẩm Amazon (shop-amazon.md)
+
+> Skill workflow định nghĩa quy trình chuẩn để thu thập và trích xuất dữ liệu sản phẩm có cấu trúc từ sàn thương mại điện tử Amazon sử dụng hệ thống `data-crawler-be`.
+
+---
+
+## 1. Thông Tin Tác Vụ (Task Metadata)
+
+- **Mục tiêu:** Thu thập thông tin danh sách sản phẩm hoặc chi tiết sản phẩm Amazon (Tiêu đề, Giá bán, Đánh giá sao, Số lượng review, Ảnh đại diện, ASIN, Tình trạng còn hàng).
+- **Target Domain:** `amazon.com`, `amazon.co.jp`, `amazon.de`, ...
+- **Chế độ khuyến nghị:** `CRAWL` (cho trang danh mục) hoặc `SCRAPE` (cho từng sản phẩm cụ thể).
+
+---
+
+## 2. Cấu Hình Job Đề Xuất (Job Parameters)
+
+Khi khởi tạo `CrawlJob` qua API `POST /api/v1/crawl-jobs`, sử dụng payload mẫu sau:
+
+```json
+{
+  "startUrl": "https://www.amazon.com/s?k=laptop",
+  "mode": "CRAWL",
+  "maxPages": 25,
+  "maxDepth": 2,
+  "delayMs": 2500,
+  "timeoutMs": 45000,
+  "retryCount": 3,
+  "respectRobotsTxt": true,
+  "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
+}
+```
+
+> **Lưu ý chống chặn (Anti-bot):**
+>
+> - Luôn thiết lập `delayMs` tối thiểu `2000`ms để tránh bị hệ thống Amazon chặn IP hoặc hiển thị CAPTCHA.
+> - Đảm bảo worker bắt cờ `CAPTCHA_DETECTED` và `BLOCKED` trong bảng `crawl_pages` để cảnh báo người dùng.
+
+---
+
+## 3. Extraction Template Cho Amazon (Bóc Tách Dữ Liệu)
+
+Áp dụng template trích xuất có cấu trúc tương ứng với domain `amazon.com`:
+
+```json
+{
+  "domain": "amazon.com",
+  "name": "Amazon Product Standard Template",
+  "fields": [
+    {
+      "name": "title",
+      "selector": "#productTitle, h2 a.a-link-normal span",
+      "type": "text",
+      "required": true
+    },
+    {
+      "name": "price",
+      "selector": ".a-price .a-offscreen, span.a-price-whole",
+      "type": "text",
+      "required": false
+    },
+    {
+      "name": "rating",
+      "selector": "span[data-hook='rating-out-of-text'], span.a-icon-alt",
+      "type": "text",
+      "required": false
+    },
+    {
+      "name": "reviewCount",
+      "selector": "#acrCustomerReviewText, span[data-hook='total-review-count']",
+      "type": "number",
+      "required": false
+    },
+    {
+      "name": "mainImage",
+      "selector": "#landingImage, .s-image",
+      "type": "attribute",
+      "attributeName": "src",
+      "required": false
+    },
+    {
+      "name": "availability",
+      "selector": "#availability span",
+      "type": "text",
+      "required": false
+    }
+  ]
+}
+```
+
+---
+
+## 4. Quy Trình Xuất Báo Cáo Kết Quả
+
+1. Sau khi Job chuyển trạng thái `COMPLETED`:
+2. Gọi API `POST /api/v1/exports` với:
+   ```json
+   {
+     "jobId": "<crawl-job-id>",
+     "exportType": "XLSX"
+   }
+   ```
+3. File Excel sinh ra sẽ có cột phân tách rõ ràng cho từng thuộc tính sản phẩm, sẵn sàng phân tích hoặc nhập liệu.

.env.example

 NODE_ENV=development
-PORT=3000
+PORT=9898
 TRUST_PROXY=false
 
 # Database Configuration (PostgreSQL / Supabase)
@@ -14,8 +14,9 @@ DB_SSL=true
 # Cách 2 (Tùy chọn): Sử dụng trực tiếp connection string từ Supabase Dashboard
 # DATABASE_URL="postgresql://postgres:[YOUR-PASSWORD]@db.[YOUR-PROJECT-REF].supabase.co:5432/postgres?sslmode=require"
 
-JWT_ACCESS_SECRET=change_me_access_secret
-JWT_REFRESH_SECRET=change_me_refresh_secret
+# REQUIRED - minimum 32 characters each. Generate with: openssl rand -hex 32
+JWT_ACCESS_SECRET=
+JWT_REFRESH_SECRET=
 JWT_ACCESS_EXPIRES_IN=1d
 JWT_REFRESH_EXPIRES_IN=7d
 
@@ -47,9 +48,12 @@ WORKER_CONCURRENCY=3
 WORKER_JOB_TIMEOUT_MS=300000
 WORKER_MAX_STALLED_COUNT=1
 
+# Default User Quota Limits
 USER_MAX_PAGES=100
 USER_MAX_JOBS_PER_DAY=10
 USER_MAX_CONCURRENT_JOBS=3
+USER_MAX_PAGES_PER_MONTH=1000
+USER_MAX_JOBS_PER_MONTH=100
 
 # SMTP Configuration
 SMTP_HOST=smtp.gmail.com
@@ -60,7 +64,8 @@ SMTP_USER=your_smtp_user
 SMTP_PASS=your_16_character_app_password
 SMTP_FROM="Data Crawler <no-reply@datacrawler.com>"
 FRONTEND_URL=http://localhost:5173
+CORS_ALLOWED_ORIGINS=http://localhost:5173,http://localhost:3000
 
-# Webhook Configuration
-WEBHOOK_ENCRYPTION_KEY=a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2
+# REQUIRED - must be a 64-character lowercase hex string. Generate with: openssl rand -hex 32
+WEBHOOK_ENCRYPTION_KEY=
 WEBHOOK_QUEUE_NAME=webhook-delivery

.gitignore

@@ -7,4 +7,8 @@ storage/exports/*
 .DS_Store
 nopush/
 coverage/
-frontend-data-crawler-be/
\ No newline at end of file
+
+# Claude local overrides
+.claude/*.local.*
+.claude/CLAUDE.local.md
+.claude/settings.local.json
\ No newline at end of file

AGENTS.md

-# Backend guidance
+# Backend Guidance & Architecture Rules (data-crawler-be)
 
-## Architecture
+> Tài liệu quy tắc chuẩn cho **Antigravity** khi phát triển và bảo trì mã nguồn trong workspace `data-crawler-be`.
 
-- Keep the request path `Route -> Controller -> Service -> Repository -> Prisma`.
-- Only `*.repository.ts` files may import or call the Prisma client.
-- Keep module code under `src/modules/<feature>/`; use the existing route, controller, service, repository, DTO, validation, and test conventions in the nearest module.
-- Put shared errors, helpers, constants, storage code, and types under `src/common/` only when they are genuinely reused.
-- Mount module routes through `src/routes/index.ts`.
+---
 
-## Data and integrations
+## 1. Kiến Trúc Phân Tầng Tuyệt Đối (Strict 5-Layer Pattern)
 
-- Treat `prisma/schema.prisma` as the database schema source of truth.
-- Make schema changes through Prisma migrations and commit the schema and generated migration together.
-- Never run `pnpm db:migrate:reset` unless the user explicitly requests destructive local reset.
-- Keep queue processors idempotent where retries are possible, and preserve valid crawl-job state transitions.
-- Parse and validate external input at boundaries. Reuse Zod validation and the existing application error shape.
-- Keep configuration access in `src/config/`; document new variables in `.env.example` without real values.
-- Preserve the clean/raw output contract described in `docs/DATA_CONTRACT_V1.md`.
+Mọi luồng dữ liệu nghiệp vụ bắt buộc tuân theo thứ tự phân tầng đơn hướng:
 
-## Generated and runtime files
+```
+Route  →  Controller  →  Service  →  Repository  →  Prisma Client  →  PostgreSQL
+```
 
-- Do not hand-edit `dist/`, `coverage/`, `storage/exports/`, or generated Swagger JSON.
-- When API annotations or routes change, regenerate Swagger with `pnpm swagger`.
+### Quy tắc bất biến:
 
-## Validation
+- **Độc quyền Prisma:** Chỉ duy nhất các file `*.repository.ts` được phép import và gọi `prisma` hoặc `PrismaClient`. Service, Controller, Worker, Helper và Middleware **tuyệt đối không** được gọi Prisma trực tiếp. Đồng thời, các tầng ngoài Repository (DTO, Service, Controller, Middleware) **không import enum từ `@prisma/client`** (ví dụ: `UserRole`, `CrawlJobStatus`, `CrawlMode`, `ExportType`, `AssetType`), mà phải sử dụng types từ `src/common/constants/`.
+- **Tổ chức Module chuẩn (`src/modules/<feature>/`):**
+  - `<feature>.route.ts`: Khai báo endpoints, gắn middleware (`auth`, `role`, `validate`, `rateLimit`).
+  - `<feature>.controller.ts`: Nhận HTTP request, trích xuất parameters, gọi Service, trả response HTTP chuẩn.
+  - `<feature>.service.ts`: Chứa toàn bộ Business Logic, điều phối các Repository và đẩy job vào BullMQ.
+  - `<feature>.repository.ts`: Chịu trách nhiệm duy nhất về tương tác dữ liệu với Prisma (select, filter, transaction).
+  - `<feature>.dto.ts` & `<feature>.validation.ts`: Định nghĩa kiểu TypeScript và schema kiểm thực Zod.
+  - `__tests__/`: Chứa colocated unit/integration tests cho module.
+- **Routing:** Mọi router module mới phải được mount tập trung trong `src/routes/index.ts` với prefix `/api/v1/`.
+- **Mã dùng chung (`src/common/`):** Chỉ đặt vào `src/common/` (errors, helpers, constants, types, storage) khi code thực sự được tái sử dụng qua ít nhất 2 modules.
+- **Chuẩn Hóa Constants & Cấm Tuyệt Đối Hardcode (Zero Hardcode Principle):**
+  - **Tập trung tại `src/common/constants/`:** Mọi chuỗi trạng thái (`JOB_STATUS`), vai trò người dùng (`ROLES`), chế độ thu thập (`CRAWL_MODE`), tần suất lịch (`SCHEDULE_FREQUENCY`), kiểu xuất dữ liệu (`EXPORT_TYPE`), loại tài nguyên (`ASSET_TYPE`), múi giờ (`DEFAULT_TIMEZONE`), v.v. **bắt buộc** phải được định nghĩa trong `src/common/constants/*.constant.ts` dưới dạng object `as const` và export type `keyof typeof CONSTANT`. Barrel export tập trung tại `src/common/constants/index.ts`.
+  - **Cấm hardcode chuỗi / mảng trong code:** Tuyệt đối không viết trực tiếp string literal (ví dụ: `'COMPLETED'`, `'SCRAPE'`, `'ADMIN'`, `'Asia/Ho_Chi_Minh'`) trong Controllers, Services, Workers, DTOs, Repositories, Helpers hoặc Schemas. Luôn dùng `CONSTANT.KEY` hoặc `Object.values(CONSTANT)`.
+  - **Sử dụng trong Zod Validation (`*.validation.ts`):** Luôn dùng `z.nativeEnum(CONSTANT)` thay vì khai báo mảng chuỗi `z.enum(['VAL1', 'VAL2'])`.
+  - **Pattern chuẩn:**
+    ```typescript
+    // 1. Khai báo (src/common/constants/crawl-mode.constant.ts):
+    export const CRAWL_MODE = {
+      SCRAPE: 'SCRAPE',
+      CRAWL: 'CRAWL',
+      SITEMAP: 'SITEMAP',
+      URL_LIST: 'URL_LIST',
+    } as const;
+    export type CrawlMode = keyof typeof CRAWL_MODE;
 
-Run commands from `data-crawler-be/`.
+    // 2. Validation (crawl-job.validation.ts):
+    mode: z.nativeEnum(CRAWL_MODE).optional().default(CRAWL_MODE.SCRAPE)
 
-- Focused test: `pnpm test -- <path-to-test> --runInBand`
-- Test suite: `pnpm test -- --runInBand`
-- Lint: `pnpm lint`
-- Production compile: `pnpm build`
+    // 3. Logic nghiệp vụ (crawl-job.service.ts / crawl.worker.processor.ts):
+    if (job.mode === CRAWL_MODE.URL_LIST) { ... }
+    ```
 
-Add colocated Jest tests under `__tests__/` for service, worker, repository-boundary, export, or contract behavior that changes.
+---
 
+## 2. Dữ Liệu & Tích Hợp (Data, Prisma & Workers)
+
+### A. Cơ sở dữ liệu & Prisma Migrations
+
+- `prisma/schema.prisma` là nguồn chân lý duy nhất (Single Source of Truth) của database schema.
+- Thao tác Prisma thông qua script runner: `node scripts/prisma-run.js <cmd>`. Runner tự động tổng hợp `DATABASE_URL` từ các biến môi trường cấu hình (`DB_HOST`, `DB_PORT`, `DB_USER`, `DB_PASSWORD`, `DB_NAME`, `DB_SSL`).
+- Mọi thay đổi schema phải sinh migration tương ứng bằng `pnpm db:migrate` và commit đồng thời cả `schema.prisma` lẫn thư mục migration.
+- **CẤM:** Không bao giờ chạy `pnpm db:migrate:reset` trừ khi người dùng yêu cầu rõ ràng việc xóa trắng dữ liệu.
+
+### B. Hàng đợi bất đồng bộ & Worker (BullMQ + Redis)
+
+- Các tác vụ nặng (thu thập web, gửi webhook, chạy lịch cron) phải chuyển qua hàng đợi BullMQ:
+  - `crawl.queue.ts` / `crawl.worker.ts` / `crawl.worker.processor.ts`
+  - `webhook.queue.ts` / `webhook.worker.ts`
+  - `schedule.worker.ts`
+- **Idempotency:** Worker processor phải có khả năng chạy lại mà không tạo trùng dữ liệu (dựa trên `url`, `jobId`, `contentHash`).
+- **State Lifecycle:** Bảo toàn chuyển đổi trạng thái hợp lệ của `CrawlJob`:
+  $$\text{PENDING} \longrightarrow \text{QUEUED} \longrightarrow \text{RUNNING} \longrightarrow \text{PROCESSING\_EXPORT} \longrightarrow \text{COMPLETED} \ / \ \text{FAILED} \ / \ \text{CANCELED}$$
+- Ghi log chi tiết theo từng step vào `crawl_job_logs`.
+
+---
+
+## 3. Xác Thực, Xử Lý Lỗi & Hợp Đồng Dữ Liệu
+
+- **Xác thực tại tầng biên:** Toàn bộ Body, Query và Params phải được định nghĩa bằng **Zod** và gắn middleware `validate(schema)`.
+- **Chuẩn hóa lỗi:** Sử dụng `AppError` kèm mã định danh từ `src/common/errors/error-code.ts` (`ERROR_CODE.*`), không dùng `throw new Error()`.
+- **Data Contract V1:** Tuân thủ cấu trúc dữ liệu theo `docs/DATA_CONTRACT_V1.md`:
+  - Phân định rõ ràng giữa dữ liệu thô (raw HTML từ Firecrawl) và dữ liệu đã làm sạch (clean Markdown, structuredData, dataQualityScore).
+- **Lưu trữ tệp:** Sử dụng abstraction `src/common/storage/` (hỗ trợ `local` disk và `s3` / MinIO).
+
+---
+
+## 4. Các Tệp Sinh Tự Động & Runtime (Do Not Hand-Edit)
+
+- **Không chỉnh sửa thủ công:**
+  - `dist/` (build artifacts)
+  - `coverage/` (test reports)
+  - `storage/exports/*` (runtime export files)
+  - `src/docs/swagger.json` (sinh tự động qua Swagger Autogen)
+- Khi thay đổi router, query params, request body hoặc Swagger tags, chạy lại:
+  ```bash
+  pnpm swagger
+  ```
+
+---
+
+## 5. Quy Trình Kiểm Thử & Kiểm Định Chất Lượng (QA Commands)
+
+Thực hiện tất cả các lệnh từ thư mục `data-crawler-be/`:
+
+```bash
+# 1. Chạy test đơn lẻ hoặc theo thư mục
+pnpm test -- <path-to-test> --runInBand
+
+# 2. Chạy toàn bộ test suite
+pnpm test -- --runInBand
+
+# 3. Kiểm tra cú pháp và quy chuẩn mã nguồn
+pnpm lint
+
+# 4. Format mã nguồn
+pnpm format
+
+# 5. Build kiểm tra biên dịch TypeScript
+pnpm build
+```
+
+Bắt buộc bổ sung Jest test trong thư mục `__tests__/` cho mọi Service, Worker, Repository logic, Export pipeline hoặc Data Contract mới được thêm vào hoặc chỉnh sửa.

CRAWL_DATA_EXPRESS_TYPESCRIPT_PRISMA_MIGRATION_INIT.md

@@ -583,10 +583,10 @@ src/database/prisma.client.ts
 ```
 
 ```ts
-import { PrismaClient } from '@prisma/client';
+import { PrismaClient } from "@prisma/client";
 
 export const prisma = new PrismaClient({
-  log: ['error', 'warn'],
+  log: ["error", "warn"],
 });
 ```
 
@@ -594,9 +594,10 @@ Nếu cần log query khi development:
 
 ```ts
 export const prisma = new PrismaClient({
-  log: process.env.NODE_ENV === 'development'
-    ? ['query', 'error', 'warn']
-    : ['error', 'warn'],
+  log:
+    process.env.NODE_ENV === "development"
+      ? ["query", "error", "warn"]
+      : ["error", "warn"],
 });
 ```
 
@@ -628,17 +629,17 @@ Không để repository xử lý nghiệp vụ.
 ### 11.1. Route
 
 ```ts
-import { Router } from 'express';
-import { CrawlJobController } from './crawl-job.controller';
-import { authMiddleware } from '../../middlewares/auth.middleware';
+import { Router } from "express";
+import { CrawlJobController } from "./crawl-job.controller";
+import { authMiddleware } from "../../middlewares/auth.middleware";
 
 const router = Router();
 const controller = new CrawlJobController();
 
-router.post('/', authMiddleware, controller.create);
-router.get('/', authMiddleware, controller.findAll);
-router.get('/:id', authMiddleware, controller.findById);
-router.post('/:id/cancel', authMiddleware, controller.cancel);
+router.post("/", authMiddleware, controller.create);
+router.get("/", authMiddleware, controller.findAll);
+router.get("/:id", authMiddleware, controller.findById);
+router.post("/:id/cancel", authMiddleware, controller.cancel);
 
 export default router;
 ```
@@ -648,8 +649,8 @@ export default router;
 ### 11.2. Controller
 
 ```ts
-import { Request, Response, NextFunction } from 'express';
-import { CrawlJobService } from './crawl-job.service';
+import { Request, Response, NextFunction } from "express";
+import { CrawlJobService } from "./crawl-job.service";
 
 export class CrawlJobController {
   private readonly service = new CrawlJobService();
@@ -717,9 +718,9 @@ export class CrawlJobController {
 ### 11.3. Service
 
 ```ts
-import { CrawlJobRepository } from './crawl-job.repository';
-import { AppError } from '../../common/errors/app-error';
-import { crawlQueue } from '../../queues/crawl.queue';
+import { CrawlJobRepository } from "./crawl-job.repository";
+import { AppError } from "../../common/errors/app-error";
+import { crawlQueue } from "../../queues/crawl.queue";
 
 export class CrawlJobService {
   private readonly repository = new CrawlJobRepository();
@@ -738,7 +739,7 @@ export class CrawlJobService {
       maxDepth: payload.maxDepth,
     });
 
-    await crawlQueue.add('crawl-job', {
+    await crawlQueue.add("crawl-job", {
       jobId: job.id,
     });
 
@@ -753,7 +754,7 @@ export class CrawlJobService {
     const job = await this.repository.findById(jobId);
 
     if (!job || job.userId !== userId) {
-      throw new AppError('Crawl job not found', 404);
+      throw new AppError("Crawl job not found", 404);
     }
 
     return job;
@@ -762,11 +763,11 @@ export class CrawlJobService {
   async cancel(userId: string, jobId: string) {
     const job = await this.findById(userId, jobId);
 
-    if (job.status === 'COMPLETED') {
-      throw new AppError('Completed job cannot be canceled', 400);
+    if (job.status === "COMPLETED") {
+      throw new AppError("Completed job cannot be canceled", 400);
     }
 
-    return this.repository.updateStatus(jobId, 'CANCELED');
+    return this.repository.updateStatus(jobId, "CANCELED");
   }
 }
 ```
@@ -776,8 +777,8 @@ export class CrawlJobService {
 ### 11.4. Repository
 
 ```ts
-import { prisma } from '../../database/prisma.client';
-import { CrawlJobStatus } from '@prisma/client';
+import { prisma } from "../../database/prisma.client";
+import { CrawlJobStatus } from "@prisma/client";
 
 export class CrawlJobRepository {
   create(data: {
@@ -801,7 +802,7 @@ export class CrawlJobRepository {
   findAllByUser(userId: string, query: any) {
     return prisma.crawlJob.findMany({
       where: { userId },
-      orderBy: { createdAt: 'desc' },
+      orderBy: { createdAt: "desc" },
       include: {
         exports: true,
       },
@@ -839,27 +840,27 @@ prisma/seed.ts
 ```
 
 ```ts
-import { PrismaClient, UserRole } from '@prisma/client';
-import bcrypt from 'bcryptjs';
+import { PrismaClient, UserRole } from "@prisma/client";
+import bcrypt from "bcryptjs";
 
 const prisma = new PrismaClient();
 
 async function main() {
-  const passwordHash = await bcrypt.hash('Admin@123456', 10);
+  const passwordHash = await bcrypt.hash("Admin@123456", 10);
 
   await prisma.user.upsert({
-    where: { email: 'admin@crawl.local' },
+    where: { email: "admin@crawl.local" },
     update: {},
     create: {
-      email: 'admin@crawl.local',
+      email: "admin@crawl.local",
       passwordHash,
-      fullName: 'System Admin',
+      fullName: "System Admin",
       role: UserRole.ADMIN,
       isActive: true,
     },
   });
 
-  console.log('Seed completed');
+  console.log("Seed completed");
 }
 
 main()

README.md

@@ -4,16 +4,16 @@ Backend API cho hệ thống crawl dữ liệu web. Người dùng dán link, h
 
 ## Tech Stack
 
-| Thành phần      | Công nghệ                              |
-| --------------- | -------------------------------------- |
-| Runtime         | Node.js + TypeScript                   |
-| Framework       | Express.js                             |
-| ORM             | Prisma (Code First Migration)          |
-| Database        | PostgreSQL                             |
-| Queue / Cache   | Redis + BullMQ                         |
-| Crawl Engine    | Firecrawl API                          |
-| Export          | Archiver, ExcelJS, json2csv, Turndown  |
-| Package Manager | pnpm@9.15.0                            |
+| Thành phần      | Công nghệ                             |
+| --------------- | ------------------------------------- |
+| Runtime         | Node.js + TypeScript                  |
+| Framework       | Express.js                            |
+| ORM             | Prisma (Code First Migration)         |
+| Database        | PostgreSQL                            |
+| Queue / Cache   | Redis + BullMQ                        |
+| Crawl Engine    | Firecrawl API                         |
+| Export          | Archiver, ExcelJS, json2csv, Turndown |
+| Package Manager | pnpm@9.15.0                           |
 
 ## Kiến trúc Service Layer
 
@@ -111,10 +111,10 @@ docker ps
 
 Hai container cần chạy:
 
-| Container            | Service    | Port   |
-| -------------------- | ---------- | ------ |
-| `crawl_data_postgres`| PostgreSQL | `5432` |
-| `crawl_data_redis`   | Redis      | `6379` |
+| Container             | Service    | Port   |
+| --------------------- | ---------- | ------ |
+| `crawl_data_postgres` | PostgreSQL | `5432` |
+| `crawl_data_redis`    | Redis      | `6379` |
 
 > DB name mặc định trong Docker là `crawl_data_db` — đảm bảo `DB_NAME` trong `.env` khớp với giá trị này.
 
@@ -129,6 +129,7 @@ pnpm db:migrate:init
 ```
 
 Lệnh này sẽ:
+
 1. Đọc `prisma/schema.prisma`
 2. Tạo folder migration đầu tiên trong `prisma/migrations/`
 3. Apply migration xuống PostgreSQL
@@ -153,11 +154,11 @@ pnpm db:seed
 
 Seed tạo 3 tài khoản mặc định để test:
 
-| Email                 | Password         | Role          |
-| --------------------- | ---------------- | ------------- |
-| `admin@crawl.local`   | `Admin@123456`   | ADMIN         |
-| `crawl@crawl.local`   | `Crawler@123456` | CRAWLER_USER  |
-| `viewer@crawl.local`  | `Viewer@123456`  | VIEWER        |
+| Email                | Password         | Role         |
+| -------------------- | ---------------- | ------------ |
+| `admin@crawl.local`  | `Admin@123456`   | ADMIN        |
+| `crawl@crawl.local`  | `Crawler@123456` | CRAWLER_USER |
+| `viewer@crawl.local` | `Viewer@123456`  | VIEWER       |
 
 ---
 
@@ -212,20 +213,20 @@ pnpm worker
 
 ## Các lệnh hữu ích
 
-| Lệnh                       | Mô tả                                                      |
-| -------------------------- | ---------------------------------------------------------- |
-| `pnpm db:migrate:init`     | Tạo migration lần đầu (`--name init`)                      |
-| `pnpm db:migrate`          | Tạo migration mới sau khi sửa `schema.prisma`              |
-| `pnpm db:migrate:deploy`   | Apply migration lên staging/production (không dùng dev)    |
-| `pnpm db:migrate:reset`    | Xóa toàn bộ DB và chạy lại migration — **chỉ dùng local** |
-| `pnpm db:migrate:status`   | Xem trạng thái các migration đã apply                      |
-| `pnpm prisma:generate`     | Regenerate Prisma Client sau khi sửa schema thủ công       |
-| `pnpm prisma:studio`       | Mở Prisma Studio — GUI quản lý dữ liệu trực quan          |
-| `pnpm db:seed`             | Chạy seed tạo dữ liệu mẫu                                  |
-| `pnpm swagger`             | Regenerate file `src/docs/swagger.json`                    |
-| `pnpm build`               | Build production bundle ra thư mục `dist/`                 |
-| `pnpm lint`                | Kiểm tra lỗi ESLint                                        |
-| `pnpm format`              | Format code bằng Prettier                                  |
+| Lệnh                     | Mô tả                                                     |
+| ------------------------ | --------------------------------------------------------- |
+| `pnpm db:migrate:init`   | Tạo migration lần đầu (`--name init`)                     |
+| `pnpm db:migrate`        | Tạo migration mới sau khi sửa `schema.prisma`             |
+| `pnpm db:migrate:deploy` | Apply migration lên staging/production (không dùng dev)   |
+| `pnpm db:migrate:reset`  | Xóa toàn bộ DB và chạy lại migration — **chỉ dùng local** |
+| `pnpm db:migrate:status` | Xem trạng thái các migration đã apply                     |
+| `pnpm prisma:generate`   | Regenerate Prisma Client sau khi sửa schema thủ công      |
+| `pnpm prisma:studio`     | Mở Prisma Studio — GUI quản lý dữ liệu trực quan          |
+| `pnpm db:seed`           | Chạy seed tạo dữ liệu mẫu                                 |
+| `pnpm swagger`           | Regenerate file `src/docs/swagger.json`                   |
+| `pnpm build`             | Build production bundle ra thư mục `dist/`                |
+| `pnpm lint`              | Kiểm tra lỗi ESLint                                       |
+| `pnpm format`            | Format code bằng Prettier                                 |
 
 ---
 
@@ -300,11 +301,11 @@ Hệ thống hỗ trợ chuẩn hóa dữ liệu đầu ra **Data Contract v1**,
 
 ### 8.1 Mô hình Hai Lớp Output (Clean vs. Raw Output Model)
 
-| Lớp Output | Trường Dữ Liệu | Đặc Điểm & Mô Tả | Đối Tượng Sử Dụng |
-| :--- | :--- | :--- | :--- |
-| **Raw Output** | `rawMarkdown` | Nội dung Markdown thô nguyên bản thu thập từ crawler, giữ nguyên menu, header, sidebar và footer. | FE Debug / Reconstruct trang gốc |
-| **Clean Output** | `mainContent` | Thân bài chính đã qua thuật toán lọc nhiễu tự động (`extractMainContent`), loại bỏ menu nav, liên kết mạng xã hội, bài viết liên quan và copyright footer. Vẫn giữ cú pháp Markdown. | **AI Agent / LLM Prompt Context / RAG** |
-| **Clean Text** | `cleanText` | Plain text thuần túy đã xóa sạch toàn bộ ký tự định dạng Markdown (`stripMarkdown`). | Đếm từ (`wordCount`) & Hash (`contentHash`) |
+| Lớp Output       | Trường Dữ Liệu | Đặc Điểm & Mô Tả                                                                                                                                                                     | Đối Tượng Sử Dụng                           |
+| :--------------- | :------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------------------------------------------ |
+| **Raw Output**   | `rawMarkdown`  | Nội dung Markdown thô nguyên bản thu thập từ crawler, giữ nguyên menu, header, sidebar và footer.                                                                                    | FE Debug / Reconstruct trang gốc            |
+| **Clean Output** | `mainContent`  | Thân bài chính đã qua thuật toán lọc nhiễu tự động (`extractMainContent`), loại bỏ menu nav, liên kết mạng xã hội, bài viết liên quan và copyright footer. Vẫn giữ cú pháp Markdown. | **AI Agent / LLM Prompt Context / RAG**     |
+| **Clean Text**   | `cleanText`    | Plain text thuần túy đã xóa sạch toàn bộ ký tự định dạng Markdown (`stripMarkdown`).                                                                                                 | Đếm từ (`wordCount`) & Hash (`contentHash`) |
 
 ### 8.2 API Endpoints Preview & Assets
 
@@ -321,6 +322,7 @@ Hệ thống hỗ trợ chuẩn hóa dữ liệu đầu ra **Data Contract v1**,
 ### 8.3 Chỉ số Chất lượng Dữ liệu & Cảnh báo (Quality Metrics & Warnings)
 
 Mỗi bản ghi trang đã crawl trả về đầy đủ các trường đo lường chất lượng:
+
 - **`normalizedUrl`**: URL đã loại bỏ các tham số tracking (`utm_*`, `fbclid`, `gclid`), loại bỏ fragment và chuẩn hóa host/scheme để tránh trùng lặp.
 - **`wordCount`**: Số từ tính trên `cleanText`.
 - **`contentHash`**: Mã SHA-256 tính từ `cleanText` phục vụ deduplication trên Vector DB.
@@ -348,5 +350,3 @@ export-job-c4b8e21a.zip
 ```
 
 Xem chi tiết Data Contract đầy đủ tại [DATA_CONTRACT_V1.md](docs/DATA_CONTRACT_V1.md).
-
-

docker-compose.yml

-version: '3.8'
+version: "3.8"
 
 services:
-  postgres:
-    image: postgres:16-alpine
-    container_name: crawl_data_postgres
-    environment:
-      POSTGRES_USER: ${DB_USER:-postgres}
-      POSTGRES_PASSWORD: ${DB_PASSWORD:-postgres}
-      POSTGRES_DB: ${DB_NAME:-crawl_data_db}
-    ports:
-      - "${DB_PORT:-5432}:5432"
-    volumes:
-      - postgres_data:/var/lib/postgresql/data
+  # postgres:
+  #   image: postgres:16-alpine
+  #   container_name: crawl_data_postgres
+  #   environment:
+  #     POSTGRES_USER: ${DB_USER:-postgres}
+  #     POSTGRES_PASSWORD: ${DB_PASSWORD:-postgres}
+  #     POSTGRES_DB: ${DB_NAME:-crawl_data_db}
+  #   ports:
+  #     - "${DB_PORT:-5432}:5432"
+  #   volumes:
+  #     - postgres_data:/var/lib/postgresql/data
 
   redis:
     image: redis:7-alpine

docs/STORAGE_S3_MINIO.md

@@ -4,10 +4,10 @@
 
 Storage được chọn bằng `STORAGE_DRIVER`:
 
-| Cấu hình | Nơi lưu file |
-| --- | --- |
-| `STORAGE_DRIVER=local` | Lưu trực tiếp trong `STORAGE_EXPORT_DIR` |
-| `STORAGE_DRIVER=s3` | Dùng AWS S3, MinIO hoặc dịch vụ tương thích S3 |
+| Cấu hình               | Nơi lưu file                                   |
+| ---------------------- | ---------------------------------------------- |
+| `STORAGE_DRIVER=local` | Lưu trực tiếp trong `STORAGE_EXPORT_DIR`       |
+| `STORAGE_DRIVER=s3`    | Dùng AWS S3, MinIO hoặc dịch vụ tương thích S3 |
 
 Với JSON, CSV, XLSX và Markdown, hệ thống tạo file staging rồi upload lên storage. ZIP được stream trực tiếp lên storage, không tạo file ZIP local.
 

docs/postman/README.md

@@ -7,6 +7,7 @@ Tài liệu này hướng dẫn cách vận hành, cấu trúc và danh sách c
 ## 🚀 1. Cách chạy kiểm thử
 
 ### Cách 1: Chạy trực tiếp bằng Postman (GUI)
+
 1. **Import vào Postman**:
    - File Collection: [data-crawler.postman_collection.json](./data-crawler.postman_collection.json)
    - File Environment: [data-crawler.postman_environment.json](./data-crawler.postman_environment.json)
@@ -17,6 +18,7 @@ Tài liệu này hướng dẫn cách vận hành, cấu trúc và danh sách c
    - Nhấn **Run Data Crawler BE API**.
 
 ### Cách 2: Chạy tự động bằng Newman (CLI)
+
 Yêu cầu đã khởi chạy Server Backend tại `http://localhost:3000`. Chạy lệnh sau tại thư mục gốc của dự án:
 
 ```bash
@@ -24,6 +26,7 @@ npx newman run "docs/postman/data-crawler.postman_collection.json" -e "docs/post
 ```
 
 Chạy từng folder kịch bản cụ thể:
+
 ```bash
 npx newman run "docs/postman/data-crawler.postman_collection.json" -e "docs/postman/data-crawler.postman_environment.json" --folder "Crawl Jobs" --reporters cli
 ```
@@ -33,72 +36,75 @@ npx newman run "docs/postman/data-crawler.postman_collection.json" -e "docs/post
 ## 📋 2. Chi tiết các Nhóm Kiểm thử (Test Suites)
 
 ### 🔐 A. Nhóm Auth (Xác thực & Ủy quyền)
+
 Kiểm tra luồng đăng nhập, lấy thông tin cá nhân, cập nhật tài khoản và cơ chế refresh token.
 
 1. **Login**:
-   - *Endpoint*: `POST /auth/login`
-   - *Test Assertions*: Trả về HTTP 200, `success: true`, sinh ra `accessToken` và `refreshToken`, tự động lưu vào môi trường Postman (`token`, `refreshToken`).
+   - _Endpoint_: `POST /auth/login`
+   - _Test Assertions_: Trả về HTTP 200, `success: true`, sinh ra `accessToken` và `refreshToken`, tự động lưu vào môi trường Postman (`token`, `refreshToken`).
 2. **Get Me**:
-   - *Endpoint*: `GET /auth/me`
-   - *Test Assertions*: Đính kèm Bearer token. Trả về chính xác thông tin User (`id`, `email`, `role`, `isActive`).
+   - _Endpoint_: `GET /auth/me`
+   - _Test Assertions_: Đính kèm Bearer token. Trả về chính xác thông tin User (`id`, `email`, `role`, `isActive`).
 3. **Refresh Token**:
-   - *Endpoint*: `POST /auth/refresh`
-   - *Test Assertions*: Nhận `refreshToken`, cấp lại `accessToken` mới, cập nhật lại biến môi trường.
+   - _Endpoint_: `POST /auth/refresh`
+   - _Test Assertions_: Nhận `refreshToken`, cấp lại `accessToken` mới, cập nhật lại biến môi trường.
 4. **Logout**:
-   - *Endpoint*: `POST /auth/logout`
-   - *Test Assertions*: Thu hồi token trong database, xóa khỏi biến môi trường Postman.
+   - _Endpoint_: `POST /auth/logout`
+   - _Test Assertions_: Thu hồi token trong database, xóa khỏi biến môi trường Postman.
 5. **Login with Invalid Credentials (Edge Case)**:
-   - *Test Assertions*: Trả về `401 Unauthorized`.
+   - _Test Assertions_: Trả về `401 Unauthorized`.
 6. **Get Me without Token (Edge Case)**:
-   - *Test Assertions*: Trả về `401 Unauthorized`.
+   - _Test Assertions_: Trả về `401 Unauthorized`.
 
 ---
 
 ### ⚙️ B. Nhóm Crawl Jobs & Clean/Raw Preview
+
 Kiểm tra các hoạt động tạo tác vụ crawl, lấy danh sách trang, xem trước dữ liệu sạch/thô và lọc chất lượng.
 
 1. **Create Crawl Job**:
-   - *Endpoint*: `POST /crawl-jobs`
-   - *Body Payload*: `{ "startUrl": "https://example.com", "mode": "CRAWL", "maxPages": 50, "maxDepth": 2 }`
-   - *Test Assertions*: HTTP 201 Created, trả về job ID mới (`job_id`).
+   - _Endpoint_: `POST /crawl-jobs`
+   - _Body Payload_: `{ "startUrl": "https://example.com", "mode": "CRAWL", "maxPages": 50, "maxDepth": 2 }`
+   - _Test Assertions_: HTTP 201 Created, trả về job ID mới (`job_id`).
 2. **Get Crawl Jobs (Paginated & Filtered)**:
-   - *Endpoint*: `GET /crawl-jobs?status=PENDING&page=1&limit=10`
-   - *Test Assertions*: Trả về danh sách phân trang `{ items: Array, meta: { total, page, limit, totalPages } }`.
+   - _Endpoint_: `GET /crawl-jobs?status=PENDING&page=1&limit=10`
+   - _Test Assertions_: Trả về danh sách phân trang `{ items: Array, meta: { total, page, limit, totalPages } }`.
 3. **Get Crawl Job by ID**:
-   - *Endpoint*: `GET /crawl-jobs/:id`
-   - *Test Assertions*: Trả về chi tiết các thông số của Job (`startUrl`, `mode`, `status`, `totalPages`, `successPages`, `failedPages`).
+   - _Endpoint_: `GET /crawl-jobs/:id`
+   - _Test Assertions_: Trả về chi tiết các thông số của Job (`startUrl`, `mode`, `status`, `totalPages`, `successPages`, `failedPages`).
 4. **Get Crawled Pages (Metadata List)**:
-   - *Endpoint*: `GET /crawl-jobs/:id/pages`
-   - *Query Params*: Supports `status`, `statusCode`, `search`, `dataQualityScore`, `hasTables`, `hasImages`, `hasLinks`, `wordCount`, `sortBy`, `order`.
-   - *Test Assertions*: Trả về mảng danh sách trang kèm theo `normalizedUrl`, `dataQualityScore`, `wordCount`, `warnings`, `hasSensitiveData` (không chứa payload markdown dài để tối ưu băng thông).
+   - _Endpoint_: `GET /crawl-jobs/:id/pages`
+   - _Query Params_: Supports `status`, `statusCode`, `search`, `dataQualityScore`, `hasTables`, `hasImages`, `hasLinks`, `wordCount`, `sortBy`, `order`.
+   - _Test Assertions_: Trả về mảng danh sách trang kèm theo `normalizedUrl`, `dataQualityScore`, `wordCount`, `warnings`, `hasSensitiveData` (không chứa payload markdown dài để tối ưu băng thông).
 5. **Get Crawled Pages Preview (Clean vs. Raw Output)**:
-   - *Endpoint*: `GET /crawl-jobs/:id/pages/preview?minQualityScore=50` (hoặc `GET /crawl-jobs/:id/pages?preview=true`)
-   - *Test Assertions*:
+   - _Endpoint_: `GET /crawl-jobs/:id/pages/preview?minQualityScore=50` (hoặc `GET /crawl-jobs/:id/pages?preview=true`)
+   - _Test Assertions_:
      - Phải chứa đủ 3 trường nội dung đại diện cho hai lớp Output:
        - `rawMarkdown`: Markdown thô nguyên bản thu thập được.
        - `mainContent`: Thân bài chính đã qua lọc bỏ nhiễu nav/footer/sidebar **(Khuyến nghị cho AI Agents / LLM)**.
        - `cleanText`: Văn bản thuần túy đã xóa sạch ký tự định dạng Markdown.
      - Hỗ trợ kiểm tra các chỉ số chất lượng: `dataQualityScore` (0-100), `wordCount`, `contentHash` (SHA-256), `warnings` (`NAV_NOISE`, `TOO_SHORT`, `DUPLICATE_CONTENT`, ...).
 6. **Get Job Assets**:
-   - *Endpoint*: `GET /crawl-jobs/:id/assets?assetType=IMAGE`
-   - *Query Params*: `assetType` (enum: `IMAGE`, `LINK`, `PDF`, `FILE`, `VIDEO`, `OTHER`).
-   - *Test Assertions*: Trả về danh sách tài nguyên hình ảnh/liên kết thu thập được từ các trang.
+   - _Endpoint_: `GET /crawl-jobs/:id/assets?assetType=IMAGE`
+   - _Query Params_: `assetType` (enum: `IMAGE`, `LINK`, `PDF`, `FILE`, `VIDEO`, `OTHER`).
+   - _Test Assertions_: Trả về danh sách tài nguyên hình ảnh/liên kết thu thập được từ các trang.
 
 ---
 
 ### 💾 C. Nhóm Export & Download
+
 Kiểm tra luồng khởi tạo và tải về các tập tin xuất bản cho Crawl Job đã hoàn thành (`COMPLETED`).
 
 1. **Get Crawl Job Exports**:
-   - *Endpoint*: `GET /crawl-jobs/:id/exports`
-   - *Test Assertions*: Trả về danh sách các tệp tin xuất bản đã tạo của Job.
+   - _Endpoint_: `GET /crawl-jobs/:id/exports`
+   - _Test Assertions_: Trả về danh sách các tệp tin xuất bản đã tạo của Job.
 2. **Create Export for Job**:
-   - *Endpoint*: `POST /crawl-jobs/:id/exports`
-   - *Body Payload*: `{ "exportType": "ZIP" }` (Các định dạng hỗ trợ: `JSON`, `CSV`, `XLSX`, `MARKDOWN`, `ZIP`).
-   - *Test Assertions*: HTTP 201 Created, khởi tạo bản export thành công và lưu `export_id`.
+   - _Endpoint_: `POST /crawl-jobs/:id/exports`
+   - _Body Payload_: `{ "exportType": "ZIP" }` (Các định dạng hỗ trợ: `JSON`, `CSV`, `XLSX`, `MARKDOWN`, `ZIP`).
+   - _Test Assertions_: HTTP 201 Created, khởi tạo bản export thành công và lưu `export_id`.
 3. **Download Export File**:
-   - *Endpoint*: `GET /exports/:exportId/download` (hoặc `GET /crawl-jobs/:id/download`)
-   - *Test Assertions*: Trả về stream binary tệp tin kèm theo đúng Header `Content-Disposition`.
+   - _Endpoint_: `GET /exports/:exportId/download` (hoặc `GET /crawl-jobs/:id/download`)
+   - _Test Assertions_: Trả về stream binary tệp tin kèm theo đúng Header `Content-Disposition`.
    - **Đặc quyền cấu trúc ZIP Output**:
      - Thư mục `/data/raw/pages.raw.json` & `/data/clean/pages.clean.json`.
      - Thư mục `/markdown/raw/` & `/markdown/clean/`.
@@ -107,6 +113,7 @@ Kiểm tra luồng khởi tạo và tải về các tập tin xuất bản cho C
 ---
 
 ### 🛡️ D. Nhóm Permission & Edge Case Tests (Phân quyền & Lỗi nghiệp vụ)
+
 1. **Get Non-Existent Job**:
    - Gửi ID UUID không tồn tại -> Kiểm tra phản hồi `404 Not Found` và `code: "CRAWL_JOB_NOT_FOUND"`.
 2. **Cancel Completed Job**:

jest.config.js

 module.exports = {
-  preset: 'ts-jest',
-  testEnvironment: 'node',
-  testMatch: ['**/*.test.ts'],
-  moduleFileExtensions: ['ts', 'js', 'json'],
+  preset: "ts-jest",
+  testEnvironment: "node",
+  testMatch: ["**/*.test.ts"],
+  moduleFileExtensions: ["ts", "js", "json"],
   moduleNameMapper: {
-    'node-html-parser': '<rootDir>/src/__mocks__/node-html-parser.ts',
+    "node-html-parser": "<rootDir>/src/__mocks__/node-html-parser.ts",
   },
-  modulePathIgnorePatterns: ['<rootDir>/dist/'],
-};
\ No newline at end of file
+  modulePathIgnorePatterns: ["<rootDir>/dist/"],
+  setupFiles: ["<rootDir>/jest.setup.ts"],
+};