Commit 5b123692 authored by ThinhNC's avatar ThinhNC

feat(security,rbac): resolve 17 audit findings and migrate routes to dynamic permissions

parent e0406ff9
......@@ -110,14 +110,14 @@ Small, focused changes are easier to review, faster to merge, and safer to deplo
~1000 lines changed → Too large. Split it.
```
**Watch file size, not just diff size.** A small diff can still push a file past a healthy boundary — around 1000 *total* lines in a single file (distinct from the ~1000 *changed*-lines threshold above) is a common inspection signal, not a hard cap. When a change materially grows an already-large file, ask whether to extract helpers, subcomponents, or modules *first*, before piling more on. Decompose, then add.
**Watch file size, not just diff size.** A small diff can still push a file past a healthy boundary — around 1000 _total_ lines in a single file (distinct from the ~1000 _changed_-lines threshold above) is a common inspection signal, not a hard cap. When a change materially grows an already-large file, ask whether to extract helpers, subcomponents, or modules _first_, before piling more on. Decompose, then add.
**What counts as "one change":** A single self-contained modification that addresses one thing, includes related tests, and keeps the system functional after submission. One part of a feature — not the whole feature.
**Splitting strategies when a change is too large:**
| Strategy | How | When |
|----------|-----|------|
| ----------------- | ------------------------------------------------------- | ----------------------- |
| **Stack** | Submit a small change, start the next one based on it | Sequential dependencies |
| **By file group** | Separate changes for groups needing different reviewers | Cross-cutting concerns |
| **Horizontal** | Create shared code/stubs first, then consumers | Layered architecture |
......@@ -179,8 +179,8 @@ For each file changed:
Label every comment with its severity so the author knows what's required vs optional:
| Prefix | Meaning | Author Action |
|--------|---------|---------------|
| *(no prefix)* | Required change | Must address before merge |
| ----------------------------- | ------------------ | ------------------------------------------------------- |
| _(no prefix)_ | Required change | Must address before merge |
| **Critical:** | Blocks merge | Security vulnerability, data loss, broken functionality |
| **Nit:** | Minor, optional | Author may ignore — formatting, style preferences |
| **Optional:** / **Consider:** | Suggestion | Worth considering but not required |
......@@ -188,7 +188,7 @@ Label every comment with its severity so the author knows what's required vs opt
This prevents authors from treating all feedback as mandatory and wasting time on optional suggestions.
**Lead with what matters.** Order findings by leverage: correctness and security first, then structural regressions and missed simplifications, then everything else. Don't bury a real issue under cosmetic nits — a few high-conviction comments beat a long list. If you have one structural problem and ten nits, the structural problem *is* the review.
**Lead with what matters.** Order findings by leverage: correctness and security first, then structural regressions and missed simplifications, then everything else. Don't bury a real issue under cosmetic nits — a few high-conviction comments beat a long list. If you have one structural problem and ten nits, the structural problem _is_ the review.
### Step 5: Verify the Verification
......@@ -222,6 +222,7 @@ Human makes the final call
This catches issues that a single model might miss — different models have different blind spots.
**Example prompt for a review agent:**
```
Review this code change for correctness, security, and adherence to
our project conventions. The spec says [X]. The change should [Y].
......@@ -281,6 +282,7 @@ When reviewing code — whether written by you, another agent, or a human:
Part of code review is dependency review:
**Before adding any dependency:**
1. Does the existing stack solve this? (Often it does.)
2. How large is the dependency? (Check bundle impact.)
3. Is it actively maintained? (Check last commit, open issues.)
......@@ -293,11 +295,11 @@ Part of code review is dependency review:
1. **Read the changelog, not just the version number.** Semver is a promise the maintainer may not have kept — a "patch" can carry a behavioral change. For a major bump, read the migration notes and find what breaks.
2. **One dependency per change.** Upgrade and merge them individually (or in small related groups). When a bulk bump breaks the build, you've lost which package did it; a single-package change makes the cause obvious and the revert clean.
3. **Let the tests decide.** The upgrade is verified by a green suite before *and* after, not by "it installed." If coverage around the dependency's behavior is thin, that gap is the real finding — add a test first.
3. **Let the tests decide.** The upgrade is verified by a green suite before _and_ after, not by "it installed." If coverage around the dependency's behavior is thin, that gap is the real finding — add a test first.
4. **Mind the transitive graph.** Most installed packages are ones nobody chose directly. Review the lockfile diff, not just `package.json`; a single direct bump can pull in dozens of indirect changes.
5. **Keep the lockfile honest.** Commit it, review its diff, and never hand-edit it. The lockfile is the thing that actually pins what ships.
For triaging `npm audit` findings and supply-chain risk (typosquatting, compromised maintainers), follow the `security-and-hardening` skill — this section covers the upgrade *workflow*, that one covers the security verdict.
For triaging `npm audit` findings and supply-chain risk (typosquatting, compromised maintainers), follow the `security-and-hardening` skill — this section covers the upgrade _workflow_, that one covers the security verdict.
## The Review Checklist
......@@ -305,20 +307,24 @@ For triaging `npm audit` findings and supply-chain risk (typosquatting, compromi
## Review: [PR/Change title]
### Context
- [ ] I understand what this change does and why
### Correctness
- [ ] Change matches spec/task requirements
- [ ] Edge cases handled
- [ ] Error paths handled
- [ ] Tests cover the change adequately
### Readability
- [ ] Names are clear and consistent
- [ ] Logic is straightforward
- [ ] No unnecessary complexity
### Architecture
- [ ] Follows existing patterns
- [ ] No unnecessary coupling or dependencies
- [ ] Appropriate abstraction level
......@@ -326,6 +332,7 @@ For triaging `npm audit` findings and supply-chain risk (typosquatting, compromi
- [ ] No feature logic in shared modules; file stays within a healthy size
### Security
- [ ] No secrets in code
- [ ] Input validated at boundaries
- [ ] No injection vulnerabilities
......@@ -333,19 +340,23 @@ For triaging `npm audit` findings and supply-chain risk (typosquatting, compromi
- [ ] External data sources treated as untrusted
### Performance
- [ ] No N+1 patterns
- [ ] No unbounded operations
- [ ] Pagination on list endpoints
### Verification
- [ ] Tests pass
- [ ] Build succeeds
- [ ] Manual verification done (if applicable)
### Verdict
- [ ] **Approve** — Ready to merge
- [ ] **Request changes** — Issues must be addressed
```
## See Also
- For detailed security review guidance, see `../../references/security-checklist.md`
......@@ -354,7 +365,7 @@ For triaging `npm audit` findings and supply-chain risk (typosquatting, compromi
## Common Rationalizations
| Rationalization | Reality |
|---|---|
| ------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| "It works, that's good enough" | Working code that's unreadable, insecure, or architecturally wrong creates debt that compounds. |
| "I wrote it, so I know it's correct" | Authors are blind to their own assumptions. Every change benefits from another set of eyes. |
| "We'll clean it up later" | Later never comes. The review is the quality gate — use it. Require cleanup before merge, not after. |
......
/*
Warnings:
- You are about to drop the `CrawlJobLog` table. If the table is not empty, all the data it contains will be lost.
*/
-- DropForeignKey
ALTER TABLE "CrawlJobLog" DROP CONSTRAINT "CrawlJobLog_job_id_fkey";
-- DropForeignKey
ALTER TABLE "crawl_assets" DROP CONSTRAINT "crawl_assets_crawl_job_id_fkey";
-- DropTable
DROP TABLE "CrawlJobLog";
-- CreateTable
CREATE TABLE "crawl_job_logs" (
"id" UUID NOT NULL,
"job_id" UUID NOT NULL,
"level" "LogLevel" NOT NULL,
"step" TEXT NOT NULL,
"message" TEXT NOT NULL,
"created_at" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
CONSTRAINT "crawl_job_logs_pkey" PRIMARY KEY ("id")
);
-- CreateIndex
CREATE INDEX "crawl_job_logs_job_id_created_at_idx" ON "crawl_job_logs"("job_id", "created_at");
-- CreateIndex
CREATE INDEX "audit_logs_user_id_created_at_idx" ON "audit_logs"("user_id", "created_at");
-- CreateIndex
CREATE INDEX "audit_logs_action_idx" ON "audit_logs"("action");
-- CreateIndex
CREATE INDEX "audit_logs_created_at_idx" ON "audit_logs"("created_at");
-- CreateIndex
CREATE INDEX "audit_logs_ip_address_idx" ON "audit_logs"("ip_address");
-- CreateIndex
CREATE INDEX "crawl_assets_crawl_job_id_idx" ON "crawl_assets"("crawl_job_id");
-- CreateIndex
CREATE INDEX "crawl_jobs_user_id_created_at_idx" ON "crawl_jobs"("user_id", "created_at");
-- AddForeignKey
ALTER TABLE "crawl_assets" ADD CONSTRAINT "crawl_assets_crawl_job_id_fkey" FOREIGN KEY ("crawl_job_id") REFERENCES "crawl_jobs"("id") ON DELETE CASCADE ON UPDATE CASCADE;
-- AddForeignKey
ALTER TABLE "crawl_job_logs" ADD CONSTRAINT "crawl_job_logs_job_id_fkey" FOREIGN KEY ("job_id") REFERENCES "crawl_jobs"("id") ON DELETE CASCADE ON UPDATE CASCADE;
......@@ -152,6 +152,7 @@ model CrawlJob {
@@index([status])
@@index([createdAt])
@@index([userId, status])
@@index([userId, createdAt])
@@index([scheduleId])
@@map("crawl_jobs")
}
......@@ -206,7 +207,7 @@ model CrawlAsset {
createdAt DateTime @default(now()) @map("created_at")
page CrawlPage? @relation(fields: [pageId], references: [id], onDelete: SetNull)
crawlJob CrawlJob? @relation(fields: [crawlJobId], references: [id])
crawlJob CrawlJob? @relation(fields: [crawlJobId], references: [id], onDelete: Cascade)
crawlJobId String? @map("crawl_job_id") @db.Uuid
@@index([pageId])
......
......@@ -19,19 +19,12 @@ const app = express();
app.set("trust proxy", parseTrustProxy(envConfig.trustProxy));
app.use(
helmet({
contentSecurityPolicy: false, // Vô hiệu hóa CSP để Swagger UI load stylesheet bình thường
}),
);
app.use(helmet());
app.use(
cors({
origin: (origin, callback) => {
if (!origin) return callback(null, true);
if (
envConfig.cors.allowedOrigins.includes(origin) ||
envConfig.cors.allowedOrigins.includes("*")
) {
if (envConfig.cors.allowedOrigins.includes(origin)) {
return callback(null, true);
}
return callback(new Error(`Origin ${origin} not allowed by CORS`));
......@@ -46,7 +39,12 @@ app.use(express.json());
app.use(express.urlencoded({ extended: true }));
app.use("/health", healthRoute);
app.use("/api-docs", swaggerUi.serve, swaggerUi.setup(swaggerDocument));
app.use(
"/api-docs",
helmet({ contentSecurityPolicy: false }),
swaggerUi.serve,
swaggerUi.setup(swaggerDocument),
);
app.use("/api/v1", rateLimitMiddleware, routes);
app.use(notFoundMiddleware);
......
......@@ -41,6 +41,26 @@ export const PERMISSIONS = {
EXPORTS_READ_ALL: "exports.read_all",
EXPORTS_CREATE: "exports.create",
EXPORTS_DOWNLOAD: "exports.download",
EXPORTS_DELETE: "exports.delete",
// Webhooks
WEBHOOKS_READ: "webhooks.read",
WEBHOOKS_CREATE: "webhooks.create",
WEBHOOKS_UPDATE: "webhooks.update",
WEBHOOKS_DELETE: "webhooks.delete",
WEBHOOKS_TEST: "webhooks.test",
// Extraction Templates
EXTRACTION_TEMPLATES_READ: "extraction_templates.read",
EXTRACTION_TEMPLATES_CREATE: "extraction_templates.create",
EXTRACTION_TEMPLATES_UPDATE: "extraction_templates.update",
EXTRACTION_TEMPLATES_DELETE: "extraction_templates.delete",
// API Keys
API_KEYS_READ: "api_keys.read",
API_KEYS_CREATE: "api_keys.create",
API_KEYS_UPDATE: "api_keys.update",
API_KEYS_DELETE: "api_keys.delete",
// Audit Logs
AUDIT_LOGS_READ: "audit_logs.read",
......@@ -305,6 +325,124 @@ export const SYSTEM_PERMISSIONS_CATALOG: PermissionDefinition[] = [
action: "download",
isSystem: true,
},
{
name: "Delete Export",
slug: PERMISSIONS.EXPORTS_DELETE,
description: "Xóa tệp trích xuất dữ liệu",
resource: "exports",
action: "delete",
isSystem: true,
},
// Webhooks
{
name: "View Webhooks",
slug: PERMISSIONS.WEBHOOKS_READ,
description: "Xem cấu hình và nhật ký gửi webhook",
resource: "webhooks",
action: "read",
isSystem: true,
},
{
name: "Create Webhook",
slug: PERMISSIONS.WEBHOOKS_CREATE,
description: "Tạo cấu hình webhook mới",
resource: "webhooks",
action: "create",
isSystem: true,
},
{
name: "Update Webhook",
slug: PERMISSIONS.WEBHOOKS_UPDATE,
description: "Cập nhật cấu hình webhook hoặc gửi lại",
resource: "webhooks",
action: "update",
isSystem: true,
},
{
name: "Delete Webhook",
slug: PERMISSIONS.WEBHOOKS_DELETE,
description: "Xóa cấu hình webhook",
resource: "webhooks",
action: "delete",
isSystem: true,
},
{
name: "Test Webhook",
slug: PERMISSIONS.WEBHOOKS_TEST,
description: "Gửi kiểm thử ping webhook",
resource: "webhooks",
action: "test",
isSystem: true,
},
// Extraction Templates
{
name: "View Extraction Templates",
slug: PERMISSIONS.EXTRACTION_TEMPLATES_READ,
description: "Xem mẫu bóc tách dữ liệu",
resource: "extraction_templates",
action: "read",
isSystem: true,
},
{
name: "Create Extraction Template",
slug: PERMISSIONS.EXTRACTION_TEMPLATES_CREATE,
description: "Tạo mẫu bóc tách dữ liệu mới",
resource: "extraction_templates",
action: "create",
isSystem: true,
},
{
name: "Update Extraction Template",
slug: PERMISSIONS.EXTRACTION_TEMPLATES_UPDATE,
description: "Cập nhật mẫu bóc tách dữ liệu",
resource: "extraction_templates",
action: "update",
isSystem: true,
},
{
name: "Delete Extraction Template",
slug: PERMISSIONS.EXTRACTION_TEMPLATES_DELETE,
description: "Xóa mẫu bóc tách dữ liệu",
resource: "extraction_templates",
action: "delete",
isSystem: true,
},
// API Keys
{
name: "View API Keys",
slug: PERMISSIONS.API_KEYS_READ,
description: "Xem danh sách khóa API",
resource: "api_keys",
action: "read",
isSystem: true,
},
{
name: "Create API Key",
slug: PERMISSIONS.API_KEYS_CREATE,
description: "Tạo khóa API mới",
resource: "api_keys",
action: "create",
isSystem: true,
},
{
name: "Update API Key",
slug: PERMISSIONS.API_KEYS_UPDATE,
description: "Kích hoạt hoặc vô hiệu hóa khóa API",
resource: "api_keys",
action: "update",
isSystem: true,
},
{
name: "Delete API Key",
slug: PERMISSIONS.API_KEYS_DELETE,
description: "Thu hồi và xóa khóa API",
resource: "api_keys",
action: "delete",
isSystem: true,
},
// Audit Logs
{
......@@ -370,6 +508,20 @@ export const SYSTEM_ROLE_DEFAULT_PERMISSIONS: Record<
PERMISSIONS.EXPORTS_READ_ALL,
PERMISSIONS.EXPORTS_CREATE,
PERMISSIONS.EXPORTS_DOWNLOAD,
PERMISSIONS.EXPORTS_DELETE,
PERMISSIONS.WEBHOOKS_READ,
PERMISSIONS.WEBHOOKS_CREATE,
PERMISSIONS.WEBHOOKS_UPDATE,
PERMISSIONS.WEBHOOKS_DELETE,
PERMISSIONS.WEBHOOKS_TEST,
PERMISSIONS.EXTRACTION_TEMPLATES_READ,
PERMISSIONS.EXTRACTION_TEMPLATES_CREATE,
PERMISSIONS.EXTRACTION_TEMPLATES_UPDATE,
PERMISSIONS.EXTRACTION_TEMPLATES_DELETE,
PERMISSIONS.API_KEYS_READ,
PERMISSIONS.API_KEYS_CREATE,
PERMISSIONS.API_KEYS_UPDATE,
PERMISSIONS.API_KEYS_DELETE,
PERMISSIONS.AUDIT_LOGS_READ,
PERMISSIONS.DASHBOARD_READ,
PERMISSIONS.DASHBOARD_READ_ALL,
......@@ -379,6 +531,7 @@ export const SYSTEM_ROLE_DEFAULT_PERMISSIONS: Record<
PERMISSIONS.CRAWL_JOBS_READ,
PERMISSIONS.CRAWL_JOBS_CANCEL,
PERMISSIONS.CRAWL_JOBS_RETRY,
PERMISSIONS.CRAWL_JOBS_DELETE,
PERMISSIONS.CRAWL_SCHEDULES_CREATE,
PERMISSIONS.CRAWL_SCHEDULES_READ,
PERMISSIONS.CRAWL_SCHEDULES_UPDATE,
......@@ -387,12 +540,30 @@ export const SYSTEM_ROLE_DEFAULT_PERMISSIONS: Record<
PERMISSIONS.EXPORTS_READ,
PERMISSIONS.EXPORTS_CREATE,
PERMISSIONS.EXPORTS_DOWNLOAD,
PERMISSIONS.EXPORTS_DELETE,
PERMISSIONS.DASHBOARD_READ,
PERMISSIONS.WEBHOOKS_READ,
PERMISSIONS.WEBHOOKS_CREATE,
PERMISSIONS.WEBHOOKS_UPDATE,
PERMISSIONS.WEBHOOKS_DELETE,
PERMISSIONS.WEBHOOKS_TEST,
PERMISSIONS.EXTRACTION_TEMPLATES_READ,
PERMISSIONS.EXTRACTION_TEMPLATES_CREATE,
PERMISSIONS.EXTRACTION_TEMPLATES_UPDATE,
PERMISSIONS.EXTRACTION_TEMPLATES_DELETE,
PERMISSIONS.API_KEYS_READ,
PERMISSIONS.API_KEYS_CREATE,
PERMISSIONS.API_KEYS_UPDATE,
PERMISSIONS.API_KEYS_DELETE,
],
[SYSTEM_ROLE_SLUGS.VIEWER]: [
PERMISSIONS.CRAWL_JOBS_READ,
PERMISSIONS.CRAWL_SCHEDULES_READ,
PERMISSIONS.EXPORTS_READ,
PERMISSIONS.EXPORTS_DOWNLOAD,
PERMISSIONS.DASHBOARD_READ,
PERMISSIONS.WEBHOOKS_READ,
PERMISSIONS.EXTRACTION_TEMPLATES_READ,
PERMISSIONS.API_KEYS_READ,
],
};
......@@ -31,6 +31,7 @@ export const ERROR_CODE = {
PRIVILEGE_ESCALATION_DENIED: "PRIVILEGE_ESCALATION_DENIED",
SYSTEM_ROLE_PROTECTED: "SYSTEM_ROLE_PROTECTED",
CANNOT_REMOVE_LAST_SUPER_ADMIN: "CANNOT_REMOVE_LAST_SUPER_ADMIN",
RATE_LIMIT_EXCEEDED: "RATE_LIMIT_EXCEEDED",
} as const;
export type ErrorCode = keyof typeof ERROR_CODE;
......@@ -14,6 +14,8 @@ import {
DATA_QUALITY_MIN_SCORE,
DATA_CONTRACT_HASH_ALGORITHM,
} from "../constants/data-contract.constant";
import { ASSET_TYPE } from "../constants/asset-type.constant";
import { CRAWL_PAGE_STATUS } from "../constants/crawl-page-status.constant";
/**
* Normalize một URL để phục vụ deduplicate và so sánh.
......@@ -321,7 +323,7 @@ export function transformImages(assets: CrawlAsset[]): ImageRecord[] {
const seenUrls = new Set<string>();
return assets
.filter((a) => {
if (a.assetType !== "IMAGE") return false;
if (a.assetType !== ASSET_TYPE.IMAGE) return false;
if (seenUrls.has(a.url)) return false;
seenUrls.add(a.url);
return true;
......@@ -366,7 +368,7 @@ export function transformPageToRecord(
const { page, assets, tables = [], jobDomain, seenContentHashes } = options;
const normalizedUrl = page.normalizedUrl || normalizeUrl(page.url);
const isSuccess = page.status === "SUCCESS";
const isSuccess = page.status === CRAWL_PAGE_STATUS.SUCCESS;
// Clean text từ markdownContent
const rawMarkdown = page.markdownContent ?? null;
......
......@@ -32,11 +32,60 @@ export function errorMiddleware(
return;
}
// Handle Prisma Known Request Errors
const prismaError = error as { code?: string; meta?: { target?: string[] } };
if (
prismaError.code &&
typeof prismaError.code === "string" &&
prismaError.code.startsWith("P")
) {
switch (prismaError.code) {
case "P2002": {
const target = Array.isArray(prismaError.meta?.target)
? prismaError.meta.target.join(", ")
: "field";
res.status(409).json({
success: false,
message: `A record with this ${target} already exists.`,
code: ERROR_CODE.DUPLICATE_ENTRY,
});
return;
}
case "P2023": {
res.status(400).json({
success: false,
message: "Invalid input format or malformed identifier.",
code: ERROR_CODE.VALIDATION_ERROR,
});
return;
}
case "P2025": {
res.status(404).json({
success: false,
message: "Requested record not found.",
code: ERROR_CODE.NOT_FOUND,
});
return;
}
case "P2003": {
res.status(400).json({
success: false,
message:
"Referenced record does not exist or relation constraint failed.",
code: ERROR_CODE.VALIDATION_ERROR,
});
return;
}
default:
break;
}
}
console.error("[Unhandled Error]", error);
res.status(500).json({
success: false,
message: "Internal server error",
code: "INTERNAL_SERVER_ERROR",
code: ERROR_CODE.INTERNAL_SERVER_ERROR,
});
}
import { Request, Response, NextFunction } from "express";
import { PermissionSlug } from "../common/constants/permission.constant";
import {
PermissionSlug,
SYSTEM_ROLE_DEFAULT_PERMISSIONS,
} from "../common/constants/permission.constant";
import { SystemRoleSlug } from "../common/constants/system-role.constant";
import { AppError } from "../common/errors/app-error";
import { ERROR_CODE } from "../common/errors/error-code";
import { PermissionService } from "../modules/permissions/permission.service";
......@@ -12,12 +16,19 @@ async function resolveUserPermissions(req: Request): Promise<string[]> {
}
const permissions = await permissionService.getUserPermissions(req.user.id);
req.user.permissions = permissions;
if (!req.user.roles) {
req.user.roles = await permissionService.getUserRoles(req.user.id);
}
if (permissions.length === 0 && req.user.role) {
const defaultPerms =
SYSTEM_ROLE_DEFAULT_PERMISSIONS[req.user.role as SystemRoleSlug] || [];
req.user.permissions = defaultPerms;
return defaultPerms;
}
req.user.permissions = permissions;
return permissions;
}
......
import rateLimit, { RateLimitRequestHandler } from "express-rate-limit";
import { envConfig } from "../config/env.config";
import { ERROR_CODE } from "../common/errors/error-code";
/**
* Global API rate limit per IP, configurable for each environment.
......@@ -14,7 +15,7 @@ export const rateLimitMiddleware: RateLimitRequestHandler = rateLimit({
message: {
success: false,
message: "Bạn đã gửi quá nhiều yêu cầu. Vui lòng thử lại sau.",
code: "RATE_LIMIT_EXCEEDED",
code: ERROR_CODE.RATE_LIMIT_EXCEEDED,
},
});
......@@ -26,6 +27,6 @@ export const authRateLimiter: RateLimitRequestHandler = rateLimit({
message: {
success: false,
message: "Quá nhiều yêu cầu xác thực. Vui lòng thử lại sau 1 phút.",
code: "RATE_LIMIT_EXCEEDED",
code: ERROR_CODE.RATE_LIMIT_EXCEEDED,
},
});
......@@ -7,6 +7,7 @@ export const ALLOWED_AVATAR_MIME_TYPES = [
"image/jpeg",
"image/png",
"image/webp",
"image/gif",
] as const;
export const MAX_AVATAR_SIZE_BYTES = 5 * 1024 * 1024; // 5MB
......
import { Router } from "express";
import { ApiKeyController } from "./api-key.controller";
import { authMiddleware } from "../../middlewares/auth.middleware";
import { requirePermission } from "../../middlewares/permission.middleware";
import { PERMISSIONS } from "../../common/constants/permission.constant";
import { validate } from "../../middlewares/validate.middleware";
import {
createApiKeySchema,
......@@ -13,16 +15,28 @@ const controller = new ApiKeyController();
router.post(
"/",
authMiddleware,
requirePermission(PERMISSIONS.API_KEYS_CREATE),
validate(createApiKeySchema),
controller.create,
);
router.get("/", authMiddleware, controller.list);
router.get(
"/",
authMiddleware,
requirePermission(PERMISSIONS.API_KEYS_READ),
controller.list,
);
router.patch(
"/:id",
authMiddleware,
requirePermission(PERMISSIONS.API_KEYS_UPDATE),
validate(updateApiKeyStatusSchema),
controller.setActive,
);
router.delete("/:id", authMiddleware, controller.revoke);
router.delete(
"/:id",
authMiddleware,
requirePermission(PERMISSIONS.API_KEYS_DELETE),
controller.revoke,
);
export default router;
......@@ -3,8 +3,8 @@ import { AuditLogController } from "./audit-log.controller";
import { authMiddleware } from "../../middlewares/auth.middleware";
import { validateQuery } from "../../middlewares/validate.middleware";
import { listAuditLogsQuerySchema } from "./audit-log.validation";
import { requireRole } from "../../middlewares/role.middleware";
import { ROLES } from "../../common/constants/role.constant";
import { requirePermission } from "../../middlewares/permission.middleware";
import { PERMISSIONS } from "../../common/constants/permission.constant";
const router = Router();
const controller = new AuditLogController();
......@@ -12,7 +12,7 @@ const controller = new AuditLogController();
router.get(
"/",
authMiddleware,
requireRole(ROLES.ADMIN),
requirePermission(PERMISSIONS.AUDIT_LOGS_READ),
validateQuery(listAuditLogsQuerySchema),
controller.findAll,
);
......
import { prisma } from "../../database/prisma.client";
import { ROLES } from "../../common/constants/role.constant";
import { SYSTEM_ROLE_SLUGS } from "../../common/constants/system-role.constant";
export class AuthRepository {
findByEmail(email: string) {
......@@ -14,13 +15,14 @@ export class AuthRepository {
});
}
createUser(data: {
async createUser(data: {
email: string;
passwordHash: string;
fullName?: string;
isActive?: boolean;
}) {
return prisma.user.create({
return prisma.$transaction(async (tx) => {
const user = await tx.user.create({
data: {
email: data.email,
passwordHash: data.passwordHash,
......@@ -29,6 +31,22 @@ export class AuthRepository {
isActive: data.isActive ?? true,
},
});
const defaultRole = await tx.role.findUnique({
where: { slug: SYSTEM_ROLE_SLUGS.CRAWLER_USER },
});
if (defaultRole) {
await tx.userRoleAssignment.create({
data: {
userId: user.id,
roleId: defaultRole.id,
},
});
}
return user;
});
}
updateUser(
......
......@@ -5,7 +5,10 @@ import {
copyRefreshTokenToBody,
} from "../../middlewares/auth.middleware";
import { authRateLimiter } from "../../middlewares/rate-limit.middleware";
import { validate } from "../../middlewares/validate.middleware";
import {
validate,
validateParams,
} from "../../middlewares/validate.middleware";
import { uploadAvatarMiddleware } from "../../middlewares/upload.middleware";
import {
loginSchema,
......@@ -20,6 +23,7 @@ import {
changePasswordSchema,
requestDeactivationSchema,
confirmDeactivationSchema,
avatarFileNameParamsSchema,
} from "./auth.validation";
const router = Router();
......@@ -75,9 +79,13 @@ router.post(
controller.uploadAvatar(req, res, next);
},
);
router.get("/avatar/:fileName", (req, res, next) => {
router.get(
"/avatar/:fileName",
validateParams(avatarFileNameParamsSchema),
(req, res, next) => {
controller.getAvatar(req, res, next);
});
},
);
router.post(
"/change-password",
authMiddleware,
......
......@@ -104,3 +104,14 @@ export const requestDeactivationSchema = z.object({
export const confirmDeactivationSchema = z.object({
token: z.string().min(1, "Thiếu mã xác nhận vô hiệu hóa."),
});
export const avatarFileNameParamsSchema = z.object({
fileName: z
.string()
.trim()
.regex(
/^[a-zA-Z0-9_.-]+\.(jpg|jpeg|png|webp|gif)$/i,
"Invalid avatar filename format",
)
.refine((name) => !name.includes(".."), "Path traversal is not allowed"),
});
......@@ -14,6 +14,7 @@ import { buildJobRootFilePath } from "../../common/helpers/file.helper";
import { normalizeUrl } from "../../common/helpers/data-contract.helper";
import { AppError } from "../../common/errors/app-error";
import { ERROR_CODE } from "../../common/errors/error-code";
import { CrawlPageStatus } from "../../common/constants/crawl-page-status.constant";
/**
* Minimal page shape required for diff comparison.
......@@ -26,7 +27,7 @@ type DiffPage = {
normalizedUrl: string;
contentHash: string | null;
wordCount: number;
status: import("@prisma/client").CrawlPageStatus;
status: CrawlPageStatus;
statusCode: number | null;
title: string | null;
crawledAt: Date | null;
......
......@@ -41,11 +41,14 @@ export class CrawlExportController {
const result = await this.service.findAllByUser(req.user.id, page, limit);
res.json({
success: true,
data: result.items,
pagination: {
data: {
items: result.items,
meta: {
total: result.total,
page: result.page,
limit: result.limit,
totalPages: Math.ceil(result.total / (result.limit || 1)),
},
},
});
} catch (error) {
......
import { Router } from "express";
import { CrawlExportController } from "./crawl-export.controller";
import { authMiddleware } from "../../middlewares/auth.middleware";
import { requireRole } from "../../middlewares/role.middleware";
import { ROLES } from "../../common/constants/role.constant";
import { requirePermission } from "../../middlewares/permission.middleware";
import { PERMISSIONS } from "../../common/constants/permission.constant";
import {
validateQuery,
validateParams,
} from "../../middlewares/validate.middleware";
import {
crawlExportQuerySchema,
crawlExportParamsSchema,
} from "./crawl-export.validation";
const router = Router();
const controller = new CrawlExportController();
......@@ -10,19 +18,22 @@ const controller = new CrawlExportController();
router.get(
"/",
authMiddleware,
requireRole(ROLES.ADMIN, ROLES.CRAWLER_USER, ROLES.VIEWER),
requirePermission(PERMISSIONS.EXPORTS_READ),
validateQuery(crawlExportQuerySchema),
controller.findAll,
);
router.get(
"/:exportId/download",
authMiddleware,
requireRole(ROLES.ADMIN, ROLES.CRAWLER_USER, ROLES.VIEWER),
requirePermission(PERMISSIONS.EXPORTS_DOWNLOAD),
validateParams(crawlExportParamsSchema),
controller.download,
);
router.delete(
"/:exportId",
authMiddleware,
requireRole(ROLES.ADMIN, ROLES.CRAWLER_USER),
requirePermission(PERMISSIONS.EXPORTS_DELETE),
validateParams(crawlExportParamsSchema),
controller.delete,
);
......
import { z } from "zod";
export const crawlExportQuerySchema = z.object({
page: z.coerce.number().int().min(1).default(1),
limit: z.coerce.number().int().min(1).max(100).default(20),
});
export const crawlExportParamsSchema = z.object({
exportId: z.string().uuid("Invalid export ID format"),
});
......@@ -7,10 +7,12 @@ import {
createExportSchema,
listCrawlJobsQuerySchema,
getAssetsQuerySchema,
jobLogsQuerySchema,
diffQuerySchema,
} from "./crawl-job.validation";
import { crawlPageQuerySchema } from "../crawl-pages/crawl-page.validation";
import { requireRole } from "../../middlewares/role.middleware";
import { ROLES } from "../../common/constants/role.constant";
import { requirePermission } from "../../middlewares/permission.middleware";
import { PERMISSIONS } from "../../common/constants/permission.constant";
const router = Router();
const controller = new CrawlJobController();
......@@ -18,7 +20,7 @@ const controller = new CrawlJobController();
router.post(
"/",
apiKeyOrAuthMiddleware,
requireRole(ROLES.ADMIN, ROLES.CRAWLER_USER),
requirePermission(PERMISSIONS.CRAWL_JOBS_CREATE),
validate(createCrawlJobSchema),
(req, res, next) => {
controller.create(req, res, next);
......@@ -27,70 +29,71 @@ router.post(
router.get(
"/",
apiKeyOrAuthMiddleware,
requireRole(ROLES.ADMIN, ROLES.CRAWLER_USER, ROLES.VIEWER),
requirePermission(PERMISSIONS.CRAWL_JOBS_READ),
validateQuery(listCrawlJobsQuerySchema),
controller.findAll,
);
router.get(
"/:id",
apiKeyOrAuthMiddleware,
requireRole(ROLES.ADMIN, ROLES.CRAWLER_USER, ROLES.VIEWER),
requirePermission(PERMISSIONS.CRAWL_JOBS_READ),
controller.findById,
);
router.delete(
"/:id",
apiKeyOrAuthMiddleware,
requireRole(ROLES.ADMIN, ROLES.CRAWLER_USER),
requirePermission(PERMISSIONS.CRAWL_JOBS_DELETE),
controller.delete,
);
router.post(
"/:id/rerun",
apiKeyOrAuthMiddleware,
requireRole(ROLES.ADMIN, ROLES.CRAWLER_USER),
requirePermission(PERMISSIONS.CRAWL_JOBS_RETRY),
controller.rerun,
);
router.get(
"/:id/logs",
apiKeyOrAuthMiddleware,
requireRole(ROLES.ADMIN, ROLES.CRAWLER_USER, ROLES.VIEWER),
requirePermission(PERMISSIONS.CRAWL_JOBS_READ),
validateQuery(jobLogsQuerySchema),
controller.getLogs,
);
router.get(
"/:id/events",
apiKeyOrAuthMiddleware,
requireRole(ROLES.ADMIN, ROLES.CRAWLER_USER, ROLES.VIEWER),
requirePermission(PERMISSIONS.CRAWL_JOBS_READ),
controller.streamEvents,
);
router.post(
"/:id/cancel",
apiKeyOrAuthMiddleware,
requireRole(ROLES.ADMIN, ROLES.CRAWLER_USER),
requirePermission(PERMISSIONS.CRAWL_JOBS_CANCEL),
controller.cancel,
);
router.get(
"/:id/pages",
apiKeyOrAuthMiddleware,
requireRole(ROLES.ADMIN, ROLES.CRAWLER_USER, ROLES.VIEWER),
requirePermission(PERMISSIONS.CRAWL_JOBS_READ),
validateQuery(crawlPageQuerySchema),
controller.getPages,
);
router.get(
"/:id/pages/preview",
apiKeyOrAuthMiddleware,
requireRole(ROLES.ADMIN, ROLES.CRAWLER_USER, ROLES.VIEWER),
requirePermission(PERMISSIONS.CRAWL_JOBS_READ),
validateQuery(crawlPageQuerySchema),
controller.getPagesPreview,
);
router.get(
"/:id/exports",
apiKeyOrAuthMiddleware,
requireRole(ROLES.ADMIN, ROLES.CRAWLER_USER, ROLES.VIEWER),
requirePermission(PERMISSIONS.EXPORTS_READ),
controller.getExports,
);
router.post(
"/:id/exports",
apiKeyOrAuthMiddleware,
requireRole(ROLES.ADMIN, ROLES.CRAWLER_USER),
requirePermission(PERMISSIONS.EXPORTS_CREATE),
validate(createExportSchema),
(req, res, next) => {
controller.createExport(req, res, next);
......@@ -99,26 +102,28 @@ router.post(
router.get(
"/:id/download",
apiKeyOrAuthMiddleware,
requireRole(ROLES.ADMIN, ROLES.CRAWLER_USER, ROLES.VIEWER),
requirePermission(PERMISSIONS.EXPORTS_DOWNLOAD),
controller.download,
);
router.get(
"/:id/assets",
apiKeyOrAuthMiddleware,
requireRole(ROLES.ADMIN, ROLES.CRAWLER_USER, ROLES.VIEWER),
requirePermission(PERMISSIONS.CRAWL_JOBS_READ),
validateQuery(getAssetsQuerySchema),
controller.getAssets,
);
router.get(
"/:id/diff",
apiKeyOrAuthMiddleware,
requireRole(ROLES.ADMIN, ROLES.CRAWLER_USER, ROLES.VIEWER),
requirePermission(PERMISSIONS.CRAWL_JOBS_READ),
validateQuery(diffQuerySchema),
controller.getDiff,
);
router.get(
"/:id/diff/download",
apiKeyOrAuthMiddleware,
requireRole(ROLES.ADMIN, ROLES.CRAWLER_USER, ROLES.VIEWER),
requirePermission(PERMISSIONS.CRAWL_JOBS_READ),
validateQuery(diffQuerySchema),
controller.downloadDiff,
);
export default router;
......@@ -12,7 +12,10 @@ import {
import { crawlQueue } from "../../queues/crawl.queue";
import { ROLES } from "../../common/constants/role.constant";
import { JOB_STATUS } from "../../common/constants/job-status.constant";
import { EXPORT_TYPE } from "../../common/constants/export-type.constant";
import {
EXPORT_TYPE,
EXPORT_STATUS,
} from "../../common/constants/export-type.constant";
import { DEFAULT_TIMEZONE } from "../../common/constants/timezone.constant";
import { CRAWL_MODE } from "../../common/constants/crawl-mode.constant";
import { CreateCrawlJobDto, CrawlJobQueryDto } from "./crawl-job.dto";
......@@ -246,7 +249,7 @@ export class CrawlJobService {
for (const exportRecord of exports) {
if (
exportRecord.exportType === EXPORT_TYPE.ZIP &&
exportRecord.status === JOB_STATUS.COMPLETED &&
exportRecord.status === EXPORT_STATUS.COMPLETED &&
(await storage.exists(exportRecord.filePath))
) {
return exportRecord;
......
......@@ -74,3 +74,15 @@ export const getAssetsQuerySchema = z.object({
page: z.coerce.number().int().min(1).default(1),
limit: z.coerce.number().int().min(1).max(500).default(50),
});
export const jobLogsQuerySchema = z.object({
page: z.coerce.number().int().min(1).default(1),
limit: z.coerce.number().int().min(1).max(100).default(50),
});
export const diffQuerySchema = z.object({
compareWithJobId: z
.string()
.uuid("Invalid compareWithJobId format")
.optional(),
});
......@@ -6,9 +6,10 @@ import {
createCrawlScheduleSchema,
updateCrawlScheduleSchema,
crawlScheduleQuerySchema,
crawlScheduleHistoryQuerySchema,
} from "./crawl-schedule.validation";
import { requireRole } from "../../middlewares/role.middleware";
import { ROLES } from "../../common/constants/role.constant";
import { requirePermission } from "../../middlewares/permission.middleware";
import { PERMISSIONS } from "../../common/constants/permission.constant";
const router = Router();
const controller = new CrawlScheduleController();
......@@ -16,7 +17,7 @@ const controller = new CrawlScheduleController();
router.post(
"/",
apiKeyOrAuthMiddleware,
requireRole(ROLES.ADMIN, ROLES.CRAWLER_USER),
requirePermission(PERMISSIONS.CRAWL_SCHEDULES_CREATE),
validate(createCrawlScheduleSchema),
controller.create,
);
......@@ -24,7 +25,7 @@ router.post(
router.get(
"/",
apiKeyOrAuthMiddleware,
requireRole(ROLES.ADMIN, ROLES.CRAWLER_USER, ROLES.VIEWER),
requirePermission(PERMISSIONS.CRAWL_SCHEDULES_READ),
validateQuery(crawlScheduleQuerySchema),
controller.findAll,
);
......@@ -32,14 +33,14 @@ router.get(
router.get(
"/:id",
apiKeyOrAuthMiddleware,
requireRole(ROLES.ADMIN, ROLES.CRAWLER_USER, ROLES.VIEWER),
requirePermission(PERMISSIONS.CRAWL_SCHEDULES_READ),
controller.findById,
);
router.patch(
"/:id",
apiKeyOrAuthMiddleware,
requireRole(ROLES.ADMIN, ROLES.CRAWLER_USER),
requirePermission(PERMISSIONS.CRAWL_SCHEDULES_UPDATE),
validate(updateCrawlScheduleSchema),
controller.update,
);
......@@ -47,21 +48,22 @@ router.patch(
router.delete(
"/:id",
apiKeyOrAuthMiddleware,
requireRole(ROLES.ADMIN, ROLES.CRAWLER_USER),
requirePermission(PERMISSIONS.CRAWL_SCHEDULES_DELETE),
controller.delete,
);
router.post(
"/:id/run",
apiKeyOrAuthMiddleware,
requireRole(ROLES.ADMIN, ROLES.CRAWLER_USER),
requirePermission(PERMISSIONS.CRAWL_SCHEDULES_RUN),
controller.triggerRun,
);
router.get(
"/:id/history",
apiKeyOrAuthMiddleware,
requireRole(ROLES.ADMIN, ROLES.CRAWLER_USER, ROLES.VIEWER),
requirePermission(PERMISSIONS.CRAWL_SCHEDULES_READ),
validateQuery(crawlScheduleHistoryQuerySchema),
controller.getHistory,
);
......
......@@ -106,3 +106,8 @@ export const crawlScheduleQuerySchema = z.object({
.default("createdAt"),
order: z.enum(["asc", "desc"]).optional().default("desc"),
});
export const crawlScheduleHistoryQuerySchema = z.object({
page: z.coerce.number().int().min(1).default(1),
limit: z.coerce.number().int().min(1).max(100).default(20),
});
import { prisma } from "../../database/prisma.client";
import { ROLES } from "../../common/constants/role.constant";
import { JOB_STATUS } from "../../common/constants/job-status.constant";
import { CrawlPageStatus } from "@prisma/client";
import { CRAWL_PAGE_STATUS } from "../../common/constants/crawl-page-status.constant";
export class DashboardRepository {
async getStats(userId: string, role: string) {
......@@ -12,41 +12,24 @@ export class DashboardRepository {
const exportWhere = isGlobal ? {} : { job: { userId } };
const [
totalJobs,
completedJobs,
failedJobs,
runningJobs,
pendingJobs,
jobStatusGroups,
pageStatusGroups,
totalPagesCrawled,
successfulPages,
failedPages,
activeSchedules,
totalSchedules,
totalExports,
] = await Promise.all([
prisma.crawlJob.count({ where: jobWhere }),
prisma.crawlJob.count({
where: { ...jobWhere, status: JOB_STATUS.COMPLETED },
prisma.crawlJob.groupBy({
by: ["status"],
_count: { status: true },
where: jobWhere,
}),
prisma.crawlJob.count({
where: { ...jobWhere, status: JOB_STATUS.FAILED },
}),
prisma.crawlJob.count({
where: { ...jobWhere, status: JOB_STATUS.RUNNING },
}),
prisma.crawlJob.count({
where: {
...jobWhere,
status: { in: [JOB_STATUS.PENDING, JOB_STATUS.QUEUED] },
},
prisma.crawlPage.groupBy({
by: ["status"],
_count: { status: true },
where: pageWhere,
}),
prisma.crawlPage.count({ where: pageWhere }),
prisma.crawlPage.count({
where: { ...pageWhere, status: CrawlPageStatus.SUCCESS },
}),
prisma.crawlPage.count({
where: { ...pageWhere, status: CrawlPageStatus.FAILED },
}),
prisma.crawlSchedule.count({
where: { ...scheduleWhere, isActive: true },
}),
......@@ -54,18 +37,32 @@ export class DashboardRepository {
prisma.crawlExport.count({ where: exportWhere }),
]);
const jobCounts: Record<string, number> = {};
let totalJobs = 0;
for (const group of jobStatusGroups) {
jobCounts[group.status] = group._count.status;
totalJobs += group._count.status;
}
const pageCounts: Record<string, number> = {};
for (const group of pageStatusGroups) {
pageCounts[group.status] = group._count.status;
}
return {
jobs: {
total: totalJobs,
completed: completedJobs,
failed: failedJobs,
running: runningJobs,
pending: pendingJobs,
completed: jobCounts[JOB_STATUS.COMPLETED] ?? 0,
failed: jobCounts[JOB_STATUS.FAILED] ?? 0,
running: jobCounts[JOB_STATUS.RUNNING] ?? 0,
pending:
(jobCounts[JOB_STATUS.PENDING] ?? 0) +
(jobCounts[JOB_STATUS.QUEUED] ?? 0),
},
pages: {
total: totalPagesCrawled,
successful: successfulPages,
failed: failedPages,
successful: pageCounts[CRAWL_PAGE_STATUS.SUCCESS] ?? 0,
failed: pageCounts[CRAWL_PAGE_STATUS.FAILED] ?? 0,
},
schedules: {
total: totalSchedules,
......
import { Router } from "express";
import { DashboardController } from "./dashboard.controller";
import { authMiddleware } from "../../middlewares/auth.middleware";
import { requirePermission } from "../../middlewares/permission.middleware";
import { PERMISSIONS } from "../../common/constants/permission.constant";
const router = Router();
const controller = new DashboardController();
router.get("/stats", authMiddleware, controller.getStats);
router.get(
"/stats",
authMiddleware,
requirePermission(PERMISSIONS.DASHBOARD_READ),
controller.getStats,
);
export default router;
......@@ -60,17 +60,22 @@ export class JsonExportService extends BaseExportService {
fs.writeFileSync(filePath, JSON.stringify(wrapper, null, 2), "utf-8");
// 1. Xuất pages.raw.json (Lọc bỏ các trường dữ liệu sạch & chất lượng)
const rawPages = pages.map(
({
cleanText: _cleanText,
mainContent: _mainContent,
wordCount: _wordCount,
contentHash: _contentHash,
dataQualityScore: _dataQualityScore,
warnings: _warnings,
...rawFields
}) => rawFields,
);
const rawPages = pages.map((page) => ({
id: page.id,
jobId: page.jobId,
url: page.url,
normalizedUrl: page.normalizedUrl,
status: page.status,
statusCode: page.statusCode,
errorMessage: page.errorMessage,
title: page.title,
description: page.description,
rawMarkdown: page.rawMarkdown,
links: page.links,
images: page.images,
tables: page.tables,
crawledAt: page.crawledAt,
}));
const { filePath: rawFilePath } = buildJobDataRawFilePath(
job.id,
JOB_EXPORT_FILES.PAGES_RAW_JSON,
......@@ -79,9 +84,27 @@ export class JsonExportService extends BaseExportService {
fs.writeFileSync(rawFilePath, JSON.stringify(rawWrapper, null, 2), "utf-8");
// 2. Xuất pages.clean.json (Lọc bỏ trường rawMarkdown)
const cleanPages = pages.map(
({ rawMarkdown: _rawMarkdown, ...cleanFields }) => cleanFields,
);
const cleanPages = pages.map((page) => ({
id: page.id,
jobId: page.jobId,
url: page.url,
normalizedUrl: page.normalizedUrl,
status: page.status,
statusCode: page.statusCode,
errorMessage: page.errorMessage,
title: page.title,
description: page.description,
cleanText: page.cleanText,
mainContent: page.mainContent,
wordCount: page.wordCount,
contentHash: page.contentHash,
dataQualityScore: page.dataQualityScore,
warnings: page.warnings,
links: page.links,
images: page.images,
tables: page.tables,
crawledAt: page.crawledAt,
}));
const { filePath: cleanFilePath } = buildJobDataCleanFilePath(
job.id,
JOB_EXPORT_FILES.PAGES_CLEAN_JSON,
......
......@@ -11,6 +11,7 @@ import {
buildCrawlResultZipName,
} from "../../common/constants/storage-path.constant";
import { EXPORT_MIME_TYPES } from "../../common/constants/export-type.constant";
import { CRAWL_PAGE_STATUS } from "../../common/constants/crawl-page-status.constant";
import {
buildJobLogsFilePath,
buildJobRootFilePath,
......@@ -101,9 +102,13 @@ export class ZipExportService extends BaseExportService {
private writeSummary(job: CrawlJob & { pages: CrawlPage[] }): void {
const { filePath } = buildJobRootFilePath(job.id, JOB_EXPORT_FILES.SUMMARY);
const successPages = job.pages.filter((p) => p.status === "SUCCESS").length;
const successPages = job.pages.filter(
(p) => p.status === CRAWL_PAGE_STATUS.SUCCESS,
).length;
const failedPages = job.pages.filter(
(p) => p.status !== "SUCCESS" && p.status !== "SKIPPED",
(p) =>
p.status !== CRAWL_PAGE_STATUS.SUCCESS &&
p.status !== CRAWL_PAGE_STATUS.SKIPPED,
).length;
const summary = {
......@@ -125,9 +130,9 @@ export class ZipExportService extends BaseExportService {
const errors = job.pages
.filter(
(p) =>
p.status !== "SUCCESS" &&
p.status !== "PENDING" &&
p.status !== "SKIPPED",
p.status !== CRAWL_PAGE_STATUS.SUCCESS &&
p.status !== CRAWL_PAGE_STATUS.PENDING &&
p.status !== CRAWL_PAGE_STATUS.SKIPPED,
)
.map((p) => ({
url: p.url,
......@@ -145,12 +150,14 @@ export class ZipExportService extends BaseExportService {
JOB_EXPORT_FILES.DATA_QUALITY_JSON,
);
const successPages = job.pages.filter((p) => p.status === "SUCCESS");
const successPages = job.pages.filter(
(p) => p.status === CRAWL_PAGE_STATUS.SUCCESS,
);
const errorPages = job.pages.filter(
(p) =>
p.status !== "SUCCESS" &&
p.status !== "SKIPPED" &&
p.status !== "PENDING",
p.status !== CRAWL_PAGE_STATUS.SUCCESS &&
p.status !== CRAWL_PAGE_STATUS.SKIPPED &&
p.status !== CRAWL_PAGE_STATUS.PENDING,
);
const seenHashes = new Set<string>();
......
import { Router } from "express";
import { ExtractionTemplateController } from "./extraction-template.controller";
import { authMiddleware } from "../../middlewares/auth.middleware";
import { requirePermission } from "../../middlewares/permission.middleware";
import { PERMISSIONS } from "../../common/constants/permission.constant";
import { validate } from "../../middlewares/validate.middleware";
import {
createExtractionTemplateSchema,
......@@ -12,14 +14,32 @@ const controller = new ExtractionTemplateController();
router.use(authMiddleware);
router.post("/", validate(createExtractionTemplateSchema), controller.create);
router.get("/", controller.findAll);
router.get("/:id", controller.findById);
router.post(
"/",
requirePermission(PERMISSIONS.EXTRACTION_TEMPLATES_CREATE),
validate(createExtractionTemplateSchema),
controller.create,
);
router.get(
"/",
requirePermission(PERMISSIONS.EXTRACTION_TEMPLATES_READ),
controller.findAll,
);
router.get(
"/:id",
requirePermission(PERMISSIONS.EXTRACTION_TEMPLATES_READ),
controller.findById,
);
router.patch(
"/:id",
requirePermission(PERMISSIONS.EXTRACTION_TEMPLATES_UPDATE),
validate(updateExtractionTemplateSchema),
controller.update,
);
router.delete("/:id", controller.delete);
router.delete(
"/:id",
requirePermission(PERMISSIONS.EXTRACTION_TEMPLATES_DELETE),
controller.delete,
);
export default router;
import { HealthService } from "../health.service";
import { prisma } from "../../../database/prisma.client";
import { HealthRepository } from "../health.repository";
jest.mock("../../../database/prisma.client", () => ({
prisma: {
$queryRaw: jest.fn(),
},
}));
jest.mock("../health.repository");
jest.mock("../../../queues/crawl.queue", () => ({
crawlQueue: {
......@@ -28,10 +24,12 @@ jest.mock("../../../queues/webhook.queue", () => ({
describe("HealthService", () => {
let service: HealthService;
let mockHealthRepo: jest.Mocked<HealthRepository>;
beforeEach(() => {
jest.clearAllMocks();
service = new HealthService();
mockHealthRepo = new HealthRepository() as jest.Mocked<HealthRepository>;
service = new HealthService(mockHealthRepo);
});
describe("getLiveness", () => {
......@@ -45,7 +43,7 @@ describe("HealthService", () => {
describe("getReadiness", () => {
it("returns ready status when database is up", async () => {
(prisma.$queryRaw as jest.Mock).mockResolvedValue([{ 1: 1 }]);
mockHealthRepo.pingDatabase.mockResolvedValue();
const result = await service.getReadiness();
expect(result.status).toBe("ready");
......@@ -54,7 +52,7 @@ describe("HealthService", () => {
});
it("returns unhealthy status when database query fails", async () => {
(prisma.$queryRaw as jest.Mock).mockRejectedValue(
mockHealthRepo.pingDatabase.mockRejectedValue(
new Error("Connection timeout"),
);
......
import { prisma } from "../../database/prisma.client";
export class HealthRepository {
async pingDatabase(): Promise<void> {
await prisma.$queryRaw`SELECT 1`;
}
}
import { prisma } from "../../database/prisma.client";
import { HealthRepository } from "./health.repository";
import { crawlQueue } from "../../queues/crawl.queue";
import { webhookQueue } from "../../queues/webhook.queue";
import { getErrorMessage } from "../../common/helpers/error-mapping.helper";
......@@ -13,6 +13,8 @@ export interface QueueCountMetrics {
export type QueueMetricsResult = QueueCountMetrics | "unavailable" | null;
export class HealthService {
constructor(private readonly repository = new HealthRepository()) {}
getLiveness() {
return {
status: "ok",
......@@ -32,7 +34,7 @@ export class HealthService {
// 1. Check Database
const dbStart = Date.now();
try {
await prisma.$queryRaw`SELECT 1`;
await this.repository.pingDatabase();
checks.database = {
status: "up",
latencyMs: Date.now() - dbStart,
......
......@@ -76,6 +76,22 @@ export class RoleRepository {
});
}
async findByIds(ids: string[]) {
return prisma.role.findMany({
where: { id: { in: ids } },
include: {
rolePermissions: {
include: {
permission: true,
},
},
_count: {
select: { userRoles: true },
},
},
});
}
async findBySlug(slug: string) {
return prisma.role.findUnique({
where: { slug },
......
......@@ -154,6 +154,21 @@ export class UserRepository {
where: { userId: id },
});
await tx.crawlSchedule.updateMany({
where: { userId: id },
data: { isActive: false },
});
await tx.apiKey.updateMany({
where: { userId: id },
data: { isActive: false },
});
await tx.webhookConfig.updateMany({
where: { userId: id },
data: { isActive: false },
});
return user;
});
}
......
......@@ -261,12 +261,9 @@ export class UserService {
await this.findById(targetUserId);
// 2. Fetch target roles to validate
const targetRoles = await Promise.all(
roleIds.map((id) => this.roleRepository.findById(id)),
);
const targetRoles = await this.roleRepository.findByIds(roleIds);
const missingRole = targetRoles.find((r) => !r);
if (missingRole || targetRoles.length !== roleIds.length) {
if (targetRoles.length !== roleIds.length) {
throw new AppError(
"One or more roles not found",
404,
......
......@@ -7,6 +7,7 @@ import { getErrorMessage } from "../../common/helpers/error-mapping.helper";
import { AppError } from "../../common/errors/app-error";
import { ERROR_CODE } from "../../common/errors/error-code";
import { WEBHOOK_DELIVERY_STATUS } from "../../common/constants/webhook.constant";
export class WebhookDeliveryService {
private readonly repository = new WebhookRepository();
......@@ -40,7 +41,7 @@ export class WebhookDeliveryService {
crawlJobId,
event,
payload,
status: "PENDING",
status: WEBHOOK_DELIVERY_STATUS.PENDING,
attempt: 1,
});
......@@ -100,7 +101,7 @@ export class WebhookDeliveryService {
: JSON.stringify(response.data);
await this.repository.updateDelivery(deliveryId, {
status: "SUCCESS",
status: WEBHOOK_DELIVERY_STATUS.SUCCESS,
statusCode: response.status,
responseBody: responseBody.substring(0, 2000), // Limit size stored in DB
deliveredAt: new Date(),
......@@ -141,7 +142,7 @@ export class WebhookDeliveryService {
*/
async markFailed(deliveryId: string, errorReason: string): Promise<void> {
await this.repository.updateDelivery(deliveryId, {
status: "FAILED",
status: WEBHOOK_DELIVERY_STATUS.FAILED,
errorMessage:
`Max attempts exhausted. Last error: ${errorReason}`.substring(0, 1000),
});
......@@ -162,7 +163,7 @@ export class WebhookDeliveryService {
}
const updated = await this.repository.updateDelivery(deliveryId, {
status: "PENDING",
status: WEBHOOK_DELIVERY_STATUS.PENDING,
attempt: 1,
errorMessage: null,
});
......
import { Router } from "express";
import { WebhookController } from "./webhook.controller";
import { authMiddleware } from "../../middlewares/auth.middleware";
import { requirePermission } from "../../middlewares/permission.middleware";
import { PERMISSIONS } from "../../common/constants/permission.constant";
import { validate, validateQuery } from "../../middlewares/validate.middleware";
import {
createWebhookConfigSchema,
......@@ -14,24 +16,47 @@ const controller = new WebhookController();
router.post(
"/configs",
authMiddleware,
requirePermission(PERMISSIONS.WEBHOOKS_CREATE),
validate(createWebhookConfigSchema),
controller.createConfig,
);
router.get("/configs", authMiddleware, controller.listConfigs);
router.get(
"/configs",
authMiddleware,
requirePermission(PERMISSIONS.WEBHOOKS_READ),
controller.listConfigs,
);
router.patch(
"/configs/:id",
authMiddleware,
requirePermission(PERMISSIONS.WEBHOOKS_UPDATE),
validate(updateWebhookConfigSchema),
controller.updateConfig,
);
router.delete("/configs/:id", authMiddleware, controller.deleteConfig);
router.post("/configs/:id/test", authMiddleware, controller.testConfig);
router.delete(
"/configs/:id",
authMiddleware,
requirePermission(PERMISSIONS.WEBHOOKS_DELETE),
controller.deleteConfig,
);
router.post(
"/configs/:id/test",
authMiddleware,
requirePermission(PERMISSIONS.WEBHOOKS_TEST),
controller.testConfig,
);
router.get(
"/deliveries",
authMiddleware,
requirePermission(PERMISSIONS.WEBHOOKS_READ),
validateQuery(listWebhookDeliveriesQuerySchema),
controller.listDeliveries,
);
router.post("/deliveries/:id/redeliver", authMiddleware, controller.redeliver);
router.post(
"/deliveries/:id/redeliver",
authMiddleware,
requirePermission(PERMISSIONS.WEBHOOKS_UPDATE),
controller.redeliver,
);
export default router;
......@@ -19,6 +19,9 @@ import {
import { runExtractionIfTemplate } from "../modules/extraction-templates/extraction-runner";
import { JOB_STATUS } from "../common/constants/job-status.constant";
import { CRAWL_MODE } from "../common/constants/crawl-mode.constant";
import { ASSET_TYPE } from "../common/constants/asset-type.constant";
import { CRAWL_PAGE_STATUS } from "../common/constants/crawl-page-status.constant";
import { WEBHOOK_EVENT } from "../common/constants/webhook.constant";
// Lazy getters — instantiated on first use so Jest mocks replace constructors before creation
const getJobRepository = () => new CrawlJobRepository();
const getPageRepository = () => new CrawlPageRepository();
......@@ -58,7 +61,7 @@ export async function savePageAssets(
assetsBatch.push({
jobId,
pageId,
assetType: "IMAGE" as const,
assetType: ASSET_TYPE.IMAGE,
url: img.url,
sourceUrl: item.url,
altText: img.alt || undefined,
......@@ -73,7 +76,7 @@ export async function savePageAssets(
assetsBatch.push({
jobId,
pageId,
assetType: "LINK" as const,
assetType: ASSET_TYPE.LINK,
url: link.url,
sourceUrl: item.url,
altText: link.text || undefined,
......@@ -87,7 +90,7 @@ export async function savePageAssets(
assetsBatch.push({
jobId,
pageId,
assetType: "PDF" as const,
assetType: ASSET_TYPE.PDF,
url: pdfUrl,
sourceUrl: item.url,
orderIndex: index + 1,
......@@ -196,7 +199,7 @@ export async function persistBatchResults(
const normalized = getPageProcessor().normalizeFailedPage(
{ url: blockedUrl, error: "Blocked by robots.txt" },
jobId,
"BLOCKED",
CRAWL_PAGE_STATUS.BLOCKED,
);
await getPageRepository().upsert(normalized);
failedCount++;
......@@ -581,8 +584,8 @@ export async function processCrawlJob(job: Job): Promise<void> {
const event =
updatedJob.status === JOB_STATUS.COMPLETED
? "job.completed"
: "job.failed";
? WEBHOOK_EVENT.JOB_COMPLETED
: WEBHOOK_EVENT.JOB_FAILED;
void logStep(
jobId,
updatedJob.status === JOB_STATUS.COMPLETED ? "INFO" : "ERROR",
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment