Skip to content

  • Projects
  • Groups
  • Snippets
  • Help
    • Loading...
    • Help
    • Submit feedback
    • Contribute to GitLab
  • Sign in
U
upgrade-data-crawler-be
  • Project
    • Project
    • Details
    • Activity
    • Releases
    • Cycle Analytics
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
    • Charts
  • Issues 0
    • Issues 0
    • List
    • Board
    • Labels
    • Milestones
  • Merge Requests 0
    • Merge Requests 0
  • CI / CD
    • CI / CD
    • Pipelines
    • Jobs
    • Schedules
    • Charts
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Members
    • Members
  • Collapse sidebar
  • Activity
  • Graph
  • Charts
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
  • ThinhNC
  • upgrade-data-crawler-be
  • Merge Requests
  • !23

Merged
Opened Sep 12, 2026 by ThinhNC@ThinhNC
  • Report abuse
Report abuse

fix(crawl-jobs): include complete page content and details in job query

Problem Description

In CrawlJobRepository.findById, the Prisma query previously specified a restrictive select projection for related pages (id, url, normalizedUrl, contentHash, wordCount, status, statusCode, title, crawledAt).

This projection omitted critical crawled data fields such as content, markdown, and htmlContent, preventing the API and frontend from accessing and viewing the actual crawled content of pages within a crawl job.


Proposed Changes

src/modules/crawl-jobs/crawl-job.repository.ts

  • Replaced the restricted projection pages: { select: { ... } } with pages: true.
  • Ensured all page entity fields (including parsed content, markdown, error logs, and metadata) are retrieved when fetching a job by ID.

Check out, review, and merge locally

Step 1. Fetch and check out the branch for this merge request

git fetch origin
git checkout -b fix/content-pages origin/fix/content-pages

Step 2. Review the changes locally

Step 3. Merge the branch and fix any conflicts that come up

git fetch origin
git checkout origin/develop
git merge --no-ff fix/content-pages

Step 4. Push the result of the merge to GitLab

git push origin develop

Note that pushing to GitLab requires write access to this repository.

Tip: You can also checkout merge requests locally by following these guidelines.

  • Discussion 0
  • Commits 1
  • Changes 1
Assignee
No assignee
Assign to
None
Milestone
None
Assign milestone
Time tracking
0
Labels
None
Assign labels
  • View project labels
Reference: ThinhNC/upgrade-data-crawler-be!23

Revert this merge request

This will create a new commit in order to revert the existing changes.

Switch branch
Cancel
A new branch will be created in your fork and a new merge request will be started.

Cherry-pick this merge request

Switch branch
Cancel
A new branch will be created in your fork and a new merge request will be started.