Summary of Changes

This PR optimizes and standardizes the export pipeline across all supported formats (Markdown, CSV, XLSX, JSON, and ZIP) according to Data Contract v1. It eliminates stranded server-side export files by bundling individual exports into comprehensive .zip archives, unifies data schemas with cross-referencing keys (Page ID), and aligns architectural documentation.

Key Improvements

1. Individual Export ZIP Bundling

Previously, individual export endpoints only returned a single file while leaving secondary data files stranded on the server:

• exportType: "MARKDOWN": Bundles all files into markdown.zip containing:
  • clean/*.md: Sanitized Markdown without nav/footer boilerplate (ideal for AI prompt contexts).
  • raw/*.md: Original Markdown for debugging and audit.
  • Removed duplicate root-level .md files that cluttered the export directory.
• exportType: "CSV": Bundles into csv.zip containing pages.csv, links.csv, and images.csv.
• exportType: "XLSX": Bundles into xlsx.zip containing:
  • pages.xlsx: Standardized page inventory.
  • tables.xlsx: Dedicated workbook for all extracted HTML tables.
• exportType: "JSON": Bundles into json.zip containing:
  • pages.json: Master envelope with full crawl page records.
  • clean/pages.clean.json: Token-optimized dataset for AI/LLM/RAG pipelines.
  • raw/pages.raw.json: Raw Markdown audit dataset.
  • structured.json: Schema.org / JSON-LD structured data (omitted if empty).

2. Schema Standardization & Quality Metrics

• pages.csv:
  • Added pageId as primary identifier to link with links.csv and images.csv.
  • Added quality metrics: wordCount, dataQualityScore, errorMessage.
  • Replaced multiple long text columns with clean mainContent.
• pages.xlsx:
  • Added Page ID, Word Count, Quality Score (0-100).
  • Replaced truncated 500-char content columns with a single clean Content Preview.
  • Removed redundant table sheet embedding to keep the workbook lightweight.
• tables.xlsx:
  • Added Page ID column to the Summary sheet for cross-sheet and cross-file JOIN operations with pages.xlsx.
• structured.json:
  • Avoids generating empty { jobId, records: [] } files when no structured data is detected.

3. Overall ZIP Hierarchy & Contract Alignment (Option 2B)

• Preserved executive summary and metadata files at the ZIP root (metadata.json, summary.json, data_quality.json, diff_report.json) for quick inspection upon unzipping.
• Organized crawled data under data/, logs under logs/, and markdown files under markdown/.
• Updated DATA_CONTRACT_V1.md and README.md to accurately document all export bundles and file structures.