feat(be): exempt failed connection & provider quota jobs from daily usage quota (!15) · Merge Requests · ThinhNC / upgrade-data-crawler-be

feat(be): exempt failed connection & provider quota jobs from daily usage quota

Pull Request Overview: Client Resilience, Quota Waiver Policy & Bilingual Localization

This pull request introduces connection resilience, client-side data handling utilities, an automatic quota exemption policy for connection/provider failures, and complete English/Vietnamese bilingual localization across the platform.

Backend Changes (`data-crawler-be`)

1. Quota Exemption Policy for Failed Connection & Provider Outages

Quota Exemption Logic (crawl-job.repository.ts):
- Updated countJobsSince to exclude crawl jobs that failed due to connection disruption or provider limitations (e.g., Firecrawl HTTP 402, timeouts, DNS failures) where successPages === 0.
- Updated sumPagesCrawledByUser to avoid deducting pages from user limits when a job fails without scraping any page.
Error Mapping Expansion (error-mapping.helper.ts):
- Enhanced isConnectionLossError to identify provider credit exhaustion (402, credit), network drops (ECONNRESET, ECONNREFUSED), timeouts, and DNS failures.
Test Suite:
- All Jest unit test suites pass with 100% success rate (39 suites, 430 tests passing).

## Pull Request Overview: Client Resilience, Quota Waiver Policy & Bilingual Localization

---

### Backend Changes (`data-crawler-be`)

#### 1. Quota Exemption Policy for Failed Connection & Provider Outages
- **Quota Exemption Logic** (`crawl-job.repository.ts`):
  - Updated `countJobsSince` to exclude crawl jobs that failed due to connection disruption or provider limitations (e.g., Firecrawl HTTP 402, timeouts, DNS failures) where `successPages === 0`.
  - Updated `sumPagesCrawledByUser` to avoid deducting pages from user limits when a job fails without scraping any page.
- **Error Mapping Expansion** (`error-mapping.helper.ts`):
  - Enhanced `isConnectionLossError` to identify provider credit exhaustion (`402`, `credit`), network drops (`ECONNRESET`, `ECONNREFUSED`), timeouts, and DNS failures.
- **Test Suite**:
  - All Jest unit test suites pass with 100% success rate (39 suites, 430 tests passing).