Document Language Router Benchmark Report

Date: 2026-03-18 GPU: Apple M2 (MPS), 8 GB unified memory Test: 32 pages across 5 PDFs — English tables, Arabic text, mixed bilingual, blank/decorative, photos Goal: Classify page images as "pure English" vs "everything else" for OCR model routing

Context

The OCR pipeline uses two models on a single NVIDIA L40S (46 GB VRAM):

• LightOnOCR-2 (1B): Best English OCR (92.8/100, 3.0s/page, ~2 GB) — English only
• DotsOCR v1.5 (14B): Best Arabic OCR (6/6 Arabic numerals, ~14 GB) — handles English too (84.4/100)

A fast router is needed to classify each page image and send it to the right model:

• Pure English -> LightOnOCR-2 (faster, higher quality on English)
• Everything else (Arabic, mixed, blank, unknown) -> DotsOCR v1.5 (safe default)

Key constraint: false positives (Arabic page sent to English-only model) are dangerous — LightOnOCR-2 garbles Arabic. False negatives (English page sent to DotsOCR) are merely suboptimal.

Approach

Models Tested

Zero-shot CLIP-family models that classify images against text prompts, requiring no training data:

Key Finding: Binary Prompts Required

Initial tests used 3 prompts: ["English text", "Arabic text", "mixed text"]. All models scored ~33% English / ~67% Arabic on every page — no discrimination. The 3-prompt softmax creates a structural bias: "Arabic + mixed" always sums to 2/3.

Binary prompts ["English text", "Arabic text"] fixed this completely. The raw cosine similarity (logit) margin between the two prompts clearly separates English and Arabic pages.

Confidence Threshold

Instead of a simple argmax, a margin threshold is used:

• margin = en_logit - ar_logit
• margin > 0.02 -> English (LightOnOCR-2)
• margin <= 0.02 -> DotsOCR (safe default)

This eliminates all false positives by sending borderline cases to DotsOCR.

Threshold Analysis

Test Corpus

32 pages across 5 PDFs, ground truth verified by visual inspection of each page:

Total: 13 English, 19 Arabic/other

Results

Model Comparison (binary prompts, threshold=0.02)

MobileCLIP models failed completely — they predict the same class for every page regardless of content. Trained on natural image-caption pairs (DataComp), they never learned to recognize document scripts. SigLIP2's WebLI training data (multilingual web content) is the differentiator.

SigLIP2 ViT-B-32-256 Per-Page Breakdown (threshold=0.02)

The 2 False Negatives (English -> DotsOCR)

Both are English pages with weak text signal (margin +0.003, below 0.02 threshold):

• oman p10: infographic with sparse text, mostly visual chart
• kfd p1: table with large car photo occupying most of the page

Impact is minimal: DotsOCR handles English at 84.4/100 (vs LightOnOCR-2's 92.8/100).

Margin Distribution

Clear separation between English and Arabic text pages:

• English text pages: +0.030 to +0.054 (well above 0.02 threshold)
• Arabic text pages: -0.006 to -0.055 (well below threshold)
• Mixed/blank pages: -0.002 to +0.019 (correctly below threshold)
• Weak English: +0.003 (sparse text / photo-heavy — below threshold, safe fallback)

Also Tested: PyMuPDF Text Extraction

For PDF inputs (not scanned images), PyMuPDF can extract the embedded text layer and detect language via Unicode character ranges:

text = page.get_text()
arabic = sum(1 for c in text if '\u0600' <= c <= '\u06FF' or ...)
latin = sum(1 for c in text if 'A' <= c <= 'Z' or 'a' <= c <= 'z')

PyMuPDF is faster and more accurate when a text layer exists, but fails on scanned documents. Can be used as a first-pass optimization before SigLIP2 fallback.

Triton Deployment Validation

The SigLIP2 vision encoder was exported to ONNX and deployed inside Triton Inference Server with a BLS (Business Logic Scripting) pipeline. Full end-to-end validation on the L40S GPU confirmed identical results to the PyTorch benchmark.

ONNX Export

• Vision encoder only (text embeddings pre-computed as static .npy file)
• ONNX opset 17, native IR version (no downgrade needed on Triton 26.02+)
• Model size: 1.4 MB ONNX + 361 MB weights
• Validation: 0.000000 max difference in features and logits vs PyTorch

Triton Architecture

triton-server (GPU)
  └── router_onnx (ONNX Runtime, GPU) — vision encoder inference

TritonClient (direct gRPC)
  └── classify(data_url) → gRPC → router_pipeline (BLS, CPU)
        └── image decode, PIL BICUBIC resize, normalize, call router_onnx,
            cosine similarity with text embeddings, margin threshold → route decision

Triton Results (L40S, 10 runs per page)

All 32 page logits match the PyTorch benchmark to 4 decimal places. The same 2 false negatives (kfd p1, oman p10 — sparse text with large photos) fall below the 0.02 threshold.

The 55ms end-to-end time includes HTTP request parsing, base64 decode, PIL preprocessing, gRPC to Triton, GPU inference, cosine similarity, and JSON response serialization.

Key Preprocessing Detail

PIL BICUBIC resize must be used (not OpenCV INTER_CUBIC) to match open_clip's preprocessing pipeline. Using OpenCV produces slightly different pixel values that shift margins enough to affect threshold-sensitive pages.

Concurrency

The router handles ~32 RPS on L40S with original instance counts (1 GPU, 2 CPU). Increasing instances caused GPU contention and degraded performance. See Triton Concurrency Tuning for full analysis.

Recommendation

Primary router: SigLIP2 ViT-B-32-256 (google/siglip2-base-patch32-256)

Proposed routing logic:

page_image -> SigLIP2 encode_image -> cosine similarity with pre-computed text embeddings
  if (en_logit - ar_logit) > 0.02:
    route to LightOnOCR-2  (pure English, fast)
  else:
    route to DotsOCR v1.5  (Arabic/mixed/unknown, safe)

Optional optimization for PDF inputs:

if pdf has text layer:
  extract text -> count Arabic vs Latin Unicode chars
  if >95% Latin: route to LightOnOCR-2
  else: route to DotsOCR v1.5
else:
  fall through to SigLIP2 visual classification

Both LightOnOCR-2 (~2 GB) + DotsOCR (~14 GB) + SigLIP2 (~350 MB) fit on a single L40S with ~30 GB to spare.

How to Reproduce

PyTorch Benchmark

pip install torch open_clip_torch transformers pymupdf pillow

# Run the benchmark (default: SigLIP2 B32 on MPS)
python docs/research/benchmark_router/benchmark_router.py

# Run on CUDA
python docs/research/benchmark_router/benchmark_router.py --device cuda --runs 10

ONNX Export & Triton Deployment

The router model is automatically exported and baked into the Triton Docker image via a multi-stage build. No manual setup needed:

# Build Triton image (exports SigLIP2 to ONNX in a builder stage)
docker compose build triton

# Start Triton
docker compose up -d triton

# Validate against the 32-page test corpus
python scripts/validate_router_triton.py --runs 10

For standalone export without Docker (e.g., local development):

pip install torch open_clip_torch transformers onnx onnxscript onnxruntime
python scripts/export_router_onnx.py

Test PDFs must be in data/pdfs/ relative to the repo root.