Arabic OCR Needs Script-Aware Evaluation, Not Just a Headline Score

Adaptation targets more than an alphabet

Falcon-OCR-Arabic is a 270-million-parameter document model adapted from Falcon OCR without an architectural change. The source describes two training stages: supervised fine-tuning on real and synthetic Arabic documents, followed by reinforcement learning on curated samples. Its early-fusion design processes image patches and text tokens in a shared Transformer, while prompting selects plain text, LaTeX or HTML output.

That recipe addresses a problem broader than recognizing Arabic characters. An OCR system must preserve dots and optional diacritics, handle connected letterforms, reconcile Arabic and Latin text, distinguish numeral systems, reconstruct right-to-left tables and emit a sensible reading order. A page can look mostly correct while still being unusable because a name changed, an amount moved to the wrong row or columns were serialized backward.

What the reported benchmark establishes

The authors evaluate 11,974 real-world samples across 15 document categories. They report 81.87% text accuracy, defined from normalized page-level edit distance with diacritics included, and 59.95% Table TEDS. In their 17-model comparison, that places the model second for text and first for tables. The unadapted Falcon OCR baseline reports 55.39% text accuracy and 24.83% Table TEDS on the same benchmark.

Those figures provide evidence that targeted adaptation substantially helps this base model on the authors' Arabic test set. They should not be read as a universal ranking for every Arabic OCR workload. Results depend on the document mixture, reference transcriptions, preprocessing, prompts, decoding settings and scoring implementation. The benchmark is also reported by the model's creators, so independent reproduction remains valuable.

Category results reinforce the need for a granular reading. The source reports leading scores for official documents, administrative forms, receipts and invoices, but wider gaps on newspapers, magazines and comics. A single average can conceal exactly the layouts that dominate a prospective deployment.

Build an evaluation set around failure cost

A useful pilot starts with documents sampled from the real intake stream rather than polished examples. Preserve scans from different devices, compression levels, lighting conditions and page orientations. Include stamps, handwriting, skew, bleed-through and partial crops in proportions that match production. Separate results by document type and capture condition so an apparent improvement cannot be produced by an easier mix.

References should encode the output the application actually needs. For archival search, normalized text may be enough. For invoices, row and column associations are essential. For legal or identity material, exact spelling, digits and diacritics may carry disproportionate risk. Define normalization rules in advance: whether to preserve tashkeel, how to represent Arabic and Western numerals, and how to order mixed-direction lines.

Measure at least three layers. Character or word error reveals transcription fidelity. A layout-aware metric tests tables and reading order. Field-level accuracy tests the downstream task, such as matching an invoice total to its label. Report missing content and invented content separately; the operational response to each may differ.

Treat tables as structured predictions

A high table metric is promising, but it does not guarantee that every exported table is safe to consume automatically. Small structural mistakes can be expensive: a merged cell can shift values, a reversed column order can detach quantities from descriptions, and a plausible hallucinated row can pass superficial review.

Test tables with explicit invariants. Totals should reconcile with line items when arithmetic permits. Dates, currencies and identifiers should remain attached to their labels. Row and column counts can be compared with human references, while HTML output should be parsed rather than judged only as visible text. For mixed Arabic and Latin content, verify both logical serialization and rendered directionality. These checks turn OCR evaluation from a screenshot comparison into a data-quality exercise.

Compact models still require system measurements

Parameter count is not a deployment benchmark. A 270-million-parameter model may offer attractive storage characteristics, but latency, memory use, image resolution, token output length and runtime support determine whether it fits a device or service. Measure cold start and steady-state throughput on the intended hardware. Track peak memory and worst-case latency for dense pages, not only medians for short receipts.

The same caution applies to privacy and review workflows. Local execution can reduce data exposure only if the complete pipeline, including image conversion, logging and fallback processing, stays within the required boundary. High-risk fields should receive validation rules or human review based on confidence and business impact.

Falcon-OCR-Arabic is therefore best treated as a strong candidate for an Arabic-specific bake-off. Its reported gains make adaptation look worthwhile; a deployment decision should follow only after representative, category-level and downstream evaluation under the exact output conventions and hardware constraints of the intended system.