Guía del desarrollador sobre OmniParseBench: benchmarks de clasificación y pruebas de OCR

En este tutorial, implementamos OmniParseBench, el benchmark abierto de Datalab para parsers de documentos, que califica la salida de un parser con 16,288 pequeñas pruebas de aprobar-o-reprobar en 2,937 páginas, desde formularios escaneados y escritura a mano hasta matemáticas de arXiv y tablas densas. En lugar de confiar en el leaderboard, lo reconstruimos a partir de los veredictos que Datalab publicó para 27 parsers, después de alimentar al scorer con una página hecha a mano para ver qué errores cambian un veredicto. Luego medimos la certeza del ranking, la segmentamos por tipo de documento, comprobamos si los dos modelos que la ficha menciona como ayudantes en la construcción de pruebas se comportan de manera distinta en las pruebas corregidas por un modelo, y ejecutamos nosotros mismos Tesseract e Indra OCR en una muestra de 50 páginas. En el camino, descubrimos que los dos mejores parsers están estadísticamente empatados, que el líder cambia en la mayoría de los segmentos, y que una cifra del lanzamiento no coincide con los archivos publicados.

import os
import re
import sys
import json
import time
import html
import shutil
import platform
import traceback
import subprocess
import collections
import urllib.request
from pathlib import Path
 
RESULTS = {}          # step -> one-line result, for the summary table
CLAIMS = []           # (step, claim, held)
CURRENT = ["1"]
 
 
def banner(title):
    print("\n" + "=" * 78)
    print(title)
    print("=" * 78)
 
 
def section(name):
    def wrap(fn):
        def run(*a, **kw):
            CURRENT[0] = name.split(".")[0]
            banner(name)
            t0 = time.time()
            try:
                out = fn(*a, **kw)
                RESULTS[name] = out if isinstance(out, str) else "ok"
                return out
            except Exception as e:
                RESULTS[name] = f"SKIPPED / FAILED -> {type(e).__name__}: {e}"
                print(f"\n[!] {name} did not complete: {type(e).__name__}: {e}")
                traceback.print_exc(limit=3)
                return None
            finally:
                print(f"\n[{time.time() - t0:.1f} s]")
        return run
    return wrap
 
 
def claim(text, held):
    CLAIMS.append((CURRENT[0], text, bool(held)))
    return held
 
 
banner("1. Install omni-parse-bench 1.0.0rc1 and Tesseract, and pin the dataset revision")
SYSTEM = platform.system()
OPB_VERSION = "1.0.0rc1"        # a pre-release: a bare `pip install omni-parse-bench` resolves to 0.2.0, whose API differs
REPO = "datalab-to/omni_parse_bench"
REV = "961c0f89a2da701f5ce53afbd5e80aa469b0369a"   # the dataset with the published verdicts of 27 parsers
INDRA = ["indra-ocr==0.3.2", "numpy==2.5.4", "onnxruntime==1.31.0", "opencv-python-headless==5.0.0.93",
         "pypdfium2==5.14.0", "huggingface_hub==2.2.0"]   # the pins of the package's own Indra server image
WORK = Path(os.environ.get("OPB_WORK") or ("/content/opb" if os.path.isdir("/content") else "opb_work")).resolve()
WORK.mkdir(parents=True, exist_ok=True)
subprocess.run([sys.executable, "-m", "pip", "install", "-q", f"omni-parse-bench[benchmark]=={OPB_VERSION}", "uv"], check=True)
if SYSTEM == "Linux" and shutil.which("tesseract") is None:
    scripts = ("arab armn beng cans cher cyrl deva ethi frak geor grek gujr guru hang hang-vert hans hans-vert hant "
               "hant-vert hebr jpan jpan-vert knda khmr laoo latn mlym mymr orya sinh syrc taml telu thaa thai tibt viet").split()
    subprocess.run(["apt-get", "-qq", "update"], check=True)
    subprocess.run(["apt-get", "-qq", "install", "-y", "tesseract-ocr", "tesseract-ocr-eng", "tesseract-ocr-osd"]
                   + [f"tesseract-ocr-script-{s}" for s in scripts], check=True, stdout=subprocess.DEVNULL)
 
import numpy as np
import pandas as pd
import importlib.metadata as md
from huggingface_hub import snapshot_download
from omni_parse_bench import benchmark, data, harness, markdown, vocab
from omni_parse_bench.scorers import unit_tests
 
print(f"omni-parse-bench {md.version('omni-parse-bench')}, Python {platform.python_version()}, {SYSTEM} {platform.machine()}, "
      f"{os.cpu_count()} CPU cores")
if SYSTEM == "Linux":
    print("os:", open("/etc/os-release").read().split('PRETTY_NAME="')[1].split('"')[0])
tess = subprocess.run(["tesseract", "--version"], capture_output=True, text=True)
print("tesseract:", (tess.stdout or tess.stderr).splitlines()[0])
TESS_MAJOR = int((tess.stdout or tess.stderr).split()[1].split(".")[0])
print("test families:", {f: [t for t, g in vocab.FAMILIES.items() if g == f] for f in dict.fromkeys(vocab.FAMILIES.values())})
print(f"dataset {REPO} @ {REV[:7]}; work folder {WORK}")
RESULTS["1. Install omni-parse-bench 1.0.0rc1 and Tesseract, and pin the dataset revision"] = (
    f"omni-parse-bench {md.version('omni-parse-bench')}, tesseract {(tess.stdout or tess.stderr).split()[1]}, dataset @ {REV[:7]}")

Instalamos omni-parse-bench fijado a 1.0.0rc1, porque un pip install sin fijar aún resuelve a 0.2.0, una versión anterior con una API diferente. El extra benchmark trae los adaptadores de parsers, el cliente de Hugging Face y pyarrow para el manifiesto, y en Colab añadimos Tesseract con sus modelos de escritura mediante apt. También fijamos el dataset a un commit, ya que el repositorio cambia a medida que se añaden parsers. Imprimir las familias muestra la estructura sobre la que descansa el benchmark: las pruebas present, order y repeat forman la familia text; table_cell forma la familia tables; y layout_kind forma la familia layout.

@section("2. How the scorer decides: one hand-built page, five test types, one change at a time")
def scorer_anatomy():
    T = lambda i, tt, out, tags, args, st=None: {"id": i, "test_type": tt, "output_type": out, "derivation": "authored",
                                                 "tags": tags, "args": args, "settings": st or {}}
    exact = lambda s: {"text": s, "max_diffs": 0}
    gold = [
        T("title", "present", "html", ["heading"], {"content": exact("Quarterly Report")}),
        T("total", "present", "html", ["eng"], {"content": exact("Total due")}),
        T("growth", "present", "html", ["eng"], {"content": {"number": "-4.8%"}}),
        T("amount", "present", "html", ["eng"], {"content": {"number": "1,240.00"}}),
        T("eq_ratio", "present", "html", ["math"], {"content": {"latex": r"\alpha_{1} + \beta"}}),
        T("eq_half", "present", "html", ["math"], {"content": {"latex": r"\frac{1}{2} \to x"}}),
        T("order", "order", "html", ["eng"], {"before": exact("Introduction: this report covers the second quarter."),
                                              "after": exact("Conclusion: the plan holds.")}),
        T("repeat", "repeat", "html", ["eng"], {"content": exact("Revenue")}, {"expected_count": 2}),
        T("cell_top", "table_cell", "html", ["table"], {"cell": exact("1,240"), "top_heading": exact("2024")}),
        T("cell_left", "table_cell", "html", ["table"], {"cell": exact("1,240"), "left_heading": exact("Income")}),
        T("cell_price", "table_cell", "html", ["table"], {"cell": exact("9.50"), "top_heading": exact("Price"),
                                                          "left": exact("Widget")}),
        T("block_title", "layout_kind", "blocks", ["heading"],
          {"content": dict(exact("Quarterly Report"), place=[[0.10, 0.04, 0.90, 0.09]])}, {"kind": "heading"}),
        T("block_total", "layout_kind", "blocks", ["eng"],
          {"content": dict(exact("Total due"), place=[[0.10, 0.12, 0.90, 0.16]])}, {"kind": "text"}),
    ]
    gold_file = WORK / "fixture" / "gold" / "demo-p1.json"           # a gold file is named after its page
    gold_file.parent.mkdir(parents=True, exist_ok=True)
    gold_file.write_text(json.dumps({"tests": gold}, indent=1))
    page = data.Page(page_id="demo-p1", document="demo", suite="demo", doc_path=WORK / "fixture" / "demo-p1.pdf",
                     gt_path=gold_file, outputs=("blocks", "html"))
    tests = unit_tests.gold(page)
 
    table_a = ("<table><tr><th></th><th>2023</th><th>2024</th></tr><tr><th>Income</th><td>1,000</td><td>1,240</td></tr>"
               "<tr><th>Costs</th><td>600</td><td>700</td></tr></table>")
    table_b = "<table><tr><th>Product</th><th>Price</th></tr><tr><td>Widget</td><td>9.50</td></tr></table>"
    md_table_b = "| Product | Price |\n|---|---|\n| Widget | 9.50 |"
    parts = {"title": "<h1>Quarterly Report</h1>", "total": "<p>Total due: 1,240.00 USD.</p>",
             "growth": "<p>Growth was -4.8% year on year.</p>", "ratio": r"<p>Ratio: 1+ holds.</p>",
             "half": r"<p>Half: 12x in the model.</p>",
             "intro": "<p>Introduction: this report covers the second quarter.</p>",
             "revenue": "<p>Revenue is shown in thousands. Revenue grew in the second quarter.</p>",
             "table_a": table_a, "conclusion": "<p>Conclusion: the plan holds.</p>", "table_b": table_b}
    blocks = [{"label": "heading", "bbox": [0.10, 0.04, 0.90, 0.09], "text": "Quarterly Report"},
              {"label": "text", "bbox": [0.10, 0.12, 0.90, 0.16], "text": "Total due: 1,240.00 USD."},
              {"label": "table", "bbox": [0.10, 0.40, 0.90, 0.60], "text": "Income 1,000 1,240 Costs 600 700"}]
 
    def pred(change=None, order=None, new_blocks=None):
        p = dict(parts, **(change or {}))
        return {"html": "\n".join(p[k] for k in (order or list(parts))), "blocks": blocks if new_blocks is None else new_blocks}
 
    base = {v.id: v.result for v in unit_tests.score(pred(), tests)}
    print(f"base prediction: {sum(r == 'pass' for r in base.values())} of {len(tests)} tests pass\n")
    swapped = ["title", "total", "growth", "ratio", "half", "conclusion", "intro", "revenue", "table_a", "table_b"]
    variants = [
        ("flip", "a typo: 'Total dve'", pred({"total": "<p>Total dve: 1,240.00 USD.</p>"}), ()),
        ("flip", "the minus sign dropped: 4.8%", pred({"growth": "<p>Growth was 4.8% year on year.</p>"}), ()),
        ("flip", "another symbol: alpha_2", pred({"ratio": r"<p>Ratio: 2+ holds.</p>"}), ()),
        ("flip", "conclusion before introduction", pred(order=swapped), ()),
        ("flip", "'Revenues' for the second 'Revenue'",
         pred({"revenue": "<p>Revenue is shown in thousands. Revenues grew in the second quarter.</p>"}), ()),
        ("flip", "2023 and 2024 headings swapped", pred({"table_a": table_a.replace("<th>2023</th><th>2024</th>",
                                                                                     "<th>2024</th><th>2023</th>")}), ()),
        ("flip", "row label 'Margin' for 'Income'", pred({"table_a": table_a.replace("<th>Income</th>", "<th>Margin</th>")}), ()),
        ("flip", "a markdown table left in the html", pred({"table_b": md_table_b}), ()),
        ("flip", "the title's block labelled text", pred(new_blocks=[dict(blocks[0], label="text")] + blocks[1:]), ()),
        ("control", r"\dfrac12\rightarrow for \frac{1}{2}\to", pred({"half": r"<p>Half: $\dfrac12\rightarrow x$ in the model.</p>"}), ()),
        ("control", "'TOTAL DUE' and '1240.00'", pred({"total": "<p>TOTAL DUE: 1240.00 USD.</p>"}), ()),
        ("control", "markdown table through markdown.to_html", pred({"table_b": markdown.to_html(md_table_b)}), ()),
        ("unsupported", "no blocks, declared unsupported", {"html": pred()["html"]}, ("blocks",)),
    ]
    as_expected = 0
    for kind, name, p, unsupported in variants:
        changed = [v for v in unit_tests.score(p, tests, unsupported) if v.result != base[v.id]]
        if kind == "flip":
            ok = len(changed) == 1 and changed[0].result == "fail"
        elif kind == "control":
            ok = not changed
        else:
            ok = bool(changed) and all(v.result == "unsupported" for v in changed)
        as_expected += ok
        shown = "; ".join(f"{v.id} -> {v.result}" + (f" ({v.why})" if v.result == "fail" else "") for v in changed) or "no verdict changes"
        print(f"{kind:11s} {name:42s} {shown}")
    s = unit_tests.summarize(unit_tests.lines(unit_tests.score(variants[0][2], tests), tests, page))
    print(f"\nsummary of the typo variant: headline {s['headline']:.4f} = mean of "
          + ", ".join(f"{k} {v['score']:.3f} ({v['tests']} tests)" for k, v in s["families"].items()))
    claim("every one-change variant flips exactly one verdict to fail, and no control flips any", as_expected == len(variants))
    return f"{as_expected} of {len(variants)} hand-built variants behave as expected (9 flips, 3 controls, 1 unsupported)"
 
 
scorer_anatomy()

Antes de confiar en una puntuación, queremos ver qué significa un veredicto, así que escribimos a mano un archivo gold con trece pruebas sobre una página inventada y calificamos predicciones con unit_tests.score, la función que el propio benchmark usa. Una predicción base pasa todo, y cada variante cambia una sola cosa. Un error tipográfico de una letra, un signo menos perdido, un símbolo diferente en una ecuación, encabezados intercambiados y un bloque mal etiquetado cambian exactamente una prueba a fallida, con una razón adjunta. Los controles siguen pasando: mayúsculas, un separador de miles perdido y dfrac en lugar de frac, porque el texto se pliega y las ecuaciones se comparan después de que KaTeX las renderiza.

@section("3. The dataset: 16,288 tests, what they check, and how each answer was established")
def dataset_step():
    global DATA, PAGES, TESTS, CARD_TEXT
    snapshot_download(REPO, repo_type="dataset", revision=REV, local_dir=str(WORK / "dataset"),
                      allow_patterns=["manifest.parquet", "gold/*", "results.parquet", "README.md"])
    DATA, PAGES = data.read_manifest(str(WORK / "dataset"))
    rows = []
    for p in PAGES:
        for t in unit_tests.gold(p):
            rows.append(dict(test_id=t.id, page_id=p.page_id, suite=p.suite, document=p.document, test_type=t.test_type,
                             derivation=t.derivation, tags=list(t.tags), pdf=p.doc_path.suffix == ".pdf"))
    TESTS = pd.DataFrame(rows)
    tags = collections.Counter(g for ts in TESTS.tags for g in ts)
    langs = {g for g in tags if re.fullmatch(r"[a-z]{3}", g)}
    scripts = {g for g in tags if re.fullmatch(r"[A-Z][a-z]{3}", g)}
    n_pdf = sum(p.doc_path.suffix == ".pdf" for p in PAGES)
    print(f"{len(TESTS):,} tests on {TESTS.page_id.nunique():,} pages from {TESTS.document.nunique():,} documents in "
          f"{TESTS.suite.nunique()} suites; {n_pdf:,} PDF pages and {len(PAGES) - n_pdf:,} images")
    print(f"tags: {len(langs)} language codes ({len(langs - {'und', 'zxx'})} without und and zxx), {len(scripts)} scripts")
    counts = lambda col: {k: int(v) for k, v in col.value_counts().items()}
    print("\ntests by type:          ", counts(TESTS.test_type))
    print("tests by family:        ", counts(TESTS.test_type.map(vocab.FAMILIES)))
    print("structure and role tags:", {g: n for g, n in tags.most_common() if g not in langs | scripts})
    print("\nhow each test's answer was established (derivation):")
    for d, n in TESTS.derivation.value_counts().items():
        print(f"   {d:22s} {n:6,d}   {', '.join(f'{k} {v}' for k, v in TESTS[TESTS.derivation == d].test_type.value_counts().items())}")
    CARD_TEXT = (DATA / "README.md").read_text()
    claim("16,288 tests on 2,937 pages from 2,343 documents in 22 suites, as the card says",
          (len(TESTS), TESTS.page_id.nunique(), TESTS.document.nunique(), TESTS.suite.nunique()) == (16288, 2937, 2343, 22))
    return f"{len(TESTS):,} tests, {TESTS.page_id.nunique():,} pages, {TESTS.document.nunique():,} documents, {len(langs - {'und', 'zxx'})} languages"
 
 
dataset_step()

Con el scorer entendido, descargamos el manifiesto, los archivos gold y los veredictos publicados en la revisión fijada, saltamos los 1.5 GB de imágenes de páginas, y leemos las pruebas de cada página mediante unit_tests.gold. Los recuentos coinciden con la ficha del dataset, incluidos 92 idiomas una vez eliminados los códigos de idioma indeterminado y ausente. Dos campos merecen atención. Etiquetas como handwriting, rotated o table nunca deciden un veredicto; solo nos permiten segmentar resultados. Derivation registra cómo se estableció cada respuesta, desde el marcado del editor y capas de texto hasta paneles de modelos y revisión humana, lo que nos permite comprobar más tarde si la construcción de las pruebas favoreció a algún parser.

COLS = ["page_id", "suite", "document", "test_id", "test_type", "output_type", "derivation", "tags", "result", "why"]
STARRED = ["Gemini 3.8 Flash", "Claude Sonnet 5.5"]    # the card: models "used in creating and verifying at least some tests"
 
 
@section("4. Recompute the published leaderboard from 439,776 verdicts")
def recompute_table():
    global VERDICTS, TABLE
    VERDICTS = pd.read_parquet(DATA / "results.parquet")
    VERDICTS["output_type"] = VERDICTS.test_type.map(vocab.READS)
    VERDICTS["tags"] = VERDICTS.tags.apply(list)
    VERDICTS["family"] = VERDICTS.test_type.map(vocab.FAMILIES)
    rows = []
    for system, g in VERDICTS.groupby("system", sort=False):
        s = unit_tests.summarize(g[COLS].to_dict("records"))
        fam = {k: (np.nan if v["score"] is None else 100 * v["score"]) for k, v in s["families"].items()}
        scored = g[g.result != "unsupported"]
        rows.append(dict(system=system, headline=np.nan if s["headline"] is None else 100 * s["headline"],
                         text=fam.get("text"), tables=fam.get("tables"), layout=fam.get("layout"),
                         pooled=100 * (scored.result == "pass").mean()))
    TABLE = pd.DataFrame(rows).set_index("system")
    card = {}
    for line in CARD_TEXT.splitlines():
        cells = [c.strip().strip("*").strip() for c in line.strip().strip("|").split("|")]
        if len(cells) == 6 and cells[1] in ("open", "closed"):
            card[cells[0]] = [None if c in ("none", "unsupported") else float(c) for c in cells[2:]]
    mismatch = []
    for name, vals in card.items():
        mine = [None if pd.isna(x) else round(x, 1) for x in TABLE.loc[name, ["headline", "text", "tables", "layout"]]]
        mismatch += [(name, a, b) for a, b in zip(mine, vals) if a != b]
    print(f"{len(VERDICTS):,} verdicts = {VERDICTS.system.nunique()} parsers x {VERDICTS.test_id.nunique():,} tests: "
          f"{ {k: int(v) for k, v in VERDICTS.result.value_counts().items()} }")
    numeric = sum(v is not None for vals in card.values() for v in vals)
    print(f"the card's table: {len(card)} parsers, {4 * len(card)} entries ({numeric} numbers, the rest none or unsupported); "
          f"recomputed with unit_tests.summarize, {len(mismatch)} differ {mismatch[:3] if mismatch else ''}\n")
    show = TABLE.sort_values("headline", ascending=False, na_position="last")
    f = lambda x: "none" if pd.isna(x) else f"{x:.2f}"
    print(f"{'parser':24s} {'headline':>9s} {'text':>7s} {'tables':>7s} {'layout':>7s}   {'pooled':>7s}")
    for name, r in show.iterrows():
        if name in show.index[:8] or pd.isna(r.headline):
            print(f"{name + ('*' if name in STARRED else ''):24s} {f(r.headline):>9s} {f(r.text):>7s} {f(r.tables):>7s} "
                  f"{f(r.layout):>7s}   {r.pooled:7.2f}")
    a, g = TABLE.loc["Datalab accurate", "headline"], TABLE.loc["Gemini 3.8 Flash", "headline"]
    ranked = TABLE.dropna(subset=["headline"])
    print(f"\nDatalab accurate {a:.3f} vs Gemini 3.8 Flash {g:.3f}: both {a:.1f} at one decimal, {a - g:+.3f} points apart")
    print("top 3 by headline (mean of the three families):", list(ranked.headline.nlargest(3).index))
    print("top 3 by pooled pass rate over every scored test:", list(ranked.pooled.nlargest(3).index))
    claim("every number in the card's results table is reproduced from results.parquet", not mismatch)
    return f"{4 * len(card) - len(mismatch)} of {4 * len(card)} card entries reproduced; Datalab accurate {a:.2f}, Gemini {g:.2f}"
 
 
recompute_table()

Como results.parquet contiene una fila por parser y por prueba, podemos reconstruir el leaderboard pasando las filas de cada parser a unit_tests.summarize. La cifra principal es la media de las tasas de aprobación de text, tables y layout con las pruebas no soportadas excluidas, de modo que un parser que no puede devolver bloques de layout, como Tesseract, no obtiene ninguna cifra principal. Las 108 entradas de la tabla de la ficha coinciden con sus valores de un decimal. El recálculo también revela un casi empate en la cima: Datalab accurate y Gemini 3.8 Flash ambos redondean a 92.9, con 0.024 puntos de diferencia, y agrupar todas las pruebas en lugar de promediar las familias coloca a Gemini primero.

def per_document(df, systems, groups):
    """For each system: an array (documents x groups x [passes, scored tests]); `groups` maps a name to a row mask."""
    docs = {d: i for i, d in enumerate(sorted(df.document.unique()))}
    s = df[df.result != "unsupported"]
    out = {}
    for name in systems:
        g = s[s.system == name]
        arr = np.zeros((len(docs), len(groups), 2))
        for k, mask in enumerate(groups.values()):
            sel = g[mask(g)]
            d = sel.document.map(docs).to_numpy()
            arr[:, k, 0] = np.bincount(d, weights=(sel.result == "pass").to_numpy(), minlength=len(docs))
            arr[:, k, 1] = np.bincount(d, minlength=len(docs))
        out[name] = arr
    return out, len(docs)
 
 
def resample(n_docs, B, seed):
    """B bootstrap resamples of the documents, each as a count per document."""
    rng = np.random.default_rng(seed)
    return np.stack([np.bincount(rng.integers(0, n_docs, n_docs), minlength=n_docs) for _ in range(B)]).astype(float)
 
 
@section("5. How sure is the order at the top? A document-level bootstrap")
def bootstrap_step():
    top = list(TABLE.headline.nlargest(6).index)
    families = {f: (lambda g, f=f: g.family == f) for f in ("text", "tables", "layout")}
    arrs, n_docs = per_document(VERDICTS[VERDICTS.system.isin(top)], top, families)
    W = resample(n_docs, 2000, seed=20261022)
    head = {}
    for s in top:
        tot = np.tensordot(W, arrs[s], axes=([1], [0]))            # resamples x families x [passes, tests]
        head[s] = 100 * (tot[..., 0] / tot[..., 1]).mean(axis=1)
    point = {s: 100 * (arrs[s].sum(0)[:, 0] / arrs[s].sum(0)[:, 1]).mean() for s in top}
    print(f"2,000 resamples of {n_docs:,} documents; a document's tests stay together\n")
    print(f"{'parser':22s} {'headline':>9s}  {'95% interval':>15s}   {'gap to ' + top[0]:>26s}  {'resamples where it leads':>24s}")
    leader = np.argmax(np.stack([head[x] for x in top]), axis=0)
    for s in top:
        lo, hi = np.percentile(head[s], [2.5, 97.5])
        d = head[top[0]] - head[s]
        gap = "" if s == top[0] else f"{point[top[0]] - point[s]:+.2f} [{np.percentile(d, 2.5):+.2f}, {np.percentile(d, 97.5):+.2f}]"
        print(f"{s:22s} {point[s]:9.2f}  {lo:6.2f} - {hi:6.2f}   {gap:>26s}  {100 * np.mean(leader == top.index(s)):23.1f}%")
    d = head[top[0]] - head[top[1]]
    lo, hi = np.percentile(d, [2.5, 97.5])
    claim(f"{top[0]} and {top[1]} cannot be told apart (the 95% interval of their gap includes zero)", lo < 0 < hi)
    return f"{top[0]} vs {top[1]}: {point[top[0]] - point[top[1]]:+.2f} points, 95% interval [{lo:+.2f}, {hi:+.2f}]"
 
 
bootstrap_step()

Una brecha tan pequeña plantea la pregunta de cuánto del orden es ruido. Las pruebas en un mismo documento tienden a aprobar o fallar juntas, así que remuestreamos documentos en lugar de pruebas: cada uno de 2,000 sorteos elige 2,343 documentos con reemplazo y recalcula cada cifra principal a partir de recuentos de aprobación por documento con un solo producto de matrices. La brecha entre los dos primeros oscila entre aproximadamente -0.9 y +0.9 puntos, y cada uno lidera en aproximadamente la mitad de los sorteos, por lo que la lectura honesta es un empate. Datalab balanced, a 1.35 puntos del líder, está claramente separado de él, y también lo está cada parser por debajo.

def slice_headline(df):
    """Per system: family pass rates on a slice of tests and their mean, as the benchmark's headline is built."""
    s = df[df.result != "unsupported"]
    fam = s.groupby(["system", "family"]).result.apply(lambda r: 100 * (r == "pass").mean()).unstack()
    return fam.mean(axis=1), fam
 
 
@section("6. Slices where a different parser comes first")
def slices_step():
    ranked = TABLE.dropna(subset=["headline"]).sort_values("headline", ascending=False)
    rank = {s: i + 1 for i, s in enumerate(ranked.index)}
    V = VERDICTS[VERDICTS.system.isin(ranked.index)]
    whole, _ = slice_headline(V)
    assert np.allclose(whole[ranked.index], ranked.headline), "slice_headline disagrees with unit_tests.summarize"
    print(f"{'tag':13s} {'tests':>6s} {'docs':>5s}  {'first (overall rank)':34s} {'second':26s} families")
    leaders = collections.Counter()
    for tag in ("handwriting", "tiny", "rotated", "degraded", "math", "table", "form", "multi_column",
                "Grek", "Hani", "Cyrl", "Armn", "Laoo"):
        sl = V[V.tags.apply(lambda t: tag in t)]
        head, fam = slice_headline(sl)
        order = head.sort_values(ascending=False)
        (a, x), (b, y) = list(order.items())[:2]
        docs = sl.document.nunique()
        leaders[a] += 1
        note = ("  (tie for first)" if x == y else "") + ("  (10 or fewer documents)" if docs <= 10 else "")
        print(f"{tag:13s} {sl.test_id.nunique():6d} {docs:5d}  {f'{a} {x:.1f} (#{rank[a]})':34s} {f'{b} {y:.1f}':26s} "
              f"{'+'.join(c for c in ('text', 'tables', 'layout') if c in fam.columns)}{note}")
    first = ranked.index[0]
    claim("the overall leader is not first on every slice", leaders[first] < 13)
    return f"{first} is first on {leaders[first]} of 13 slices; {len(leaders) - (first in leaders)} other parsers lead the rest"
 
 
slices_step()

Segmentar por etiqueta revela lo que un solo titular oculta sobre los documentos que realmente tenemos. Nosotros recalculamos el titular sobre escritura a mano, páginas rotadas, tablas, escrituras únicasy otras segmentaciones con la misma regla de media por familia, después de que una aserción confirme que la regla reproduce los titulares oficiales en el conjunto completo. El líder general va primero en solo tres de trece segmentaciones. Gemini 3.8 Flash lidera en escritura a mano, matemáticas y formularios; Claude Sonnet 5.5 en páginas rotadas; y Mistral OCR en griego. El armenio y el lao se apoyan en seis y cinco documentos, por lo que los tratamos como indicios más que como clasificaciones.

@section("7. Do the two models named on the card hold up better on tests a model corrected? A difference-in-differences check")
def creation_check():
    others = [s for s in TABLE.dropna(subset=["headline"]).index if s not in STARRED]
    V = VERDICTS[VERDICTS.system.isin(STARRED + others)]
    print(f"starred: {STARRED}; compared with the {len(others)} other parsers that have a headline")
    print("DiD = (starred: pass rate on the split - elsewhere) - (others: the same gap); 1,000 document resamples\n")
    print(f"{'tests established by':21s} {'tests':>6s}  {'starred in / out':>17s}  {'others in / out':>16s}  {'DiD [95% interval]':>23s}")
    W = None
    out = {}
    for deriv, note in (("model_review", "a vision model corrected the test"),
                        ("model_panel", "independent models agreed"),
                        ("human_review", "placebo: a person decided")):
        groups = {"in": lambda g, d=deriv: g.derivation == d, "out": lambda g, d=deriv: g.derivation != d}
        arrs, n_docs = per_document(V, STARRED + others, groups)
        W = resample(n_docs, 1000, seed=7) if W is None else W
 
        def gaps(weights):
            r = {}
            for s, a in arrs.items():
                tot = np.tensordot(weights, a, axes=([-1], [0]))
                with np.errstate(invalid="ignore", divide="ignore"):
                    r[s] = 100 * tot[..., 0] / tot[..., 1]
            did = (np.nanmean([r[s][..., 0] - r[s][..., 1] for s in STARRED], axis=0)
                   - np.nanmean([r[s][..., 0] - r[s][..., 1] for s in others], axis=0))
            return r, did
        r, d = gaps(np.ones(n_docs))
        _, boot = gaps(W)
        lo, hi = np.nanpercentile(boot, [2.5, 97.5])
        si, so = np.mean([r[s][0] for s in STARRED]), np.mean([r[s][1] for s in STARRED])
        oi, oo = np.mean([r[s][0] for s in others]), np.mean([r[s][1] for s in others])
        out[deriv] = (d, lo, hi)
        print(f"{deriv:21s} {(TESTS.derivation == deriv).sum():6,d}  {si:7.1f} / {so:6.1f}  {oi:6.1f} / {oo:6.1f}  "
              f"{d:+6.1f} [{lo:+5.1f}, {hi:+5.1f}]   {note}")
    panel = TESTS[TESTS.derivation == "model_panel"]
    layout = TESTS[TESTS.test_type == "layout_kind"]
    print(f"\nmodel_panel tests are {100 * (panel.test_type == 'layout_kind').mean():.0f}% layout tests, and "
          f"{100 * (layout.derivation == 'model_panel').mean():.0f}% of layout tests are model_panel")
    review = VERDICTS[VERDICTS.derivation == "model_review"]
    rate = lambda s: 100 * (review[review.system == s].result == "pass").mean()
    print(f"pass rate on the {(TESTS.derivation == 'model_review').sum()} model_review tests: "
          + ", ".join(f"{s} {rate(s):.1f}%" for s in STARRED) + f", median of the others {np.median([rate(s) for s in others]):.1f}%")
    claim("the starred models gain more on model_review tests than on the placebo split", out["model_review"][1] > out["human_review"][2])
    claim("the placebo split shows no gap (its interval includes zero)", out["human_review"][1] < 0 < out["human_review"][2])
    return (f"DiD model_review {out['model_review'][0]:+.1f}, model_panel {out['model_panel'][0]:+.1f}, "
            f"placebo human_review {out['human_review'][0]:+.1f} points")
 
 
creation_check()

La tarjeta del conjunto de datos marca a Gemini 3.8 Flash y Claude Sonnet 5.5 como usados en la creación y verificación de algunas pruebas sin decir cuáles, así que planteamos una pregunta más concreta con un difference-in-differences: ¿la tasa de acierto de un parser en una división de derivación, en relación con su tasa en el resto, cambia de forma distinta para el par marcado? En las 102 pruebas model_review, corregidas por un modelo de visión, ambos grupos aciertan mucho menos que en el resto, pero el par marcado cae unos 25 puntos menos. El placebo human_review muestra una brecha menor en la misma dirección y model_panel va al revés, así que reportamos una señal, no una prueba.

def predict_pages(run, pages, alive=lambda: True):
    """Predict every page the run still lacks. A failed call is recorded, not raised, so we count failures ourselves."""
    t0, n, failed = time.time(), 0, []
    for p in pages:
        if not alive():
            raise RuntimeError(f"{run.provider}: the server stopped after {n} pages")
        if need := benchmark.needs(run, p):
            got = harness.predict(run.provider, p.doc_path, need, timeout=900, **run.options)
            benchmark.record(run, p.page_id, got)
            n += 1
            if any(c.get("failure") for c in got["calls"]):
                failed.append(f"{p.page_id}: {got['errors']}")
    if failed:
        print(f"{len(failed)} of {n} pages failed, e.g. {failed[0][:160]}")
        if 2 * len(failed) >= n:
            raise RuntimeError(f"{run.provider} failed on {len(failed)} of {n} pages")
    return n, (time.time() - t0) / max(n, 1)
 
 
@section("8. Run Tesseract ourselves on 50 pages that span all 22 suites")
def tesseract_step():
    global SAMPLE
    if TESS_MAJOR < 5:
        raise RuntimeError(f"tesseract {TESS_MAJOR}.x is installed; this tutorial was verified with tesseract 5")
    SAMPLE = data.select(PAGES, sample=50)
    snapshot_download(REPO, repo_type="dataset", revision=REV, local_dir=str(DATA),
                      allow_patterns=sorted(str(p.doc_path.relative_to(DATA)) for p in SAMPLE))
    n_pdf = sum(p.doc_path.suffix == ".pdf" for p in SAMPLE)
    print(f"sample: {len(SAMPLE)} pages from {len({p.suite for p in SAMPLE})} suites, {n_pdf} PDF and {len(SAMPLE) - n_pdf} images, "
          f"{sum(p.doc_path.stat().st_size for p in SAMPLE) / 1e6:.1f} MB, {sum(len(unit_tests.gold(p)) for p in SAMPLE)} tests")
    # The adapter asks Tesseract for '-l script/<Name>'. Homebrew keeps those models in a script/ folder; Ubuntu installs
    # them flat, so on Linux we point Tesseract at a folder of symlinks with the layout it expects.
    listing = subprocess.run(["tesseract", "--list-langs"], capture_output=True, text=True)
    listing = listing.stdout + listing.stderr
    if "script/Latin" not in listing:
        src = Path(re.search(r'"(.+?)"', listing).group(1))
        dst = WORK / "tessdata"
        (dst / "script").mkdir(parents=True, exist_ok=True)
        for f in src.glob("*.traineddata"):
            for link in [dst / f.name] + ([dst / "script" / f.name] if f.name[0].isupper() else []):
                if not link.exists():
                    link.symlink_to(f)
        os.environ["TESSDATA_PREFIX"] = str(dst)
        print(f"script models linked under {dst}")
    run = benchmark.Run("tesseract", {}, WORK / "runs" / "tesseract")
    benchmark.prepare(run, str(DATA), 900)
    n, per_page = predict_pages(run, SAMPLE)
    s = benchmark.grade(run, SAMPLE, say=lambda *_: None)
    fam = s["families"]
    print(f"\nTesseract: {n} pages read, {per_page:.1f} s per page")
    for k, v in fam.items():
        print(f"   {k:7s} " + ("unsupported (no blocks output)" if v["score"] is None else f"{100 * v['score']:5.1f}% of {v['tests']} tests"))
    print(f"   headline {'none' if s['headline'] is None else round(100 * s['headline'], 1)} (a family it cannot produce leaves no headline)")
    return f"Tesseract on 50 pages: text {100 * fam['text']['score']:.1f}%, tables {100 * fam['tables']['score']:.1f}%, {per_page:.1f} s/page"
 
 
tesseract_step()

Reconstruir una tabla de clasificación a partir de veredictos publicados es una cosa; producir veredictos por nosotros mismos es otra. data.select elige una muestra fija de 50 páginas repartidas entre las 22 suites, y descargamos solo esas páginas. El bucle es el propio del benchmark: needs lista las salidas que aún faltan de una página, harness. predict ejecuta el parser; record guarda la predicción y califica la carpeta, de modo que una ejecución interrumpida puede reanudarse. En Ubuntu también enlazamos los modelos de escritura de Tesseract en la disposición que espera el adaptador. Tesseract aprueba el 30.4% de las pruebas de texto de la muestra frente al 17.6% en el conjunto completo, una brecha que depende de qué páginas se muestrearon.

def agreement(folder, published):
    """Our verdicts against the published ones for the same tests, where both scored the test."""
    ours = pd.read_json(folder / "scores.jsonl", lines=True)[["test_id", "result", "why"]]
    pub = VERDICTS[VERDICTS.system == published][["test_id", "result"]]
    m = ours.merge(pub, on="test_id", suffixes=("_ours", "_pub"))
    m = m[(m.result_ours != "unsupported") & (m.result_pub != "unsupported")]
    return m, int((m.result_ours == m.result_pub).sum()), len(m)
 
 
@section("9. Indra OCR on the same pages, served on the CPU from the package's own server code")
def indra_step():
    if sys.version_info < (3, 12):
        raise RuntimeError("the Indra pins (numpy 2.5.4) need Python 3.12 or newer")
    venv = WORK / "indra-venv"
    py = venv / "bin" / "python"
    if not py.exists():
        uv = shutil.which("uv") or str(Path(sys.executable).parent / "uv")
        subprocess.run([uv, "venv", "-q", "--python", sys.executable, str(venv)], check=True)
        subprocess.run([uv, "pip", "install", "-q", "--python", str(py)] + INDRA, check=True)
    server = Path(harness.__file__).parent / "providers" / "oss" / "indra" / "server.py"
    # server.py counts cores with os.sched_getaffinity, which macOS lacks; on Linux the shim changes nothing
    launcher = WORK / "indra_launch.py"
    launcher.write_text("import os, runpy, sys\n"
                        "if not hasattr(os, 'sched_getaffinity'):\n"
                        "    os.sched_getaffinity = lambda pid: set(range(os.cpu_count()))\n"
                        "sys.argv = ['server.py', '--port', '8001', '--workers', '2']\n"
                        f"runpy.run_path({str(server)!r}, run_name='__main__')\n")
    log = open(WORK / "indra_server.log", "w")
    proc = subprocess.Popen([str(py), str(launcher)], stdout=log, stderr=subprocess.STDOUT)
    t0 = time.time()
    while True:
        try:
            urllib.request.urlopen("http://127.0.0.1:8001/health", timeout=2)
            break
        except Exception:
            if proc.poll() is not None or time.time() - t0 > 600:
                proc.terminate()
                raise RuntimeError("the Indra server did not start; see " + log.name)
            time.sleep(2)
    print(f"server up in {time.time() - t0:.0f} s: {INDRA[0]}, 2 worker processes")
    try:
        run = benchmark.Run("indra-ocr", {"base_url": "http://127.0.0.1:8001"}, WORK / "runs" / "indra-ocr")
        benchmark.prepare(run, str(DATA), 900)
        n, per_page = predict_pages(run, SAMPLE, alive=lambda: proc.poll() is None)
        s = benchmark.grade(run, SAMPLE, say=lambda *_: None)
    finally:
        proc.terminate()
    fam = {k: 100 * v["score"] for k, v in s["families"].items()}
    print(f"Indra OCR: {n} pages, {per_page:.2f} s per page; text {fam['text']:.1f}%, tables {fam['tables']:.1f}%, "
          f"layout {fam['layout']:.1f}%; headline on these pages {100 * s['headline']:.1f}\n")
    ids = {p.page_id for p in SAMPLE}
    fmt = lambda x: "none" if x is None else f"{100 * x:.1f}"
    print(f"{'on the 50 pages':22s} {'headline':>9s} {'text':>6s} {'tables':>7s} {'layout':>7s}")
    for name, folder in (("Tesseract", WORK / "runs" / "tesseract"), ("Indra OCR", WORK / "runs" / "indra-ocr")):
        pub = unit_tests.summarize(VERDICTS[(VERDICTS.system == name) & VERDICTS.page_id.isin(ids)][COLS].to_dict("records"))
        ours = unit_tests.summarize(benchmark.read_scores(benchmark.Run(name, {}, folder)))
        for who, x in (("ours", ours), ("published", pub)):
            f = x["families"]
            print(f"{name + ', ' + who:22s} {fmt(x['headline']):>9s} " + " ".join(f"{fmt(f[k]['score']) if k in f else 'none':>6s}"
                                                                         for k in ("text", "tables", "layout")))
    print()
    for name, folder in (("Tesseract", WORK / "runs" / "tesseract"), ("Indra OCR", WORK / "runs" / "indra-ocr")):
        m, same, total = agreement(folder, name)
        diff = m[m.result_ours != m.result_pub]
        print(f"{name:10s} our verdicts equal the published ones on {same} of {total} tests ({100 * same / total:.1f}%); "
              f"of the {len(diff)} that differ, {(diff.result_ours == 'pass').sum()} pass only for us and "
              f"{(diff.result_ours == 'fail').sum()} only in the published run")
        if len(diff):
            by_page = diff.test_id.map(TESTS.set_index("test_id").page_id).value_counts()
            print("           differing tests by type:", {k: int(v) for k, v in diff.test_id.map(TESTS.set_index("test_id").test_type).value_counts().items()},
                  f"| on {len(by_page)} pages, {by_page.iloc[0]} of them on {by_page.index[0]}")
        claim(f"{name}: our verdicts equal the published ones on at least 95% of the sample's tests", same >= 0.95 * total)
    return f"Indra OCR on 50 pages: headline {100 * s['headline']:.1f} (text {fam['text']:.1f}, tables {fam['tables']:.1f}, layout {fam['layout']:.1f}), {per_page:.2f} s/page"
 
 
indra_step()

Indra OCR es un modelo de CPU de código abierto, y el paquete incluye el servidor HTTP con el que habla su adaptador, así que instalamos las versiones fijadas en el Dockerfile de ese servidor en un entorno aparte y lo iniciamos en segundo plano. Frente a los veredictos publicados en las mismas páginas, Tesseract coincide en cada prueba puntuada, por lo que su puntuación de muestra coincide con la publicada. Indra coincide en 203 de 221, y todas sus diferencias en tablas están en una sola página, lo que explica toda la brecha entre su puntuación publicada en tablas de 36.1 y nuestra 69.4. Que una sola página mueva una familia tanto es el argumento más claro para remuestrear documentos.

@section("10. Bring your own parser: grade a PDF text-layer baseline with the same code")
def byo_step():
    import pypdfium2 as pdfium
 
    def text_layer(path):
        if path.suffix.lower() != ".pdf":
            return ""                                    # an image has no text layer to read
        doc = pdfium.PdfDocument(str(path))
        try:
            return "<pre>" + html.escape(doc[0].get_textpage().get_text_range()) + "</pre>"
        finally:
            doc.close()
 
    out = WORK / "runs" / "text-layer"
    (out / "predictions").mkdir(parents=True, exist_ok=True)
    for p in SAMPLE:
        (out / "predictions" / f"{p.page_id}.json").write_text(json.dumps({"html": text_layer(p.doc_path)}))
    s = benchmark.grade(benchmark.Run("text-layer", {}, out), SAMPLE, say=lambda *_: None, settings={"unsupported": ["blocks"]})
    kind = {p.page_id: "PDF" if p.doc_path.suffix == ".pdf" else "image" for p in SAMPLE}
    print(f"{'parser on the same 50 pages':28s} {'text: PDF':>10s} {'images':>8s} {'all':>6s} {'tables':>8s}")
    for name in ("text-layer", "tesseract", "indra-ocr"):
        sc = pd.read_json(WORK / "runs" / name / "scores.jsonl", lines=True)
        sc = sc[sc.result.isin(["pass", "fail"])]
        txt = sc[sc.test_type.map(vocab.FAMILIES) == "text"]
        by = txt.groupby(txt.page_id.map(kind)).result.apply(lambda r: 100 * (r == "pass").mean())
        tab = sc[sc.test_type == "table_cell"]
        print(f"{name:28s} {by.get('PDF', np.nan):10.1f} {by.get('image', np.nan):8.1f} "
              f"{100 * (txt.result == 'pass').mean():6.1f} {100 * (tab.result == 'pass').mean():8.1f}")
    fam = s["families"]
    return f"text layer, no OCR: text {100 * fam['text']['score']:.1f}%, tables {100 * fam['tables']['score']:.1f}% on the 50 pages"
 
 
byo_step()

Cualquier parser puede evaluarse sin un adaptador: escribimos un archivo de predicciones por página y llamamos a benchmark.grade, declarando como no admitidas las salidas que no podemos producir. Nuestro parser lee la capa de texto incrustada de un PDF con pypdfium2 y devuelve una cadena vacía para las imágenes, que no tienen ninguna. En páginas PDF aprueba exactamente tantas pruebas de texto como Tesseract, 42.3% cada uno, sin ningún OCR, mientras que en imágenes no tiene nada e Indra OCR aún aprueba el 43.1%. El texto plano falla todas las pruebas de tablas, ya que un acierto requiere estructura de celdas, así que la mezcla de PDF e imágenes importa siempre que se comparen dos parsers.

@section("11. The launch claims, checked against the files")
def claims_step():
    a, t = TABLE.loc["Datalab accurate"], TABLE.loc["Tesseract"]
    g = TABLE.loc["Gemini 3.8 Flash", "headline"]
    c = TABLE.loc["Claude Sonnet 5.5"]
    tags = TESTS.tags
    langs = {x for ts in tags for x in ts if re.fullmatch("[a-z]{3}", x)} - {"und", "zxx"}
    pkg = Path(vocab.__file__).parent       # the scorer and the modules it reads; data.py and the harness only fetch and run
    scoring = sorted(pkg.glob("scorers/**/*.py")) + [pkg / f"{m}.py" for m in ("normalize", "tables", "equations", "views", "vocab", "markdown")]
    clients = [f.name for f in scoring if re.search(r"^\s*(import|from)\s+(httpx|requests|urllib|huggingface_hub|openai|anthropic|google"
                                                    r"|transformers|torch|omni_parse_bench\.harness)\b", f.read_text(), re.M)]
    n_eng, n_table = tags.apply(lambda x: "eng" in x).sum(), tags.apply(lambda x: "table" in x).sum()
    rows = [
        ("16,288 tests, 2,937 pages, 2,343 documents", f"{len(TESTS):,} / {TESTS.page_id.nunique():,} / {TESTS.document.nunique():,}",
         (len(TESTS), TESTS.page_id.nunique(), TESTS.document.nunique()) == (16288, 2937, 2343)),
        ("92 languages", f"{len(langs)} codes besides und and zxx", len(langs) == 92),
        ("10,089 English tests, 6,767 table tests", f"{n_eng:,} and {n_table:,}", (n_eng, n_table) == (10089, 6767)),
        ("Datalab accurate scores 92.7", f"{a.headline:.2f} in results.parquet", round(a.headline, 1) == 92.7),
        ("Datalab accurate ranks second, behind Gemini", f"{a.headline:.3f} vs {g:.3f}", a.headline < g),
        ("Claude Sonnet 5.5: text 93.1, tables 95.4, layout 77.5", f"{c.text:.1f} / {c.tables:.1f} / {c.layout:.1f}",
         [round(c[k], 1) for k in ("text", "tables", "layout")] == [93.1, 95.4, 77.5]),
        ("Tesseract: no headline, text 17.6, tables 0.0",
         f"{'none' if pd.isna(t.headline) else round(t.headline, 1)} / {t.text:.1f} / {t.tables:.1f}",
         pd.isna(t.headline) and round(t.text, 1) == 17.6 and round(t.tables, 1) == 0.0),
        ("no model judges the outputs", f"{len(clients)} of {len(scoring)} scorer modules import a model or HTTP client", not clients),
    ]
    print(f"{'claim (launch article and blog)':56s} {'what the files say':38s} verdict")
    for text, found, ok in rows:
        print(f"{text:56s} {found:38s} {'holds' if ok else 'does not hold'}")
        claim(text, ok)
    print("\nnot checkable from these files: latency, price per 1,000 pages, the other benchmarks' test counts")
    return f"{sum(bool(ok) for *_, ok in rows)} of {len(rows)} launch claims hold against the files"
 
 
claims_step()

En lugar de repetir las afirmaciones del lanzamiento, comprobamos cada una contra los archivos. Los recuentos se sostienen exactamente, desde pruebas y páginas hasta idiomas y subconjuntos, y las filas de Claude y Tesseract coinciden. Datalabno lo hace en su propia fila: el artículo de lanzamiento y el blog dan a Datalab un 92.7 exacto y lo sitúan segundo detrás de Gemini, mientras que los veredictos publicados dan 92.92 frente a 92.89, un empate que se redondea a la misma cifra. La afirmación de no-judge se sostiene para el propio evaluador, cuyos módulos no importan ningún modelo ni cliente HTTP; KaTeX se ejecuta en un motor de JavaScript incrustado para comparar ecuaciones. La latencia y el precio no se pueden comprobar a partir de estos archivos.

banner("SUMMARY")
for name, res in RESULTS.items():
    print(f"  {name:<100s}  {res}")
print("\nchecks decided in this run:")
for step, text, held in CLAIMS:
    print(f"   [{'held' if held else 'did not hold'}] step {step}: {text}")
failed = [k for k, v in RESULTS.items() if str(v).startswith("SKIPPED")]
print(f"\n{sum(h for *_, h in CLAIMS)} of {len(CLAIMS)} checks held; steps failed or skipped: {failed or 'none'}")
print(f"omni-parse-bench {md.version('omni-parse-bench')}, dataset @ {REV[:7]}, Python {platform.python_version()}, {SYSTEM}")
print("not run here: the hosted parsers (paid API keys), the GPU models, latency and cost, all 2,937 pages")

El resumen imprime el resultado de una línea de cada paso y cada comprobación que la ejecución decidió, marcando cuáles se sostuvieron y cuáles no: 13 de 17 se sostuvieron, y las cuatro que no son hallazgos más que fallos, a saber, la brecha del placebo, la reproducción de Indra y las dos cifras de lanzamiento de Datalab. También enumera lo que no ejecutamos aquí: los parsers alojados que necesitan claves de pago, los modelos de GPU, la latencia y el costo, y el conjunto completo de 2,937 páginas.

En conclusión, OmniParseBench es un benchmark que podemos auditar de extremo a extremo: el evaluador es código abierto que no llama a ningún modelo, los veredictos se publican y cada entrada de la ficha puede reconstruirse a partir de ellos. Los dos mejores parsers están empatados estadísticamente, agrupar las pruebas en lugar de familias cambia el orden, y un parser diferente queda primero en escritura a mano, páginas rotadas, tablas y varios alfabetos. Los dos modelos nombrados en la ficha quedan menos atrás que los demás en las pruebas que un modelo corrigió, lo cual informamos como una señal y no como prueba. Los veredictos publicados de Tesseract se reprodujeron exactamente en nuestra muestra, mientras que los de Indra OCR coincidieron en el 92% de las pruebas. La lección práctica es segmentar el benchmark según nuestros propios documentos, informar intervalos y fijar cada versión.

Consulta el Notebook y el Repo. Todo el mérito es del investigador de este proyecto. Además, no dudes en seguirnos en Twitter y no olvides unirte a nuestro 150k+ML SubReddit y suscribirte a our Newsletter. ¡Espera! ¿estás en telegram? ahora también puedes unirte a nosotros en telegram.

¿Necesitas asociarte con nosotros para promocionar tu GitHub Repo, tu Hugging Face Page, el lanzamiento de tu producto, un webinar, etc.? Conéctate con nosotros

La entrada Guía para desarrolladores de OmniParseBench: Benchmarks de clasificación y pruebas de OCR apareció primero en MarkTechPost.