Run reproducible document-parser evaluations with the OmniDocBench skill
Validate parser predictions, run OmniDocBench evaluations, and report formula, table, text, and reading-order scores with runtime diagnostics.
npx skills add agentskillexchange/skills --skill run-reproducible-document-parser-evaluations-with-omnidocbench-skill
Use the upstream OmniDocBench evaluation helper when an agent must produce a reproducible quality report for existing document-parser predictions. Supply ground-truth JSON, a directory of page-level Markdown predictions, an output directory, the evaluation mode, and whether formula CDM is required. Invoke it when a parser upgrade or a difficult table, formula, or reading-order failure needs a measured comparison with runtime evidence.
What this skill actually does
The agent checks input paths, filename alignment, missing or empty prediction files, available memory, and Docker access. It generates an isolated end2end configuration, mounts ground truth and predictions read-only, and writes results into a separate output directory. The documented entrypoint is python pdf_validation.py --config configs/custom_end2end.yaml. Use the bundled skills/scripts/generate_end2end_config.py helper to prepare configuration and skills/scripts/parse_results.py to inspect results. For a constrained 4 CPU / 8 GB host, the upstream skill recommends two workers, reducing to one if necessary.
The deliverable combines Overall, text edit distance, formula CDM, table TEDS, and reading-order measurements with metric JSON, run summaries, stage execution records, and runtime environment records. The operator inspects page coverage, timeout counts, errors, and exceptions before accepting comparisons. If CDM is disabled, label that omission and avoid presenting the report as a complete formula-inclusive evaluation. Specify a result prefix when parsing a directory containing several runs.
The scope is evaluation of supplied predictions and investigation of scoring failures. Keep parser inference, model deployment, and downstream ingestion outside this task. This is useful for agents that can operate a shell across runtimes: the upstream skill depends on Python and Docker or a configured source environment, with no required agent SDK or MCP server. CDM requires TeX Live/CJK, ImageMagick PDF support, and Ghostscript; follow the documented Docker setup where available. Record the evaluator commit and ground-truth version so comparisons remain interpretable. The upstream benchmark dataset is restricted to research and noncommercial use; check its terms before choosing evaluation data.