Skill Detail

Run reproducible document-parser evaluations with the OmniDocBench skill

Validate parser predictions, run OmniDocBench evaluations, and report formula, table, text, and reading-order scores with runtime diagnostics.

Data Extraction & TransformationMulti-Framework
Data Extraction & Transformation Multi-Framework Security Reviewed
โญ 2.1k GitHub stars
COPY SKILL INSTRUCTIONS (OPTIONAL)
npx skills add agentskillexchange/skills --skill run-reproducible-document-parser-evaluations-with-omnidocbench-skill Copy
Uses the third-party skills CLI, not an ASE-owned installer. Check your agent’s compatibility. This copies instructions; complete the upstream tool setup below separately.
At a glance
Tools required
Git; shell-capable agent; Docker with accessible daemon or Python 3.10 source environment; ground-truth JSON and page Markdown predictions; writable output storage; TeX Live/CJK, ImageMagick and Ghostscript for CDM.
Install & setup
Clone the complete upstream repository using git clone https://github.com/opendatalab/OmniDocBench.git, enter OmniDocBench, and check out ea2d138c7d2e185feb71ef8014e97d7206820b34. Keep skills/SKILL.md with skills/scripts/ and the evaluator source; configure your agent to read the skill using its own skill-loading mechanism. Follow the pinned README Docker setup; its documented image is ghcr.io/zeng-weijun/omnidocbench-eval:repro-ubuntu2204. Supply existing ground-truth JSON, page Markdown predictions, and a separate output directory. For a source environment, use Python 3.10 and pip install -e . from the repository root; CDM also needs the documented TeX Live/CJK, ImageMagick and Ghostscript setup. Generate configuration using python skills/scripts/generate_end2end_config.py –gt /absolute/path/OmniDocBench.json –pred /absolute/path/predictions –out configs/custom_end2end.yaml –workers 2. Run python pdf_validation.py –config configs/custom_end2end.yaml in that configured runtime. Inspect results using python skills/scripts/parse_results.py /absolute/path/results –prefix YOUR_RESULT_PREFIX –pred /absolute/path/predictions. Use paths as seen by the chosen runtime and read-only input mounts. These are setup instructions; no evaluation was run during discovery.
Author
OpenDataLab
Publisher
Organization
Last updated
Oct 29, 2026
Quick brief

Use the upstream OmniDocBench evaluation helper when an agent must produce a reproducible quality report for existing document-parser predictions. Supply ground-truth JSON, a directory of page-level Markdown predictions, an output directory, the evaluation mode, and whether formula CDM is required. Invoke it when a parser upgrade or a difficult table, formula, or reading-order failure needs a measured comparison with runtime evidence.

How it works

What this skill actually does

The agent checks input paths, filename alignment, missing or empty prediction files, available memory, and Docker access. It generates an isolated end2end configuration, mounts ground truth and predictions read-only, and writes results into a separate output directory. The documented entrypoint is python pdf_validation.py --config configs/custom_end2end.yaml. Use the bundled skills/scripts/generate_end2end_config.py helper to prepare configuration and skills/scripts/parse_results.py to inspect results. For a constrained 4 CPU / 8 GB host, the upstream skill recommends two workers, reducing to one if necessary.

The deliverable combines Overall, text edit distance, formula CDM, table TEDS, and reading-order measurements with metric JSON, run summaries, stage execution records, and runtime environment records. The operator inspects page coverage, timeout counts, errors, and exceptions before accepting comparisons. If CDM is disabled, label that omission and avoid presenting the report as a complete formula-inclusive evaluation. Specify a result prefix when parsing a directory containing several runs.

The scope is evaluation of supplied predictions and investigation of scoring failures. Keep parser inference, model deployment, and downstream ingestion outside this task. This is useful for agents that can operate a shell across runtimes: the upstream skill depends on Python and Docker or a configured source environment, with no required agent SDK or MCP server. CDM requires TeX Live/CJK, ImageMagick PDF support, and Ghostscript; follow the documented Docker setup where available. Record the evaluator commit and ground-truth version so comparisons remain interpretable. The upstream benchmark dataset is restricted to research and noncommercial use; check its terms before choosing evaluation data.