Benchmark CLI agents on autonomous LLM post-training with PostTrainBench
Run Claude Code, Codex CLI, Gemini CLI, or OpenCode through bounded H100 post-training tasks and compare how well each agent improves a base LLM.
npx skills add agentskillexchange/skills --skill benchmark-cli-agents-on-autonomous-llm-post-training-with-posttrainbench
Use PostTrainBench when the operator needs to evaluate whether CLI agents can conduct autonomous AI R&D: set up the benchmark container, provide the allowed credentials and compute budget, submit runs for selected agent scaffolds, and compare resulting model performance across reasoning, tool-use, math, health, and coding tasks. The upstream project is a source-backed benchmark for Claude Code, Codex CLI, Gemini CLI, and OpenCode, with an arXiv paper and explicit run rules for 10-hour H100 post-training jobs.
What this skill actually does
Invoke this instead of using a coding agent normally when the goal is controlled evaluation of agent-led post-training under benchmark constraints, not day-to-day software work. The scope boundary is narrow: prepare and run benchmark jobs, preserve contamination and final-model rules, collect scores, and inspect failure modes such as reward hacking or unauthorized data use. It is not a general LLM training framework, a hosted leaderboard card, or a broad CLI-agent listing.
Inputs and prerequisites: Python, apptainer, fuse-overlayfs, Hugging Face cache, H100 GPU access, currently HTCondor scheduler support, and credentials for the selected CLI agent scaffolds.
Setup notes: Clone https://github.com/aisa-group/PostTrainBench, install requirements including apptainer and fuse-overlayfs, build the standard container with bash containers/build_container.sh standard, download the Hugging Face cache with bash containers/download_hf_cache/download_hf_cache.sh, copy example.env to .env, set API keys and paths, then submit jobs with bash src/commit_utils/commit.sh.
Source and verification boundary: use http://posttrainbench.com/ as the canonical reference before running the workflow; keep commands, API calls, CLI usage, and generated outputs reviewable against that upstream source.
Framework fit: publish this as a Multi-Framework workflow only when the operator can invoke the documented toolchain directly, rather than treating the upstream project as a generic product listing.