Regression-test coding agents and skills with Coder Eval
Run sandboxed YAML evaluation suites against Claude Code, Codex, Gemini, OpenCode, or Pi agents, then gate CI on task scores and skill-trigger checks.
npx skills add agentskillexchange/skills --skill regression-test-coding-agents-and-skills-with-coder-eval
Use Coder Eval when an operator needs to prove that a coding agent, skill, MCP server, or CLI still works when a real agent uses it. The workflow is concrete: define YAML tasks, run an agent in a sandbox, score the produced files and commands, compare agent or prompt variants, and fail CI when quality regresses. Invoke this instead of a generic benchmark or observability tool when the target is agent behavior on your own repeatable coding tasks. The scope boundary is sandboxed, agent-agnostic evaluation and CI gating for coding-agent workflows, not a fixed leaderboard or plain Python library listing.
What this skill actually does
Inputs and prerequisites: Python 3.13+, uv or pip, at least one supported coding-agent runtime such as Claude Code, Codex, Gemini Antigravity, OpenCode, or Pi, model credentials or existing agent login, optional GitHub Actions runner.
Setup notes: Install with uv tool install coder-eval, or uv tool install coder-eval[codex,antigravity] for bundled agent extras. Define or reuse YAML tasks, run coder-eval plan, coder-eval run, and coder-eval report locally or through the documented GitHub Action.
Source and verification boundary: use https://coder-eval.com as the canonical reference before running the workflow; keep commands, API calls, CLI usage, and generated outputs reviewable against that upstream source.
Framework fit: publish this as a Multi-Framework workflow only when the operator can invoke the documented toolchain directly, rather than treating the upstream project as a generic product listing.