A/B test agent skills and MCP changes with Caliper
Use Caliper to run real agent tasks with and without a skill, MCP server, or rule change so reliability and token cost are measurable.
npx skills add agentskillexchange/skills --skill ab-test-agent-skills-mcp-changes-with-caliper
pipx install caliper-eval for direct runs, or add the agent skill with npx skills@latest add edonadei/caliper; then create an .eval.yaml, run caliper run ... --k 3, run an ablated control with --ablate, and compare results with caliper compare.Use Caliper when a team needs to know whether an agent skill, MCP server, prompt rule, or model change actually improves task outcomes. The operator writes or generates an evaluation YAML, runs the real agent multiple times, runs an ablated control without the skill or tool, and compares success rates, task outcomes, token cost, and run history across Claude Code, Codex, Pi, or Hermes.
What this skill actually does
Invoke this instead of relying on a one-off manual demo when the question is whether a skill should stay installed, ship to users, or survive a model/runtime change. The scope boundary is repeatable evaluation and ablation of agent skills, MCPs, and rules. It is not a generic benchmark suite, observability platform, or prompt-writing framework.
Inputs and prerequisites: Python 3.10+, Caliper CLI, target agent runtime such as Claude Code, Codex, Pi, or Hermes, skills or MCP servers under test.
Setup notes: Install the CLI with pipx install caliper-eval for direct runs, or add the agent skill with npx skills@latest add edonadei/caliper ; then create an .eval.yaml , run caliper run … –k 3 , run an ablated control with –ablate , and compare results with caliper compare .
Source and verification boundary: use https://github.com/edonadei/caliper as the canonical reference before running the workflow; keep commands, API calls, CLI usage, and generated outputs reviewable against that upstream source.
Framework fit: publish this as a Multi-Framework workflow only when the operator can invoke the documented toolchain directly, rather than treating the upstream project as a generic product listing.