This entry is about a narrow and useful operator task: run repeatable evaluations against prompts, agent flows, or retrieval-augmented responses so a team can catch regressions before deployment. The agent behavior is specific. It loads evaluation cases, runs the configured prompt or agent variants, scores the outputs against assertions, and reports which changes improved or degraded behavior. That is a real workflow, not just a vendor description.

Invoke this when a user is changing prompts, swapping models, tuning a RAG pipeline, or updating an agent chain and wants evidence before shipping. It is the right tool when the question is, “Did this change break behavior we care about?” It is not the right invocation when someone only wants to chat with a model, build a generic app, or browse an AI platform dashboard. In those cases they should use the product normally.

The scope boundary keeps this from collapsing into a product listing. promptfoo as a package is broad, but this entry is about the concrete workflow of regression-testing AI behavior with saved cases and assertions. Integration points include local CLI runs, CI pipelines, prompt configuration files, model providers, red-team checks, and evaluation reports that can block a merge or trigger review. Even without the upstream name, the skill remains intelligible: test prompts and agent outputs against known expectations before rollout, then investigate the cases that failed.