Build a Smoke Test Skill That Catches Breakage Before Users Do

The fastest way to lose trust in an AI coding agent is to let it ship code without checking whether the basics still work. Logins break. Buttons stop submitting. Health endpoints return 500s. The deployment technically succeeds, but the product is already broken.

That is exactly where a smoke test skill earns its place. Instead of asking an agent to “test the app” in a vague way, you give it a narrow verification workflow: check the few paths that must work, collect evidence, and return a clear verdict fast enough to run every day.

This tutorial shows how to build that kind of skill. We will keep it intentionally simple: core flow coverage, lightweight artifacts, and a result format that humans can trust. If you have read our guide to product verification skills or our argument for evidence over vague confidence, this is the hands-on next step.

What is a smoke test skill?

A smoke test skill is a reusable verification skill that tells an agent how to confirm the product is not obviously broken. It is not a full regression suite. It is not exploratory QA. It is a fast release gate built around the question, does the app still do the few things users absolutely need?

In practice, that usually means checks like:

• the app loads and returns a healthy status code
• the login page renders
• a critical form can submit
• a core API endpoint responds
• the main dashboard shows expected content instead of an error state

The category overlap is useful here. A smoke test skill sits inside the broader verification family, but it often draws on patterns from observability and operational skills too. For example, a team may pair browser checks with log inspection from Agents Observe, or use a browser-first approach like Anthropic’s Playwright-based webapp testing skill when UI evidence matters.

When should you build one?

Build a smoke test skill when the same release questions keep coming up: “Did login survive this change?” “Can the checkout still load?” “Did the new deploy break the health endpoint?” If the answer requires the same small set of checks every time, it should become a skill.

Smoke test skills are especially useful for:

• pre-merge verification on risky pull requests
• post-deploy checks in staging or production
• handoffs between implementation agents and review agents
• small teams that need fast confidence before announcing a release

If you try to cram the entire QA strategy into one smoke test, it will become slow, flaky, and ignored. The goal is not total coverage. The goal is a dependable early warning system.

Choose the right flows

The hardest part is deciding what not to test. Good smoke test skills cover three to five critical paths, not twenty. Start by asking which failures would make the release feel immediately broken to a customer or operator.

A practical selection framework looks like this:

1. Entry point: Can the app or service start and respond?
2. Authentication: Can a user or service reach the protected area?
3. Primary task: Can the user do the one job the product is for?
4. Exit signal: Is there evidence the action completed successfully?

For a SaaS dashboard, that might be homepage load, login, dashboard render, and one create action. For an API product, it might be health endpoint, auth token exchange, one read request, and one write request in a safe test environment. For a content site, it may be homepage, article page, search, and form submission.

If you need inspiration from live ASE catalog entries, look at how focused skills narrow the problem. CodSpeed is useful because it targets one verification job well, and Hadrian works because it turns one class of failure into a repeatable check instead of a generic security promise.

Recommended file structure

Keep the skill compact. The agent should understand the workflow immediately, then read deeper references only when needed.

smoke-test-skill/
├── SKILL.md
├── config.example.json
├── references/
│   ├── target-flows.md
│   ├── environments.md
│   └── verdict-rules.md
├── scripts/
│   ├── smoke_test.ts
│   └── collect_artifacts.sh
└── artifacts/
    └── .gitkeep

SKILL.md should explain when to invoke the skill, what environments are safe, and how to report the outcome. The scripts folder should hold the actual checks. The references folder documents selectors, test accounts, environment caveats, and known weak points.

The SKILL.md pattern

Your description field should help the model recognize release-verification requests instantly.

---
name: smoke-test-verifier
description: >
  Use when verifying that a web app or API still works after code changes,
  before merges, after deploys, or during incident triage.
  Triggers on: "run smoke test", "verify staging", "did deploy break login",
  "check critical flow", "confirm app still works".
  Not for: exhaustive QA, visual polish review, or load testing.
---

# Smoke Test Verifier

## Workflow
1. Confirm the target environment and base URL.
2. Read references/target-flows.md and choose only the required checks.
3. Run scripts/smoke_test.ts or the equivalent test runner.
4. Save artifacts for every failure and any important pass evidence.
5. Return one verdict: PASS, FAIL, or BLOCKED.

## Output rules
- Include environment, commit or build reference, and timestamp.
- Summarize each check in one line.
- Link or attach artifacts for failures.
- Do not claim success without evidence.

That last line matters more than most teams realize. Verification skills fail when they start sounding confident instead of being confident.

Write checks for speed first

The best smoke tests are short and boring. They avoid fragile selectors, sleep statements, and sprawling setup. If a check routinely takes minutes or fails randomly, it does not belong in the smoke layer.

A lightweight Playwright-style example might look like this:

import { test, expect } from '@playwright/test';

test('homepage loads', async ({ page }) => {
  await page.goto(process.env.APP_URL!);
  await expect(page.locator('h1')).toContainText('Dashboard');
});

test('login form renders', async ({ page }) => {
  await page.goto(`${process.env.APP_URL}/login`);
  await expect(page.locator('input[type="email"]')).toBeVisible();
  await expect(page.locator('input[type="password"]')).toBeVisible();
});

test('health endpoint returns ok', async ({ request }) => {
  const res = await request.get(`${process.env.APP_URL}/api/health`);
  expect(res.ok()).toBeTruthy();
});

If your stack is API-first, use a shell script, Python, or the team test harness instead. The language does not matter much. Repeatability does.

Also keep environment safety explicit. If a flow creates data, direct it to a test tenant or make the script clean up after itself. Smoke test skills should reduce risk, not create production noise.

Evidence and verdicts

Every smoke test skill needs a strict reporting format. Without that, agents tend to over-explain or bury the real issue.

A solid output template looks like this:

Verdict: FAIL
Environment: staging
Build: 1f3a92c
Time: 2026-04-29T11:45:00Z

Checks:
- PASS: Homepage loaded in 1.2s
- PASS: /api/health returned 200
- FAIL: Login form submitted but redirected to 500 page

Evidence:
- Screenshot: artifacts/login-failure.png
- Console log: artifacts/browser-console.txt
- Response body: artifacts/login-response.txt

Next step:
- Inspect auth callback handler and recent env var changes

This format gives a human reviewer what they need immediately: status, scope, evidence, and likely direction. It also makes the skill easier to compose with incident or triage workflows later.

Common gotchas

This section is where the skill becomes genuinely useful. Add the failure modes your team keeps rediscovering.

• False passes from cached pages: force a fresh load or disable caching where appropriate.
• Wrong environment: staging and production URLs often look similar; make the base URL explicit in the report.
• Flaky selectors: prefer stable test IDs or API checks over fragile CSS paths.
• Sleeping instead of waiting: use reachability or locator waits, not arbitrary delays.
• Missing artifacts: on failure, always capture screenshot plus at least one machine-readable trace such as logs or response text.
• Too many checks: once the suite grows beyond a few fast checks, split it into smoke versus deeper regression skills.

These are exactly the kinds of details that separate a skill that sounds smart from a skill teams actually rely on.

A quick publish checklist

• ☐ Description field includes release-verification trigger phrases
• ☐ Only 3 to 5 critical flows are covered
• ☐ Scripts produce artifacts automatically on failure
• ☐ Verdict format is PASS, FAIL, or BLOCKED only
• ☐ Environment safety rules are documented
• ☐ Selectors, endpoints, and test credentials live outside the main SKILL.md body
• ☐ The full run finishes fast enough to use before real shipping decisions

Smoke test skills are not glamorous, but they save teams from deeply unglamorous outages. Build one well, and your agents stop guessing whether the release worked. They can prove it.

If you want to study more verification patterns, browse the ASE catalog entries for Playwright-based app verification, agent observability, and targeted checks like Hadrian. The common thread is not complexity. It is narrow scope, real evidence, and a result people can act on.