Skip to Content
Commands/conclave-qa

/conclave-qa US-NNN [US-NNN …] [—lab]

Verify one or more user stories in status: review against their acceptance criteria — or, with --lab, execute a post-merge lab test.

/conclave-qa US-001                   # single story — identical to previous behaviour
/conclave-qa US-001 US-002 US-003     # three stories in parallel, each on the integration branch
/conclave-qa US-001 --lab             # execute US-001's lab test (post-merge, single story only)
/conclave-qa BUG-004 --lab            # execute BUG-004's lab test

--lab is only accepted for a single story or bug ID, and only after the PR is already merged. It does not run Gherkin verification — it runs the Tech Lead’s lab spec instead.

When this finishes, each story has moved to one of four states:

  • verified — QA passed, awaiting Tech Lead PR approval. Happens when peer_pr_review.required: true.
  • done — QA passed and there is no separate TL gate. Happens when peer_pr_review.required: false.
  • review (pending UAT) — UAT artifacts were generated and pushed and nothing has failed, but a mobile checklist is awaiting a human, or was left incomplete. Not a defect.
  • review (with blockers) — QA found failures — in the Gherkin scenarios, the DoD, or the generated UAT tests’ CI run. The dev (or tester) fixes, and QA re-runs.

This is one of the two structural Scrum gates Conclave enforces (along with Sprint Planning). It cannot be skipped by any profile. QA does not approve the PR itself — code-level approval belongs to the Tech Lead via /conclave-pr-review. QA’s verdict goes into the verification report and a PR comment.

Multi-story behaviour

When more than one story ID is passed, the same upfront-validation → concurrent-batches-of-≤-3 → failure-isolation → final-summary-table pattern applies as in /conclave-dev. Each QA subagent runs fully independently on the integration branch, including UAT generation, CI wait, and PR comment. A final summary table shows the verdict for every story:

| Story  | PR                           | Verdict             |
|--------|------------------------------|---------------------|
| US-001 | https://github.com/…/pull/42 | ✓ passed            |
| US-002 | https://github.com/…/pull/43 | ✗ blocked (2 items) |
| US-003 | https://github.com/…/pull/44 | ⏳ pending_uat       |

What it does

  1. Resolves the workspace and confirms qa_verification.required: true (always true — refuses if a malformed config says otherwise).
  2. Resolves the story — must be status: review. Reads repo.integration_branch from config.md (defaults to develop).
  3. Switches to the integration branch (develop or the configured repo.integration_branch). Captures the PR URL for this story’s feature branch (feat/US-NNN-<slug>) via gh pr list — the feature branch does not need to be checked out. Verification is anchored to the integrated state of the codebase, not the isolated feature branch.
  4. Loads context in parallel: config, DoD, story, acceptance file, conclave/team/testing-environments.md, the verification-report/UAT-report templates, and PR metadata (reviews, CI status) if gh is available. If testing-environments.md is missing or every row is still TBD, UAT is skipped for this run — QA verifies exactly as it always has.
  5. Generates UAT test artifacts, when UAT is enabled, based on the story’s discipline:
    • frontend / multi → a Playwright spec (tests/uat/US-NNN.spec.ts), one test case per Gherkin scenario.
    • backend / multi → merges this story’s endpoints into the single, evolving tests/uat/api-collection.postman_collection.json, run via Newman — never overwrites another story’s requests.
    • mobile → a manual functional checklist (tests/uat/US-NNN-UAT.md) for a human tester to execute and record a verdict into. No CI wait for mobile.
    • If no CI job runs tests/uat/ yet, proposes the minimal addition and confirms with the human via AskUserQuestion before writing it.
  6. Pushes and waits for CI (skipped for mobile) — polls the CI run triggered by the pushed commit, up to ceremonies.qa_verification.ci_wait_timeout_minutes (default 20). A CI failure, or no run found / a timed-out wait, is treated exactly like a failing Gherkin scenario.
  7. Delegates to the QA subagent, which re-derives PASS/FAIL for every Gherkin scenario from first principles — adversarially, not by trusting the dev’s own claims — and folds in the UAT/CI outcome (or the current state of a mobile checklist on a later run).
  8. Writes outputs (below):
    • 8.1 — Passed: standard verification outputs (below).
    • 8.2 — Blocked: in addition to the standard ## QA blockers section, every reproducible defect that is linked to a specific story is written as a sprint bug file:
      • Path: conclave/sprints/<SPRINT_ID>/bugs/BUG-NNN-<slug>.md
      • Frontmatter includes: linked_story, linked_acceptance (file path), introduced_by_pr (the feature PR URL), severity (minor | major | critical), status: open.
      • A sprint cannot close if any file in conclave/sprints/<SPRINT_ID>/bugs/ has severity: critical and status other than done.

QA never executes the generated tests itself — the target repo’s own CI does, and never resolves, reads, or writes a secret value at any point.

What it produces

  • A ## Verification — <date> section appended to acceptance/AC-US-NNN.md — never overwrites previous verification runs, now including a UAT execution subsection (CI run + result, or mobile checklist state).
  • New UAT artifacts committed to the target repo under tests/uat/ (when UAT is enabled).
  • Story frontmatter update:
    • Verdict passed + peer_pr_review.required: true → status: verified.
    • Verdict passed + peer_pr_review.required: false → status: done directly.
    • Verdict pending_uat → stays status: review, with a ## QA pending section naming what a human needs to complete — worded as awaiting completion, not a defect.
    • Verdict blocked → stays status: review, with a ## QA blockers section listing each failing scenario/DoD item/CI failure and its reproduction or evidence (log excerpt + CI run URL).
  • conclave/sprints/<SPRINT_ID>/bugs/BUG-NNN-<slug>.md — one file per reproducible defect found (blocked verdict only).
  • A PR comment with the verdict (never a review/approval — gh pr review is never called from here).

Sprint bug files

When QA is blocked, each reproducible defect becomes a sprint bug file at:

conclave/sprints/SPRINT-NNN/bugs/BUG-NNN-<slug>.md

The file links back to the story, its acceptance criteria, and the PR that introduced the regression. severity: critical bugs block sprint closing. These files are separate from bugs reported manually via /conclave-bug report (which go to conclave/product/bugs/).

Lab test execution (--lab)

When --lab is passed, the standard Gherkin verification steps are replaced by lab test execution. This requires:

  • lab_test.enabled: true in conclave/config.md
  • A lab spec file (US-NNN-lab.md or BUG-NNN-lab.md) already written by the Tech Lead
  • conclave/lab-config.md present and gitignored (hard stop if missing)
  • The PR for this entity is already merged into lab_test.integration_branch

What --lab does

  1. Loads conclave/lab-config.md, reads the variable registry, confirms the file is gitignored.
  2. Switches to LAB_TEST_BRANCH, captures the commit SHA (LAB_SHA).
  3. Selects the integration environment vars (with user confirmation if integration.vars is empty, fallback to local.vars).
  4. Exports integration env vars to the shell (values never printed). Generates LAB_TEST_TAG=lab-<timestamp>.
  5. Safety pre-checks (hard abort if either fails):
    • payment_mode must equal test
    • STRIPE_SECRET_KEY must start with sk_test_
  6. Executes the Verify command from the lab spec, with the configured timebox. Captures exit code and 200 chars of output.
  7. Appends a row to the Evidence log in the lab spec (Tier A: command + exit code + output + SHA). Updates status, last_run_at, last_run_sha.
  8. If failed: dispatches the QA subagent to write a bug report. Bug is written to conclave/product/bugs/BUG-NNN-lab-failure-<slug>.md. Lab spec’s linked_bug is updated.
  9. Commits and pushes the lab spec (with updated Evidence log) to LAB_TEST_BRANCH.

A timeout during execution results in status: blocked (not failed). Blocked specs can be re-run after the timeout is resolved.

See Lab tests for the full reference including safety rules, idempotency patterns, and bug filing.

Guardrails

  • Only ever modifies conclave/, this story’s acceptance file, and tests/uat/US-NNN.spec.ts / the shared Postman collection+environment files / tests/uat/US-NNN-UAT.md — QA writes verification reports and UAT artifacts, QA does not fix code.
  • May propose (with human confirmation) adding the one CI job/step that runs tests/uat/ — no other pipeline changes.
  • Never deletes prior verification sections — the acceptance file is the story’s full audit trail.
  • Never overwrites another story’s requests in the shared Postman collection.
  • Never resolves, reads, or writes a secret value, anywhere, at any step — including during --lab execution (values are exported to the shell; QA never sees them).
  • Never runs gh pr review --approve or --request-changes. That is the Tech Lead’s call via /conclave-pr-review.
  • Never passes a story when any scenario fails, any required DoD item is unmet, or the generated UAT tests’ CI run failed or timed out.
  • Never conflates pending_uat with blocked.
  • Does not merge the PR, even when the verdict directly produces status: done.
  • --lab is single-entity, post-merge only — never accepted in a batch or before the PR merges.
  • --lab never prints env var values — only variable names appear in logs.

After it runs

  • Verdict passed, peer_pr_review.required: true — Tech Lead runs /conclave-pr-review US-NNN to approve and merge.
  • Verdict passed, peer_pr_review.required: false — no separate gate; the PR is ready to merge.
  • Verdict pending_uat — a human completes tests/uat/US-NNN-UAT.md (mobile checklist), then re-run /conclave-qa US-NNN.
  • Verdict blocked — dev fixes and pushes (or a mobile tester records a FAIL with notes), then re-run /conclave-qa US-NNN. Sprint bug files remain open until the fix is verified.

After the PR is merged (and lab_test.enabled: true):

  • Lab test passed — Tier A evidence is in the lab spec’s Evidence log. Done.
  • Lab test failed — a bug file was written to conclave/product/bugs/. Fix and re-run /conclave-qa US-NNN --lab.
  • Lab test blocked — the Verify command timed out or the lab spec had no runnable command. Check the lab spec and re-run after the issue is resolved.
Last updated on
/conclave-adr/conclave-pr-review