Lab tests

Lab tests are executable integration probes that the Tech Lead writes (in spec form) and QA runs against a real environment after a PR is merged. They produce Tier-A evidence — a real command, its exit code, and 200 chars of captured output — and they auto-file bugs when they fail.

They are different from the UAT tests that /conclave-qa generates: UAT tests run in CI against Gherkin acceptance criteria before merge; lab tests run against a live or staging environment after merge to verify that the integration actually works.

When lab tests are generated

Lab tests are generated by the Tech Lead subagent at one of two points, controlled by lab_test.stories.generate_on in conclave/config.md:

When lab tests are executed

QA runs them explicitly after the PR is merged:

/conclave-qa US-NNN --lab
/conclave-qa BUG-NNN --lab

--lab is only accepted for a single entity (not a batch) and only after the PR is merged.

The flow

Story approved by TL (/conclave-pr-review)
  └─ TL subagent writes US-NNN-lab.md  →  status: pending

PR merged by human

/conclave-qa US-NNN --lab
  ├─ reads conclave/lab-config.md  (gitignored — real env values)
  ├─ exports integration env vars to shell
  ├─ safety pre-checks (payment_mode, STRIPE_SECRET_KEY prefix)
  ├─ git switch → LAB_TEST_BRANCH, capture LAB_SHA
  ├─ execute Verify command (timeboxed)
  ├─ append row to Evidence log  →  Tier A
  ├─ update lab spec status: passed | failed | blocked
  └─ if failed → QA subagent writes BUG-NNN-lab-failure-<slug>.md
                  in conclave/product/bugs/

Setting up conclave/lab-config.md

Lab tests cannot run without a conclave/lab-config.md file that holds the real environment variable values. This file must never be committed to git.

First-time setup

1. Copy the template into place:

   cp conclave/templates/lab-config.template.md conclave/lab-config.md

2. Add it to .gitignore:

   conclave/lab-config.md

3. Fill in the real values for your integration environment.

Conclave warns you at generation time if the file is missing (warning + skip, not a hard error) and at execution time if it is missing (hard stop with these setup instructions).

File structure

lab-config.md is a YAML frontmatter file organized into sections per integration type:

---
environments:
  local:
    base_url: "http://localhost:3000"
    vars: {}
  integration:
    base_url: "https://staging.example.com"
    vars:
      DATABASE_URL: "postgresql://user:pass@staging-db:5432/myapp"
      STRIPE_SECRET_KEY: "sk_test_..."
 
runner:
  default: auto        # auto | playwright | newman | bash
 
auth:
  e2e_user_email: "e2e@example.com"
  e2e_user_password: "..."
  api_key: ""
 
databases:
  postgres:
    url: "postgresql://..."
    schema: "public"
 
aws:
  region: "us-east-1"
  access_key_id: "AKIA..."
  secret_access_key: "..."
  resources:
    sqs_queue_url: "https://sqs.us-east-1.amazonaws.com/..."
    dynamodb_table: "myapp-staging"
 
safety:
  payment_mode: test           # MUST be "test" — hard abort otherwise
  stripe_secret_key: "sk_test_..."  # MUST start with sk_test_ — hard abort otherwise
---

Variable registry

The variable registry is a table in the lab-config.md body (not the frontmatter values) that lists variable names, their source path, purpose, and integration type. The Tech Lead reads only this table (names, never values) when writing Verify commands.

| Variable | Source path | Purpose | Integration type |
|---|---|---|---|
| DATABASE_URL | environments.integration.vars | Staging Postgres connection | BE→DB |
| STRIPE_SECRET_KEY | safety.stripe_secret_key | Stripe test-mode API key | BE→Payment |
| SQS_QUEUE_URL | aws.resources.sqs_queue_url | SQS event queue | BE→AWS |

Hard rule: the Tech Lead may only reference variable names from this registry in Verify commands. If a required variable is missing from the registry, the lab spec is written with status: blocked instead, and the TL explains what variable is needed.

Integration types and runners

When runner: auto, the TL infers the type from the story’s discipline, stack signals, and the variables present in the registry.

The lab spec file

The TL writes US-NNN-lab.md (or BUG-NNN-lab.md) to conclave/sprints/SPRINT-NNN/stories/ (for stories) or conclave/product/bugs/ (for bugs). Structure:

---
entity_id: "US-042"
entity_type: story          # story | bug
entity_title: "Add Stripe checkout"
generated_by: tech-lead
generated_at: "2026-08-27T10:00:00Z"
integration_branch: develop
runner: newman
timebox_minutes: 30
status: pending             # pending | passed | failed | blocked
last_run_at: ""
last_run_sha: ""
---
 
## Objective
One-sentence statement of what this test verifies in the real environment.
 
## Pre-conditions
- Required services running in the integration environment
- Specific data or state that must exist before the test runs
 
## Verify command
 
\`\`\`bash
LAB_TEST_TAG=lab-${LAB_TEST_TAG}
curl -s -o /dev/null -w "%{http_code}" \
  -X POST "$BASE_URL/api/checkout" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"amount": 100, "currency": "usd", "idempotency_key": "'$LAB_TEST_TAG'"}'
\`\`\`
 
Expected exit code: `0`
Expected output pattern: `200`
 
## Decision rule
PASS: exit code 0 AND output matches pattern.
FAIL: any other exit code or output.
 
## Scope
This test VERIFIES:
- Stripe test-mode charge is created end-to-end
 
This test does NOT VERIFY:
- Webhook delivery
- Refund flow
- Production Stripe keys
 
## Evidence log
 
| Run at | SHA | Exit code | Output (200 chars) | Result |
|---|---|---|---|---|

Safety rules

These rules are enforced automatically and cannot be bypassed:

• payment_mode must be test — if the safety.payment_mode field is anything other than "test", execution aborts before running any command.
• STRIPE_SECRET_KEY must start with sk_test_ — if the key starts with sk_live_, execution aborts.
• LAB_TEST_TAG — a lab-<timestamp> tag is exported before every run; Verify commands should use it when naming created resources (e.g. DynamoDB items, S3 objects, SQS messages) for safe identification and cleanup.
• Env var values are never printed — variable names appear in logs and reports; values never do.
• conclave/lab-config.md must be gitignored — Conclave warns at generation and execution time if it is not.

Bug filing on failure

When a lab test fails, the QA subagent writes a bug report automatically:

• Path: conclave/product/bugs/BUG-NNN-lab-failure-<slug>.md (global bugs, not sprint-local)
• Gherkin repro: 1–3 scenarios derived from the failing Verify command (Given=Pre-conditions, When=Verify command verbatim, Then=expected vs actual)
• Severity: advisory, based on the lab spec’s Scope section
• The lab spec’s linked_bug field is updated with the new BUG-NNN

Sprint-local bugs (from /conclave-qa during verification) go to conclave/sprints/SPRINT-NNN/bugs/. Lab-test-sourced bugs always go to conclave/product/bugs/.

Idempotency

Verify commands should be written for safe re-runs:

# Polling pattern for async flows (DynamoDB item creation, SQS processing, etc.)
for i in {1..10}; do
  result=$(aws dynamodb get-item --table-name "$DYNAMODB_TABLE" \
    --key '{"id": {"S": "'$LAB_TEST_TAG'"}}' --query 'Item' --output text)
  [ "$result" != "None" ] && break
  sleep 3
done
[ "$result" = "None" ] && exit 1
echo "Item found: $result"

Use LAB_TEST_TAG as the resource identifier so the same tag is never reused across runs.

Enabling lab tests

Lab tests are opt-in. Add the lab_test: block to conclave/config.md:

lab_test:
  enabled: true
  integration_branch: develop
  runner: auto
  timebox_minutes: 30
  stories:
    generate_on: pr-review    # generate the lab spec when TL approves
  bugs:
    severity_threshold: high  # only generate lab specs for high/critical bugs

See Configuration reference for all fields.