QA overview

QA overview 문서는 OpenClaw 공식 문서(concepts/qa-e2e-automation)를 한국어로 정리한 가이드입니다. QA stack overview: qa-lab, qa-channel, repo-backed scenarios, live transport lanes, transport adapters, and reporting. 명령·설정 키·코드 예시는 공식 문서를 그대로 보존하며, 해석과 절차 안내는 한국어로 제공합니다. 최종 동작은 설치된 CLI 버전과 공식 원문을 확인하세요.

핵심 요약

QA stack overview: qa-lab, qa-channel, repo-backed scenarios, live transport lanes, transport adapters, and reporting.

한국어 가이드 범위: concepts/qa-e2e-automation 경로의 설정·명령·제약·예시를 학습용으로 재구성합니다.

문서 구성

공식 문서의 주요 섹션은 다음과 같습니다.

Command surface
Profile-backed qa run
Operator flow
Observability smokes
Matrix smoke lanes
Discord Mantis scenarios
Mantis Slack desktop and visual-task runners
Credential pool health check
Canonical scenario coverage
Discord, Slack, Telegram, and WhatsApp QA reference
Shared CLI flags
Telegram QA
Discord QA
Slack QA
WhatsApp QA
Convex credential pool
Repo-backed seeds
Provider mock lanes
Transport adapters
Adding a channel
Scenario helper names
Reporting
Related docs

상세 내용

본문

The private QA stack exercises OpenClaw in a realistic, channel-shaped way that a unit test cannot.

주요 항목:

extensions/qa-channel: synthetic message channel with DM, channel, thread,
extensions/qa-lab: debugger UI, QA bus, scenario profiles, and live
qa/: repo-backed seed assets for the kickoff task and baseline QA
Mantis: before/after live verification for bugs that

Command surface

Every QA flow runs under pnpm openclaw qa . Many have pnpm qa:* script aliases; both forms work.

Command	Purpose
`qa run`	Bundled QA self-check without `--qa-profile`; taxonomy-backed maturity profile runner with `--qa-profile smoke-ci`, `--qa-profile release`, or `--qa-profile all`.
`qa suite`	Run repo-backed scenarios against the QA gateway lane. `--runner multipass` uses a disposable Linux VM instead of the host.
`qa coverage`	Print the YAML scenario-coverage inventory (`--json` for machine output; `--match <query>` to find scenarios for a touched behavior; `--tools` for runtime tool fixture coverage).
`qa parity-report`	Compare two `qa-suite-summary.json` files for a model-axis parity gate, or use `--runtime-axis --token-efficiency` to write Codex-vs-OpenClaw runtime parity and token-efficiency reports.
`qa confidence-report`	Classify QA proof artifacts against a manifest into a zero-unknown confidence report.
`qa confidence-self-test`	Write seeded negative-control canaries proving the confidence gate detects drift.
`qa jsonl-replay`	Replay curated JSONL transcripts through the runtime parity replay harness.
`qa character-eval`	Run the character QA scenario across multiple live models with a judged report. See Reporting.
`qa manual`	Run a one-off prompt against the selected provider/model lane.
`qa ui`	Start the QA debugger UI and local QA bus (alias: `pnpm qa:lab:ui`).
`qa docker-build-image`	Build the prebaked QA Docker image.
`qa docker-scaffold`	Write a docker-compose scaffold for the QA dashboard + gateway lane.
`qa up`	Build the QA site, start the Docker-backed stack, print the URL (alias: `pnpm qa:lab:up`; `:fast` variant adds `--use-prebuilt-image --bind-ui-dist --skip-ui-build`).
`qa aimock`	Start only the AIMock provider server.
`qa mock-openai`	Start only the scenario-aware `mock-openai` provider server.
`qa credentials doctor` / `add` / `list` / `remove`	Manage the shared Convex credential pool.
`qa discord`	Live transport lane against a real private Discord guild channel.
`qa matrix`	QA Lab Matrix profiles against a disposable Tuwunel homeserver. See Matrix smoke lanes.
`qa slack`	Live transport lane against a real private Slack channel.
`qa telegram`	Live transport lane against a real private Telegram group.
`qa whatsapp`	Live transport lane against real WhatsApp Web accounts.
`qa mantis`	Before/after verification runner for live transport bugs, with Discord status-reactions evidence, Crabbox desktop/browser smoke, and Slack-in-VNC smoke. See Mantis and Mantis Slack Desktop Runbook.

Profile-backed qa run

Profile-backed qa run reads membership from taxonomy.yaml, then dispatches the resolved scenarios through qa suite. --surface and --category filter the selected profile instead of defining separate lanes. The resulting qa-evidence.json includes a profile scorecard summary with selected-category counts and missing coverage IDs; the individual evidence entries remain the source of truth for the tests, coverage roles, and results. Taxonomy feature coverage IDs are exact proof targets, not aliases: primary scenario coverage fulfills matching IDs, while secondary coverage stays advisory. Every coverage ID is exactly taxonomy-surface.feature, using the short surface ID from taxonomy.yaml. A scenario's separate surface field is an execution/reporting label (예를 들어, channel or runtime-tool); it does not define taxonomy ownership.

pnpm openclaw qa run \
  --qa-profile smoke-ci \
  --category channels.conversation-routing-and-delivery \
  --provider-mode mock-openai \
  --output-dir .artifacts/qa-e2e/smoke-ci-profile-dispatch

pnpm openclaw --profile work qa run --qa-profile smoke-ci

Operator flow

The current QA operator flow is a two-pane QA site:

주요 항목:

Left: Gateway dashboard (Control UI) with the agent.
Right: QA Lab, showing the Slack-ish transcript and scenario plan.

pnpm qa:lab:up

pnpm openclaw qa docker-build-image
pnpm qa:lab:build
pnpm qa:lab:up:fast
pnpm qa:lab:watch

Observability smokes

Observability QA stays source-checkout only. The npm tarball intentionally omits QA Lab (and qa-channel), so package Docker release lanes do not run qa commands. Run these from a built source checkout when changing diagnostics instrumentation.

Alias	What it runs
`pnpm qa:otel:smoke`	Local OpenTelemetry receiver plus the `otel-trace-smoke` scenario with `diagnostics-otel` enabled.
`pnpm qa:otel:collector-smoke`	Same lane behind a real OpenTelemetry Collector Docker container. Use it when changing endpoint wiring or collector/OTLP compatibility.
`pnpm qa:prometheus:smoke`	The `docker-prometheus-smoke` scenario with `diagnostics-prometheus` enabled.
`pnpm qa:observability:smoke`	`qa:otel:smoke` followed by `qa:prometheus:smoke`.
`pnpm qa:observability:collector-smoke`	`qa:otel:collector-smoke` followed by `qa:prometheus:smoke`.

Matrix smoke lanes

For a transport-real Matrix smoke lane that does not require model-provider credentials, run the release profile with the deterministic mock OpenAI provider:

Profile	Scenarios	Purpose
`all`	93	Complete catalog (default).
`release`	2	Release-critical channel baseline and live allowlist reload.
`fast`	12	Focused threading, reactions, approvals, policy, bot-gating, and encrypted-reply coverage.
`transport`	50	Threading, DM/room routing, autojoin, approvals, reactions, restarts, mention/allowlist policy, edits, and multi-actor ordering.
`media`	7	Image, generated-image, voice, attachment, unsupported-media, and encrypted-media coverage.
`e2ee-smoke`	8	Minimum encrypted reply, threading, bootstrap, recovery, restart, redaction, and failure coverage.
`e2ee-deep`	18	State-loss, backup, key recovery, device hygiene, and SAS/QR/DM verification.
`e2ee-cli`	9	`openclaw matrix encryption setup`, recovery-key, multi-account, gateway round-trip, and self-verification commands through the harness.

Flag	Default	Purpose
`--profile <profile>`	`all`	Select one of the profiles above.
`--scenario <id>`	-	Select one scenario; repeatable.
`--fail-fast`	off	Stop after the first failed check or scenario.
`--allow-failures`	off	Write artifacts without returning a failing exit code for scenario failures.
`--provider-mode <mode>`	`live-frontier`	Use `mock-openai` for deterministic dispatch or `live-frontier` for a live provider.
`--model <ref>`	provider default	Set the primary `provider/model` reference.
`--alt-model <ref>`	provider default	Set the alternate model used by scenarios that switch models.
`--fast`	off	Enable provider fast mode where supported.
`--output-dir <path>`	generated	Choose the report directory; relative paths resolve against `--repo-root`.
`--repo-root <path>`	current directory	Run from a neutral working directory.
`--sut-account <id>`	`sut`	Select the Matrix account id in the child gateway config.

pnpm openclaw qa matrix --provider-mode mock-openai --profile release

OPENCLAW_LIVE_OPENAI_KEY="${OPENAI_API_KEY}" \
  pnpm openclaw qa matrix --provider-mode live-frontier --profile release

Discord Mantis scenarios

Discord also has Mantis-only opt-in scenarios for bug reproduction. Use --scenario discord-status-reactions-tool-only for the explicit status reaction timeline, or --scenario discord-thread-reply-filepath-attachment to create a real Discord thread and verify that message.thread-reply preserves a filePath attachment. These scenarios stay out of the default live Discord lane because they are before/after repro probes rather than broad smoke coverage. The thread-attachment Mantis workflow can also add a logged-in Discord Web witness video when MANTIS_DISCORD_VIEWER_CHROME_PROFILE_DIR or MANTIS_DISCORD_VIEWER_CHROME_PROFILE_TGZ_B64 is configured in the QA environment. That viewer profile is only for visual capture; the pass/fail decision still comes from the Discord REST oracle.

pnpm openclaw qa discord
pnpm openclaw qa slack
pnpm openclaw qa telegram
pnpm openclaw qa whatsapp

Mantis Slack desktop and visual-task runners

For a full Slack desktop VM run with VNC rescue, run:

pnpm openclaw qa mantis slack-desktop-smoke \
  --gateway-setup \
  --scenario slack-canary \
  --keep-lease

pnpm openclaw qa mantis slack-desktop-smoke \
  --approval-checkpoints \
  --credential-source convex \
  --credential-role maintainer

pnpm openclaw qa mantis visual-task \
  --browser-url https://example.net \
  --expect-text "Example Domain" \
  --vision-model openai/gpt-5.6-luna

Credential pool health check

Before using pooled live credentials, run:

pnpm openclaw qa credentials doctor

Canonical scenario coverage

The root taxonomy.yaml defines semantic coverage IDs. Scenario YAML files under qa/scenarios/ map each scenario to those IDs and own execution metadata: channel is the only channel requirement, and profiles declare named run membership. The channel driver is an interchangeable run-level implementation choice. TypeScript runners query that catalog; they do not maintain parallel scenario or coverage inventories.

pnpm openclaw qa suite --runner multipass --scenario channel-chat-baseline

Discord, Slack, Telegram, and WhatsApp QA reference

The Matrix adapter uses the disposable Docker-backed lane documented above. Discord, Slack, Telegram, and WhatsApp run against pre-existing real transports, so their reference lives here.

Shared CLI flags

These lanes register through extensions/qa-lab/src/live-transports/shared/live-transport-cli.ts and accept the same flags:

Flag	Default	Description
`--scenario <id>`	-	Run only this scenario. Repeatable.
`--output-dir <path>`	`<repo>/.artifacts/qa-e2e/<transport>-<timestamp>`	Where reports, summaries, evidence, transport-specific artifacts, and the output log are written. Relative paths resolve against `--repo-root`.
`--repo-root <path>`	`process.cwd()`	Repository root when invoking from a neutral cwd.
`--sut-account <id>`	`sut`	Temporary account id inside the QA gateway config.
`--provider-mode <mode>`	`live-frontier`	`mock-openai`, `aimock`, or `live-frontier`.
`--model <ref>` / `--alt-model <ref>`	provider default	Primary/alternate model refs.
`--fast`	off	Provider fast mode where supported.
`--credential-source <env\|convex>`	`env`	See Convex credential pool.
`--credential-role <maintainer\|ci>`	`ci` in CI, `maintainer` otherwise	Role used when `--credential-source convex`.
`--allow-failures`	off	Write artifacts without returning a failing exit code when scenarios fail.

Telegram QA

Targets one real private Telegram group with two distinct bots (driver + SUT). The SUT bot must have a Telegram username; bot-to-bot observation works best when both bots have Bot-to-Bot Communication Mode enabled in @BotFather.

주요 항목:

OPENCLAW_QA_TELEGRAM_GROUP_ID - numeric chat id (string).
OPENCLAW_QA_TELEGRAM_DRIVER_BOT_TOKEN
OPENCLAW_QA_TELEGRAM_SUT_BOT_TOKEN
channel-canary
channel-mention-gating
telegram-help-command
telegram-commands-command
telegram-tools-compact-command
telegram-whoami-command
telegram-status-command
telegram-repeated-command-authorization
telegram-other-bot-command-gating
telegram-context-command
telegram-current-session-status-tool
telegram-tool-only-usage-footer
telegram-reply-chain-exact-marker
telegram-stream-final-single-message
telegram-long-final-reuses-preview
telegram-long-final-three-chunks
qa-suite-report.md
qa-suite-summary.json
qa-evidence.json - evidence entries for the live transport checks,

pnpm openclaw qa telegram

OPENCLAW_QA_CREDENTIAL_SOURCE=convex \
pnpm test:docker:npm-telegram-live

Discord QA

Targets one real private Discord guild channel with two bots: a driver bot controlled by the harness and a SUT bot started by the child OpenClaw gateway through the bundled Discord plugin. Verifies channel mention handling, that the SUT bot has registered the native /help command with Discord, and opt-in Mantis evidence scenarios.

주요 항목:

OPENCLAW_QA_DISCORD_GUILD_ID
OPENCLAW_QA_DISCORD_CHANNEL_ID
OPENCLAW_QA_DISCORD_DRIVER_BOT_TOKEN
OPENCLAW_QA_DISCORD_SUT_BOT_TOKEN
OPENCLAW_QA_DISCORD_SUT_APPLICATION_ID - must match the SUT bot user id
OPENCLAW_QA_DISCORD_VOICE_CHANNEL_ID selects the voice/stage channel for
discord-canary
discord-mention-gating
discord-native-help-command-registration
discord-voice-autojoin - opt-in voice scenario. Runs by itself, enables
discord-status-reactions-tool-only - opt-in Mantis scenario. Runs by
discord-thread-reply-filepath-attachment - opt-in Mantis scenario; see
qa-suite-report.md
qa-suite-summary.json
qa-evidence.json - evidence entries for the live transport checks.
discord-qa-reaction-timelines.json and

pnpm openclaw qa discord

pnpm openclaw qa discord \
  --scenario discord-voice-autojoin \
  --provider-mode mock-openai

pnpm openclaw qa discord \
  --scenario discord-status-reactions-tool-only \
  --provider-mode live-frontier \
  --model openai/gpt-5.6-luna \
  --alt-model openai/gpt-5.6-luna \
  --fast

Slack QA

Targets one real private Slack channel with two distinct bots: a driver bot controlled by the harness and a SUT bot started by the child OpenClaw gateway through the bundled Slack plugin.

주요 항목:

OPENCLAW_QA_SLACK_CHANNEL_ID
OPENCLAW_QA_SLACK_DRIVER_BOT_TOKEN
OPENCLAW_QA_SLACK_SUT_BOT_TOKEN
OPENCLAW_QA_SLACK_SUT_APP_TOKEN
OPENCLAW_QA_SLACK_APPROVAL_CHECKPOINT_DIR enables visual approval
OPENCLAW_QA_SLACK_APPROVAL_CHECKPOINT_TIMEOUT_MS overrides the checkpoint
thread-follow-up
thread-isolation
slack-canary
slack-mention-gating
slack-allowlist-block
slack-channel-disabled-warning - opt-in real-Slack probe that confirms a
slack-top-level-reply-shape
slack-restart-resume
slack-progress-commentary-true, slack-progress-commentary-false,
slack-reaction-glyph-native - opt-in live message-tool reaction scenario.
slack-chart-presentation-native - opt-in portable chart scenario that
slack-table-presentation-native - opt-in portable table scenario that
slack-table-invalid-blocks-fallback - opt-in direct-transport scenario
slack-approval-exec-native - opt-in native Slack exec approval scenario.
slack-approval-plugin-native - opt-in native Slack plugin approval
slack-codex-approval-exec-native - opt-in Codex Guardian command approval
slack-codex-approval-plugin-native - opt-in Codex Guardian file approval
qa-suite-report.md
qa-suite-summary.json

pnpm openclaw qa slack

{
  "display_information": {
    "name": "OpenClaw QA Driver",
    "description": "Test driver bot for OpenClaw QA Slack live lane"
  },
  "features": {
    "bot_user": {
      "display_name": "OpenClaw QA Driver",
      "always_online": true
    }
  },
  "oauth_config": {
    "scopes": {
      "bot": ["chat:write", "channels:history", "groups:history", "users:read"]
    }
  },
  "settings": {
    "socket_mode_enabled": false
  }
}

{
  "display_information": {
    "name": "OpenClaw QA SUT",
    "description": "OpenClaw QA SUT connector for OpenClaw"
  },
  "features": {
    "bot_user": {
      "display_name": "OpenClaw QA SUT",
      "always_online": true
    },
    "app_home": {
      "home_tab_enabled": true,
      "messages_tab_enabled": true,
      "messages_tab_read_only_enabled": false
    }
  },
  "oauth_config": {
    "scopes": {
      "bot": [
        "app_mentions:read",
        "assistant:write",
        "channels:history",
        "channels:read",
        "chat:write",
        "commands",
        "emoji:read",
        "files:read",
        "files:write",
        "groups:history",
        "groups:read",
        "im:history",
        "im:read",
        "im:write",
        "mpim:history",
        "mpim:read",
        "mpim:write",
        "pins:read",
        "pins:write",
        "usergroups:read",
        "users:read"
      ]
    }
  },
  "settings": {
    "socket_mode_enabled": true,
    "event_subscriptions": {
      "bot_events": [
        "app_home_opened",
        "app_mention",
        "channel_rename",
        "member_joined_channel",
        "member_left_channel",
        "message.channels",
        "message.groups",
        "message.im",
        "message.mpim",
        "pin_added",
        "pin_removed"
      ]
    }
  }
}

/invite @OpenClaw QA Driver
/invite @OpenClaw QA SUT

WhatsApp QA

Targets two dedicated WhatsApp Web accounts: a driver account controlled by the harness and a SUT account started by the child OpenClaw gateway through the bundled WhatsApp plugin.

주요 항목:

OPENCLAW_QA_WHATSAPP_DRIVER_PHONE_E164
OPENCLAW_QA_WHATSAPP_SUT_PHONE_E164
OPENCLAW_QA_WHATSAPP_DRIVER_AUTH_ARCHIVE_BASE64
OPENCLAW_QA_WHATSAPP_SUT_AUTH_ARCHIVE_BASE64
OPENCLAW_QA_WHATSAPP_GROUP_JID enables group scenarios such as
Baseline and group gating: whatsapp-canary, whatsapp-pairing-block,
Native commands: whatsapp-help-command, whatsapp-status-command,
Reply and final-output behavior: whatsapp-tool-only-usage-footer,
User-path message actions: whatsapp-agent-message-action-react starts
Group fanout: whatsapp-broadcast-group-fanout starts from one mentioned
Group activation: whatsapp-group-activation-always changes a real group
Inbound media and structured messages: whatsapp-inbound-image-caption,
Direct Gateway contract probes: whatsapp-outbound-media-matrix,
Access-control coverage: whatsapp-access-control-dm-open,
Native approvals: whatsapp-approval-exec-deny-native,
Status reactions: whatsapp-status-reactions,
qa-suite-report.md
qa-suite-summary.json
qa-evidence.json - evidence entries for the live transport checks.

pnpm openclaw qa whatsapp

Convex credential pool

Discord, Slack, Telegram, and WhatsApp lanes can lease credentials from a shared Convex pool instead of reading the env vars above. Pass --credential-source convex (or set OPENCLAW_QA_CREDENTIAL_SOURCE=convex); QA Lab acquires an exclusive lease, heartbeats it for the duration of the run, and releases it on shutdown. Pool kinds are "discord", "slack", "telegram", and "whatsapp".

주요 항목:

Discord (kind: "discord"): `{ guildId: string, channelId: string,
Telegram (kind: "telegram"): `{ groupId: string, driverToken: string,
Telegram real user (kind: "telegram-user"): `{ groupId: string, sutToken:
WhatsApp (kind: "whatsapp"): `{ driverPhoneE164: string, sutPhoneE164:

Repo-backed seeds

These are intentionally in git so the QA plan is visible to both humans and the agent.

주요 항목:

qa/scenarios/index.yaml
qa/scenarios/<theme>/*.yaml
top-level title
scenario metadata
optional category, capability, lane, and risk metadata in scenario
docs and code refs in scenario
optional plugin requirements in scenario
optional gateway config patch in scenario
executable top-level flow for flow scenarios, or
DM and channel chat
thread behavior
message action lifecycle
cron callbacks
memory recall
model switching
subagent handoff
repo-reading and docs-reading
one small build task such as Lobster Invaders

Provider mock lanes

qa suite has two local provider mock lanes:

주요 항목:

mock-openai is the scenario-aware OpenClaw mock. It remains the default
aimock starts an AIMock-backed provider server for experimental

Transport adapters

qa-lab owns a generic transport seam for YAML QA scenarios. qa-channel is the synthetic default. crabline starts local provider-shaped servers and runs OpenClaw's normal channel plugins against them. live is reserved for real provider credentials and external channels.

주요 항목:

qa-lab owns generic scenario execution, worker concurrency, artifact
The transport adapter owns gateway config, readiness, inbound and outbound
YAML scenario files under qa/scenarios/ define the test run; qa-lab

Adding a channel

Adding a channel to the YAML QA system requires the channel implementation plus a scenario pack that exercises the channel contract. For smoke CI coverage, add the matching Crabline local provider server and expose it through the crabline driver.

주요 항목:

the openclaw qa command root
suite startup and teardown
worker concurrency
artifact writing
report generation
scenario execution
compatibility aliases for older qa-channel scenarios
how openclaw qa <runner> is mounted beneath the shared qa root
how the gateway is configured for that transport
how readiness is checked
how inbound events are injected
how outbound messages are observed
how transcripts and normalized transport state are exposed
how transport-backed actions are executed
how transport-specific reset or cleanup is handled
If behavior can be expressed once in qa-lab, put it in qa-lab.
If behavior depends on one channel transport, keep it in that runner
If a scenario needs a new capability that more than one channel can use,
If a behavior is only meaningful for one transport, keep the scenario

Scenario helper names

Preferred generic helpers for new scenarios:

주요 항목:

waitForTransportReady
waitForChannelReady
injectInboundMessage
injectOutboundMessage
waitForTransportOutboundMessage
waitForChannelOutboundMessage
waitForNoTransportOutbound
getTransportSnapshot
readTransportMessage
readTransportTranscript
formatTransportTranscript
resetTransport

Reporting

qa-lab exports a Markdown protocol report from the observed bus timeline. The report should answer:

주요 항목:

What worked
What failed
What stayed blocked
What follow-up scenarios are worth adding

pnpm openclaw qa character-eval \
  --model openai/gpt-5.6-luna,thinking=medium,fast \
  --model openai/gpt-5.2,thinking=xhigh \
  --model openai/gpt-5,thinking=xhigh \
  --model anthropic/claude-opus-4-8,thinking=high \
  --model anthropic/claude-sonnet-4-6,thinking=high \
  --model zai/glm-5.1,thinking=high \
  --model moonshot/kimi-k2.5,thinking=high \
  --model google/gemini-3.1-pro-preview,thinking=high \
  --judge-model openai/gpt-5.6-sol,thinking=xhigh,fast \
  --judge-model anthropic/claude-opus-4-8,thinking=high \
  --blind-judge-models \
  --concurrency 16 \
  --judge-concurrency 16

주요 항목:

Maturity scorecard
Personal agent benchmark pack
QA Channel
Testing
Dashboard

실습 체크리스트

공식 문서와 로컬 버전을 대조합니다: https://docs.openclaw.ai/concepts/qa-e2e-automation
관련 CLI는 openclaw --help 및 하위 명령 --help로 옵션을 확인합니다.
설정 변경 시 openclaw config / openclaw doctor로 유효성을 검사합니다.
Gateway·채널·플러그인 변경 후에는 필요 시 Gateway를 재시작합니다.

자주 쓰는 명령·설정 예시

pnpm openclaw qa run \
  --qa-profile smoke-ci \
  --category channels.conversation-routing-and-delivery \
  --provider-mode mock-openai \
  --output-dir .artifacts/qa-e2e/smoke-ci-profile-dispatch

pnpm openclaw --profile work qa run --qa-profile smoke-ci

pnpm qa:lab:up

pnpm openclaw qa docker-build-image
pnpm qa:lab:build
pnpm qa:lab:up:fast
pnpm qa:lab:watch

pnpm openclaw qa matrix --provider-mode mock-openai --profile release

OPENCLAW_LIVE_OPENAI_KEY="${OPENAI_API_KEY}" \
  pnpm openclaw qa matrix --provider-mode live-frontier --profile release

QA overview

핵심 요약

QA stack overview: qa-lab, qa-channel, repo-backed scenarios, live transport lanes, transport adapters, and reporting.

한국어 가이드 범위: concepts/qa-e2e-automation 경로의 설정·명령·제약·예시를 학습용으로 재구성합니다.

문서 구성

공식 문서의 주요 섹션은 다음과 같습니다.

Command surface
Profile-backed qa run
Operator flow
Observability smokes
Matrix smoke lanes
Discord Mantis scenarios
Mantis Slack desktop and visual-task runners
Credential pool health check
Canonical scenario coverage
Discord, Slack, Telegram, and WhatsApp QA reference
Shared CLI flags
Telegram QA
Discord QA
Slack QA
WhatsApp QA
Convex credential pool
Repo-backed seeds
Provider mock lanes
Transport adapters
Adding a channel
Scenario helper names
Reporting
Related docs

상세 내용

본문

The private QA stack exercises OpenClaw in a realistic, channel-shaped way that a unit test cannot.

주요 항목:

extensions/qa-channel: synthetic message channel with DM, channel, thread,
extensions/qa-lab: debugger UI, QA bus, scenario profiles, and live
qa/: repo-backed seed assets for the kickoff task and baseline QA
Mantis: before/after live verification for bugs that

Command surface

Every QA flow runs under pnpm openclaw qa . Many have pnpm qa:* script aliases; both forms work.

Command	Purpose
`qa run`	Bundled QA self-check without `--qa-profile`; taxonomy-backed maturity profile runner with `--qa-profile smoke-ci`, `--qa-profile release`, or `--qa-profile all`.
`qa suite`	Run repo-backed scenarios against the QA gateway lane. `--runner multipass` uses a disposable Linux VM instead of the host.
`qa coverage`	Print the YAML scenario-coverage inventory (`--json` for machine output; `--match <query>` to find scenarios for a touched behavior; `--tools` for runtime tool fixture coverage).
`qa parity-report`	Compare two `qa-suite-summary.json` files for a model-axis parity gate, or use `--runtime-axis --token-efficiency` to write Codex-vs-OpenClaw runtime parity and token-efficiency reports.
`qa confidence-report`	Classify QA proof artifacts against a manifest into a zero-unknown confidence report.
`qa confidence-self-test`	Write seeded negative-control canaries proving the confidence gate detects drift.
`qa jsonl-replay`	Replay curated JSONL transcripts through the runtime parity replay harness.
`qa character-eval`	Run the character QA scenario across multiple live models with a judged report. See Reporting.
`qa manual`	Run a one-off prompt against the selected provider/model lane.
`qa ui`	Start the QA debugger UI and local QA bus (alias: `pnpm qa:lab:ui`).
`qa docker-build-image`	Build the prebaked QA Docker image.
`qa docker-scaffold`	Write a docker-compose scaffold for the QA dashboard + gateway lane.
`qa up`	Build the QA site, start the Docker-backed stack, print the URL (alias: `pnpm qa:lab:up`; `:fast` variant adds `--use-prebuilt-image --bind-ui-dist --skip-ui-build`).
`qa aimock`	Start only the AIMock provider server.
`qa mock-openai`	Start only the scenario-aware `mock-openai` provider server.
`qa credentials doctor` / `add` / `list` / `remove`	Manage the shared Convex credential pool.
`qa discord`	Live transport lane against a real private Discord guild channel.
`qa matrix`	QA Lab Matrix profiles against a disposable Tuwunel homeserver. See Matrix smoke lanes.
`qa slack`	Live transport lane against a real private Slack channel.
`qa telegram`	Live transport lane against a real private Telegram group.
`qa whatsapp`	Live transport lane against real WhatsApp Web accounts.
`qa mantis`	Before/after verification runner for live transport bugs, with Discord status-reactions evidence, Crabbox desktop/browser smoke, and Slack-in-VNC smoke. See Mantis and Mantis Slack Desktop Runbook.

Profile-backed qa run

pnpm openclaw qa run \
  --qa-profile smoke-ci \
  --category channels.conversation-routing-and-delivery \
  --provider-mode mock-openai \
  --output-dir .artifacts/qa-e2e/smoke-ci-profile-dispatch

pnpm openclaw --profile work qa run --qa-profile smoke-ci

Operator flow

The current QA operator flow is a two-pane QA site:

주요 항목:

Left: Gateway dashboard (Control UI) with the agent.
Right: QA Lab, showing the Slack-ish transcript and scenario plan.

pnpm qa:lab:up

pnpm openclaw qa docker-build-image
pnpm qa:lab:build
pnpm qa:lab:up:fast
pnpm qa:lab:watch

Observability smokes

Alias	What it runs
`pnpm qa:otel:smoke`	Local OpenTelemetry receiver plus the `otel-trace-smoke` scenario with `diagnostics-otel` enabled.
`pnpm qa:otel:collector-smoke`	Same lane behind a real OpenTelemetry Collector Docker container. Use it when changing endpoint wiring or collector/OTLP compatibility.
`pnpm qa:prometheus:smoke`	The `docker-prometheus-smoke` scenario with `diagnostics-prometheus` enabled.
`pnpm qa:observability:smoke`	`qa:otel:smoke` followed by `qa:prometheus:smoke`.
`pnpm qa:observability:collector-smoke`	`qa:otel:collector-smoke` followed by `qa:prometheus:smoke`.

Matrix smoke lanes

For a transport-real Matrix smoke lane that does not require model-provider credentials, run the release profile with the deterministic mock OpenAI provider:

Profile	Scenarios	Purpose
`all`	93	Complete catalog (default).
`release`	2	Release-critical channel baseline and live allowlist reload.
`fast`	12	Focused threading, reactions, approvals, policy, bot-gating, and encrypted-reply coverage.
`transport`	50	Threading, DM/room routing, autojoin, approvals, reactions, restarts, mention/allowlist policy, edits, and multi-actor ordering.
`media`	7	Image, generated-image, voice, attachment, unsupported-media, and encrypted-media coverage.
`e2ee-smoke`	8	Minimum encrypted reply, threading, bootstrap, recovery, restart, redaction, and failure coverage.
`e2ee-deep`	18	State-loss, backup, key recovery, device hygiene, and SAS/QR/DM verification.
`e2ee-cli`	9	`openclaw matrix encryption setup`, recovery-key, multi-account, gateway round-trip, and self-verification commands through the harness.

Flag	Default	Purpose
`--profile <profile>`	`all`	Select one of the profiles above.
`--scenario <id>`	-	Select one scenario; repeatable.
`--fail-fast`	off	Stop after the first failed check or scenario.
`--allow-failures`	off	Write artifacts without returning a failing exit code for scenario failures.
`--provider-mode <mode>`	`live-frontier`	Use `mock-openai` for deterministic dispatch or `live-frontier` for a live provider.
`--model <ref>`	provider default	Set the primary `provider/model` reference.
`--alt-model <ref>`	provider default	Set the alternate model used by scenarios that switch models.
`--fast`	off	Enable provider fast mode where supported.
`--output-dir <path>`	generated	Choose the report directory; relative paths resolve against `--repo-root`.
`--repo-root <path>`	current directory	Run from a neutral working directory.
`--sut-account <id>`	`sut`	Select the Matrix account id in the child gateway config.

pnpm openclaw qa matrix --provider-mode mock-openai --profile release

OPENCLAW_LIVE_OPENAI_KEY="${OPENAI_API_KEY}" \
  pnpm openclaw qa matrix --provider-mode live-frontier --profile release

Discord Mantis scenarios

pnpm openclaw qa discord
pnpm openclaw qa slack
pnpm openclaw qa telegram
pnpm openclaw qa whatsapp

Mantis Slack desktop and visual-task runners

For a full Slack desktop VM run with VNC rescue, run:

pnpm openclaw qa mantis slack-desktop-smoke \
  --gateway-setup \
  --scenario slack-canary \
  --keep-lease

pnpm openclaw qa mantis slack-desktop-smoke \
  --approval-checkpoints \
  --credential-source convex \
  --credential-role maintainer

pnpm openclaw qa mantis visual-task \
  --browser-url https://example.net \
  --expect-text "Example Domain" \
  --vision-model openai/gpt-5.6-luna

Credential pool health check

Before using pooled live credentials, run:

pnpm openclaw qa credentials doctor

Canonical scenario coverage

pnpm openclaw qa suite --runner multipass --scenario channel-chat-baseline

Discord, Slack, Telegram, and WhatsApp QA reference

The Matrix adapter uses the disposable Docker-backed lane documented above. Discord, Slack, Telegram, and WhatsApp run against pre-existing real transports, so their reference lives here.

Shared CLI flags

These lanes register through extensions/qa-lab/src/live-transports/shared/live-transport-cli.ts and accept the same flags:

Flag	Default	Description
`--scenario <id>`	-	Run only this scenario. Repeatable.
`--output-dir <path>`	`<repo>/.artifacts/qa-e2e/<transport>-<timestamp>`	Where reports, summaries, evidence, transport-specific artifacts, and the output log are written. Relative paths resolve against `--repo-root`.
`--repo-root <path>`	`process.cwd()`	Repository root when invoking from a neutral cwd.
`--sut-account <id>`	`sut`	Temporary account id inside the QA gateway config.
`--provider-mode <mode>`	`live-frontier`	`mock-openai`, `aimock`, or `live-frontier`.
`--model <ref>` / `--alt-model <ref>`	provider default	Primary/alternate model refs.
`--fast`	off	Provider fast mode where supported.
`--credential-source <env\|convex>`	`env`	See Convex credential pool.
`--credential-role <maintainer\|ci>`	`ci` in CI, `maintainer` otherwise	Role used when `--credential-source convex`.
`--allow-failures`	off	Write artifacts without returning a failing exit code when scenarios fail.

Telegram QA

주요 항목:

OPENCLAW_QA_TELEGRAM_GROUP_ID - numeric chat id (string).
OPENCLAW_QA_TELEGRAM_DRIVER_BOT_TOKEN
OPENCLAW_QA_TELEGRAM_SUT_BOT_TOKEN
channel-canary
channel-mention-gating
telegram-help-command
telegram-commands-command
telegram-tools-compact-command
telegram-whoami-command
telegram-status-command
telegram-repeated-command-authorization
telegram-other-bot-command-gating
telegram-context-command
telegram-current-session-status-tool
telegram-tool-only-usage-footer
telegram-reply-chain-exact-marker
telegram-stream-final-single-message
telegram-long-final-reuses-preview
telegram-long-final-three-chunks
qa-suite-report.md
qa-suite-summary.json
qa-evidence.json - evidence entries for the live transport checks,

pnpm openclaw qa telegram

OPENCLAW_QA_CREDENTIAL_SOURCE=convex \
pnpm test:docker:npm-telegram-live

Discord QA

주요 항목:

OPENCLAW_QA_DISCORD_GUILD_ID
OPENCLAW_QA_DISCORD_CHANNEL_ID
OPENCLAW_QA_DISCORD_DRIVER_BOT_TOKEN
OPENCLAW_QA_DISCORD_SUT_BOT_TOKEN
OPENCLAW_QA_DISCORD_SUT_APPLICATION_ID - must match the SUT bot user id
OPENCLAW_QA_DISCORD_VOICE_CHANNEL_ID selects the voice/stage channel for
discord-canary
discord-mention-gating
discord-native-help-command-registration
discord-voice-autojoin - opt-in voice scenario. Runs by itself, enables
discord-status-reactions-tool-only - opt-in Mantis scenario. Runs by
discord-thread-reply-filepath-attachment - opt-in Mantis scenario; see
qa-suite-report.md
qa-suite-summary.json
qa-evidence.json - evidence entries for the live transport checks.
discord-qa-reaction-timelines.json and

pnpm openclaw qa discord

pnpm openclaw qa discord \
  --scenario discord-voice-autojoin \
  --provider-mode mock-openai

pnpm openclaw qa discord \
  --scenario discord-status-reactions-tool-only \
  --provider-mode live-frontier \
  --model openai/gpt-5.6-luna \
  --alt-model openai/gpt-5.6-luna \
  --fast

Slack QA

Targets one real private Slack channel with two distinct bots: a driver bot controlled by the harness and a SUT bot started by the child OpenClaw gateway through the bundled Slack plugin.

주요 항목:

OPENCLAW_QA_SLACK_CHANNEL_ID
OPENCLAW_QA_SLACK_DRIVER_BOT_TOKEN
OPENCLAW_QA_SLACK_SUT_BOT_TOKEN
OPENCLAW_QA_SLACK_SUT_APP_TOKEN
OPENCLAW_QA_SLACK_APPROVAL_CHECKPOINT_DIR enables visual approval
OPENCLAW_QA_SLACK_APPROVAL_CHECKPOINT_TIMEOUT_MS overrides the checkpoint
thread-follow-up
thread-isolation
slack-canary
slack-mention-gating
slack-allowlist-block
slack-channel-disabled-warning - opt-in real-Slack probe that confirms a
slack-top-level-reply-shape
slack-restart-resume
slack-progress-commentary-true, slack-progress-commentary-false,
slack-reaction-glyph-native - opt-in live message-tool reaction scenario.
slack-chart-presentation-native - opt-in portable chart scenario that
slack-table-presentation-native - opt-in portable table scenario that
slack-table-invalid-blocks-fallback - opt-in direct-transport scenario
slack-approval-exec-native - opt-in native Slack exec approval scenario.
slack-approval-plugin-native - opt-in native Slack plugin approval
slack-codex-approval-exec-native - opt-in Codex Guardian command approval
slack-codex-approval-plugin-native - opt-in Codex Guardian file approval
qa-suite-report.md
qa-suite-summary.json

pnpm openclaw qa slack

{
  "display_information": {
    "name": "OpenClaw QA Driver",
    "description": "Test driver bot for OpenClaw QA Slack live lane"
  },
  "features": {
    "bot_user": {
      "display_name": "OpenClaw QA Driver",
      "always_online": true
    }
  },
  "oauth_config": {
    "scopes": {
      "bot": ["chat:write", "channels:history", "groups:history", "users:read"]
    }
  },
  "settings": {
    "socket_mode_enabled": false
  }
}

{
  "display_information": {
    "name": "OpenClaw QA SUT",
    "description": "OpenClaw QA SUT connector for OpenClaw"
  },
  "features": {
    "bot_user": {
      "display_name": "OpenClaw QA SUT",
      "always_online": true
    },
    "app_home": {
      "home_tab_enabled": true,
      "messages_tab_enabled": true,
      "messages_tab_read_only_enabled": false
    }
  },
  "oauth_config": {
    "scopes": {
      "bot": [
        "app_mentions:read",
        "assistant:write",
        "channels:history",
        "channels:read",
        "chat:write",
        "commands",
        "emoji:read",
        "files:read",
        "files:write",
        "groups:history",
        "groups:read",
        "im:history",
        "im:read",
        "im:write",
        "mpim:history",
        "mpim:read",
        "mpim:write",
        "pins:read",
        "pins:write",
        "usergroups:read",
        "users:read"
      ]
    }
  },
  "settings": {
    "socket_mode_enabled": true,
    "event_subscriptions": {
      "bot_events": [
        "app_home_opened",
        "app_mention",
        "channel_rename",
        "member_joined_channel",
        "member_left_channel",
        "message.channels",
        "message.groups",
        "message.im",
        "message.mpim",
        "pin_added",
        "pin_removed"
      ]
    }
  }
}

/invite @OpenClaw QA Driver
/invite @OpenClaw QA SUT

WhatsApp QA

Targets two dedicated WhatsApp Web accounts: a driver account controlled by the harness and a SUT account started by the child OpenClaw gateway through the bundled WhatsApp plugin.

주요 항목:

OPENCLAW_QA_WHATSAPP_DRIVER_PHONE_E164
OPENCLAW_QA_WHATSAPP_SUT_PHONE_E164
OPENCLAW_QA_WHATSAPP_DRIVER_AUTH_ARCHIVE_BASE64
OPENCLAW_QA_WHATSAPP_SUT_AUTH_ARCHIVE_BASE64
OPENCLAW_QA_WHATSAPP_GROUP_JID enables group scenarios such as
Baseline and group gating: whatsapp-canary, whatsapp-pairing-block,
Native commands: whatsapp-help-command, whatsapp-status-command,
Reply and final-output behavior: whatsapp-tool-only-usage-footer,
User-path message actions: whatsapp-agent-message-action-react starts
Group fanout: whatsapp-broadcast-group-fanout starts from one mentioned
Group activation: whatsapp-group-activation-always changes a real group
Inbound media and structured messages: whatsapp-inbound-image-caption,
Direct Gateway contract probes: whatsapp-outbound-media-matrix,
Access-control coverage: whatsapp-access-control-dm-open,
Native approvals: whatsapp-approval-exec-deny-native,
Status reactions: whatsapp-status-reactions,
qa-suite-report.md
qa-suite-summary.json
qa-evidence.json - evidence entries for the live transport checks.

pnpm openclaw qa whatsapp

Convex credential pool

주요 항목:

Discord (kind: "discord"): `{ guildId: string, channelId: string,
Telegram (kind: "telegram"): `{ groupId: string, driverToken: string,
Telegram real user (kind: "telegram-user"): `{ groupId: string, sutToken:
WhatsApp (kind: "whatsapp"): `{ driverPhoneE164: string, sutPhoneE164:

Repo-backed seeds

These are intentionally in git so the QA plan is visible to both humans and the agent.

주요 항목:

qa/scenarios/index.yaml
qa/scenarios/<theme>/*.yaml
top-level title
scenario metadata
optional category, capability, lane, and risk metadata in scenario
docs and code refs in scenario
optional plugin requirements in scenario
optional gateway config patch in scenario
executable top-level flow for flow scenarios, or
DM and channel chat
thread behavior
message action lifecycle
cron callbacks
memory recall
model switching
subagent handoff
repo-reading and docs-reading
one small build task such as Lobster Invaders

Provider mock lanes

qa suite has two local provider mock lanes:

주요 항목:

mock-openai is the scenario-aware OpenClaw mock. It remains the default
aimock starts an AIMock-backed provider server for experimental

Transport adapters

주요 항목:

qa-lab owns generic scenario execution, worker concurrency, artifact
The transport adapter owns gateway config, readiness, inbound and outbound
YAML scenario files under qa/scenarios/ define the test run; qa-lab

Adding a channel

주요 항목:

the openclaw qa command root
suite startup and teardown
worker concurrency
artifact writing
report generation
scenario execution
compatibility aliases for older qa-channel scenarios
how openclaw qa <runner> is mounted beneath the shared qa root
how the gateway is configured for that transport
how readiness is checked
how inbound events are injected
how outbound messages are observed
how transcripts and normalized transport state are exposed
how transport-backed actions are executed
how transport-specific reset or cleanup is handled
If behavior can be expressed once in qa-lab, put it in qa-lab.
If behavior depends on one channel transport, keep it in that runner
If a scenario needs a new capability that more than one channel can use,
If a behavior is only meaningful for one transport, keep the scenario

Scenario helper names

Preferred generic helpers for new scenarios:

주요 항목:

waitForTransportReady
waitForChannelReady
injectInboundMessage
injectOutboundMessage
waitForTransportOutboundMessage
waitForChannelOutboundMessage
waitForNoTransportOutbound
getTransportSnapshot
readTransportMessage
readTransportTranscript
formatTransportTranscript
resetTransport

Reporting

qa-lab exports a Markdown protocol report from the observed bus timeline. The report should answer:

주요 항목:

What worked
What failed
What stayed blocked
What follow-up scenarios are worth adding

pnpm openclaw qa character-eval \
  --model openai/gpt-5.6-luna,thinking=medium,fast \
  --model openai/gpt-5.2,thinking=xhigh \
  --model openai/gpt-5,thinking=xhigh \
  --model anthropic/claude-opus-4-8,thinking=high \
  --model anthropic/claude-sonnet-4-6,thinking=high \
  --model zai/glm-5.1,thinking=high \
  --model moonshot/kimi-k2.5,thinking=high \
  --model google/gemini-3.1-pro-preview,thinking=high \
  --judge-model openai/gpt-5.6-sol,thinking=xhigh,fast \
  --judge-model anthropic/claude-opus-4-8,thinking=high \
  --blind-judge-models \
  --concurrency 16 \
  --judge-concurrency 16

주요 항목:

Maturity scorecard
Personal agent benchmark pack
QA Channel
Testing
Dashboard

실습 체크리스트

공식 문서와 로컬 버전을 대조합니다: https://docs.openclaw.ai/concepts/qa-e2e-automation
관련 CLI는 openclaw --help 및 하위 명령 --help로 옵션을 확인합니다.
설정 변경 시 openclaw config / openclaw doctor로 유효성을 검사합니다.
Gateway·채널·플러그인 변경 후에는 필요 시 Gateway를 재시작합니다.

자주 쓰는 명령·설정 예시

pnpm openclaw qa run \
  --qa-profile smoke-ci \
  --category channels.conversation-routing-and-delivery \
  --provider-mode mock-openai \
  --output-dir .artifacts/qa-e2e/smoke-ci-profile-dispatch

pnpm openclaw --profile work qa run --qa-profile smoke-ci

pnpm qa:lab:up

pnpm openclaw qa docker-build-image
pnpm qa:lab:build
pnpm qa:lab:up:fast
pnpm qa:lab:watch

pnpm openclaw qa matrix --provider-mode mock-openai --profile release

OPENCLAW_LIVE_OPENAI_KEY="${OPENAI_API_KEY}" \
  pnpm openclaw qa matrix --provider-mode live-frontier --profile release

QA overview

핵심 요약

문서 구성

상세 내용

본문

Command surface

Profile-backed qa run

Operator flow

Observability smokes

Matrix smoke lanes

Discord Mantis scenarios

Mantis Slack desktop and visual-task runners

Credential pool health check

Canonical scenario coverage

Discord, Slack, Telegram, and WhatsApp QA reference

Shared CLI flags

Telegram QA

Discord QA

Slack QA

WhatsApp QA

Convex credential pool

Repo-backed seeds

Provider mock lanes

Transport adapters

Adding a channel

Scenario helper names

Reporting

Related docs

실습 체크리스트

자주 쓰는 명령·설정 예시

관련 링크

관련 가이드

QA overview

핵심 요약

문서 구성

상세 내용

본문

Command surface

Profile-backed qa run

Operator flow

Observability smokes

Matrix smoke lanes

Discord Mantis scenarios

Mantis Slack desktop and visual-task runners

Credential pool health check

Canonical scenario coverage

Discord, Slack, Telegram, and WhatsApp QA reference

Shared CLI flags

Telegram QA

Discord QA

Slack QA

WhatsApp QA

Convex credential pool

Repo-backed seeds

Provider mock lanes

Transport adapters

Adding a channel

Scenario helper names

Reporting

Related docs

실습 체크리스트

자주 쓰는 명령·설정 예시

관련 링크

관련 가이드