Testing

Testing 문서는 OpenClaw 공식 문서(help/testing)를 한국어로 정리한 가이드입니다. Testing kit: unit/e2e/live suites, Docker runners, and what each test covers 명령·설정 키·코드 예시는 공식 문서를 그대로 보존하며, 해석과 절차 안내는 한국어로 제공합니다. 최종 동작은 설치된 CLI 버전과 공식 원문을 확인하세요.

핵심 요약

Testing kit: unit/e2e/live suites, Docker runners, and what each test covers

한국어 가이드 범위: help/testing 경로의 설정·명령·제약·예시를 학습용으로 재구성합니다.

문서 구성

공식 문서의 주요 섹션은 다음과 같습니다.

빠른 시작
Test Temp Directories
Live and Docker/Parallels workflows
QA-specific runners
Shared Telegram credentials via Convex (v1)
Adding a channel to QA
Test suites (what runs where)
Unit / integration (default)
Stability (gateway)
E2E (repo aggregate)
E2E (gateway smoke)
E2E (Control UI mocked browser)
E2E: OpenShell backend smoke
Live (real providers + real models)
Which suite should I run?
Live (network-touching) tests
Docker runners (optional "works in Linux" checks)
Docs sanity
Offline regression (CI-safe)
Agent reliability evals (skills)
Contract tests (plugin and channel shape)
명령
Channel contracts
Provider contracts
When to run
Adding regressions (guidance)
관련 문서

상세 내용

본문

OpenClaw has three Vitest suites (unit/integration, e2e, live) plus Docker runners. This page covers what each suite covers, which command to run for a given workflow, how live tests discover credentials, and how to add regressions for real-world provider/model bugs.

주요 항목:

QA overview - architecture, command surface, scenario authoring, and Matrix profiles.
Maturity scorecard - how release QA evidence supports stability and LTS decisions.
QA channel - the synthetic transport plugin used by repo-backed scenarios.

빠른 시작

When you touch tests or want extra confidence:

주요 항목:

Full gate (expected before push): pnpm build && pnpm check && pnpm check:test-types && pnpm test
Faster local full-suite run on a roomy machine: pnpm test:max
Direct Vitest watch loop: pnpm test:watch
Direct file targeting routes plugin/channel paths too: pnpm test extensions/discord/src/monitor/message-handler.preflight.test.ts
Prefer targeted runs first when iterating on a single failure.
Docker-backed QA site: pnpm qa:lab:up
Linux VM-backed QA lane: pnpm openclaw qa suite --runner multipass --scenario channel-chat-baseline
Informational V8 coverage report: pnpm test:coverage
E2E suite: pnpm test:e2e

Test Temp Directories

Use the shared helpers in test/helpers/temp-dir.ts for test-owned temporary directories so ownership is explicit and cleanup stays in the test lifecycle:

const tempDirs = useAutoCleanupTempDirTracker(afterEach);

it("uses a temp workspace", () => {
  const workspace = tempDirs.make("openclaw-example-");
  // use workspace
});

// openclaw-temp-dir: allow verifies raw fs cleanup behavior
const workspace = fs.mkdtempSync(prefix);

Live and Docker/Parallels workflows

When debugging real providers/models (requires real creds):

주요 항목:

Live suite (models + gateway tool/image probes): pnpm test:live
Target one live file quietly: pnpm test:live -- src/agents/models.profiles.live.test.ts
Runtime performance reports: dispatch OpenClaw Performance with
Docker live model sweep: pnpm test:docker:live-models
Each selected model runs a text turn plus a small file-read-style probe.
CI coverage: daily OpenClaw Scheduled Live And E2E Checks and manual
For focused CI reruns, dispatch OpenClaw Live And E2E Checks (Reusable)
Add new high-signal provider secrets to scripts/ci-hydrate-live-auth.sh
Native Codex bound-chat smoke: pnpm test:docker:live-codex-bind
Runs a Docker live lane against the Codex app-server path, binds a
Codex app-server harness smoke: pnpm test:docker:live-codex-harness
Runs gateway agent turns through the plugin-owned Codex app-server
Codex on-demand install smoke: pnpm test:docker:codex-on-demand
Installs the packaged OpenClaw tarball in Docker, runs OpenAI API-key
Codex npm-plugin live package smoke: pnpm test:docker:live-codex-npm-plugin
Installs the candidate OpenClaw package and exact Codex plugin into Docker,
Its zero-retry medium-thinking follow-through turn must send progress, keep
Live plugin tool dependency smoke: pnpm test:docker:live-plugin-tool
Packs a fixture plugin with a real slugify dependency, installs it
OpenClaw rescue command smoke: pnpm test:live:system-agent-rescue-channel
Opt-in belt-and-suspenders check for the message-channel rescue command
OpenClaw first-run Docker smoke: pnpm test:docker:system-agent-first-run
Starts from an empty OpenClaw state dir and first proves the packaged
Moonshot/Kimi cost smoke: with MOONSHOT_API_KEY set, run

QA-specific runners

These commands sit beside the main test suites when you need QA-lab realism.

주요 항목:

pnpm openclaw qa suite
Runs repo-backed QA scenarios directly on the host.
Writes top-level qa-evidence.json, qa-suite-summary.json, and
When dispatched by pnpm openclaw qa run --qa-profile <profile>, embeds
Runs multiple selected scenarios in parallel by default with isolated
Exits non-zero when any scenario fails. Use --allow-failures for
Supports provider modes live-frontier, mock-openai, and aimock.
pnpm openclaw qa coverage --match <query>
Searches scenario IDs, titles, surfaces, coverage IDs, docs refs, code
Use this before a QA Lab run when you know the touched behavior or file
pnpm test:plugins:kitchen-sink-live
Runs the live OpenAI Kitchen Sink plugin gauntlet through QA Lab.
pnpm test:gateway:cpu-scenarios
Runs the gateway startup bench plus a small mock QA Lab scenario pack
Flags only sustained hot CPU observations by default (--cpu-core-warn,
Runs against built dist artifacts; run a build first when the checkout
pnpm openclaw qa suite --runner multipass
Runs the same QA suite inside a disposable Multipass Linux VM, keeping
Live runs forward the QA auth inputs practical for the guest:
Output dirs must stay under the repo root so the guest can write back
Writes the normal QA report + summary plus Multipass logs under
pnpm qa:lab:up
Starts the Docker-backed QA site for operator-style QA work.
pnpm test:docker:npm-onboard-channel-agent
Builds an npm tarball from the current checkout, installs it globally in

gh workflow run package-acceptance.yml --ref main \
  -f source=npm \
  -f package_spec=openclaw@beta \
  -f suite_profile=product \
  -f telegram_mode=mock-openai

gh workflow run package-acceptance.yml --ref main \
  -f source=url \
  -f package_url=https://registry.npmjs.org/openclaw/-/openclaw-VERSION.tgz \
  -f package_sha256=<sha256> \
  -f suite_profile=package

gh workflow run package-acceptance.yml --ref main \
  -f source=trusted-url \
  -f trusted_source_id=enterprise-artifactory \
  -f package_url=https://packages.example.internal:8443/artifactory/openclaw/openclaw-VERSION.tgz \
  -f package_sha256=<sha256> \
  -f suite_profile=package

gh workflow run package-acceptance.yml --ref main \
  -f source=artifact \
  -f artifact_run_id=<run-id> \
  -f artifact_name=<artifact-name> \
  -f suite_profile=smoke

Shared Telegram credentials via Convex (v1)

When --credential-source convex (or OPENCLAW_QA_CREDENTIAL_SOURCE=convex) is enabled for live transport QA, QA lab acquires an exclusive lease from a Convex-backed pool, heartbeats that lease while the lane is running, and releases the lease on shutdown. The section name predates Discord, Slack, and WhatsApp support; the lease contract is shared across kinds.

주요 항목:

OPENCLAW_QA_CONVEX_SITE_URL (for example https://your-deployment.convex.site)
One secret for the selected role:
OPENCLAW_QA_CONVEX_SECRET_MAINTAINER for maintainer
OPENCLAW_QA_CONVEX_SECRET_CI for ci
Credential role selection:
CLI: --credential-role maintainer|ci
Env default: OPENCLAW_QA_CREDENTIAL_ROLE (defaults to ci in CI, maintainer otherwise)
OPENCLAW_QA_CREDENTIAL_LEASE_TTL_MS (default 1200000)
OPENCLAW_QA_CREDENTIAL_HEARTBEAT_INTERVAL_MS (default 30000)
OPENCLAW_QA_CREDENTIAL_ACQUIRE_TIMEOUT_MS (default 90000)
OPENCLAW_QA_CREDENTIAL_HTTP_TIMEOUT_MS (default 15000)
OPENCLAW_QA_CONVEX_ENDPOINT_PREFIX (default /qa-credentials/v1)
OPENCLAW_QA_CREDENTIAL_OWNER_ID (optional trace id)
OPENCLAW_QA_ALLOW_INSECURE_HTTP=1 allows loopback http:// Convex URLs for local-only development.
POST /acquire
Request: { kind, ownerId, actorRole, leaseTtlMs, heartbeatIntervalMs }
Success: { status: "ok", credentialId, leaseToken, payload, leaseTtlMs?, heartbeatIntervalMs? }
Exhausted/retryable: { status: "error", code: "POOL_EXHAUSTED" | "NO_CREDENTIAL_AVAILABLE", ... }
POST /payload-chunk
Request: { kind, ownerId, actorRole, credentialId, leaseToken, index }
Success: { status: "ok", index, data }
POST /heartbeat
Request: { kind, ownerId, actorRole, credentialId, leaseToken, leaseTtlMs }
Success: { status: "ok" } (or empty 2xx)
POST /release

pnpm openclaw qa credentials doctor
pnpm openclaw qa credentials add --kind telegram --payload-file qa/telegram-credential.json
pnpm openclaw qa credentials list --kind telegram
pnpm openclaw qa credentials remove --credential-id <credential-id>

Adding a channel to QA

The architecture and scenario-helper names for new channel adapters live in QA overview - Adding a channel. The minimum bar: implement the transport runner on the shared qa-lab host seam, add an adapterFactory for shared scenarios, declare qaRunners in the plugin manifest, mount as openclaw qa , and author scenarios under qa/scenarios/.

Test suites (what runs where)

Think of the suites as "increasing realism" (and increasing flakiness/cost).

Unit / integration (default)

expand multi-project shards into per-project configs for parallel scheduling

주요 항목:

Command: pnpm test
Config: untargeted runs use the vitest.full-*.config.ts shard set and may
Files: core/unit inventories under src/**/*.test.ts,
Scope:
Pure unit tests
In-process integration tests (gateway auth, routing, tooling, parsing, config)
Deterministic regressions for known bugs
Expectations:
Runs in CI
No real keys required
Should be fast and stable
Resolver and public-surface loader tests must prove broad api.js and
Default test installs skip optional native Discord opus builds. Discord
Compare native opus performance in the libopus-wasm benchmark repo, not
Untargeted pnpm test runs thirteen smaller shard configs (core-unit-fast, core-unit-src, core-unit-security, core-unit-ui, core-unit-support, core-support-boundary, core-tooling, core-contracts, core-bundled, core-runtime, agentic, auto-reply, extensions) instead of one giant native root-project process. This cuts peak RSS on loaded machines and avoids auto-reply/plugin work starving unrelated suites.
pnpm test --watch still uses the native root vitest.config.ts project graph, because a multi-shard watch loop is not practical.
pnpm test, pnpm test:watch, and pnpm test:perf:imports route explicit file/directory targets through scoped lanes first, so pnpm test extensions/discord/src/monitor/message-handler.preflight.test.ts avoids paying the full root project startup tax.
pnpm test:changed expands changed git paths into cheap scoped lanes by default: direct test edits, sibling *.test.ts files, explicit source mappings, and local import-graph dependents. Config/setup/package edits do not broad-run tests unless you explicitly use OPENCLAW_TEST_CHANGED_BROAD=1 pnpm test:changed.
pnpm check:changed is the normal smart local check gate for narrow work. It classifies the diff into core, core tests, extensions, extension tests, apps, docs, release metadata, live Docker tooling, and tooling, then runs the matching typecheck, lint, and guard commands. It does not run Vitest tests; call pnpm test:changed or explicit pnpm test <target> for test proof. Release metadata-only version bumps run targeted version/config/root-dependency checks, with a guard that rejects package changes outside the top-level version field.
Live Docker ACP harness edits run focused checks: shell syntax for the live Docker auth scripts and a live Docker scheduler dry-run. package.json changes are included only when the diff is limited to scripts["test:docker:live-*"]; dependency, export, version, and other package-surface edits still use the broader guards.
Import-light unit tests from agents, commands, plugins, auto-reply helpers, plugin-sdk, and similar pure utility areas route through the unit-fast lane, which skips test/setup-openclaw-runtime.ts; stateful/runtime-heavy files stay on the existing lanes.
Selected plugin-sdk and commands helper source files also map changed-mode runs to explicit sibling tests in those light lanes, so helper edits avoid rerunning the full heavy suite for that directory.
auto-reply has dedicated buckets for top-level core helpers, top-level reply.* integration tests, and the src/auto-reply/reply/** subtree. CI further splits the reply subtree into agent-runner, dispatch, and commands/state-routing shards so one import-heavy bucket does not own the full Node tail.
Normal PR/main CI intentionally skips the bundled plugin batch sweep and release-only agentic-plugins shard. Full Release Validation dispatches the separate Plugin Prerelease child workflow for those plugin-heavy suites on release candidates.
When you change message-tool discovery inputs or compaction runtime

Stability (gateway)

주요 항목:

Command: pnpm test:stability:gateway
Config: test/vitest/vitest.gateway.config.ts, test/vitest/vitest.logging.config.ts, and test/vitest/vitest.infra.config.ts, each forced to one worker
Scope:
Starts a real loopback Gateway with diagnostics enabled by default
Drives synthetic gateway message, memory, and large-payload churn through the diagnostic event path
Queries diagnostics.stability over the Gateway WS RPC
Covers diagnostic stability bundle persistence helpers
Asserts the recorder remains bounded, synthetic RSS samples stay under the pressure budget, and per-session queue depths drain back to zero
Expectations:
CI-safe and keyless
Narrow lane for stability-regression follow-up, not a substitute for the full Gateway suite

E2E (repo aggregate)

주요 항목:

Command: pnpm test:e2e
Scope:
Runs the gateway smoke E2E lane
Runs the mocked Control UI browser E2E lane
Expectations:
CI-safe and keyless
Requires Playwright Chromium to be installed

E2E (gateway smoke)

주요 항목:

Command: pnpm test:e2e:gateway
Config: test/vitest/vitest.e2e.config.ts
Files: src/**/*.e2e.test.ts, test/**/*.e2e.test.ts, and bundled-plugin E2E tests under extensions/
Runtime defaults:
Uses Vitest threads with isolate: false, matching the rest of the repo.
Uses adaptive workers (CI: up to 2, local: 1 by default).
Runs in silent mode by default to reduce console I/O overhead.
Useful overrides:
OPENCLAW_E2E_WORKERS=<n> to force worker count (capped at 16).
OPENCLAW_E2E_VERBOSE=1 to re-enable verbose console output.
Scope:
Multi-instance gateway end-to-end behavior
WebSocket/HTTP surfaces, node pairing, and heavier networking
Expectations:
Runs in CI (when enabled in the pipeline)
No real keys required
More moving parts than unit tests (can be slower)

E2E (Control UI mocked browser)

주요 항목:

Command: pnpm test:ui:e2e
Config: test/vitest/vitest.ui-e2e.config.ts
Files: ui/src/**/*.e2e.test.ts
Scope:
Starts the Vite Control UI
Drives a real Chromium page through Playwright
Replaces the Gateway WebSocket with deterministic in-browser mocks
Expectations:
Runs in CI as part of pnpm test:e2e
No real Gateway, agents, or provider keys required
Browser dependency must be present (pnpm --dir ui exec playwright install chromium)

E2E: OpenShell backend smoke

주요 항목:

Command: pnpm test:e2e:openshell
File: extensions/openshell/src/backend.e2e.test.ts
Scope:
Reuses an active local OpenShell gateway
Creates a sandbox from a temporary local Dockerfile
Exercises OpenClaw's OpenShell backend over real sandbox ssh-config + SSH exec
Verifies remote-canonical filesystem behavior through the sandbox fs bridge
Expectations:
Opt-in only; not part of the default pnpm test:e2e run
Requires a local openshell CLI plus a working Docker daemon
Requires an active local OpenShell gateway and its config source
Uses isolated HOME / XDG_CONFIG_HOME, then destroys the test sandbox
Useful overrides:
OPENCLAW_E2E_OPENSHELL=1 to enable the test when running the broader e2e suite manually
OPENCLAW_E2E_OPENSHELL_COMMAND=/path/to/openshell to point at a non-default CLI binary or wrapper script
OPENCLAW_E2E_OPENSHELL_CONFIG_HOME=/path/to/config to expose the registered gateway config to the isolated test
OPENCLAW_E2E_OPENSHELL_HOST_IP=172.18.0.1 to override the Docker gateway IP used by the host policy fixture

Live (real providers + real models)

주요 항목:

Command: pnpm test:live
Config: test/vitest/vitest.live.config.ts
Files: src/**/*.live.test.ts, test/**/*.live.test.ts, and bundled-plugin live tests under extensions/
기본값: enabled by pnpm test:live (sets OPENCLAW_LIVE_TEST=1)
Scope:
"Does this provider/model actually work today with real creds?"
Catch provider format changes, tool-calling quirks, auth issues, and rate limit behavior
Expectations:
Not CI-stable by design (real networks, real provider policies, quotas, outages)
Costs money / uses rate limits
Prefer running narrowed subsets instead of "everything"
Live runs use already-exported API keys and staged auth profiles.
By default, live runs still isolate HOME and copy config/auth material into a temp test home so unit fixtures cannot mutate your real ~/.openclaw.
Set OPENCLAW_LIVE_USE_REAL_HOME=1 only when you intentionally need live tests to use your real home directory.
pnpm test:live defaults to a quieter mode: it keeps [live] ... progress output and mutes gateway bootstrap logs/Bonjour chatter. Set OPENCLAW_LIVE_TEST_QUIET=0 if you want the full startup logs back.
API key rotation (provider-specific): set *_API_KEYS with comma/semicolon format or *_API_KEY_1, *_API_KEY_2 (for example OPENAI_API_KEYS, ANTHROPIC_API_KEYS, GEMINI_API_KEYS) or per-live override via OPENCLAW_LIVE_*_KEY; tests retry on rate limit responses.
Progress/heartbeat output:
Live suites emit progress lines to stderr so long provider calls are visibly active even when Vitest console capture is quiet.
test/vitest/vitest.live.config.ts disables Vitest console interception so provider/gateway progress lines stream immediately during live runs.
Tune direct-model heartbeats with OPENCLAW_LIVE_HEARTBEAT_MS.
Tune gateway/probe heartbeats with OPENCLAW_LIVE_GATEWAY_HEARTBEAT_MS.

Which suite should I run?

주요 항목:

Editing logic/tests: run pnpm test (and pnpm test:coverage if you changed a lot)
Touching gateway networking / WS protocol / pairing: add pnpm test:e2e
Debugging "my bot is down" / provider-specific failures / tool calling: run a narrowed pnpm test:live

Live (network-touching) tests

For the live model matrix, CLI backend smokes, ACP smokes, Codex app-server harness, and all media-provider live tests (Deepgram, BytePlus, ComfyUI, image, music, video, media harness) - plus credential handling for live runs

주요 항목:

see Testing live suites. For the dedicated update and

Docker runners (optional "works in Linux" checks)

These Docker runners split into two buckets:

주요 항목:

Live-model runners: test:docker:live-models and test:docker:live-gateway run only their matching profile-key live file inside the repo Docker image (src/agents/models.profiles.live.test.ts and src/gateway/gateway-models.profiles.live.test.ts), mounting your local config dir, workspace, and optional profile env file. The matching local entrypoints are test:live:models-profiles and test:live:gateway-profiles.
Docker live runners keep their own practical caps where needed:
test:docker:all builds the live Docker image once via test:docker:live-build, packs OpenClaw once as an npm tarball through scripts/package-openclaw-for-docker.mjs, then builds/reuses two scripts/e2e/Dockerfile images. The bare image is only the Node/Git runner for install/update/plugin-dependency lanes; those lanes mount the prebuilt tarball. The functional image installs the same tarball into /app for built-app functionality lanes. Docker lane definitions live in scripts/lib/docker-e2e-scenarios.mjs; planner logic lives in scripts/lib/docker-e2e-plan.mjs; scripts/test-docker-all.mjs executes the selected plan. The aggregate uses a weighted local scheduler: OPENCLAW_DOCKER_ALL_PARALLELISM controls process slots, while resource caps keep heavy live, npm-install, and multi-service lanes from all starting at once. If a single lane is heavier than the active caps, the scheduler can still start it when the pool is empty and then keeps it running alone until capacity is available again. Defaults are 10 slots, OPENCLAW_DOCKER_ALL_LIVE_LIMIT=9, OPENCLAW_DOCKER_ALL_NPM_LIMIT=5, and OPENCLAW_DOCKER_ALL_SERVICE_LIMIT=7; tune OPENCLAW_DOCKER_ALL_WEIGHT_LIMIT or OPENCLAW_DOCKER_ALL_DOCKER_LIMIT (and other OPENCLAW_DOCKER_ALL_<RESOURCE>_LIMIT overrides) only when the Docker host has more headroom. The runner performs a Docker preflight by default, removes stale OpenClaw E2E containers, prints status every 30 seconds, stores successful lane timings in .artifacts/docker-tests/lane-timings.json, and uses those timings to start longer lanes first on later runs. Use OPENCLAW_DOCKER_ALL_DRY_RUN=1 to print the weighted lane manifest without building or running Docker, or node scripts/test-docker-all.mjs --plan-json to print the CI plan for selected lanes, package/image needs, and credentials.
Package Acceptance is the GitHub-native package gate for "does this installable tarball work as a product?" It resolves one candidate package from source=npm, source=ref, source=url, source=trusted-url, or source=artifact, uploads it as package-under-test, then runs the reusable Docker E2E lanes against that exact tarball instead of repacking the selected ref. Profiles are ordered by breadth: smoke, package, product, and full (plus custom for an explicit lane list). See Testing updates and plugins for the package/update/plugin contract, published-upgrade survivor matrix, release defaults, and failure triage.
Build and release checks run scripts/check-cli-bootstrap-imports.mjs after tsdown. The guard walks the static built graph from dist/entry.js and dist/cli/run-main.js and fails if that pre-dispatch bootstrap graph statically imports any external package (Commander, prompt UI, undici, logging, and similar startup-heavy deps all count) before command dispatch; it also caps the bundled gateway run chunk at 70 KB and rejects static imports of known cold gateway paths (control-ui-assets, diagnostic-stability-bundle, onboard-helpers, process-respawn, restart-sentinel, server-close, server-reload-handlers) from that chunk. scripts/release-check.ts separately smoke-tests the packed CLI with --help, onboard --help, doctor --help, status --json --timeout 1, config schema, and models list --provider openai.
Package Acceptance legacy compatibility is capped at 2026.4.25 (2026.4.25-beta.* included). Through that cutoff, the harness tolerates only shipped-package metadata gaps: omitted private QA inventory entries, missing gateway install --wrapper, missing patch files in the tarball-derived git fixture, missing persisted update.channel, legacy plugin install-record locations, missing marketplace install-record persistence, and config metadata migration during plugins update. For packages after 2026.4.25, those paths are strict failures.
Container smoke runners: test:docker:openwebui, test:docker:onboard, test:docker:npm-onboard-channel-agent, test:docker:release-user-journey, test:docker:release-typed-onboarding, test:docker:release-media-memory, test:docker:release-upgrade-user-journey, test:docker:release-plugin-marketplace, test:docker:skill-install, test:docker:update-channel-switch, test:docker:upgrade-survivor, test:docker:published-upgrade-survivor, test:docker:session-runtime-context, test:docker:agents-delete-shared-workspace, test:docker:gateway-network, test:docker:browser-cdp-snapshot, test:docker:mcp-channels, test:docker:agent-bundle-mcp-tools, test:docker:cron-mcp-cleanup, test:docker:plugins, test:docker:plugin-update, test:docker:plugin-lifecycle-matrix, and test:docker:config-reload boot one or more real containers and verify higher-level integration paths.
Docker/Bash E2E lanes that install the packed OpenClaw tarball through scripts/lib/openclaw-e2e-instance.sh cap npm install at OPENCLAW_E2E_NPM_INSTALL_TIMEOUT (default 600s; set 0 to disable the wrapper for debugging).
Direct models: pnpm test:docker:live-models (script: scripts/test-live-models-docker.sh)
ACP bind smoke: pnpm test:docker:live-acp-bind (script: scripts/test-live-acp-bind-docker.sh; covers Claude, Codex, and Gemini by default, with strict Droid/OpenCode coverage via pnpm test:docker:live-acp-bind:droid and pnpm test:docker:live-acp-bind:opencode)
CLI backend smoke: pnpm test:docker:live-cli-backend (script: scripts/test-live-cli-backend-docker.sh)
Codex app-server harness smoke: pnpm test:docker:live-codex-harness (script: scripts/test-live-codex-harness-docker.sh)
Gateway + dev agent: pnpm test:docker:live-gateway (script: scripts/test-live-gateway-models-docker.sh)
Observability smokes: pnpm qa:otel:smoke, pnpm qa:prometheus:smoke, and pnpm qa:observability:smoke are private QA source-checkout lanes. They are intentionally not part of package Docker release lanes because the npm tarball omits QA Lab.
Open WebUI live smoke: pnpm test:docker:openwebui (script: scripts/e2e/openwebui-docker.sh)
Onboarding wizard (TTY, full scaffolding): pnpm test:docker:onboard (script: scripts/e2e/onboard-docker.sh)
Npm tarball onboarding/channel/agent smoke: pnpm test:docker:npm-onboard-channel-agent installs the packed OpenClaw tarball globally in Docker, configures OpenAI via env-ref onboarding plus Telegram by default, runs doctor, and runs one mocked OpenAI agent turn. Reuse a prebuilt tarball with OPENCLAW_CURRENT_PACKAGE_TGZ=/path/to/openclaw-*.tgz, skip the host rebuild with OPENCLAW_NPM_ONBOARD_HOST_BUILD=0, or switch channel with OPENCLAW_NPM_ONBOARD_CHANNEL=discord or OPENCLAW_NPM_ONBOARD_CHANNEL=slack.
Release user journey smoke: pnpm test:docker:release-user-journey installs the packed OpenClaw tarball globally in a clean Docker home, runs onboarding, configures a mocked OpenAI provider, runs an agent turn, installs/uninstalls external plugins, configures ClickClack against a local fixture, verifies outbound/inbound messaging, restarts Gateway, and runs doctor.
Release typed onboarding smoke: pnpm test:docker:release-typed-onboarding installs the packed tarball, drives openclaw onboard through a real TTY, configures OpenAI as an env-ref provider, verifies no raw key persistence, and runs a mocked agent turn.
Release media/memory smoke: pnpm test:docker:release-media-memory installs the packed tarball, verifies image understanding from a PNG attachment, OpenAI-compatible image generation output, memory search recall, and recall survival across Gateway restart.
Release upgrade user journey smoke: pnpm test:docker:release-upgrade-user-journey installs the newest published baseline older than the candidate tarball by default, configures provider/plugin/ClickClack state on the published package, upgrades to the candidate tarball, then reruns the core agent/plugin/channel journey. If no older published baseline exists, it reuses the candidate version. Override the baseline with OPENCLAW_RELEASE_UPGRADE_BASELINE_SPEC=openclaw@<version>.
Release plugin marketplace smoke: pnpm test:docker:release-plugin-marketplace installs from a local fixture marketplace, updates the installed plugin, uninstalls it, and verifies the plugin CLI disappears with install metadata pruned.
Skill install smoke: pnpm test:docker:skill-install installs the packed OpenClaw tarball globally in Docker, disables uploaded archive installs in config, resolves the current live ClawHub skill slug from search, installs it with openclaw skills install, and verifies the installed skill plus .clawhub origin/lock metadata.
Update channel switch smoke: pnpm test:docker:update-channel-switch installs the packed OpenClaw tarball globally in Docker, switches from package stable to git dev, verifies the persisted channel and plugin post-update work, then switches back to package stable and checks update status.
Upgrade survivor smoke: pnpm test:docker:upgrade-survivor installs the packed OpenClaw tarball over a dirty old-user fixture with agents, channel config, plugin allowlists, stale plugin dependency state, and existing workspace/session files. It runs package update plus non-interactive doctor without live provider or channel keys, then starts a loopback Gateway and checks config/state preservation plus startup/status budgets.

OPENCLAW_DOCKER_E2E_IMAGE=openclaw-docker-e2e-functional:local pnpm test:docker:e2e-build
OPENCLAW_DOCKER_E2E_IMAGE=openclaw-docker-e2e-functional:local OPENCLAW_SKIP_DOCKER_BUILD=1 pnpm test:docker:mcp-channels

Docs sanity

Run docs checks after doc edits: pnpm check:docs. Run full Mintlify anchor validation when you need in-page heading checks too: pnpm docs:check-links:anchors.

Offline regression (CI-safe)

These are "real pipeline" regressions without real providers:

주요 항목:

Gateway tool calling (mock OpenAI, real gateway + agent loop): src/gateway/gateway.test.ts (case: "runs a mock OpenAI tool call end-to-end via gateway agent loop")
Gateway wizard (WS wizard.start/wizard.next, writes config + auth enforced): src/gateway/gateway.test.ts (case: "runs wizard over ws and writes auth token config")

Agent reliability evals (skills)

We already have a few CI-safe tests that behave like "agent reliability evals":

주요 항목:

Mock tool-calling through the real gateway + agent loop (src/gateway/gateway.test.ts).
End-to-end wizard flows that validate session wiring and config effects (src/gateway/gateway.test.ts).
Decisioning: when skills are listed in the prompt, does the agent pick the right skill (or avoid irrelevant ones)?
Compliance: does the agent read SKILL.md before use and follow required steps/args?
Workflow contracts: multi-turn scenarios that assert tool order, session history carryover, and sandbox boundaries.
A scenario runner using mock providers to assert tool calls + order, skill file reads, and session wiring.
A small suite of skill-focused scenarios (use vs avoid, gating, prompt injection).
Optional live evals (opt-in, env-gated) only after the CI-safe suite is in place.

Contract tests (plugin and channel shape)

Contract tests verify that every registered plugin and channel conforms to its interface contract. They iterate over all discovered plugins and run a suite of shape and behavior assertions. The default pnpm test unit lane intentionally skips these shared seam and smoke files; run the contract commands explicitly when you touch shared channel or provider surfaces.

명령

주요 항목:

All contracts: pnpm test:contracts
Channel contracts only: pnpm test:contracts:channels
Provider contracts only: pnpm test:contracts:plugins

Channel contracts

Located in src/channels/plugins/contracts/*.contract.test.ts. Current top-level categories:

주요 항목:

channel-catalog - bundled/registry channel catalog entry metadata
plugin (registry-backed, sharded) - basic plugin registration shape
surfaces-only (registry-backed, sharded) - per-surface shape checks for actions, setup, status, outbound, messaging, threading, directory, and gateway
session-binding (registry-backed) - session binding behavior
outbound-payload - message payload structure and normalization
group-policy (fallback) - default group policy enforcement per channel
threading (registry-backed, sharded) - thread id handling
directory (registry-backed, sharded) - directory/roster API
registry and plugins-core.* - channel plugin registry, loader, and config-write authorization internals

Provider contracts

Located in src/plugins/contracts/*.contract.test.ts. Current categories include:

주요 항목:

shape - plugin manifest, API, and runtime export shape
plugin-registration (+ parallel) - manifest registration cases
package-manifest - package manifest requirements
loader - plugin loader setup/teardown behavior
registry - plugin contract registry contents and lookup
providers - shared provider behavior across bundled providers, plus web-search providers
auth-choice - auth choice metadata and setup behavior
provider-catalog-deprecation - deprecated provider catalog metadata
wizard.choice-resolution, wizard.model-picker, wizard.setup-options - provider setup wizard contracts
embedding-provider, memory-embedding-provider, web-fetch-provider, tts - capability-specific provider contracts
session-actions, session-attachments, session-entry-projection - plugin-owned session state contracts
scheduled-turns - plugin scheduled turn metadata and timestamp bounds
host-hooks, run-context-lifecycle, runtime-import-side-effects, runtime-seams - plugin host/runtime lifecycle and import-boundary contracts
extension-runtime-dependencies - runtime dependency placement for extensions

When to run

Contract tests run in CI and do not require real API keys.

주요 항목:

After changing plugin-sdk exports or subpaths
After adding or modifying a channel or provider plugin
After refactoring plugin registration or discovery

Adding regressions (guidance)

When you fix a provider/model issue discovered in live:

주요 항목:

Add a CI-safe regression if possible (mock/stub provider, or capture the exact request-shape transformation)
If it's inherently live-only (rate limits, auth policies), keep the live test narrow and opt-in via env vars
Prefer targeting the smallest layer that catches the bug:
provider request conversion/replay bug -> direct models test
gateway session/history/tool pipeline bug -> gateway live smoke or CI-safe gateway mock test
SecretRef traversal guardrail:
src/secrets/exec-secret-ref-id-parity.test.ts derives one sampled target per SecretRef class from registry metadata (listSecretTargetRegistryEntries()), then asserts traversal-segment exec ids are rejected.
If you add a new includeInPlan SecretRef target family in src/secrets/target-registry-data.ts, update classifyTargetClass in that test. The test intentionally fails on unclassified target ids so new classes cannot be skipped silently.

실습 체크리스트

공식 문서와 로컬 버전을 대조합니다: https://docs.openclaw.ai/help/testing
관련 CLI는 openclaw --help 및 하위 명령 --help로 옵션을 확인합니다.
설정 변경 시 openclaw config / openclaw doctor로 유효성을 검사합니다.
Gateway·채널·플러그인 변경 후에는 필요 시 Gateway를 재시작합니다.

자주 쓰는 명령·설정 예시

const tempDirs = useAutoCleanupTempDirTracker(afterEach);

it("uses a temp workspace", () => {
  const workspace = tempDirs.make("openclaw-example-");
  // use workspace
});

// openclaw-temp-dir: allow verifies raw fs cleanup behavior
const workspace = fs.mkdtempSync(prefix);

gh workflow run package-acceptance.yml --ref main \
  -f source=npm \
  -f package_spec=openclaw@beta \
  -f suite_profile=product \
  -f telegram_mode=mock-openai

gh workflow run package-acceptance.yml --ref main \
  -f source=url \
  -f package_url=https://registry.npmjs.org/openclaw/-/openclaw-VERSION.tgz \
  -f package_sha256=<sha256> \
  -f suite_profile=package

gh workflow run package-acceptance.yml --ref main \
  -f source=trusted-url \
  -f trusted_source_id=enterprise-artifactory \
  -f package_url=https://packages.example.internal:8443/artifactory/openclaw/openclaw-VERSION.tgz \
  -f package_sha256=<sha256> \
  -f suite_profile=package

gh workflow run package-acceptance.yml --ref main \
  -f source=artifact \
  -f artifact_run_id=<run-id> \
  -f artifact_name=<artifact-name> \
  -f suite_profile=smoke

Testing

핵심 요약

Testing kit: unit/e2e/live suites, Docker runners, and what each test covers

한국어 가이드 범위: help/testing 경로의 설정·명령·제약·예시를 학습용으로 재구성합니다.

문서 구성

공식 문서의 주요 섹션은 다음과 같습니다.

빠른 시작
Test Temp Directories
Live and Docker/Parallels workflows
QA-specific runners
Shared Telegram credentials via Convex (v1)
Adding a channel to QA
Test suites (what runs where)
Unit / integration (default)
Stability (gateway)
E2E (repo aggregate)
E2E (gateway smoke)
E2E (Control UI mocked browser)
E2E: OpenShell backend smoke
Live (real providers + real models)
Which suite should I run?
Live (network-touching) tests
Docker runners (optional "works in Linux" checks)
Docs sanity
Offline regression (CI-safe)
Agent reliability evals (skills)
Contract tests (plugin and channel shape)
명령
Channel contracts
Provider contracts
When to run
Adding regressions (guidance)
관련 문서

상세 내용

본문

주요 항목:

QA overview - architecture, command surface, scenario authoring, and Matrix profiles.
Maturity scorecard - how release QA evidence supports stability and LTS decisions.
QA channel - the synthetic transport plugin used by repo-backed scenarios.

빠른 시작

When you touch tests or want extra confidence:

주요 항목:

Full gate (expected before push): pnpm build && pnpm check && pnpm check:test-types && pnpm test
Faster local full-suite run on a roomy machine: pnpm test:max
Direct Vitest watch loop: pnpm test:watch
Direct file targeting routes plugin/channel paths too: pnpm test extensions/discord/src/monitor/message-handler.preflight.test.ts
Prefer targeted runs first when iterating on a single failure.
Docker-backed QA site: pnpm qa:lab:up
Linux VM-backed QA lane: pnpm openclaw qa suite --runner multipass --scenario channel-chat-baseline
Informational V8 coverage report: pnpm test:coverage
E2E suite: pnpm test:e2e

Test Temp Directories

Use the shared helpers in test/helpers/temp-dir.ts for test-owned temporary directories so ownership is explicit and cleanup stays in the test lifecycle:

const tempDirs = useAutoCleanupTempDirTracker(afterEach);

it("uses a temp workspace", () => {
  const workspace = tempDirs.make("openclaw-example-");
  // use workspace
});

// openclaw-temp-dir: allow verifies raw fs cleanup behavior
const workspace = fs.mkdtempSync(prefix);

Live and Docker/Parallels workflows

When debugging real providers/models (requires real creds):

주요 항목:

Live suite (models + gateway tool/image probes): pnpm test:live
Target one live file quietly: pnpm test:live -- src/agents/models.profiles.live.test.ts
Runtime performance reports: dispatch OpenClaw Performance with
Docker live model sweep: pnpm test:docker:live-models
Each selected model runs a text turn plus a small file-read-style probe.
CI coverage: daily OpenClaw Scheduled Live And E2E Checks and manual
For focused CI reruns, dispatch OpenClaw Live And E2E Checks (Reusable)
Add new high-signal provider secrets to scripts/ci-hydrate-live-auth.sh
Native Codex bound-chat smoke: pnpm test:docker:live-codex-bind
Runs a Docker live lane against the Codex app-server path, binds a
Codex app-server harness smoke: pnpm test:docker:live-codex-harness
Runs gateway agent turns through the plugin-owned Codex app-server
Codex on-demand install smoke: pnpm test:docker:codex-on-demand
Installs the packaged OpenClaw tarball in Docker, runs OpenAI API-key
Codex npm-plugin live package smoke: pnpm test:docker:live-codex-npm-plugin
Installs the candidate OpenClaw package and exact Codex plugin into Docker,
Its zero-retry medium-thinking follow-through turn must send progress, keep
Live plugin tool dependency smoke: pnpm test:docker:live-plugin-tool
Packs a fixture plugin with a real slugify dependency, installs it
OpenClaw rescue command smoke: pnpm test:live:system-agent-rescue-channel
Opt-in belt-and-suspenders check for the message-channel rescue command
OpenClaw first-run Docker smoke: pnpm test:docker:system-agent-first-run
Starts from an empty OpenClaw state dir and first proves the packaged
Moonshot/Kimi cost smoke: with MOONSHOT_API_KEY set, run

QA-specific runners

These commands sit beside the main test suites when you need QA-lab realism.

주요 항목:

pnpm openclaw qa suite
Runs repo-backed QA scenarios directly on the host.
Writes top-level qa-evidence.json, qa-suite-summary.json, and
When dispatched by pnpm openclaw qa run --qa-profile <profile>, embeds
Runs multiple selected scenarios in parallel by default with isolated
Exits non-zero when any scenario fails. Use --allow-failures for
Supports provider modes live-frontier, mock-openai, and aimock.
pnpm openclaw qa coverage --match <query>
Searches scenario IDs, titles, surfaces, coverage IDs, docs refs, code
Use this before a QA Lab run when you know the touched behavior or file
pnpm test:plugins:kitchen-sink-live
Runs the live OpenAI Kitchen Sink plugin gauntlet through QA Lab.
pnpm test:gateway:cpu-scenarios
Runs the gateway startup bench plus a small mock QA Lab scenario pack
Flags only sustained hot CPU observations by default (--cpu-core-warn,
Runs against built dist artifacts; run a build first when the checkout
pnpm openclaw qa suite --runner multipass
Runs the same QA suite inside a disposable Multipass Linux VM, keeping
Live runs forward the QA auth inputs practical for the guest:
Output dirs must stay under the repo root so the guest can write back
Writes the normal QA report + summary plus Multipass logs under
pnpm qa:lab:up
Starts the Docker-backed QA site for operator-style QA work.
pnpm test:docker:npm-onboard-channel-agent
Builds an npm tarball from the current checkout, installs it globally in

gh workflow run package-acceptance.yml --ref main \
  -f source=npm \
  -f package_spec=openclaw@beta \
  -f suite_profile=product \
  -f telegram_mode=mock-openai

gh workflow run package-acceptance.yml --ref main \
  -f source=url \
  -f package_url=https://registry.npmjs.org/openclaw/-/openclaw-VERSION.tgz \
  -f package_sha256=<sha256> \
  -f suite_profile=package

gh workflow run package-acceptance.yml --ref main \
  -f source=trusted-url \
  -f trusted_source_id=enterprise-artifactory \
  -f package_url=https://packages.example.internal:8443/artifactory/openclaw/openclaw-VERSION.tgz \
  -f package_sha256=<sha256> \
  -f suite_profile=package

gh workflow run package-acceptance.yml --ref main \
  -f source=artifact \
  -f artifact_run_id=<run-id> \
  -f artifact_name=<artifact-name> \
  -f suite_profile=smoke

Shared Telegram credentials via Convex (v1)

주요 항목:

OPENCLAW_QA_CONVEX_SITE_URL (for example https://your-deployment.convex.site)
One secret for the selected role:
OPENCLAW_QA_CONVEX_SECRET_MAINTAINER for maintainer
OPENCLAW_QA_CONVEX_SECRET_CI for ci
Credential role selection:
CLI: --credential-role maintainer|ci
Env default: OPENCLAW_QA_CREDENTIAL_ROLE (defaults to ci in CI, maintainer otherwise)
OPENCLAW_QA_CREDENTIAL_LEASE_TTL_MS (default 1200000)
OPENCLAW_QA_CREDENTIAL_HEARTBEAT_INTERVAL_MS (default 30000)
OPENCLAW_QA_CREDENTIAL_ACQUIRE_TIMEOUT_MS (default 90000)
OPENCLAW_QA_CREDENTIAL_HTTP_TIMEOUT_MS (default 15000)
OPENCLAW_QA_CONVEX_ENDPOINT_PREFIX (default /qa-credentials/v1)
OPENCLAW_QA_CREDENTIAL_OWNER_ID (optional trace id)
OPENCLAW_QA_ALLOW_INSECURE_HTTP=1 allows loopback http:// Convex URLs for local-only development.
POST /acquire
Request: { kind, ownerId, actorRole, leaseTtlMs, heartbeatIntervalMs }
Success: { status: "ok", credentialId, leaseToken, payload, leaseTtlMs?, heartbeatIntervalMs? }
Exhausted/retryable: { status: "error", code: "POOL_EXHAUSTED" | "NO_CREDENTIAL_AVAILABLE", ... }
POST /payload-chunk
Request: { kind, ownerId, actorRole, credentialId, leaseToken, index }
Success: { status: "ok", index, data }
POST /heartbeat
Request: { kind, ownerId, actorRole, credentialId, leaseToken, leaseTtlMs }
Success: { status: "ok" } (or empty 2xx)
POST /release

pnpm openclaw qa credentials doctor
pnpm openclaw qa credentials add --kind telegram --payload-file qa/telegram-credential.json
pnpm openclaw qa credentials list --kind telegram
pnpm openclaw qa credentials remove --credential-id <credential-id>

Adding a channel to QA

Test suites (what runs where)

Think of the suites as "increasing realism" (and increasing flakiness/cost).

Unit / integration (default)

expand multi-project shards into per-project configs for parallel scheduling

주요 항목:

Command: pnpm test
Config: untargeted runs use the vitest.full-*.config.ts shard set and may
Files: core/unit inventories under src/**/*.test.ts,
Scope:
Pure unit tests
In-process integration tests (gateway auth, routing, tooling, parsing, config)
Deterministic regressions for known bugs
Expectations:
Runs in CI
No real keys required
Should be fast and stable
Resolver and public-surface loader tests must prove broad api.js and
Default test installs skip optional native Discord opus builds. Discord
Compare native opus performance in the libopus-wasm benchmark repo, not
Untargeted pnpm test runs thirteen smaller shard configs (core-unit-fast, core-unit-src, core-unit-security, core-unit-ui, core-unit-support, core-support-boundary, core-tooling, core-contracts, core-bundled, core-runtime, agentic, auto-reply, extensions) instead of one giant native root-project process. This cuts peak RSS on loaded machines and avoids auto-reply/plugin work starving unrelated suites.
pnpm test --watch still uses the native root vitest.config.ts project graph, because a multi-shard watch loop is not practical.
pnpm test, pnpm test:watch, and pnpm test:perf:imports route explicit file/directory targets through scoped lanes first, so pnpm test extensions/discord/src/monitor/message-handler.preflight.test.ts avoids paying the full root project startup tax.
pnpm test:changed expands changed git paths into cheap scoped lanes by default: direct test edits, sibling *.test.ts files, explicit source mappings, and local import-graph dependents. Config/setup/package edits do not broad-run tests unless you explicitly use OPENCLAW_TEST_CHANGED_BROAD=1 pnpm test:changed.
pnpm check:changed is the normal smart local check gate for narrow work. It classifies the diff into core, core tests, extensions, extension tests, apps, docs, release metadata, live Docker tooling, and tooling, then runs the matching typecheck, lint, and guard commands. It does not run Vitest tests; call pnpm test:changed or explicit pnpm test <target> for test proof. Release metadata-only version bumps run targeted version/config/root-dependency checks, with a guard that rejects package changes outside the top-level version field.
Live Docker ACP harness edits run focused checks: shell syntax for the live Docker auth scripts and a live Docker scheduler dry-run. package.json changes are included only when the diff is limited to scripts["test:docker:live-*"]; dependency, export, version, and other package-surface edits still use the broader guards.
Import-light unit tests from agents, commands, plugins, auto-reply helpers, plugin-sdk, and similar pure utility areas route through the unit-fast lane, which skips test/setup-openclaw-runtime.ts; stateful/runtime-heavy files stay on the existing lanes.
Selected plugin-sdk and commands helper source files also map changed-mode runs to explicit sibling tests in those light lanes, so helper edits avoid rerunning the full heavy suite for that directory.
auto-reply has dedicated buckets for top-level core helpers, top-level reply.* integration tests, and the src/auto-reply/reply/** subtree. CI further splits the reply subtree into agent-runner, dispatch, and commands/state-routing shards so one import-heavy bucket does not own the full Node tail.
Normal PR/main CI intentionally skips the bundled plugin batch sweep and release-only agentic-plugins shard. Full Release Validation dispatches the separate Plugin Prerelease child workflow for those plugin-heavy suites on release candidates.
When you change message-tool discovery inputs or compaction runtime

Stability (gateway)

주요 항목:

Command: pnpm test:stability:gateway
Config: test/vitest/vitest.gateway.config.ts, test/vitest/vitest.logging.config.ts, and test/vitest/vitest.infra.config.ts, each forced to one worker
Scope:
Starts a real loopback Gateway with diagnostics enabled by default
Drives synthetic gateway message, memory, and large-payload churn through the diagnostic event path
Queries diagnostics.stability over the Gateway WS RPC
Covers diagnostic stability bundle persistence helpers
Asserts the recorder remains bounded, synthetic RSS samples stay under the pressure budget, and per-session queue depths drain back to zero
Expectations:
CI-safe and keyless
Narrow lane for stability-regression follow-up, not a substitute for the full Gateway suite

E2E (repo aggregate)

주요 항목:

Command: pnpm test:e2e
Scope:
Runs the gateway smoke E2E lane
Runs the mocked Control UI browser E2E lane
Expectations:
CI-safe and keyless
Requires Playwright Chromium to be installed

E2E (gateway smoke)

주요 항목:

Command: pnpm test:e2e:gateway
Config: test/vitest/vitest.e2e.config.ts
Files: src/**/*.e2e.test.ts, test/**/*.e2e.test.ts, and bundled-plugin E2E tests under extensions/
Runtime defaults:
Uses Vitest threads with isolate: false, matching the rest of the repo.
Uses adaptive workers (CI: up to 2, local: 1 by default).
Runs in silent mode by default to reduce console I/O overhead.
Useful overrides:
OPENCLAW_E2E_WORKERS=<n> to force worker count (capped at 16).
OPENCLAW_E2E_VERBOSE=1 to re-enable verbose console output.
Scope:
Multi-instance gateway end-to-end behavior
WebSocket/HTTP surfaces, node pairing, and heavier networking
Expectations:
Runs in CI (when enabled in the pipeline)
No real keys required
More moving parts than unit tests (can be slower)

E2E (Control UI mocked browser)

주요 항목:

Command: pnpm test:ui:e2e
Config: test/vitest/vitest.ui-e2e.config.ts
Files: ui/src/**/*.e2e.test.ts
Scope:
Starts the Vite Control UI
Drives a real Chromium page through Playwright
Replaces the Gateway WebSocket with deterministic in-browser mocks
Expectations:
Runs in CI as part of pnpm test:e2e
No real Gateway, agents, or provider keys required
Browser dependency must be present (pnpm --dir ui exec playwright install chromium)

E2E: OpenShell backend smoke

주요 항목:

Command: pnpm test:e2e:openshell
File: extensions/openshell/src/backend.e2e.test.ts
Scope:
Reuses an active local OpenShell gateway
Creates a sandbox from a temporary local Dockerfile
Exercises OpenClaw's OpenShell backend over real sandbox ssh-config + SSH exec
Verifies remote-canonical filesystem behavior through the sandbox fs bridge
Expectations:
Opt-in only; not part of the default pnpm test:e2e run
Requires a local openshell CLI plus a working Docker daemon
Requires an active local OpenShell gateway and its config source
Uses isolated HOME / XDG_CONFIG_HOME, then destroys the test sandbox
Useful overrides:
OPENCLAW_E2E_OPENSHELL=1 to enable the test when running the broader e2e suite manually
OPENCLAW_E2E_OPENSHELL_COMMAND=/path/to/openshell to point at a non-default CLI binary or wrapper script
OPENCLAW_E2E_OPENSHELL_CONFIG_HOME=/path/to/config to expose the registered gateway config to the isolated test
OPENCLAW_E2E_OPENSHELL_HOST_IP=172.18.0.1 to override the Docker gateway IP used by the host policy fixture

Live (real providers + real models)

주요 항목:

Command: pnpm test:live
Config: test/vitest/vitest.live.config.ts
Files: src/**/*.live.test.ts, test/**/*.live.test.ts, and bundled-plugin live tests under extensions/
기본값: enabled by pnpm test:live (sets OPENCLAW_LIVE_TEST=1)
Scope:
"Does this provider/model actually work today with real creds?"
Catch provider format changes, tool-calling quirks, auth issues, and rate limit behavior
Expectations:
Not CI-stable by design (real networks, real provider policies, quotas, outages)
Costs money / uses rate limits
Prefer running narrowed subsets instead of "everything"
Live runs use already-exported API keys and staged auth profiles.
By default, live runs still isolate HOME and copy config/auth material into a temp test home so unit fixtures cannot mutate your real ~/.openclaw.
Set OPENCLAW_LIVE_USE_REAL_HOME=1 only when you intentionally need live tests to use your real home directory.
pnpm test:live defaults to a quieter mode: it keeps [live] ... progress output and mutes gateway bootstrap logs/Bonjour chatter. Set OPENCLAW_LIVE_TEST_QUIET=0 if you want the full startup logs back.
API key rotation (provider-specific): set *_API_KEYS with comma/semicolon format or *_API_KEY_1, *_API_KEY_2 (for example OPENAI_API_KEYS, ANTHROPIC_API_KEYS, GEMINI_API_KEYS) or per-live override via OPENCLAW_LIVE_*_KEY; tests retry on rate limit responses.
Progress/heartbeat output:
Live suites emit progress lines to stderr so long provider calls are visibly active even when Vitest console capture is quiet.
test/vitest/vitest.live.config.ts disables Vitest console interception so provider/gateway progress lines stream immediately during live runs.
Tune direct-model heartbeats with OPENCLAW_LIVE_HEARTBEAT_MS.
Tune gateway/probe heartbeats with OPENCLAW_LIVE_GATEWAY_HEARTBEAT_MS.

Which suite should I run?

주요 항목:

Editing logic/tests: run pnpm test (and pnpm test:coverage if you changed a lot)
Touching gateway networking / WS protocol / pairing: add pnpm test:e2e
Debugging "my bot is down" / provider-specific failures / tool calling: run a narrowed pnpm test:live

Live (network-touching) tests

주요 항목:

see Testing live suites. For the dedicated update and

Docker runners (optional "works in Linux" checks)

These Docker runners split into two buckets:

주요 항목:

Live-model runners: test:docker:live-models and test:docker:live-gateway run only their matching profile-key live file inside the repo Docker image (src/agents/models.profiles.live.test.ts and src/gateway/gateway-models.profiles.live.test.ts), mounting your local config dir, workspace, and optional profile env file. The matching local entrypoints are test:live:models-profiles and test:live:gateway-profiles.
Docker live runners keep their own practical caps where needed:
test:docker:all builds the live Docker image once via test:docker:live-build, packs OpenClaw once as an npm tarball through scripts/package-openclaw-for-docker.mjs, then builds/reuses two scripts/e2e/Dockerfile images. The bare image is only the Node/Git runner for install/update/plugin-dependency lanes; those lanes mount the prebuilt tarball. The functional image installs the same tarball into /app for built-app functionality lanes. Docker lane definitions live in scripts/lib/docker-e2e-scenarios.mjs; planner logic lives in scripts/lib/docker-e2e-plan.mjs; scripts/test-docker-all.mjs executes the selected plan. The aggregate uses a weighted local scheduler: OPENCLAW_DOCKER_ALL_PARALLELISM controls process slots, while resource caps keep heavy live, npm-install, and multi-service lanes from all starting at once. If a single lane is heavier than the active caps, the scheduler can still start it when the pool is empty and then keeps it running alone until capacity is available again. Defaults are 10 slots, OPENCLAW_DOCKER_ALL_LIVE_LIMIT=9, OPENCLAW_DOCKER_ALL_NPM_LIMIT=5, and OPENCLAW_DOCKER_ALL_SERVICE_LIMIT=7; tune OPENCLAW_DOCKER_ALL_WEIGHT_LIMIT or OPENCLAW_DOCKER_ALL_DOCKER_LIMIT (and other OPENCLAW_DOCKER_ALL_<RESOURCE>_LIMIT overrides) only when the Docker host has more headroom. The runner performs a Docker preflight by default, removes stale OpenClaw E2E containers, prints status every 30 seconds, stores successful lane timings in .artifacts/docker-tests/lane-timings.json, and uses those timings to start longer lanes first on later runs. Use OPENCLAW_DOCKER_ALL_DRY_RUN=1 to print the weighted lane manifest without building or running Docker, or node scripts/test-docker-all.mjs --plan-json to print the CI plan for selected lanes, package/image needs, and credentials.
Package Acceptance is the GitHub-native package gate for "does this installable tarball work as a product?" It resolves one candidate package from source=npm, source=ref, source=url, source=trusted-url, or source=artifact, uploads it as package-under-test, then runs the reusable Docker E2E lanes against that exact tarball instead of repacking the selected ref. Profiles are ordered by breadth: smoke, package, product, and full (plus custom for an explicit lane list). See Testing updates and plugins for the package/update/plugin contract, published-upgrade survivor matrix, release defaults, and failure triage.
Build and release checks run scripts/check-cli-bootstrap-imports.mjs after tsdown. The guard walks the static built graph from dist/entry.js and dist/cli/run-main.js and fails if that pre-dispatch bootstrap graph statically imports any external package (Commander, prompt UI, undici, logging, and similar startup-heavy deps all count) before command dispatch; it also caps the bundled gateway run chunk at 70 KB and rejects static imports of known cold gateway paths (control-ui-assets, diagnostic-stability-bundle, onboard-helpers, process-respawn, restart-sentinel, server-close, server-reload-handlers) from that chunk. scripts/release-check.ts separately smoke-tests the packed CLI with --help, onboard --help, doctor --help, status --json --timeout 1, config schema, and models list --provider openai.
Package Acceptance legacy compatibility is capped at 2026.4.25 (2026.4.25-beta.* included). Through that cutoff, the harness tolerates only shipped-package metadata gaps: omitted private QA inventory entries, missing gateway install --wrapper, missing patch files in the tarball-derived git fixture, missing persisted update.channel, legacy plugin install-record locations, missing marketplace install-record persistence, and config metadata migration during plugins update. For packages after 2026.4.25, those paths are strict failures.
Container smoke runners: test:docker:openwebui, test:docker:onboard, test:docker:npm-onboard-channel-agent, test:docker:release-user-journey, test:docker:release-typed-onboarding, test:docker:release-media-memory, test:docker:release-upgrade-user-journey, test:docker:release-plugin-marketplace, test:docker:skill-install, test:docker:update-channel-switch, test:docker:upgrade-survivor, test:docker:published-upgrade-survivor, test:docker:session-runtime-context, test:docker:agents-delete-shared-workspace, test:docker:gateway-network, test:docker:browser-cdp-snapshot, test:docker:mcp-channels, test:docker:agent-bundle-mcp-tools, test:docker:cron-mcp-cleanup, test:docker:plugins, test:docker:plugin-update, test:docker:plugin-lifecycle-matrix, and test:docker:config-reload boot one or more real containers and verify higher-level integration paths.
Docker/Bash E2E lanes that install the packed OpenClaw tarball through scripts/lib/openclaw-e2e-instance.sh cap npm install at OPENCLAW_E2E_NPM_INSTALL_TIMEOUT (default 600s; set 0 to disable the wrapper for debugging).
Direct models: pnpm test:docker:live-models (script: scripts/test-live-models-docker.sh)
ACP bind smoke: pnpm test:docker:live-acp-bind (script: scripts/test-live-acp-bind-docker.sh; covers Claude, Codex, and Gemini by default, with strict Droid/OpenCode coverage via pnpm test:docker:live-acp-bind:droid and pnpm test:docker:live-acp-bind:opencode)
CLI backend smoke: pnpm test:docker:live-cli-backend (script: scripts/test-live-cli-backend-docker.sh)
Codex app-server harness smoke: pnpm test:docker:live-codex-harness (script: scripts/test-live-codex-harness-docker.sh)
Gateway + dev agent: pnpm test:docker:live-gateway (script: scripts/test-live-gateway-models-docker.sh)
Observability smokes: pnpm qa:otel:smoke, pnpm qa:prometheus:smoke, and pnpm qa:observability:smoke are private QA source-checkout lanes. They are intentionally not part of package Docker release lanes because the npm tarball omits QA Lab.
Open WebUI live smoke: pnpm test:docker:openwebui (script: scripts/e2e/openwebui-docker.sh)
Onboarding wizard (TTY, full scaffolding): pnpm test:docker:onboard (script: scripts/e2e/onboard-docker.sh)
Npm tarball onboarding/channel/agent smoke: pnpm test:docker:npm-onboard-channel-agent installs the packed OpenClaw tarball globally in Docker, configures OpenAI via env-ref onboarding plus Telegram by default, runs doctor, and runs one mocked OpenAI agent turn. Reuse a prebuilt tarball with OPENCLAW_CURRENT_PACKAGE_TGZ=/path/to/openclaw-*.tgz, skip the host rebuild with OPENCLAW_NPM_ONBOARD_HOST_BUILD=0, or switch channel with OPENCLAW_NPM_ONBOARD_CHANNEL=discord or OPENCLAW_NPM_ONBOARD_CHANNEL=slack.
Release user journey smoke: pnpm test:docker:release-user-journey installs the packed OpenClaw tarball globally in a clean Docker home, runs onboarding, configures a mocked OpenAI provider, runs an agent turn, installs/uninstalls external plugins, configures ClickClack against a local fixture, verifies outbound/inbound messaging, restarts Gateway, and runs doctor.
Release typed onboarding smoke: pnpm test:docker:release-typed-onboarding installs the packed tarball, drives openclaw onboard through a real TTY, configures OpenAI as an env-ref provider, verifies no raw key persistence, and runs a mocked agent turn.
Release media/memory smoke: pnpm test:docker:release-media-memory installs the packed tarball, verifies image understanding from a PNG attachment, OpenAI-compatible image generation output, memory search recall, and recall survival across Gateway restart.
Release upgrade user journey smoke: pnpm test:docker:release-upgrade-user-journey installs the newest published baseline older than the candidate tarball by default, configures provider/plugin/ClickClack state on the published package, upgrades to the candidate tarball, then reruns the core agent/plugin/channel journey. If no older published baseline exists, it reuses the candidate version. Override the baseline with OPENCLAW_RELEASE_UPGRADE_BASELINE_SPEC=openclaw@<version>.
Release plugin marketplace smoke: pnpm test:docker:release-plugin-marketplace installs from a local fixture marketplace, updates the installed plugin, uninstalls it, and verifies the plugin CLI disappears with install metadata pruned.
Skill install smoke: pnpm test:docker:skill-install installs the packed OpenClaw tarball globally in Docker, disables uploaded archive installs in config, resolves the current live ClawHub skill slug from search, installs it with openclaw skills install, and verifies the installed skill plus .clawhub origin/lock metadata.
Update channel switch smoke: pnpm test:docker:update-channel-switch installs the packed OpenClaw tarball globally in Docker, switches from package stable to git dev, verifies the persisted channel and plugin post-update work, then switches back to package stable and checks update status.
Upgrade survivor smoke: pnpm test:docker:upgrade-survivor installs the packed OpenClaw tarball over a dirty old-user fixture with agents, channel config, plugin allowlists, stale plugin dependency state, and existing workspace/session files. It runs package update plus non-interactive doctor without live provider or channel keys, then starts a loopback Gateway and checks config/state preservation plus startup/status budgets.

OPENCLAW_DOCKER_E2E_IMAGE=openclaw-docker-e2e-functional:local pnpm test:docker:e2e-build
OPENCLAW_DOCKER_E2E_IMAGE=openclaw-docker-e2e-functional:local OPENCLAW_SKIP_DOCKER_BUILD=1 pnpm test:docker:mcp-channels

Docs sanity

Run docs checks after doc edits: pnpm check:docs. Run full Mintlify anchor validation when you need in-page heading checks too: pnpm docs:check-links:anchors.

Offline regression (CI-safe)

These are "real pipeline" regressions without real providers:

주요 항목:

Gateway tool calling (mock OpenAI, real gateway + agent loop): src/gateway/gateway.test.ts (case: "runs a mock OpenAI tool call end-to-end via gateway agent loop")
Gateway wizard (WS wizard.start/wizard.next, writes config + auth enforced): src/gateway/gateway.test.ts (case: "runs wizard over ws and writes auth token config")

Agent reliability evals (skills)

We already have a few CI-safe tests that behave like "agent reliability evals":

주요 항목:

Mock tool-calling through the real gateway + agent loop (src/gateway/gateway.test.ts).
End-to-end wizard flows that validate session wiring and config effects (src/gateway/gateway.test.ts).
Decisioning: when skills are listed in the prompt, does the agent pick the right skill (or avoid irrelevant ones)?
Compliance: does the agent read SKILL.md before use and follow required steps/args?
Workflow contracts: multi-turn scenarios that assert tool order, session history carryover, and sandbox boundaries.
A scenario runner using mock providers to assert tool calls + order, skill file reads, and session wiring.
A small suite of skill-focused scenarios (use vs avoid, gating, prompt injection).
Optional live evals (opt-in, env-gated) only after the CI-safe suite is in place.

Contract tests (plugin and channel shape)

명령

주요 항목:

All contracts: pnpm test:contracts
Channel contracts only: pnpm test:contracts:channels
Provider contracts only: pnpm test:contracts:plugins

Channel contracts

Located in src/channels/plugins/contracts/*.contract.test.ts. Current top-level categories:

주요 항목:

channel-catalog - bundled/registry channel catalog entry metadata
plugin (registry-backed, sharded) - basic plugin registration shape
surfaces-only (registry-backed, sharded) - per-surface shape checks for actions, setup, status, outbound, messaging, threading, directory, and gateway
session-binding (registry-backed) - session binding behavior
outbound-payload - message payload structure and normalization
group-policy (fallback) - default group policy enforcement per channel
threading (registry-backed, sharded) - thread id handling
directory (registry-backed, sharded) - directory/roster API
registry and plugins-core.* - channel plugin registry, loader, and config-write authorization internals

Provider contracts

Located in src/plugins/contracts/*.contract.test.ts. Current categories include:

주요 항목:

shape - plugin manifest, API, and runtime export shape
plugin-registration (+ parallel) - manifest registration cases
package-manifest - package manifest requirements
loader - plugin loader setup/teardown behavior
registry - plugin contract registry contents and lookup
providers - shared provider behavior across bundled providers, plus web-search providers
auth-choice - auth choice metadata and setup behavior
provider-catalog-deprecation - deprecated provider catalog metadata
wizard.choice-resolution, wizard.model-picker, wizard.setup-options - provider setup wizard contracts
embedding-provider, memory-embedding-provider, web-fetch-provider, tts - capability-specific provider contracts
session-actions, session-attachments, session-entry-projection - plugin-owned session state contracts
scheduled-turns - plugin scheduled turn metadata and timestamp bounds
host-hooks, run-context-lifecycle, runtime-import-side-effects, runtime-seams - plugin host/runtime lifecycle and import-boundary contracts
extension-runtime-dependencies - runtime dependency placement for extensions

When to run

Contract tests run in CI and do not require real API keys.

주요 항목:

After changing plugin-sdk exports or subpaths
After adding or modifying a channel or provider plugin
After refactoring plugin registration or discovery

Adding regressions (guidance)

When you fix a provider/model issue discovered in live:

주요 항목:

Add a CI-safe regression if possible (mock/stub provider, or capture the exact request-shape transformation)
If it's inherently live-only (rate limits, auth policies), keep the live test narrow and opt-in via env vars
Prefer targeting the smallest layer that catches the bug:
provider request conversion/replay bug -> direct models test
gateway session/history/tool pipeline bug -> gateway live smoke or CI-safe gateway mock test
SecretRef traversal guardrail:
src/secrets/exec-secret-ref-id-parity.test.ts derives one sampled target per SecretRef class from registry metadata (listSecretTargetRegistryEntries()), then asserts traversal-segment exec ids are rejected.
If you add a new includeInPlan SecretRef target family in src/secrets/target-registry-data.ts, update classifyTargetClass in that test. The test intentionally fails on unclassified target ids so new classes cannot be skipped silently.

실습 체크리스트

공식 문서와 로컬 버전을 대조합니다: https://docs.openclaw.ai/help/testing
관련 CLI는 openclaw --help 및 하위 명령 --help로 옵션을 확인합니다.
설정 변경 시 openclaw config / openclaw doctor로 유효성을 검사합니다.
Gateway·채널·플러그인 변경 후에는 필요 시 Gateway를 재시작합니다.

자주 쓰는 명령·설정 예시

const tempDirs = useAutoCleanupTempDirTracker(afterEach);

it("uses a temp workspace", () => {
  const workspace = tempDirs.make("openclaw-example-");
  // use workspace
});

// openclaw-temp-dir: allow verifies raw fs cleanup behavior
const workspace = fs.mkdtempSync(prefix);

gh workflow run package-acceptance.yml --ref main \
  -f source=npm \
  -f package_spec=openclaw@beta \
  -f suite_profile=product \
  -f telegram_mode=mock-openai

gh workflow run package-acceptance.yml --ref main \
  -f source=url \
  -f package_url=https://registry.npmjs.org/openclaw/-/openclaw-VERSION.tgz \
  -f package_sha256=<sha256> \
  -f suite_profile=package

gh workflow run package-acceptance.yml --ref main \
  -f source=trusted-url \
  -f trusted_source_id=enterprise-artifactory \
  -f package_url=https://packages.example.internal:8443/artifactory/openclaw/openclaw-VERSION.tgz \
  -f package_sha256=<sha256> \
  -f suite_profile=package

gh workflow run package-acceptance.yml --ref main \
  -f source=artifact \
  -f artifact_run_id=<run-id> \
  -f artifact_name=<artifact-name> \
  -f suite_profile=smoke

Testing

핵심 요약

문서 구성

상세 내용

본문

빠른 시작

Test Temp Directories

Live and Docker/Parallels workflows

QA-specific runners

Shared Telegram credentials via Convex (v1)

Adding a channel to QA

Test suites (what runs where)

Unit / integration (default)

Stability (gateway)

E2E (repo aggregate)

E2E (gateway smoke)

E2E (Control UI mocked browser)

E2E: OpenShell backend smoke

Live (real providers + real models)

Which suite should I run?

Live (network-touching) tests

Docker runners (optional "works in Linux" checks)

Docs sanity

Offline regression (CI-safe)

Agent reliability evals (skills)

Contract tests (plugin and channel shape)

명령

Channel contracts

Provider contracts

When to run

Adding regressions (guidance)

관련 문서

실습 체크리스트

자주 쓰는 명령·설정 예시

관련 링크

관련 가이드

Testing

핵심 요약

문서 구성

상세 내용

본문

빠른 시작

Test Temp Directories

Live and Docker/Parallels workflows

QA-specific runners

Shared Telegram credentials via Convex (v1)

Adding a channel to QA

Test suites (what runs where)

Unit / integration (default)

Stability (gateway)

E2E (repo aggregate)

E2E (gateway smoke)

E2E (Control UI mocked browser)

E2E: OpenShell backend smoke

Live (real providers + real models)

Which suite should I run?

Live (network-touching) tests

Docker runners (optional "works in Linux" checks)

Docs sanity

Offline regression (CI-safe)

Agent reliability evals (skills)

Contract tests (plugin and channel shape)

명령

Channel contracts

Provider contracts

When to run

Adding regressions (guidance)

관련 문서

실습 체크리스트

자주 쓰는 명령·설정 예시

관련 링크

관련 가이드