Computer use

기준일: 2026-07-26
공식 기준: Computer use

Computer use 문서는 OpenClaw 공식 문서(nodes/computer-use)를 한국어로 정리한 가이드입니다. Capability-based desktop control through the computer tool and computer.act node command 명령·설정 키·코드 예시는 공식 문서를 그대로 보존하며, 해석과 절차 안내는 한국어로 제공합니다. 최종 동작은 설치된 CLI 버전과 공식 원문을 확인하세요.

핵심 요약

Capability-based desktop control through the computer tool and computer.act node command

한국어 가이드 범위: nodes/computer-use 경로의 설정·명령·제약·예시를 학습용으로 재구성합니다.

문서 구성

공식 문서의 주요 섹션은 다음과 같습니다.

• 요구사항
• The computer agent tool
• Windows and Linux (experimental, via cua-driver)
• 트러블슈팅
• The computer.act node command
• Enable and arm
• Safety
• Relationship to other desktop-control paths

상세 내용

본문

Computer use lets the gateway agent see and control a capable paired desktop. Eligibility is capability-based: the connected node must advertise both computer.act and screen.snapshot, whose result must include a displayFrameId. The tool captures a screenshot as its reference frame, then drives the pointer and keyboard through the dangerous computer.act command. The action set follows the core Anthropic computer-use actions; optional computer_20251124 zoom is not exposed. A vision-capable model drives it through the built-in computer agent tool.

위 내용은 공식 문서의 해당 섹션 요지입니다. 세부 플래그·기본값은 원문과 --help를 확인하세요.

요구사항

주요 항목:

• A paired, connected node advertising both computer.act and screen.snapshot, with screen.snapshot returning displayFrameId.
• macOS fulfiller: app setting Allow Computer Control enabled (default: off).
• macOS fulfiller: Accessibility permission granted to OpenClaw (for pointer/keyboard injection) and Screen Recording permission (for screen.snapshot).
• Windows/Linux fulfiller: bundled cua-computer plugin enabled and a compatible cua-driver 0.10.x executable installed.
• The computer.act command armed on the gateway (it is dangerous and disarmed by default).
• A vision-capable agent model.
• Tool policy that exposes computer. The default coding profile does not. Add computer to tools.alsoAllow; sandboxed agents also need it in tools.sandbox.tools.alsoAllow.

The computer agent tool

The built-in computer tool takes one action per call. Coordinates are non-negative integer pixels in the most recent screenshot; the node maps them to display points. Coordinate actions must echo the screenshot result's frameId, and an explicit screenIndex must match that frame. OpenClaw also carries a node-issued display identity from the screenshot into the action, so a display reconnect or geometry change fails closed instead of silently retargeting the same index. These checks reject guessed tokens and tokens from another delivered frame or display. A token is not a freshness guarantee: apps can change pixels on the same display after capture, so take a new screenshot whenever the scene may have changed.

위 내용은 공식 문서의 해당 섹션 요지입니다. 세부 플래그·기본값은 원문과 --help를 확인하세요.

주요 항목:

• Reads: screenshot.
• Pointer: left_click, right_click, middle_click, double_click, triple_click, mouse_move, left_click_drag (with startCoordinate), left_mouse_down, left_mouse_up.
• Scroll: scroll with scrollDirection (up|down|left|right) and scrollAmount (wheel ticks).
• Keyboard: type (text), key (combo such as cmd+shift+t or Return), hold_key (text combo held for duration seconds).
• Pacing: wait (duration seconds).

Windows and Linux (experimental, via cua-driver)

The bundled cua-computer plugin provides an experimental fulfiller for Windows and Linux node hosts. It is disabled by default and requires the prerelease 0.10.x driver contract:

위 내용은 공식 문서의 해당 섹션 요지입니다. 세부 플래그·기본값은 원문과 --help를 확인하세요.

openclaw plugins enable cua-computer

트러블슈팅

The cua-computer fulfiller surfaces typed error codes in the tool result and node logs. Common ones:

위 내용은 공식 문서의 해당 섹션 요지입니다. 세부 플래그·기본값은 원문과 --help를 확인하세요.

Code	Cause	Fix
COMPUTER_DRIVER_UNAVAILABLE	The cua-driver binary is not on PATH (or driverPath is wrong), the daemon did not become ready in time, or the node is not Windows/Linux.	Install cua-driver 0.10.x on PATH or set driverPath. Run openclaw node run inside the interactive desktop session; on Linux ensure an X11 DISPLAY (or a WAYLAND_DISPLAY with CUA_DRIVER_RS_ENABLE_WAYLAND) is present.
COMPUTER_DRIVER_UNSUPPORTED	The connected driver is not cua-driver 0.10.x, or its capability/schema version differs.	Install a supported 0.10.x build. The plugin re-probes about 30 seconds after you correct it, so no node restart is needed.
COMPUTER_REFUSED_<code>	The driver refused the action with a structured code such as background_unavailable, background_occluded, or foreground_unavailable (KDE/KWin Wayland).	Bring the target window forward, switch to X11, or use a supported compositor. See the compatibility notes above.
COMPUTER_STALE_FRAME	The coordinates referenced a screenshot that is no longer current (context compaction, a display geometry change, or a reference-width change).	Take a fresh screenshot before the coordinate action.
COMPUTER_UNSUPPORTED_ACTION	An action this fulfiller cannot faithfully deliver: hold_key, left_mouse_down, left_mouse_up, modifier-held drag/scroll, or modifier-held click on Linux.	Use a supported action. cua-driver 0.10.x has no desktop-scope held-input contract.
COMPUTER_UNSUPPORTED_DISPLAY	A non-primary screenIndex, a capture/screen geometry mismatch, or a cursor outside the primary display.	Drive the primary display only.
COMPUTER_UNSUPPORTED_KEY	A key value the driver cannot reproduce reliably: a digit or punctuation key whose shift state is layout-dependent, or an unknown key.	Send that text through the type action instead.
COMPUTER_DRIVER_ERROR / COMPUTER_INVALID_REQUEST	The driver failed without a structured code, or the action arguments were malformed.	Check the driver state and retake a screenshot; correct the action arguments.

The computer.act node command

computer.act is the single node command the tool routes input through (node.invoke with command: "computer.act"). It is:

위 내용은 공식 문서의 해당 섹션 요지입니다. 세부 플래그·기본값은 원문과 --help를 확인하세요.

주요 항목:

• Dangerous by default: listed in the built-in dangerous node commands and excluded from the runtime allowlist until explicitly armed. macOS, Windows, and Linux desktop nodes may still declare it at pairing so the surface is approved once.
• Capability-based: the tool requires a connected node to advertise both computer.act and screen.snapshot. The bundled macOS app and the opt-in experimental cua-computer plugin fulfill the same command pair.

Enable and arm

1. Enable the platform fulfiller: on macOS, enable Settings → Allow Computer Control, then grant Accessibility and Screen Recording under Settings → Permissions; on Windows/Linux, follow the experimental cua-computer setup above. 2. Approve the pairing update on the gateway (a new command forces re-pairing). 3. Expose the tool to the vision-capable agent. For the default coding profile:

위 내용은 공식 문서의 해당 섹션 요지입니다. 세부 플래그·기본값은 원문과 --help를 확인하세요.

{
     tools: {
       alsoAllow: ["computer"],
       // Sandboxed agents need this second gate too:
       sandbox: { tools: { alsoAllow: ["computer"] } },
     },
   }

/phone arm computer 30m
   /phone status
   /phone disarm

Safety

주요 항목:

• Before authorization, every layer (tool policy, gateway command policy, node-app setting, and platform permissions) must agree. For the current macOS fulfiller, that includes Allow Computer Control, Accessibility, and Screen Recording. Once armed, actions execute without a per-action confirmation until expiry or /phone disarm.
• The macOS fulfiller posts text one grapheme at a time, so cancellation, disconnect, pause, disable, or endpoint replacement stops it before the next grapheme. The experimental cua-driver fulfiller cannot cancel a type_text call mid-typing.
• Screenshots are model-only and never auto-sent to chat (issue #44759).
• Treat screen content as untrusted; it can carry prompt injection.

Relationship to other desktop-control paths

This is the agent-driven path. See Peekaboo bridge for how it relates to the PeekabooBridge host, Codex Computer Use, and the direct cua-driver MCP.

위 내용은 공식 문서의 해당 섹션 요지입니다. 세부 플래그·기본값은 원문과 --help를 확인하세요.

실습 체크리스트

1. 공식 문서와 로컬 버전을 대조합니다: https://docs.openclaw.ai/nodes/computer-use
2. 관련 CLI는 openclaw --help 및 하위 명령 --help로 옵션을 확인합니다.
3. 설정 변경 시 openclaw config / openclaw doctor로 유효성을 검사합니다.
4. Gateway·채널·플러그인 변경 후에는 필요 시 Gateway를 재시작합니다.

자주 쓰는 명령·설정 예시

openclaw plugins enable cua-computer

{
     tools: {
       alsoAllow: ["computer"],
       // Sandboxed agents need this second gate too:
       sandbox: { tools: { alsoAllow: ["computer"] } },
     },
   }

/phone arm computer 30m
   /phone status
   /phone disarm

관련 링크

• 공식 원문: nodes/computer-use
• OpenClaw 문서 홈

이 가이드는 공식 문서를 한국어 학습용으로 재구성한 것입니다. 옵션 기본값·플래그 이름은 설치 버전에 따라 달라질 수 있습니다.