Media understanding

기준일: 2026-07-26
공식 기준: Media understanding

Media understanding 문서는 OpenClaw 공식 문서(nodes/media-understanding)를 한국어로 정리한 가이드입니다. Inbound image/audio/video understanding (optional) with provider + CLI fallbacks 명령·설정 키·코드 예시는 공식 문서를 그대로 보존하며, 해석과 절차 안내는 한국어로 제공합니다. 최종 동작은 설치된 CLI 버전과 공식 원문을 확인하세요.

핵심 요약

Inbound image/audio/video understanding (optional) with provider + CLI fallbacks

한국어 가이드 범위: nodes/media-understanding 경로의 설정·명령·제약·예시를 학습용으로 재구성합니다.

문서 구성

공식 문서의 주요 섹션은 다음과 같습니다.

• 동작 방식
• 구성
• Model entries
• Provider credentials
• Rules and behavior
• Auto-detect (default)
• Proxy support (audio/video provider calls)
• Capabilities
• Provider support matrix
• Model selection guidance
• Attachment policy
• File-attachment extraction
• Config examples
• Status output
• 참고
• 관련 문서

상세 내용

본문

OpenClaw can summarize inbound media (image/audio/video) before the reply pipeline runs, so command parsing and routing work off short text instead of raw bytes. Understanding auto-detects local tools or provider keys, or you can configure explicit models. Original media is always delivered to the model as usual; when understanding fails or is disabled, the reply flow continues unchanged.

위 내용은 공식 문서의 해당 섹션 요지입니다. 세부 플래그·기본값은 원문과 --help를 확인하세요.

동작 방식

Collect ordered inbound media facts (path, url, contentType, and kind).

위 내용은 공식 문서의 해당 섹션 요지입니다. 세부 플래그·기본값은 원문과 --help를 확인하세요.

구성

tools.media holds one capability-tagged model list plus small per-capability controls:

위 내용은 공식 문서의 해당 섹션 요지입니다. 세부 플래그·기본값은 원문과 --help를 확인하세요.

Key	Type	Default	Notes
enabled	boolean	auto (false disables)	Set false to turn off auto-detect for this capability
preferredModel	string	first compatible entry	Prefer provider/model, model id, provider:<id>, or cli:command
prompt	string	capability default	Default prompt when an entry does not override it
maxChars	number	500 image/video, unset audio	Default output limit
maxBytes	number	10MB image, 20MB audio, 50MB video	Default input limit
timeoutSeconds	number	60 image/audio, 120 video	Default request timeout
language	string	unset	Audio transcription hint
scope	object	unset	Gate by channel/chat type/source key
attachments	object	{ mode: "first", maxAttachments: 1 }	Select which matching attachments are processed
echoTranscript	boolean	false	Audio only: echo the transcript before agent processing
echoFormat	string	'📝 "{transcript}"'	Audio only: format for the echoed transcript

{
  tools: {
    media: {
      concurrency: 2, // max concurrent capability runs (default)
      models: [
        { provider: "openai", model: "gpt-4o-mini-transcribe", capabilities: ["audio"] },
        { provider: "google", model: "gemini-3-flash-preview", capabilities: ["image", "video"] },
      ],
      image: { preferredModel: "google/gemini-3-flash-preview" },
      audio: { enabled: true },
      video: { enabled: true },
    },
  },
}

Model entries

Each models[] entry is a provider entry (default) or a CLI entry:

위 내용은 공식 문서의 해당 섹션 요지입니다. 세부 플래그·기본값은 원문과 --help를 확인하세요.

{
      type: "provider", // default if omitted
      provider: "openai",
      model: "gpt-5.6-sol",
      prompt: "Describe the image in <= 500 chars.",
      maxChars: 500,
      maxBytes: 10485760,
      timeoutSeconds: 60,
      capabilities: ["image"],
      profile: "vision-profile",
      preferredProfile: "vision-fallback",
    }

{
      type: "cli",
      command: "gemini",
      args: [
        "-m",
        "gemini-3-flash",
        "--allowed-tools",
        "read_file",
        "Read the media at {{AttachmentPath}} and describe it in <= {{MaxChars}} characters.",
      ],
      maxChars: 500,
      maxBytes: 52428800,
      timeoutSeconds: 120,
      capabilities: ["video", "image"],
    }

Provider credentials

Provider media understanding uses the same auth resolution as normal model calls: auth profiles, environment variables, then models.providers..apiKey. tools.media.models[] entries do not accept an inline apiKey field.

위 내용은 공식 문서의 해당 섹션 요지입니다. 세부 플래그·기본값은 원문과 --help를 확인하세요.

{
  models: {
    providers: {
      openai: { apiKey: "<OPENAI_API_KEY>" },
      moonshot: { apiKey: "<MOONSHOT_API_KEY>" },
    },
  },
}

Rules and behavior

주요 항목:

• Media exceeding maxBytes skips that model and tries the next one.
• Audio files under 1024 bytes are treated as empty/corrupt and skipped before transcription; the agent gets a deterministic placeholder transcript instead.
• If the active primary image model already supports vision natively, OpenClaw skips the [Image] summary block and passes the original image into the model directly. MiniMax is an exception: minimax, minimax-cn, minimax-portal, and minimax-portal-cn always route image understanding through the plugin-owned MiniMax-VL-01 media provider, even if legacy MiniMax M2.x chat metadata claims image input (only MiniMax-M3 and later are treated as natively vision-capable).
• If a Gateway/WebChat primary model is text-only, image attachments are preserved as offloaded media://inbound/* refs so image/PDF tools or a configured image model can still inspect them instead of losing the attachment.
• Explicit openclaw infer image describe --file <path> --model <provider/model> (alias: openclaw capability image describe) runs that image-capable provider/model directly, including Ollama refs such as ollama/qwen2.5vl:7b when a matching image-capable model is configured under models.providers.ollama.models[].
• If <capability>.enabled is not false but no models are configured, OpenClaw tries the active reply model when its provider supports the capability.

Auto-detect (default)

When tools.media..enabled is not false and no models are configured, OpenClaw tries these in order and stops at the first working option:

위 내용은 공식 문서의 해당 섹션 요지입니다. 세부 플래그·기본값은 원문과 --help를 확인하세요.

주요 항목:

• whisper-cli first only after an earlier model invocation in the current process observed Metal or CUDA
• CPU-default sherpa-onnx-offline (requires SHERPA_ONNX_MODEL_DIR with tokens.txt/encoder.onnx/decoder.onnx/joiner.onnx)
• whisper-cli when acceleration is merely build-capable or unobserved
• parakeet-mlx on Apple Silicon (MLX-capable, device use unobserved)
• whisper (Python CLI; defaults to the turbo model, downloads automatically)
• Image: Anthropic/OpenAI → Google → MiniMax → Deepinfra → MiniMax Portal → Z.AI
• Video: Google → Qwen → Moonshot

{
  tools: {
    media: {
      audio: {
        enabled: false,
      },
    },
  },
}

Proxy support (audio/video provider calls)

Provider-based audio and video understanding honors standard outbound proxy environment variables, including NO_PROXY/no_proxy bypass rules: HTTPS_PROXY, HTTP_PROXY, ALL_PROXY, https_proxy, http_proxy, all_proxy. Lowercase vars take precedence over uppercase. If none are set, media understanding uses direct egress; if the proxy value is malformed, OpenClaw logs a warning and falls back to direct fetch. Image understanding does not go through this proxy path.

위 내용은 공식 문서의 해당 섹션 요지입니다. 세부 플래그·기본값은 원문과 --help를 확인하세요.

Capabilities

Set capabilities on a models[] entry to restrict it to specific media types. For shared lists, OpenClaw infers defaults per bundled provider:

위 내용은 공식 문서의 해당 섹션 요지입니다. 세부 플래그·기본값은 원문과 --help를 확인하세요.

Provider	Capabilities
openai, anthropic, minimax	image
minimax-portal	image
moonshot	image + video
openrouter	image + audio
google (Gemini API)	image + audio + video
qwen	image + video
deepinfra	image + audio
mistral	audio
zai	image
groq, xai, deepgram, senseaudio	audio
Any models.providers.<id>.models[] catalog with an image-capable model	image

Provider support matrix

Capability	Providers	Notes
Image	Anthropic, Codex app-server, Deepinfra, Google, MiniMax, MiniMax Portal, Moonshot, OpenAI, OpenAI Codex OAuth, OpenRouter, Qwen, Z.AI, config providers	Vendor plugins register image support; openai/* can use API-key or Codex OAuth routing; codex/* uses a bounded Codex app-server turn; image-capable config providers auto-register.
Audio	Deepgram, Deepinfra, ElevenLabs, Google, Groq, Mistral, OpenAI, OpenRouter, SenseAudio, xAI	Provider transcription (Whisper/Groq/xAI/Deepgram/OpenRouter STT/Gemini/SenseAudio/Scribe/Voxtral).
Video	Google, Moonshot, Qwen	Provider video understanding via vendor plugins; Qwen video understanding uses the standard DashScope endpoints.

Model selection guidance

주요 항목:

• Prefer the strongest current-generation model for each media capability when quality and safety matter.
• For tool-enabled agents handling untrusted inputs, avoid older/weaker media models.
• Keep at least one fallback per capability for availability (quality model + faster/cheaper model).
• CLI fallbacks (whisper-cli, whisper, gemini) help when provider APIs are unavailable.
• Known file-output modes are authoritative: an empty or missing inferred transcript file produces no transcript instead of falling back to CLI progress output.
• parakeet-mlx: use --output-format txt (or all) with --output-dir and the default {filename} output template. The upstream PARAKEET_OUTPUT_FORMAT and PARAKEET_OUTPUT_TEMPLATE environment variables are also honored. OpenClaw reads <output-dir>/<media-basename>.txt; the default srt format, other formats, and custom output templates continue to use stdout.

Attachment policy

Per-capability attachments controls which attachments are processed:

위 내용은 공식 문서의 해당 섹션 요지입니다. 세부 플래그·기본값은 원문과 --help를 확인하세요.

File-attachment extraction

주요 항목:

• Extracted file text is wrapped as untrusted external content before it's appended to the media prompt, using boundary markers like <<<EXTERNAL_UNTRUSTED_CONTENT id="...">>> / <<<END_EXTERNAL_UNTRUSTED_CONTENT id="...">>> plus a Source: External metadata line.
• This path intentionally omits the long SECURITY NOTICE: banner to keep the media prompt short; the boundary markers and metadata still apply.
• A file with no extractable text gets [No extractable text].
• If a PDF falls back to rendered page images, OpenClaw forwards those images to vision-capable reply models and keeps the placeholder [PDF content rendered to images] in the file block.

Config examples

{
      tools: {
        media: {
          models: [
            { provider: "openai", model: "gpt-5.6-sol", capabilities: ["image"] },
            {
              provider: "google",
              model: "gemini-3-flash-preview",
              capabilities: ["image", "audio", "video"],
            },
            {
              type: "cli",
              command: "gemini",
              args: [
                "-m",
                "gemini-3-flash",
                "--allowed-tools",
                "read_file",
                "Read the media at {{AttachmentPath}} and describe it in <= {{MaxChars}} characters.",
              ],
              capabilities: ["image", "video"],
            },
          ],
          audio: {
            attachments: { mode: "all", maxAttachments: 2 },
          },
          video: {
            maxChars: 500,
          },
        },
      },
    }

{
      tools: {
        media: {
          audio: {
            enabled: true,
            models: [
              { provider: "openai", model: "gpt-4o-mini-transcribe" },
              {
                type: "cli",
                command: "whisper",
                args: ["--model", "base", "{{AttachmentPath}}"],
              },
            ],
          },
          video: {
            enabled: true,
            maxChars: 500,
            models: [
              { provider: "google", model: "gemini-3-flash-preview" },
              {
                type: "cli",
                command: "gemini",
                args: [
                  "-m",
                  "gemini-3-flash",
                  "--allowed-tools",
                  "read_file",
                  "Read the media at {{AttachmentPath}} and describe it in <= {{MaxChars}} characters.",
                ],
              },
            ],
          },
        },
      },
    }

{
      tools: {
        media: {
          image: {
            enabled: true,
            maxBytes: 10485760,
            maxChars: 500,
            models: [
              { provider: "openai", model: "gpt-5.6-sol" },
              { provider: "anthropic", model: "claude-opus-5" },
              {
                type: "cli",
                command: "gemini",
                args: [
                  "-m",
                  "gemini-3-flash",
                  "--allowed-tools",
                  "read_file",
                  "Read the media at {{AttachmentPath}} and describe it in <= {{MaxChars}} characters.",
                ],
              },
            ],
          },
        },
      },
    }

{
      tools: {
        media: {
          image: {
            models: [
              {
                provider: "google",
                model: "gemini-3.1-pro-preview",
                capabilities: ["image", "video", "audio"],
              },
            ],
          },
          audio: {
            models: [
              {
                provider: "google",
                model: "gemini-3.1-pro-preview",
                capabilities: ["image", "video", "audio"],
              },
            ],
          },
          video: {
            models: [
              {
                provider: "google",
                model: "gemini-3.1-pro-preview",
                capabilities: ["image", "video", "audio"],
              },
            ],
          },
        },
      },
    }

Status output

When media understanding runs, /status includes a per-capability summary line:

위 내용은 공식 문서의 해당 섹션 요지입니다. 세부 플래그·기본값은 원문과 --help를 확인하세요.

📎 Media: image ok (openai/gpt-5.6-sol) · audio ok (whisper-cli observed=metal)

openclaw doctor --lint --only core/doctor/local-audio-acceleration --severity-min info

참고

주요 항목:

• Understanding is best-effort. Errors do not block replies.
• Attachments are still passed to models even when understanding is disabled.
• Use scope to limit where understanding runs (예를 들어, only DMs).

관련 문서

주요 항목:

• Configuration
• Image & media support

실습 체크리스트

1. 공식 문서와 로컬 버전을 대조합니다: https://docs.openclaw.ai/nodes/media-understanding
2. 관련 CLI는 openclaw --help 및 하위 명령 --help로 옵션을 확인합니다.
3. 설정 변경 시 openclaw config / openclaw doctor로 유효성을 검사합니다.
4. Gateway·채널·플러그인 변경 후에는 필요 시 Gateway를 재시작합니다.

자주 쓰는 명령·설정 예시

{
  tools: {
    media: {
      concurrency: 2, // max concurrent capability runs (default)
      models: [
        { provider: "openai", model: "gpt-4o-mini-transcribe", capabilities: ["audio"] },
        { provider: "google", model: "gemini-3-flash-preview", capabilities: ["image", "video"] },
      ],
      image: { preferredModel: "google/gemini-3-flash-preview" },
      audio: { enabled: true },
      video: { enabled: true },
    },
  },
}

{
      type: "provider", // default if omitted
      provider: "openai",
      model: "gpt-5.6-sol",
      prompt: "Describe the image in <= 500 chars.",
      maxChars: 500,
      maxBytes: 10485760,
      timeoutSeconds: 60,
      capabilities: ["image"],
      profile: "vision-profile",
      preferredProfile: "vision-fallback",
    }

{
      type: "cli",
      command: "gemini",
      args: [
        "-m",
        "gemini-3-flash",
        "--allowed-tools",
        "read_file",
        "Read the media at {{AttachmentPath}} and describe it in <= {{MaxChars}} characters.",
      ],
      maxChars: 500,
      maxBytes: 52428800,
      timeoutSeconds: 120,
      capabilities: ["video", "image"],
    }

{
  models: {
    providers: {
      openai: { apiKey: "<OPENAI_API_KEY>" },
      moonshot: { apiKey: "<MOONSHOT_API_KEY>" },
    },
  },
}

{
  tools: {
    media: {
      audio: {
        enabled: false,
      },
    },
  },
}

{
      tools: {
        media: {
          models: [
            { provider: "openai", model: "gpt-5.6-sol", capabilities: ["image"] },
            {
              provider: "google",
              model: "gemini-3-flash-preview",
              capabilities: ["image", "audio", "video"],
            },
            {
              type: "cli",
              command: "gemini",
              args: [
                "-m",
                "gemini-3-flash",
                "--allowed-tools",
                "read_file",
                "Read the media at {{AttachmentPath}} and describe it in <= {{MaxChars}} characters.",
              ],
              capabilities: ["image", "video"],
            },
          ],
          audio: {
            attachments: { mode: "all", maxAttachments: 2 },
          },
          video: {
            maxChars: 500,
          },
        },
      },
    }

관련 링크

• 공식 원문: nodes/media-understanding
• OpenClaw 문서 홈

이 가이드는 공식 문서를 한국어 학습용으로 재구성한 것입니다. 옵션 기본값·플래그 이름은 설치 버전에 따라 달라질 수 있습니다.