Agent Integration

Start with an observation-only adapter

The binary provides local actions; your agent chooses the task and decides when an action is authorized. The simplest integration doesn't need a model SDK, an API key, or any shell command generation.

Save the following as observe.py in the checkout from Quick start, then run python3 observe.py. It accepts exactly one structured action, starts the binary with an argument array, checks the exit status, decodes the output, and prints the path of observation.png.

import base64
import binascii
import subprocess
from pathlib import Path

binary = Path("target/release/nano-use").resolve(strict=True)

def observe(action):
    if action != {"command": "screenshot"}:
        raise ValueError("This adapter accepts screenshot only")
    result = subprocess.run(
        [str(binary), "screenshot"],
        shell=False, check=True, capture_output=True, timeout=30,
    )
    try:
        png = base64.b64decode(result.stdout.strip(), validate=True)
    except binascii.Error as error:
        raise RuntimeError("Invalid base64 output") from error
    if not png.startswith(b"\x89PNG\r\n\x1a\n"):
        raise RuntimeError("Output is not a PNG")
    Path("observation.png").write_bytes(png)
    return Path("observation.png").resolve()

print(observe({"command": "screenshot"}))

Open that PNG to inspect the observation. FileNotFoundError means the binary is missing or the working directory is wrong; CalledProcessError means capture failed; TimeoutExpired means the 30-second limit was reached. Don't continue to an input action after any of these failures.

Extend one command at a time

The example deliberately only accepts {"command": "screenshot"}. It will reject {"command": "type", "text": "hello"} or any additional fields—and that's intentional. This is an adapter example, not a complete agent or security sandbox.

When you add input commands, validate the command name, exact arity, finite coordinates, and bounded text length. Keep a fixed binary path and shell=False; never interpolate model text into shell code. Put -- before positional values that may start with -, as described in Commands.

Observe, act, verify

1. Capture and inspect the current desktop.
2. Identify a concrete target and make sure the next action fits the user's task.
3. Issue one allowed input action.
4. Capture again and verify the intended result before continuing.

A zero exit code doesn't prove the input event reached the intended window. Display scaling, focus, and permissions are still the caller's responsibility. The tool has no autonomous planner, remote transport, or MCP server. Sending a screenshot to a model is a separate operation handled by your integration.