Drawgent: Coding agent on a live Excalidraw canvas

README.md

drawgent: your coding agent on a live Excalidraw canvas#

drawgent connects your own Claude Code, Codex or opencode (your install, login, config and repo) to an Excalidraw whiteboard. Ask for a diagram in the chat panel, or write AGENT: … next to the part of a drawing you want changed. The agent looks at the canvas (screenshot + scene), edits it live, checks the result, and marks the note DONE.

Quick start#

Prerequisite: one of claude, codex or opencode installed and logged in. The Claude and Codex bridges also need Node.js ≥ 18 (npm).

drawgent setup claude        # once per agent: claude | codex | opencode
cd ~/my-repo
drawgent up                  # new agent session in this repo + canvas in your browser
drawgent up --attach         # or: pick one of your running sessions and connect the canvas to it

drawgent setup <agent>#

Checks everything once, fails with the exact fix when something is missing, and writes ~/.config/drawgent/config.toml:

1. The agent CLI. Your claude / codex / opencode on PATH.

2. Login. claude auth status, codex login status or opencode auth list.

3. ACP bridge.

   • opencode speaks ACP itself (opencode acp).
   • Claude Code and Codex use the official ACP adapters. They are installed once into ~/.cache/drawgent/adapters (~60 MB), without their bundled agent binaries, and pointed at your CLI (CLAUDE_CODE_EXECUTABLE, CODEX_PATH).
   • Setup then verifies the ACP handshake.

4. Canvas tools for attached sessions. Only Codex needs a change: codex mcp add drawgent -- drawgent mcp. Claude and opencode get the tools at attach time.

5. Headless Chrome for the renderer. Setup uses your Chrome/Chromium if you have one. Otherwise it proposes:

   • downloading Chrome Headless Shell (Chrome for Testing, ~120 MB, no sudo) into ~/.cache/drawgent/chrome, and telling you exactly which system libraries are missing, if any;
   • installing Chromium with your package manager (apt, snap, dnf, pacman, zypper, apk, brew, or nix without sudo).

   Non-interactive: --chrome download | system | /path/to/chrome.

drawgent up refuses to start until setup succeeded for that agent, or if something setup recorded disappeared.

drawgent up#

Runs in the current directory (the workspace):

• starts the editor + API on 127.0.0.1:7300 (next free port if taken);
• starts a new session of your agent over ACP, working in the workspace;
• opens your browser. On a headless box it prints the ssh -L command instead.

The scene is kept in .drawgent/scene.json, which is git-ignored automatically.

drawgent up --attach [id]#

Connects the canvas to a session you already run. Without an id it lists the sessions it finds (current directory first) and lets you pick one:

Using the canvas#

• Chat panel (right side): send a request, watch replies and tool calls stream in, approve permission prompts, press Stop to cancel a turn.
• On the canvas: write a text starting with AGENT: next to or inside a shape, or draw an arrow from the note to a shape. It fires about 2.5 s after you stop typing, with its position, what it points at and what is nearby. The agent resolves it into a green DONE: … note. Edit it back to AGENT: to send it again.
• Prompts are queued and run one at a time. A queued note that was already handled is skipped.

drawgent up --room 'https://excalidraw.com/#room=<id>,<key>'

• drawgent joins the room as a collaborator (🤖 Agent, with a cursor that follows its edits). Humans can stay on excalidraw.com: their AGENT: notes reach your agent, and its edits appear there live.
• Traffic is end-to-end encrypted with the room key. Empty rooms are loaded from and saved to excalidraw's Firestore storage.
• The local editor mirrors the room and still has the chat panel.

Other commands#

• drawgent mcp: stdio MCP server with the canvas tools. Agents launch it; it finds the running drawgent up by itself.
• drawgent serve: low-level server with explicit agents (--agent claude,codex,opencode as set up, or name=command for any ACP agent); for scripts and containers.
• Options for up / serve: --port, --room, --token (API bearer + ?token= in the URL), --permissions canvas|ask|all (default canvas: drawing tools auto-approved, anything else asked in the chat panel), --data, --settle-ms.

Optional: Docker#

docker compose up --build runs a canvas server (drawgent + Chromium, no agents), e.g. to host a shared canvas or a room bridge on a server. Agents are never bundled: they always run with your own setup.

Build from source#

npm ci && npm run build      # editor + renderer pages -> dist/ (embedded into the binary)
cargo install --path .       # or: cargo build --release

Agent tools (MCP)#

get_scene, get_screenshot (vision; zoom with element_ids), add_elements (Excalidraw skeletons; arrows bind by id and are routed edge-to-edge), add_mermaid (auto-layout), update_elements (labels follow shapes, bound arrows re-route), delete_elements, clear_canvas, list_instructions, resolve_instruction, set_status.

API#

GET /api/health · GET /api/scene · GET /api/screenshot?ids=&padding=&max= · POST|PATCH|DELETE /api/elements · POST /api/mermaid · POST /api/clear · GET /api/instructions · POST /api/instructions/{id}/resolve · POST /api/status · POST /api/chat {agent?, text} · WS /ws (browser sync + chat events)

Tests#

cargo test                          # fractional indices, room crypto/framing
node scripts/smoke.mjs [url]        # chat turn + AGENT: note against a running drawgent
node scripts/e2e-browser.mjs [url]  # real browser: chat panel + note typed on the canvas
node scripts/room-e2e.mjs           # fresh excalidraw.com room ↔ drawgent, both directions

Layout#

src/ (Rust):

• main.rs: CLI.
• setup.rs, config.rs, chrome.rs: setup, config, renderer install.
• attach.rs: session discovery and picker.
• agents.rs + acp.rs: ACP driver (new / fork sessions).
• live.rs: live opencode / Codex drivers.
• hub.rs: routing, notes, chat log.
• scene.rs: store and edit operations.
• renderer.rs: Chrome over CDP.
• mcp.rs: MCP server.
• room.rs: excalidraw.com client.
• fractional.rs, geometry.rs, el.rs: helpers.

web/: editor (main.jsx, chat.jsx) and renderer page (render.jsx).

Limits#

• The renderer needs Chrome (a native renderer is planned).
• Claude "attach" is a fork, because Claude Code has no public way to inject into a running terminal session.
• Codex live attach is implemented, but was not yet tested against a logged-in Codex.
• One scene per workspace. Images/files are not synced.