Aiden: a physical AI agent that controls phones over USB HID

Most GUI agent work assumes the agent lives inside the device it operates — installed as an app, granted accessibility permissions, or driven through a debugging bridge like ADB. That assumption is convenient, but it constrains what you can automate and forces the host to trust software running on it. We went the other way.

Aiden is a physical mobile AI agent device: a small board that sits outside the host, plugs in over USB, and operates the phone or computer the way a person would. It sees the screen, decides what to do, and acts — without anything installed on the target device.

This post covers how it works, why the architecture is shaped the way it is, and the problems we haven't solved yet.

The core idea: perceive out-of-band, act as a human input device

Aiden separates the two things a GUI agent has to do — see the screen and act on it — onto two independent channels, neither of which requires cooperation from the host OS:

• Perception happens over HDMI. The board captures the host's display through an HDMI-to-CSI bridge. To the host, this looks like an ordinary external monitor. It has no idea it's being read.
• Action happens over USB HID. The board presents itself as a standard USB keyboard and mouse. When the agent decides to tap or type, it writes to the HID device nodes, and the host receives what looks like ordinary human input.

The host sees a monitor and a keyboard/mouse. Nothing else. No app to install, no root, no ADB, no accessibility API, no cloud round-trip. If a device can output video and accept USB input, Aiden can drive it — which is most phones and computers people already own.

This is the key difference from the dominant approaches. Software agents need something on the device (an app, a permission, a debug bridge). Cloud "computer use" agents stream a virtual screen to a remote model and act inside a controlled environment. Aiden does neither: the perception-action loop closes locally, on hardware, against a real physical device.

Runtime architecture

The agent runtime is written in Go. The design leans on goroutines to keep the four concurrent concerns independent and non-blocking:

• Frame capture — pulls the display feed from the capture pipeline and hands screenshots to the agent loop.
• Audio — full-duplex, with voice activity detection (VAD) so the device can take spoken instructions and respond without a wake word.
• The agent loop — sends the current screen (and any voice input) to a configured multimodal model, gets back a decision, and translates it into concrete input events.
• HID output — writes those events to the USB HID gadget, so they land on the host as keyboard and pointer actions.

Hardware-adjacent services (the HDMI capture path, audio recording and playback) are handled in a lower-level C++ layer; the agent logic, skills, and memory live in Go.

There is no backend. All model calls go to whatever endpoint you configure — a hosted API, a local model, your own deployment. Aiden collects nothing. This isn't a marketing line about privacy; it's the only defensible design when the input to your system is a live feed of someone's phone screen. The moment you route that through a server you control, you've created a target and a liability. Keeping the loop on-device — self-hostable, auditable, model-agnostic — is what makes the whole premise acceptable.

Open questions we haven't solved

This is a dev board and an early runtime, not a finished product. The honest part of the writeup is the part that isn't working yet:

• Action verification. The agent infers whether an action succeeded by re-reading the screen. That breaks the moment a loading state lies to you — a spinner that resolves to an error, an optimistic UI that shows success before the network confirms it, a screen that looks done but isn't. Reliable success detection from pixels alone is unsolved here.
• Prompt injection. An agent that reads screens reads whatever an attacker puts on them. A malicious notification, a crafted web page, text planted in an app — any of it can become instructions the agent wasn't meant to follow. Screen-reading agents inherit the entire injection surface of everything they look at, and we don't have a general defense.
• iOS pointer control. Driving an iPhone via HID requires AssistiveTouch to be enabled. It works, but it's a setup step and a caveat, not the zero-configuration story it is on other targets.

None of these are unique to Aiden — they're structural problems for any agent that operates real devices by looking at them. But putting the agent on external hardware makes them concrete rather than abstract, which is part of why we open-sourced it.

Try it / read the code

The repository includes the USB HID gadget configuration, the HDMI capture pipeline, and the Go agent runtime:

github.com/AidenAI-IO/aiden-hardware-demo

If you want the conceptual comparison rather than the code — how a physical, HID-based mobile agent differs from cloud-based computer use agents — we wrote that up separately:

Mobile AI Agent vs Computer Use Agent: What's the Difference?

---

Note: the current hardware is a development board, not a finished consumer product. The architecture described here is real and runnable; the form factor is still evolving. Aiden is open source — bring your own model, self-host the whole thing, and read every line that touches the screen feed. Project home: aidenai.io · @AidenAI_IO