Ghost OS
1,200+ starsFull computer-use for AI agents on macOS. One install and any AI agent can see and operate every app on your Mac.
AI agents live in chat boxes.
Your AI agent can write code, run tests, search files. But it can't click a button, send an email, fill out a form, organize files, or use any of the apps you use every day. It lives inside a terminal.
The existing solutions (Anthropic Computer Use, OpenAI Operator) take screenshots and guess what's on screen. That's slow, fragile, and expensive. Every action costs a full vision model inference. Operator is browser-only. Computer Use needs a VM.
Ghost OS takes a different approach. It reads the macOS accessibility tree: structured, labeled data about every element in every app. It knows what's a button, what's a text field, what's a link. No guessing.
How it works
Ghost OS reads your screen like structured data, not pixels. When the accessibility tree falls short, it cascades through fallbacks until it finds the element.
Accessibility tree (native)
Query the macOS AX API for structured element data. Buttons, fields, labels, positions, available actions. 50-500ms. Works with every native app.
Chrome DevTools Protocol
For web apps where Chrome flattens elements to AXGroup, fall back to CDP to query the DOM directly. Gets accurate element data from the page.
ShowUI-2B vision model
When structured data is unavailable, a local vision model (ShowUI-2B) runs on Apple Silicon via MLX. Visual grounding finds elements by description. Runs entirely on-device.
Synthetic input
As a last resort, generate CGEvent synthetic input at coordinates. System-level events that work in any app, any context.
Self-learning recipes
Show the agent once, it remembers forever. The most differentiated feature in Ghost OS.
Transparent
Recipes are just JSON. Read every step before running. No black box.
Shareable
One person learns the workflow, everyone benefits. Share with your team.
Cost-efficient
Write once with Claude or GPT-4. Run forever with Haiku. No repeat reasoning costs.
See it working
Sending an email and downloading papers from a single prompt. The recipe was learned once and runs instantly every time after.
29 MCP tools
A complete toolkit for computer use. Perceive, act, navigate, wait, learn, and replay. All through the open MCP protocol.
Perceive
ghost_context, ghost_state, ghost_find, ghost_read, ghost_inspect, ghost_screenshot, ghost_annotate
Act
ghost_click, ghost_type, ghost_hover, ghost_drag, ghost_long_press
Navigate
ghost_scroll, ghost_press, ghost_hotkey
Windows
ghost_window, ghost_focus
Wait
ghost_wait
Recipes
ghost_recipes, ghost_run, ghost_recipe_save, ghost_recipe_show, ghost_recipe_delete
Learn
ghost_learn_start, ghost_learn_stop, ghost_learn_status
Vision
ghost_ground, ghost_parse_screen, ghost_element_at
Why Ghost OS
Accessibility tree, not screenshots
Every macOS app exposes a structured hierarchy of every UI element: buttons, text fields, links, labels. Ghost OS reads this data in 50-500ms. Vision models take 3 seconds per inference. Structured data over pixel guessing. Faster, more reliable, cheaper.
Self-learning recipes
Show Ghost OS how to do something by doing it yourself. It watches through the accessibility tree and CGEvent tap, captures every click and keystroke with full element context, then synthesizes a reusable recipe. A frontier model figures it out once. A small model runs it forever.
Any app, not just browsers
Slack, Finder, Messages, System Settings, Xcode, Preview, Notes. Anything on your Mac with an accessibility tree. Cross-app workflows (copy from Chrome, paste in TextEdit) just work. Not limited to the browser like Operator or OpenClaw.
Local and private
Everything runs on your Mac. The vision model (ShowUI-2B) runs locally via MLX on Apple Silicon. Your data never leaves your machine. Recipes are local JSON files you can read, share, and chain together. Open source, MIT licensed.
Works with your tools
Ghost OS speaks MCP, the open standard for AI tool interfaces. Connect it to any MCP client and your agent gains computer use.
Get started
Two lines. No Docker, no VM, no config files. You're automating your Mac in 60 seconds.
Requires macOS 14+ (Sonoma) on Apple Silicon. Ghost OS will request Accessibility and Screen Recording permissions during setup. For self-learning recipes, Input Monitoring is also needed.
Run ghost doctor to check all permissions, processes, configs, and models.
Your AI agent can write code.
Ghost OS lets it use the rest of your computer.