Chalk
A Mac screenshot and markup tool for people and agents: every box, arrow and pin reaches the agent as data, not just pixels
- Role
- Sole designer and builder
- Status
- In development
- Period
- Nov 2026
Swift · AppKit · ScreenCaptureKit · Accessibility API · Vision · SQLite · FTS5 · SwiftPM · MCP
Problem
When I show an agent something on my screen, I hand it a PNG and it guesses which button I circled, in which app, on which page. Going the other way, an agent has no good way to point at something on my screen and ask.
Mac capture apps are fast and polished, but their marks are pixels. The tools that give agents structured annotations work only inside a web page they own. Nothing captured any Mac app and recorded what was under each mark.
What it does
Press Shift-Cmd-2 and the screen freezes. Drag over any app and let go. Draw a box, an arrow, a highlight, a numbered pin, text or a redaction, type a comment on any mark, and press Cmd-Return. The person gets a marked image on the clipboard. The agent gets the same shot as a bundle on disk: shot.png, marked.png and a marks.json with each mark's position and comment, the app, window title, page URL and the accessibility element under it.
It also works the other way. An agent calls annotate with its marks and a question, Chalk opens a review window on my screen, and my reply comes back as data through await_review. Also:
- History: every shot is kept and searchable by app, window, URL, comment and the text in the picture, read on device with OCR.
- Scrolling capture: a selection can grow into a tall capture that scrolls by itself, and I can take over by scrolling.
- Agent access: the
chalkCLI (capture,marks,latest,get,search) andchalk mcp, a stdio MCP server for Claude Code and other clients.
Architecture
Capture. The hotkey freezes every display in parallel through ScreenCaptureKit. Context is read before the overlay opens, because once it is open the overlay is what accessibility returns.
Context. Each mark records the most specific accessibility element at its anchor point. Browser URLs come from the accessibility tree, with an AppleScript fallback. Any value it cannot read is null with a *_reason beside it, never a guess.
The agent path. macOS attributes a command-line tool's permissions to the app that launched it, so the CLI asks the running app over a local Unix socket and the app captures with its own grants. Every agent capture shows on screen, and agent capture can be switched off from the menu.
Key decisions
-
The marks are a frozen contract.
marks.jsonhas a JSON Schema that forbids extra fields, and readers check its id. When the second milestone needed OCR text, replies and review state, I leftmarks/v0unchanged byte for byte and put the additions in a sidecar,extra.json, so v0 readers keep working. -
No Xcode. Chalk builds with SwiftPM and the Command Line Tools alone. That SDK has no SwiftUI macro plugin, so the app is plain AppKit.
-
Nothing leaves the Mac. There is no network code and no network entitlement, and a test fails if any network code appears in the sources. OCR runs through Vision on device.
-
A redaction hides text, not only pixels. A redact is burned into both images, and every text that came from under it is dropped: another mark's element keeps only its role, and the window title and URL go when the redact covers them.
-
Hand-off by clipboard and files. Cmd-Return puts the image, the marks and a text summary on the clipboard and writes the bundle to disk. Any agent that accepts a paste works.
Numbers
Measured inside the installed app on my Mac with two 6400x3600 displays; the overlay figures are over 50 runs of the hotkey path:
| Measurement | Result |
|---|---|
| Hotkey to frozen overlay, first milestone | p50 61 ms, p95 80 ms, max 128 ms (target p95 under 150 ms) |
| Hotkey to frozen overlay, second milestone | p95 118 ms |
| Per-display capture | p95 47.9 ms and 40.4 ms |
Agent annotate to review window, 20 runs | p95 55 ms |
| History search over 200 shots | p95 2.1 ms |
| OCR on 12 fixture images | 100% word recall, median 43 ms |
| Full-screen OCR, 6400x3600 synthetic | 257 ms down to 127 ms |
Browser URLs came back on 19 of 20 shots. A specific element, not a window or bare group, resolved at 9 of 9 grid points in each of six apps swept. The test suite went from 204 tests at the first milestone to 464 at the second.
What broke
- The redaction leaked text, twice. The first independent review found that a redact blacked out pixels while the window title, URL and other marks' element values still reached
marks.json, the clipboard and MCP. I fixed it with a test that the redacted string appears nowhere. In the second milestone it came back: the review window had its own bundle writer, so a redact there only touchedmarked.png. It now saves through the shared edit path. - Rebuilds lost the permissions. Each ad-hoc signed build changed the code signature, so macOS dropped the Screen Recording grant every time. A stable local signing identity fixed that.
- The first flow was not obvious. I could select an area but not tell how to capture it. The editor now opens on mouse-up with a visible toolbar and a comment field on each mark.
- OCR was far too slow on big shots. A full Retina screen took about 39 s. Now each mark's area is read at its mouse-up, the clipboard waits at most 150 ms for those reads, and the whole-shot read runs afterwards at 1x on one background queue.
- Abuse cases fell between workstreams. An agent could flood me with reviews, abandoned long polls held socket slots, and a pending review was lost on relaunch. Reviews are now capped, restored and expired.
Results
Two milestones shipped in two days, 16 and 17 November, across 24 pull requests. The build is installed on my Mac, and I tried the review window, History and scrolling capture on screen and accepted the second milestone. It is on hold until 25 November.
The repository is private and there is no release. Chalk runs only on my own Macs, signed ad hoc or with a local signing identity. The public site describes it and takes waitlist sign-ups.
What I'd change
- Elements are read at save, not at freeze. A page that scrolls or updates while I mark can give elements that do not match the pixels. The marks and the image are always exact; the element data can drift.
- Chromium and Electron cost. Reading them keeps their full accessibility tree on, which costs some CPU until the app quits.
- Review leftovers. Some lower-priority review findings are still open, including tiling for very tall scroll captures.