A hotkey has a latency budget that a chat window does not. Below roughly 400 ms an edit feels like part of your keystroke. Above about 800 ms you start watching for it, and once you are watching for it you may as well have opened a window.

We shipped at a p95 of 1.2 s. We are now at 380 ms. This is where the time was.

The original budget

Measured on an M2 Air over home wifi, replacing a 19-word selection:

Stage p50 p95
Read selection 12 ms 260 ms
Request setup (TLS, DNS) 40 ms 310 ms
Model time to first token 210 ms 380 ms
Model remaining tokens 180 ms 290 ms
Write replacement 8 ms 190 ms
Total 450 ms 1230 ms

Two things stand out. The model is roughly half the p50 but well under half the p95 — most of the tail was ours. And reading a 19-word selection should not cost 260 ms at p95.

Change 1: stop synthesising ⌘C when we do not have to

The 260 ms p95 on selection reads was the pasteboard fallback. We were using it more than necessary: the original code tried AXSelectedText on the focused element and fell back immediately on failure, but many apps expose a writable text element one or two levels below the element the system reports as focused.

Walking down kAXChildrenAttribute looking for a text role before giving up moved a meaningful set of apps — including Slack’s message composer — onto the fast path.

func selectedText(in element: AXUIElement, depth: Int = 0) -> String? {
  if let direct = copyAttribute(element, kAXSelectedTextAttribute) as String?,
     !direct.isEmpty { return direct }

  guard depth < 3 else { return nil }   // deeper than this is never a text field

  let children = copyAttribute(element, kAXChildrenAttribute) as? [AXUIElement]
  return children?.lazy.compactMap { selectedText(in: $0, depth: depth + 1) }.first
}

The depth cap matters. Without it, an unbounded walk of a Chrome accessibility tree costs more than the pasteboard round trip it was meant to avoid.

Selection read p95: 260 ms → 45 ms.

Change 2: warm the connection before the keypress

Forty milliseconds of DNS and TLS at p50, 310 ms at p95, all of it spent on work that could have happened while the user was still typing.

We now open and hold a connection when the app detects a text selection anywhere on the system — an event we are already subscribed to for the menu-bar indicator. By the time the hotkey arrives, the socket is warm.

// Kept alive with an idle timeout slightly above the observed
// server-side close, so the common case is zero handshakes.
config.httpMaximumConnectionsPerHost = 2
config.httpShouldUsePipelining = true

Request setup p95: 310 ms → 15 ms.

Change 3: stream, and write as tokens arrive

We were waiting for the complete response before touching the selection. For a 19-word rewrite that meant carrying the full generation time, not just time to first token.

Streaming into the selection is harder than streaming into a chat bubble, because every intermediate write goes onto the host app’s undo stack — you do not want ⌘Z to walk back through forty partial rewrites. The fix is to buffer until the first sentence boundary, write once, then coalesce subsequent tokens on a 60 ms timer and collapse the whole sequence into a single undo group.

Perceived completion p50: 450 ms → 240 ms.

Change 4: cap the prompt

Adherence to instructions starts degrading past roughly 150 words of system prompt, and so does time to first token. Our default prompt had grown to 310 words through a series of well-intentioned additions, each fixing a real edge case.

We cut it to 120 words and moved the edge cases into profiles, where they only cost latency for the people who need them.

Time to first token p50: 210 ms → 130 ms.

Where we ended up

Stage p50 p95
Read selection 9 ms 45 ms
Request setup 3 ms 15 ms
Model time to first token 130 ms 220 ms
Write first replacement 8 ms 40 ms
Perceived total 150 ms 380 ms

None of the four changes involved a faster model. Three of them were about not doing work at the moment the user is waiting, and the fourth was about asking for less.

That is the general shape of latency work on an interactive tool. The model is the part you cannot control and usually not the part costing you the tail.

What we are still carrying

The p95 selection read is 45 ms rather than 9 ms because a handful of Electron apps still require the pasteboard fallback, and there is no way around it without them implementing accessibility properly. We poll changeCount at 8 ms intervals, which is as tight as we can go without burning noticeable CPU on a laptop.

If you build a Mac app with a text surface, implementing AXSelectedText on your focused element is about thirty lines and makes every accessibility tool on the platform — screen readers very much included — work properly with it.

← All posts