Stream Deck integration with cmux

What this integration controls

Recommended architecture

Stream Deck hardware
  -> Elgato Stream Deck application
     -> native Stream Deck plugin
        -> cmux CLI / socket RPC
        -> GJC SDK WebSocket endpoints
        -> local worktree launcher
        -> generated mascot key art

A native plugin is the useful boundary: it can render dynamic titles, track key settings, subscribe to SDK events, guard GJC-only actions, and route every operation to the exact cmux surface.

Keep the editable plugin under a user-owned source directory and synchronize it to ~/Library/Application Support/com.elgato.StreamDeck/Plugins/<plugin>.sdPlugin/. Never commit local profiles, SDK state, API keys, endpoint tokens, or browser credentials.

Three-page layout

Page 1 — daily web shortcuts

Focus existing matching Chrome or Safari tabs. Create a Chrome tab only when no matching tab exists in either browser.

Page 2 — cmux and session entry

TAB PREV | TAB NEXT | NEW SESSION | CLOSE TAB | GJC FOCUS
PANE PREV | PANE NEXT | VOICE | STEER | ESC X2
BACK | PROJECT 1 | PROJECT 2 | HOME | NEXT

• NEW SESSION creates a terminal surface and asks for a worktree name. A blank answer starts a plain gjc session; a name starts gjc --worktree <name>. Model profiles are applied later from page 3.
• VOICE invokes local Whisper STT through the operator remap app.stt.toggle = Ctrl+H on the focused GJC: surface.
• GJC FOCUS submits proceed plus Enter only when the focused surface title starts with GJC:.
• STEER sends Escape, waits 100 ms, then sends Enter.
• ESC X2 sends Escape, waits 100 ms, then sends Escape again.
• The first two project keys are automatically bound to the repositories with the highest GJC session counts, merging the SDK session index with saved session headers and canonicalizing managed worktrees back to their parent repository. The third key always opens $HOME.

The portable source, profile page manifests, and icon assets live under integrations/streamdeck-cmux/ in the main repository. The installer derives runtime paths from $HOME, the plugin directory, and optional environment overrides; it does not embed operator-specific project paths.

Page 3 — focused GJC operations

SET FRONTIER | SET GPT | SET GLM DS | KIMI GPT | BTW EXPLAIN
RESUME | EXIT | PR TO DEV | THINK LEVEL | CLEAR CTX
BACK | DEEP INTERVIEW | RALPLAN | ULTRAGOAL | NEXT

Profile keys submit /model gajae-code/<profile> to the focused session. A profile such as kimi-gpt may be user-defined; it must exist in the operator's model configuration before the button can apply it.

PR TO DEV submits the convenience prompt make a PR targeting dev and make it LGTM. The prompt does not bypass repository rules, verification, review, or branch availability.

Skill keys type /skill:deep-interview, /skill:ralplan, or /skill:ultragoal without pressing Enter, leaving room for arguments.

Use cmux as the topology authority

CMUX=/Applications/cmux.app/Contents/Resources/bin/cmux

$CMUX identify --no-caller
$CMUX tree --all
$CMUX focus-panel --panel surface:7 --workspace workspace:1 --window window:1
$CMUX new-surface --type terminal --pane pane:1 --focus true
$CMUX close-surface --surface surface:7 --workspace workspace:1 --window window:1

Use cmux identify --no-caller for the actual focused window, workspace, pane, and surface. Tree decorations are useful for display, but they are not sufficient routing authority.

Send keyboard controls precisely

Text macros should use cmux send, followed by cmux send-key ... enter. Remap app.stt.toggle to Ctrl+H in ~/.gjc/agent/keybindings.json; the plugin then sends atomic ctrl+h to the focused surface. New sessions load the remap, while already-running sessions retain their startup keybindings and are left untouched. Send thinking-level changes as one atomic key event:

cmux send-key \
  --surface surface:7 \
  --workspace workspace:1 \
  --window window:1 \
  'shift+tab'

The expected terminal bytes are [27, 91, 90]. Sending escape-prefixed text through a text API can let the TUI consume the leading Escape first and report Operation aborted.

Turn questions into an answer pad

GJC sessions publish a token-authenticated loopback SDK endpoint under <cwd>/.gjc/state/sdk/. Resolve live GJC PIDs to their TTY and current working directory so managed .gajae-code-worktrees sessions are discovered as well. Treat the token as a credential and never log or persist it elsewhere.

For a focused session with one to five scalar options, temporarily replace all five top-row controls—the four profile keys plus BTW EXPLAIN—with ANSWER 1 through ANSWER 5. Render the real option labels with bounded wrapping and highlight a valid recommended index.

{
  "type": "action_needed",
  "id": "act_9e31",
  "kind": "ask",
  "question": "Choose a target",
  "options": ["A", "B"],
  "recommendedIndex": 1
}

Reply with the exact active presentation ID:

{
  "type": "reply",
  "id": "act_9e31",
  "answer": 1,
  "token": "<session token>",
  "idempotencyKey": "streamdeck-act_9e31-1"
}

For checkbox questions, negotiate ask_controls_v1. Render up to four options as ☐/☑ buttons and reserve the fifth top-row key for the typed navigation_forward control labeled Done or Next. An option press sends its numeric index against the exact current action ID; GJC reissues a fresh action with updated selectedOptionIndices. The final key sends { "controlId": "navigation_forward" } only when the negotiated control is enabled.

Restore the profile keys after action_resolved. Do not infer authority or controls from question text, option labels, workflow IDs, or an earlier presentation. Leave free-text, checkbox questions with five or more options, malformed controls, and other unsupported shapes to the native GJC UI.

Generate readable mascot keys

Use the official character artwork as the identity reference. Give every key a distinct pose, expression, prop, and action. Optimize for 144-by-144 output with dark backgrounds, strong silhouettes, short high-contrast labels, and generous safe margins.

Suitable scenes include pane dividers, terminal tabs, browser windows, model cores, emergency controls, git branches, approval checks, interview notebooks, planning blueprints, and goal summits. Dim the art behind dynamic session or folder titles.

Verification checklist

• Back up the Stream Deck profile before changing page manifests.
• Build the plugin with Bun and compare the source and installed copies.
• Confirm every referenced image exists and every dynamic-title key has titles enabled.
• Use temporary cmux surfaces to verify navigation, browser creation, folder entry, tab close, exact macros, Shift+Tab, steer, and abort.
• Use a temporary SDK server to verify question rendering, recommended highlighting, exact zero-based replies, and resolution restoration.
• Restore the original cmux focus after every destructive or visual test.
• Restart Stream Deck and confirm every plugin-driven key renders.

AI skill workflow

The source repository carries the long-form guide in an installable skill-template structure. An AI operator following that skill should first inventory hardware, application paths, profiles, cmux topology, GJC endpoints, and existing backups; then make one bounded layout change at a time; then prove the exact observable behavior before deployment.

The skill should report only observed profile names, page coordinates, backup paths, commands run, render counts, and unresolved environment-specific inputs. It should never report secrets or claim a test it did not execute.

Troubleshooting