macos-agent ops

Contract

Prereqs:

• •macOS host with Accessibility and Automation permissions granted for Terminal and target apps.
• •Homebrew-installed macos-agent available on PATH.
• •cliclick, osascript, and im-select available on PATH.
• •ABC/US input source enabled in macOS Input Sources.
• •For full AX surface (ax attr/action/session/watch), Hammerspoon runtime should be healthy (hs + hs.ipc).

Inputs:

• •Command entrypoint: $AGENTS_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh.
• •Optional env vars:
  • •MACOS_AGENT_OPS_INPUT_SOURCE_ID (preferred; example: abc) for deterministic input source target.
  • •MACOS_AGENT_REAL_E2E_INPUT_SOURCE (legacy fallback) if MACOS_AGENT_OPS_INPUT_SOURCE_ID is unset.
  • •MACOS_AGENT_OPS_SKIP_INPUT_SOURCE_SWITCH=1 to bypass automatic input source switch.
  • •CODEX_MACOS_AGENT_AX_BACKEND=auto|hammerspoon|applescript to control AX backend preference.

Outputs:

• •macos-agent JSON/text command output.
• •Artifacts generated by macos-agent under $AGENTS_HOME/out/ (screenshots, traces, debug bundles, step ledgers).

Exit codes:

• •0: success
• •1: runtime failure or missing runtime dependency
• •2: usage error

Failure modes:

• •wait app-active timeouts because focus was stolen (Control Center/Spotlight/notifications).
• •window.activate failure when app relaunch/update state is stuck; wrapper now uses --reopen-on-fail.
• •missing im-select causes input-source failure before keyboard-input actions.
• •not authorized / Apple Events denial from TCC permissions.
• •AX target mismatch (selector too broad / stale node id) during ax operations.
• •AX gate failures (--gate-*) when app/window/selector readiness conditions are not met.
• •AX postcondition failures (--postcondition-*) when state does not change as expected after mutation.
• •observe screenshot --if-changed-baseline path missing or unreadable.
• •trace directory validation failures when using --trace/--trace-dir and path is not writable.

Scripts (only entrypoints)

• •$AGENTS_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh

Workflow

1. •Resolve Homebrew macos-agent binary path:

$AGENTS_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh where

1. •Ensure deterministic input source (default target: abc):

$AGENTS_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh input-source
# or explicit source id
$AGENTS_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh input-source --id com.apple.keylayout.ABC

1. •Run readiness checks (preflight + AX list smoke check):

$AGENTS_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh doctor
# optional target for AX smoke
$AGENTS_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh doctor --ax-app Arc
$AGENTS_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh doctor --ax-bundle-id com.google.Chrome

1. •Inspect unified permission state directly (screen_recording / accessibility / automation / ready):

$AGENTS_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh run -- \
  --format json preflight --include-probes

1. •Run quick app foreground check with reopen recovery:

$AGENTS_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh app-check --app Finder
$AGENTS_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh app-check --app Arc --timeout-ms 15000
$AGENTS_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh app-check --bundle-id com.google.Chrome

1. •Run AX-only health check (AX tree probe):

$AGENTS_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh ax-check --app Arc --role AXWindow
$AGENTS_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh ax-check --app Arc --role AXTextField --title-contains Search

1. •Use AX wait primitives before mutation to stabilize dynamic UI:

$AGENTS_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh run -- \
  --format json wait ax-present --app Arc --role AXButton --title-contains "Compose" \
  --match-strategy contains --timeout-ms 2500 --poll-ms 80

$AGENTS_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh run -- \
  --format json wait ax-unique --app Arc --role AXTextField --title-contains "Search" \
  --selector-explain --timeout-ms 2500 --poll-ms 80

1. •Run robust mutating AX actions with gating + postconditions:

$AGENTS_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh run -- \
  --format json ax click --app Arc --role AXButton --title-contains "Play" \
  --match-strategy contains --selector-explain --reselect-before-click \
  --fallback-order ax-press,ax-confirm,frame-center,coordinate \
  --gate-app-active --gate-window-present --gate-ax-unique \
  --postcondition-focused true --postcondition-timeout-ms 2000 \
  --allow-coordinate-fallback

$AGENTS_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh run -- \
  --format json ax type --app Arc --role AXTextField --title-contains "Search" \
  --text "lofi" --clear-first --submit --paste \
  --gate-app-active --gate-window-present --gate-ax-present \
  --postcondition-attribute AXValue --postcondition-attribute-value "lofi" \
  --postcondition-timeout-ms 2000 --allow-keyboard-fallback

1. •Use selector-frame screenshots for visual proof of target alignment:

$AGENTS_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh run -- \
  --format json observe screenshot --active-window \
  --role AXButton --title-contains "Play" --selector-padding 12 \
  --path "$AGENTS_HOME/out/macos-agent-selector-$(date +%Y%m%d-%H%M%S).png"

1. •Use diff-aware screenshots to avoid noisy artifacts when nothing changed:

$AGENTS_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh run -- \
  --format json observe screenshot --active-window \
  --path "$AGENTS_HOME/out/macos-agent-state.png" \
  --if-changed --if-changed-threshold 2

1. •Use one-shot debug bundle for triage artifacts:

$AGENTS_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh run -- \
  --format json debug bundle --active-window \
  --output-dir "$AGENTS_HOME/out/macos-agent-debug-$(date +%Y%m%d-%H%M%S)"

1. •Run routine scripted scenarios:

$AGENTS_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh scenario --file /path/to/scenario.json

1. •Pass through raw macos-agent commands with JSON errors + trace when needed:

$AGENTS_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh run -- \
  --format json --error-format json --trace \
  input click --x 200 --y 160

Screenshot-Based Triage Rules

• •During runtime failure triage, the agent SHOULD capture an active-window screenshot before retry/remediation.
• •For wait app-active and window.activate failures, screenshot capture SHOULD be first-line diagnostics.
• •Screenshot artifacts MUST be written under $AGENTS_HOME/out/ and SHOULD use timestamped filenames.
• •The agent SHOULD include screenshot path + failing command + error message in the final diagnostic summary.

Recommended command (via the skill entrypoint):

$AGENTS_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh run -- \
  --format json observe screenshot --active-window \
  --path "$AGENTS_HOME/out/macos-agent-failure-$(date +%Y%m%d-%H%M%S).png"

Common Errors And Prevention

• •

  error: input source mismatch after switch attempt ...

  • •Run .../macos-agent-ops.sh input-source --id abc and verify current is com.apple.keylayout.ABC or ...US.
  • •Ensure im-select is installed: brew install im-select.
  • •If you intentionally need non-ABC IME, set MACOS_AGENT_OPS_SKIP_INPUT_SOURCE_SWITCH=1.
• •

  error: timed out waiting for app-active ...

  • •Keep hands off keyboard/mouse while checks are running.
  • •Close Control Center/Spotlight overlays and reduce notification interruptions.
  • •Increase timeout (--timeout-ms 12000 or higher) for slow app transitions.
• •

  error: ax.list ... timed out during wait ax-present / wait ax-unique

  • •Run app-check first, then retry AX waits after app focus is stable.
  • •Increase AX wait timeout/poll (--timeout-ms, --poll-ms) and narrow selectors.
  • •Keep backend on auto unless you need to force Hammerspoon behavior (CODEX_MACOS_AGENT_AX_BACKEND).
• •

  window activate failed ...

  • •Wrapper already uses --reopen-on-fail; rerun once to allow quit/relaunch recovery.
  • •For flaky apps (e.g., Spotify updater), kill stale relauncher and retry.
• •

  not authorized / Apple Events failures

  • •Re-run doctor and fix Accessibility/Automation in System Settings.
• •

  selector returned zero AX matches or ambiguous matches

  • •Narrow selectors and use --match-strategy exact|prefix|suffix|regex when contains is too broad.
  • •Use --selector-explain and wait ax-unique before ax click/ax type.
  • •Prefer --reselect-before-click in dynamic pages where node ids churn.
• •

  gate failure (--gate-*) before mutation

  • •Add/adjust readiness gates based on context (--gate-app-active, --gate-window-present, --gate-ax-present, --gate-ax-unique).
  • •Increase gate timeout/poll for slower app transitions (--gate-timeout-ms, --gate-poll-ms).
• •

  postcondition failure (--postcondition-*) after mutation

  • •Verify you are asserting the correct target/attribute (AXValue, focus state, etc.).
  • •Increase postcondition timeout on animation-heavy apps (--postcondition-timeout-ms).
• •

  error: --if-changed-baseline path does not exist ...

  • •Verify baseline path exists before running capture.
  • •If baseline is optional, remove --if-changed-baseline and let current output file be baseline.
• •

  trace directory is not writable (trace.write errors)

  • •Use a writable custom path with --trace-dir.
  • •Prefer $AGENTS_HOME/out/... for reproducible artifacts.
• •

  Homebrew macos-agent missing

  • •Install and link with Homebrew: brew install macos-agent.

Practical Notes

• •Prefer canonical flags in examples:
  • •--window-title-contains (instead of legacy --window-name)
  • •input type --submit (instead of legacy --enter)
• •Keep --error-format json enabled in automation loops when you need machine-parseable failures.
• •Use debug bundle first when a run fails and root cause is unclear.
• •Parse preflight JSON permissions (screen_recording / accessibility / automation / ready) before mutating runs.

References

• •skills/tools/macos-agent-ops/references/e2e-ref-01-preflight-focus.md
• •skills/tools/macos-agent-ops/references/e2e-ref-02-finder-routine.md
• •skills/tools/macos-agent-ops/references/e2e-ref-03-matrix-routine.md