Live observation for agents

Copy Markdown View Source

Insider.Live lets you watch a running development, test, or CI node while it keeps running: one bounded, leased session per node records Logger, Telemetry, and Poller evidence, and you read it after a cursor. This page is the operating contract. The authoritative, always-current version is the welcome document the node itself returns; read it first.

Read the welcome first

The welcome document is generated from the code the node runs. By default it is an agent quick start (insider.live-welcome-quick-start/v1): the steps in order, the six core observer commands with real flags, --load for first contact, symptom recipes (triage, latency, crash loop, 5xx, growing mailbox, memory growth, missing values, several agents, and unknown event names), tandem for sharing a session, and runtime: the node, its environment, the active session, and its quiet default Telemetry events. The full reference (insider.live-welcome/v2; --full, or %{"full" => true}) adds every operation with its options, defaults, and bounds, every error code of the node and the observer, the observer's value syntax for every flag (such as module:NAME), both transports with example invocations, and a longer guide.

  • Observer node: mix insider.live welcome --node NAME@HOST --cookie-file PATH from tools/insider_lab in a checkout, or insider-lab live welcome … after mix escript.install hex insider_lab; add --full for the full reference. The first command run against a target node, whatever it is, also carries the quick start under "welcome"; later ones carry a pointer. mix insider.live --help and help OPERATION print the flags with their value syntax and examples, without a target.
  • Parse stdout alone. It carries only the observer's documents; Mix prints its Compiling and Waiting for lock on the build directory notices (several mix commands in one checkout share that lock) on stderr. If your tool merges the two streams, or several agents share the checkout, run the escript instead (insider-lab live … from Hex, or mix escript.build once in the checkout, then ./insider-lab live …): it starts no Mix, so neither notice appears.
  • Tidewave: project_eval with Insider.Live.JSON.welcome(), or Insider.Live.JSON.welcome(%{"full" => true}).
  • capabilities carries welcome.hash and the same commands, so you learn about the document even if you skipped it.

hash is the SHA-256 of the canonical JSON of every key of the full reference except hash and runtime; the quick start carries the same hash. It changes exactly when what the document promises changes. Keep it; if capabilities later reports another hash, read the welcome again.

Act, then read

  1. Check runtime.allowed. If runtime.active_session_id is set, a session is already running; only one exists per node.
  2. start with the fewest components that can show the effect. Without options a session is quiet: Logger warnings and worse, OTP reports, slow or failed queries, every request, nothing from Oban's own machinery (the welcome's filters.quiet_defaults; this node's default events are in runtime.quiet_default_telemetry_events). Widen it with the filter options, for example logger_level debug or telemetry_min_duration_ms 0. Do not guess Telemetry event names: discover (--prefix my_app) lists the names the node can emit, with each name's source and module, and --telemetry-event 'my_app.*' subscribes to every discovered name below the prefix. status then reports telemetry_subscriptions: subscribed with seen 0 means the event was not emitted.
  3. Act: send the request, run the job, or ask the developer to use the app. Pairing with a developer who clicks in the app: mark the moment first (--label 'about to click Confirm'), ask them to act, then read since the mark (digest --since-mark --human, events --since-mark): no time or cursor to guess.
  4. digest the session first: counts by kind, severity, and offender, crash loops with their period, the top error signatures, latency tables, the longest mailboxes, and VM deltas in one call (digest --human prints a few lines to paste for the developer). Then read events_since from your cursor for detail (the observer's --format compact prints one line per event) and keep next_cursor. A non-empty gaps list means evidence was lost and says how much; truncated means more events are waiting, and truncated_by says which bound ended the read (limit and max_bytes count only events you are shown; scan_limit means a sparse view used its scan budget). Silence is not inactivity: filtered counts what each filter and your view did not show you. Joining a running session, read from cursor "latest" to skip its history; status reports next_receipt and the recorder's fill.
  5. Repeat 3 and 4. Reads renew the lease; status and subscriptions do not, so call renew during long pauses.
  6. stop when done and check the receipt's outcome. A session you abandon is stopped by its lease through the same cleanup.

Sharing a session with other agents

  • Name yourself on every read, watch, stop, and amend with --reader NAME ("reader"); status lists each named reader's cursor and when it last read. Join a running session from cursor latest instead of stopping it.
  • Recording options (components, events, measurements, filters) apply to every reader. A view applies only to your own read. To change what is recorded, amend the running session: its history and every cursor stay, and every reader sees a live.session_amended event.
  • stop refuses with readers_active while another named reader is active; --force stops anyway. A stopped session stays readable until readable_until, so peers can drain it.
  • handoff prints what another agent needs to continue: node, welcome hash, session id, configuration, and each reader's cursor.
  • Run several agents from the escript (mix escript.install hex insider_lab, or mix escript.build in tools/insider_lab), which never waits on Mix's build lock.

What leaves the node depends on its disclosure profile, which the welcome states as effective_disclosure. Development and test default to rich: events carry Logger message text and metadata, Telemetry metadata and measurements, with credentials masked. Only in the strict profile (the ci default) do event payloads, Logger message text, and Telemetry metadata never leave the node. See the welcome's disclosure rules.

Pairs well with

Insider shows what a running node does. Reach for these when the question is a different one. The welcome's pairs_with lists the same table, and runtime.pairs_with_available says which the node has.

ToolUse it forIs it there?
CredoStatic style and consistency rules: code-shape checks Insider leaves out. mix credo --strict."credo" in mix.lock
ArgusA static analysis tool for compiled beams: OTP startup, shutdown and supervision structure, races, swallowed failures, unanswered messages, effects inside a transaction, ETS ownership, exposed secrets. argus list names the analyses; after mix compile (it never builds) run argus analyze --project beams --root . --ebin _build/dev/lib/APP/ebin --analyses default,effects,ets,exposure --format json, since the default set alone leaves out effects, ets, and exposure.argus on PATH (argus version), and souffle on PATH
reconLive process, memory, allocator and port inspection beyond Insider's bounded process, bin_leak, ports, and alloc reads, run in the node (Tidewave project_eval or a remote shell)."recon" in mix.lock, or runtime.pairs_with_available.recon (true also after --load pushed it)
TidewaveEvaluating code and reading logs in the running dev node (project_eval, get_logs) when Insider's evidence needs a why."tidewave" in mix.lock, and the dev server answers at /tidewave/mcp
observer_cliA terminal UI for a human watching the node. Not for agents; use digest, trends, and process instead."observer_cli" in mix.lock

Report Insider's own bugs

You must report Insider bugs and every behaviour that differs from the welcome document. Categories are a closed list: bug, doc_mismatch (cite the welcome field you relied on), missing_capability, and confusing_output. Write what you expected and what happened; never include event payloads, log text, option values, or secrets. Insider adds the context itself: its version and commit, the welcome hash, the node, the environment, the last target operation and error code, and the observer's last local failure (such as a usage error).

mix insider.live report --category doc_mismatch --message 'welcome says X; observed Y'
Insider.Live.JSON.report(%{"category" => "doc_mismatch", "message" => "welcome says X; observed Y"})

Reports stay on the machine: the observer keeps a bounded JSONL queue, and the node keeps a bounded in-memory queue that mix insider.live reports --node … copies into it. Both deduplicate by fingerprint (the category and the normalised message). mix insider.live reports submit files GitHub issues on hiivemarkets/insider through gh, one per new fingerprint, only with --confirm or when the host sets config :insider, :live_report_submission, :github. Without either it prints what it would send and runs nothing; --dry-run always previews, overriding every opt-in. Submitting is the developer's decision, not yours.

The observer queue is shared by every run on the machine. A report's context cites report --session ID when given, else the session of your last target command; reports groups reports by node and session, and --session or --fingerprint selects what reports and reports submit cover.

Where the details are

The reference under docs/live/ covers the environment rule, leases, cursors and gaps, subscriptions, every error code, Tidewave setup, and the observer node's target setup, --load, and exit statuses. Start with Live observation, the cheat sheet, or find your task in the documentation index.

Other questions

Each operation that this guide does not cover has its own section: