Insider.Live lets you watch a running development, test, or CI node while
it keeps running: one bounded, leased session per node records Logger,
Telemetry, and Poller evidence, and you read it after a cursor. This page is
the operating contract. The authoritative, always-current version is the
welcome document the node itself returns; read it first.
Read the welcome first
The welcome document is generated from the code the node runs. By default
it is an agent quick start (insider.live-welcome-quick-start/v1): the steps
in order, the six core observer commands with real flags, --load for first
contact, symptom recipes (triage, latency, crash loop, 5xx, growing mailbox,
memory growth, missing values, several agents, and unknown event names),
tandem for sharing a session, and runtime: the node, its
environment, the active
session, and its quiet default Telemetry events. The full reference
(insider.live-welcome/v2; --full, or
%{"full" => true}) adds every operation with its options, defaults, and
bounds, every error code of the node and the observer, the observer's value
syntax for every flag (such as module:NAME), both transports with example
invocations, and a longer guide.
- Observer node:
mix insider.live welcome --node NAME@HOST --cookie-file PATHfromtools/insider_labin a checkout, orinsider-lab live welcome …aftermix escript.install hex insider_lab; add--fullfor the full reference. The first command run against a target node, whatever it is, also carries the quick start under"welcome"; later ones carry a pointer.mix insider.live --helpandhelp OPERATIONprint the flags with their value syntax and examples, without a target. - Parse stdout alone. It carries only the observer's documents; Mix prints
its
CompilingandWaiting for lock on the build directorynotices (severalmixcommands in one checkout share that lock) on stderr. If your tool merges the two streams, or several agents share the checkout, run the escript instead (insider-lab live …from Hex, ormix escript.buildonce in the checkout, then./insider-lab live …): it starts no Mix, so neither notice appears. - Tidewave:
project_evalwithInsider.Live.JSON.welcome(), orInsider.Live.JSON.welcome(%{"full" => true}). capabilitiescarrieswelcome.hashand the same commands, so you learn about the document even if you skipped it.
hash is the SHA-256 of the canonical JSON of every key of the full
reference except hash and runtime; the quick start carries the same
hash. It changes exactly when what the document promises changes. Keep it;
if capabilities later reports another hash, read the welcome again.
Act, then read
- Check
runtime.allowed. Ifruntime.active_session_idis set, a session is already running; only one exists per node. startwith the fewest components that can show the effect. Without options a session is quiet: Logger warnings and worse, OTP reports, slow or failed queries, every request, nothing from Oban's own machinery (the welcome'sfilters.quiet_defaults; this node's default events are inruntime.quiet_default_telemetry_events). Widen it with the filter options, for examplelogger_leveldebugortelemetry_min_duration_ms0. Do not guess Telemetry event names:discover(--prefix my_app) lists the names the node can emit, with each name's source and module, and--telemetry-event 'my_app.*'subscribes to every discovered name below the prefix.statusthen reportstelemetry_subscriptions:subscribedwithseen0 means the event was not emitted.- Act: send the request, run the job, or ask the developer to use the app.
Pairing with a developer who clicks in the app:
markthe moment first (--label 'about to click Confirm'), ask them to act, then read since the mark (digest --since-mark --human,events --since-mark): no time or cursor to guess. digestthe session first: counts by kind, severity, and offender, crash loops with their period, the top error signatures, latency tables, the longest mailboxes, and VM deltas in one call (digest --humanprints a few lines to paste for the developer). Then readevents_sincefrom your cursor for detail (the observer's--format compactprints one line per event) and keepnext_cursor. A non-emptygapslist means evidence was lost and says how much;truncatedmeans more events are waiting, andtruncated_bysays which bound ended the read (limitandmax_bytescount only events you are shown;scan_limitmeans a sparse view used its scan budget). Silence is not inactivity:filteredcounts what each filter and yourviewdid not show you. Joining a running session, read from cursor"latest"to skip its history;statusreportsnext_receiptand the recorder's fill.- Repeat 3 and 4. Reads renew the lease;
statusand subscriptions do not, so callrenewduring long pauses. stopwhen done and check the receipt'soutcome. A session you abandon is stopped by its lease through the same cleanup.
Sharing a session with other agents
- Name yourself on every read, watch, stop, and amend with
--reader NAME("reader");statuslists each named reader's cursor and when it last read. Join a running session from cursorlatestinstead of stopping it. - Recording options (components, events, measurements, filters) apply to
every reader. A view applies only to your own read. To change what is
recorded,
amendthe running session: its history and every cursor stay, and every reader sees alive.session_amendedevent. stoprefuses withreaders_activewhile another named reader is active;--forcestops anyway. A stopped session stays readable untilreadable_until, so peers can drain it.handoffprints what another agent needs to continue: node, welcome hash, session id, configuration, and each reader's cursor.- Run several agents from the escript (
mix escript.install hex insider_lab, ormix escript.buildintools/insider_lab), which never waits on Mix's build lock.
What leaves the node depends on its disclosure profile, which the welcome
states as effective_disclosure. Development and test default to rich:
events carry Logger message text and metadata, Telemetry metadata and
measurements, with credentials masked. Only in the strict profile (the ci
default) do event payloads, Logger message text, and Telemetry metadata never
leave the node. See the welcome's disclosure rules.
Pairs well with
Insider shows what a running node does. Reach for these when the question
is a different one. The welcome's pairs_with lists the same table, and
runtime.pairs_with_available says which the node has.
| Tool | Use it for | Is it there? |
|---|---|---|
| Credo | Static style and consistency rules: code-shape checks Insider leaves out. mix credo --strict. | "credo" in mix.lock |
| Argus | A static analysis tool for compiled beams: OTP startup, shutdown and supervision structure, races, swallowed failures, unanswered messages, effects inside a transaction, ETS ownership, exposed secrets. argus list names the analyses; after mix compile (it never builds) run argus analyze --project beams --root . --ebin _build/dev/lib/APP/ebin --analyses default,effects,ets,exposure --format json, since the default set alone leaves out effects, ets, and exposure. | argus on PATH (argus version), and souffle on PATH |
| recon | Live process, memory, allocator and port inspection beyond Insider's bounded process, bin_leak, ports, and alloc reads, run in the node (Tidewave project_eval or a remote shell). | "recon" in mix.lock, or runtime.pairs_with_available.recon (true also after --load pushed it) |
| Tidewave | Evaluating code and reading logs in the running dev node (project_eval, get_logs) when Insider's evidence needs a why. | "tidewave" in mix.lock, and the dev server answers at /tidewave/mcp |
| observer_cli | A terminal UI for a human watching the node. Not for agents; use digest, trends, and process instead. | "observer_cli" in mix.lock |
Report Insider's own bugs
You must report Insider bugs and every behaviour that differs from the
welcome document. Categories are a closed list: bug, doc_mismatch (cite
the welcome field you relied on), missing_capability, and
confusing_output. Write what you expected and what happened; never include
event payloads, log text, option values, or secrets. Insider adds the
context itself: its version and commit, the welcome hash, the node, the
environment, the last target operation and error code, and the observer's
last local failure (such as a usage error).
mix insider.live report --category doc_mismatch --message 'welcome says X; observed Y'
Insider.Live.JSON.report(%{"category" => "doc_mismatch", "message" => "welcome says X; observed Y"})Reports stay on the machine: the observer keeps a bounded JSONL queue, and
the node keeps a bounded in-memory queue that mix insider.live reports --node … copies into it. Both deduplicate by fingerprint (the category and
the normalised message). mix insider.live reports submit files GitHub
issues on hiivemarkets/insider through gh, one per new fingerprint, only
with --confirm or when the host sets
config :insider, :live_report_submission, :github. Without either it
prints what it would send and runs nothing; --dry-run always previews,
overriding every opt-in. Submitting is the developer's decision, not yours.
The observer queue is shared by every run on the machine. A report's context
cites report --session ID when given, else the session of your last target
command; reports groups reports by node and session, and --session or
--fingerprint selects what reports and reports submit cover.
Where the details are
The reference under docs/live/ covers the environment rule, leases,
cursors and gaps, subscriptions, every error code, Tidewave setup, and the
observer node's target setup, --load, and exit statuses. Start with
Live observation,
the cheat sheet, or find your task in the
documentation index.
Other questions
Each operation that this guide does not cover has its own section:
- One request as a span tree:
trace, in Read one request as a trace. Keep it past eviction with--retain, in Pin a trace. - GraphQL response values:
trace --values, in Response values. - Everything one process or one request did:
timeline, in Timeline of a process or request. - Who changed an in-memory value:
start --watch-write, thenchanges, in Changes of in-memory values. - Mutations that outlive the Recorder:
mutations, in Kept mutation records. - Anomalies you already know about:
baseline, in Known-normal baseline. - A frontend's spans in the same trace:
otlp-receive, in Frontend traces. - Reads without
--session: the node's active session, in Default session. - What
stopproves and what it does not: Stop and check the cleanup receipt.