Analyzer Workflow

The analyzer runs plugins against an experiment's artifact and produces reports you read back through Falcon. You submit one or more analysis records for an experiment — either inline when you create the experiment (an analyses: block) or after the fact with falcon analysis create. Once the artifact succeeds, Falcon dispatches an analyzer pod and moves the record through PENDING → RUNNING → SUCCEEDED | FAILED.

Submit an analysis

If you didn't add an analyses: block at experiment-create time, submit a manual analysis request:

falcon analysis create \
  --exp-id exp-arf29g1hql \
  --name late-hlo-pass \
  --plugin hlo-analyzer \
  --params /tmp/hlo-params.json

analysis_id:        an-3k9qz1p7
name:               late-hlo-pass
status:             ANALYSIS_STATUS_PENDING

Falcon picks it up on its next tick once the artifact is ready.

Inspect records

List the analyses on an experiment:

falcon analysis records --exp-id exp-arf29g1hql

ANALYSIS_ID   NAME                     STATUS                     ATTEMPT  RESULT_GCS_PATH
an-3k9qz1p7   profiling-trace-summary  ANALYSIS_STATUS_SUCCEEDED  1        gs://.../analysis/exp-arf29g1hql/an-3k9qz1p7/

falcon analysis list --exp-id <exp_id> is an alias, matching the exp list / cluster list muscle memory.

Other inspection commands:

falcon analysis get an-3k9qz1p7         # full key:value detail
falcon analysis outputs an-3k9qz1p7     # declared plugin output paths
falcon analysis cat an-3k9qz1p7 report.md   # read one declared file
falcon analysis plugins list --scope CUSTOM  # available plugin versions

Register a custom plugin

falcon analysis plugins create -f plugin.yaml

The manifest declares the command (or inline script) and the outputs the plugin writes:

name: my-plugin
command: /usr/local/bin/analyzer.sh   # OR script_content (mutually exclusive)
outputs:
  - path: report.md
    type: markdown
    description: optional

Registration is idempotent on identical (command, script_content, outputs) — re-running with the same content returns the existing version.

Clone an analysis spec

To re-run the same plugin / params against a different experiment's artifact, render a reproduction manifest from the stored record:

falcon analysis clone an-3k9qz1p7 > clone.yaml

clone outputs YAML by default. The rendered shape carries exp_id at the top plus the spec fields (name, plugin_names, params, command, script_content, outputs), so a single document is self-contained.

The common case — re-run against a different experiment with one override:

falcon analysis clone an-3k9qz1p7 \
  --exp-id exp-newrun \
  --name baseline-vs-new \
  --set params.threshold=2.5 \
  > rerun.yaml

Supported --set paths:

• top-level: name, exp_id, command, script_content
• params overrides: params.<key> (string-typed; edit the rendered YAML directly for non-string values)

Unknown paths reject hard. clone never submits — that stays your explicit analysis create (or analysis rerun) call.

Rerun (clone + submit in one step)

falcon analysis rerun an-3k9qz1p7 \
  --exp-id exp-newrun \
  --name baseline-vs-new

Default behaviour is submit. Add --dry-run to preview the manifest without submitting:

falcon analysis rerun an-3k9qz1p7 --exp-id exp-newrun --dry-run

Same --set rules as clone. If the source record has no exp_id and you don't pass --exp-id, rerun fails fast — re-runs need a destination.

Read outputs

Falcon stores analyzer bytes in object storage, but you read them through the declared-output API rather than listing storage yourself:

falcon analysis outputs an-3k9qz1p7
falcon analysis cat an-3k9qz1p7 report.md

analysis outputs returns the plugin's declared output paths; analysis cat reads one of those paths through Falcon. Undeclared paths are rejected. A typical analysis writes:

File	What it is
output.log	plugin stdout/stderr (always present on a terminal record)
report.md	optional aggregated markdown report
<plugin>.jsonl	per-plugin structured output

Example: profile a workload and summarize it

Profiling isn't a separate system — it's a normal experiment whose workload writes profile data to the artifact directory, plus an analyzer that summarizes it.

1. Submit an experiment whose workload writes profile data to $ARTIFACT_LOCAL_DIR (see Experiment Lifecycle for the submit mechanics).

2. When it succeeds, run the xprof-summary analyzer against its artifact:

   falcon analysis create --exp-id <exp_id> --name xprof-summary --plugin xprof-summary

3. Read the summary:

   falcon analysis outputs <analysis_id>
   falcon analysis cat <analysis_id> report.md

For agent-driven profiling runs, the falcon-workflow skill has a profiling recipe — see Using Falcon with an Agent.

Failure and retry

A failed record leaves the artifact eligible for rediscovery. If the record's auto_retry is true (the default), the next tick mints a fresh attempt with the same correlation key; if auto_retry: false, the failed record stays terminal and no retry happens. Inspect both attempts with falcon analysis records --exp-id ....

Multi-artifact analysis (precision compare and beyond)

A multi-artifact analysis (e.g. comparing a baseline dump against candidate dumps) compares two or more artifacts in one analysis record that consumes several role-tagged inputs. There is no separate Alignment object — it's the same analysis record, just with more than one input.

Each input names an experiment (exp_id) and a role (baseline / candidate); roles are free-form strings and the plugin decides how to combine them. An input whose artifact is missing or not SUCCEEDED fails the request before the analysis is enrolled, and the error names the offending input.

The manifest carries one baseline and one or more candidates:

schema_version: 1
name: align-llama-bf16
owner_exp: exp-baseline
plugins:
  - precision-compare
inputs:
  - { name: baseline,    role: baseline,  exp_id: exp-base }
  - { name: candidate-a, role: candidate, exp_id: exp-a    }
  - { name: candidate-b, role: candidate, exp_id: exp-b    }
config_refs:
  mapping:    gs://<bucket>/mappings/llama/sgl-jax__hf-torch.json
  thresholds: gs://<bucket>/thresholds/bf16.json
  argus:      v0.4.2

Each input's exp_id is resolved to its artifact and the locked falcon.multi_artifact.v1 params envelope is stamped into the analysis; the analyzer then picks it up like any other offline analysis. The full envelope schema is the falcon.multi_artifact.v1 params schema. For the automated submit path, see Automation below.

v1 artifact contract (precision-compare)

The precision-compare plugin (v1) consumes a strict per-step layout under each dump experiment's artifact directory:

gs://<bucket>/experiments/<exp_id>/artifacts/<artifact_id>/
  step_000000.npz
  step_000001.npz
  step_000002.npz
  ...

Rules:

• File names match step_*.npz. v1 does not recurse into subdirectories, does not handle sharded layouts, and ignores non-.npz files.
• Each .npz is a flat tensor_name -> numpy.ndarray mapping.
• Baseline and each candidate pair by exact filename; the plugin compares the union of tensor keys per step.
• Missing step files, missing tensor keys, shape mismatches, and uncomparable dtypes are recorded per-tensor and summarised in metrics.json — the plugin does not abort.
• config_refs.mapping / config_refs.thresholds are gs://... URIs reachable from the analyzer pod. v1 reads global max_abs / max_rel from the thresholds JSON.
• config_refs.argus is an opaque traceability tag persisted into metrics.json and the report.

Troubleshooting

Read this before escalating to the on-call operator.

Authentication failures

Symptom	Likely cause	Fix
Unauthenticated: ... from any command	No Cloudflare Access credential in env or on disk	Run falcon auth login, or set FALCON_CF_ACCESS_CLIENT_ID / FALCON_CF_ACCESS_CLIENT_SECRET
HTTP 401: Cloudflare Access token is missing, expired, or invalid	The stored credential expired or was revoked	Run falcon auth login again — a fresh login captures a refresh token and the CLI then refreshes silently
Cf-Access-Client-Secret rejected	Service token expired or rotated	Generate a new token in the Cloudflare Zero Trust dashboard and update the env vars
Browser doesn't open	No browser launcher on PATH	Copy the URL the CLI printed and open it manually; the loopback callback still completes. As a last resort, falcon auth login --use-cloudflared

No active cluster

falcon: ...: rpc error: code = FailedPrecondition desc = no active cluster

falcon cluster list shows no row with STATUS active. An operator must register or re-activate a cluster — you can't fix this client-side.

Job stuck in pending

A TPU node pool create is async and can take 2–4 minutes for v7x. Wait first. After ~10 minutes, collect the Falcon-visible state and escalate:

falcon exp get <exp_id>
falcon exp logs <exp_id>

Common causes: TPU reservation exhausted, IAM permissions missing on the scheduler's service account, or the cluster temporarily busy with another node-pool operation. A ClusterBusy reason is retryable; leave the job pending unless it stays there well past the normal provisioning window.

Job ran but no artifact / no logs

falcon exp get <exp_id>
falcon exp logs <exp_id>

If Falcon can't return logs, include this output when escalating. Likely causes are cluster logging configuration or per-cluster log bucket permissions.

Analyzer doesn't dispatch

falcon analysis records --exp-id <exp_id>

No record means nothing to dispatch — submit one with analysis create. If a record stays PENDING after the artifact is SUCCEEDED, escalate with the exp_id, artifact_id, and analysis_id.

Analyzer record failed

falcon analysis get <analysis_id>
falcon analysis outputs <analysis_id>

If auto_retry: true, the next tick mints a new attempt. If false, fix the root cause and re-submit with falcon analysis create.

What you can't fix yourself

Anything cluster/infra-side — TPU reservations, GKE node pool errors, GCS bucket permissions, Cloudflare Access policy. Read the Production Runbook for operator-side diagnosis, then escalate to the on-call with the exp_id, job_id, any analysis_id, and the relevant exp get / exp logs / analysis get / analysis outputs output.

Automation

For automated or agent-driven analysis — blocking submit/wait, a stable JSON envelope (--output json), exit-code contracts, and the multi-artifact submit path — use the falcon-workflow skill. It wraps the falcon analysis commands above into end-to-end recipes against a locked v1 envelope contract, so skills and CI can parse results without scraping text. This page covers the interactive human path only.

Related docs

• Deploy Guide
• Production Runbook
• Contributing