Experiment Lifecycle

An experiment is one logical run — typically one node pool, one job, one output artifact. The pieces you'll see:

• Experiment — the user-facing record (status, creator, lineage).
• Job — the work Falcon runs to execute it.
• Artifact — the output the run produces (e.g. an HLO dump) that an analyzer can consume.
• Analysis (optional) — a request to run an analyzer plugin against the artifact when it succeeds.

Submit an experiment

Describe the run in a YAML file:

name: trial-001
exp_type: training
cluster_name: tpu7x-cluster
artifact_type: hlo
role_to_task_spec:
  worker:
    command: "echo hello && sleep 5"
    replica: 1
    image: "busybox:1.37"
    device_type: "v7x"
    device_topo: "2x2x1"

Falcon derives the artifact location for you — you don't put a gs:// URI in your YAML.

To give each submission a unique name, build the file from the shell with a heredoc (YAML can't expand $(date +%s) on its own):

cat <<EOF > my-exp.yaml
name: trial-$(date +%s)
exp_type: training
cluster_name: tpu7x-cluster
artifact_type: hlo
role_to_task_spec:
  worker:
    command: "echo hello && sleep 5"
    replica: 1
    image: "busybox:1.37"
    device_type: "v7x"
    device_topo: "2x2x1"
EOF

Submit it:

falcon exp create -f my-exp.yaml

EXP_ID          JOB_ID          ARTIFACT_ID   NAME       STATUS   ARTIFACT_URI
exp-arf29g1hql  job-8kd2m1xqp4  art-9plzq2v8  trial-001  pending  gs://.../experiments/exp-arf29g1hql/artifacts/art-9plzq2v8/

Notes:

• creator / creator_id come from your authenticated identity; any value you put in the YAML is ignored.
• role_to_task_spec must contain exactly one role.
• device_type / device_topo must match what the cluster can schedule. v7x + 2x2x1 works on the production tpu7x-cluster.

Choose a supply mode

Falcon normally chooses the node pool supply path from the cluster configuration. On clusters with a default_reservation, the default path uses reservation capacity. If the cluster has no default reservation, Falcon creates a flex-start pool.

You can override that per experiment with supply_mode under the role's task spec:

name: flex-start-smoke
exp_type: training
cluster_name: tpu7x-cluster
artifact_type: gcs
role_to_task_spec:
  worker:
    command: "echo flex-start && sleep 30"
    replica: 1
    image: "busybox:1.37"
    device_type: "v7x"
    device_topo: "2x2x1"
    device_count: 8
    supply_mode: flex-start

Supported values are:

supply_mode	Meaning
omitted	Use the cluster default: reservation when default_reservation is configured, otherwise flex-start.
reservation	Require the cluster's configured reservation. This needs a default_reservation on the target cluster.
flex-start	Force a GKE flex-start TPU node pool and skip reservation lookup, even if the cluster has a default reservation.
on-demand	Force on-demand provisioning and skip reservation lookup. Use only when the target cluster and quota support it.

Flex-start provisioning is asynchronous. falcon exp get <exp_id> can show pending with NodeRegistration/WaitingForNode after the GKE node pool reaches RUNNING; that means the flex-start pool exists but no Kubernetes node is Ready yet. Falcon keeps the lease active and waits for GKE capacity instead of deleting and recreating the pool. Use falcon exp abort <exp_id> if you want to stop waiting.

Mount GCS buckets

Use mounts when your workload needs a local filesystem path backed by a GCS bucket or prefix. Falcon mounts each entry with the GKE GCSFuse CSI driver. This example exposes gs://inference-model-storage-poc-tpu-hns/logs at /mnt/falcon-bucket-logs inside the container:

name: mount-smoke
exp_type: training
cluster_name: tpu7x-cluster
artifact_type: hlo
role_to_task_spec:
  worker:
    command: "ls -la /mnt/falcon-bucket-logs | head"
    replica: 1
    image: "busybox:1.37"
    device_type: "v7x"
    device_topo: "2x2x1"
    mounts:
      - name: falcon-logs
        type: gcs
        bucket: inference-model-storage-poc-tpu-hns
        prefix: logs
        mount_path: /mnt/falcon-bucket-logs
        read_only: true
        mount_options:
          - implicit-dirs
          - metadata-cache:ttl-secs:-1
          - metadata-cache:stat-cache-max-size-mb:-1
          - metadata-cache:type-cache-max-size-mb:-1
          - file-cache:max-size-mb:10240
          - file-cache:cache-file-for-range-read:true
          - file-system:kernel-list-cache-ttl-secs:-1
          - file-cache:enable-parallel-downloads:true
          - file-cache:parallel-downloads-per-file:128
          - read_ahead_kb=1024

Rules:

• type currently supports only gcs.
• bucket is the bucket name only — no gs://, no object prefix.
• prefix is optional and is mounted at exactly mount_path; omit it to mount the bucket root.
• mount_path must be absolute.
• mount_options are passed to GCSFuse as comma-separated options.
• file-cache:enable-parallel-downloads:true requires a file-cache:max-size-mb:<size> option too. Avoid very large file-cache values unless the node storage budget is explicit — checkpoint restores and dependency installs can otherwise exhaust node ephemeral-storage.

For GCP TPU jobs, route temporary/cache-heavy writes to /tmp/tpu_logs when available — set TMPDIR, PIP_CACHE_DIR, UV_CACHE_DIR under it before runtime installs or checkpoint restores. Also run ulimit -c 0 early in the command so a crashing rank doesn't fill node storage with multi-GB core files.

If the pod fails with GCS permission errors, ask an operator to grant the cluster's workload identity access to the bucket.

Training framework example: MaxText

MaxText jobs usually need three extra pieces beyond the minimal schema: a checked-out framework tree (usually baked into the image), a high-throughput GCSFuse mount for model/data paths, and framework-specific TPU/JAX environment variables.

A parameterized template lives at skills/falcon-workflow/references/training/maxtext-training.yaml. Fill it with your cluster, image, workdir, script, model/data bucket, checkpoint path, hardware shape, and framework environment variables.

Prefer an image that already contains dependencies. If the framework source is private or you need to pin a branch at submission time, create a reusable Git credential once and reference it from sources[].

Falcon supports two credential methods:

• ssh (default) — git@host:org/repo.git URLs, authenticated with an unencrypted SSH private key plus a known_hosts allowlist.
• token — https://host/org/repo.git URLs, authenticated with a personal access token (GitHub, GitLab, Bitbucket, Gitea, etc.).

SSH example:

falcon cred git create maxtext-ro \
  --ssh-key ~/.ssh/maxtext_ro_ed25519 \
  --known-hosts ~/.ssh/known_hosts \
  --repo git@github.com:primatrix/maxtext.git

falcon cred git list
falcon cred git get maxtext-ro
falcon cred git delete maxtext-ro

HTTPS token example (a fine-grained PAT, scoped read-only to one repo):

gh auth token | falcon cred git create gh-pat \
  --method token --token - \
  --repo https://github.com/primatrix/maxtext.git

The token can be a literal, a file path, or - to read from stdin. GitLab uses the same shape (default --token-username oauth2). Bitbucket and GitHub App installation tokens need an explicit --token-username x-token-auth.

The key or token is encrypted by Falcon and projected into a short-lived Kubernetes Secret only the source-checkout step can read. get, list, experiment readbacks, and logs never return the secret material. Credentials are owner-scoped — you cannot reference another user's credential by name.

Reference the credential from your manifest:

role_to_task_spec:
  worker:
    sources:
      - name: maxtext
        type: git
        repo: git@github.com:primatrix/maxtext.git
        ref: fix/ling3-moe-weight-gather-hoist
        path: /workspace/maxtext
        auth:
          cred: maxtext-ro
    command: |
      bash -lc 'cd /workspace/maxtext && bash train.sh'

Inspect status

falcon exp get exp-arf29g1hql

exp_id:       exp-arf29g1hql
name:         trial-001
status:       running
exp_type:     training
cluster:      tpu7x-cluster
job_id:       job-8kd2m1xqp4
job_status:   RUNNING
conditions:
  - type:                 ResourceReady
    status:               True
    reason:               reservation
  - type:                 K8sScheduling
    status:               False
    reason:               Unschedulable
    message:              0/4 nodes are available: ...
artifact_id:  art-9plzq2v8
artifact_uri: gs://.../experiments/exp-arf29g1hql/artifacts/art-9plzq2v8/

falcon exp get returns the experiment plus its bundled job and artifact. The conditions block shows the durable scheduling, provisioning, K8s job submission, and terminal-sync trail. A ResourceProvisioning condition with status=False and reason=ClusterBusy is retryable — Falcon keeps the job pending and tries again.

status	What it means
created	Row exists; not picked up yet.
pending	Selecting capacity, provisioning a node pool, or waiting for the pod to schedule.
running	The pod is executing the workload.
succeeded	The pod exited 0; the artifact succeeded.
failed	The pod exited non-zero, or the cluster surfaced a failure.
aborting / aborted	Stopped via falcon exp abort or by an operator.

Why isn't my experiment scheduling?

A newly submitted experiment sits in pending until Falcon finds (or provisions) capacity. To see why, first read the conditions on the experiment, then check the cluster's live capacity.

1. Read the experiment's conditions

falcon exp get <exp_id>

Look at the conditions: block:

Condition	What it tells you
ResourceProvisioning = False, reason ClusterBusy	The reserved capacity is momentarily exhausted. This is retryable — Falcon keeps the job pending and retries automatically.
K8sScheduling = False, reason Unschedulable	Kubernetes can't place the pod yet (e.g. 0/4 nodes are available). Usually means no node of the requested shape is free.
K8sJobSubmitted = True, no running yet	Submitted to Kubernetes; the pod is still waiting for a node.

2. Check live cluster capacity

falcon cluster list                       # find the cluster_id / name
falcon cluster resources get <cluster_id>

You'll see something like (numbers and pool names below are illustrative, not real cluster data):

=== SUMMARY ===
NODE_POOLS  NODES  OBSERVED_PROVISIONED_CHIPS  PODS  RUNNING  PENDING  FAILED  JOBS  EXPS
6           12     48                          9     7        2        0       8     8

=== CAPACITY ===
DEVICE_TYPE  TOPOLOGY  SUPPLY       NODE_POOLS  NODES  CHIPS
v7x          2x2x1     reservation  4           8      32
v7x          2x2x2     reservation  2           4      16

=== OCCUPANCY ===
NODE_POOL          DEVICE_TYPE  TOPOLOGY  SUPPLY       NODES  CHIPS  RUNNING  PENDING  FAILED  OTHER  WORKLOADS
falcon-job-7f3a…   v7x          2x2x1     reservation  2      8      1        0        0       0      exp-7f3a…

=== IDLE ===
DEVICE_TYPE  TOPOLOGY  SUPPLY       NODE_POOLS  NODES  CHIPS  EXAMPLES
v7x          2x2x2     reservation  2           4      16     falcon-job-2b9c…

How to read it for "why am I pending":

• IDLE has a row matching your device_type + device_topo → capacity exists; your job should schedule shortly (or is briefly held by ClusterBusy retry).
• No matching IDLE row and CAPACITY is fully reflected in OCCUPANCY → every pool of your shape is busy; you're waiting for one to free up or for Falcon to provision more.
• Your shape isn't in CAPACITY at all → the cluster has no node pools of that device_type/topology; fix your manifest's device_type / device_topo, or ask an operator.
• PENDING count is high → a queue of work is ahead of you on this cluster.

Pool reuse between experiments is tracked by leases; that's operator-side detail. As a user, the CAPACITY / OCCUPANCY / IDLE view above is enough to tell whether you're waiting on capacity.

If conditions and capacity both look fine but the pod still won't start, check the workload itself with falcon exp logs <exp_id>.

Or just ask your agent: "exp-… is stuck in pending, why?" — see Using Falcon with an Agent.

List your experiments

falcon exp list

EXP_ID          NAME       STATUS     CREATOR  EXP_TYPE   CLUSTER          CREATED_AT
exp-arf29g1hql  smoke-v7x  running    you      profiling  tpu7x-cluster    2026-05-09 18:23 CST
exp-vm69bcalkx  bench-v6e  succeeded  you      benchmark  tpuv6e-256-node  2026-05-09 17:50 CST

--status filters to a single status, --cluster <name> narrows to one cluster, --creator-id <id> to one owner, and --limit caps the page.

Reproduce a prior run

To rerun a past experiment without hunting for the original YAML, render a reproduction manifest from the server:

falcon exp inspect exp-arf29g1hql > rerun.yaml

exp inspect outputs YAML by default (the same shape exp create -f accepts): name, exp_type, cluster_name, role_to_task_spec, and so on. Per-experiment ids, status, and timestamps are intentionally omitted — this is the input shape, not a database dump. Edit name (and anything else you want to vary), then submit:

falcon exp create -f rerun.yaml

Clone (inspect + override, don't submit)

When you only want a few fields changed, clone applies overrides for you:

falcon exp clone exp-arf29g1hql \
  --name trial-002 \
  --set role_to_task_spec.worker.replica=4 \
  --set role_to_task_spec.worker.image=gcr.io/my/runner:v2 \
  --set role_to_task_spec.worker.envs.SEED=43 \
  > clone.yaml

falcon exp create -f clone.yaml

clone never submits — the explicit exp create -f stays your decision, so a typo in --set can't accidentally launch a wrong run.

Supported --set paths:

• top-level scalars: name, exp_type, cluster_name, priority, model_version, artifact_type, config
• per-role task spec: role_to_task_spec.<role>.<field> where field is command / replica / image / device_count / device_type / device_topo
• per-role env: role_to_task_spec.<role>.envs.<KEY>

Unknown paths are rejected, so a typo (replicas vs replica) fails fast instead of silently doing nothing.

Rerun (clone + submit in one step)

Once you trust the clone shape, rerun rolls clone + submit into one command:

falcon exp rerun exp-arf29g1hql \
  --name trial-002 \
  --set role_to_task_spec.worker.replica=4

Default behaviour is submit. Add --dry-run to preview the manifest without submitting:

falcon exp rerun exp-arf29g1hql --set role_to_task_spec.worker.replica=4 --dry-run

Same --set rules as clone. The preview-then-submit two-step with clone | create -f stays the recommended workflow when you want a final visual check; rerun is for when you've already done that check.

Abort

falcon exp abort exp-arf29g1hql --reason "wrong config"

The handler is idempotent — re-aborting a terminal experiment is a no-op. Falcon tears down the TPU resources asynchronously. --reason records a free-form audit note alongside who initiated the abort; a later falcon exp get surfaces these as abort_reason / aborted_by so you can answer "why was this aborted?" without log diving.

Current status	abort outcome
created / pending / running	Experiment → aborting, then aborted once teardown completes. Audit fields persisted.
aborting / aborted	Idempotent no-op — returns current state.
succeeded / failed	Rejected — terminal experiments can't be re-aborted.

Logs

Stream the workload pod's stdout/stderr — works for live and retained pods:

falcon exp logs exp-arf29g1hql                          # one-shot dump
falcon exp logs exp-arf29g1hql -f                        # follow
falcon exp logs exp-arf29g1hql --container task          # only the workload container
falcon exp logs exp-arf29g1hql --container sidecar       # only the logging sidecar
falcon exp logs exp-arf29g1hql --container all           # both, interleaved
falcon exp logs exp-arf29g1hql --pod 2 --tail 500 --since 30m  # multi-pod: pick one + window

exp logs reads through Falcon — no direct Kubernetes or GCS access needed. Logs survive the workload pod for a retention window; after that, the archived copy is returned automatically.

An experiment can be submitted to the cluster while still showing pending. running means at least one workload pod has actually started; before that, inspect the conditions in exp get for scheduling diagnostics.

Operate on a live experiment

While an experiment is running, run commands inside its pods, attach an interactive shell, or transfer files — no Kubernetes credentials required:

falcon exp exec exp-arf29g1hql -- env | sort
falcon exp exec exp-arf29g1hql --rank 0 -- ls -lah /tmp
falcon exp exec exp-arf29g1hql --no-wait -- pkill -USR1 python
falcon exp exec -it exp-arf29g1hql --rank 0 -- bash

falcon exp cp ./local.txt exp-arf29g1hql:/tmp/local.txt
falcon exp cp exp-arf29g1hql:/tmp/output.log ./output.log
falcon exp cp --rank 3 exp-arf29g1hql:/tmp/rank3.json ./rank3.json

Non-interactive exp exec defaults to all ranks. Each rank receives the same command and can branch on injected environment variables such as FALCON_RANK, FALCON_WORLD_SIZE, FALCON_EXP_ID, and FALCON_JOB_ID. Use --rank N when you want one rank.

Interactive exp exec -it attaches to one rank only — rank 0 unless --rank N is set.

exp cp is a transfer, not a shell command. It defaults to rank 0 and waits for the transfer to finish before returning success.

Delete

falcon exp delete exp-arf29g1hql                        # single
falcon exp delete exp-aaa exp-bbb exp-ccc               # batch

Multi-id delete continues on error: a failure on one id doesn't abort the loop, and the exit code is non-zero if any id failed. Deletes are soft — the underlying job and artifact rows remain for operators.

Scripting & automation

Every command above supports --output json for a stable machine-readable form. exp inspect and exp clone default to YAML and also accept --output json. For pipeline-friendly submit/wait/collect flows, see the Analyzer Workflow page and the falcon-workflow skill.