Skip to content

Error/Status 字符串模板

本页所有偏移、符号和计数均适用于 libtpu-0.0.40-cp314 wheel 中的 libtpu.solibtpu_lts_20260413_b_RC00,ELF x86-64,781,691,048 字节,BuildID md5 89edbbe81c5b328a958fe628a9f2207d,未 stripped)。其他构建会有所不同。

摘要

libtpu 的诊断信息由内嵌在 .rodata 中的格式字符串模板构成,并在调用点填充。有两种格式化习惯用法和一种包装约定。主导习惯用法是 C printf 风格,即 absl::StrFormat("…%s…%d…", args)LOG(...) << absl::StrFormat(...),覆盖了绝大多数表面;少数习惯用法是 absl 位置替换,即 absl::Substitute("…$0…$1…", args),主要限于 megascale DCN runtime、collective 缓冲区大小校验器,以及一个静态链接的 protobuf descriptor 校验器。随后,这些模板大多会由三个 factory 家族之一包装成 absl::Status(见 Status-Code 映射),因此同一个字符串同时是人类可读错误,也是经由 PJRT 传回 JAX/XLA 用户的 StatusOr<T> 载荷。

本页是参考目录,不是算法轨迹。它的价值在于按组去重的真实模板表,并解释其占位符、在字节已确认时标注其 absl::Status code。恢复出的表面大约有 2,937 个不同的 error/status 模板;本页不会全部复现,而是记录这个空间的结构(占位符语法、子系统划分、三种 Status 构造习惯用法、字节已确认的参数类型),并为每个子系统给出有来源锚点的代表性样本,使重新实现者可以预测未列出模板的形态。

占位符语义是贯穿全篇的主线。%s 几乎从来不是原始 C 字符串:它是某个对象的 ToString() / AbslStringify,例如 ShapeLayoutHloInstruction 名称、设备名称或 opcode 助记符。%d/%lld/%zu 是维度、ordinal、计数或字节大小。指针(%p)和浮点数(%f)只出现在低层 driver 和 cost-model 路径中。本目录会逐个模板指出每个占位符的含义,因为单看 printf 类型(%s = char*)会隐藏真实的 C++ 参数。

注意 — 此处的 error/status 模板不同于面向用户的提示字符串(以 "try…"、"consider…" 这类建议形式表达,而非失败),后者位于 hint-strings.md;也不同于 internal-pass-names.md 上的内部 pass-name 字符串(pipeline 阶段标识符,不是诊断)。本页负责 error/status 格式字符串目录、占位符语义和 status-code 映射。

不同的 error/status 模板~2,937(~2,799 个 printf 风格,~138 个位置 $N/%v
Status 构造习惯用法3(<Code>StrCat factory、MakeErrorStream、散文式 absl::<Code>Error
字节已确认的参数类型 factory99 个 xla::<Code>StrCat<Types…> 实例化
MakeErrorStream 返回类型操作390 个 MakeErrorStreamWithOutput<T>(每个 StatusOr<T> 一个)
最常见占位符%s(~2778)、%d(~2031)、%u(116)、%zu(84)、%x(55)、%lld(52)
最大子系统块Compile / HLO / verifier(~875 个候选)
经反编译抽查确认12/12 个代表性模板(见报告)

如何阅读本目录

两种格式化习惯用法

text
printf-style   "Argument to Cholesky must have rank >= 2; shape was %s"
               filled by  absl::StrFormat(template, shape.ToString())
               or         LOG(ERROR) << absl::StrFormat(template, …)

positional     "ALL_TO_ALL not supported when buffer size is not divisble
                by number of endpoints. Buffer Size: $0 Number of
                endpoints: $1. MegascaleInfo: $2"
               filled by  absl::Substitute(template, size, n, info)

printf 表面使用 % spec;位置表面使用 $0$1、…(以及少量 %v,即 absl 的 stringify-any 标记)。两者绝不会混在同一个模板里。只依据 % spec 的重新实现者会完全漏掉约 138 个位置模板。

占位符语法

所有模板中 % spec 出现次数的完整分布明显偏向字符串和整数:

Spec计数C 类型真实 C++ 参数(典型)
%s~2778char*ToString() / AbslStringify 结果,例如 Shape、Layout、instruction 名称、device 名称、opcode
%d~2031int维度索引、ordinal、计数、status/state enum
%u116unsigned不可能为负的计数或索引
%zu / %zd84 / 19size_t / ssize_t字节大小或元素计数
%x / %#x / %02x55 / 19 / 11unsigned(十六进制)地址片段、寄存器/位掩码、chip ID 字节
%lld / %ld / %lu / %llu52 / 48 / 40 / 4long long / long / unsigned long大字节大小或 64 位计数
%p32void*缓冲区/driver 对象指针(仅 driver + cost-model 路径)
%c24char大括号/方括号字面量或轴字母('{''}''X'
%f19double/float比率/分数(cost model、hbm-fraction 配置)
%v6absl-stringify带有 AbslStringify overload 的对象(例如 TensorCoreBundle

陷阱 — %s说明程序中的参数是字符串。它只说明调用点传入了 const char*,而这几乎总是某个对象 ToString() 的产物。重新实现时,%s 背后的真实类型只能在调用点恢复(例如 "%s vs. %s" 的操作数到底是 Shape::ToString() 还是 HloInstruction::name());本目录会注明可能类型,但不会对 printf 家族做字节确认。

置信度与归属

标记为 CERTAIN 的模板已在反编译调用点逐字抽查确认。子系统分组是对文本的关键词分类,不是逐模板调用点轨迹;对文本内容为 HIGH 置信,对精确所属 pass 为 MEDIUMstatus code 列只有对 99 个 <Code>StrCat factory(mangled symbol 命名 code)和 390 个 MakeErrorStream 位置(macro 命名 code)为 CERTAIN;其他所有模板的 code 都依据文本和所属子系统的主导习惯用法推断,并相应标记为 MEDIUMLOW


Compile / HLO / Codegen / Verifier — ~875 个模板

最大的单一块。这些是 JAX/XLA 用户在程序格式错误时遇到的 shape-inference 和 HLO-verifier 诊断:rank 检查、维度范围检查、shape 相等检查、layout 不匹配。几乎全部重度使用 %s/%d,并且几乎全部包装 InvalidArgument(少数 RET_CHECK → Internal)。verifier 的特征是 vs. 比较风格,即 "%d vs. %d""%s vs. %s",会出现在两个量必须匹配的任何位置。

偏移模板(% spec)占位符Code
0x857d595Argument to Cholesky must have rank >= 2; shape was %s%s=ShapeInvalidArgument
0x857c6efArgument to symmetrize must have >= 2 dimensions, got %s%s=ShapeInvalidArgument
0x8580369All reduced tensors must have the same dimension. Tensor 0 has shape %s, Tensor %d has shape %s%s=Shape, %d=索引, %s=ShapeInvalidArgument
0x85803c9All operands to AfterAll must be tokens; operand %d has shape %s%d=operand idx, %s=ShapeInvalidArgument
0x872a259broadcast_dimensions contains invalid value %d for result with rank %d%d=值, %d=rankInvalidArgument
0xa02f654Broadcast dimension %d mismatch: %d != %d; %s and %s.%d=dim, %d/%d=大小, %s/%s=ShapesInvalidArgument
0xa02c26fCannot concatenate arrays that differ in dimensions other than the one being concatenated. Dimension %d in both shapes must be equal (or compatible): %s vs %s.%d=dim, %s/%s=ShapesInvalidArgument
0xa02f9baCannot bitcast types with undivisible bit-widths: %s => %s.%s/%s=PrimitiveTypeInvalidArgument
0xa01cdc3Bitcast requires a new on-device shape to have the same size of %d bytes, but got %d bytes.%d/%d=字节大小InvalidArgument
0xa030ae8Cannot infer shape: attempting to index into non-tuple: %s.%s=ShapeInvalidArgument
0x857b3fasharding's tile count and device count does not match: %d vs. %d; shape=%s, sharding=%s%d/%d=计数, %s/%s=Shape/ShardingInvalidArgument
0x858b317Arguments to TriangularSolve have shapes with different ranks: %s vs. %s%s/%s=ShapesInvalidArgument
0xa0a2a82Binary op shape inference: %s; lhs: %s; rhs: %s is not implemented.%s=op, %s/%s=ShapesUnimplemented
0x8728000Binary op expects 2 operands, but got %d%d=计数RET_CHECK→Internal
0x858400cBad scalar opcode in slot 0, opcode: %d bundle: %v, bits: %s%d=opcode, %v=TensorCoreBundle, %s=bitsInvalidArgument
0x857b280Cannot feed constants into bundle packer. Copy them to registers first. instr=%s%s=instructionInternal
0x8584e00Cannot find a free bundle slot in bundle %s: %s%s/%s=bundle/reasonInternal

注意 — 0x858400c("Bad scalar opcode in slot 0…")在反编译中解析到 platforms_deepsea::jellyfish::isa::DecoderBcsDf::DecodeScalar0Slot(以及一个逐代 DecoderJf twin)。它的 %vTensorCoreBundleAbslStringify,也是 6 个带 TensorCoreBundle<Code>StrCat factory 之一(见 参数类型解码)。逐代 decoder 变体(gxc/gfc、gxc/glc、vxc)各自发出自己的副本,因此文本几乎相同的模板、即使 prose 相同,也是在 rodata 中分离的字符串。


Scheduler / Fuel / FIFO — ~15 个模板

Latency-hiding scheduler、annotation 范围检查、--xla_fuel 预算,以及设备端 FIFO push/pop 顺序。annotation 模板依赖 %c 表示大括号/方括号字面字符;FIFO 模板串联 %s :: %s 来附加 instruction 上下文。

偏移模板占位符Code
0x8796ba8annotation arg must be in correct order as given; expected %c{%d%c but got %c{%d%c%c=brace, %d=idInternal
0x8571b33annotation %c{%d%c is out of bounds%c=brace, %d=idInternal
0x858a654annotation range was not closed; expected %c}%c: %s%c=brace, %s=contextInternal
0x857fa91async-done for %s must be scheduled before %s%s/%s=instructionsRET_CHECK→Internal
0x857fabfasync-done for %s must be scheduled on core %d before %s%s=instr, %d=coreRET_CHECK→Internal
0x858a9b8Cannot schedule FIFO pop instruction when the FIFO is empty %s :: %s%s :: %s=instr contextInternal
0x857ad7eCannot schedule FIFO push instruction when the FIFO is full. FIFO name: %s. (element count %d vs %d). %s :: %s%s%s=name, %d vs %d=计数Internal
0xa02bf0cConflicting schedule type requirements in computation rooted at %s.%s=computationInternal
0xa086733Reference instruction %s was not found in the schedule.%s=instructionInternal
0x862868fGVN: Not replacing %s because GVN is out of fuel%s=instruction(LOG,非 Status)
0x8628660halt before %s because lowering is out of fuel%s=instruction(LOG,非 Status)
0xa03ca6aIllegal value for --xla_fuel. Saw %s, but expected token %s to be an integer.%s/%s=value/tokenInvalidArgument

注意 — 0xa03ca6a("Illegal value for --xla_fuel…")已确认位于 xla::MakeDebugOptionsFlags flag 解析闭包内,它是 flag 值校验器,也就是 flag-name 侧所记录 --xla_fuel flag 的错误对应项。0x8628660/0x862868f 两个 "out of fuel" 字符串是诊断 LOG 输出,不是 Status 载荷(没有 factory 包装);这里为完整性收录,但不应假定它们能恢复为 Status code。


MSA / Memory / Allocation — ~79 个模板

Memory-space assignment、prefetch/alternate-memory、HBM defragmentation 和 heap allocator。字节大小占位符占主导(%lld%zu)。两个超预算模板包装 ResourceExhausted;不匹配/校验模板包装 Internal

偏移模板占位符Code
0xa030cd9AllocateBufferForMemorySpace: Unsupported memory space: %s.%s=memory spaceInvalidArgument
0x858aa58Allocation (size=%lld) would exceed memory (size=%lld) :: %s :: %s%lld/%lld=大小, %s :: %s=contextResourceExhausted
0xa083c62BufferAllocation::Slice for instruction %s at index %s cannot be determined at compile-time.%s/%s=instr/indexInternal
0x857ce50DefineBuffer: Mismatch in memory spaces: %s vs %s%s vs %s=spacesInternal
0x8584ecdError defragmenting HBM %s: %s%s/%s=region/reasonInternal
0xa1300d0Failed to allocate %zu bytes. Memory limit: %zu bytes. Used: %zu bytes.)%zu×3=req/limit/usedResourceExhausted
0x8728f50Invalid HBM offset %d%d=offsetInvalidArgument
0x872cb26Invalid memory space for input memory space colors: %d%d=colorInvalidArgument
0xa01cf08Out of memory allocating %d bytes.%d=字节大小ResourceExhausted
0xa09a63eNumber of bytes %lld allocated must be a multiple of chunk size %lld.%lld/%lld=size/chunkInvalidArgument
0x857eed1Register allocator verification failure: live range %s; instruction %s%s/%s=range/instrInternal
0xa02b12fScoped allocation with size %s and limit %s exceeded scoped %s limit by %s.%s×4=sizes/labelsResourceExhausted

注意 — 不要把近似相同的模板 0xa13e8e5("Failed to allocate node (%zu bytes). Memory limit: %zu [bytes]. Used: %zu [bytes].)")归因于 TPU memory-space assignment。它位于 perfetto::protovm::RwProtoCursor::CreateNodeFromField,即 Perfetto tracing library 的 arena allocator,而非 TPU MSA 路径。真正的 TPU heap-allocator 超预算字符串是 0xa1300d0("Failed to allocate %zu bytes…",上表行),已在 xla::AlignedAllocator::Allocate 中确认。


ICI / Collective — ~90 个模板

Inter-chip-interconnect 链路健康、routing、GTC 同步,以及 collective(all-reduce / all-gather / reduce-scatter / all-to-all)缓冲区大小校验器。本块混合了 printf 风格的 ICI driver 错误和位置 $N collective 校验器(带 megascale 标记,见下一节)。Code 倾向于 InternalDeadlineExceeded

偏移模板习惯用法占位符Code
0xa0b3abcCannot find unicast link next hop routing table for link port %d.printf%d=portInternal
0xa030c51Coordinate assignment failed for the slice's target %s ICI network because there are chips disconnected from the rest of the slice: %s.printf%s/%s=target/chipsInternal
0xa0d5412Detected ICI link failures along %d dimensions, but only 1-dimensional link fault is allowed..printf%d=dim 计数FailedPrecondition
0xa05e59aFailed to add link information: chip %d already has a %c direction link.printf%d=chip, %c=axisInternal
0x855f8c6Failed to detect GTC reset before timeout %s expiresprintf%s=durationDeadlineExceeded
0x8727005Failed to turn down ICI link %d during slice reset, state=%dprintf%d=link, %d=stateInternal
0x871106bGTC failed to converge (max diff %d > %d) before timeout (%s) expiredprintf%d > %d=diff, %s=timeoutDeadlineExceeded
0x872a345Hop ID %d is out of bound of ICI route path with length %dprintf%d/%d=hop/lenInternal
0x8583d20ICI Probe failed. local port: %d name: %s took %d us. status: %sprintf%d=port, %s=name, %d=us, %s=statusInternal
0xa0a81a9ICI resiliency only allow 1-dimensional link failures, but link failures along %d dimensions are discovered.printf%d=dim 计数FailedPrecondition
0xa0ba533ICI routing failed to retrieve %dth hop dimension from bit encoded cache data.printf%d=hopInternal
0x9a573e9ALL_REDUCE Output buffer size is not == Input buffer size. Input size: $0 Output size: $1 Group Size $2 Key: $3 Module: $4 MegascaleInfo: $5positional$0..$5=sizes/key/module/infoInvalidArgument
0x9c142f9ALL_GATHER Input buffer size is not (Output buffer size / group size). Input size: $0 …positional$0…=sizes/infoInvalidArgument
0x9c14380REDUCE_SCATTER Output buffer size is not (Input buffer size / group size). …positional$0…=sizes/infoInvalidArgument
0x9d1493eALL_TO_ALL not supported when buffer size is not divisble by number of endpoints. Buffer Size: $0 Number of endpoints: $1. MegascaleInfo: $2positional$0/$1/$2Unimplemented

注意 — 0x8583d20("ICI Probe failed…")已在 asic_sw::driver::deepsea::ici::SliceConfiguration::GetLocalTopology 中确认;0x9d1493e("ALL_TO_ALL not supported…")已在 xla::megascale::runtime::HostCommandSchedulerFactory…GenerateCommunicationIrsFromTransferRegistry 中确认。collective 缓冲区大小校验器虽然位于 ICI/collective 路径上,却是位置形式($N):它们由 megascale runtime 发出,而这里是位置习惯用法的大本营。


Megascale(DCN Runtime / Aggregator)— ~21 个模板

跨主机 data-center-network 协调:barrier 参与者记账、损坏缓冲区检测器、launch-id timeout,以及 coordinator 的错误摘要。这是位置 $N 习惯用法的核心。coordinator 的 hang 摘要会为每个 cause branch 发出一种 prose 变体。

偏移模板占位符Code
0x9e6f85eExtra barrier participant. Expected: $0 Message $1$0=expected, $1=msgInternal
0x9e6fa4dMismatched number of barrier participants: Expected: $0 Msg: $1$0=expected, $1=msgInternal
0x9d149cbMegaScale Corrupted Buffer Detected. Key: $0 Checksum at Sender: $1 Current checksum: $2$0=key, $1/$2=checksumsDataLoss/Internal
0x9b273a4Timed out waiting for $0 graphs to complete at launch_id $1. Already completed: $2. StepGloballyInProgress: $3 Timeout: $4$0..$4=count/id/stateDeadlineExceeded
0xa122d03MegaScale devices cannot be queried except from jax. (%d)%d=error codeFailedPrecondition

coordinator 的 hang 摘要会针对每个 cause branch(BAD_TPU_CHIP、BAD_SC_CHIP、DATA_INPUT_STALL、DIFFERENT_MODULE、FINGERPRINT_MISMATCH、NETWORKING_ISSUE、PROGRAM_NOT_QUEUED、UNKNOWN_CAUSE)发出 prose "Megascale detects a hang that is likely caused by …",以及面向 operator、可行动的后续建议("Please remove the hosts from the fleet and restart the workload""Please check the workers to make sure the data input pipeline is working properly")。abort 路径见 Fatal / Abort 表面


SparseCore / Embedding — ~68 个模板

SparseCore(xla_sc_)和 BarnaCore embedding 配置:alignment 要求、table/feature 计数、partitioner objective enum,以及 SMEM row-pointer 预算。面向用户(模型配置错误),且几乎全部为 InvalidArgument

偏移模板占位符Code
0xa0a9715barna_core_infeed_queue_hbm_address must be %d-byte aligned.%d=alignmentInvalidArgument
0xa0b6320barna_core_infeed_queue_hbm_size must be a multiple of %d.%d=multipleInvalidArgument
0xa030dd4Could not find valid TPU batch of length at least %d at position %d for row %d. The embedding work in one sample exceeds what the BarnaCore can process: %s. %s.%d×3=len/pos/row, %s/%s=detailInvalidArgument
0x872d950Dynamic learning rate tag: %d not found in the TPU embedding configuration, instead found: %d. tag set size: %d%d×3=tag/found/sizeInvalidArgument
0xa02cd93Embedding table is expected to have element type %s or %s.%s/%s=typesInvalidArgument
0x86fa1adFailed to parse TPU embedding partitioner optimization objective "%s". Valid options: performance, hbm_usage, hybrid%s=valueInvalidArgument
0xa11773ahbm_limits_for_embeddings.min_fraction (%f) must be <= hbm_limits_for_embeddings.max_fraction (%f)%f/%f=fractionsInvalidArgument
0xa0d36d1Invalid num_features: %d found for table: %s in the TPU embedding configuration. Valid values are >0.%d=计数, %s=tableInvalidArgument
0xa0b7076Logical replicas must evenly divide the SparseCores in the system. logical_replicas = %d, physical_sparse_cores = %d.%d/%d=计数InvalidArgument
0xa069f4eNumber of TPU tables on row: %d exceeds what the BarnaCore hardware supports: %d > %d. This is mostly likely a result of incorrect partitioning.%d×3=row/got/maxInvalidArgument
0xa0ff40eRow pointers would exceed available SCS Smem (%d bytes > %d bytes)%d/%d=used/availResourceExhausted
0xa07d172Scatter operand has %d elements, which exceeds the 32-bit limit. Unsupported on SparseCore.%d=计数Unimplemented

注意 — 0xa0d36d1("Invalid num_features…")已在 tensorflow::PopulateMissingFieldsInTPUEmbeddingConfig 中确认。0xa11773a 中的 %f 浮点数是少见情形:%f 真的是配置比率,而非 cost-model 内部值。


Runtime / Driver / PJRT — ~177 个模板

driver state machine、device/ordinal 校验、firmware-queue 转换、DMA-buffer 记账,以及 PJRT C-API 边界。习惯用法最混杂的一块:这里会出现 %perrno%d),code 在 FailedPrecondition(state-machine guard)和 Internal 之间分裂。

偏移模板占位符Code
0xa0b7366Attempted to register programmable interrupt with bad index: %d. Number of programmable interrupts: %d.%d/%d=index/countInvalidArgument
0xa0430a3Cannot remove a driver for %s, was not found in map.%s=driver nameNotFound
0xa077a78Cannot transition to %s: the firmware queues are not in %s state; they are in %s state.%s×3=statesFailedPrecondition
0x96c33b2Can't close driver while in state %s; are multiple threads trying to open / close?%s=stateFailedPrecondition
0x94b68ceCan't get the optimized program for executable \%s`: MPMD execution is not supported by PJRT C API`%s=executableUnimplemented
0xa09fdcdChip count (%d) is not supported.%d=计数InvalidArgument
0x872d1a0Close of core dump fd failed with errno: %d%d=errnoInternal
0xa0a9937%d DMA buffers were still outstanding when the driver was re-opened. These buffers must be unmapped before the driver can be re-opened.%d=计数FailedPrecondition
0xa0d10bdDevice id '$0' is out of bound. Number of devices is $1.$0/$1=id/countInvalidArgument
0x8679159device ordinal value (%d) must be non-negative%d=ordinalInvalidArgument
0xa1a96baexecutable is built for device %s of type "%s"; cannot run it on device %s of type "%s"%s×4=device/typeInvalidArgument
0xa00ab4bExpected %d chips per tray, actually found a tray with %d chips.%d/%d=expected/foundFailedPrecondition
0x858a3defailed initializing StreamExecutor for device ordinal %d: %s%d=ordinal, %s=reasonInternal
0xa09b555Failed to convert multipod chip id %d to single-pod chip id.%d=chip idInternal

注意 — 0x8679159("device ordinal value (%d) must be non-negative")已在 stream_executor::StreamExecutorAddressAllocator::GetStreamExecutor 中确认。许多 runtime 模板会进入 executor 的 async stream 路径;参见 execute-async-on-stream.md,了解它们在入队期间出现的位置。

PJRT-C-API / protobuf-descriptor(链接库,非 TPU)

位置 $N 家族也由一个静态链接的 protobuf extension-declaration 校验器填充。这里为完整性收录,但它们不是 TPU code,不应归因于 libtpu 自身表面:

偏移模板说明
0xa0cf40b"$0" extension field $1 is expected to be $2.protobuf descriptor 校验器
0xa0eba97"$0" extension field $1 is expected to be type "$2", not "$3".protobuf descriptor 校验器
0xa0d0fab$0 cannot declare both \metadata` and `declaration` as extension declaration for extension #$1.`protobuf descriptor 校验器

CHECK / RET_CHECK / 自检模板

absl CHECK/QCHECK/DCHECK 家族和 XLA 的 TPU_RET_CHECK 不携带完整消息模板:它们发出固定前缀,随后追加字符串化的源表达式和任何流式传入的 << "msg"。比较 macro(CHECK_EQ/NE/GE)会用 verifier 块中反复出现的 "%d vs. %d" / "%s vs. %s" 格式追加操作数值。

偏移模板Macro / 来源
0xa1a64deCheck failed: 'absl CHECK/QCHECK 前缀(带引号)
0xa1f4a87Check failed in带 file/line 的 absl CHECK
0xa285fb1Check failed:absl CHECK 前缀(无引号)
0xa183292TPU_RET_CHECK failure (XLA TPU RET_CHECK macro
0xa0ab3caHostname Verification Check failed.gRPC TLS hostname-verify CHECK
0xa2300c5MakeErrorStream destructed without getting absl::Status:XLA status_macros 自检
0xa2300ffMakeErrorStream shift called after getting absl::Status:XLA status_macros 自检
0xa27f1b3MakeErrorStream got absl::Status more than once:XLA status_macros 自检

陷阱 — CHECK(expr) << "msg" macro 会把源表达式文本spmem_buffer_type != nullptrdynamic_size, nullptr)内联进 .rodata。大量 C++ 源码字面片段,甚至整段 lambda body,都会作为副作用出现在字符串表中。这些是 CHECK 条件证据,不是 printf 意义上的错误模板,已从模板计数中排除。扫描字符串寻找 "templates" 的重新实现者必须过滤它们,否则会被源码片段淹没。

0xa183292 已在 tpu::internal::RetCheckFailSlowPath 中确认;0xa2300c5 已在 xla::status_macros::MakeErrorStream::Impl::~Impl 中确认(当一个已构造但未消费的 Status 被丢弃时触发的析构自检)。


Fatal / Abort 表面 {#fatal--abort-surface}

有意 abort 的表面很小且集中:ICI 硬故障、megascale coordinator abort、内部 bug LOG(FATAL),以及一个非 TPU 专属的 library guard。

偏移模板路径
0xa1e7cb3!!!! FATAL ERROR !!!! forICI FatalErrorCheck
0x8864055!!!! FATAL ERROR !!!! observed errors are: [AsyncDriver::HandleFatalError composite
0xa046045Fatal error occurred. Data links will go down.ICI hard-failure marker
0xa1b3810FATAL ERROR RECEIVED FROM HARDWARE!!!hardware fatal interrupt
0x8a2941dFatal error in creation of RWB Fusion. Please file a bug with XLA-TPUfusion internal-bug LOG(FATAL)
0xbe7d460FATAL ERROR: This binary was compiled with <isa> enabled, but this feature is not available on this processor (go/sigill-fail-fast).absl CPU-feature startup guard(12 个 ISA 变体)

megascale coordinator 的 abort prose:"Aborting the coordinator after collecting errors from all workers as megascale_error_reporter_abort_on_hang is set to true. All workers will also abort after they detect the coordinator is shutdown.",是由 flag gated 的 LOG(FATAL)

注意 — 0xbe7d460 家族(12 个变体:aes、avx、mmx、pclmul、popcnt、sse、sse2、sse3、sse4.1、sse4.2、ssse3、…)是 absl CPU-feature startup guard,即 library abort:如果 host CPU 缺少该 binary 构建时启用的 ISA,它会在任何 TPU code 运行前触发。它不是 TPU 诊断。0x8a2941d("Fatal error in creation of RWB Fusion…")已在 xla::jellyfish::TpuInstructionFusion::RwbFusionHelper 中确认,是带有 "file a bug" 标记的真正内部 bug fatal。


Status-Code 映射 {#status-code-mapping}

模板通过三种 factory 习惯用法之一变为 absl::Status。只有前两种在 binary 中命名 code;prose 家族和裸 printf-into-LOG 路径需要调用点反汇编才能确认 code。

习惯用法计数含义Code 归属
xla::status_macros::MakeErrorStream390RET_CHECK(c) << … / return InvalidArgument(…) << …;390 个 MakeErrorStreamWithOutput<T> conversion op,每个 StatusOr<T> 返回类型一个。按调用点计数是主导 XLA 习惯用法。code 由该位置的 macro 命名
xla::InvalidArgumentStrCat<…>38交错 literal/typed-arg factoryCERTAIN → InvalidArgument
xla::UnimplementedStrCat<…>31同上CERTAIN → Unimplemented
xla::InternalStrCat<…>29同上CERTAIN → Internal
xla::ResourceExhaustedStrCat<…>1同上CERTAIN → ResourceExhausted
absl::<Code>Error("…") prose1+直接 prose factory(例如 absl::InternalError("Invalid error type")code 由函数命名

<Code>StrCat factory 是唯一一个无需反汇编即可恢复 format-arg C++ 类型的地方:每个实例化都会发出 Itanium-mangled symbol _ZN3xla<len><Code>StrCatIJ<typepack>EEC2E…,其中 <typepack> 会逐字节命名每个交错 literal segment 和 typed argument。

参数类型解码(mangled type pack)

text
RA<N>_Kc   const char (&)[N]  — a literal segment of N chars (N counts the
                                 NUL, so the visible literal is N-1 chars)
RA<N>_S1_  another const char (&)[N] — a later literal (S1_ back-references
                                 the already-named const char type)
m / l / i / f          unsigned long(size_t) / long / int / float, by value
Rm / Rl / Ri / Rf      &  of each
RKm / RKi              const unsigned long& / const int&
NSt…basic_string…      std::string by value;  RKNSt… = const std::string&
NSt…basic_string_view… std::string_view
N3tpu10TpuVersionE     tpu::TpuVersion (enum), by value
N…isa16TensorCoreBundleE  the per-gen ISA TensorCoreBundle (rendered via its
                          AbslStringify → the %v sigil)
N4absl8StatusOrI…E     absl::StatusOr<…>, by value

解码示例(每个都是真实的实例化 symbol):

text
InvalidArgumentStrCatIJRA74_KcmEE
    → InvalidArgument("<73-char literal>", size_t)
InvalidArgumentStrCatIJRA16_KcRfRA20_S1_EE
    → InvalidArgument("<15ch>", float&, "<19ch>")     (the only float-bearing
      factory family — a cost/ratio message)
InvalidArgumentStrCatIJRA19_KcRlRA38_S1_RKiEE
    → InvalidArgument("<18ch>", long&, "<37ch>", const int&)
InvalidArgumentStrCatIJRA26_KcN3tpu10TpuVersionEEE
    → InvalidArgument("<25ch>", tpu::TpuVersion)
InvalidArgumentStrCatIJRA65_Kc…isa16TensorCoreBundleEEE
    → InvalidArgument("<64ch>", const TensorCoreBundle&)   (the "Bad scalar
      opcode … bundle: %v" family — per-gen gxc/gfc, gxc/glc, vxc variants)
UnimplementedStrCatIJRA253_KcEE
    → Unimplemented("<252-char literal>")   (the longest single-literal
      Unimplemented message — a "not supported" explanation block)

99 个 factory 中字节已确认的 arg-type 频率:std::string 29、std::string_view 15、long 13、TensorCoreBundle 6、size_t 5、StatusOr<…> 3、const int& 2、float 2、TpuVersion 1。即使在字节已确认层面,args 也绝大多数是类字符串 + longfloat 和 pointer 极少,这与上面的 printf-spec 分布一致。

StatusCode 关键词分布

字符串表关键词出现次数(已过滤噪声),这是表面偏向哪些 code 的文本信号上界,不是逐模板计数:

text
InvalidArgument 300 | Unimplemented 169 | NotFound 75 | FailedPrecondition 56
ResourceExhausted 21 | Unavailable 15 | OutOfRange 14 | Aborted 8
DeadlineExceeded 3 | AlreadyExists 3 | PermissionDenied 2 | DataLoss 1

陷阱 — 原始 Internal 关键词计数(~16,823)主要由内联 __FILE__ 字符串中的 /internal/ source-path 组件贡献,而非错误模板。不要把它解读为 Internal-status 计数。字节已确认的 Internal 总数是 29 个 InternalStrCat factory 加上 MakeErrorStream Internal 位置。


面向用户与内部的划分

prose 本身会标示受众。面向用户(operator / JAX-user 可行动、无 bug id):HLO verifier shape/rank/dimension 块、--xla_fuel flag 值错误、带有效值提示的 embedding-config 错误、megascale 摘要后续建议,以及 device/executable 不匹配。内部(XLA-bug 标记):带有 b/<id>go/<link>"please file a bug""should not happen" 的字符串。

偏移内部 bug 标记模板标记
0x858c562Kernel body fingerprint collision detected for key: … Please file a bug with the XLA team …"file a bug"
0x8a2941dFatal error in creation of RWB Fusion. Please file a bug with XLA-TPU"file a bug"
0x96c1211XLA has not implemented dynamic sized slice with non-trival stride yet. Please file a bug against XLA"file a bug"
0x96c12d4Unimplemented reduce-window in fusion cost modeling. Please file a bug with XLA"file a bug"
0x9fd476dtightened domains are empty. This should not happen except if we proven infeasibility or optimality."should not happen"
0xa0c9452Encountered unexpected layout … This should not happen - please file a bug against XLA."should not happen" + "file a bug"
0x99e1405Close() appears to be hanging, this might be a deadlock see b/147787375b/<id>

注意 — b/<id>go/<link> token 是最强的内部信号。它们也标记已知空缺 TODO("TODO(b/157237781) support VFIO device %s""TODO: b/475913712 - Expected Gather indices to be bitpacked …"),这些不是活动错误,但当其受保护路径被命中时会变为 Unimplemented 消息。


概览:各子系统模板数

子系统候选主导习惯用法主导 code受众
Compile / HLO / verifier~875StrFormat %s/%d + RET_CHECKInvalidArgument面向用户
Runtime / driver / PJRT~177StrFormat %s/%d/%p + errnoInternal / FailedPrecondition混合
ICI / collective~90StrFormat %d/%s + $N (msc)Internal / DeadlineExceeded混合
MSA / memory / allocation~79StrFormat %zu/%lldResourceExhausted / Internal混合
SparseCore / embedding~68StrFormat %d/%s/%fInvalidArgument面向用户
Megascale (DCN runtime)~21Substitute $0/$1Internal / LOG(FATAL)混合
Scheduler / fuel / FIFO~15StrFormat %s/%d/%cInternal / RET_CHECK混合
PJRT-C-API / protobuf-desc(lib)Substitute $0/$1InvalidArgument(非 TPU)
不同模板总数~2,9372,799 printf + 138 positional

陷阱 — 子系统计数是在 prose 关键词集上按 first-match-wins 计算的(顺序:compile → MSA → ICI → SparseCore → runtime → scheduler → megascale)。匹配两个组的模板只计一次,归入较早的组。总数(~2,937)带有 ±~50 的 caveat,因为 error-prose gate 会纳入少量 help strings 或排除少量短错误;这些正确归属为提示字符串(hint-strings.md),而不是错误。


交叉引用