# Clockless Inkling Architecture Atlas — evidence Snapshot: September 11, 2026, America/Toronto. This is a research explanation and visualization, not a new experiment, training run or deployment. ## Sources and authority - Current goal: `PROJECT_GOAL.md`; current evidence: `autoresearch/current_state.md`; local agreements: `AGENTS.local.md`. These include uncommitted working-tree updates. - Local HEAD: `4977b750a63e7ff12a5012365615861138ba15e5`. Source notes below distinguish local files from pinned source. - Fresh listener branch: `codex/listener-handoff-2026-09-06` at `35f1233d54fe675f6b8dfd34a740b2065db076b3`, fetched without switching the dirty checkout. - Linear: Clockless Inkling project, all current project issues, architecture tickets SPE-123/126/129/132/135/138, add-ons SPE-147/148/100/149, listener SPE-91/92/93/94/96/97, scoring and later delivery documents. - Latest primary training issues amend older architecture-plan prose: matched SFT-only and SFT+OPD after common warmup is a proposal, not selected training. ## Listener viewer access The first isolated browser attempt returned401. After the user authorized reading Zayaan’s iMessage thread, the original access link was recovered and used to verify bounded metadata for Oto69. The credential stays outside the repository and is not included in this guide. Oto69 is a training recording, task-oriented,838.042 seconds,341 raw annotations across two speakers. No audio was downloaded in this verification. ## Diagram conventions Solid paths describe supported computation or transport. Dashed paths mark proposals, missing learned integration or schematic expansion. An implemented component is not necessarily a qualified or deployed component. The complete system view is the target architecture; the preserved companion starts speech at trunk exit. Six named arms times sixteen optional add-on subsets creates 96 visual configurations. A6 is an initialization experiment over another selected topology, not a sixth distinct network topology. Reader placement, feature view and training recipe are separate axes. The planned execution strategy screens add-ons and transfers winners rather than launching the full Cartesian sweep. The spatial rendering is an independently authored Three.js scene. Its depth, animation and spacing are illustrative, not hardware placement, measured execution time or tensor-value visualizations. ## Scope of the source review The review covered repository structure, authority documents, relevant runtime/model/data families and latest Linear plans. It did not read every training record or sealed test, replay every past experiment, load large model checkpoints, or measure deployed behavior. Source-level geometry and recorded results are distinguished below. --- # Clockless Inkling: independent architecture/source review Reviewed 2026-09-11; repo `/Users/parsabahraminejad/Desktop/Experiments/clockless-inkling`. Source inspected in the working tree at HEAD `4977b750a63e7ff12a5012365615861138ba15e5`; this is not a claim that all inspected bytes are committed. Scope: trunk internals, mouth computation, data/recipe contracts and Sources coverage. No six-candidate mapping or UI work. Method: local file reads only; no model loading, inference, tests, GPU calls, network calls or external writes. Only this report was written. Paths below are repo-relative, including `.venv/` dependency sources. Those dependency sources require separate packaging if exposed in the guide. Current authority: `AGENTS.local.md:21-38`, `PROJECT_GOAL.md:55-89`, `autoresearch/current_state.md:5-16`. The four-depth architecture is an existing candidate; depth/composition/recipes remain open. ## 1. Trunk geometry and one-block diagram - Inkling-Small has hidden width **4096**, **42 trunk blocks**, indexed **0–41**, and 32 query / 8 KV heads of dimension 128. Source: `export/inkling_small_meta/config.json:7-22`. - Configured maximum context is 1,048,576 tokens; this is configuration, not a measured end-to-end context/latency guarantee. Source: `export/inkling_small_meta/config.json:8`. - Trunk attention pattern: five local blocks, then one global, repeated seven times. Global indices are **5,11,17,23,29,35,41**; local window is **512**. Derived from `export/inkling_small_meta/config.json:24-70` and construction at `autoresearch/notes/sglang_pinned/python__sglang__srt__models__inkling.py.txt:592-606`. - First **two** blocks (0,1) use dense MLPs; blocks 2–41 use MoE. The condition is `layer_id < dense_mlp_idx`, not “block 2 is the only dense block.” Sources: `export/inkling_small_meta/config.json:61-79`; `autoresearch/notes/sglang_pinned/python__sglang__srt__models__inkling.py.txt:187-210`. - Dense intermediate width is **16384**; each routed expert's configured width is **2048**; MoE uses **256 routed experts, top-6 per token, plus 2 shared experts**. Source: `export/inkling_small_meta/config.json:71-81`. - The local Torch router chooses top-6 with sigmoid scores plus correction bias, normalizes selected routed and shared scores together, then applies route/global scales. Shared expert contributions are added to routed outputs. This is local implementation evidence, not a new native-kernel parity check. Source: `.venv/lib/python3.13/site-packages/transformers/models/inkling/modeling_inkling.py:342-425`. - Semantic block diagram, before considering SGLang's deferred execution: `x → RMSNorm → Attention → attention-output SConv → +x → RMSNorm → dense/MoE → MLP-output SConv → +residual`. Source: `.venv/lib/python3.13/site-packages/transformers/models/inkling/modeling_inkling.py:575-592`; the mouth's equivalent is `mouth_engine/row_depth.py:30-45`. - Inside attention: Q projection; K projection→K SConv; V projection→V SConv; Q/K per-head RMSNorm; grouped-query attention; output projection. K/V are cached after convolution, and K after its norm. Sources: `.venv/lib/python3.13/site-packages/transformers/models/inkling/modeling_inkling.py:228-243`; `mouth_engine/row_attention.py:121-152`. - Each block therefore has **four** depthwise causal SConvs: K, V, attention output, MLP output. Kernel width is **4**, and each SConv also adds its own input residual. Sources: `export/inkling_small_meta/config.json:62-63`; `mouth_engine/row_depth.py:48-61`; `mouth_engine/row_attention.py:27-46`. - The attention position mechanism is learned, hidden-conditioned **relative logit bias**, not a generic RoPE box: `r=W_r h`, learned distance profiles, zero bias beyond the configured extent. Sources: `mouth_engine/row_attention.py:49-74`; `mouth_engine/inkling_layers.py:160-175,193`. - Attention uses **1/head_dim**, not the usual 1/sqrt(head_dim), after Q/K normalization. Nonlocal layers apply long-context log scaling to Q and relative bias beyond the 128,000 floor. Sources: `mouth_engine/inkling_layers.py:163`; `mouth_engine/row_attention.py:134-149`; `export/inkling_small_meta/config.json:20-21`. - Trunk input: normalized text embeddings plus native multimodal embedding routes. Final output: final RMSNorm, then divide by **16**, then unembed. Sources: `autoresearch/notes/sglang_pinned/python__sglang__srt__models__inkling.py.txt:578-587,927-932,1165-1198`. - Semantic vocabulary is **200,058**. Config `vocab_size=201024` is not the legal output-token count; the previous 200,064 mouth size also included six invalid tokenizer rows. Sources: `export/inkling_small_meta/config.json:12,64`; `mouth_engine/constants.py:1-10`; `mouth_engine/load.py:91-99`. ## 2. Exact meaning of source taps and the wider layer views - Standard capture selects **13,27,41**, three evenly spaced zero-based trunk indices. Source: `modal_mouth_warmup.py:82`; `patch_sglang_native_readout.py:21-28`. - Critical native detail: SGLang defers each block's MLP-output SConv until the next block's entry. Its auxiliary tap is `hidden_states + residual` **before that block's deferred MLP SConv**. Source: `autoresearch/notes/sglang_pinned/python__sglang__srt__models__inkling.py.txt:813-838`. - Thus an auxiliary “post-layer” tap is not interchangeable with the completed HF decoder-block output. Put the tap before deferred MLP SConv in an implementation-level drawing; annotate the distinction in a simplified drawing. - For layer 41, native final is `final_RMSNorm(MLP_SConv41(MLP_branch41) + residual41)`, whereas aux41 is `MLP_branch41 + residual41`. Source: `autoresearch/notes/sglang_pinned/python__sglang__srt__models__inkling.py.txt:831-838,927-932`. - **RMSNorm(aux41) ≠ native final**; applying a norm to the already-added auxiliary tensor cannot reconstruct the omitted branch convolution. Source: `patch_sglang_native_readout.py:4-19`. - Preserved four-depth companion view is `[aux13, aux27, aux41, RMSNorm(aux41)]`, shape `[B,4,4096]`; the fourth member is a derived legacy stream. Source: `mouth_engine/companion.py:447-455`; `graft/clean_mouth_sft.py:1075-1088`. - Correct native capture view is `[aux13, aux27, aux41, actual_native_final]`; native final is undivided, captured before the muP division. Source: `patch_sglang_native_readout.py:51-73`. - The clean-SFT preparation API explicitly distinguishes `legacy_aux_norm` and `native_final`; the latter requires actual captured arrays and cannot silently synthesize them. Source: `graft/clean_mouth_sft.py:1111-1155`. - All-layer view is **42 raw auxiliary streams + native final = 43 streams**, not 43 trunk blocks. Source: `graft/shared_mouth_feature_views.py:46-72`. - Stored all-layer rows have **176,128 FP32 values**, or **704,512 bytes / 688 KiB per token row** (derived from 43×4096×4). Names are `raw_aux_0` … `raw_aux_41`, then `native_final`. Source: `graft/shared_mouth_feature_views.py:46-72`. - The all-layer/final patch is a separate mutually exclusive capture mode from the three-tap/native-final patch; it validates exactly layers 0–41, 4096 width and identical row shapes. Source: `patch_sglang_feature_view_readout.py:27-68`. - Capture uses FP32 because deep residuals can overflow FP16; a stale embedded comment says “fp16” but the executable cast is FP32. Source: `patch_sglang_hidden_all_layers.py:77-94`. - All-layer recapture replays exact previously consumed prefixes, preserves target/history bindings, and discards one instrumentation sample. New processing clocks do **not** establish original runtime or layer-by-layer availability. Sources: `graft/shared_mouth_feature_views.py:1-7,90-107,173-180`. - Feature views identify exact manifest, source identity, cohort, thresholds, stream names and optional sidecar manifest by hashes. A view is data access, not a trained all-layer reader. Sources: `graft/shared_mouth_dataset.py:245-264`; `graft/shared_mouth_feature_views.py:1-7`. ## 3. Existing mouth equations and native MTP distinction - Shipped model has **8 MTP depths**. Native local depths are **0,2,4,5,6,7**; depths **1,3** are global. `chain_hidden_post_norm=false`. Source: `export/inkling_small_meta/config.json:102-112`. - Every MTP depth uses a **dense** MLP, even though most trunk blocks use MoE. Sources: `autoresearch/notes/sglang_pinned/python__sglang__srt__models__inkling.py.txt:1759-1779`; `.venv/lib/python3.13/site-packages/transformers/models/inkling/configuration_inkling.py:144-148`. - One depth takes two H-wide inputs: state and token embedding. It computes `D_k(h,e) = Block_k(W_k [N_hidden,k(h); N_embed,k(e)])`, with `W_k:8192→4096`. Source: `inkling_mtp.py:55-85`. - Token route includes **two distinct learned norms**: base embedding norm, then the depth's own embedding norm. Sources: `mouth_warmup.py:116`; `inkling_mtp.py:73-80`; native implementation `autoresearch/notes/sglang_pinned/python__sglang__srt__models__inkling.py.txt:1789-1796`. - Existing four-depth front end is one shared fusion and one shared token gate, followed serially by depths 0→1→2→3. There are not four independently trained fusion routers. Sources: `mouth_engine/load.py:85-103`; `mouth_engine/mouth_step.py:101-111`. - Fusion: `f = W_f [N_0(s_0); …; N_3(s_3)] + b_f`. Each stream has its own trainable RMSNorm; concat width 16384 projects to 4096. Source: `mouth_conditioning.py:185-206`. - Gate: `g = sigmoid(W_g [f;e_gate] + b_g)` and `c = g⊙f + (1−g)⊙e_gate`. This is a **4096-dimensional elementwise gate**, not one scalar. Source: `mouth_conditioning.py:50-80`. - The gate's embedding table is frozen/nonpersistent. Gate token `−1` uses a learned no-token embedding. The separate block-input token must remain a legal semantic token. Sources: `mouth_conditioning.py:54-68`; `mouth_engine/functional.py:132-146`. - Same-token composition: `h_0=D_0(c,e_t)`, `h_1=D_1(h_0,e_t)`, …, `h_3=D_3(h_2,e_t)`; **the same token embedding enters every depth**. The chain emits one next-token distribution from its last depth. Sources: `mtp_chain.py:18-35`; `mouth_engine/mouth_step.py:105-111`. - Preserved custom readout is `logits = (N_trunk_final(h_3)/16) Uᵀ`; the norm and unembed are frozen base buffers. Source: `mouth_warmup.py:97-104`. - Native MTP readout with the released false chain-norm flag is **raw depth output/16 → Uᵀ**, with undivided raw output handed onward. Source: `autoresearch/notes/sglang_pinned/python__sglang__srt__models__inkling.py.txt:1752-1757,1831-1845`. - Native future-offset indexing: trunk state at position `i` plus token `x[i+1]` supports depth-0 prediction of `x[i+2]`; the next native depth advances the token to the next candidate, targeting `x[i+3]`, etc. Local diagnostic explicitly advances `current=out`, `fed=pred`, and scores `tokens[p+2+k]`. Sources: `mouth_warmup.py:3-14`; `modal_mtp_diagnose.py:838-855`. - That diagnostic has historical readout choices; use it only for its offset indexing. Original upstream pretraining trainer/loss was not located in inspected sources, so no claim about its exact loss weights, full training schedule or measured native acceptance follows. - Same-token four-depth composition is therefore a learned repurposing of native NextN blocks, not native eight-offset prediction and not speculative trunk verification. Sources: `mtp_chain.py:1-6,26-35`; `PROJECT_GOAL.md:62-65`. - `NativeDepth0Head` bypasses fusion/gate, selects the last/native-final stream, uses raw readout, and has one depth. It is explicitly an out-of-distribution whole-reply decoder, not evidence of native standalone speech capability. Source: `mouth316_native_head.py:13-40,129-147`. - `LayerMixer` is a learned scalar-softmax weighted sum; `AttentionMixer` attends **across layer streams at one position**, not across time. Both are alternatives in source, not the default trained fusion. Sources: `mouth_conditioning.py:26-47,159-182`; `mouth_warmup.py:73-87`. - Optional `CopyPointer` mixes generation and causal context-copy distributions; the default preserved loader does not enable it, and the functional four-depth primitive rejects it. Sources: `mouth_conditioning.py:99-124`; `mouth_warmup.py:91-93`; `mouth_engine/functional.py:73-77`. ## 4. Time, recurrence, caches and actual companion behavior - Keep three axes distinct in diagrams: **trunk layer index**, **MTP depth**, and **mouth token/time step**. Serial depth arrows alone do not indicate temporal recurrence. - Each mouth depth owns independent KV and four SConv histories. Slots isolate requests; source and speech tokens advance the same mouth sequence. Sources: `mouth_engine/mouth_step.py:55-69,101-111`; `mouth_engine/functional.py:39-61,158-168`. - KV persistence and feeding generated tokens back already provide history. The default cached mouth nevertheless rereads a fixed exit-state row at every reply step; it does **not** feed last raw mouth hidden into its next conditioner. Sources: `graft/cached_mouth.py:429-455,539-541`; `mouth_engine/mouth_step.py:101-108`. - Explicit native0 `self_predicted` recurrence changes the next state input to the previous raw depth output after the first reply step; no extra final norm is applied. `exit_constant` preserves the fixed conditioner. Source: `mouth316_native_head.py:240-249`. - Functional native0 training implements the same choice, retains gradients through KV/SConv/hidden history, and uses teacher tokens for later block inputs. Sources: `mouth316_recurrent_training.py:194-241`; `mouth316_recurrent_training.py:70-150`. - Functional four-depth code stores each depth's `last_hidden`, but each new query still starts from `gate(fusion(rows))`; storage alone does not wire a cross-token raw-hidden feedback edge. Source: `mouth_engine/functional.py:143-170`. - Four-depth query/commit is explicit: query computes a candidate state; HOLD retains the original state; commit rejects a stale candidate. This is an arithmetic/ownership primitive, not trained turn-taking. Source: `mouth_engine/functional.py:7-23,174-186`. - Preserved mouth attention is windowed at **512 across all depths**, including native-global depths 1/3. This custom mask/cache contract must not be diagrammed as native unlimited history. Sources: `mouth_warmup.py:119-134`; `mouth_engine/mouth_step.py:55-68`; `mouth_engine/row_attention.py:92-99`. - KV rings evict direct attention access, not earlier effects carried through subsequent states. Physical ring position is modulo capacity, while logical positions remain absolute for relative bias. Source: `mouth_engine/slot_cache.py:83-89`; `mouth_engine/row_attention.py:68-74`; `mouth316_recurrent_training.py:112-124`. - SConv receptive field is current input plus three previous positions. Engine buffers physically allocate four entries; functional training retains three prior inputs. Avoid literal cache-size claims from the engine's “kernel−1” docstring alone. Sources: `mouth_engine/slot_cache.py:50-55`; `mouth_engine/inkling_layers.py:112-120`; `mouth316_recurrent_training.py:74-85`. - Trace alignment: consume each trace pair once; drop the pair whose next token was `<|content_text|>`; feed that marker once as seed, paired with the state **at the marker's consumed position**. Source: `graft/trace_batch.py:40-50`; `graft/cached_mouth.py:12-22,350-378`. - The companion currently **LISTENS** through trunk rows and surfaces speech only at the seed or while already SPEAKING. Its exit tail rereads the last row. This does not implement freely chosen speech throughout trunk work. Source: `mouth_engine/companion.py:432-457,957-989`. - Cross-request continuation restores mouth caches, token state and last rows by session/turn identity; root/new-turn paths reset. This is narrower than long-call semantic continuity or learned listener control. Source: `mouth_engine/companion.py:277-315,375-385`. - `mouth_engine/listener.py` adds a detachable zero-initialized language residual and a control head reading committed state; its initial outputs have no trained behavior. Sources: `mouth_engine/listener.py:1-11`; `mouth_engine/functional.py:147-157`. ## 5. Data and training contracts to expose - Freeze the trunk; train detachable MTP/conditioning artifacts. `WarmupHead.trainable_parameters` selects depth, fusion, gate and optional pointer, excluding base embedding/unembedding/norm buffers. Sources: `PROJECT_GOAL.md:62-65`; `mouth_warmup.py:97-100,210-217`. - Native-style text warmup uses aligned source streams at `i`, gate token `x[i]`, block input `x[i+s]`, target `x[i+s+1]`. Loss is `Σ_s 0.8^(s−1) CE_s`, with full backpropagation through normalized own-hidden unroll and teacher-forced tokens. Source: `mouth_warmup.py:161-208`. - Warmup unroll time is separate from `DepthChain` depth: one unroll step can execute all four depths. Do not draw four-depth training as automatically four future-label losses. Sources: `mouth_warmup.py:184-208`; `mtp_chain.py:26-35`. - Clean SFT sequence: trimmed trace inputs → content-text seed → approved target tokens except the last; loss covers **all target tokens including end-message**, never prefix tokens. Source: `graft/clean_mouth_sft.py:1156-1184,1195-1215`. - SFT objective aggregates CE sums divided by actual total target-token count, not an unweighted mean of example means. Only train split enters `sft_step`; mixed feature views are rejected. Source: `graft/clean_mouth_sft.py:1229-1249`. - Strict capture readers require explicit eligibility and hashes; successful capture does not automatically admit labels. Sources: `graft/clean_mouth_sft.py:5-20,194-214`; `graft/shared_mouth_dataset.py:559-594`. - Shared records separately bind conversation, visible events, decision availability, native captures, speech history, reviewed target, tool executions and listener/playback evidence. Source: `graft/shared_mouth_data.py:205-229,471-506`. - Causality checks reject unavailable states, wrong exact prefixes, post-input teacher-answer states and stale speech history. Candidate speech requires fresh downstream native recapture. Sources: `graft/shared_mouth_data.py:1190-1228`; `graft/shared_mouth_opd_data.py:1364-1388`. - OPD preparation freezes source prompts/tool worlds; only a specific current student's sampled trajectory is on-policy. Identity includes checkpoint hash, optimizer step and both full-model and actual sampling log-probabilities. Source: `graft/shared_mouth_opd_data.py:1-6,1197-1235,1525-1533`. - Teacher scoring consumes the exact student prefix, `prompt + sampled_ids[:-1]`, with one predictor position per sampled token; scoring artifacts are privileged. Source: `graft/shared_mouth_opd_data.py:1239-1263`. - New teacher policy specifies actual native-final readout, semantic vocabulary, divisor 16, temperature and top-K plus the **full normalizer and sampled-token score**. Source: `graft/shared_mouth_opd_data.py:47-75,1684-1748`. - New CPU KL reference uses teacher top-K plus one residual-mass bucket; this is a **coarsened forward KL**, equaling full-vocabulary KL only at K=V. It is not an optimizer implementation. Source: `graft/shared_mouth_opd_data.py:1752-1789`. - Historical 296 warmup uses CE+0.5 KL on captured agent-leg trajectories; historical 297 uses student rollouts, teacher scoring, global-token normalization and separate replay. Those recipes are lineage evidence. Sources: `modal_mtp_legs_warmup.py:1-13,35-40`; `modal_mtp_opd.py:1024-1039`. - Historical top-K KL renormalizes teacher top-K to one, unlike the newer residual-mass format; historical OPD also constructs its “final” stream from RMSNorm(aux41). Sources: `modal_opd_distill.py:118-129`; `modal_mtp_opd.py:367-370,401-404`. - Some OPD header prose predates exit seeding: executable default trace/exit logic uses the last context token as seed, while tracking uses the first speech token. Source: `modal_mtp_opd.py:1-13,529-538`. - Parent checkpoints bind feature view, readout, window, config/base-buffer hashes and optimizer schema. Weights-only A800, full SFT1 and one-depth native formats are distinct. Source: `mouth_engine/checkpoint.py:1-12,50-67`. - W13 raw official tensors are interleaved `[gate0,up0,…]`: split even/odd rows. Never apply that repair to learned separate gate/up tensors. Source: `mouth_engine/mtp_keys.py:75-86`. - Current recipe is undecided: common SFT warmup, matched SFT-only and SFT+OPD are proposals awaiting approval. Reference data and new feature capture are not completed student training. Sources: `AGENTS.local.md:59-75,94-100`; `PROJECT_GOAL.md:157-169`. - Preserve real source partitions and 60 sealed test conversations; the shared loader exposes only explicit train/dev and has no sealed-test override. Sources: `AGENTS.local.md:77-80`; `graft/shared_mouth_dataset.py:1043-1048,1082-1084`. ## 6. Sources coverage map for the coordinator | Family | What it should explain | Concrete entry points | | --- | --- | --- | | Root model modules | Conditioning math, depth chain, warmup, native0 and recurrence | `mouth_conditioning.py:26`; `mtp_chain.py:13`; `mouth_warmup.py:46`; `mouth316_native_head.py:102`; `mouth316_recurrent_training.py:154` | | `mouth_engine/` | Exact cached compute, slots, live companion, functional ownership and optional listener integration | `mouth_engine/row_depth.py:17`; `mouth_engine/mouth_step.py:39`; `mouth_engine/slot_cache.py:21`; `mouth_engine/companion.py:401`; `mouth_engine/episode.py:1`; `mouth_engine/joint_training.py:1` | | `graft/` | Trace alignment, current mouth rollouts, SFT/OPD contracts, reviewed data and feature views | `graft/trace_batch.py:23`; `graft/cached_mouth.py:235`; `graft/clean_mouth_sft.py:1111`; `graft/shared_mouth_data.py:485`; `graft/shared_mouth_dataset.py:526`; `graft/shared_mouth_opd_data.py:1197`; `graft/shared_mouth_feature_views.py:53` | | `clockless_live/` | Session/API/event, transport and delivery infrastructure; multiple historical runtime lanes | `clockless_live/session.py:40`; `clockless_live/contracts.py:175`; `clockless_live/api_gateway.py:1`; `clockless_live/sglang_transport.py:1`; `clockless_live/self_hearing.py:1`; `clockless_live/speech_delivery_adapter.py:1` | | `experiments/` | Local run drivers, durable launch/recovery, historical experiment entry points | `experiments/run_mtp_legs_warmup.py:1`; `experiments/run_mtp_opd.py:1`; `experiments/mouth_warmup_run.py:1` | | `autoresearch/` | Current evidence vs goals, lineage notes, frozen protocols and per-run artifacts | `autoresearch/current_state.md:1`; `autoresearch/notes/four_depth_lineage_reconstruction_2026-09-06.md:37`; pinned engine source `autoresearch/notes/sglang_pinned/VERSION.txt:1-2` | | Patches | Native source capture/readout and companion hooks; older multistream kernel patches separately | `patch_sglang_hidden_all_layers.py:35`; `patch_sglang_native_readout.py:51`; `patch_sglang_feature_view_readout.py:27`; `patch_sglang_companion_mouth.py:1`; `patches/sglang_multistream_stage_c_model_runtime_b725.patch:1` | | Modal family | Root-level `modal_*.py` files, not one `modal/` implementation directory; cloud execution recipes are not model blocks | `modal_mouth_warmup.py:247`; `modal_mtp_legs_warmup.py:1`; `modal_mtp_opd.py:111`; `modal_shared_mouth_dataset.py:1,408` | | Model identity/dependency sources | Checkpoint-specific geometry and native vs local layer implementations | `export/inkling_small_meta/config.json:7`; `mouth_engine/constants.py:8-10`; `mouth_engine/inkling_layers.py:1-10`; `autoresearch/notes/sglang_pinned/python__sglang__srt__models__inkling.py.txt:148` | ## 7. Uncertainties and visual pitfalls - Do not use generic HF config defaults (6144/66 layers/divisor24) for this Small checkpoint (4096/42/divisor16). Source: `.venv/lib/python3.13/site-packages/transformers/models/inkling/configuration_inkling.py:67-99` versus `export/inkling_small_meta/config.json:10-11,65`. - Keep distinct labels for **raw aux41**, **legacy normalized aux41**, **actual native final**, and **mouth hidden output**. “Final state” alone hides materially different computation. - Draw one frozen trunk, detachable trainable MTP/conditioning blocks, frozen shared vocab readout, separate per-depth histories, and an explicit optional raw-hidden feedback edge only where implemented. - All-layer data availability does not establish a trained router, per-depth mixture, selective delivery mechanism, or per-layer arrival timestamps. Existing functional four-depth code still requires exactly four named streams. Source: `mouth_engine/functional.py:75-78,119-131`. - Reading source/config proves architecture contracts, not current deployed configuration or numerical parity. Pinned engine snapshot, mutable local HF sources, capture patch variants and preserved learned weights must remain identifiable. - No checkpoint tensors were opened; trained weight completeness, actual optimizer contents and current deployment identity were not independently verified in this pass. - Native0 class docstring says no training path; its inherited loss methods still raise, but the separate functional trainer now exists. Do not label native0 “untrainable.” Sources: `mouth316_native_head.py:142-147`; `mouth316_recurrent_training.py:154-241`. - Existing transport, TTS and listener scaffolding must not appear as measured mouth capability. Generated, buffered, delivered and heard speech are different states. Source: `PROJECT_GOAL.md:69-89`. - Existing exit-only companion behavior must be labeled separately from the prospective useful-speech-during-work objective; neither static CE nor cache plumbing establishes that objective. Sources: `PROJECT_GOAL.md:33-46`; `mouth_engine/companion.py:432-457`. - No performance, quality, cost, throughput or latency was newly measured here. Historical numbers embedded in comments are not current benchmarks. --- # Clockless Inkling — listener, request interruption policy, and live transport evidence Reviewed 2026-09-11; scope: Zayaan's listener, request interruption policy, and live transport, with mouth internals outside scope. All repository references below are pinned to **35f1233d54fe675f6b8dfd34a740b2065db076b3** (`origin/codex/listener-handoff-2026-09-06`). Resolve `path:lines` with `git show 35f1233d54fe675f6b8dfd34a740b2065db076b3:path`; line numbers refer to that blob, not the current checkout. Read branch AGENTS.local, historical handoff, amended runbook, latest data handoff and fit status, then source and current Linear descriptions/comments. Evidence classes: **CODE** = implemented at this commit; **RECORDED** = historical result retained in artifacts; **PENDING** = required but unqualified or missing. No checkout switch, model inference, training, tests, provider requests, deployment, or external mutation was performed for this review. This is source/evidence inspection, not a deployment certification; reported training and runtime checks were not rerun. ## Current status and September 11 training - **SPE-93: In Progress.** Registration and controlled fit complete; development selection/calibration, sealed qualification, CPU/consumer qualification and packaging remain open. - **SPE-97, SPE-94, SPE-96: Todo.** Delivery order is listener qualification → compatible trained mouth → trained request interruption policy that uses listener outputs → final integrated acceptance. - The Sep11 study trains **only the VAP readout**: bias-only control (8 trainable scalars) versus weight+bias candidate (384×8+8 = 3,080 scalars). - Encoder, input projection, positional/start embeddings, clock projection, action and wake parameters are frozen; no cause heads are added. - Both arms: original Oto train340; 21,760 fixed boundary/role keys; 173,847 valid binary target cells; 1,020 matched updates / 12 passes; CPU fit recorded at 21.22 s. - Objective: individually masked eight-output BCE; AdamW LR 0.003, batch 256, zero weight decay; cached frozen encoder vectors, zero encoder forwards during the fit. - Retained parent SHA256: `3205498da5c603445e03078477269f6e6348642d8f260898a1f375f2cfc4b752`. - Candidate SHA256: `4f150659d6102b7db1c490059df23ccb675465527419ad14e2c5d0db804d4e16`. - Control SHA256: `e3b394bd60a301545a2da85a0fc051f182eab93079ba8fea06bc4a2582baa599`. - The committed verification records matched orders, finite gradients, frozen tensors, checkpoint reloads and W&B steps; it explicitly says `selection_or_qualification: false`. - Latest SPE-93 comment (23:04 UTC) says full native38 evaluation was running locally. No completed selection result is established by that comment or this pinned commit. - Native38 means 38 development recordings / 76 role views, with the frozen 25-risk ranking; final116 is reserved, excluded from fit/selection, and still needs final references/custom scoring. - Two failed starts and a client-side post-fit deserialization failure are preserved; recorded recovery did not retrain. No serving artifact was replaced. - Exact refs: `autoresearch/campaigns/listener-controlled-fit-v1.json:56–167`; `experiments/train_listener_controlled.py:124–180`; `autoresearch/runs/listener-controlled-fit-20260911-local1/README.md:3–29`; same directory `fit-verification.json:1–35`. - Live tracker: [SPE-93](https://linear.app/speedtrain/issue/SPE-93), [SPE-97](https://linear.app/speedtrain/issue/SPE-97), [SPE-94](https://linear.app/speedtrain/issue/SPE-94), [SPE-96](https://linear.app/speedtrain/issue/SPE-96). Listener W&B: `speedtrain/clockless-timing`, run `listener-vap-readout-20260911-v1`. ## Diagram skeleton: distinct paths and owners 1. **Timing:** microphone + actual rendered self PCM → browser AudioWorklet → CLP1 stereo → CPU resampler → dMel + tau → small causal listener → decoder/eligibility → timing evidence. 2. **Semantic input:** independent microphone content stream + text/images/tool results → observation queues → native request admission → frozen Inkling perception/reasoning. 3. **Fast stop:** eligible listener YIELD → runtime stop transaction → provider freeze + browser rendered-prefix settlement + native settlement. 4. **Refresh:** pending observation metadata + optional eligible listener snapshot → `refresh_policy` → CUT/CONTINUE → native continuation/refresh transaction. 5. **Speech feedback:** in-engine mouth text → TTS → CPO1 → browser render → paired self PCM back to listener; playback receipts + word alignment → heard text back to native history. 6. **Dashed/pending edge:** qualified live listener features → native trained mouth speech/hold conditioning. The current local mailbox does not implement this end-to-end edge. ## Audio geometry and clocks - **CODE:** CLP1 header is 29 bytes (`>4sIIQIIB`): magic, graph generation, sequence, AudioContext start frame, sample rate, frame count, availability flags. - Each packet contains 960 stereo frames / 20 ms at 48 kHz, interleaved int16 little-endian **[self, microphone/other]**; complete packet size is 3,869 bytes. - Self samples are exactly those the worklet renders from the output queue, including real zeros when the connected queue is empty; they are not server-send timestamps or guessed TTS activity. - Worklet output receives 24 kHz TTS source PCM and repeats each source sample twice into its 48 kHz graph. Graph incarnation and speech-output generation are different identities. - `PairedAudio16k` applies one continuous libav stereo resampler and emits 3,200-sample/channel, 200 ms ticks. Packet loss, missing channel flags, changed rate/generation or discontinuity invalidate transport. - The declared first 200 ms signal tick is available after the source packet ending at 220 ms; this is source-publication geometry, not measured inference latency. - Per channel dMel has 80 bins per 50 ms / 800-sample row; channels concatenate to width160; renderer dMel values are converted to float32 and divided by16. - Model input is `[B,80,160]`; live observed context grows 4,8,…,76 rows (maximum3.8 s), with remaining rows padded. PCM rendering retains at most4 s. - `PolicyDecision.model_position` is the action output **slot** `context_frames/4` (maximum19); corresponding transformer sequence position is four times that slot. - VAP uses frame index `context_frames−1`, maximum75. A decision-call counter does not advance audio or reveal more history. - **Tau is computed input, not a prediction head:** `[B,80,2]`, channel order **[other,self]**, `log1p(seconds since previous activity)`, strictly before each row, saturated at120 s. - Activity for tau is the rule `50 ms peak >=0.02`; unknown prehistory uses the historical saturated numerical sentinel with a separate false identifiability mask. - Fully identifying tau at the earliest row of a76-row context can require123.8 s prehistory; the partial-context startup contract does not impose that as a universal startup delay. - Browser graph→server monotonic mapping uses a one-use round-trip bracket,128-frame quantum allowance and50 ppm drift envelope; expires after30 s. Device drift qualification remains required. - Exact refs: `clockless_live/webrtc_media.py:33–42,162–245,478–635`; `vap_runtime.py:235–289,298–399,422–441`; `listener_clock.py:1–54`; `autoresearch/notes/listener_runtime_contract_v1.json:22–46`. ## Listener model blocks and dimensions The retained artifact's metadata was inspected directly from its Git blob without constructing/running a model: **384 width,6 layers,6 attention heads,use_clock=true,13 wake buckets,no cause-head tensors**. | Block | Actual geometry / mechanism | State | |---|---|---| | Audio projection | Linear160→384 → GELU → Linear384→384 | Learned; frozen in Sep11 fit | | Start and position | Learned start `[1,1,384]`; learned absolute position `[1,81,384]` | Learned; frozen in fit | | Wristwatch projection | Linear2→384 → GELU → Linear384→384; added to frame embeddings, zero start row | Learned projection of rule-computed tau | | Causal encoder | 6 pre-norm TransformerEncoder layers;6 heads,64 dimensions/head;FFN384→1536→384;dropout0.1 | Learned; frozen in fit | | Attention mask | Upper-triangular81×81 future mask | Fixed causal rule | | Action readout | Linear384→4 at sequence positions0,4,…,76 | Learned; frozen in fit | | VAP readout | Linear384→8 at all80 frame states | Only block adapted Sep11 | | Wake readout | Linear384→13 at the20 action positions | Learned historical head; diagnostic only | | Optional cause heads | Named Linear384→K modules, appended after wake outputs when artifact declares them | Supported architecture; absent in retained parent | Constructor defaults192/4/4 are **not** the retained artifact geometry. The loader derives dimensions/flags from artifact metadata and verifies action order. Exact refs: `vap_controller.py:104–249`; `vap_runtime.py:78–128`; `autoresearch/artifacts/vap_controller_clock_v1.pt` at the parent hash above. ## Head semantics: learned evidence versus rule control | Output | Values / timing | What a diagram may claim | |---|---|---| | Action | softmax `[SILENT,SPEAK,BACKCHANNEL,YIELD]`, tensor `[B,20,4]` | Learned timing/floor evidence; no words or semantic reasoning | | VAP | eight independent sigmoid probabilities, `[B,80,8]` | Any future activity per channel; not a joint256-class code | | VAP bins | per self/other:0–400,400–800,800–1200,1200–2000 ms | Acoustic forecasts; not desired assistant speech/hold labels | | EOT | current evaluator maps SPEAK/BACKCHANNEL to end-of-turn boundary events | No dedicated EOT scalar head in clock-v1 or the Sep11 descendants | | INT | current event projection maps YIELD to interruption transactions | A separate historical `interruption` scalar head exists in experimental variants, not this retained parent | | Optional scalar auxiliaries | `sigmoid(Linear384→1)` at action slot | Generic runtime support; packaged INT threshold can override to YIELD, then EOT threshold to SPEAK | | Historical cause hazards | `floor_return`,`silence_tail`:12 hazard outputs or13 CE buckets, variant-dependent | Diagnostic research heads; size12 hazards are not surfaced by `_aux_probs` | | Learned wake | argmax over13 buckets, lower edges0.2…60 s | Logged only; no live learned-wake actuation | | `next_wake_ms` | rule: first either-channel VAP bin >=0.5; clamp minimum200 ms, otherwise2000 ms | Derived rule, not learned wake; diagnostic in integrated runtime, which processes every200 ms tick | Runtime decoding is action argmax unless exact named auxiliary heads/thresholds are packaged; adding decoder support does not establish trained or qualified heads. Eligible YIELD is level-sensitive while output exists, before changed-action filtering. Sustained YIELD can stop a successor generation; count generation+stop_id transactions. SPEAK/BACKCHANNEL changes may close an energy-opened utterance via `_fire_boundary`; default behavior is boundary logging/callback, not learned word emission or native KV admission. Exact refs: `vap_runtime.py:340–349,402–464`; `listener_contract.py:150–183`; `clockless_live/integrated_live.py:854–918,1093–1147,2957–2961`; `experiments/evaluate_listener_events.py:477–560`; `experiments/train_listener_v3.py:60,100`; `experiments/train_listener_v3_293b.py:1–19`. ## Checks before using listener outputs and listener-to-mouth integration - **CODE:** shared IDs are `listener-consumer-v1`, `listener-decoder-v1`, `listener-eligibility-v1`; startup mode `partial_context_v1`. - Snapshot binds artifact hash, sequence, source sample interval/publication/completion, clock uncertainty/expiry, observed-row and tau masks, session/request/phase/output generation, final heard prefix and per-output validity. - Four distinct checks matter: transport continuity, observed feature rows, tau identifiability, and artifact/output qualification. Real silence and missing input are not interchangeable. - Source-age upper bound includes clock error: completion≤250 ms, actuation≤350 ms, clock error≤50 ms; recheck original evidence after waiting for locks and before side effects. - Only4…76 contiguous observed rows are valid; padding is expected. Unknown tau in observed rows is allowed only as explicitly masked partial-context input. - A missing qualification ID or unqualified required output rejects actuation. `consumer_snapshot` removes unqualified probabilities/tau and always removes wake/deadline fields. - Factory still mounts clock-v1; optional `LISTENER_POLICY_PACKAGE` binds qualification/thresholds to its exact bytes. With no package the default artifact remains diagnostic and cannot acquire policy permission merely by producing numbers. - **CODE PROTOTYPE:** `NativeListenerMailbox` specifies `listener-native-update-v1` / `listener_update_v1`: one pending coalesced update, one immutable in-flight transfer, exact owner/consumer hash and delivered/consumed/rejected receipts. - Only a receipt emitted after native model feature use can prove consumption. Expired unacknowledged transfers remain quarantined; delivery is not consumption. - **PENDING:** repository Python search finds the mailbox/update method only in its contract definition outside tests; IntegratedSession and the companion patch do not instantiate this bridge. - Current `latest_listener_snapshot` reaches the CPU request-interruption logic after masking/eligibility. Recorded “listener→mouth” fields alone do not show GPU/model conditioning. - SPE-97 owns verification of Parsa-delivered trained model-side speech/hold interface; SPE-94 owns minimum live integration needed to record request-interruption decisions; SPE-96 owns final bridge hardening/acceptance. - Exact refs: `listener_contract.py:19–90,204–440,444–602`; `clockless_live/integrated_live.py:920–1095`; `modal_thinking_live.py:175–207,414–431`; current SPE-97/SPE-94/SPE-96. ## Policy for interrupting the current request - **CODE BASELINE:** `IntegratedDeps.refresh_policy` is configured as `cut_for_pending_input`: CUT whenever pending observations exist in planning/companion_exit/playback; otherwise CONTINUE. It ignores listener probabilities. - Available `RefreshContext`: native request ID, phase, completion count, observation IDs/modality/hash/provenance/ready time, decision time, optional eligible listener snapshot and selected-token logprob. - **Missing semantic features:** observation hashes are identities, not content; current context does not carry observation semantic embeddings, trunk hidden states or full-distribution entropy. - Custom policy execution has a separate one-worker executor, no queued backlog and100 ms timeout; timeout rejects a result without claiming the Python worker was killed. - Ownership/source/phase is checked after inference and immediately before refresh/stop. Fast listener YIELD does not wait for the request interruption policy. - CUT while planning calls one-shot `RoundHandle.refresh` and awaits settlement. CUT with exit/unheard output suppresses stale speech and uses the shared stop transaction before reopening generation. - **PENDING:** trained request interruption policy that uses listener outputs, eligible semantic preference dataset, feature route, qualified policy loader/package and listener-contribution evidence. A small CPU regularized classifier is specified, not delivered. - **RECORDED FAILURE:** old D3 had6 scheduled pairs/12 sessions,5 prefix-comparable pairs,1 mismatch,0 naturally completed mouth arms and0 eligible semantic labels; faster CUT did not license fitting. - Exact refs: `live_observations.py:10–49`; `listener_contract.py:638–674`; `clockless_live/integrated_live.py:1370–1718`; `experiments/capture_d3_referee.py:1–6,58–88`; `autoresearch/runs/d3-explicit-refresh-baselines-20260906-run1/recovered_mechanics.json` (`counts`,`interpretation`). ## Native admission and playback-feedback ownership - Semantic microphone intake is separate from paired timing audio: `MicChunker` keeps completed1 s mono16 kHz chunks only when peak≥0.02. This is an energy rule, not listener EOT; paired timing still includes real silence. - Text/image/tool events retain original payload identities and availability; closure remains first, then independent blocks are sorted by availability, then the model marker. Do not diagram a permanent tool→text→audio priority. - Runtime assembles native `input_ids`, `audio_data`, optional `image_data`, `clock_times_ms` and persistent `session_params`; the native engine owns materialized multimodal tokens, inherited prefix and KV state. - `ObservationLedger` separates received/submitted/admitted. Duplicate event IDs with changed payload/provenance fail; second admission fails. `_on_handle` admission time is explicitly the first-observed-decode upper bound. - Native mouth protocol is root/continue/new_turn/exit/off; new observations preempt pending stale exit. `<|content_text|>` is the planning exit boundary; exit uses offset−1 and exactly one discarded sampled token. - Native prefix receipts distinguish origin tokens, sampled output and completed-forward prefix; the latest sampled token is not yet forwarded. These distinctions must survive cancellation and successors. - `SessionClock` is an experimental positional warp: generated progress adds100 ms/token plus mapped inter-event gaps. It is not physical elapsed time or evidence of learned time understanding. - In-engine mouth owns speech text; web `mouth_loader.py` is only Cartesia TTS wiring. `meta_info["mouth_text"]` supplies voice text, with transport/control-marker handling rather than a second language model. - **Browser owns rendering:** CPO1 carries24 kHz PCM/source positions; CPS1 requests an ordered stop; CPA1 is final rendered-prefix acknowledgment; CPP1 is progress; CPF1 retires a turn. - Current stop order: suppress/freeze provider output → ordered CPS1/final CPA1 barrier → cancel/refresh native handle and await its finish → queue heard prefix → allow successor. Immediate local mute is supplementary, not the receipt authority. - Final CPA must match generation+stop_id, cover delivered turns and never exceed/regress source positions; duplicate final receipts are idempotent and cannot settle a different successor stop. - Heard-prefix text includes only fully rendered aligned words (or a fully played, normally finished utterance); no characters-per-second guess. It re-enters native self history; rendered PCM independently feeds the listener. - Post-stop listener checks reject obsolete output ownership and pre-stop source-publication envelopes while retaining the resampler/model history. A fresh output generation does not make old evidence fresh. - Native cleanup is bounded and owner-specific; HTTP200 close is DISPATCH_ACK, not remote purge. Browser digital render receipts do not establish physical audibility/echo behavior. - Exact refs: `clockless_live/thinking_live.py:505–568`; `clockless_live/integrated_live.py:1275–1368,1887–2116,2226–2236,2283–2510,2832–2873`; `live_observations.py:52–138`; `live_request_modes.py:45–169`; `live_prefix_receipt.py:110–165`; `live_playout_output.py:10–28,260–264,355–430`; `clockless_live/native_cleanup.py:1–87`. ## Evidence limits to keep visible in the guide - Retained native38 baseline failed7/8 VAP Brier and7/8 ECE point targets; the Sep11 readout-only fit cannot fix unchanged action/event-head failures by construction. - Historical512-update causal fit remained unqualified: control YIELD recall5/20, onset arm2/20; onset arm0/13 AMI yields. Preserve these as failures of that recipe, not universal impossibility. - V8 source `31102c7bfcde224aab25c536173aef4b1773fe9e` recorded mixed-input/native mechanics, browser transport and one reconnect pair; it is a different source from this review. - V8 browser replay reproduced100 actions but had **zero YIELD**; both stops were semantic_input. Four API mouths remained partial, new_turn was client-canceled, mixed output repetitive/unfinished. - V8 was recorded stopped; its readiness expired. No current serving health was probed here. Current default source and historical deployed behavior must not be conflated. - Final12/12 tasks,≥18/20 listener interruptions with rendered-stop p95≤500 ms,≤1 false stop/10 negative playback minutes,10/10 starts/reconnects,30-minute soak and physical acceptance remain **targets**, not achievements. - Guide rendering recommendation: solid edges for implemented plumbing, separately labeled historical receipts, dashed edges for trained live mouth conditioning/request interruption policy/wake; no “deployed success” badge. - Exact refs: `autoresearch/runs/listener-controlled-fit-20260911-local1/README.md:3–15`; `autoresearch/notes/listener_duration512_review_2026-09-06.md:37–52`; `autoresearch/notes/listener_v8_validation_results_2026-09-06.md:3–41,54–59`; current SPE-96. - Status precedence: `autoresearch/notes/listener_agent_runbook_2026-09-07.md:5–23` and `listener_spe92_final_handoff_2026-09-10.md:1–14` amend older handoff/startup prose; latest SPE-93 comment and committed fit evidence establish only fit completion. --- # Oto69 viewer data evidence Verified metadata fetch: 2026-09-12T02:20:14.745Z (UTC). Viewer URL without credentials: https://aptly-underfoot-showman.ngrok-free.dev/?conversation=69&at=0.0&source=all&split=all&annotator=a Recording metadata URL: https://aptly-underfoot-showman.ngrok-free.dev/api/recordings/69 The viewer URL is safe to quote but may require an already authenticated browser session. No access-link route, fragment, cookie, credential value, Messages identifier or transcript is included in these evidence files. ## Verified recording - Recording: Oto69; source ID 69; dataset Oto. - Recording-level type: `Task-Oriented/Transactional`; split: `train`. This describes the conversation genre, separate from each interval’s tag. - Duration: 838.042 seconds (13:58.042). - 341 raw annotation cues, all annotator `a`; speaker1: 172, speaker2: 169. - 12 distinct tag strings; exact counts below were recomputed from all341 cues and checked against the endpoint’s `tags` object. - `quarantined=false`; all cues have `held=false` and `past_eof=false`. These are record/interval validity flags, not model qualification or independent annotation-accuracy results. - 311 cues have `vocal=true`;30 have `vocal=false`. Counts describe annotation entries, not seconds or model decisions. - Scope: this one recording metadata response and its viewer page. No audio endpoint, bulk catalog, other recording or model inference was requested. ## Exact source tags and counts | Source tag, spelling preserved | Cue count | Repository source-activity grouping | vocal | |---|---:|---|---| | Normal Turn | 136 | floor | true | | Non-Speech Noise | 30 | non-vocal | false | | Filler | 59 | non-floor vocalization | true | | Acknowledgement Backchannel | 52 | non-floor vocalization | true | | Reaction Backchannel | 7 | non-floor vocalization | true | | Speech, Non-Linguistic | 9 | non-floor vocalization | true | | Laughter | 24 | non-floor vocalization | true | | Overlap | 9 | floor | true | | Bounded Response | 3 | non-floor vocalization | true | | Continuer Backchannel | 4 | non-floor vocalization | true | | Non-floor Taking Cooperative Interruption | 6 | non-floor vocalization | true | | Floor-taking Cooperative Interruption | 2 | floor | true | | **Total** | **341** | | | The grouping column explains the inspected repository mapping; it is not an additional field returned on each API cue. These12 labels are the labels present in Oto69, not the complete dataset vocabulary. ## Interval schema and visualization semantics | Safe JSON field | Type | Meaning | |---|---|---| | recording.id | string | Recording/source identifier69; no access credential | | recording.conversation_type | string | Original API `type`, here Task-Oriented/Transactional | | recording.split | string | Original source assignment, here train | | recording.duration_s | number | Original API duration in seconds | | tag_counts | object | Number of annotation entries per exact source tag | | intervals[].interval_index | integer | Local1-based position in the returned cue sequence; added for drawing references | | intervals[].speaker | integer1 or2 | Original speaker/channel; neither number implies assistant or user | | intervals[].annotator | string | Source annotation track, here a for every cue | | intervals[].start_s / end_s | number | Original API start/end values, in seconds from recording start | | intervals[].tag | string | One raw category per source annotation cue | | intervals[].vocal | boolean | Category-based source vocal-activity mapping, described below | | intervals[].held | boolean | Source cue appears in the prepared record’s held-source list | | intervals[].past_eof | boolean | Source cue end is beyond the recording duration | - Intervals use half-open `[start_s,end_s)` semantics; duration is `end_s-start_s`. Preserve the original floating-point values; do not round stored boundaries to ticks. - Draw separate speaker1/speaker2 lanes with a shared seconds axis. Concurrent intervals remain concurrent; flattening into a single categorical lane would hide overlap. - Use tag for color and speaker for lane. Use vocal/held/past_eof as separate attributes, not replacement labels. - A backchannel can overlap a Normal Turn without either cue being tagged Overlap. The table below contains real examples. - Unannotated gaps remain unlabeled; they are not automatically Awkward Silence, missing data, a SILENT action or a hold target. - SRT timestamps have millisecond representation. This is annotation precision, not a claim of millisecond acoustic accuracy. - JSON is an allowlisted projection of the response: source times/tags are unchanged, start/end are renamed start_s/end_s, and transcript-bearing cue text and source cue IDs are omitted. ## Eight example intervals entirely within the first35 seconds | Speaker | Start(s) | End(s) | Source tag | |---|---:|---:|---| | 1 | 0.100 | 1.000 | Normal Turn | | 1 | 1.776 | 2.252 | Non-Speech Noise | | 1 | 8.214 | 8.551 | Filler | | 1 | 19.217 | 29.600 | Normal Turn | | 2 | 28.200 | 28.700 | Acknowledgement Backchannel | | 1 | 32.400 | 33.900 | Normal Turn | | 2 | 33.224 | 34.124 | Reaction Backchannel | | 1 | 33.904 | 34.492 | Speech, Non-Linguistic | These are illustrative selected intervals, not every cue in0–35s. The JSON includes all341 safe intervals. In particular, speaker2’s28.200–28.700 acknowledgment overlaps speaker1’s19.217–29.600 normal turn;33.224–34.124 reaction overlaps both the33.900 end of one speaker1 turn and the33.904 start of a non-linguistic vocalization. ## Source vocal activity versus desired assistant action - Viewer code sets `vocal = tag not in NON_VOCAL_LABELS`. The inspected non-vocal set is exactly Awkward Silence, Channel Bleed, and Non-Speech Noise; only the last occurs in Oto69. - Consequently fillers, laughter, backchannels and non-linguistic speech are vocal=true, even when they do not claim the conversational floor. “Vocal” is not synonymous with intelligible words, floor-taking, or a request to interrupt. - The flag is derived from the source category, not a learned acoustic classifier or a new listening judgment performed in this review. - The raw tag describes recorded human behavior. It does not directly prescribe the assistant’s SILENT/SPEAK/BACKCHANNEL/YIELD action or train a semantic refresh decision. - For example, the two Floor-taking Cooperative Interruption cues are two source labels, not proof of two desired assistant stops. Correct assistant action also depends on role assignment, timing, playback exposure and the reviewed target policy. - Source vocal intervals can support acoustic future-activity/VAP targets when causal alignment and validity are respected. The September11 registered fit uses eight masked source-activity targets, not action or wake labels; future self activity is recorded human behavior, not automatically desired generated assistant speech. - Runtime time-since-activity feature tau has a separate energy-based definition (peak>=0.02), so do not label the viewer vocal flag as the runtime tau detector. ## Retrieval and source provenance, with credentials omitted | Request / read | Method or operation | Observed result | |---|---|---| | Viewer access-link exchange; credential-bearing route omitted | GET |303; 267 ms; no credential values copied into evidence files | | oto69_recording_metadata | GET /api/recordings/69 |200; 247.819 ms through full response-body read; 84663 response bytes | | oto69_viewer_page | GET /?conversation=69&at=0.0&source=all&split=all&annotator=a |200; 210.532 ms; 2026-09-12T02:21:28.203Z; HTML retrieved without executing page scripts | | Source mapping inspection | git show at pinned commit, files below | Read only; no source or external modification | - Metadata request started: 2026-09-12T02:20:14.496Z. - Metadata response fully read: 2026-09-12T02:20:14.745Z. - SHA256 of the received metadata body: `73baf4b13a1bc8ea6d0e2cb0311cf33e15ef3e40ea115542cd7aa532d9ba4045`. This hashes the original response; that response was not written to disk. - Endpoint-reported source dataset: `otoearth/otoSpeech-full-duplex-turn-104h`. - Endpoint-reported source revision: `46f520297f434edf804389f82f9075a59d2f8268`. - Source annotation files named by the inspected viewer implementation: `69/speaker_1_annotation_a.srt`, `69/speaker_2_annotation_a.srt`. - Repository mapping reference: `TalkingComputers/edgewise@35f1233d54fe675f6b8dfd34a740b2065db076b3`, `tools/oto_viewer/serve.py:Corpus.__init__` and `experiments/oto_timelines.py:NON_VOCAL_LABELS` / interval semantics. - Training-contract reference at the same commit: `autoresearch/campaigns/listener-controlled-fit-v1.json` (objective and frozen groups); runtime tau reference: `vap_runtime.py:StreamingVapPolicy.feed/_tau`. - The endpoint’s source revision identifies the source dataset; the inspected repository commit identifies mapping code. Neither is a verified hash of the currently deployed viewer process. - Fetch durations are network/retrieval observations for these requests, not model latency, audio timing, rendered playback or listener qualification. Sanitized machine-readable data: `/tmp/clockless-viewer-oto69-safe.json`. ## Final 3D source-review corrections The final view keeps the fixed checkpoint-bound four-stream parent route separate from C1 wider-layer evidence. SPE-110/native-final and A800/legacy-normalized-aux41 are explicit selectable four-depth contracts. A5 shows a distribution at each native offset and advancing candidate-token inputs; autonomous commitment is still an open research rule. SPEAK/HOLD control receives committed mouth hidden and listener evidence separately from the vocabulary path. Tool results and request-interruption decisions return to native input admission rather than injecting into hidden layers. Listener drill-down uses LayerNorm, ordinary ReLU FFN, and recomputed sequence K/V projections; it does not inherit Inkling SConv or persistent KV semantics. ## MTP mathematical architecture update Source inspection: October 5, 2026. This supports the mathematical Atlas descriptions in `mtp-math-core.js`. It preserves the six comparison definitions from the September 11 Atlas and distinguishes them from subsequently added source implementations. No model, checkpoint tensors, training job or research-team runtime was invoked. These are source facts and proposed comparisons, not newly measured model results. The complete model remains: native inputs → all 42 Inkling trunk blocks → native final normalization/readout, with selected internal source representations supplied to an MTP mouth. Listener, synthesis, playback and task-control routes retain their separate contracts. This audit covers the trunk and six MTP variants only. ### Notation Batch is omitted. A token state is a row vector of width `H=4096`; matrices are written `[input,output]`, so `xW` is a row-vector projection. PyTorch stores a Linear weight as its transpose `[output,input]`. The legal semantic output vocabulary has `V=200058` rows, distinct from the checkpoint's padded allocation of `201024`. ```text N_γ(x) = γ ⊙ x / sqrt(mean_j(x_j²) + 1e−6) e(b_t) = N_base_embed(E[b_t]) [4096] Σ_k,t = persistent KV, logical-position and SConv state of depth k (y, Σ′_k) = D_k(h, e; Σ_k) ``` `b_t` is the block-input token. `q_t` is the separate source-gate token, where that gate exists. In the preserved cached reply, after sampling the next token, `q_(t+1)=b_t` and `b_(t+1)=sampled_token`. Four same-token depths share `e(b_t)`, not necessarily the gate token `q_t`. ### Complete trunk block and capture boundary The checkpoint has 42 blocks, 32 query heads, 8 KV heads, width 128 per head, and four depthwise short convolutions per block. Its native global trunk indices are `5,11,17,23,29,35,41`; the other 35 layers use a 512-position causal window. Blocks 0–1 use dense width-16384 MLPs; blocks 2–41 use the MoE below. ```text a = N_attn(x) o = Attention(a; KV, SC_K, SC_V) r = x + SC_attn(o) b = FFN(N_mlp(r)) y = r + SC_mlp(b) all state rows [4096] SC(z)[n,c] = z[n,c] + sum_(j=0…3) w[c,j] z[n−3+j,c] ``` There is a short-convolution input residual **and** a separate block residual. The convolution has no bias or activation, uses current plus three prior inputs, and mixes positions independently per channel. K/V convolutions use 1024 channels; branch-output convolutions use 4096. Pinned SGLang defers an MLP-output convolution into the next entry. A captured auxiliary tensor is therefore not the complete block output: ```text aux41 = r41 + b41 x42 = r41 + SC_mlp,41(b41) native_final = N_final(x42) trunk logits = (native_final / 16) W_U [200058] ``` `N_final(aux41)` does not reconstruct `native_final`. The native-final capture is before the `/16` readout calibration. The historical four-stream view uses `[aux13,aux27,aux41,N_final(aux41)]`; the corrected native view uses actual native final as its fourth stream. The checkpoint/feature-view contract determines which can be consumed. A 43-stream capture means 42 auxiliaries plus a separate native final, not 43 trunk blocks. ### Exact attention operations Inkling uses **hidden-conditioned learned relative logit bias, not RoPE**. With the already normalized input `a[4096]`: ```text W_Q,W_O [4096,4096]; W_K,W_V [4096,1024]; W_R [4096,512] Q = N_q(reshape(a W_Q, [32,128])) K_new = N_k(reshape(SC_K(a W_K), [8,128])) V_new = reshape(SC_V(a W_V), [8,128]) R = reshape(a W_R, [32,16]) ``` Q has no short convolution. V has no per-head normalization. The KV cache receives convolved/normalized K and convolved V. Four query heads share each KV head; the index for query head `h` is `floor(h/4)`. ```text P_rel [16,L_rel]; B = R P_rel [32,L_rel] β[h,j] = B[h,n−j] if 0≤n−j