PersonaPlex native LM audit

Scope: the released temporal model, depth model, code-stream alignment and generation loop. This is the native model, not the proposed Inkling connector. Source inspected on 2026-10-04. No model inference, weight loading, checkpoint mutation or deployment occurred.

1. Evidence and exact limits

• NVIDIA source: commit 3428dfd95309a7f3c84fd93259ded0f810d1ff91.
• Published model: nvidia/personaplex-7b-v1, revision fdaf4090a61cb315c138a1faee287ffd6c716309.
• Hugging Face model metadata reports 8,371,408,896 BF16 parameters, all in model.safetensors, file size 16,742,874,000 bytes. Publisher LFS SHA256: db1290db583cdaa6cb4de444ed279e0b586ca2a372b41434b07a7461c8c0e2f4.
• The complete source-derived parameter inventory below totals exactly 8,371,408,896, independently matching that published aggregate. Its temporal/input/text side is 6,973,366,272 parameters; its depth side is 1,398,042,624. The release's “7B” name must not be displayed as an exact full-file parameter count.
• Existing protected Hugging Face authentication was used only for metadata requests. File metadata/header access returned GatedRepoError/HTTP 403. No safetensors header or tensor payload was read. The shapes below are exact constructor-derived shapes, corroborated by the aggregate count, not a inventory verified from the file bytes of individual stored tensors. No credential value or signed download URL was retained here.
• Hugging Face API config metadata reports model_type: personaplex; the hosted config.json is 56 bytes. Its file contents were not fetched. In the pinned server it is downloaded for the download counter and not passed to the model constructor.

Sources: published model revision, Hugging Face revision metadata, server loading call, NVIDIA loader. The metadata/LFS digest above are publisher-provided identities, not hashes recomputed over downloaded weights.

2. Effective constructor: why 16 depth positions is supported

The default _lm_kwargs dictionary declares n_q=16, dep_q=8. The actual get_moshi_lm() entry copies that dictionary and sets dep_q=16 before creating LMModel. The server calls this entry directly and does not override it with the hosted config. The CPU-offload branch receives the same modified dictionary.

The loader contains compatibility code to expand depth-attention matrices and copy missing step-0…7 groups into step-8…15 groups. That code's existence does not prove it runs for the released checkpoint. The published parameter total already matches the 16-position constructor exactly. An 8-position constructor would have 7,687,729,152 parameters, 683,679,744 fewer.

The correct diagram distinction is:

17 temporal input streams = 1 agent-text + 8 agent-audio + 8 user-audio
16 sequential depth positions = 8 agent positions + 8 user positions
8 agent-audio codebooks are decoded to outgoing PCM

The code computes/samples depth logits at all 16 positions. At a provided user position, the supplied code replaces the sampled code as the next depth input and remains in the generator cache. These are not 16 independently generated agent-audio streams. Loader · depth loop · server agent decode

3. Native temporal computation

Let B be batch size, T temporal positions, D=4096, H=32, d_h=128. The server uses B=1. Each temporal position is a position in the delayed token streams, not an arbitrary text-token position.

Input embedding sum

There are 16 separate audio embedding tables, each [2049,4096], and one text embedding table [32001,4096]. Audio IDs 0…2047 are codec symbols; 2048 is the initial audio ID. The text initial ID is 32000; the text padding ID used by this construction is 3. The special no-input ID -1 is mapped to an exact zero embedding by ScaledEmbedding; it is not an audio-silence codebook vector. -2 is the ungenerated-cache sentinel and must not become a model input.

At each temporal slot:

x = E_text(text) + Σ(i=0…7) E_agent,i(agent_i) + Σ(i=0…7) E_user,i(user_i)
x shape: [B,T,4096]

The 17 embeddings are summed, not concatenated, averaged or gated. Embedding normalization is disabled (norm_emb=False); the embedding wrapper's name does not introduce a scale multiplier in this path. Tables are separate across speakers/codebooks, and the temporal text embedding and text output projection are untied. LM constructor/embedding implementation

One of the 32 temporal blocks

Every block performs pre-normalized self-attention followed by a pre-normalized SiLU-gated feed-forward network:

u = RMSNorm32(x)
[Q,K,V] = Linear(u; W_QKV)                 W_QKV: [12288,4096], no bias
Q,K,V → [B,32,T,128]
Q,K = RoPE(Q,K, temporal_offset)
A = SDPA(Q,K,V, causal_context_mask)        attention dropout = 0
a = concat_heads(A) W_Oᵀ                 W_O: [4096,4096], no bias
y = x + a
v = RMSNorm32(y)
[g,z] = Linear(v; W_in)                   W_in: [22528,4096], no bias
g,z → [B,T,11264]
f = Linear(SiLU(g) ⊙ z; W_out)            W_out: [4096,11264], no bias
block_output = y + f

RMSNorm32 means FP32 normalization arithmetic, with epsilon 1e-8 and a learned scale of shape [1,1,4096]; its output is cast back to the input dtype. It does not mean LayerNorm or mean subtraction. The checkpoint metadata reports BF16 stored parameters; the constructor's norm computation still casts to FP32. Each block has two independent scales, one before attention and one before the FFN.

Attention is ordinary multi-head self-attention: 32 query, 32 key and 32 value heads, each width 128. It is not GQA/MQA, and there is no separate Q/K normalization, attention output gate, convolution or private cross-attention in the native temporal block. Residual LayerScale is identity because the configured layer scale is None. FFN dropout is absent; SDPA receives dropout_p=0.0.

The configured FFN argument dim_feedforward=16896 is not the gated hidden width. ActivationGating uses hidden=2×16896/3=11264, projects to 2×hidden, applies SiLU to the first half, multiplies by the second, and projects back. Attention/block source · gating source

Positions, attention mask and final text head

Temporal RoPE uses maximum period 10,000 over all 128 head coordinates in adjacent coordinate pairs. Frequencies are exp(-log(10000)×2j/128) for j=0…63. Rotation is applied to Q and K, not V. There is no learned absolute position table and no additional sinusoidal vector added on the configured rope path. RoPE implementation

The declared causal mask permits keys with valid positions and 0 ≤ query_position − key_position < 3000. See the ring-cache limitation below before calling all 3,000 positions usable.

After block 31, one final RMSNorm computed in FP32 produces h:[B,T,4096]. An untied, bias-free text head with weight [32000,4096] gives [B,T,32000]; forward_embeddings() adds the text-stream dimension, returning [B,1,T,32000]. There is no configured vocabulary softcap or additional width-divisor readout in this source. Text is sampled in FP32 and then conditions the depth model. LM readout

4. Native depth computation: six blocks, sixteen sequential positions

The depth model does not run at a separate wall-clock rate. One temporal prediction feeds a within-frame loop over codebook positions q=0…15. The temporal model's long history persists; the depth transformer's streaming context is opened fresh for each depformer_step() and closed after its 16-position loop.

For position q:

c_q = Linear(h_temporal; W_in,q)          W_in,q: [1024,4096], no bias
q=0:  z_q = c_q + E_depth_text(selected agent-text token)
q>0:  z_q = c_q + E_depth_audio,q−1(previous selected code)
z_q shape: [B,1,1024]
z_q → six causal depth blocks → logits_q:[B,1,1,2048]
sample a code; choose supplied code instead where this position is provided
pass the selected code to position q+1

There are 16 distinct temporal-to-depth projections, one depth-text embedding [32001,1024], 15 distinct preceding-audio embedding tables [2049,1024], and 16 distinct output heads [2048,1024]. The final codebook does not need its own input embedding because no next depth position consumes it in the same loop. No final depth RMSNorm is added after its sixth block in forward_depformer().

One depth block at position q

The operation is the same pre-RMSNorm→self-attention→residual→pre-RMSNorm→SwiGLU→residual structure, but with D=1024, 16 Q/K/V heads of width 64, and no positional embedding or RoPE. Its configured FFN dimension 4224 yields actual gated hidden width 2816.

The attention and FFN weights are different for each codebook position. The two RMSNorm scales are shared across the 16 positions within that block; different blocks have different scales.

multi_linear() selects the attention weight slice by offset_cpu; the block selects its FFN from gating[offset+t]. Thus “six depth blocks” must not be drawn as the same six attention/FFN weight sets reused unchanged for all codebooks. It is also not 96 independent temporal blocks: codebook positions are sequential inside a frame and attend to earlier depth positions.

Although the loader dictionary contains depformer_context=8, LMModel.__init__() explicitly replaces the depth context with None. Streaming capacity then comes from weights_per_step=16. No temporal RoPE is inherited because depformer_pos_emb='none'. Depth construction and per-position input · per-step attention/FFN · depth loop

5. Complete constructor-derived parameter inventory

Shapes use PyTorch's [out_features,in_features] orientation for linear weights. * denotes the stated range, not sharing. These 475 tensor objects are derived from source, not an enumerated checkpoint header.

No bias tensors, embedding normalization tensors, learned RoPE tables or LayerScale tensors are added by this constructor. The sum excludes Mimi, which has its own checkpoint. Matching the published aggregate strongly corroborates the effective 16-position construction, but does not establish individual checkpoint values, ties, loading success or numerical behavior.

6. Three distinct caches

A KV cache stores the key and value vectors used by attention so earlier positions need not be recomputed. It is distinct from the ring buffer of token IDs described below.

Temporal attention cache

Each temporal layer has a RingKVCache allocated as [2,B,32,3000,128], where axis 0 selects K/V. At BF16 and B=1 this is 49,152,000 bytes per layer, or 1,572,864,000 bytes across 32 layers, before offset storage, allocator overhead and other runtime memory. Keys have already received RoPE; values have not. The ring has its own end_offset; attention also tracks tensor and CPU offsets. Reset clears validity/offset state rather than requiring every cache byte to be zeroed.

Observed source-level boundary: RingKVCache.complete() maps positions with delta <= 0. When full, the slot at end_index is labelled end_offset (a future position) and rejected by the causal mask. Reproducing those integer operations gives:

capacity4, end_offset4 → reported positions [4,1,2,3]
query_position3 → visible positions [1,2,3]
capacity3000, end_offset3000 → 2999 visible positions

This is a potential off-by-one in the pinned ring-position mapping. It was reproduced with small integer arithmetic, not a model run; its numerical/behavioral impact was not measured and no source was changed. The diagram should say allocated/configured capacity 3000, with this actual source behavior in the detailed cache view, rather than promise 3,000 visible keys after wrap. Ring mapping · causal mask

Depth attention cache

Each of six depth layers allocates [2,B,16,16,64] for a fresh within-frame streaming loop. At BF16/B=1, all six total 393,216 bytes. Its offsets run over codebook q0…15 and do not represent conversation time. The same ring mapping means that at the final q15 step a full 16-slot ring reports 15 causally visible keys in the pinned implementation. No claim is made that this changes the released model's measured quality.

Generator delayed-code cache

LMGen separately allocates an integer token ring [B,17,max_delay+3] = [B,17,4] and a boolean provided ring of the same shape. These are not attention KV caches. They hold delayed text/audio codes, initial tokens and flags distinguishing supplied tokens from generated tokens. _LMGenState.offset is separate from the temporal/depth attention offsets. Generator state

7. One exact generator step

Let s be LMGen.offset at call entry and C=4 the code-ring length. Delays in stream order are:

k0: agent text             delay0
k1: agent codebook0        delay0
k2…8: agent codebooks1…7   delay1
k9: user codebook0         delay0
k10…16: user codebooks1…7  delay1

1. Place incoming user codes in k9…16 at (s+delay[k]) mod C and mark them provided. Optional forced agent codes/text use their corresponding slots and delays.
2. Initial calls insert native initial tokens. At s=0, initialize the first ring position, increment the offset and return no model output.
3. Read temporal input from (s−1) mod C; select the current delayed target/provided slice at s mod C.
4. Compute the temporal output and agent-text logits from that input. Sample text; if text was supplied, use the supplied text for depth conditioning.
5. Run q0…15 depth steps. At each position compute/sample logits, then use the supplied target instead where provided is true. The selected token is the next position's input.
6. Write sampled text/audio codes only into unprovided target slots. Clear the consumed input's provided flags.
7. Once the initial delay padding is complete, gather aligned output stream k from (s−max_delay+delay[k]) mod C, then increment s.

With a maximum delay of 1 frame, output uses text/semantic codes from ring time s−1 and delayed acoustic codes from ring time s. This is why an individual current prediction and the emitted aligned audio frame are not interchangeable. The returned code tensor is [B,17,1]; the server passes tokens[:,1:9] to Mimi to render the eight agent codebooks. Input preparation · selection/output alignment · depth loop · server decode

Default generator sampling is enabled: audio temperature 0.8/top-k 250, text temperature 0.7/top-k 25. The pinned ServerState does not override these values. They are released-path defaults, not a recommendation or latency/quality measurement.

8. Native training alignment

forward_train(codes:[B,17,T]) applies the per-stream delays, prepends initial tokens, and runs the temporal model on the first T delayed input positions. It constructs each depth input from the appropriate target text/previous code plus that position's temporal projection. Depth training reshapes [B,T,16,1024] to [B×T,16,1024], treating each frame's codebooks as a separate sequence.

It then undelays the logits back to the input code timeline and supplies validity masks. Output shapes are audio logits[B,16,T,2048], audio mask[B,16,T], text logits[B,1,T,32000], text mask[B,1,T]. Tail positions without valid delayed targets are marked invalid and filled with NaN; native no-input codes are masked. These outputs are already aligned: adding another generic next-token shift is wrong. The provided user-audio channels remain distinguishable from generated agent-audio channels when defining an adaptation loss. Native train path and LMOutput

9. Startup prompt and clock facts

The server uses a 24 kHz/12.5 Hz Mimi instance, so a live audio frame is 1,920 samples/80 ms. It passes these rates to LMGen; its class default 32 kHz is not the effective server rate. The constructor's prompt-silence count is int(0.5×12.5)=6 frames: 0.48 seconds of model-frame positions, not exactly 0.5 seconds. Prefill computation is not paced to make its wall time equal that duration.

Startup order is voice prompt, six special silence/sine frames, system-tagged text prompt, six silence/sine frames. Text is wrapped as <system> … <system>. Each text token advances one model step, paired with the provided agent-silence/user-sine codec inputs. Cached voice embeddings replay the native summed embeddings, rebuilding temporal history, then restore the short token ring. Prompts and initial delay padding therefore consume offsets before live audio begins.

These startup mechanisms do not constitute a dynamic private tool/latent channel. The native server accepts live audio, not mid-call evidence updates. Agent text-token overrides alter the speech-aligned text stream. A proposed Inkling connector must be visibly separate from this native architecture. Server startup · prompt routines

10. Instructions for the native diagram

• Label the base as PersonaPlex-7B-v1; published full language-model total 8.371B, with individual tensor shapes calculated from source code and the fact that the checkpoint header was not read available in its details.
• Expand the temporal block into exact pre-RMSNorm, packed QKV, 32 query/key/value heads, full-head RoPE, causal SDPA, output projection, residual, pre-RMSNorm, split SiLU-gated FFN and residual. Do not draw GQA, cross-attention, convolution or a separate attention gate as native.
• Draw the final temporal RMSNorm/text head outside the32-block stack. Show that the normalized temporal output also feeds the per-position depth projections.
• Draw six depth blocks with a q-selected weight set, not six globally shared matrices. Show 16 sequential positions, q0…7 agent/q8…15 user, and the supplied-code selector between positions. No depth RoPE or conversation-time cache.
• Draw three separate cache concepts: long temporal KV, fresh per-frame depth KV and the four-slot delayed-code ring. Expose their different shapes and offsets.
• Place undelay/output alignment after depth prediction and show only the first eight audio codebooks entering the agent Mimi decode path.
• Keep the proposed private connector optional and labelled untrained. No animation should imply measured activations, actual generated speech, or proved live integration.

Verification performed: primary source inspection; authenticated/public metadata queries without retained credentials; exact parameter-family arithmetic and aggregate reconciliation; integer reproduction of the ring-position boundary. Not performed: safetensors-header read, tensor hashing, checkpoint loading, GPU/CPU model execution, training, numerical parity, latency measurement or native code repair.

PersonaPlex's actual Mimi codec: source audit

Audit date: 2026-10-04. Source: NVIDIA/personaplex commit 3428dfd95309a7f3c84fd93259ded0f810d1ff91, specifically its own moshi/moshi implementation. This is not an architecture inferred from the generic Mimi paper.

Mimi converts one mono waveform into eight discrete codes every 80 ms, and converts generated codes back into waveform audio. Its internal convolution and transformer rate is 25 Hz, not 12.5 Hz. An additional learned convolution halves that rate before quantization; a learned transposed convolution doubles it before decoding.

No model, tensor checkpoint or audio was loaded. Dimensions, parameter arithmetic and clock relationships below are derived from the pinned constructors and execution paths. One cache-index edge case was reproduced using scalar arithmetic only. GPU execution, waveform quality, streaming/nonstreaming numerical equivalence and measured latency remain untested.

A codebook is a table of learned vectors indexed by discrete codes. Residual vector quantization (RVQ) represents a vector using successive codebooks that encode the remaining error. Mimi uses separate semantic and acoustic quantizers; residual subtraction happens within each quantizer, not between these two branches.

Complete encode/decode order

ENCODE
mono PCM [B, 1, N] at 24,000 Hz
  → SEANet encoder: total stride 960, output width 512 at 25 Hz
  → encoder ProjectedTransformer: 8 layers, width 512, 25 Hz
  → learned Conv1d: 512 → 512, kernel 4, stride 2, groups 1
  → width-512 latent at 12.5 Hz
  → split quantizer:
       same latent → semantic input projection 512 → 256 → one codebook
       same latent → acoustic input projection 512 → 256 → seven active RVQ codebooks
  → concatenate semantic then acoustic indices: [B, 8, F]

DECODE
indices [B, 8, F]
  → semantic code lookup → semantic output projection 256 → 512
  + sum of seven acoustic code lookups → acoustic output projection 256 → 512
  → summed width-512 latent at 12.5 Hz
  → learned depthwise ConvTranspose1d: 512 → 512, kernel 4, stride 2, groups 512
  → decoder ProjectedTransformer: 8 separate layers, width 512, 25 Hz
  → SEANet decoder: total upsampling 960
  → mono PCM [B, 1, 1920 × F] at 24,000 Hz, subject to stream/padding boundaries

MimiModel.encode calls the waveform encoder, encoder transformer, downsampler and quantizer in that order. decode calls quantizer decoding, upsampler, decoder transformer and waveform decoder. The two transformers are independently constructed modules; they do not share weights. Output audio decoding does not rerun the waveform encoder. Mimi produces PCM directly; no additional TTS model follows it in this path.[1][2]

Exact loaded configuration

The dilation_base=2 setting does not produce a dilation-2 convolution here. The loop uses dilation_base ** j, and n_residual_layers=1 means only j=0: every residual convolution has dilation 1. The SEANet constructors' generic defaults—32 filters, three residual layers and ratios [8,5,4,2]—are overridden by NVIDIA's loader.[1][3]

The loader says weight normalization was used before inference weights were processed. The runtime constructors select norm="none", so the diagram should not insert weight-normalization, batch-normalization or layer-normalization blocks between SEANet convolutions. Transformer LayerNorm is a separate operation.[1][4]

SEANet encoder, every stage

For the table, a frame-aligned 80 ms input contains 1,920 samples. Shape entries are the source-derived lengths for that input; they are not measured execution results. All convolutions below have dilation 1, groups 1 and a bias.

The encoder reverses the loader ratio list. Its actual downsampling order is 4, 5, 6, 8, not 8, 6, 5, 4.[3]

Each residual block of width C is exactly:

                  ┌──────────────── identity ──────────────────┐
x [B,C,T] ────────┤                                            + → [B,C,T]
                  └→ ELU → Conv(C,C/2,k3) → ELU → Conv(C/2,C,k1) ┘

There is no activation after the addition inside the residual block. The next downsampling stage starts with its own ELU. StreamingAdd aligns and adds available prefixes of the two branches while retaining any unmatched suffixes.[3][8]

For explicit weight-shape labels, a Conv1d weight is [out_channels, in_channels, kernel] here. Thus E0 is [64,1,7], E4b is [1024,512,16] and E5 is [512,1024,3]. Every listed bias has length equal to the output channel count.[4]

Both codec transformer stacks

The encoder and decoder each use the following separate stack. The wrapper accepts [B,512,T], transposes to [B,T,512], applies eight blocks and transposes back. Its input projection is absent because input_dimension == d_model == 512; its only output projection is Identity. There is no extra final stack normalization in StreamingTransformer.forward.[5]

Each block implements:

x
  → LayerNorm(512, eps=1e-5)
  → ordinary multihead self-attention
  → learned elementwise LayerScale γ_attention
  → add original x
  → LayerNorm(512, eps=1e-5)
  → Linear 512→2048 → GELU → Linear 2048→512
  → learned elementwise LayerScale γ_ffn
  → add attention-block result

Exact attention/FFN details:

• 8 Q heads, 8 K heads and 8 V heads, each width 64. This is ordinary MHA, not grouped-query attention.
• Combined bias-free QKV projection: weight shape [1536,512], split into three [512,512] projections.
• Queries, keys and values become [B,8,T,64].
• RoPE rotates Q and K, not V. It uses adjacent real/imaginary pairs across all 64 dimensions: 32 frequencies 10000^(-m/32), m=0…31, with positions offset by the streaming position counter. Rotary arithmetic uses float32 before casting back to the input dtype.[6]
• Attention calls PyTorch scaled dot-product attention with the explicit boolean causal/context mask and dropout_p=0.0.
• Bias-free attention output weight: [512,512].
• Bias-free FFN weights: [2048,512] and [512,2048]. Activation is the constructor's F.gelu; there is no GLU/SwiGLU gate in this codec configuration.
• Both LayerNorm modules have 512 learned scale and 512 learned bias elements. They are ordinary LayerNorm, not the rms_norm_f32 used elsewhere in the speech language model.
• Each residual branch has a learned 512-element LayerScale. 0.01 is the initialization supplied to the constructor, not a verified value of the trained checkpoint.
• weights_per_step=0; these codec blocks share their own weights across time, without the per-codebook weight banks of the language model's depth transformer.
• positional_embedding="rope" means the separate additive sinusoidal-position branch does not execute. max_period=10000 is the rotary frequency parameter, not the attention-window length.

Each block has 3,148,800 constructor-derived nn.Parameter elements: 1,048,576 attention weights,2,097,152 FFN weights,2,048 LayerNorm parameters and 1,024 LayerScale parameters. Each eight-block stack therefore has 25,190,400. This is architecture arithmetic, not a tensor-header or runtime parameter count.[5]

Attention clocks and cache

Each codec transformer runs at 25 positions/second. Its nominal 250-position attention window is therefore a 10-second position grid, with the exact nonstreaming mask accepting 0 ≤ query_position − key_position < 250: current position plus at most 249 earlier positions, a maximum direct attention lag of 9.96 seconds. This is a per-layer constraint; it is not a fixed 10-second limit on information represented by the entire multi-layer model.

For each layer, streaming K/V capacity is [2,B,8,250,64], totaling 256,000 scalar slots per batch element. Eight layers have 2,048,000 K/V scalar slots; both codec stacks have 4,096,000. Cache dtype comes from the projection parameter dtype. These are element counts, not measured memory use; convolution state, graph storage and other buffers are additional.[5]

The pinned ring cache has a source-level indexing caveat. At capacity 250, after writing 250 positions, its delta <= 0 position reconstruction labels slot 0 as position 250 instead of 0. Query 249 then masks that slot as future. The same scalar calculation gives 249 valid keys at end_offset=250,251 and 500 for a single final query, despite capacity 250. This was reproduced from the exact index formula without tensors, models or audio. Do not describe the streaming implementation as empirically attending to exactly 250 valid keys on every step; acoustic consequences and any repair are untested and outside this audit.[5]

Learned resampling around the quantizer

SEANet's hop is 4 × 5 × 6 × 8 = 960 samples. At 24,000 Hz, encoder_frame_rate = 24000 / 960 = 25 Hz. The requested quantizer rate is 12.5 Hz, so MimiModel constructs a stride-2 pair.[1][2][7]

The upsampler is depthwise, while the downsampler mixes channels. This asymmetry is explicit: MimiModel defaults upsample_channel_wise_bug=True to reproduce its historical implementation, NVIDIA's loader does not override it, and the value is passed as channel_wise only to the upsampler. Silently replacing it with a dense 512×512 upsampler would change the network and checkpoint shapes.[2][7]

Neither resampler performs linear interpolation in this configuration. The latent upsampler sits before the decoder transformer, so that transformer sees two positions per 80 ms codec tick. The encoder transformer also sees 25 Hz positions, before downsampling.

Split quantizer: exact paths and active capacity

NVIDIA constructs SplitResidualVectorQuantizer(dimension=256, input_dimension=512, output_dimension=512, n_q=32, bins=2048). Its default n_q_semantic=1 creates a semantic quantizer with one codebook and an acoustic quantizer with 31 codebooks. After loading, set_num_codebooks(8) changes only the acoustic active count to 7. All 32 codebooks are still configured in the module; the production encoding route uses 1 semantic +7 acoustic.[1][9]

e ∈ R^512
  ├→ W_sem,in [256,512,1] → z_sem ∈ R^256
  │    → nearest vector in semantic codebook 0 [2048,256] → code0
  │
  └→ W_ac,in  [256,512,1] → r0 ∈ R^256
       → acoustic codebook 1 → code1, r1 = r0 − embedding1[code1]
       → acoustic codebook 2 → code2, r2 = r1 − embedding2[code2]
       → …
       → acoustic codebook 7 → code7

decoded = W_sem,out embedding0[code0]
        + W_ac,out Σ(j=1…7) embeddingj[codej]

All four outer projections are bias-free kernel-1 convolutions. Input projection weights are [256,512,1]; output weights are [512,256,1]. The semantic and acoustic paths receive the same 512-dimensional latent through separate projections. The acoustic path does not take e − decoded_semantic as its input. Residual subtraction occurs within the acoustic RVQ chain, in its own 256-dimensional space. Decoding sums the two separately projected 512-dimensional outputs.[9][10]

Within each active codebook:

• The input is flattened over batch/time for vector lookup.
• Centroids have shape [2048,256].
• The effective centroid matrix is embedding_sum / clamp(cluster_usage, min=1e-5) and is cached after construction.
• Encoding uses torch.cdist(..., p=2) and argmin; no softmax or sampling is applied by this codec quantizer.
• Decoding performs an embedding lookup. Acoustic decoded vectors are summed in codebook order before the acoustic output projection.
• The inner VectorQuantization input/output projections are identities because its codebook dimension equals 256.
• Output codes have shape [B,8,F] and integer values 0…2047. “Semantic” is the branch's learned role; these codes are not literal text tokens or explicit symbolic facts.

Configured codebook state consists of 32 embedding_sum matrices, 32 cluster_usage vectors and 32 initialization scalars. These are registered buffers, not nn.Parameter objects. The inference implementation's quantization forward returns zero loss in its inner vector-quantization module; its comments about EMA/commitment training should not be substituted for the actual executed inference code.[10]

Eight codebooks ×11 bits/index ×12.5 frames/second gives a nominal 1,100 bits/second per mono stream for fixed-width code indices. That is codec-symbol arithmetic, not the websocket, Opus, PCM or container bitrate. The configured 32-codebook capacity would correspond to 4,400 bits/second; PersonaPlex's loader selects eight.

SEANet decoder, every stage

The decoder keeps loader order 8,6,5,4. Every stage upsamples before its residual block. This differs from the encoder, whose residual block precedes downsampling.[3]

After one codec frame is looked up and upsampled, the decoder transformer returns [B,512,2]. All SEANet convolutions below have groups 1 and a bias; Conv1d dilation is 1, and the streaming ConvTranspose implementation requires dilation 1.

A ConvTranspose1d weight uses [in_channels, out_channels/groups, kernel], so D1a is [1024,512,16]. Its bias is 512 elements. The initial D0 Conv1d weight is [1024,512,7]; final D5 is [1,64,3] with a one-element bias.

There is no final tanh or other requested final activation. final_activation remains None. There is also no LSTM in this instantiated SEANet path: the long-context codec processing is in the two transformer stacks. The decoder is a separately learned synthesis network, not the transpose of all encoder weights.

Stateful streaming, padding and dependency boundaries

StreamingConv1d uses total padding effective_kernel − stride, where effective_kernel=(kernel−1)×dilation+1. For the SEANet operators here, that means 6 initial positions for kernel 7 / stride 1, 2 for a residual kernel 3 / stride 1, and exactly the stride for each kernel-2×stride downsampler. SEANet uses constant zero padding; the latent stride-2 downsampler instead uses replicate padding.[4][7]

In nonstreaming mode, causal convolution adds that padding on the left and enough extra padding on the right to complete the final stride. In streaming mode, initial left padding is inserted once. RawStreamingConv1d concatenates a retained suffix from the previous call, emits only complete convolution windows, then retains the unused suffix. A partial final audio chunk therefore cannot be assumed to flush itself into a complete codec frame.[4][8]

For transposed convolution, the raw streaming implementation retains kernel − stride unfinished output positions. The next call adds its overlapping contribution to that partial tail, subtracting the previously included bias once when necessary. It emits the completed prefix and retains the new tail. With kernel2×stride, each stage retains one stride of partial output at that stage's rate. Nonstreaming causal transposed convolution instead trims kernel−stride positions from the right because trim_right_ratio=1.0.[4][8]

The two residual branches use StreamingAdd to avoid adding misaligned or unavailable suffixes. Codec transformers retain their own K/V rings and absolute position counters. MimiModel's streaming state also wraps encoder/decoder transformer execution in CUDAGraphed on CUDA; that wrapper is an execution optimization, not an additional neural layer. It is disabled when the device is not CUDA.[2][8]

Consequences for the system diagram:

• One 80 ms packet is not decoded as a fresh independent audio clip. Decoder convolution overlaps, transformer K/V and position state continue across packets.
• Encoder and decoder caches are separate, even within one Mimi instance. Resetting a codec between every frame changes the computation.
• A user-audio stream and a generated-agent stream must keep the intended stream/session state ownership. Shared weights do not mean arbitrary reuse of mutable stream caches.
• The 80 ms clock is the codec frame period, not measured compute latency, time to first audible output, or total conversational delay.
• An 80 ms aligned input block produces two 40 ms encoder positions and one codec position by shape arithmetic. Model/token delays, packet transport, decoding compute, queueing and physical playback are separate.
• MimiModel.forward truncates reconstructed audio back to the original waveform length. Standalone decode(codes) has no original waveform length to recover; the caller owns any final-length handling.[2]

Source-derived parameter inventory and public file metadata

The following totals were calculated from the exact dimensions above, including convolution biases, bias-free transformer linears, LayerNorm scale/bias and LayerScale vectors. They are not the result of importing the model or opening tensor metadata.

The 32 codebooks add 16,842,784 persistent buffer elements: 32 × (2048×256 + 2048 + 1). If all these parameters and buffers are stored as FP32, their tensor payload would be 384,605,572 bytes, before safetensors headers. Dtype and tensor inventory were not opened in this audit; this is an explicit conditional arithmetic check. Derived centroid caches and streaming buffers are additional runtime allocations.

The unauthenticated public Hugging Face metadata endpoint returned the following actual file metadata on the audit date:[11]

Only metadata was read. The file contents were not downloaded, its tensor shapes/dtypes were not inspected and no conclusion about numerical equivalence during inference follows. NVIDIA's loader creates Mimi without an explicit parameter dtype, then loads weights and moves the model to the selected device; the caller/default construction dtype and actual loaded tensors must be checked in a real runtime measurement.[1]

Diagram checklist

1. Expand SEANet encoder → encoder transformer → learned downsample → split quantizer, in that order.
2. Draw the semantic and acoustic branches in parallel from the same latent, with separate projections. Show the acoustic branch's seven active residual lookups, not a residual chain that starts from the semantic branch.
3. Expand decode as lookup/projection/sum → learned depthwise upsample → decoder transformer → SEANet decoder.
4. Label the three clocks: 24,000 samples/s →25 latent positions/s →12.5 code frames/s, then their inverse.
5. Mark 32 configured /8 active codebooks, 1 semantic +7 acoustic, each 2,048 entries×256 dimensions.
6. Keep codec transformers 512-dimensional, with 8 layers and 8 heads, separate from PersonaPlex's much wider language-model transformers.
7. Show stateful convolution overlap and K/V caches. Do not show packet boundaries as model resets or 80 ms as a measured response time.
8. Place the cache-index caveat and source-only verification limits in the detailed evidence view.

Pinned primary sources

1. models/loaders.py: audio/codec constants, SEANet, transformer and quantizer construction, selection of active codebooks.
2. models/compression.py: MimiModel.__init__, _encode_to_unquantized_latent, encode, decode and forward; resampling placement and streaming graph wrappers.
3. modules/seanet.py: residual block; encoder ratio reversal and exact stage construction; decoder ordering and optional final activation.
4. modules/conv.py: runtime normalization, default groups/bias, causal padding and transposed-convolution trimming.
5. modules/transformer.py: LayerScale, LayerNorm, ring-cache indexing, attention, FFN and projection wrapper.
6. modules/rope.py: exact rotary frequency/pairing and dtype behavior.
7. modules/resample.py: learned versus fixed resampling, group selection, kernel/stride and padding.
8. modules/streaming.py: recursive state reset, streaming addition, convolution suffix and transposed-convolution overlap state.
9. quantization/vq.py: separate branch projections, same-input split, selection of active codebooks and decoded branch sum.
10. quantization/core_vq.py: codebook buffers/centroids, nearest-neighbor encoding, lookup decoding and residual iteration.
11. Public model metadata and revision-pinned model tree: repository revision, filename, LFS hash and byte count.

PersonaPlex reference client and transport

Source snapshot: NVIDIA/personaplex 3428dfd95309a7f3c84fd93259ded0f810d1ff91, reviewed on 2026-10-04. These are the reference client/server source operations. No microphone, audio playback experiment, model inference or equivalence to the deployed client bundle was measured.

PCM (pulse-code modulation) represents audio as numerical samples. Opus compresses audio for transport. An AudioWorklet runs the browser processing code that supplies samples to the audio output.

Input transport

The browser obtains a microphone stream and passes it to opus-recorder. Its explicit configuration is mono, encoderSampleRate=24000, encoderFrameSize=20 ms, maxFramesPerPage=2, recordingGain=1, resampleQuality=3, encoderComplexity=0, and encoderApplication=2049. Its nominal buffer length is round(960 × AudioContext.sampleRate / 24000). The source itself notes that this buffer-length value has no impact for mono audio; frame size and frames per page are the relevant controls. These settings are not a measured 40 ms network-latency guarantee.

The recorder emits encoded pages. The socket audio message is a leading 0x01 byte followed by those bytes. The server gives the payload to sphn.OpusStreamReader, obtains PCM at the Mimi sample rate, accumulates a full native frame, and runs Mimi. The browser microphone sample rate and the server's 24 kHz model PCM are therefore distinct. The source computes microphone duration from the recorder's encoded-sample counter divided by 48,000; it does not use that counter as 24 kHz raw PCM samples.

Sources: microphone recorder, wire encoder, server receive and PCM accumulation.

Output transport and text

The native generator returns re-aligned text and audio channels. The server decodes only channels 1:9 as agent audio through Mimi. This yields model-rate PCM, which is encoded by sphn.OpusStreamWriter. Outgoing socket audio carries 0x01 plus the Opus bytes. The browser's decoder worker receives encoded pages, decodes them with decoderSampleRate=24000, and resamples to the actual AudioContext.sampleRate. Its requested output buffer length is 960 × AudioContext.sampleRate / 24000, corresponding to 40 ms; message scheduling and buffering are separate.

The server's text output is a separate 0x02 + UTF-8 message. It excludes its designated text padding/end-padding tokens. This text stream is not a per-word device-playback acknowledgment and does not itself prove exact correspondence to intelligible played speech.

Sources: server output, audio reception, decoder initialization, text/audio protocol.

AudioWorklet buffering is part of the actual path

The reference worklet maintains a queue of decoded PCM frames, an offset in the first frame, a start/rebuffer state and counters. It initially waits for 80 ms of buffered samples, then a further 10 ms partial-buffer allowance. The queue threshold is the sum of initial, partial and maximum-buffer allowances: initially 80 + 10 + 10 = 100 ms.

When the queue reaches the threshold, the code drops its oldest samples until the queue falls to the initial-plus-partial allowance (initially 90 ms), then increases the maximum allowance by 5 ms, capped at 80 ms. On underflow, it increases the partial allowance by 5 ms, also capped at 80 ms, and returns to the rebuffer state. Both values reset to 10 ms on initialization. The source applies a fade-in to the first resumed output and a fade-out to the available prefix when an output block underflows. These are reference-client buffering operations, not learned PersonaPlex behavior or measurements of conversational interruption.

The worklet distinguishes totalAudioPlayed from actualAudioPlayed; the latter counts samples actually copied into the output block. The UI derives a missed-audio duration from their difference. It also reports timing counters to the main thread. The inspected server receiver does not implement a played-range acknowledgment channel back into the model or Inkling. A new feedback protocol would have to connect output generations, played sample ranges and text alignment explicitly.

Source: actual worklet, playback counters in the UI.

Protocol declarations versus implemented server behavior

The client protocol declares handshake 0x00, audio 0x01, text 0x02, control 0x03, metadata 0x04, error 0x05 and ping 0x06. This enumeration is not evidence that the Python server implements all those incoming message types. Its live receive loop handles audio kind 1 and logs other kinds as unknown. In particular, there is no released private Inkling-memory message or feedback handler linking playback records to generated output.

The initial role/voice prompt follows the separate prefill path. The client waits for the server handshake before starting its connected audio flow. The reference decoder can be initialized before use, but a timer for initialization is not a measured audio readiness guarantee.

Sources: message definitions, socket handshake, server.

Diagram interpretation

Show microphone/recorder → Opus/WebSocket → server PCM → Mimi encoder. Show Mimi decoder → PCM/Opus/WebSocket → browser decoder/resampler → AudioWorklet → audio output. Show UTF-8 text separately. Native generated-token self-history loops through the model's delay and attention caches, not through decoding and re-encoding the played waveform.

Private Inkling state delivery, task-control decisions and feedback identifying the portion played by the device are proposed additions. The physical sample rate of a particular browser output device, actual network latency, intelligibility, and human hearing were not measured by this source audit.

Historical design note. The implemented Stage 1 architecture, current losses and evaluation are documented in Current sequence memory.

Inkling → PersonaPlex: a private evidence connector

Research snapshot: 2026-10-04. Proposed design; no connector has been implemented, trained or measured. This document supports the new Atlas research tab. It does not change the running architecture study or authorize model execution.

The proposal is to retain PersonaPlex's pretrained listening, speech, voice and turn-taking computation, while giving its temporal language model a small cross-attention module that uses only Inkling states available at that time. Start with one zero-gated cross-attention module before temporal block 16, using zero-based block indices. Inkling continues to choose and execute task-related tool calls. PersonaPlex generates its own speech and audio; the connector supplies evidence, not words it must recite.

A frozen model keeps its weights unchanged during training. Cross-attention lets PersonaPlex use its current state to select information from Inkling states. The residual gate scales the added contribution; setting it to zero removes that contribution. Here, private means that these states are supplied internally rather than spoken to the caller.

1. What the released system actually contains

The source below is pinned to NVIDIA/personaplex commit 3428dfd95309a7f3c84fd93259ded0f810d1ff91. These are published-source construction facts, not a new checkpoint-tensor or live-runtime audit.

Sources: loader geometry and effective construction, LM construction, embedding sum and depth execution, gated feed-forward geometry, streaming transformer, Mimi implementation.

How the effective depth setting was checked: the pinned server downloads config.json to increment the download counter, discards the returned path, then calls get_moshi_lm(weight_path, device, cpu_offload) without a config argument. That loader applies the 16-position override and can copy/expand missing depth weights. A public, unauthenticated request for the hosted config.json returned HTTP 401 during this review; its contents and checkpoint shapes were not inspected. The diagram should describe the pinned server construction, and verify the loaded model before presenting these dimensions as measurements of a running deployment. See server lines 427–445 and loader.

The 3,000-position context is nominally 240 seconds of frame positions. Startup prompt positions also consume model history. It is not proof of complete four-minute recall or indefinite task retention. The 80 ms frame and one-frame codebook staggering are not a measured end-to-end latency guarantee.

2. The present external-input boundary

At connection start, the server reads the voice and text prompt, resets streaming state, and prefills voice → silence → text → silence before its handshake. During the call, the websocket receiver accepts audio messages; it has no tool-result or latent-state message handler. Server

Two low-level entry points should not be misrepresented:

• step(text_token=...) forces the agent speech text stream. That token conditions the depth model's audio. It is a speech-control primitive, not a private knowledge channel.
• step_embeddings(...) is used for cached voice-prompt replay. It replaces combined input embeddings and advances frame machinery with dummy inputs. It is not an Inkling adapter trained to coexist with live user audio.

The connector therefore needs integration into the model and an internal evidence input. Changing a startup prompt string alone does not update the state of an ongoing call. Replaying system prompts mid-call has not been shown to preserve continuous conversation.

3. The first connector to build

Keep the native sum of agent text, agent audio and user audio embeddings. Run temporal blocks 0–15 unchanged. Insert one small cross-attention residual, then run blocks 16–31, the original output normalization/text head, and the original depth model/Mimi decoder.

caller PCM → Mimi user codes ─┐
prior agent audio codes ─────┼→ native embedding sum → temporal blocks 0…15
prior agent text ────────────┘                         │
                                                     ↓ query
Inkling + actual tools → causal state memory → private cross-attention
                                                     │ gated residual
                                                     ↓
                                  temporal blocks 16…31
                                      ├→ agent text
                                      └→ depth model → agent codes → Mimi → PCM
                                                                             │
                               record of played audio samples ← playback ────┘
                                             ↓
                           feedback describing played speech → Inkling

This is a concrete starting placement, not a claim that the middle block is optimal. It gives the cross-attention module a query based on the current conversation while leaving 16 pretrained temporal blocks to integrate evidence into both text and audio decisions. Do not add connectors at every layer initially. The existing StreamingTransformer loop and layer interface are the natural integration points; both full-sequence training and streamed inference must use the same augmented computation. Transformer

For frame decision t, let h_t ∈ R^4096 be the state after block 15, and let S_t ∈ R^(M×4096) contain the selected, causally available Inkling rows. Start with one explicitly identified source feature view; preserve the native capture location and normalization semantics. Additional layer mixtures are separate input comparisons.

For PersonaPlex, S0–S41 are the completed outputs of Inkling blocks 0–41: f_l = MLP_l(N_mlp,l(r_l)), then S_l = r_l + SC_mlp,l(f_l), after the MLP-output short convolution and residual addition. Final = RMSNorm_final(S41) is the separate native final state before readout division by 16. The MTP/SGLang raw auxiliary taps precede that convolution and are not interchangeable with these completed outputs. This choice gives the capture point a consistent meaning across runtimes; it does not establish better speech quality.

With connector width a=512, eight heads of width 64, bias-free projections and trainable RMSNorm scales:

Q_t = RMSNorm_q(h_t) W_Q                  W_Q : 4096 × 512
K_t = RMSNorm_s(S_t) W_K                  W_K : 4096 × 512
V_t = RMSNorm_s(S_t) W_V                  W_V : 4096 × 512
A_t = softmax(Q_t K_tᵀ / sqrt(64) + causal_availability_mask)
r_t = concat_heads(A_t V_t) W_O           W_O : 512 × 4096
h'_t = h_t + tanh(g) r_t

Initialize scalar g=0, with ordinary nonzero projection initialization. The closed gate preserves the original forward function for the same native input/state. Do not zero both the gate and output projection: that would prevent either from receiving the initial learning signal. Initially the gate learns; projection gradients appear as it opens. Empty memory has the defined result r_t=0, without applying softmax to an all-masked row.

This design is motivated by the zero-gated residual in OpenFlamingo's implementation. It is not a claim that Flamingo already supports PersonaPlex, or that its image results demonstrate an Inkling speech benefit. Unlike inserting soft tokens into PersonaPlex, reading the memory creates no extra audio frame or temporal position. Unlike overriding agent text, it allows the model to decide what to say or whether to remain quiet.

4. What is in private memory

Memory is an ordered view of real Inkling states, with source positions and the point at which each row became available. Keep source row IDs, model/view identity, source event, tool-result identity, branch/version and arrival timestamps in provenance. A state captured after a tool return cannot be supplied at a frame before that return. The model may learn from event type or age if explicitly represented and trained; provenance IDs are not predictive prompt features.

Keep three facts separate: what the user requested, what the reasoning model represents or plans, and what a tool actually confirmed. Latent vectors are not guaranteed truth. For exact new names/numbers, retain a later alternative that supplies the factual text explicitly; do not assume a cross-attention module for hidden states has lossless access to literal values.

During inference at fixed weights, reuse projected keys/values for unchanged rows rather than recomputing them every frame. Training must retain differentiable projections and must not reuse values computed before an optimizer update. The first implementation needs one component that writes the memory and publishes versions that remain unchanged while being consumed, not a new distributed storage system. Store the version actually consumed by each frame. Old evidence can be marked superseded for subsequent decisions without rewriting the past frame's evidence.

M is a capacity variable, not a selected context-length limit. Any bounded selection/compression must retain the required task/history scope and be evaluated for lost facts and corrections. The numerical examples below do not authorize dropping difficult sources or older commitments.

5. Frame clock and update contract

1. Preserve PersonaPlex's native 12.5 Hz frame schedule, codebook delays and temporal positions. Inkling token positions and wall time are different clocks.
2. At the start of a frame computation, choose the newest memory version whose transfer is complete and whose evidence is available. Hold that version fixed through the frame's temporal, text and depth computation. An update arriving during that computation becomes eligible for the next frame.
3. Append or supersede memory evidence without resetting the temporal model, Mimi state or speech history. Private memory changes affect new decisions; they do not retroactively change generated or rendered audio.
4. Keep a mapping from session audio frame/sample positions to LMGen.offset. That offset includes startup prompt/initial-delay steps and must not be treated as elapsed conversation time.
5. Train with the same frame-level availability table. Native prepare_step_input() uses delayed input/target positions; do not attach a tool result to a teacher-forced target frame merely because that result exists later in the record.

These boundaries are part of the proposed model contract. The released server and its CUDA-graph wrappers do not yet accept this memory. The implementation must pass the memory to both ordinary forward calls and CUDA-graph execution; accepting a Python argument that the captured graph never consumes would not implement the connector. LM step and delayed training path

6. Learning plan

First, establish the unmodified reference. Preserve the loaded checkpoint, voice prompt, text prompt, tokenizer, codec, delay policy and sampling settings. Verify that a closed connector gate reproduces its forward values on the same input, and that full replay and streamed execution agree across a real evidence-arrival boundary. This is a focused verification of the new computation, not a capability claim.

Then train the connector with the base model frozen. Use complete causal examples containing known/unknown facts, tool returns, user corrections, useful quiet and spoken responses. Questions and answers about model state may help train the connector to interpret Inkling states, but the speech objective needs time-aligned user audio, agent text and agent audio. Backpropagate through the frozen downstream computation; do not use the @torch.no_grad() inference generator as the training entry.

Use the existing native delayed-code training formulation. Score grounded agent text and agent audio with declared weights/masks, preserving silence and padding distinctions. The effective 16-position depth path includes provided user codebooks; do not report its outputs as 16 agent codebooks. If an inherited user-audio auxiliary loss is retained, describe it separately and keep it common across comparisons. Mask unavailable/invalid labels explicitly rather than inserting successful defaults.

Only then consider low-rank adaptation (LoRA) of the temporal model if the connector learns evidence but cannot integrate it adequately. Keep Mimi, voice conditioning and the depth decoder fixed initially so the first comparison isolates evidence integration. LoRA/full adaptation is an experiment, not an automatic remedy for bad labels or stale evidence. Reassess native speech quality, turn-taking and task correctness after each substantive adaptation.

Upstream Moshi-Finetune supplies real stereo-audio/transcript training, LoRA/full tuning and masked text/audio losses. It is not drop-in PersonaPlex support: its model wrapper imports upstream CheckpointInfo and requests LoRA-aware construction; the pinned NVIDIA loader exposes a different interface. Verify one implementation using the actual PersonaPlex weights, delay schedule and forward methods. Reuse its data/loss ideas without claiming its example checkpoints establish compatibility with this fork.

The central behavioral comparison is native PersonaPlex versus connected PersonaPlex on identical calls, supplemented by matched text-evidence and correct/shuffled/removed-memory controls. An effective connector must change its answer when the relevant evidence changes, preserve uncertainty before a result, and keep speech grounded after a correction. Fluent audio alone is insufficient. A poor result with incorrect labels or unconsumed memory is an implementation/data failure, not evidence that model capacity is insufficient.

7. Retention, feedback and stopping

• Native history and task memory have different jobs. The temporal cache retains conversational computation; private memory supplies current task/evidence state. Neither guarantees that Inkling knows what the user actually heard.
• Feed back delivered speech. Associate generated text/audio with output generations and rendered sample ranges. Only confirmed rendered words belong in the reasoning model's record of commitments communicated to the user. The speech-aligned text head is useful, but it is not by itself proof of exact word-to-sample alignment or intelligible delivery; validate that association against actual audio.
• Preserve unplayed suffix distinctions. Generated, queued, sent and rendered audio differ. Cancelling playback does not automatically erase the corresponding model history. If the speech model must forget a generated suffix that was not played, restore a coherent full state or replay the retained prefix; do not trim only text while retaining contradictory audio/transformer state.
• Keep listening when speech pauses. Quiet audio or a pause is not cancellation of a tool, end of the user's goal or end of the session. Let learned acoustic behavior handle natural pauses and interruptions; do not rename text padding or the native no-input sentinel a learned decision to wait.
• Task control stays explicit. A user stop/correction can require cancelling or revising work, but that decision belongs to the task-control implementation with the actual instruction and tool state. An acknowledgement or sentence ending is not authority to cancel necessary work. Test the effect of feedback on later reasoning-model actions, not merely whether a feedback message was sent.

8. Proposed parameter, memory and arithmetic costs

These are design arithmetic, not measured footprint, latency or throughput. For one bias-free connector with d=4096, a=512, two trainable RMSNorm vectors and one scalar gate:

P = 4 d a + 2 d + 1 = 8,396,801 parameters ≈ 8.40M
BF16 connector weights = 2P bytes ≈ 16.79 MB
Illustrative mixed-precision Adam training state = 16P bytes ≈ 134.35 MB

The 16-byte example assumes BF16 weights/gradients, FP32 master weights and two FP32 moments. Actual optimizer/gradient precision may differ. It excludes all base-model weights, activations, gradients through frozen layers, checkpoint workspace, CUDA graphs, Mimi, transport and audio buffers.

For an illustrative M=128, one call's BF16 raw memory is M×4096×2 = 1 MiB; projected K/V are 2×M×512×2 = 0.25 MiB. This is not a selected 128-row limit. Retaining concurrent versions or training graphs costs more.

With source K/V already cached, the query/output projections plus attention require approximately 4da + 4Ma multiply/add FLOPs per frame: about 8.65M at M=128, or 108M/second at 12.5 Hz. New source rows incur their own projection and transfer cost. Kernel launches, synchronization and memory traffic can dominate small arithmetic; this estimate cannot establish negligible overhead.

Report actual source-ready→memory-published→memory-consumed→first useful audio→rendered timing, steady-state frame deadlines, correction latency and peak memory. Do not substitute codec frame size or a live process for these measurements.

9. Alternatives and the decisions they answer

10. What would constitute progress

The next useful result is a complete live or faithful replay case where a private tool result changes PersonaPlex's subsequent grounded speech without disrupting caller audio, voice, timing or ongoing Inkling work. Preserve failures and full outputs. A learned gate opening, falling QA loss, working websocket, equivalent outputs with and without CUDA graphs or attractive attention map is not that result.

No new framework, comparison of every model variant or automatic expansion of the experiment follows from this proposal. Start with the single connector, real causal data and the native reference. Expand only to resolve an observed limitation, while retaining the original full-conversation, grounding, feedback and task obligations.

Historical design note. The implemented Stage 1 architecture, current losses and evaluation are documented in Current sequence memory.

Four cross-attention connections and a played-audio return

Diagram scope: the diagram shows the specified tensor graph and the proposed return of digitally rendered audio through Inkling's existing audio frontend. Output resampling, queue copying and fade multiplication are explicit numerical operations, not additional neural layers. Selecting source rows, encoding tool text and combining source representations remain outside the forward connector graph. S_t is a supplied matrix and the attention mask is a supplied vector.

The tool-text branch remains an open research option; it is omitted from the graph until its encoder and combination equations are selected, not silently abandoned.

Status: proposed experiment; unimplemented and untrained. This is a separate Atlas view alongside native PersonaPlex and the single-connector proposal. The specified dimensions below describe a candidate architecture, not a demonstrated improvement, optimal design, available runtime or training authorization. No model run was performed for this document.

The objective is useful, continuous speech while Inkling reasons and uses tools. PersonaPlex should be able to continue an explanation, clarify something, correct itself or remain quiet as information changes. It should not merely announce that a model token, tool call or result arrived. This proposal gives the speech model more opportunities and capacity to use available information, then trains its temporal model to use those connections.

Inkling remains the source model

The mathematical graph includes frozen Inkling Small with 42 trunk blocks of width 4096 and its native final normalization. Let H^(r) be the selected captured feature matrix of shape [N,4096], and J_t the selected row indices. The source edge is S_t = H^(r)[J_t,:], shape [M_t,4096]. This indexing adds no learned layer. The feature choice r selects among completed block outputs S0–S41 and the separate Final state; the chosen layers and row-selection rule remain experiment choices.

The return path described below enters that same full model through native audio embeddings. It does not replace the trunk with an audio encoder. The chosen source is the application's post-buffer, post-fade digital output, with new capture and range-acknowledgment integration; it is not a claim about physical hearing.

Specified geometry and open choices

The following remain unresolved, not silently supplied by the diagram:

• Which current and earlier Inkling rows to retain, for how long, and which layer views to use. No relative Inkling/PersonaPlex throughput or benefit of a longer retained sequence is assumed.
• Whether to add learned multi-layer fusion at all. Fusion would combine selected layer representations; its method, sharing across sites and additional parameter count are open. It is optional, not a required warmup module.
• The tool-text encoder, its initialization, dimensions and context processing. There is no selected new pretrained text language model.
• How source order and event type are encoded. Availability must be enforced by masks; it must not depend only on a learned timestamp feature.
• Whether state and tool-token rows receive one joint attention normalization or separate attention operations. The equations below define the core module; separate source branches may require additional parameters and work.
• RMSNorm epsilon and numerical policy, optimizer settings, loss weights, training doses and the final data allocation.

This distinction matters: the four attention modules have an exact parameter count. The complete input pipeline and complete trained model do not yet have one.

1. Information entering the speech model

Native PersonaPlex inputs

Retain the original sum of agent-text, agent-audio and user-audio embeddings. The native model receives 17 delayed streams: one text stream, eight agent-audio codebooks and eight user-audio codebooks. None is replaced with an Inkling vector. The short delayed-code ring, temporal positions, codebook ordering and Mimi state keep their original jobs. Native LM · native generator

Current and selected earlier Inkling states

Use actual captured or live feature representations with their model/view identity, source row and causal history intact. For PersonaPlex, S0–S41 are completed block outputs after each MLP-output short convolution and residual addition. Final applies native final RMSNorm to S41, before readout division by 16. These differ from the MTP/SGLang pre-convolution raw auxiliary taps. Storage index 42 is the separate native final state, not a forty-third backbone block. Learned multi-layer fusion is an optional, unselected extension; neither fusion nor pre-averaging is required by this proposal.

Keeping earlier rows is an information-retention hypothesis. It can make a prior constraint or correction directly accessible, but can also retain obsolete or distracting information. A longer row list is not automatically better. The existing capture regime, source context and timing determine what a row actually represents; one vector is not necessarily one isolated fact.

Source order, event type and the time information became available must be explicit. A plain cross-attention operation is invariant to jointly permuting its key/value rows. It cannot recover chronological order from list order alone. The donor states may already contain contextual information about order, but that does not supply a separate, guaranteed recency signal.

Actual tool-result text

Retain the literal returned text and its field structure, call/result identity and status. Tokenize with PersonaPlex's tokenizer, not Inkling token IDs. Produce accessible representations for individual tokens so exact names, dates, amounts and identifiers need not rely solely on a compressed hidden-state representation.

This is a new learned encoding route. PersonaPlex's existing text embeddings may be reused where appropriate, but embeddings alone are not a demonstrated contextual tool-text encoder. The representation must distinguish, for example, an original charge, a pending credit and an applied credit. Exact token/type/position encodings and the mapping to the 4,096-wide attention-source interface remain to be selected and trained.

The tool-text route does not force the agent's speech-text stream and does not insert another pretrained text-LM mouth. It supplies information from which PersonaPlex chooses its speech. Access to literal tokens may help copying, but it does not guarantee correct copying or truthful interpretation.

If state rows and token rows are combined into one attention operation, their scales, type information and relative abundance matter. A long tool response contributes many more candidate rows than a short state list. Separate attention normalization is a retained alternative, not something already implemented or included in the known parameter count.

2. One module, repeated with independent parameters at four sites

For site i, let h_i,t ∈ R^(1×4096) be the PersonaPlex state immediately before its selected native block, and let S_t ∈ R^(M_t×4096) be the supplied source matrix. Batch dimension is omitted. This input boundary applies no learned operation and does not specify an encoder or source-merging layer.

For a vector x of width d=4096:

RMSNorm(x; γ, ε) = γ ⊙ x / sqrt(mean_j(x_j²) + ε)

Normalize each vector separately, without subtracting its mean. The query and source gain arrays are independent; a source gain array is shared across rows. Gains are trainable. The exact positive epsilon remains an explicit configuration choice.

Using row-vector notation, with a single query position at each site:

h_bar_i,t = RMSNorm_q,i(h_i,t)             [1,4096]
S_bar_i,t = RMSNorm_s,i(S_t)               [M_t,4096]

Q_i,t = h_bar_i,t W_Q,i                   [1,2048]
K_i,t = S_bar_i,t W_K,i                   [M_t,2048]
V_i,t = S_bar_i,t W_V,i                   [M_t,2048]

W_Q,i, W_K,i, W_V,i: [4096,2048]
W_O,i: [2048,4096]
All four projections are bias-free.

Reshape Q to [16,1,128].
Reshape and transpose K and V to [16,M_t,128].

L_i[a,1,m] = sum_j Q_i[a,1,j] K_i[a,m,j] / sqrt(128) + A_t[1,m]
P_i = softmax(L_i, axis=M_t)              [16,1,M_t]
U_i[a,1,j] = sum_m P_i[a,1,m] V_i[a,m,j]  [16,1,128]

U_concat = concatenate heads(U_i)         [1,2048]
delta_i,t = U_concat W_O,i                [1,4096]
h_prime_i,t = h_i,t + tanh(g_i) delta_i,t  [1,4096]

The source RMS divisor is computed separately for each row over its 4,096 coordinates. One source gain vector is shared across rows at a site and used for both K and V. Query and source gains are separate. There is no mean subtraction, additive normalization bias or normalization across the source-row axis.

The supplied mask row A_t ∈ R^(1×M_t) is zero for an allowed row and negative infinity for an excluded row. If S_t already contains only allowed rows, the mask row is all zero. If no row is allowed, define delta_i,t=0; do not evaluate softmax over an all-negative-infinity vector. Input filtering and mask generation are not learned layers in this diagram.

The residual is added to the original h_i,t, not its normalized version. Each later site's query is computed from a PersonaPlex state that has already passed through the earlier connectors and intervening native blocks. All sites read the same causally fixed source version for that frame, but their independent projections can use it differently.

g_i is a learned, global scalar parameter, not a runtime confidence score or a per-token decision. tanh(g_i) can be negative. At initialization, all four gates are zero. With finite branch outputs, the new residuals are initially absent. Only the gate can initially receive a gradient through a closed branch; that gradient is not guaranteed nonzero. Projection and normalization gradients appear as the gate moves away from zero. Do not also initialize the output projection to zero and block the initial gate signal.

With exactly one allowed source row, each head's softmax is 1: Q/K cannot change the weighted value. This special case remains true with four sites. With no allowed rows, the defined correction is zero rather than softmax over an entirely masked set.

The gated residual pattern is supported by OpenFlamingo's implementation. Its use here, width, number of sites and effect on PersonaPlex remain new hypotheses. No result in the cited code demonstrates this particular combined model.

3. What four sites change

The native temporal blocks are state-dependent operations. Each block has its own K/V ring cache and position offset; current K/V are appended and RoPE/causal masking use that offset. A drawn unary block such as F_b(h) holds those state arguments fixed for notation. It is not a memoryless function; its output also updates the block’s cache and offset.

The first connector operates after four native temporal blocks, so its contribution can affect 28 subsequent blocks. The other sites provide additional opportunities after blocks 11, 19 and 27. The final connector still leaves four native blocks, final normalization and text/audio readouts to integrate its contribution.

This may let different stages use task information for different purposes, such as maintaining a constraint, selecting a relevant billing item or preparing exact wording. Those are predictions to test, not assigned semantic jobs for individual layers or heads. Four connectors may also be redundant, harder to optimize or more disruptive than one. Retain the single-connector comparison on matched information and evaluation when making an architectural claim.

No connector adds a temporal position or audio frame. That avoids changing the native frame indexing, but the four attention computations still take time. This document does not claim faster token generation, lower latency, or that Inkling produces states faster than PersonaPlex consumes them.

4. A continuous explanation, not a sequence of event announcements

Consider an illustrative bill discussion. The caller reports a £85 bill instead of the usual £60. PersonaPlex can explain how to compare the bill while Inkling retrieves the actual itemization. It need not repeatedly say that it is checking something, and useful silence remains appropriate.

The result identifies £60 base service, £10 roaming and a £15 one-time fee. PersonaPlex can continue its explanation using those amounts. A later verified result says the £10 roaming charge has been credited: the original bill was £85, but the current amount due is £75. The next eligible speech decisions should preserve both facts and explain the updated balance.

This is a desired behavior example, not an actual tool record, model output or benchmark. No literal phrase is prescribed by the connector. Its point is to support coherent explanation that evolves with reasoning and results, rather than a fixed response for every event type.

Frame-level availability

1. At the start of a PersonaPlex frame computation, select the latest fully delivered version of allowed state and tool information.
2. Hold that version fixed through all four attention sites, temporal blocks, text selection and depth decoding.
3. A result arriving during the computation becomes eligible at the next frame decision. Its wall-clock delay depends on the actual runtime; an 80 ms frame period is not a response-latency promise.
4. Retain native delayed-code alignment. Current predictions, aligned output frames and device playback are different events.
5. Do not assume already generated or queued audio will absorb a new result. Unplayed obsolete audio needs cancellation/replacement if appropriate. Already played words require a later clarification or correction.

The released PersonaPlex server does not implement this state/text input or per-frame version selection. Its native websocket accepts audio; its startup role prompt is not a live tool-information interface. Server · frame alignment

5. Training plan and exact frozen/trainable distinction

Reference: unmodified PersonaPlex

Retain the native checkpoint, voice/role prompts, codec, delay alignment and sampling settings. Check the augmented computation with all gates closed before adaptation. Measure actual grounded speech, listening, interruption and delivery on the intended calls. A closed gate only preserves the original model before its base weights change.

Warmup: new modules, PersonaPlex frozen

Train the four attention modules and selected new source/text encoders. Include layer-fusion parameters only if that optional extension is selected. Keep both existing base models, existing embeddings, depth model and Mimi frozen. Native forward/backward computation must still transmit gradients from text/audio loss to the new modules; using an inference no_grad path would prevent that.

QA about current facts, unknown information and corrections can help align the inputs. A native PersonaPlex question still needs an actual supported input, such as recorded caller audio; answer-text loss does not create a new native text-question interface. Timed speech training additionally needs aligned agent audio and text. Existing MTP captures can support compatible fixed-prefix examples, but do not constitute this complete audio dataset.

“Brief warmup” means an initial alignment stage, not an established number of updates or a promise that small-data QA will suffice. It must show use of the supplied information without treating question templates or future answer text as a shortcut.

Joint training: full temporal model and text head

Jointly train the new modules and all 32 existing temporal blocks, final output normalization and text head. This candidate specifies full temporal fine-tuning, not an implicit substitution of LoRA. Initially keep existing input embeddings, the depth model, Mimi and existing voice-prompt embedding computation fixed.

Freezing those components does not preserve voice, pronunciation, prosody or turn-taking automatically: their inputs change when the temporal model changes. Also, after full temporal fine-tuning, setting all four connector gates to zero does not recover the original pretrained model. Retain a separate original checkpoint/reference.

Training must include much broader, correctly timed speech than the existing captures alone: long multi-turn conversations, overlap, interruptions, quiet periods, evolving explanations during tool waits, exact values, obsolete results, corrections and varied domains. Include cases without new tool information so the model retains ordinary conversation. Do not replace this with a collection of isolated event announcements or only short QA answers.

Current-model conversations and feedback

Collect/evaluate valid continuations using the adapted model's actual speech. Feed confirmed played speech back to Inkling at the correct input boundary. If that feedback changes Inkling's history, its later hidden states must be recomputed for the new history even though Inkling's weights remain frozen. Old reference-history captures cannot be relabelled as current-model continuations.

Depth adaptation remains conditional on an observed limitation in audio generation after checking alignment and inputs. It is not part of the initial joint stage. Keep Mimi fixed initially. An inability to read task information is not by itself evidence that the audio decoder needs training.

The native LM provides delayed-code training and text/audio outputs, and upstream Moshi-Finetune provides useful training patterns. The NVIDIA fork still needs an exact integration: the upstream wrapper expects a different model-loading/LoRA interface. Existing support for ordinary Moshi training is not proof that this augmented PersonaPlex trainer works. Native train path · upstream wrapper

6. Known parameter count and illustrative costs

For each specified connector, with d=4096 and a=2048:

four bias-free projections: 4 × d × a = 33,554,432
two RMSNorm gain arrays:     2 × d     =      8,192
one scalar gate:                          =          1
per connector:                          = 33,562,625
four independent connectors:            = 134,250,500 parameters

This is 134.25M new connector parameters only. It excludes any optional layer fusion, tool-text encoders, type/position representations, any extra source-specific attention branches, and all existing base parameters that become trainable. It must not be presented as the full trainable-model size.

The selected audio return adds no newly trained reverse adapter. It reuses Inkling's existing 1280×4096 audio table and 4096 audio RMSNorm gains: 5,246,976 existing frozen parameters. They are excluded from the 134,250,500 new forward-connector parameters. Resampling, framing, FFT, mel filtering, quantization and placeholder replacement add no learned parameters. These counts do not settle the full training count or establish that native assistant-audio conditioning works.

Illustrative storage arithmetic for the four connectors:

• BF16 weights: 268,501,000 bytes, approximately 268.50 MB.
• A 16-byte-per-parameter mixed-precision Adam example: 2,148,008,000 bytes, approximately 2.15 GB, for weights/gradients/master weights/two moments under that specific precision choice. This excludes the full temporal-model training state, activations, codec, scratch and runtime buffers.
• If all four sites read the same M compatible rows, their separate BF16 K/V arrays require 4×2×M×2048×2 = 32768M bytes. For an illustrative M=128, that is 4 MiB. The raw 4096-wide BF16 rows add 1 MiB. Retaining multiple source layers, token encodings or concurrent versions adds more. 128 is not a selected retention limit.

At fixed inference weights, unchanged source rows can reuse projected K/V. Training must retain differentiable projections and recompute them after parameter updates.

With source K/V already projected, the four connectors' matrix-multiply arithmetic is approximately 16da + 16Ma FLOPs per frame. At illustrative M=128, this is 138,412,032 FLOPs/frame. Computing new source K/V projections costs additional work. This is arithmetic, not a wall-time, throughput, real-time feasibility or speedup measurement; transfers, synchronization and kernel execution matter.

7. Comparisons that answer the actual question

Evaluate complete spoken interactions with correct, absent and mismatched state information, including same-question/different-result pairs. Compare state-only, tool-text-only and combined information when attributing benefits to the new input route. Keep source availability and difficulty matched; do not choose easier calls for the larger model.

Report useful grounded speech, exact entities/amounts, appropriate uncertainty, adaptation to corrections, natural overlap/pausing, later task consistency and actual played speech. Measure source readiness, delivery, model use, first useful audio, queued-audio replacement and device playback separately. More parameters, an attention plot, a lower QA loss or a live process is not evidence that the speech behavior improved.

The extra sites, width, retained history, text encoding and joint fine-tuning remain hypotheses. The architecture is proposed, unimplemented and untrained.

8. Played audio back into the full Inkling model

This candidate specifies five groups of operations: digital-output construction, continuous rate conversion and framing, native dMel extraction, native audio embedding, and a model-role input wrapper. The return capture, exact rendered-range acknowledgment and assistant-audio integration are proposed. The inspected frontend, embedding table, special-token IDs and placeholder-replacement operations are existing Inkling components. No end-to-end inference, self-feedback training or behavioral result is claimed.

8.1 Digital renderer samples: Opus, resampling, queue copies and fades

Let p̂_t ∈ R^1920 be one native Mimi output at 24 kHz. The reference server encodes output PCM with Opus. The browser decodes it at 24 kHz and resamples to its actual AudioContext rate f_d, before the worklet copies queue samples into output blocks and applies fades. Server output · decoder · worklet

For a realized sequence of these operations, write:

w = OpusDecode(OpusEncode(concat_t p̂_t))
v = R_(24000→f_d)(w; ξ_out)                    [N_queued]

P_play ∈ {0,1}^[N_rendered,N_queued]
P_play[n,c(n)] = 1 when output n copies queue sample c(n)
P_play[n,:] = 0 when output n is actually zero-filled
D_g = diag(g_0, …, g_(N_rendered−1))

r = D_g P_play v                              [N_rendered]

P_play and D_g describe the actual queue schedule and fade multiplications; they are not neural layers or instructions to allocate dense matrices. Dropped queue samples never appear in a selected column. Genuine underflow/rebuffer zeros and actual fade gains are part of the resulting digital waveform. Opus is lossy: decoded, queued and original Mimi samples are not assumed equal.

The chosen return captures r after the application's buffering and fades, as float samples at f_d. New integration must associate those samples with their output-generation identity and confirmed contiguous rendered ranges. The released worklet's aggregate counters do not supply this exact capture/acknowledgment route to Inkling. Unknown intervals are not an extra model-input mask and are not silently replaced by zeros. Actual captured zero-valued output is known silence and remains in the waveform.

Here “rendered” means samples supplied by the application to the AudioContext output. It does not measure later system/device gain, acoustic speaker output, intelligibility or what a person physically heard.

8.2 Continuous f_d→16 kHz conversion and 800-sample hops

For each confirmed contiguous sample interval [a_s,b_s), the proposed frontend consumes r_s = r[a_s:b_s]. Preserve resampler filter history and fractional phase, and preserve the native-hop FIFO across output/transport callbacks:

(x_s, ξ_next) = R_(f_d→16000)(r_s; ξ)
F ← concatenate(F, x_s)
while len(F) ≥ 800:
    emit F[:800]
    F ← F[800:]

The main source is already floating-point WebAudio output. If a transport uses PCM16 instead, its amplitude conversion is x=q/32768; passing integer samples directly to the native extractor only casts them to float. The repository's current native-context path instead uses soundfile.read(..., dtype="float32") on PCM WAV, which supplies this scaling. An in-memory five-value PCM check confirmed the exact conversion without loading a model.

An 80 ms model frame spans 1920 samples at 24 kHz and 1280 samples at 16 kHz: 1.6 native hops. Five continuous frames span 400 ms, or 6400 samples at 16 kHz, and therefore eight full hops. Independently preprocessing each 1280-sample clip would produce two dMel rows per clip and ten rows total, introducing five clip boundaries and repeated right padding. Transport callbacks must not define native clip boundaries. This duration arithmetic is not a guarantee about callback timing or resampler delay.

At an actual segment end, flush the selected resampler according to its implemented filter policy, then apply native right padding once to any final short native hop. Missing capture or acknowledgment intervals end a segment; the next known interval starts fresh resampler/overlap state. Inkling keeps its existing conversation history. Real underflow zeros already captured from a continuous renderer interval do not cause a reset. The exact resampling filter, group delay and final filter flush remain implementation choices; the notation does not claim a selected zero-delay or bit-identical resampler.

8.3 Native dMel: 100 ms window, 50 ms hop, 80 mel coordinates

For a completed segment x ∈ R^N at 16 kHz, the checkpoint's frontend uses H=800, L=N_FFT=1600, a periodic Hann window and no STFT centering. Let T=ceil(N/800):

x_pad = concatenate(zeros(800), x, zeros(800T−N))
w[n] = 0.5 − 0.5 cos(2πn/1600), n=0…1599

X = STFT(x_pad, n_fft=1600, win_length=1600,
         hop_length=800, window=w, center=False)       complex [801,T]
A = sqrt(max(real(X)^2 + imag(X)^2, 1e−10))            [801,T]
M = Slaney mel filters, 0…8000 Hz, Slaney norm         [80,801]
L_mel = clip(log10(max(M A, 1e−10)), −7, 2)            [80,T]

c_k = −7 + 0.6k, k=0…15
D[t,m] = argmin_k abs(L_mel[m,t] − c_k)                [T,80]

Each D[t,m] is an integer from 0 to 15. The native processor computes nearest-center distances in float64. The mel filterbank consumes magnitude, not squared power: the square root appears before M A. The first full window is 800 zeros followed by the first 800 samples; later windows contain the previous and current hops. A live frontend can emit each complete hop while keeping the final partial hop pending until the segment ends. The 80 mel coordinates produce one native trunk-input row per hop, not 80 trunk positions.

The existing CausalAudioEncoder helper retains 800 previous samples but also emits a row for every partial final hop of each push() call. Its demonstrated caller supplies 1600-sample events. That helper is not evidence that arbitrary repeated 80 ms pushes already preserve the chosen continuous 800-sample grid; the explicit FIFO above is part of this proposed integration.

8.4 Native audio lookup, sum and RMSNorm

The checkpoint has a distinct 16-entry table slice for each of 80 mel coordinates. With m=0…79:

I[t,m] = 16m + D[t,m]                                [T,80]
E_audio                                              [1280,4096]
G[t,m,:] = E_audio[I[t,m],:]                          [T,80,4096]
u[t,:] = sum_(m=0…79) G[t,m,:]                       [T,4096]

Z[t,:] = γ_audio ⊙ u[t,:]
         / sqrt(mean_j(u[t,j]^2) + 1e−6)              [T,4096]
γ_audio                                              [4096]

This is table lookup, summation over the 80 mel coordinates and rowwise RMSNorm. There is no mean over mel coordinates, concatenation, additional audio transformer, reverse projection or channel-embedding addition. The existing CPU tower computes the RMS arithmetic in FP32 and returns a BF16-rounded result. Its 5,246,976 existing parameters remain frozen here.

8.5 Proposed model-role wrapper and native placeholder replacement

This proposal chooses the following exact wrapper for each continuous self-audio segment:

b = [200001, 200020, repeat(200053,T), 200043, 200010]  [T+4]

200001 = <|message_model|>
200020 = <|content_audio_input|>
200053 = audio soft-token placeholder
200043 = <|audio_end|>
200010 = <|end_message|>

U[j,:] = embed_norm(E_token[b_j])                     [T+4,4096]
U[2:T+2,:] ← Z                                       replace placeholders

U enters I_0 → … → I_41 → native final normalization
with the existing causal positions, KV and convolution history.

The embedding normalization and placeholder replacement are the existing native multimodal operations. Audio rows already have their native audio normalization; they replace the normalized placeholder embeddings, rather than receiving token-embedding normalization again. The ordinary model-role and delimiter rows retain their token embeddings. Prefix rows are appended once, followed by successive available Z rows, then suffix rows once when the segment ends. No message wrapper is repeated per codec frame or network callback. No new audio validity-mask input or channel vector is supplied.

The assistant-role wrapper is a selected proposal, not verified native assistant-audio semantics. The IDs exist, but the inspected repository audio wrapper currently uses <|message_user|> (200000). The renderer's types exposing AuthorKind.Model and AudioPointer do not prove that their combination has the required native rendering, training or self-feedback behavior. This candidate intentionally uses the model author and must demonstrate that the incremental integration carries the intended self-audio meaning.

The full 42-block Inkling model remains in the loop. Its weights stay frozen, while its causal state incorporates newly appended audio. Later hidden states supplied to the four forward connectors must be computed from that updated history; prior captures from a different history cannot stand in for them. Closing an audio segment or resetting its resampler does not reset the thinker.

Source evidence for the native return operators

These local references were inspected on October 5, 2026. They identify source files, not publicly served code or a new runtime test. The existing Inkling evidence report covers the full trunk and capture boundaries; the exact audio sources inspected for this addition are listed here so the diagram does not link to unavailable private paths.

Sources and relationship to the native model

All native code links below are pinned. The native audit reconciles source-derived parameter counts with published checkpoint metadata; it did not read tensor values. The new modules and training plan are proposals, not part of these upstream sources.

Historical design note. The implemented Stage 1 architecture, current losses and evaluation are documented in Current sequence memory.

Connections to semantic outputs

Status: architecture proposal. No model implementation, training or inference result is established by this diagram. The existing random-connector inference experiments are separate. This view introduces a different location for the connection; publishing it changes no running experiment.

The goal is a combined agent with Inkling's intelligence and tool use and PersonaPlex's conversational speech. Both consume native user audio. PersonaPlex must be able to start speaking while Inkling is still computing. Speech, reasoning and tool use should eventually overlap. The current Inkling worker consumes audio only; it does not generate reasoning/answer tokens or execute tools.

The proposed comparison keeps all 42 Inkling blocks, all 32 PersonaPlex temporal blocks, the native speech-text head, the position-specific depth model and Mimi. One cross-attention module reads the final PersonaPlex state and the selected available Inkling states. Its shared result can enter the speech-text prediction, the first audio-code prediction, or both.

1. Complete path

live caller audio
  ├─ stateful 16 kHz resampling → native dMel/audio embedding
  │   → I0 → I1 → … → I41 → Final
  │      └──── each selected layer publishes independently ──→ latest rows
  │                                                           → S_t
  └─ stateful 24 kHz resampling → Mimi.encode → supplied user codes
      → native delayed text/audio cache → 17-stream embedding sum
      → PersonaPlex F0… F31 → final RMSNorm → h_t
                                              │
                     S_t ── K,V ─┐            │ Q
                                 └─ cross-attention → r_t
                                              │
                 ┌────────────────────────────┴─────────────────────┐
                 │                                                  │
     h_text = h_t + g_text r_t                          h_0 = h_t + g_0 r_t
                 │                                                  │
     native text head → selected y_t ────────────────┐              │
                 │                                   ↓              ↓
                 │                              u_0 = P0 h_0 + E_text[y_t]
                 │                                   │
                 │                            native depth at q=0 → c_0
                 │                                   │
                 │                  q=1…7: Pq h_t + E_audio,q−1[c_q−1]
                 │                                   │
                 │                  same six-layer depth module → c_1…c_7
                 │                                   │
                 └──────────── native delayed cache ←┘
                                     │
                              output realignment
                                     │
                       8 agent codes → Mimi.decode → PCM → playback

The products in this overview use conventional column-vector shorthand. The detailed equations below use batched row vectors and PyTorch [output,input] weight shapes explicitly.

There is no PersonaPlex-output audio return to Inkling in this proposal. PersonaPlex's own generated text and audio codes remain in its native autoregressive history. Those two paths must not be confused.

2. Native input and state definitions

PersonaPlex operates on 24 kHz PCM, with 1,920 samples per 80 ms codec frame. Its eight supplied user codebooks enter the existing delayed-token ring. The temporal input is the sum of one speech-text embedding and sixteen audio-stream embeddings: eight for the agent and eight for the user. The native delay schedule and ring cursor determine which IDs are read; the drawing does not replace that schedule with simultaneous undelayed tokens.

Inkling receives 16 kHz audio. Its native frontend uses 1,600-sample analysis windows with 800-sample hops, producing 80 dMel indices per row. The native audio embedding turns those indices into 4,096-wide rows. All native attention and short-convolution state remains inside Inkling.

S0…S41 refer to completed block outputs, after the MLP-output short convolution and residual addition. Final is the native final RMSNorm result before the readout division by 16. These differ from the pre-convolution auxiliary taps used by other MTP experiments.

Independent startup and continuous audio

Start PersonaPlex's native voice/role initialization and Inkling's native prefix prefill independently. PersonaPlex becomes ready when its own initialization finishes; it does not wait for Inkling readiness. Every arriving microphone block enters both input paths. Audio arriving while Inkling initializes remains queued for processing.

Inkling continuously consumes audio with its native cache. It does not wait for an utterance or a turn to end. All 42 blocks still execute for each input update; publishing an early state does not skip deeper computation.

Publish each selected layer independently

When a selected block output completes, transfer its latest input-position row while later Inkling layers continue computing. Final is published after Inkling's final normalization. A row is available only after computation and transfer have completed. The design must avoid synchronous GPU-to-host copies in the layer execution path; actual transfer overhead remains to be measured.

For example, with S3, S20 and Final selected, PersonaPlex first uses S3 when it arrives, then adds S20 and Final as they arrive. There is no wait for the complete selection. If only Final is selected, PersonaPlex still speaks independently until that state arrives.

Read one available snapshot per speech frame

Let f_t be the time PersonaPlex takes the source snapshot for frame t. Let arrival_l(n) be the time the row for layer l and input position n reaches PersonaPlex. L is the live set of selected layer IDs.

J_t = {l in L with at least one row received by f_t}
n_l(t) = max {n : arrival_l(n) ≤ f_t}
S_t = stack(S_l(n_l(t)) for l in J_t, in layer order)
S_t: [B,M,4096]; M = |J_t|; 0 ≤ M ≤ |L| ≤ 43
M > 0 → r_t = cross-attention(h_t,S_t)
M = 0 → r_t = 0; continue native generation

Each selected layer supplies its latest available row. An early layer may describe newer audio than a deeper layer. The former requirement that all selected rows share one completed input position is removed. There is one latest row per layer, with no added multi-position history bank; Inkling retains its own native conversation cache.

Both output branches use the same snapshot throughout a frame. New source rows and live selection changes apply at the next frame boundary. An empty selected set or a set whose rows have not arrived adds no connector contribution; there is no attention calculation over an empty set.

What this changes in the existing implementation

The current playground already uses independent ongoing audio consumers. However, conversation startup waits for Inkling's ready message, and the Inkling worker publishes all captured states together after the full forward pass. Independent startup and per-layer publication are proposed serving changes. This diagram update does not implement them or change the current sweep.

3. Preserve the temporal computation

Let s be the native delayed-code cursor and t identify the current speech frame. The cursor includes startup processing and should not be interpreted directly as elapsed microphone time.

x_t = Σ_(k=0…16) E_k[C[k,(s−1) mod L]]
z_t = F31(…F0(x_t, KV0)…, KV31)
h_t = RMSNorm_final(z_t): [B,1,4096]

Each temporal block retains its native two pre-RMSNorm residual sublayers: 32-head causal attention with 128 features per head and RoPE, followed by a SiLU-gated feed-forward network with effective hidden width 11,264. The native temporal attention span remains a model setting; no new frame positions are inserted by this connection.

This final normalization belongs to PersonaPlex, so it creates no dependency on Inkling’s Final state. The new module comes after the final native normalization, not before block 16 or at four intermediate blocks. There is no direct Inkling addition inside F0…F31 in this proposal. The final native norm uses FP32 arithmetic and epsilon 1e-8.

4. One shared attention result

The following geometry is an explicit reference design, not an optimum measured by the running sweep:

All weight matrices above use PyTorch's [output,input] convention. The reference initialization is bias-free Xavier-uniform matrices, RMSNorm gains of one and bridge RMSNorm epsilon 1e-6. A concrete seed is to be selected when implementing a comparison; this publication does not create random weights.

n_q = RMSNorm_q(h_t)
n_s = RMSNorm_s(S_t)
Q = n_q W_Qᵀ: [B,1,512]
K = n_s W_Kᵀ: [B,M,512]
V = n_s W_Vᵀ: [B,M,512]

Q → [B,8,1,64]
K,V → [B,8,M,64]
P = softmax_M(QKᵀ / sqrt(64)): [B,8,1,M]
U = PV: [B,8,1,64]
r_t = concat_heads(U) W_Oᵀ: [B,1,4096]

These attention equations apply when M > 0. Only available selected rows enter S_t, so no additional availability mask is needed. When M = 0, define r_t = 0 without evaluating attention. No extra positional encoding or learned state-selection network is silently included. With one selected source row, softmax has one entry equal to one: the value still influences the output, but the query cannot choose among source rows.

r_t is computed once and shared by both consumers. It is not two independently initialized attention modules.

5. Text branch

h_text = h_t + g_text r_t
logits_text = h_text W_textᵀ: [B,1,32000]
W_text: [32000,4096]
y_t = native text sampling(logits_text)

The existing text projection is frozen. The selected token is PersonaPlex's own internal speech-text prediction. No transcript, completed Inkling answer, name filter or forced word is supplied to this head.

The selected text token has two native effects: it conditions the current frame's depth computation and it enters the delayed input history for later frames. Modifying this branch can therefore affect audio and future temporal states even when g_0=0.

There is no additional application of the native final RMSNorm after the addition in this specified variant. Adding one would be a different experiment.

6. First audio-code branch

h_0 = h_t + g_0 r_t
u_0 = h_0 P_0ᵀ + E_depth_text[y_t]
P_0: [1024,4096]
E_depth_text: [32001,1024]
u_0: [B,1,1024]

d_0 = D5,0(…D0,0(u_0)…)
logits_0 = d_0 W_out,0ᵀ: [B,1,2048]
c_0 = native audio sampling(logits_0)

The first Mimi codebook has semantic supervision distilled from WavLM. Kyutai's explainer demonstrates that fixing these tokens can preserve the spoken words while changing the reconstructed voice. This motivates a separate connection into the first-code prediction.

It does not establish that c_0 is pure meaning, that other codebooks carry no content, or that changing c_0 preserves timbre, pronunciation, timing or intelligibility.

7. Remaining audio-code inputs

For the active agent code positions q=1…7:

u_q = h_t P_qᵀ + E_depth_audio,q−1[c_q−1]
P_q: [1024,4096]
E_depth_audio,q−1: [2049,1024]
u_q: [B,1,1024]

d_q = D5,q(…D0,q(u_q)…)
logits_q = d_q W_out,qᵀ: [B,1,2048]
c_q = native audio sampling(logits_q)

There is no direct r_t term here. These positions still depend on earlier code selections and the within-frame depth history, including the text-conditioned position zero. The diagram therefore does not claim an independent, unaffected acoustic channel.

The same six-layer depth module runs sequentially for each code position. It uses position-specific attention and feed-forward parameters, width 1,024, 16 attention heads of width 64, no RoPE, and a SiLU-gated feed-forward hidden width of 2,816. Its streaming history is reset for each new audio frame.

The released checkpoint stores depth parameters for sixteen positions. This view chooses eight generated agent positions, as in the current inference playground; user codes are supplied from Mimi instead of generating the unused user positions. All eight agent codes are realigned through the native delay schedule before Mimi decoding.

8. Mimi and playback

aligned agent codes [B,8,1]
→ split quantizer reconstruction: semantic + acoustic residual embeddings
→ learned 12.5→25 Hz upsampling
→ eight native codec Transformer blocks
→ SEANet waveform decoder
→ 24000 Hz PCM [B,1,1920]
→ existing transport, device conversion and playback

The Mimi encoder has the corresponding native SEANet, Transformer, learned downsampling and quantizer path. Both codec directions are expandable in the scene. Generated, queued and physically played audio are distinct measurements. No latency or quality result follows from this graph.

9. The four comparison settings

The coefficients are direct manual multipliers. No tanh or square-root-of-connector-count scaling is implied here. Their values are not a percentage of Inkling intelligence. The initial comparison keeps the two nonzero coefficients equal; independent values are a later possible comparison, not another module in this diagram.

Use the same complete audio, prompt, voice, base-model weights and generation seed when comparing routes. All weights remain fixed. Garbled, muffled, fragmented or repetitive output remains eligible for investigation; fluency is not a prerequisite for a useful untrained starting state. Report name evidence, input dependence and timing separately. An empty ASR result is not proof of silence or of audible name absence.

10. Response timing

PersonaPlex's generation does not wait for a new Inkling state, a selected final layer, or completion of Inkling's current audio update. It starts with its own pretrained behavior and accepts available Inkling influence on subsequent output. An early invented identity can therefore still occur. Already generated speech cannot be revised by a later state.

The 80 ms codec frame describes audio duration, not total response latency. Native initialization, inference, output alignment, Mimi, network transport and device playback all contribute to the time to first audible response. Connector computation and state transfers add work. Preserving native response latency requires an end-to-end measurement; early state publication alone does not establish it. Early layers are also not guaranteed to contain the conclusions of the full model.

11. Scope and open questions

• This proposal does not modify the running 1/4/32 temporal-connector experiments.
• Native user audio reaches both models; no transcript context box or VAP is added.
• No agent audio is fed back into Inkling.
• The current source is audio-processing state. Actual reasoning and tool-result states are a future extension and must obey availability ordering.
• Width 512, one shared attention result and the exact output-branch locations are explicit starting choices, not measured superior architecture choices.
• No guarantee is made that the two models' equal feature width implies compatible semantics.
• Whether conditioning the text head, first code, or both offers a useful trajectory remains an empirical question.

Sources

• Pinned NVIDIA model code: temporal state, text head, depth inputs and sampling
• Pinned NVIDIA streaming Transformer
• Pinned model and Mimi construction
• Mimi's semantic-token experiment
• Released Inkling Small checkpoint
• Existing native model audit
• Existing Mimi audit

Current sequence memory: one writer, four readers

Implemented Stage 1 architecture, updated 4 November 2026. This is the architecture used by the full training run, pinned to TalkingComputers/inkling-personaplex at b52677a29409cac860ceb11bb57748117ae07413. The inspected current checkout has the same writer, readers, memory construction, PersonaPlex wrapper and training objective as this training revision. Generation details below reflect the fixed 20-second evaluation path introduced in 33b5b10.

The task is grounded speech while Inkling continues work. Inkling supplies a causal memory of its task conversation, thinking and tool results. PersonaPlex hears the caller and generates its own text and speech. Stage 1 learns only the interface between them: one memory writer and four cross-attention readers. Inkling, the native PersonaPlex temporal/depth models, their text embeddings/output heads, and Mimi remain frozen.

The other connector tabs preserve historical designs. This current design has three selected rows per token, a 40,000-token window, learned metadata buckets and four temporal readers. It has no fifth end connector, no separate text/first-code update branch, no rotary time encoding in the memory readers, and no podcast-copy objective.

1. Exact configuration

2. Actual computation graph

Recorded task inputs, media and tool results
    ↓ native Inkling embeddings
Frozen Inkling: I0 … I41 → final RMSNorm
    ↓ per-token rows 28, 34, 42 + kind + bytes + availability
Resolve cached state owners and shared conversation prefixes
    ↓
Shared learned writer
    ├─ per-row LayerNorm → concat 12,288 → Linear 1024
    ├─ whole-word PersonaPlex embeddings → mean → Linear 1024
    ├─ token-kind embedding
    ├─ log2 token-recency embedding
    └─ log2 elapsed-seconds embedding
    ↓ affine LayerNorm
Rolling bank: ≤40,000 live slots × 1024
    ↓ four independent K/V readers, same availability mask

Voice/role prefix + caller codes + previous agent text/audio
    ↓ sum 17 delayed stream embeddings
Frozen T0…T7    → Reader 1 residual
Frozen T8…T15   → Reader 2 residual
Frozen T16…T23  → Reader 3 residual
Frozen T24…T31  → Reader 4 residual
    ↓ native final RMSNorm
    ├─ native text head → 32,000 logits
    └─ native per-codebook projections + selected text/previous code
          ↓ 6-layer native depth transformer at q0…q15
          ↓ 2048 logits per position
          ↓ native code selection and delay alignment
          ↓ eight agent codes → frozen Mimi → 24 kHz PCM

The supervised optimizer does not rerun Inkling or Mimi. It reads stored Inkling state rows and pre-encoded call codes. Gradients pass through the frozen PersonaPlex computations to the writer/readers; frozen is not equivalent to detaching the entire forward pass.

3. Which information reaches the memory

The recorded task conversation includes setup, user input, thinking, tool calls, replies, tool results and other token kinds. For each computed token, the recorder stores state rows and availability. Cached prefixes refer to the request that originally computed their states. Session construction removes duplicated shared prefixes and orders newly added positions by computation time, then call index and token position.

The 40,000-token window is defined in arrival order. A memory is laid out by call then token position for state access and word construction. Attention does not receive that storage position as an absolute positional embedding.

For an audio frame at session time f_t, only tokens with arrival_n ≤ f_t are eligible. Prefill may make many token positions available together. The current recorder publishes chosen rows from a completed forward result; the earlier claim that each individual layer is delivered as soon as that layer finishes is not this implementation.

4. Shared memory writer

For token n, let H_n contain the three captured 4096-wide rows. Each row is promoted to FP32 and normalized independently with functional LayerNorm, without learned affine scale/bias:

Ĥ_n,l = (H_n,l − mean(H_n,l)) / sqrt(var(H_n,l) + 1e−5)
u_n = W_H concat(Ĥ_n,28, Ĥ_n,34, Ĥ_n,42) + b_H
W_H: [1024,12288]; b_H: [1024]

The fixed concatenation order identifies the three source depths. There is no separate learned layer-ID embedding and no 43-layer attention axis.

Causal lexical feature

Inkling token bytes are assembled into whitespace-delimited words within each call. Special tokens act as boundaries; audio/image placeholders carry no printable bytes. A word is complete only after the following whitespace token, or the call’s final token, has arrived.

The complete word is tokenized with PersonaPlex’s SentencePiece tokenizer. Each Inkling token receives the mean of the frozen 4096-wide PersonaPlex embeddings for pieces whose byte spans overlap that token. This is a content feature from the available work trace, not the target reply.

e_n = sum(overlapping PersonaPlex piece embeddings) / max(1, piece_count)
lexical_n(t) = W_text e_n + b_text   if the word is whole and its first token remains
               b_text             otherwise
W_text: [1024,4096]

The “otherwise” branch retains the projection bias. It is not an all-zero projected feature.

Kind, recency and age

The seven token kinds are setup, user, thinking, tool_call, reply, tool_result and other. Each has a learned 1024-wide vector.

The newest token has tokens_since=1. Its recency bucket is zero; ranks 2–3 share bucket one, ranks 4–7 bucket two, and so on. Elapsed time has boundaries at 1, 3, 7, 15, … seconds. Time can advance while a tool runs even if no new token arrives.

b_token = min(16, floor(log2(tokens_since)))
b_age   = min(14, floor(log2(seconds_since + 1)))

m_n(t) = LayerNorm1024(
    u_n + lexical_n(t) + E_kind[k_n]
        + E_recency[b_token] + E_age[b_age]
)

The writer’s affine LayerNorm also uses epsilon 1e−5. There is no memory RoPE, continuous sinusoidal time transform or absolute-position table. PersonaPlex’s native temporal RoPE is a separate unchanged mechanism.

5. Versions and causal visibility

A token’s representation changes at arrival, recency-bucket transitions, elapsed-time transitions, word completion, loss of its word’s first token, and eviction. The training representation stores each distinct version once, with a half-open interval of speech steps where it is valid.

seen[b,t,v] = own_call(b,v) AND start_step_v ≤ t < end_step_v

Padding steps see no versions. At most one version per retained token is visible at a frame, even when the union of versions across an entire call is much larger than 40,000.

The exact chunk planner bounds the union of relevant versions to 131,072 per chunk unless a single step alone needs more. K/V projections are formed for that chunk’s columns. Versions unseen by every step in the chunk can be omitted without changing any attention result. Activation checkpointing recomputes these chunks in backward. Free generation selects only the current step through active_steps.

An empty-memory step adds zero, including after the output bias has trained. For a fully masked attention row, the implementation temporarily admits all selected columns and then explicitly zeros the projected result. The zeroing, not the temporary mask, defines the empty-memory behavior.

6. Four independent readers

Each reader is attached by a forward hook after one native temporal block: 7, 15, 23 or 31. The fourth acts before final RMSNorm.

x = LayerNorm4096(float(z))
Q = x W_Qᵀ + b_Q
K = M W_Kᵀ + b_K
V = M W_Vᵀ + b_V

Q: [B,32,T,128]
K,V: [B,32,V_chunk,128]

A = softmax_v(QKᵀ / sqrt(128) + visibility_mask)
r = Linear4096→4096(concat_heads(A V))
r = 0 where memory is empty
z_out = z + cast(r, dtype(z))

The readers have separate Q/K/V/output weights and separate query LayerNorms. Only the written memory is shared. Attention dropout is zero. There is no scalar gate or reader MLP.

Output weights and biases initialize to zero. This preserves PersonaPlex exactly at initialization. On the first update, the output projection can learn, while its zero matrix initially blocks gradients into the reader’s Q/K/V and upstream writer. Those gradients become available as the output matrix changes.

The logged read_strength is the mean residual norm divided by mean native-state norm over real steps. It measures activation magnitude, not grounding or speech quality.

7. Exact trainable parameter count

These counts were verified by instantiating the actual writer/readers on PyTorch’s meta device, excluding the frozen PersonaPlex embedding.

At the full 40,000-token window, raw state payload is 937.5 MiB in BF16 and one FP32 slot bank is 156.25 MiB. These are tensor arithmetic sizes, excluding version unions, K/V expansion, optimizer state, gradients, native-model caches and allocator overhead.

8. Native PersonaPlex streams and output path

The temporal input sums seventeen embeddings: agent text, eight agent audio streams and eight caller audio streams. The model’s 3000-frame context corresponds to 240 seconds, including the voice and role prefix.

After reader four, the original final RMSNorm supplies h_t to the frozen 32,000-class text head and every depth input projection. There is no separate learned end connector.

At depth position zero:

d_0 = P_0 h_t + E_depth_text[y_t]

At subsequent positions:

d_q = P_q h_t + E_depth,q−1[c_(q−1)], q=1…15

The native depth transformer has six layers, width 1024, sixteen attention heads of width 64, a 2816-wide SiLU-gated branch (configured dim_feedforward is 4224, then the gating constructor uses 2×4224/3), codebook-specific weights and projections, and full causal depth history within the frame. LMModel overrides the loader’s nominal depth context with context=None; the 16 per-step weight sets determine a 16-position streaming cache. It has no depth positional embedding. The 16 depth positions represent agent q0–7 followed by caller q8–15. The semantic codebooks are q0 and q8.

During training all sixteen positions are predicted. During generation actual caller tokens are supplied and replace their sampled candidates. Only the eight agent codebooks are aligned and decoded into output speech.

Text delay is zero. Both voices have delays [0,1,1,1,1,1,1,1]. A delay of one is one 80 ms codec frame. Native output alignment undoes the schedule before Mimi decoding. This is unrelated to the retired 640 ms podcast-reconstruction target delay.

9. Stage 1 calls, prefix and text timing

Each training example is a whole timed call. Its stereo audio uses agent on the left and caller on the right. Mimi encodes each side to eight code streams. Agent speech word alignments supply stream zero.

The prefix is the selected voice example, half a second of silence, role text one piece per frame, and another half-second of silence. The caller side of the prefix is a steady tone. Prefix frames have zero loss.

The text interleaver puts a word’s first piece on its onset frame, emits remaining pieces on subsequent frames, and changes the preceding PAD frame into end-of-padding. PAD is ID 3; end-of-padding is ID 0. A new word can replace remaining pending pieces, matching the copied Kyutai interleaver. First-piece and word-piece masks describe what was actually placed.

Every frame uses:

session_time(t) = recording_starts_at + max(t − prefix_frames, 0) / 12.5

Prefix frames therefore read the memory at the recording start. Later frames see work as it arrives. The new QA caller is input to PersonaPlex; the recorded Inkling task trace is not rerun with that QA question.

10. Exact supervised objective

There are two independently normalized weighted means:

L = sum_update(w_text × CE_text) / sum_update(w_text)
  + sum_update(w_voice × CE_voice) / sum_update(w_voice)

The denominator is global over all valid weighted frames in the optimizer update, not an equal average of per-call means and not a sum of separate per-codebook means. DDP contributes scaled numerator sums against those known global totals. No extra generic next-token shift is applied to PersonaPlex’s already aligned training logits.

Text and full-voice loss update only the writer/readers. The current settings are four passes, 32 calls per update, one call per device forward, AdamW with weight decay 0.01, maximum learning rate 2e−5, a OneCycle schedule with 5% warmup and gradient norm clipping at 1. The inspected run uses the configured global batch across 32 H100 processes. Native PersonaPlex weights are BF16; added writer/readers use FP32 operations and cast residuals back to the native activation dtype.

11. Checkpoint selection and interpretation

The deciding teacher-forced metric is loss on the first piece of each word. Later pieces, padding, agent semantic audio, agent acoustic audio and caller audio are reported separately.

Periodic checks cover 48 validation tasks. They are not an evaluation of every held-out call. The chosen checkpoint minimizes first-piece loss among those passing both agent-voice conditions:

lower 95% bound of (baseline loss − checkpoint loss) ≥ −0.01 nats

The margin is 0.01 absolute nats, independently for agent semantic q0 and agent acoustic q1–7. It is not 1% of baseline. A flat “voice kept” value of one means the boolean condition passed; it does not mean voice loss is flat.

The run’s split file freezes training, validation and test membership. Repository grouping keeps related tasks together. Correct-memory versus wrong-task-memory comparisons assess memory dependence. A growing difference can reflect deterioration under wrong memory; it does not by itself mean correct-memory answers improved.

12. Implemented sampled generation

The current generator replays the native prefix and caller opening only until before the scripted response, including its end-of-padding cue. From the first free frame, neither target reply words nor target agent audio are supplied.

The response window is fixed at 20 seconds for baseline and trained checkpoints. Recorded caller codes continue, followed by silence when the recording ends. Memory grows at each frame’s session time. Agent text and agent voice are generated and become the model’s own history.

Sampling uses text temperature 0.7 with top-k 25 and native audio temperature 0.8 with top-k 250. Token timestamps account for the native generator’s maximum delay. Evaluation uses the same caller, prefix, arrivals, sampling settings, seed and window for the baseline and trained checkpoint.

The audio file is stereo: agent left, caller right. It omits the voice/role prefix. Groundedness review compares each claim with evidence available at the time it was said. Silence, repetition, clarity, completeness and claims are separate observations from teacher-forced loss.

This is causal offline replay. A fully integrated live task/caller service, training on model-generated histories, interruption learning and mouth-to-brain feedback are later stages. Joint PersonaPlex/Inkling training is not the current plan.

13. Source map

• Saved rows, token kinds and cached ownership
• Recording proxy and native task input
• Memory window, words, arrivals and versions
• Writer, readers, masks, hooks and chunking
• PersonaPlex loading, sites and delays
• Call audio, prompt prefix and code streams
• Text timing and word-piece masks
• Training objective, reductions and selection
• Fixed-window sampled generation

Spatial placement, small weight grids, animated flow and illustrated memory slots are explanatory geometry. They are not measured activations, literal counts of drawn cells or performance measurements.

Historical design note. The implemented Stage 1 architecture, current losses and evaluation are documented in Current sequence memory.

PersonaPlex as an Inkling-connected speech model: training and evaluation

Research snapshot: October 4, 2026. This document proposes an experimental variant for the Atlas. The private Inkling connector, its training, and its measured behavior are not implemented or demonstrated by this document. Existing model results and the current study remain separate. The intended outcome follows the project capability goal (PROJECT_GOAL.md, repository reference): useful conversation while reasoning and tools continue, correct handling of changing information, and memory of what the user actually heard.

Recommendation

Start from the released nvidia/personaplex-7b-v1, retain frozen Inkling-Small as the reasoning model, and add a private, causal input for its state and execution evidence. Preserve PersonaPlex's existing user-audio, agent-text, agent-audio and Mimi paths. Use the unmodified release as the speech-behavior reference; retain Qwen plus streaming TTS as a separate content/engineering reference.

Use a short connector-only alignment phase, followed by connector plus narrowly scoped Temporal Transformer LoRA. Initially freeze Mimi, the audio Depth Transformer, token embeddings and readout weights. Train with matched text and audio supervision; text-only QA is an auxiliary alignment task, not proof that the adapted model can speak well. Expand the trainable scope only if a measured failure calls for it.

This is a reasoned starting design, not a training recipe already validated for Inkling-to-PersonaPlex. In particular, freezing the audio modules does not freeze the generated voice or timing: their inputs change when the temporal model or connector changes.

Frozen means that model weights are not updated. LoRA, or low-rank adaptation, trains small additional parameter matrices while keeping the original weights fixed. QA means question answering; CE means cross-entropy loss; TTS means text-to-speech synthesis.

What the released system and source establish

PersonaPlex is a Moshi-derived model with user audio, agent text and agent audio streams. Its Hybrid System Prompt supplies role text and voice audio before dialogue. In the released implementation, text prompting forces text tokens while supplying silent agent audio and a sine wave on the user channel. That prefill mechanism is not an implemented private channel for mid-call tool results. Reusing it during a call would change the actual stream and must not be described as harmless metadata injection. [P1, P2]

The inspected NVIDIA model provides a Temporal Transformer output used by both the text readout and audio Depth Transformer. Its generator keeps delayed-code caches, an offset and CUDA-graphed forward calls. These are useful extension points, not evidence that it already accepts cross-model memory. The inspected forward_embeddings does not accept an external-memory argument. [C1]

The native training method delays the code streams, inserts their initial tokens, predicts, then undelays outputs and returns validity masks. Its LMOutput explicitly says logits are already aligned with input codes. Do not apply an additional generic causal-LM target shift. The trainer and streaming generator must consume the same source-update schedule as well as the same audio/text alignment. [C1]

Kyutai's moshi-finetune supplies reusable ingredients: frozen Mimi encoding, timestamp-based text/audio interleaving, masked text/audio cross-entropy, LoRA, checkpointing and distributed training. It is not a ready-made PersonaPlex connector trainer. Its entry calls model(codes=..., condition_tensors=...); the inspected NVIDIA implementation exposes a different training/streaming interface. Its example also targets Moshiko, not the released PersonaPlex checkpoint. Package versions, checkpoint keys, prompt construction, delay handling and trainable parameters require a concrete port and verification. [C2–C5]

Proposed private connection and causal contract

The first connector should be a learned state projection plus a small gated cross-attention module that passes information from external memory into the Temporal Transformer. Keep it separate from user audio and from the aligned agent-text stream. This avoids representing a private search result as something the caller said or something the assistant already spoke. Exact layer placement, memory capacity and compression remain experimental choices; the Atlas should not depict them as trained facts.

A zero-initialized residual gate gives a useful starting invariant: with connector contribution zero, the pretrained path is unchanged mathematically. Flamingo provides evidence for this pattern in multimodal conditioning, including an ablation favoring zero-initialized gating. It does not prove transfer to PersonaPlex. Verify the actual implementation against the original release on matched complete streams, including state, codebook outputs and sampling inputs; an equation alone does not establish numerical or runtime equivalence. Once a gate or LoRA changes, exact original behavior is no longer expected. [P4, C6]

The internal input must include enough provenance and timing metadata to establish which evidence was available for this audio frame:

• Session and branch identity, source model/layer definition, source position and monotonically ordered evidence version.
• The actual producer and delivery times. A state computed earlier but delivered later is unavailable until delivery; a later offline replay does not reconstruct its original arrival time automatically.
• A validity mask and task state such as pending, returned, failed or cancelled. A timeout is not a successful tool result. A model claim is not an execution receipt.
• Exact correspondence between the reasoning model state, its causal input history, the latest user correction and the relevant execution evidence. Raw reasoning-model states and authoritative tool status should remain distinguishable inputs.

For a claim that the model reads factual content from Inkling activations, the routing metadata must not also contain the answer, its label or a readable summary of the result. A treatment that supplies structured facts directly can be useful, but it is a separately labelled input condition. Execution receipts may provide the evaluation truth without being duplicated as an model input containing the answer.

At each existing audio step, read only memory made available by that step. Do not stop the 12.5 Hz audio loop waiting for the reasoning model, insert artificial audio steps to provide internal tokens, or expose future memory through a bidirectional training batch. The cross-attention mask must respect the per-frame availability of each memory item, not just the final bundle's contents. An absence of new evidence is a normal input condition.

Source updates must be visible through the actual serving path, including device-copy completion and CUDA-graph input handling. Do not assume a graph-captured tensor will change because its Python reference was replaced. Preserve native temporal/audio caches across normal updates; do not silently restart the conversation to make a new fact visible. These are requirements for the proposed implementation, not completed engineering claims.

Ordered training plan

Connector-only alignment is a practical starting choice, not a universal prerequisite. LLaVA and BLIP-2 demonstrate successful staged bridges; LatentQA and Activation Oracles directly adapt a decoder with LoRA in their same-model settings. If stage order becomes a material decision, use one bounded matched comparison of alignment-then-joint versus joint-from-start, with the same source data and total exposure. Do not mistake extra training for evidence that a curriculum is necessary. [P5–P8]

Initially use one connector design. Add a Q-Former or other resampler only when the actual state length, selective retrieval or compute measurements justify it. BLIP-2's 32 queries and multiple pretraining losses are properties of its visual experiment, not established settings for this task. Compression must retain needed names, numbers, corrections and older commitments; a small memory footprint is not sufficient evidence. [P6]

Losses and alignment that must survive the port

Use ordinary masked cross-entropy on the appropriate generated text and audio codebooks. Mask system-prompt/prefill targets and invalid delayed positions. Retain valid silence/audio frames and text-padding targets with their declared weights; padding, useful quiet, end of an utterance and cancellation of work are different events.

The PersonaPlex paper reports down-weighting non-semantic audio-token loss by 0.02 and padded-text loss by 0.3. The inspected Moshi-finetune example instead uses a first-codebook multiplier of 100 and text-padding weight 0.5. Its loss divides each stream's weighted loss by that stream's sum of weights, then adds the text and audio terms. Consequently, these numbers cannot be copied or inverted without checking the entire reduction and the relative text/audio contributions. The NVIDIA inference release is not a complete reproduction of the paper's training recipe. Record the chosen objective explicitly and report semantic-codebook, acoustic-codebook, non-padding text and padding losses separately. [P1, C2, C5]

Keeping the Depth Transformer frozen still permits audio loss to teach its upstream temporal inputs: retain autograd through that frozen computation. Wrapping it in no_grad would cut this learning signal. Frozen Mimi can encode fixed audio targets without gradients. Initially avoid adding cross-model hidden-state MSE, contrastive alignment, or an extra reward simply because another paper uses it; the unrelated models do not share a demonstrated numerical hidden-state target.

Text and speech targets must agree. Do not replace a transcript with the desired factual answer while retaining audio that says something else. Do not label a later tool result as available earlier to simplify synthesis. Preserve the assistant/user channel order, word timestamps, overlapping audio and quiet intervals. Kyutai's interleaver places text on Mimi's audio-frame grid, with explicit padding and end-of-padding behavior; sentence-level text inserted at an arbitrary frame is a different training distribution. [C3]

An all-short-utterance curriculum can teach the wrong conversational duration and end behavior. Conversely, filling most frames with padding can reduce aggregate loss while learning little grounded speech. Report actual conversation/source count, seconds of audio, supervised words, frame counts and important event coverage separately. The upstream README itself cautions that reducing training duration may make the model become silent sooner. This is a warning about distribution mismatch, not a requirement to copy its default duration. [C5]

Use text-only QA sparingly as an alignment diagnostic or auxiliary. The useful questions are “What changed?”, “Which constraint is current?”, “What result has arrived?” and “What remains unknown?”. LatentQA's future-completion/persona labels are not ground truth for current tool success. Activation Oracles shows a benefit from adding context-prediction data even at a matched example count, but its results do not establish that QA must precede speech or that a million-example corpus is needed here. [P7, P8]

The decisive counterfactual mid-call tool test

1. Start two runs from the same session state, voice prompt, role prompt, caller audio and sampling settings. The caller asks for current availability. The shared private state says the lookup is pending. Neither run has a result in user audio, the role prompt, a forced agent token or a future attention window.
2. At the same later audio-frame boundary, deliver different actual branch evidence: A has one available option at a specific time and price; B has no matching option. Only the private input differs. Continue without resetting the temporal or audio state. The correct response content must differ while both remain natural speech.
3. Repeat with the result delayed, absent, failed, and superseded by a user correction. The model must not assert a result before its delivery, infer success from a pending request, or retain a stale date after the correction. Before the branch evidence arrives, the runs have no causal reason to announce different availability.
4. Interrupt the response after an audible partial commitment. Measure actual stopping, cancellation of unheard queued audio, the next grounded response, and what the reasoning model retains about what was heard. Cancellation of a booking/search is a separate tool action and requires its own receipt.
5. Include missing/swapped irrelevant memory, unchanged memory, no connector, and the original PersonaPlex reference. Correct-versus-swapped memory distinguishes a useful private connection from fluent responses that ignore it. Restore the exact common prefix for each counterfactual rather than appending incompatible prerecorded user acceptance after a different assistant offer.

This is the first persuasive integration result. A fixed initial prompt containing the answer, a forced text stream, a lower teacher-forced loss, or a single good opening phrase would not satisfy it.

Teacher and rollout policy

Teachers may generate useful utterances, QA labels and corrections from the exact causal record. Execution receipts and user inputs determine factual/action labels; teacher eloquence does not. Preserve disagreement and unavailable outcomes. Synthetic response audio must match its approved text and disclose its voice/timing generation source; it does not become original human interaction evidence.

For on-policy training, the teacher must see the speech model's actual generated prefix and the evidence available at that point. It should teach a valid continuation or correction, not silently replace what the user already heard. GKD is evidence for teacher feedback on student-generated prefixes after the student can produce adequate sequences; it is not proof that every SFT run requires an OPD phase. [P9]

Inkling and PersonaPlex do not have a demonstrated shared vocabulary. Do not directly compare their text logits, and do not apply a text-only KL to heterogeneous audio-codebook outputs. A compatible teacher can score the same PersonaPlex text-prefix tokens with a separately verified text-and-evidence input containing only information available at that time; otherwise use corrected-sequence supervision. Any additional audio imitation needs matching codec identities, timing and token semantics. Name a sequence-CE method accurately rather than calling it full-distribution distillation.

Evaluation and selection

Freeze the source-level development/final separation before deriving multiple questions, audio variants, nearby cutoffs or branches. Keep those siblings together. Hold out complete conversations and task/entity combinations; multiple QA rows from one state are not independent examples of generalization.

Evaluate complete spoken responses and complete task episodes where the branch is valid. Retain failed, silent, interrupted, invalid and unattempted cases. Report:

• Grounded names/numbers, current constraints, correction acceptance, accurate pending/failed/completed action reporting and useful quiet.
• Source-following before/after private evidence changes, including stale or missing information and different history lengths.
• Voice consistency, intelligibility, text/audio agreement, pauses, backchannels and accidental takeover of the caller's turn.
• Actual audible first useful response, speech-stop time, obsolete queued-audio cancellation, sustained frame production and missed frame deadlines. Measure these while the reasoning model runs concurrently, including transport and playback, rather than adding isolated paper latencies.
• Later task behavior and memory of communicated commitments, with generated, synthesized, queued, played and cancelled content distinguished.

Use a small fixed reference/preservation panel throughout training. Select development checkpoints by the behavioral tradeoff; disclose the checkpoint choice and expose the locked selection to the final panel once. Repeat a material claimed improvement with another training seed before making it a stable architecture preference. A full factorial sweep over connector type, gate placement, LoRA rank and losses is not the starting plan.

Only three additional training comparisons are initially decision-relevant: whether joint LoRA helps beyond the frozen-only connector, whether QA changes transfer at matched exposure, and whether rollout training beats the same amount of extra supervised work. Run them when their corresponding decision arises, not as prerequisites for every next output.

Benchmark limits and public claims

The released PersonaPlex is a strong starting speech model, not a demonstrated system connected to an external reasoning model. Its paper's Service-Duplex-Bench contains 50 scenarios with seven single-turn probes each; it does not establish real tool execution or long-call task preservation. The paper lists external tools as future work. The release adds Fisher conversations and changes synthetic voice/audio generation relative to the main experimental checkpoint; keep those results separate. [P1, P2]

Do not place a single headline latency in the Atlas as the expected Inkling-connected delay. The PersonaPlex model card and the underlying Full-Duplex-Bench paper describe interruption timing differently, and the benchmark computes some latency/quality measures only for takeover cases. Moshi's architectural codec/acoustic delay is another distinct measure. The new private connector, concurrent reasoning model, network, playback and user's actual task are not measured by those numbers. [P2, P3, P10]

Sources and review scope

Primary papers and official project material were read live. The code anchors below are pinned to the repository heads inspected on October 4, 2026. Code inspection establishes available operations and contracts only: no model was loaded, no trainer was run, no latency was measured and no adaptation of the existing trainer was verified. The only repository artifact created by this research is this Markdown file.

• P1 — PersonaPlex paper: architecture, objectives, data, evaluation and released-checkpoint appendix. The main paper's training settings and release are explicitly different in Appendix A.
• P2 — Official release: model card and project page.
• P3 — Moshi: model, delayed streams, Inner Monologue and audio/text training, especially §§3.3–3.4 and 4–5.
• P4 — Flamingo: gated conditioning and controlled ablations, §§2.2 and 3.3. This is visual-language evidence, not a PersonaPlex result.
• P5 — LLaVA: simple connector and staged training, §4.
• P6 — BLIP-2: Q-Former, bottleneck and training stages, §3.
• P7 — LatentQA: data, direct decoder tuning and limitations, §§3–4, 6 and Appendix A.
• P8 — Activation Oracles: direct LoRA training, data-mixture ablation and same-model scope, §§3 and 6, Appendix A.
• P9 — GKD: on-policy distillation objective and competent-student assumption, §3.1.
• P10 — Full-Duplex-Bench: takeover, timing definitions and evaluation denominators, §III-B. Later v3 adds real-disfluency/tool-use evaluation, but does not report PersonaPlex or Moshi results.
• C1 — NVIDIA PersonaPlex 3428dfd9: LMOutput alignment, temporal/text/depth paths, native training delays/masks, generator and graph state, prompt prefilling.
• C2 — Moshi-finetune 2acc879f: frozen Mimi and interleaver setup, text/audio training objective, weighted loss reduction.
• C3 — Moshi-finetune interleaving: timestamps, padding and frame-grid construction, frozen audio encoding and paired transcript loading.
• C4 — Moshi-finetune trainable scope: LoRA initialization and loader, parameter selection. The generic selection includes all matching LoRA names; temporal-only adaptation requires an explicit narrower selection.
• C5 — Moshi-finetune example: actual example weights and configuration, README/data and duration guidance.
• C6 — OpenFlamingo 655f693f: zero-initialized gated cross-attention implementation. This is an open reproduction's component, not NVIDIA PersonaPlex code.

Historical design note. The implemented Stage 1 architecture, current losses and evaluation are documented in Current sequence memory.

Data for an Inkling → PersonaPlex speech model

Research snapshot: 2026-10-04. This describes a proposed experiment and the data formats and alignment requirements it would need. No PersonaPlex adaptation, audio generation, preparation of stored hidden-state arrays or model evaluation was performed for this report.

The existing corpus provides much of the semantic supervision and provenance of the reasoning model states. It does not yet provide a matched, continuous agent-audio stream for a PersonaPlex trained to follow Inkling. The shortest useful route is to reuse eligible native states, recorded caller audio, grounded speech text and real tool events; add explicitly synthetic agent audio; then collect actual interactive trajectories to learn and measure interruption, feedback and timing.

A hidden state is a vector computed inside the model. A causal input contains only information available at the prediction time. Private means the state is supplied internally, not as caller audio or words already spoken by the assistant.

What the upstream work actually supplies

PersonaPlex conditions a full-duplex speech model on a text role prompt and an audio voice prompt. Its public interface does not supply a trained channel for another model's changing hidden states. Its released checkpoint builds on Moshi, retaining its pretrained speech behavior; training an Inkling condition would be an additional adaptation.[1]

The paper's main experiments used synthetic service and question-answering dialogues. The released-checkpoint appendix changes that recipe: it adds 7,303 Fisher conversations, totaling 1,217 hours; uses synthetic voices sampled from TortoiseTTS and varied with Praat; and uses Chatterbox for both synthetic dialogue families. The experimental section's real-speaker samples and mixed Dia/Chatterbox pipeline should not be presented as the exact released-checkpoint recipe.[2]

This supports combining real conversational audio with synthetic task speech. It does not establish that generated voices, stitched interruptions or a QA score will transfer a new model's hidden information into accurate live speech.

The public Moshi-Finetune trainer is a useful starting point. Its documented input is stereo audio with the agent on the left and the user on the right, a JSONL inventory containing path and duration, and a neighboring JSON transcript with word timestamps. It provides LoRA and full fine-tuning, but is documented for official Moshi checkpoints; PersonaPlex compatibility and the proposed private condition remain to be implemented and verified.[3]

The inspected PersonaPlex codec loader uses 24,000 Hz audio, 12.5 codec frames per second and eight Mimi codebooks per mono stream: an 80 ms frame corresponds to 1,920 waveform samples. Preserve the checkpoint's actual stream ordering and delay schedule. The loader configures sixteen audio streams across the two speakers and changes the depth-prediction configuration during loading, so the generic Moshi README's token-count example is not an exact PersonaPlex training specification.[4]

Reuse the available evidence by modality

The maintained MTP capture path checks original audio, source and transcript identities. It distinguishes measured caller cuts from dataset-proxy cuts. Its state pack identifies 42 raw layer streams plus the actual native final stream, each of width 4,096. PersonaPlex instead uses completed block outputs S0–S41 after the MLP-output short convolution and residual addition, plus the separate Final state. Existing raw auxiliary packs cannot be relabeled as those completed outputs; any reuse needs the matching capture representation. A reference to a complete state-array file does not itself verify every array intended for a new connector.[8][9]

For the first offline adaptation, retain source-conversation and duplicate-family membership, existing source cutoffs and complete required alternatives. A fork starts from its authentic past history and current caller. Do not concatenate later recorded callers after a changed agent reply and call the result a valid interactive conversation. Compatibility with the new reply has not been observed.

Synthetic agent audio should be generated from already eligible target text using a voice with permission for the intended use. Retain the requested text, generated waveform, voice identity, generator revision and generation settings. Independently align the actual waveform and check that names, numbers, negations and commitments survived synthesis. PersonaPlex is a conversational generator rather than a guaranteed verbatim TTS interface: asking it to read a target does not prove that its audio contains that target. Transcription alone is also insufficient evidence of factual correctness.

A synthesized pause or overlap is a declared training schedule. It can train a behavior; it is not a measurement of natural turn timing or an observation that the caller interrupted at that moment. Missing target audio is not silence. Unknown labels must remain unknown, with audio loss excluded for that modality rather than supplied with an invented silent waveform.

Minimal dataset and private stream contract

Keep the ordinary Moshi audio inventory simple. This is an illustrative format example, not an existing recording:

{"path":"clips/example.wav","duration":12.0}

Its adjacent clips/example.json contains agent-word alignments of the form below. The official annotation script transcribes channel 0 and emits SPEAKER_MAIN; timestamped ASR is a proposed annotation mechanism, not ground-truth hearing or perfect word timing.[5]

{"alignments":[["example",[1.20,1.58],"SPEAKER_MAIN"]]}

The current interleaver reads those alignments, encodes both audio channels with Mimi, and combines the text and audio streams. It also accepts text_conditions. That field is not an implemented time-varying Inkling-latent channel. The following additional sidecar and collator behavior are proposed, not an upstream feature.[6]

Large states remain in immutable external arrays; a row carries small references. Cropping a training clip must crop or offset all audio, words, evidence and private updates together. Do not read the complete future sidecar into a condition available from frame zero. Preserve enough earlier context or an explicit reset so a crop does not falsely imply a new conversation.

At a prediction frame, the condition may contain only information whose causal availability and delivery have reached that frame under the declared clock mapping. Recorded audio time, source event time, native processing time and host delivery time are different clocks. If an offline schedule maps one to another, retain the mapping and label it as constructed. Current causal input code already preserves complete versus partial events and refuses future rows; reuse that boundary instead of constructing a new summary from the finished conversation.[10]

The private condition should be delivered as a model-side conditioning tensor, through the selected adapter or attention mechanism. It should not become ordinary spoken text, the model's output transcript, a cloned voice prompt or an audible channel. A minimal ablation compares the same speech model with the correct private stream, no private stream and an unrelated stream.

Learning objectives that match the evidence

Direct text-and-audio training is the central objective. With the correct private state and available user audio, predict the aligned agent text and codec streams. The open trainer sums masked text loss and masked audio-code loss, with configurable text-padding and semantic-codebook weighting.[7] The adaptation must apply those masks according to the actual PersonaPlex checkpoint and stream definitions; it must not blindly assume that every predicted channel is an agent target. Keep the codec fixed initially as the initial adaptation choice, then change it only if measurements justify that additional comparison.

Task-grounded QA is an auxiliary test or warmup. From the same causal snapshots, ask what the caller wants, which name/date/constraint is known, whether a result is pending, what changed, or whether an action can truthfully be called complete. Answers cite available events in the private training metadata. Include not-yet-known, failed, contradicted and ambiguous cases. Do not invent a single required answer when several clarifications are valid. General factual questions that the pretrained model can answer without the private stream are weak tests of latent reading.

QA can initially use text targets without manufacturing audio. If spoken QA is later trained, synthesize and align its audio with the same provenance rules. Compare direct speech training against QA-then-speech or mixed QA/speech with the same heldouts and explicit training exposure. Success on private-state questions does not demonstrate natural turn taking or dialogue that respects the tool-use rules.

Interruption and feedback require additional trajectory evidence. Teach quiet intervals, real overlap, resumption and correction from actual continuous recordings when available; constructed schedules remain a separate treatment. Preserve the model's existing conversational prior rather than interpreting every pause as an invitation to speak. A bare tool failure does not specify whether the right behavior is quiet, a clarification or an explanatory update.

Cached reasoning-model states remain valid for their exact native input. Different teacher speech can be a separately declared decoder history in an offline, one-way experiment. Once spoken output affects the reasoning model's context, tool choices or the caller's next action, the continued conversation needs newly captured states from that history. A changed speech-model history does not authorize relabeling old hidden states as if they came from that new branch.

Playback and feedback are separate observations

Store four distinct boundaries: generated, decoded/queued, device-played, and cancelled. A generated text token, codec frame or completed WAV file is not proof that a caller heard it. The repository's PlaybackReceipt already requires playback_ack with a playback_device observation before labeling a receipt heard; local rendering alone does not satisfy that requirement.[11]

For each emitted segment, retain its output sample interval, generation and enqueue times, device-played interval and cancellation boundary. If device evidence is unavailable, leave played extent unknown. Device playback establishes delivery to the output path, not that a human attended to or understood the speech.

Feedback to Inkling must state which output extent it represents. The model may know what it generated, while commitments attributed to the caller should follow the observed played extent. An interrupted suffix may have been planned or generated without being delivered. Current PersonaPlex's internal text stream is useful alignment evidence, but it is not an independent transcript of what the speaker device played. A feedback summary or vector derived from the entire planned sentence cannot be called memory of the heard prefix without verifying its correspondence to the played audio.

The same separation applies to user audio: retain capture, transport arrival, model-consumption and optional transcript-readiness boundaries. The project's goal requires adaptation to new information and memory of what was communicated, so first-token latency or an offline output file cannot complete the interaction evaluation.[12]

Smallest meaningful evaluation

1. Offline grounding: source-disjoint original conversations; correct, absent and shuffled private states; names/numbers; before/after result and correction pairs. Compare full utterances and unsupported commitments, not only loss.
2. Speech preservation: held-out user speakers and voice identities; intelligibility, word accuracy, conversational naturalness, useful quiet and overlap. Report real recorded and synthetic-audio cases separately.
3. Live task interaction: an executable tool environment with valid caller continuation, using actual candidate speech and feedback. Measure task completion, changed constraints, unsupported actions, interruption response and played-prefix consistency.
4. Timing: report user audio availability, private-state readiness, model generation, audio decoding/queueing and device playback separately. Synthetic schedule results and physical measurements stay separate.

Keep all forks, voice variants, synthetic realizations and QA questions from one original source family in its assigned split. Do not move existing evaluation sources into training to enlarge the audio pool. These tests evaluate the new conditioning mechanism; they do not replace the wider architecture study's remaining data or evaluation obligations.

Sources

Primary upstream sources were read at the revisions below. Repository sources were inspected without loading state arrays or large dataset manifests.

1. NVIDIA, PersonaPlex README: text/voice prompting, Moshi basis, public usage and released-checkpoint description.
2. Roy et al., PersonaPlex paper, sections 3.2 and 4, and Appendix A: synthetic dialogue generation and the distinct released-checkpoint recipe.
3. Kyutai, Moshi-Finetune README: stereo channel roles, inventory, transcript files and adaptation scope.
4. NVIDIA, PersonaPlex model loaders: audio/frame rates, codebooks, stream delays and loading configuration; offline inference: prompt phases and generated-audio decoding.
5. Kyutai, annotation script: channel-0 transcription and the actual alignments JSON shape.
6. Kyutai, interleaver and dataset loader: audio/text preparation, crop selection and optional text conditions.
7. Kyutai, training loop: separate masked text and audio losses.
8. Repository: main capture audio (graft/main_capture_audio.py, repository reference), check_spoken_capture_assets; environment records (graft/environment_records.py, repository reference), measured and unmeasured audio references. These retain original waveform, source and alignment provenance.
9. Repository: state pack (graft/state_pack.py, repository reference), FULL_LAYER_IDS, NAMED_LAYER_SETS, StatePackConfig; source-history capture (graft/source_history_capture.py, repository reference), build_source_capture_record. Actual native final and raw auxiliary streams remain distinct.
10. Repository: grounded supervision (graft/grounded_supervision.py, repository reference), RealGroundedSupervisionSpec and causal_input; feedback boundaries (graft/feedback_boundaries.py, repository reference), collect_feedback_boundaries. These associate histories with complete or partial events and the state rows available at each point.
11. Repository: shared speech-model data (graft/shared_mouth_data.py, repository reference), Availability, SpeechHistoryEntry, SpeechTarget, ToolExecution and PlaybackReceipt. These distinguish source, processing, availability and playback evidence.
12. Repository: capability goal (PROJECT_GOAL.md, repository reference), natural conversation, responsive audio, correction handling and memory of communicated speech.