Introducing ToddlerJev: An open-weight native-audio decision model

Today, we are releasing ToddlerJev, an open-weight model for making structured decisions directly from audio.

ToddlerJev receives an audio input, a textual question, and a set of candidate decisions. In one forward pass, it assigns a probability to every candidate. It does not need to transcribe the audio into an intermediate document, and it does not generate an answer token by token. The output is already a distribution over the choices the application cares about.

On the evaluated checkpoint, ToddlerJev reaches 68.70% on MMAU-mini, 61.30% on MMSU, and 56.90% on MMAR. The ToddlerJev weights are available alongside evaluation artifacts for inspecting, reproducing, adapting, and testing the model beyond our benchmark suite.

Audio in, decisions out

Many audio systems are assembled as a pipeline: automatic speech recognition turns sound into text, a language model reasons over the transcript, and application code extracts a final label. Every boundary can discard information. A transcript preserves words while often flattening timing, prosody, speaker state, background events, and non-speech sound.

ToddlerJev instead operates on the audio itself. The model receives an audio observation $x$, a textual question $q$, and a request-time candidate set $C = \{c_1, \ldots, c_K\}$.

The candidates are not fixed classes baked into the checkpoint. They are described in text and may change between requests. ToddlerJev scores each candidate by comparing its contextual representation $h_k$ with a learned decision-state representation $h_*$:

The scores become a probability distribution over the supplied candidate set:

This dynamic pointer interface avoids generated answers that must later be parsed. An application describes its available actions and receives a probability for each action without rebuilding a fixed classifier head whenever the decision vocabulary changes.

Training ToddlerJev

ToddlerJev adapts the Thinker component of Qwen2.5-Omni-3B with rank-16 LoRA and a dynamic pointer head. The audio and vision encoders remain frozen. The resulting adaptation contains 8,421,888 trainable parameters.

Training uses hard-label cross-entropy with shuffled candidate order. Optimization uses AdamW in BF16, an effective batch size of eight, warmup followed by cosine decay, a learning rate of $10^{-4}$ for the adapter, and $3 \times 10^{-4}$ for the pointer head.

The real-audio stage uses 47,623 task- and label-balanced presentations drawn with replacement from NSynth, NonSpeech7k, and MELD. These sources cover musical notes and instruments, environmental sound, and emotional speech rather than treating every audio input as spoken language.

The speech stage uses 20,000 examples synthesized by Teen V1 from 13 datasets, including Banking77, SST-5, ARC, and CommonsenseQA. Only the input is rendered as speech; the question, candidate descriptions, and target remain textual.

A100 40 GB BF16 inference speed

The streaming workload retains an eight-second audio context and produces updates once per second. The recording workload processes one complete 30-second input.

Native-audio benchmark results

ToddlerJev was evaluated on three native-audio decision benchmarks: MMAU-mini, MMSU, and MMAR.

The comparator values are published results rather than reruns. ToddlerJev uses 1,000 MMAU-mini items, all 5,000 MMSU items, and a 1,000-item short-clip MMAR subset. Protocols, prompts, subsets, snapshots, and scoring differ, so the table locates the release in the current landscape without claiming a controlled ranking.

Spoken-text benchmark results

We also evaluated spoken versions of public text tasks. ToddlerJev receives TeenageWorks-synthesized audio and textual instructions or options, while the comparison systems receive the original text. These results measure cross-modal transfer; they are not a controlled text-model leaderboard.

The evaluation contains 8,147 cases across 15 suites, producing 8,145 predictions and two explicit failures. The complete eleven-metric comparison is below. Values are accuracy percentages except SST-5 expected-level MAE, where lower is better. A dagger marks incomplete coverage, and an em dash means no comparable usable report.

Jev, Laya, and dJev use pinned public predictions; the other comparator configurations use publisher reports. No competitor API was rerun. Reports with mismatched case counts are excluded. The GPT-6 row refers specifically to GPT-6 Luna Decisions through OpenRouter.

Accuracy across the spoken benchmark suite

The grid below shows the ten percentage-accuracy columns from the full table. SST-5 is excluded because it reports mean absolute error, where a lower value is better, and therefore does not belong on the same accuracy scale. The outlined row is ToddlerJev.

ToddlerJev is strongest on Banking77 and ScienceQA. Its MMLU and medical question-answering results remain weak. We do not combine these heterogeneous tasks into a single frontier score.

Does the model use the audio?

An audio model can appear capable while leaning on question wording, candidate priors, or dataset artifacts. On a paired 200-item MMAU diagnostic, ToddlerJev scores 73.50% with full audio, 67.00% with only the first eight seconds, and 38.50% when the signal is replaced by silence.

The silence gap shows that the model uses evidence from the audio rather than treating it as decoration. The prefix result is an evidence ablation, not a validated streaming policy; some decisions depend on information that arrives later.

Where ToddlerJev still fails

Candidate ordering remains a failure mode. Reordering identical options changes the selected answer on 9.70% of MMAU-mini cases and 12.80% of MMAR cases. Shuffling candidates during training reduces simple positional shortcuts, but it does not make the model order-invariant.

The output probabilities should not be treated as calibrated confidence. The 15-bin expected calibration error is 0.085 on MMAU-mini and 0.104 on MMAR. Applications that attach operational meaning to a threshold should calibrate on data representative of their own deployment.

Perceptual quality is not a reliable shortcut for task success. NISQA overall MOS has a Pearson correlation of only 0.22 with correctness in the reported analysis. Clean-sounding audio can still be decisionally difficult, while imperfect audio may contain enough evidence for the correct answer.

What is included in the open release

ToddlerJev is released as an open-weight native-audio decision model so that its behavior can be examined at the level of the model, outputs, and evaluation artifacts—not only through a hosted demo.

The Hugging Face release and companion bundle include detailed metrics, per-example predictions and probabilities, uncertainty estimates, provenance, source commits, checkpoint hashes, and runnable evaluation code. This makes it possible to audit errors, test calibration, measure candidate-order sensitivity, and adapt the model to new decision spaces.

Boundaries of the evidence

ToddlerJev currently has one adaptation seed and no matched frozen-backbone or training-data ablation. Published comparisons differ in modality, prompt format, model snapshot, and benchmark subset. The spoken-text evaluation relies on synthesized speech and therefore does not establish performance on spontaneous, accented, noisy, or conversational speech.

The 2,600-example TTS holdout scores 82.12%, but 260 items were monitored during training, so it is validation rather than a fully untouched external test. Exact-text auditing found 137 external cases matching holdout content, including 13 monitor matches. No exact external-to-training text matches were found, but semantic overlap and pretraining exposure remain unknown. Six NonSpeech7k source recordings are shared with training.

ATBench was excluded after its results were inspected at the authors’ request; that post-hoc scope change is documented in the release materials. We make no claim for missing VoxCeleb1 or DiDiSpeech evaluations, hidden test sets, calibrated confidence, or sliding-window deployment.

The weights are open so the next result does not have to be ours. Reproduce it, break it, calibrate it, or teach it a new decision space.

References

• Qwen2.5-Omni-3B
• Qwen-Audio-Jev
• AudioJev
• MMSU
• MMAU code
• MMAR dataset
• JevBench, evaluated at commit 6954ce9d98d81210d043ca0e94bc72bff6443140
• DecideBench, evaluated at commit bbf8f2c77b544952b42be555866729a22c33980d
• Agent/Jev tasks, evaluated at commit 609a664e81f46e03731600bd7bd9f1b7a61257c5