LisTAya: SLAM-ASR projectors for twelve languages on the Tiny Aya decoders

All checkpoints are listed in the collection https://huggingface.co/collections/ERISLab/listaya-slam-asr-projectors-on-tiny-aya-rocling-2026-6ab031414095f3df8805ed7b.

LisTAya (Listening Tiny Aya) is a family of speech-recognition models built with the SLAM-ASR recipe: a frozen Whisper-medium speech encoder, a frozen large-language-model decoder, and a trainable linear projector between them. Only the projector is trained. The checkpoints in this collection are the models behind the paper How Does Regional Decoder Specialization Help Low-Resource ASR Based on the SLAM-ASR Framework? (ROCLING 2026). Each checkpoint is one (language, decoder) cell of that study.

Decoders

Four Tiny Aya variants (3.35B parameters, one shared base model and tokenizer, differing only in the regional weighting of their post-training data): Global, Earth, Fire and Water. Two controls: Tiny Aya Base, the model before instruction tuning, and Qwen3-4B, a decoder from another family.

Languages and training data

Every projector is trained on the WorldSpeech train split of one language (formal speech from parliamentary proceedings, broadcasts and audiobooks, clips under 30 seconds): Amharic (am_et), Hausa (ha_ng), Swahili (sw_ke), Hindi (hi_in), Marathi (mr_in), Tamil (ta_in), Urdu (ur_pk), English (en_us), Spanish (es_mx), French (fr_ca), Indonesian (id_id), Kreol Seselwa (crs_sc). The region is the Tiny Aya region each language is assigned to in the study; Kreol Seselwa is supported by neither the encoder nor any decoder and has no region. Its data come from the ERISLab/WorldSpeech config crs_sc; every other language from disco-eth/WorldSpeech.

Training recipe

AdamW at learning rate 1.5e-3, batch size 8 with gradient accumulation 64 (effective batch 512), 1,000 steps, no weight decay, one NVIDIA H200, validation every 10 steps. Checkpoints are saved every 50 steps and the trainer keeps the one with the lowest validation loss on the FLEURS validation split, with two exceptions: Hausa is validated on the WorldSpeech test split, and Kreol Seselwa, which FLEURS does not cover, on a held-out 2% split of its WorldSpeech data.

What each repository holds

Repositories are named q2a_openai_whisper-medium_<decoder>_ws-<language>-<step>. Each cell has two: the checkpoint the trainer kept as best on validation loss (the one the paper evaluates) and the final checkpoint at step 1000. Every repository holds the full trained model (encoder, projector and decoder weights, processor and tokenizer files) plus trainer_state.json with the run's validation curve.

Load a checkpoint and transcribe

Every checkpoint is a Qwen2AudioForConditionalGeneration model and loads with the standard classes of transformers 4.57.5. The prompt below is the one the projectors were trained with; the audio is a 16 kHz mono array. The weights are stored in bfloat16; the paper's evaluations load them in float32, as below. Loading in bfloat16 halves the memory and changed 3 of 40 English validation transcripts in a check.

import torch
from datasets import load_dataset
from transformers import AutoProcessor, Qwen2AudioForConditionalGeneration

repo = "ERISLab/q2a_openai_whisper-medium_CohereLabs_tiny-aya-global_ws-en_us-500"
processor = AutoProcessor.from_pretrained(repo)
model = Qwen2AudioForConditionalGeneration.from_pretrained(repo, dtype=torch.float32).to("cuda")

prompt = "<|audio_bos|><|AUDIO|><|audio_eos|>Transcribe the input speech: "
sample = next(iter(load_dataset("google/fleurs", "en_us", split="validation", streaming=True)))
audio = sample["audio"]["array"]  # 16 kHz mono float array
inputs = processor(text=prompt, audio=audio, sampling_rate=16000, return_tensors="pt").to("cuda")
with torch.no_grad():
    generated = model.generate(**inputs, max_new_tokens=256, do_sample=False)
hypothesis = processor.batch_decode(generated[:, inputs.input_ids.shape[1]:], skip_special_tokens=True)[0]

On this clip the English Global checkpoint writes "when you call someone who is 1000s of miles away you are using a satellite"; the reference is "when you call someone who is thousands of miles away you are using a satellite".

Validation CER of the kept checkpoint

Character error rate (%) of each kept checkpoint on its validation split, read from its trainer_state.json. This is the score of the released model, not the minimum of the run's validation curve, which the paper reports as best CER.

Language	Region	Global	Earth	Fire	Water	Base	Qwen3-4B
Amharic (am_et)	Earth	35.1	36.3	31.0	28.6	47.3	39.5
Hausa (ha_ng)	Earth	37.1	37.2	36.9	38.7	36.6	34.1
Swahili (sw_ke)	Earth	15.6	17.2	22.9	19.7	19.2	14.7
Hindi (hi_in)	Fire	15.7	18.8	18.5	16.4	12.2	15.5
Marathi (mr_in)	Fire	14.2	14.4	14.7	19.4	14.3	15.4
Tamil (ta_in)	Fire	84.5	72.9	47.3	80.6	45.7	53.4
Urdu (ur_pk)	Fire	33.1	21.8	26.2	25.5	25.3	48.4
English (en_us)	Water	10.2	7.2	4.4	14.8	5.5	7.2
Spanish (es_mx)	Water	3.5	5.6	3.6	3.5	3.1	3.8
French (fr_ca)	Water	8.6	7.4	8.5	7.1	11.1	7.2
Indonesian (id_id)	Water	6.1	9.1	5.9	6.1	6.1	7.4
Kreol Seselwa (crs_sc)	none	23.1	18.9	23.5	21.4	24.6	18.6

Ablations and superseded runs

• Tamil at learning rate 1.5e-2 (ws-ta_in-<step>-lr0.015): the four Tiny Aya regional variants retrained on Tamil at ten times the study's learning rate, best and final checkpoints. The best rate tried for Tamil (kept-checkpoint validation CER 49.2 Global, 51.6 Earth, 38.0 Fire, 49.6 Water); the same rate made Amharic worse on every variant, so the study keeps one rate throughout and these runs stay out of its tables.
• Spanish, first attempt on es_es (ws-es_es-<step>): the four regional variants trained on the es_es WorldSpeech config before the study switched to es_mx. One checkpoint each, stopped between step 350 and step 850; CER was not logged for these runs, so they are not comparable with the grid.

Requirements

The versions the checkpoints were trained and evaluated with:

• transformers==4.57.5 (the checkpoints' config.json records this version)
• torch==2.9.1
• datasets==4.5.0 with torchcodec==0.9.1, only for loading audio as in the example

Citation

@inproceedings{rios-etal-2026-regional,
    title = "How Does Regional Decoder Specialization Help Low-Resource {ASR} Based on the {SLAM}-{ASR} Framework?",
    author = "Rios, Edwin Arkel  and
      Zaruma, Jocelyn  and
      Ewoorkar, Girish  and
      Sourabh, Sneh  and
      Mack, Julian  and
      Juan, Hung-Hui  and
      Huang, Stephen  and
      Lai, Bo-Cheng",
    booktitle = "Proceedings of the 38th Conference on Computational Linguistics and Speech Processing (ROCLING 2026)",
    year = "2026",
    publisher = "Association for Computational Linguistics"
}