Transparent and reproducible.

Audio data

• Rioplatense: OpenSLR SLR61 (es_AR), Google, CC BY-SA 4.0.
• Colombian: OpenSLR SLR72 (es_CO), Google, CC BY-SA 4.0.
• Neutral (LatAm): FLEURS es_419, Google, CC BY 4.0.
• Portuguese (BR): FLEURS pt_br, Google, CC BY 4.0.
• Mexican (STT): Coming soon. We have not yet found an open dataset downloadable without sign-up.

Deterministic selection: files sorted alphabetically, one clip per speaker, 20 per accent.

Speech to text (STT)

• Each model transcribes each clip with forced language (es/pt), beam_size=1, no silence filter (no_speech_threshold=None), same for every model.
• Normalization: lowercase, punctuation removed, accents kept.
• WER and CER per accent with jiwer.
• Latency p50 = median total transcription time per sentence on CPU (8 threads, no GPU).

Text to speech (TTS)

• Each voice synthesizes the accent's reference sentences.
• Intelligibility: audio is transcribed with whisper-large-v3-turbo and scored against the text. Measures clarity, not naturalness.
• Latency p50 = median time until the full audio is ready on CPU.

Commercial providers

• Shown as “Coming soon” with official price and source.
• We will publish metrics only after measuring them with our own accounts, within their terms.

Limitations

• 20 clips per accent: enough for a pilot.
• Public datasets may have been seen in training.
• Local CPU latencies: comparable to each other, not to cloud APIs.

Reproduce

• python bench/fetch_data.py
• python bench/run_bench.py
• VB_LIGHT=1 python bench/run_bench.py (daily light run)