ufakzeka-kulak: A Small Turkish Speech Recognition Model Trained from Scratch
An open Turkish speech recognition model of 31,564,935 parameters, trained from scratch, that runs offline on a CPU. The post covers its training data and method, the results of the release gate and its measured limits.
PDFModel (Hugging Face)GitHub repositoryModel pageTry the model
- 31,564,935parameters
- 0.0921word error rate on FLEURS Turkish test
- 0.0916word error rate on Common Voice 27.0 Turkish test
- Apache-2.0weights
Abstract
ufakzeka-kulak is a Turkish speech recognition model of 31,564,935 parameters, trained from scratch by ufak AI and released under Apache-2.0. It writes text with capital letters and punctuation and writes numbers out in words, and the same weights run offline on a CPU in five runtimes. It was trained on 4,977 hours of speech whose licences allow commercial use. Of these, 281 hours have human transcripts; for the other 4,696 hours the training text is an uploader subtitle or a machine transcript, kept only where a second, independent reading of the audio agrees with it.
Choices were made on development data only, and the test sets were scored once, in a single release gate. The word error rate (WER) is 0.0921 on the FLEURS Turkish test set (95 percent confidence interval 0.0792 to 0.1055) and 0.0916 on the Common Voice 27.0 Turkish test set (0.0873 to 0.0958), where Whisper base scores 0.2461 and 0.3287. Compared on the same sentences, the 95 percent confidence interval of the difference lies wholly below zero on both sets, so the gate passed; Whisper large-v3 scores lower, 0.0604 and 0.0767. On conversational speech the WER is 0.3807.
1. Introduction
For speech to stay on the user’s device, a speech recognition model has to run offline on an ordinary CPU or in a browser; for it to go into a product, its licence has to allow commercial use and its training data has to be known. ufakzeka-kulak’s weights are under Apache-2.0, the licence of every training source allows commercial use, and the full list of sources, with licences and attribution, is in the ATTRIBUTION.md file of the model repository.
It is the third model of the ufakzeka family and, unlike the chat model ufakzeka-1 and the decision model ufakzeka-karar, it is not built on ufakzeka-1-base. “kulak” is Turkish for ear. Details beyond this post are in the model card.
2. Model
ufakzeka-kulak uses NeMo’s FastConformer encoder with 16 layers of hidden size 256, 4 attention heads and 8 times subsampling along the time axis. The encoder feeds two outputs. The default is a token-and-duration transducer (TDT) decoder, which predicts with each token how many audio frames to move ahead (durations 0 to 4 frames); the second is a CTC (connectionist temporal classification) head. The vocabulary is a SentencePiece BPE of 1,024 pieces, trained only on the training transcripts after decontamination. The model reads 80 log-mel features from mono audio at 16,000 Hz and writes out one utterance at a time (Section 5).
Output form. The model writes capital and lower-case letters, and for punctuation only the full stop, the comma, the question mark, the apostrophe and the hyphen inside words; other marks in the training text were mapped to these. Numbers are written as words, so the model writes “iki bin yirmi ikide”, not “2022’de”. Circumflex letters are written without the circumflex.
Runtimes. The NeMo 3.0.0 file, in float32, is the reference. The other files are TDT and CTC folders for transformers 5.20.1, float32 and int8 files for sherpa-onnx 1.13.8, f32 and q8_0 GGUF files for parakeet.cpp v0.6.1 that hold both the TDT decoder and the CTC head, and float32 weights for parakeet-mlx 0.5.3 with a feature function of their own, since parakeet-mlx computes its features differently from NeMo; they were checked at bfloat16, the precision parakeet-mlx loads them at by default. Only files whose normalised WER stayed within 0.0050 of NeMo’s on the Common Voice and FLEURS development sets, with no failed utterance, are released (Section 4.6).
3. Data and training
3.1 Sources
Hours count each recording once, without the speed-perturbed copies made during training. The first four sources train on their own transcripts, the YODAS recordings on the text of Section 3.2.
| Source | Licence | Hours |
|---|---|---|
| Mozilla Common Voice Scripted Speech 27.0 Turkish, without the sentences and speakers of its dev and test splits | CC0 1.0 | 67.9 |
| Mozilla Common Voice Spontaneous Speech 5.0 Turkish | CC0 1.0 | 0.1 |
| antalia-voice-corpus, one voice actor | CC BY 4.0 | 4.2 |
| Turkish Speech Corpus (TSC), ISSAI, Nazarbayev University, train split | CC BY 4.0 | 209.1 |
| YODAS and YODAS v3, ESPnet, Turkish | CC BY 3.0 | 4,696.2 |
ISSAI publishes the corpus under CC BY 4.0 (LICENSE.md of the TurkicASR repository); the Hugging Face copy is tagged MIT, and CC BY 4.0 is the licence followed here. The YODAS recordings are YouTube audio collected by YODAS and YODAS v3, whose authors say they took only videos published under a Creative Commons licence; they release the audio under CC BY 3.0. On YouTube the one Creative Commons licence an uploader can choose is the Creative Commons Attribution licence (CC BY). The recordings are the yodas0.10 subset of the OWSM v4 data, the yodas0.00 recordings not in it, and YODAS v3 audio cut along the spans of Alania Turkish Speech Style Captions (CC BY 4.0), of which only the span boundaries were used; its transcript column was never read.
For augmentation, noise from the MUSAN noise subset (CC BY 4.0; only files marked public domain or CC BY) was mixed into training audio at random, reverberation was added at random with simulated room responses from the OpenSLR room impulse response database (Apache-2.0), the input level was varied at random, and noise clips with an empty transcript were trained on as examples of non-speech.
3.2 Training text of the YODAS recordings
Automatic captions were never used as training text. Where an utterance has an uploader subtitle whose WER against the Whisper large-v3-turbo transcript is at most 0.2000, the subtitle is the training text. Otherwise, with no subtitle or with one that disagrees, that transcript is the training text, kept only where its character error rate (CER) against Omnilingual ASR CTC 7B v2’s reading of the same audio is at most 0.1000. Omnilingual ASR CTC 7B v2 also chose the spoken reading of number forms that have more than one. Of the 4,696.2 hours, 369.6 train on uploader subtitles and 4,326.6 on checked transcripts.
Whisper large-v3-turbo (MIT) and Omnilingual ASR CTC 7B v2 (Apache-2.0) were chosen on the TSC development split, which was never trained on. Their makers do not fully disclose their training data. No other output of these models is training text.
3.3 Decontamination
The normalised references of every development and test set used here form one exclusion set, matched against every training transcript and tokenizer line by exact match and by shared word n-grams; audio fingerprints matched the training audio against the evaluation audio. The matches removed 432 training utterances, 206 of them TSC clips matched by their audio. FLEURS, the Common Voice development and test splits, Real-TurnTurk and the 2,511 MediaSpeech clips reported in Section 4.4 were never trained on.
3.4 Training
The model was trained from randomly initialised weights, with one random seed. Design choices were tested with short pilot runs stopped at the same step and measured by WER over Common Voice and FLEURS dev together. CR-CTC in place of the intermediate CTC loss made WER worse by 7.0 percent in relative terms, a CTC-side weight of 0.1 instead of 0.3 by 1.2 percent and BPE-dropout at 0.1 by 3.7 percent, so none was adopted. Varying the input level at random changed WER at the original level by 0.03 percent and lowered it by 9.6 percent on development copies made 20 and 30 dB quieter, so it was adopted.
The run lasted 135,500 optimizer steps, with the learning rate lowered gradually at the end. The released model is the exponential moving average (EMA) weights of its last checkpoint, chosen on the development sets. From the first step to the released checkpoint, training ran on A100 80 GB GPUs for about 52 GPU session hours; the hours count setup, validation passes and checkpoint saves as well as training steps.
4. Evaluation
4.1 How the numbers were produced
Recipe, checkpoint and export choices were made on the Common Voice 27.0 Turkish and FLEURS Turkish development splits, and the models that write and check the training text were chosen on the TSC development split (Section 3.2); the test sets were scored once, for one model. Decoding is greedy, in float32, with the TDT decoder at most 10 tokens per frame. For normalised WER, reference and output go through the same Turkish normaliser, which lower-cases by Turkish rules, writes digits as words, folds circumflexes and removes punctuation, bracketed annotations and filler words such as ee and hmm.
Values in square brackets are 95 percent intervals from a blockwise percentile bootstrap (2,000 resamples, seed 0), in blocks of speakers for Common Voice and Real-TurnTurk and of sentences for FLEURS, which publishes no speaker ids. The interval of a difference between two systems is paired, with the same draws for both. The model was trained once, so the intervals come from resampling the test sets, not from repeated training. Whisper base and Whisper small (MIT; their maker does not fully disclose their training data) were scored by us with the same normaliser, greedy and with the Whisper package’s decoding settings (beam search of 5 with temperature fallback), and each comparison uses the better of the two, on every set here the second.
4.2 Release gate
The model was to ship only if it beat Whisper base on both FLEURS Turkish test and Common Voice 27.0 Turkish test, each by a paired interval lying wholly below zero. It passed on both. Whisper large-v3 (greedy) scores 0.0604 and 0.0767, lower than ufakzeka-kulak; its difference was not tested.
| Set | Utterances | Hours | ufakzeka-kulak WER | Whisper base WER | Difference to Whisper base | Difference to Whisper small |
|---|---|---|---|---|---|---|
| FLEURS Turkish test | 743 | 2.6 | 0.0921 [0.0792, 0.1055] | 0.2461 [0.2303, 0.2628] | -0.1541 [-0.1653, -0.1430] | -0.0421 [-0.0506, -0.0340] |
| Common Voice 27.0 Turkish test | 11,874 | 14.5 | 0.0916 [0.0873, 0.0958] | 0.3287 [0.3190, 0.3380] | -0.2372 [-0.2440, -0.2302] | -0.0885 [-0.0930, -0.0843] |
| Real-TurnTurk, conversational, not gating | 1,532 | 1.7 | 0.3807 [0.3270, 0.4640] | 0.7242 | -0.3435 [-0.4258, -0.3080] | -0.1367 |
A difference is ufakzeka-kulak’s WER minus the Whisper model’s, so a negative value means fewer errors. The normalised character error rate is 0.0335 [0.0258, 0.0426] on FLEURS test and 0.0242 [0.0228, 0.0258] on Common Voice test.
4.3 Punctuation and casing
The output was also scored as written, against references whose punctuation was mapped to the model’s marks. PC-WER also counts punctuation and capitals; PuncER and CaseER are the parts of it that come from each (case and punctuation error rates).
| Set | PC-WER | WER without marks or casing | PuncER | CaseER | Capital F-score | Share of references ending in a mark |
|---|---|---|---|---|---|---|
| FLEURS Turkish test | 0.1508 | 0.0890 | 0.4223 | 0.1878 | 0.8634 | 0.9960 |
| Common Voice 27.0 Turkish test | 0.1132 | 0.0963 | 0.1385 | 0.0530 | 0.9691 | 0.9832 |
| Real-TurnTurk | 0.4960 | 0.3838 | 0.8337 | 0.3817 | 0.3116 | 1.0000 |
The WER without marks or casing removes only the marks and the capitals, with the numbers in the references written as the model writes them, so it differs from the normalised WER above. The last column describes the references, not the model, since a set whose references seldom end in a mark penalises a model that writes one.
4.4 MediaSpeech Turkish
MediaSpeech (NTR Lab, CC BY 4.0) is media speech. Its makers say part of its utterances come from Mozilla Common Voice, so its clips were checked by text and audio against Common Voice and by audio against our training audio. Of its 2,513 clips none matched Common Voice and two matched training audio and were removed, so 2,511 clips (10.0 hours) are reported. The WER is 0.1042 [0.1013, 0.1073] and the CER 0.0422; the WER difference is -0.2240 to Whisper base and -0.1248 to Whisper small. The references join compound numbers into one word, which adds to the WER (the CER with spaces removed is 0.0420). The set has no speaker or video ids, so the interval treats every utterance as independent and may be too narrow, and Whisper’s training data is not fully disclosed, so Whisper may have seen this set.
4.5 Low bitrate, telephone and level
The three test sets of Section 4.2, but not MediaSpeech, were also scored on stored copies of their audio compressed with the Opus audio codec at 6,000 and 12,000 bit/s, in the 8,000 Hz telephone band with A-law coding, and 20 and 30 dB quieter. The table gives the change in WER against the original audio.
| Copy | FLEURS Turkish test | Common Voice 27.0 Turkish test | Real-TurnTurk |
|---|---|---|---|
| Opus, 6,000 bit/s | +0.0517 | +0.0752 | +0.0803 |
| Opus, 12,000 bit/s | +0.0043 | +0.0110 | +0.0135 |
| Telephone band | +0.0373 | +0.0491 | +0.0377 |
| 20 dB quieter | +0.0045 | +0.0009 | +0.0022 |
| 30 dB quieter | +0.0205 | +0.0051 | +0.0040 |
4.6 The released files
The numbers above are the NeMo file’s. On the test sets, the int8 sherpa-onnx TDT file reaches 0.0921 on FLEURS (difference to NeMo 0.0000) and 0.0932 on Common Voice (+0.0016), the int8 CTC file 0.1128 (+0.0207) and 0.1200 (+0.0284). The transformers TDT and CTC files give NeMo’s token ids on all 11,864 Common Voice and 338 FLEURS development utterances. The GGUF and MLX files were checked on the development sets only; their WER differences to NeMo on FLEURS dev range from 0.0000 to +0.0018. At one thread the int8 TDT file’s median real-time factor on 15 to 30 s utterances is 0.0101 on Apple M1, 0.0311 in Chrome there and 0.0364 on a 2-vCPU x86-64 Linux cloud server.
5. Limits
- It is trained for Turkish only. Other languages, and Turkish mixed with them, were not measured.
- Accuracy by speaker gender, age or regional accent was not measured.
- It writes out one utterance at a time. It is not a streaming model; long recordings have to be cut into utterances with voice activity detection first, as the demo does. Keep a few hundred milliseconds of audio around each utterance; a segment cut tight to the speech can come out without capitals or punctuation. It does not tell speakers apart and gives no checked word timestamps.
- It writes numbers as words. An application that needs digits has to convert them.
- It often misses commas. Its comma recall is 0.359 on FLEURS test and 0.612 on Common Voice test, against 0.839 and 0.939 for full stops.
- It is weak on conversational speech. On Real-TurnTurk its WER is 0.3807, and with 6 speakers the interval is wide.
- Its training text has errors. For 4,327 of the 4,696 hours of YODAS recordings, no uploader subtitle agreed with the transcript, so one speech recognition model wrote the training text and a second checked it; errors on which the two agreed remain in the training text.
- The sherpa-onnx files need an input gain. sherpa-onnx computes its features differently from NeMo, so the files were checked with the samples multiplied by 8, which holds at normal levels; on the FLEURS copy made 30 dB quieter the int8 TDT file’s WER rises by 0.0369, against 0.0205 for the NeMo file. The CTC file also needs zeros appended to the audio, or the last encoder frame, often carrying the final punctuation mark, can be lost.
- The MLX files were not measured on Apple silicon. They were checked on mlx’s CPU backend under Linux; the bfloat16 behaviour on Apple silicon may differ.
Attribution
The evaluation sets other than MediaSpeech (Section 4.4) are FLEURS (Google, CC BY 4.0), the Common Voice 27.0 Turkish dev and test splits (Mozilla Common Voice contributors, CC0 1.0) and Real-TurnTurk (Apache-2.0). Every training, augmentation and evaluation source is credited, with citations, in ATTRIBUTION.md in the model repository. Listing a dataset here does not mean that its authors, contributors, uploaders or publishers endorse the model or ufak AI.
Availability
The model, the files for every runtime, the Apache-2.0 LICENSE, ATTRIBUTION.md and the model card are in one repository on Hugging Face, and the code that built the model, with the result files its numbers come from, is on GitHub. The model can be tried in the browser at kulak.ufakzeka.com/en, a page that sends your voice nowhere, and its page is ufakzeka.com/en/ufakzeka-kulak. For suggestions or to report a problem, write to hello@ufakai.com.