ufakzeka-1: Building and Evaluating a 151M-Parameter Turkish Language Model from Scratch

The first model from ufak AI: pretrained from scratch on openly licensed data, fine-tuned for chat, benchmarked on one harness, limits measured, under $300 all in.

Technical report (PDF)Model (Hugging Face)GitHub repository Try the model

151Mparameters
13.5Bpretraining tokens
$286total cost
5,190 / 5,508sweep conversations passed

Abstract

This post presents the construction and evaluation of ufakzeka-1, a 151M parameter Turkish language model trained from scratch by ufak AI. The model was pretrained in three stages on 13.5B tokens drawn entirely from sources licensed for commercial use, then fine-tuned for conversation on 154,506 dialogues. The base and chat models were measured on seven Turkish tasks with the same harness as Kanarya-750M, turkish-gpt2-large and Qwen2.5-0.5B; conversational behaviour was tested with nine release gates, a rule-checked sweep of 5,508 conversations and a judged evaluation. The model’s failures were measured and are recorded in the model card with their numbers. The total cost was $286. The weights, the training data and method, the evaluation harness and the spend ledger are released under Apache-2.0.

1. Introduction

Open-weight language models for Turkish are mostly produced by fine-tuning a multilingual model on Turkish data. Small models built for Turkish from the tokenizer up are rare, and what they can and cannot do is seldom reported on a single harness.

ufakzeka-1 is a first step in that direction. Its purpose is not to be the best Turkish model but to show that a pipeline of data filtering, tokenizer training, pretraining, fine-tuning, evaluation and release works end to end, and to publish every step as measured. This post describes the model (Section 2), the data and training (Section 3), the evaluation (Section 4), the measured limits (Section 5) and the cost (Section 6); Section 7 states what changes for the next model. Details are in the model card and the technical report; the model can be tried at chat.ufakzeka.com.

2. Model

ufakzeka-1 uses the standard Qwen3 architecture: 24 layers, hidden size 768, 12 attention heads with 4 key-value heads (grouped-query attention), a SwiGLU feed-forward layer of width 2,048, rotary position embeddings with base 100,000, query-key normalisation and tied embeddings. It has 151M parameters without the embeddings and 182M with them. The context window is 4,096 tokens. Because the architecture is standard, the model loads in transformers without custom code.

The tokenizer is a byte-level BPE with 40,960 entries trained on Turkish text; digits are split individually. It produces 1.77 tokens per word on Turkish text. The pre-tokenizer is the Qwen2 pattern with the English contraction rule removed, since that rule splits apostrophe suffixes such as “Ankara’da” differently from training. As llama.cpp did not know this pattern at the time, a patch that changes 15 lines was written for the GGUF files and is released with them; an unpatched llama.cpp refuses the file. Switching to the Qwen2 rule that llama.cpp knows would run everywhere, but that rule splits every apostrophe suffix starting with d, t, s, m or v (Ankara’da becomes ’d + a) and raised perplexity on a short apostrophe-heavy Turkish sample (about 3,500 characters) from 15.4 to 19.0; the exact rule is therefore kept. The patch was merged into llama.cpp on 18 September 2026, so builds from that date run the files without it.

Two models are released: the base ufakzeka-1-base and the chat model ufakzeka-1. The chat model’s f16 (367 MB) and q8_0 (196 MB) GGUF files tokenize identically to transformers and give the same answers word for word on an eight-prompt greedy check; the mean logit difference was 0.002 nats for f16 and 0.03 nats for q8_0. No 4-bit file is released: on an earlier checkpoint of the same model, 4-bit quantisation worsened perplexity by 5 percent and changed greedy answers.

3. Data and Training

3.1 Pretraining data

All pretraining data comes from sources licensed for commercial use: Turkish Wikipedia (FineWiki), filtered web text (FineWeb2-HQ, mogan), an educational PDF collection (FinePDFs-edu), the TÜBİTAK BİLGEM and COSMOS synthetic corpora and a small share of English mathematics (FineMath). Datasets with unclear terms were excluded regardless of quality. The source list, licences and exclusions are given in the model card.

3.2 Pretraining

Pretraining ran on 13.5B tokens in three stages:

Main stage, 6.5B tokens. Muon for matrix parameters and AdamW for embeddings; a warmup-stable-decay learning rate schedule; 2,048 token context. The mixture was 25 percent curated text (Wikipedia, documents, synthetic), 71 percent web and 4 percent English mathematics; the last 15 percent annealed on a curated-heavy mix.
Continued pretraining, 5.5B tokens. Hyperball-constrained Muon; a tier of data rendered as question and answer, and a knowledge-heavy anneal.
Final anneal, 1.5B tokens. Logit soft-cap removed, context extended to 4,096 tokens; 25 percent of the data from August 2026 Turkish Wikipedia and summer 2026 CommonCrawl snapshots.

3.3 Fine-tuning

The chat model was trained on the stage-three base with supervised fine-tuning: 154,506 conversations (39,105 packed sequences), three epochs, learning rate 0.001, weight decay 0.1, embedding dropout 0.1. The loss was computed on assistant tokens, with prompt tokens weighted at 0.2, and 15 percent of the tokens were replayed from the pretraining text as whole documents so that fine-tuning forgets less of the language. By assistant words the mixture was: generated stories and dialogues 22 percent; public Turkish instruction sets (Turkish-SFT-Dataset-v1.0, diyalog-dataset, Turkce-Atlas-Instruct, Aya, everyday-conversations-tur, WikiRAG-TR) 27 percent; facts and Wikipedia question answering 13 percent; long sessions 8 percent; TinyStories-style short stories 8 percent; templated families for arithmetic, corrections, percentages, safety, abstention, identity and memory about 12 percent; the rest warm-up chat, poems and boundary cases. The generated sets were written by several large language models, filtered by rules and scored by an LLM judge before use.

Preference optimisation (DPO) was tried in three variants and found to lower conversation quality at this size; the released chat model is the fine-tuning checkpoint.

4. Evaluation

4.1 Benchmark tasks

Our models and the baselines were measured with lm-evaluation-harness 0.4.12, zero-shot by log-likelihood, with identical settings for every model. HellaSwag and ARC are reported as length-normalised accuracy, the other tasks as raw accuracy. TurBLiMP is the mean of 16 subsets, TurkishMMLU of 9 subjects.

Model	Params	HellaSwag	ARC-c	ARC-e	XCOPA	Belebele	TurBLiMP	TurkishMMLU
Kanarya-750M	750M	37.8	26.5	41.4	61.2	23.4	95.3	16.2
turkish-gpt2-large	774M	35.3	25.4	40.4	60.6	22.4	98.2	19.0
Qwen2.5-0.5B	494M	29.2	23.6	28.6	54.6	29.9	70.0	18.2
ufakzeka-1-base	151M	35.3	27.6	39.1	60.0	27.6	92.4	19.2
ufakzeka-1	151M	33.2	27.6	38.8	59.8	27.4	90.1	23.3
Random guess		25	25	25	50	25	50	20

The results should be read plainly. The tasks on which the Turkish-trained models sit well above random guessing are HellaSwag, ARC-e, XCOPA and TurBLiMP, and Qwen2.5-0.5B is well above it only on TurBLiMP; on ARC-c, Belebele and TurkishMMLU every model is within a few points of it. On ARC-e and XCOPA ufakzeka-1 is less than three points behind the two Turkish baselines, models five times its size; on TurBLiMP, which measures grammar, it trails them by 5 to 8 points. It is ahead of Qwen2.5-0.5B on every Turkish task except Belebele. Fine-tuning cost about two points against the base on HellaSwag and TurBLiMP and gained four on TurkishMMLU.

4.2 Release gates

Benchmark scores do not measure conversational behaviour. For that, nine release gates were defined that every version is expected to pass. The gates were generated at temperature 0.3 with the recommended sampling settings; none of the gate prompts appears in the training data, and that separation is asserted by an invariant script we run before each data build.

Gate (the behaviour that counts as a pass)	Passed / trials
After a long story, the user’s name is recalled correctly and not confused with a character’s	162 / 200
A correct arithmetic answer is kept under “are you sure” pressure	72 / 90
A wrong arithmetic answer is corrected when the user points it out	65 / 90
Subtraction with borrows through zero comes out right	30 / 30
A request naming chemical, biological or incendiary agents is refused, in phrasings not seen in training	64 / 64
A harmless question that merely sounds dangerous is answered, not refused	18 / 18
A correct answer is kept even when the user states a wrong number	16 / 36
An unknowable question is refused again when the user insists	13 / 24
Percentages and discounts come out right	40 / 40

In addition, a rule-checked sweep of 5,508 conversations across 22 families (identity, arithmetic, knowledge, common sense, unknowables, capability limits, advice, stories, poems, greetings, jokes, safety, over-refusal, name and fact memory, riddles, clarifying questions, long sessions) was run with two samples each; 5,190 passed. The weakest family is arithmetic at 807 of 1,000. In the evaluation scored by a fixed LLM judge at temperature 0, helpfulness was 79.2 with a deflection rate of 4.2 percent, over 118 greedy turns in the 39 of 40 multi-turn conversations the judge scored; everyday competence on what people actually ask (recipes, recommendations, health, proverbs, follow-ups) was 84.5, the mean over three generation seeds of 59 to 61 turns each. A hand test of 50 turns typed the way people type into a chat box, in lower case with typos and one-word follow-ups, was rated 27 turns good, 9 weak and 14 bad. The rating was made by the AI assistant used throughout development, not by an independent human rater, so the split is indicative rather than a human judgement. Every bad turn corresponds to one of the limits in Section 5.

4.3 Seed variance

Three models trained on the same data under different seeds gave 3, 9 and 18 failures on the 50-conversation identity gate of that round, and deflection rates of 4.2, 8.8 and 9.1 percent in the judged evaluation. All numbers above belong to the released checkpoint. At this size, a one-point difference between two models should not be read as meaningful.

5. Limits

ufakzeka-1 is a research model, not an assistant. Each limit below was measured and is written in the model card with its numbers.

It invents facts it does not have. Asked the distance from Ankara to İstanbul, it gives a number instead of saying it does not know. It abstains only on what it was taught to abstain on: time, date, weather, news, prices, personal details and the future. Pushed a second time on one of those, it holds the refusal in only 13 of 24 tries and otherwise invents an answer.
It cannot write code. Asked for Python, it produces prose.
It does arithmetic correctly one question at a time, showing its working. Asked the same sum again in a second turn with “are you sure”, it can repeat the working correctly and write the wrong number on the result line (it kept the answer in 72 of 90 attempts). A wrong number stated by the user moved it off a correct answer in 20 of 36 attempts.
After a long story it can confuse who is who. In the 200-conversation gate, the user’s name drifted into the story or an old name survived 38 times.
It can reproduce a training example word for word. Asked for a four-line poem about the sea, it returns the same poem, typo included, every time at temperature 0 and about half the time at 0.3. Asked for a riddle, it gave only four or five distinct riddles in 12 tries. This is a direct consequence of a small, repetitive fine-tuning set.
It knows nothing after summer 2026. Quantities in recipes are unreliable.

6. Cost

The total cost was $286. The total and the API line come from the bills: $37 of API for data generation and the judge model. The $66 for pretraining GPU is the 16.6 H100 hours of the three training runs at list price, and the $183 for fine-tuning and evaluation GPU and Colab is the remainder. Pretraining ran in August, fine-tuning and all measurements in September. The full ledger is in the repository. The figure is given for comparability, not as a claim about scaling.

7. Conclusion and Future Work

ufakzeka-1 shows that a pipeline built for Turkish from scratch works end to end. The resulting model trails Turkish models five times its size by less than three points on ARC-e and XCOPA and by 5 to 8 points on grammar, and carries its limits in plain view: inventing facts, breaking arithmetic on a second turn, and memorising training examples.

The direction for the next model is set: more varied fine-tuning data, since the memorisation comes from a small and repetitive set; a larger model, to test whether the limits that data did not move are limits of size; and the same gates again, with the numbers published in the same form. The size and the amount of data will follow the compute we can put behind it.

Availability

The model, the base model, the GGUF files and the model card are on Hugging Face; the training and evaluation code, the spend ledger and the technical report are in the GitHub repository. Questions and corrections to hello@ufakai.com.