A reproducible fine-tune with the measurement in plain sight: Vietnamese diacritics with CANINE-s

The problem

Anyone who wants to commission a fine-tune has a fair question: "show me a repository where you fine-tuned a model and measured something". Not a demo, but something you can clone, run and check. This is ours: a small fine-tune with the before/after measurement, the simple baselines it has to be compared against, and a section on what did not work.

What we did

The task. Restore Vietnamese diacritics: a sentence goes in with every mark stripped (đ becomes d) and the correct sentence comes out. We used CANINE-s (Google, about 0.13B parameters, character-level, no token vocabulary) with a classification head that predicts, for each character, which marks it carries (36 labels, one of them "none"). The text is rebuilt by composing each input character with its predicted marks, so the model cannot change a base letter: an assertion checks this on every test sentence (0 changes).

• Why Vietnamese and not Arabic. The standard Arabic diacritization corpus, Tashkeela, is GPL-2.0, and its re-uploads labelled MIT or Apache-2.0 are of unverified provenance (we did not check). The task has the same shape (restore marks on letters), so we used a corpus whose licence we could read at the source. Nothing has been run on Arabic.
• One command. ./run.sh creates the environment with pinned versions, downloads data and model, builds the splits, trains, evaluates and writes results.md and results.json. It needs Linux, an NVIDIA GPU, uv and about 0.8 GB of downloads.
• Split by article, no leakage. Each sentence goes to test, validation or training according to the SHA-1 hash of its article title, so one article is never in two parts. Exact duplicates are removed and an assertion checks that no training sentence appears in test. We used 400,000 training sentences, 2,000 validation and 5,000 test (2,314 articles).
• Simple baselines. A fine-tune only means something against something. We measured copying the input and a lookup of the most frequent form of each syllable seen in training (the baseline that matters), plus the model before fine-tuning.
• Confidence intervals. Sentence-level bootstrap, 1,000 resamples.

What we measured

Syllables are whitespace-separated tokens with punctuation stripped, and count only if they match the original exactly. Letter accuracy includes the many letters that carry no mark, which is why "copy the input" is not zero. The syllable error rate goes from 23.59 % with the frequent-form lookup to 5.39 %.

The "before fine-tuning" row is the pretrained encoder with a randomly initialised head. It shows the task is not solved by the pretrained model alone, but it is not a strong baseline: the lookup is the one that counts.

Rerun. We ran the whole pipeline twice from the same pinned environment. The three data files came out byte-identical (sha256). The model results did not: 94.58 % the first time and 94.61 % the second, because GPU kernels are not run in deterministic mode. Expect differences of a few hundredths of a point between reruns.

What it costs: GPU and VRAM

Estimated cost if the same card were rented, at RunPod's public list prices read from https://www.runpod.io/pricing on 2026-10-16 (the page was marked as updated 2026-10-08): RTX 4090, Community Cloud $0.34/h, Secure Cloud $0.89/h.

An estimate, not an invoice: prices change. We ran it on our own card and paid nothing for GPU time.

Error examples

Examples from the test set, with the Wikipedia article title. The input is the original sentence with its marks removed. Syllables that differ from the original are highlighted in pink. These are the first three of each list from a seeded shuffle, not selected.

Model right, baseline wrong

Model wrong

The most frequent error pairs are swaps between valid words: loại/loài, từ/tự, con/còn, cùng/cũng, quan/quân, cổ/có.

What still fails and what we did not measure

• Sentence-level success is modest. Under half of the test sentences (47.96 %) come out exactly right; 5.4 % of syllables are wrong, so a typical 22-syllable sentence has about one error (110,862 syllables over 5,000 sentences).
• Ambiguity the model cannot resolve from a sentence. Some pairs are genuinely ambiguous without wider context, and the model sees one sentence at a time.
• Spelling conventions are label noise. Wikipedia mixes hoá/hóa and Hoà/Hòa (tone mark placement). The model can only guess which style an article uses; here these count as errors.
• Names and foreign words. Those with no marks pass through unchanged; rare Sino-Vietnamese names are often wrong.
• Mistakes made on the way. The first training attempt crashed in validation because an impossible mark combination changed the string length (the base-letter assertion caught it; fixed). The first evaluation crashed because a torch native kernel needed a C compiler. In the first full run, make_results.py failed on a malformed number in timing.json, which was corrected by hand for that run only; the published run's timing.json was written by the fixed script. No logs of those failed attempts were kept, except the 1-epoch preliminary run.
• Not measured: variance across training seeds (one seed per run, two runs total); other learning rates or batch sizes; larger models or more data (validation still rose slowly at the end, 94.94 % to 95.21 % over the last 3,000 steps); text outside Wikipedia (news, chat, OCR); sentences longer than 200 characters; comparison with published Vietnamese systems (our split and filtering differ, so numbers are not comparable); CPU inference speed; any language other than Vietnamese. The test sentences have at least 5 diacritised letters, so scores on text with few marks may differ.
• Peak VRAM is the PyTorch allocator peak of our process on a card shared with other jobs; GPU time is the wall time of the train and eval steps including model loading.

What a client gets

The same thing you see here, applied to your data: a model fine-tuned for a product, character, style or task, a repository or folder you can run again, the simple baselines it is compared against, the measurement on your real cases before we hand it over, and an honest list of what fails and what we have not measured.

Custom models: from €450, excl. VAT. It is a starting price: after the first conversation, the proposal fixes scope and price. Details on the Technical teams page, or write to hello@grawwww.xyz. We answer in under 48 hours.

Credits and licences

• Model: google/canine-s (CANINE-s), Apache-2.0.
• Data: Vietnamese Wikipedia (2023-11-01 dump as published by Wikimedia on Hugging Face), CC BY-SA and GFDL. The text excerpts on this page are from Wikipedia, with the article title; authors are listed in each article's history. We do not redistribute the corpus or any trained weights.
• Our code, README and results: MIT licence.
• If you retrain and publish weights, read the CC BY-SA terms first. We have not taken legal advice on whether weights are derivative works, and we publish none.