900 Czech Questions for Five Local Decision Models
Nimble matched the reference on 11 more Czech test questions than Tev1 4B, but the two models did not make the same mistakes. We compared five local models, measured repeated-request latency, and examined the test data itself.
Jaroslav Urbánek, founder of TECHNOMATON 2 October 2026 15 min read
We tested answer selection on the Belebele dataset with five local models. Each processed all 900 Czech and 900 English questions. That gives us 4,500 answers in Czech and the same number in English. For every case, we retained the input, the model’s actual answer, and the comparison with the reference.
Nimble 9B had the highest agreement with the reference in Czech, at 91.2%, followed by Tev1 4B at 90.0%. Other details matter for practical decisions too: whether the models fail on the same questions, and what the measured time actually means. While reviewing individual answers, we also found a discrepancy in the Czech translation of the test.
What we compared and why
The starting point was Ollama’s endpoint for decision models, announced on September 29, 2026. A model receives context and a set of options described in advance. It returns a structured decision: a selected option (choice), a probability that a statement is true (noul), or a rating on a scale (score).
Such an output could, for example, route a customer request to billing, technical support, or a person for review. The model selects from the given options, and the application then acts on that result. This article focuses on passage comprehension and answer selection, not on the quality of a production application built around them.
We compared Tev1 0.8B Q8, Tev1 4B Q8, and Nimble 9B Q8 through Ollama, Laya Multilingual 322M FP16 through MLX, and Decider 2B v11 Q8 through llama.cpp. All inference ran locally on an Apple M3 Pro with 36 GiB of memory, using Metal. Ollama was pinned to version 0.35.0. Each result applies to a specific combination of model, weight precision, and runtime software.
The same task and a clearly defined metric
Belebele is a public reading-comprehension benchmark. Each short passage is accompanied by a question and four answer options; the dataset marks one as correct. We used the complete pinned Czech language version, ces_Latn, and English version, eng_Latn, with 900 questions each.
In the Czech version, the passage, question, and all options were in Czech. The general instruction was in English for both languages: “Answer the question using the passage only. Choose the best answer.” The prompt included no worked examples. A fully Czech instruction would be a separate experiment.
In this article, accuracy means agreement between the selected option and the dataset’s stored reference. The reference answer was not included in the model input. We did not truncate inputs or change our expectations after seeing the results. Beforehand, we checked that the prompts and options fit within Laya’s and Decider’s limits.
Tev1 and Nimble completed their runs on September 30, 2026; Laya and Decider did so on October 2. All five profiles received the same stored questions and references. The Czech results come from actual Czech answers, with no estimates based on English results and no imputation of missing cases.
Czech and English on the same questions
Dark bars represent Czech and light bars represent English. Each model keeps the same color throughout the article, and the shared scale starts at zero.
Dark bar: Czech. Light bar: English. Exact match between the selected answer and the reference.
Exact counts and model profiles
| Model | Czech | English |
|---|---|---|
| Tev1 0.8BQ8 · Ollama | 59.0 %531 of 900 | 72.7 %654 of 900 |
| Tev1 4BQ8 · Ollama | 90.0 %810 of 900 | 93.0 %837 of 900 |
| Nimble 9BQ8 · Ollama | 91.2 %821 of 900 | 95.4 %859 of 900 |
| Laya Multilingual322M · FP16 · MLX | 31.1 %280 of 900 | 30.6 %275 of 900 |
| Decider 2B v11Q8 · llama.cpp | 82.1 %739 of 900 | 89.9 %809 of 900 |
The highest accuracy in each column is shown in bold.
Eleven more matches still means different errors
Nimble matched the reference on 821 Czech questions, compared with 810 for Tev1 4B, a difference of about 1.2 percentage points. Comparing the same cases reveals an important detail:
- Both models matched the reference on 784 questions.
- Only Nimble matched on 37.
- Only Tev1 4B matched on 26.
- Neither model matched on 53.
Nimble’s result was not simply eleven additional reference matches. Switching from Tev1 4B to Nimble would mean 37 gained matches and 26 lost matches on this stored test. Higher overall accuracy does not guarantee that a particular decision handled by the original model will be preserved. That is why individual cases matter when changing models. We did not test whether the difference is statistically significant, and these counts do not establish a validated way to combine the two models.
English results are not enough to estimate Czech accuracy
For Tev1 0.8B, the English result exceeded the Czech result by 123 matches out of 900, or about 13.7 percentage points. For Tev1 4B, the difference was 27 matches, or 3.0 points. The language gap is therefore not a single constant that can be subtracted from every model’s English result.
Laya had similarly low agreement in both languages: 31.1% in Czech and 30.6% in English. Its result on this task cannot therefore be explained by Czech alone. Decider scored 82.1% in Czech and 89.9% in English.
A specific limitation in the reference: a reversed negative
While reviewing the stored answers, we found a case where the Czech translation changes the meaning of the options. The passage says the suspect was injured and that the authorities know his ethnicity. The question asks for the false statement. In the English source, line 557, option 4 reads “He was uninjured.” In the Czech source at the same line, it reads “Byl zraněn” (“He was injured”). The reference remains option 4 in both languages.
In Czech, option 3 also has the opposite meaning: it says that the authorities do not know the suspect’s ethnicity. Tev1 4B, Nimble, and Decider all selected option 3. In our reading of the Czech passage, they answered correctly, but automatic scoring marked them wrong. We checked the text and reference directly in the pinned source files; this was not introduced by our data conversion.
The figures in this article continue to use the original references. This single finding is not a full audit of the Czech translations and does not tell us how many similar cases the dataset contains. It does show that benchmark accuracy means agreement with a particular answer key, and why it is useful to inspect cases marked as errors.
The language versions are translations of the same questions, not independent samples. Questions based on the same passage should not automatically be treated as independent either. Belebele also does not test Czech text generation or dialogue. Because the dataset is public, we cannot rule out that some of its content appeared in a model’s training data.
Speed also depends on whether inputs repeat
On October 2, we measured all five profiles on the same Mac, with sleep disabled, running sequentially and without concurrent inference. The sample contained 20 Czech-English pairs selected by input length across five groups. After ten warm-up requests, we ran three passes of 40 cases: 120 measurements per model, 600 in total.
Time covers the full local HTTP POST for one question, excluding model loading. Values in the chart are rounded to the nearest millisecond. We did not clear the cache between passes, so the aggregate median describes a repeated sample.
Dark bar: median (p50). Light bar: 95th percentile (p95), the time within which 95% of requests completed. Time covers the full local HTTP POST for one question; model loading is excluded.
The cache was not cleared between passes. These times describe this repeated sample and the runtime used; they do not guarantee latency for a new input.
An aggregate median can hide a change in ranking
Tev1 4B and Decider have similar aggregate medians: 222 and 231 ms. In the first pass, however, Tev1 4B’s median was 885 ms, compared with 235 ms for Decider. In the third pass, they were 136 and 232 ms. The ranking of which one is faster therefore reverses depending on the pass.
Nimble’s median also fell, from 1,927 ms in the first pass to 196 ms in the third. We did not isolate the cause of this speed-up. The model order was fixed, and warm-up used a case from the sample, so even the first pass is not a clean test of entirely new inputs.
The practical implication is clear: a speed measurement should state whether the model is processing new or repeated requests. These aggregate times cannot be used as expected performance for a stream of new documents or tickets without further testing. Concurrent production load has not yet been evaluated.
How times changed across the three passes
Median for the full request, with 40 measurements per pass. Times are rounded to the nearest millisecond.
| Model | Pass 1 | Pass 2 | Pass 3 |
|---|---|---|---|
| Tev1 0.8B | 191 ms | 80 ms | 38 ms |
| Tev1 4B | 885 ms | 168 ms | 136 ms |
| Nimble 9B | 1,927 ms | 596 ms | 196 ms |
| Laya Multilingual | 50 ms | 33 ms | 51 ms |
| Decider 2B v11 | 235 ms | 229 ms | 232 ms |
On a phone, swipe horizontally to scroll the table.
Laya had the shortest median, 45 ms, but also the lowest agreement with the reference on Czech Belebele. The scatter plot therefore shows time alongside accuracy. A strong result on one dimension does not by itself determine suitability for a particular application.
Accuracy is from the full set of 900 Czech questions. Time is from a repeated sample of 40 Czech and English cases; the cache was not cleared. The chart compares specific local profiles, not the models’ independent speed.
The original timings from a broader mix of tasks on September 30, during which the Mac went to sleep, remain archived. Here we use only the shared measurement of all five profiles from October 2.
Other tasks change the model ranking
Only the two Tev1 profiles and Nimble completed the other public datasets; Laya and Decider have not been measured on them. The broader run covered 6,004 cases per model: Belebele, 13 pinned public subsets from the Bespoke Labs benchmark suite, totaling 3,880 cases, and a separate 324-case synthetic Nimble holdout.
The chart shows five selected English tasks. The metric is exact label match; for binary tasks, we use a probability threshold of 0.5. We do not combine different tasks into a single score.
MASSIVE
350 cases · routing across 18 domains
MultiNLI
299 cases · relationship between statements
PAWS
250 cases · sentence meaning equivalence
BoolQ
300 cases · yes/no based on a passage
VitaminC
599 cases · claim verification
Shared 0–100% scale for each task. Exact label match; probability threshold of 0.5 for binary tasks. Laya and Decider were not evaluated on these subsets.
Nimble led on passage comprehension in Belebele, but on PAWS, which compares sentence meaning, it had 188 matches out of 250, versus 205 for Tev1 4B. In this measurement, a higher result on one task did not mean a higher result on another. The difference between Tev1 4B and Nimble was smaller on MASSIVE EN routing: 297 versus 294 matches out of 350. These English results alone do not establish the quality of Czech request classification.
The scoring method also matters
For rating tasks, in addition to exact agreement with the most likely level, we calculated MAE, the mean absolute deviation of the weighted score from the reference scale index. A lower value means a smaller deviation; it is not a percentage of correct answers.
HelpSteer2
249 cases · mean absolute error of the weighted score
SummEval · consistency
144 cases · mean absolute error of the weighted score
SummEval · relevance
240 cases · mean absolute error of the weighted score
Shared scale from 0 to 1.25 MAE. A longer bar means a larger error; the value is not a percentage of correct answers. Laya and Decider were not evaluated on these subsets.
Nimble had the lowest MAE on all three subsets shown. Yet on SummEval · consistency, Tev1 0.8B more often selected the exact reference level: 121 of 144 cases, compared with 119 for Nimble. Nimble still had a lower weighted-score error, 0.386 versus 0.645. The two metrics can therefore produce different rankings because they answer different questions. Before evaluating a result, decide whether exact classification or the size of the deviation matters.
What to take from these measurements when evaluating your own system
On this dataset, Nimble 9B and Tev1 4B had the highest agreement with the Czech reference. Decider ranked between them and the two smallest profiles. The faster Laya showed that a short response time can coincide with low accuracy on a given task. That is the extent of what we can conclude from Belebele.
These results suggest four practical checks when evaluating a specific application:
- Compare individual errors. When changing models, inspect cases that got worse as well as those that improved. An aggregate gain can hide regressions.
- Read the references for disputed cases. Do not change them automatically to match the model’s answer. Record and version any data correction separately so it remains clear what was measured against.
- Make the timing test resemble actual use. Separate new from repeated inputs, model loading, and the expected concurrency.
- Choose data and metrics for the decision. Passage comprehension, request classification, and scale-based ratings are different tasks. Misrouting a request can also have a different impact from wrongly approving a refund.
A Czech operational dataset labeled by people, changes to a decisive fact, permutations of the answer options, and a fully Czech instruction are all useful follow-up tests. They were not part of this measurement.
Traceability and scope of verification
The audit recalculated all 18,012 answers from the broader run and another 3,600 answers from Laya and Decider. It checked that inputs and references matched the pinned datasets, case identities, model digests, stored answers, complete coverage, and metrics. It did not involve a new human review of the correctness of every reference. A separate timing audit checked 600 measured and 50 warm-up requests. The comparison of errors between Tev1 4B and Nimble in this article came from the same stored answers, with no new model calls.
Every request returned a valid answer without an API error. This verifies completion and response format; substantive correctness is assessed separately by evaluating the content. Dataset hashes, model checksums, configurations, raw answers, and recalculated summaries have been retained.
Laya’s MLX port was also checked against the original weights: all 170 tensors matched, and the same most likely option was selected on 20 check questions. This verifies the model conversion; it is not an additional accuracy measurement.
We did not call production Jev. The published Jev comparison has its own provenance and partly different methodology. This article reports results for five specific local profiles, using the original references and the run conditions described here.