Skip to content

Lab · Test bench

Test bench: tools compared

We give AI tools the same tasks on the same test data: Jev by TypeSafe, the GLiNER classifiers by Fastino and open language models on our own GPU. Every number on this page comes from the public lab-bench repository, with cases, scripts and results.

To the tool maplab-bench repository

Tool map

Which tool handles which task?

Three kinds of AI tools, the same tasks, the same test data. Every cell is a measurement from our test bench or says why it is not one. The colour shows how far a tool lands above plain guessing.

TaskJevDecision model by TypeSafe, hosted service in the USAGLiNERClassifiers and extractors by Fastino, run locally on one GPULocal language modelOpen models through Ollama, run locally on one GPU
Assign a topicPut a short spoken command into one topic from a fixed list (MASSIVE dataset).Source: test bench, results/route1/findings.mdJev74.1%Guess line 13.5%far above guessing09 Oct 2026 · n = 2,974variant jev, model jev-1.13.0GLiNER60.9%Guess line 13.5%clearly above guessing09 Oct 2026 · n = 2,974variant gliner2.5-decide, model fastino/GLiNER2.5-DecideLocal language model70.6%Guess line 13.5%clearly above guessing09 Oct 2026 · n = 2,974variant winnow-12b, model EldanRing/Winnow-12B:Q8_0
Assign an intentWith the topic known, pick the exact intent, only questions with more than one option (MASSIVE).Source: test bench, results/route1/findings.mdJev92.0%Guess line 50.2%far above guessing09 Oct 2026 · n = 2,622variant jev, model jev-1.13.0GLiNER78.3%Guess line 50.2%clearly above guessing09 Oct 2026 · n = 2,622variant gliner2.5-decide, model fastino/GLiNER2.5-DecideLocal language model91.6%Guess line 50.2%far above guessingdifference not significant09 Oct 2026 · n = 2,622variant winnow-12b, model EldanRing/Winnow-12B:Q8_0
Routing on GLiNER's home dataSort short texts into fixed classes, fast-decisions dataset. Fastino, the maker of GLiNER, built it, so a tie here is not a neutral result.Source: test bench, results/route1/findings.mdJev63.0%Guess line 31.2%clearly above guessing09 Oct 2026 · n = 2,500variant jev, model jev-1.13.0GLiNER64.4%Guess line 31.2%clearly above guessingdifference not significant09 Oct 2026 · n = 2,500variant gliner2.5-decide-1b-intext, model fastino/GLiNER2.5-Decide-1BLocal language model63.6%Guess line 31.2%clearly above guessingdifference not significant09 Oct 2026 · n = 2,500variant winnow-12b, model EldanRing/Winnow-12B:Q8_0
Area of a requirementFor one line of a project posting, pick the subject area.Source: test bench, results/pv1/score-en.md, reference claude-c · more on thisJev91.6%Guess line 16.0%far above guessing01 Oct 2026 · n = 119variant jev, model jev-1.13.0GLiNER62.2%Guess line 16.0%clearly above guessing08 Oct 2026 · n = 119variant gliner2.5-decide-1b-focus, model fastino/GLiNER2.5-Decide-1BLocal language model88.2%Guess line 16.0%far above guessingdifference not significant01 Oct 2026 · n = 119variant winnow-12b, model EldanRing/Winnow-12B:Q8_0
Extract detailsPull dates, places, names and similar details out of a spoken command, word for word (MASSIVE). Scored as F1 over all details.Source: test bench, results/ex1/findings.md · no significance testJevnot applicableJev chooses, scores and answers yes or no. It does not return text spans.GLiNER38.1%Guess line 0.0%not above guessingColour by whole frames: 22.4% against 33.5% when extracting nothing09 Oct 2026 · n = 1,000variant gliner2-large, model fastino/gliner2-large-v1Local language model48.9%Guess line 0.0%just above guessingColour by whole frames: 44.9% against 33.5% when extracting nothing09 Oct 2026 · n = 1,000variant winnow-12b, model EldanRing/Winnow-12B:Q8_0
Find evidence in a CVName the CV entry that supports a requirement, or say that there is none.Source: test bench, results/pv1/score-en.md, reference claude-c · more on thisJev79.0%Guess line 37.0%far above guessing01 Oct 2026 · n = 119variant jev, model jev-1.13.0GLiNER14.3%Guess line 37.0%not above guessing08 Oct 2026 · n = 119variant gliner2.5-decide, model fastino/GLiNER2.5-DecideLocal language model82.4%Guess line 37.0%far above guessingdifference not significant01 Oct 2026 · n = 119variant qwen3.8-27b, model qwen3.8:27b
Rate the levelRate how far the CV meets a requirement, on a fixed scale.Source: test bench, results/pv1/score-en.md, reference claude-c · more on thisJev74.8%Guess line 34.5%clearly above guessing01 Oct 2026 · n = 119variant jev, model jev-1.13.0GLiNER38.7%Guess line 34.5%just above guessing08 Oct 2026 · n = 119variant gliner2.5-decide, model fastino/GLiNER2.5-DecideLocal language model70.6%Guess line 34.5%clearly above guessingdifference not significant01 Oct 2026 · n = 119variant winnow-12b, model EldanRing/Winnow-12B:Q8_0
Yes or no about an invoiceAnswer a yes or no question about the content of an e-invoice.German onlySource: Test design on the test benchJevTest plannedTest design fixed on 9 October 2026: 34 public test invoices (KoSIT), ten questions. Before any tool is scored, a blind reference for all 340 answers is built.GLiNERTest plannedTest design fixed on 9 October 2026: 34 public test invoices (KoSIT), ten questions. Before any tool is scored, a blind reference for all 340 answers is built.Local language modelTest plannedTest design fixed on 9 October 2026: 34 public test invoices (KoSIT), ten questions. Before any tool is scored, a blind reference for all 340 answers is built.
Response timeMedian per decision when assigning a topic. The three clocks measure different things and are not directly comparable.Source: test bench, results/route1/score.mdJev231 msRound trip over the network to TypeSafe in the USA09 Oct 2026 · variant jevGLiNER26 msCompute time on the GPU only, no network09 Oct 2026 · variant gliner2.5-decideLocal language model92 msCall over the local network to the GPU09 Oct 2026 · variant winnow-12b
strong atJevAssign a topic · Assign an intent · Routing on GLiNER's home data · Area of a requirement · Find evidence in a CV · Rate the levelGLiNERRouting on GLiNER's home dataLocal language modelAssign an intent · Routing on GLiNER's home data · Area of a requirement · Find evidence in a CV · Rate the level

How to read the colours

  • far above guessing
  • clearly above guessing
  • just above guessing
  • not above guessing

The colour and label of each cell show the distance from guessing, not the raw hit rate: how much of the way from the guess line to error free a tool covers, in rough thirds. Guessing means always giving the most frequent answer. For extraction the colour counts whole frames against extracting nothing, while the number stays F1. "Difference not significant" marks cells where the difference to Jev is not significant in a paired test. The compared variant was picked as the best of its family after the measurement, and the test is not corrected for that. Below each number are the date, number of cases, variant and model.

"Strong at" lists the tasks where the tool is in the leading group: it has the highest value there, or its difference to the highest value is not significant in a paired test. It also has to be at least clearly above guessing. Tasks without a significance test do not count.

  • For each family the map shows the best variant, chosen on the same test data it is measured on. That favours families with many variants: GLiNER had up to 9, the local language models up to 4, Jev one.
  • Every number comes from a single run per variant.
  • Area, evidence and level are measured against a reference made by one language model, without a cross-check for the English rows. There is no human sample yet.
  • Two questions from the same test bench are left out on purpose: whether a line is a requirement at all, and whether it is mandatory. The reference calls 85.3% to 85.6% of all lines a requirement and 94.0% to 94.1% of the requirements mandatory, so always saying yes is already almost right and the raw rate misleads.
  • More on this in Experiment 3 on the decision models page. For the German rows it counts against the first judge, not against the adjudicated reference, so its German numbers differ slightly.

Automation

How much can be automated?

The curve shows what share of cases a tool can decide on its own if it only keeps the answers where its own probability is high enough, and the kept answers still hold the chosen target accuracy. A person checks the rest.

Task
Language of the test data
95%
Target accuracy of the kept answers

At 95% accuracy, decided without a person:

  • Jev: 42.6% of cases
  • GLiNER: 10.8% of cases
  • Local language model: 41.3% of cases
  • Classically trained: 87.7% of cases
  • Jev
  • GLiNERVariant gliner2.5-decide-1b
  • Local language modelVariant winnow-12b
  • Classically trainedVariant classic-e5-lr

Classically trained: counted on 5,596 of the 8,196 cases, the only ones with training data.

The classically trained classifier learns from the training split of MASSIVE, the other tools see not a single example. The match test has no training split.

Read with care: the threshold is chosen on the same data it is counted on, so the values are optimistic. The match test rests on only 600 German and 615 English cases, so its values move by several points. And Jev and the local language model claim too much at the top of the scale: when routing German commands, Jev is right in 93 percent of the cases where it states 0.98, Winnow-12B in 85 percent where it states 0.99.

Source: T1 findings in the lab-bench repository

The numbers as a table
Routing, German, 5,596 cases
Toolat 90%at 95%at 99%
Jev78.2%60.3%0.0%
GLiNER (gliner2.5-multi-decide)12.7%4.4%0.5%
Local language model (winnow-12b)75.4%58.0%0.0%
Classically trained (classic-e5-lr)93.3%76.4%39.4%
Routing, English, 8,196 cases
Toolat 90%at 95%at 99%
Jev64.7%42.6%0.0%
GLiNER (gliner2.5-decide-1b)23.3%10.8%1.1%
Local language model (winnow-12b)58.7%41.3%0.0%
Classically trained (classic-e5-lr)100.0%87.7%52.5%
Match, German, 600 cases
Toolat 90%at 95%at 99%
Jev90.8%53.5%40.3%
GLiNER (gliner2.5-multi-decide-focus)38.5%28.5%0.2%
Local language model (qwen3.8-27b)90.5%68.2%44.5%
Classically trainednot measurednot measurednot measured
Match, English, 615 cases
Toolat 90%at 95%at 99%
Jev79.3%51.9%21.1%
GLiNER (gliner2.5-decide-1b-focus)38.9%24.6%21.0%
Local language model (qwen3.8-27b)85.7%56.4%33.8%
Classically trainednot measurednot measurednot measured

Method

How we measure

Five decisions shape how to read the numbers. Each has a limit, and the limit is stated with it.

  1. Reference

    Every answer is scored against a reference judgment. For MASSIVE and fast-decisions these are the dataset's own labels. For the CV match, language models judged blind and a separate adjudicator decided the disagreements. A human sample is still missing.

    How judges and models decide, in Experiment 3 →

  2. Guessing line

    Next to every rate stands what you get without a model: always give the most frequent answer, or extract nothing at all. The colour of the map shows the distance to this line. Where the reference almost always says yes, a raw rate would look better than it is.

    A classifier on the guessing line →

  3. Best variant per family, chosen after the run

    For each family the map shows the best variant, chosen on the same test data it is measured on. That favours families with many variants. The paired test against Jev is not corrected for it. A difference that is not significant only means the gap is not established. It does not make the two equally good.

    What we measured and what we did not →

  4. One run per variant

    Every number comes from a single run. A second run can come out differently, for language models already because they sample when they answer. How much that matters we measured on the CV match with several runs and shuffled options.

    Shuffle the options: Experiment 5 →

  5. Test design before the first run

    A new test first gets a dated design in the plan/ folder of the repository: data, questions, variants, reference, metric and thresholds. Only then does it run. That way the design cannot be fitted to the results afterwards.

    Set the line in advance: Experiment 2 →

The tests

Four tests, each with its own question

Each test has its own data, its own reference and its own reports in the repository. The sources in the tool map lead to these sections.

pv1measured

CV match: CV against posting

Question
Does a CV fit a project posting? For every line of the posting a tool answers five questions: is it a requirement, is it mandatory, which of eight areas does it belong to, which CV entry is the evidence, and how well does it cover the line.
Data and licence
Four invented postings and eight invented CVs, German and English, combined into ten pairs. Because everything is invented, every tool may see every case, including a service in the USA. The cases are in the repository under CC BY 4.0.
Tools
Jev, four open language models through Ollama, among them Winnow-12B and Qwen3.8-27B, and nine variants of GLiNER2.5 Decide.
Result
Jev leads on area and level. Qwen3.8-27B finds the right evidence most often, 87.9 percent in German against 81.0 percent for Jev. On the mean of the five questions GLiNER2.5 Decide does not get past the most frequent answer.

Read more

route1measured

Routing: short texts into fixed classes

Question
Which of several fixed classes does a short text belong to? This is the task GLiNER2.5 Decide is built for. We ask for the topic of a spoken command, the exact intent once the topic is known, and the class in Fastino's own dataset.
Data and licence
MASSIVE 1.1 by Amazon, test split, 2,974 spoken commands each in German and English, licence CC BY 4.0. Plus the dev split of fast-decisions, published by Fastino on Hugging Face, English only, licence Apache 2.0.
Tools
Jev, the local language model Winnow-12B and six variants of GLiNER2.5 Decide. Since test T12 also eleven more open models, shown under Quality against speed.
Result
On MASSIVE Jev leads every group, with Winnow-12B close behind. GLiNER is 13 to 32 percentage points behind Jev. On fast-decisions the English Decide models, Jev and Winnow-12B reach 63.0 to 64.4 percent, and none of these differences is established.

Read more

ex1measured

Extraction: details in their own words

Question
Which details does a spoken command contain, such as date, time, place or name, and in which words? We count F1 over all details, a measure that combines precision and completeness.
Data and licence
1,000 spoken commands per language from the MASSIVE 1.1 test split, with the dataset's annotation as the reference, licence CC BY 4.0.
Tools
The local language model Winnow-12B and two GLiNER extractors, since test T12 also eleven more open models. Jev is not part of it, because it chooses and scores but returns no text spans.
Result
Winnow-12B reaches 45.1 percent in German and 48.9 percent in English, the best GLiNER 22.5 and 38.1 percent. Only Winnow-12B gets whole commands right more often than extracting nothing would. GLiNER is 13 to 21 times faster in exchange.

Read more

einv1planned

E-invoice: yes or no from the free text

Question
Does the free text of an e-invoice state a payment due date, a cash discount or a direct debit? Ten fixed yes/no questions per invoice, answered from the text only, not from the structured fields.
Data and licence
34 public test invoices from the KoSIT XRechnung test suite, licence Apache 2.0. Our page "Jev reads e-invoices" uses the same invoices.
Tools
Jev, GLiNER and a local language model, scored against a blind reference for all 340 answers.
Result
None yet. The test design has been in the repository since 9 October 2026, fixed before the first run and before the reference.

Read more

Local models

Quality against speed

Test T12 measured 11 open language models from 3.4 to 32.8 billion parameters on a single 24 GB graphics card, an RTX 4090 with Ollama. Winnow-12B joins from the run on the same day. All of them got the same questions as Jev for routing and extraction. Each dot is a model: further right means faster, higher means more accurate, and a larger dot needs more GPU memory.

Task
60.0%

12 of 12 models reach at least 60.0%. Fastest of them: Qwen3 4B with 92,880 decisions per hour.

Decisions per hour, log scale

Qwen3.8 27B

  • Accuracy: 79.1%
  • 16,632 decisions per hour
  • 18.7 GiB of GPU memory

Tap a dot or reach it with the Tab key to read another model.

  • area by GPU memory, here 5.0 GiB and 20.0 GiB
  • dashed: does not fully fit into 24 GB
  • upward arrow: accuracy is a lower bound
  • ring: the fastest model above the minimum quality
  • Jev as a service, for comparison

How full does the card get?

Pick a model or tap a dot in the chart. The bar shows how much GPU memory it took during the runs, together with what the desktop already used. Whatever no longer fits on the card runs on the CPU, and that slows it down.

Qwen3.8 27B

  • peak 20.5 GiB of 24.0 GiB
  • of which 1.8 GiB for the desktop before loading
  • 16,632 decisions per hour, 207 ms per decision

The race

All models get the same questions and work through them one after the other. Each bar fills at the measured speed, sped up. Once it is full, the result shows how many were right.

Task

Sped up: one second here is 312 seconds of the run. Every model answers the same 14,396 questions. The number right is computed from the rounded accuracy.

  1. Qwen3 4B9,962 of 14,396 right25.8 per second
  2. Granite 4 Micro9,185 of 14,396 right25.4 per second
  3. Phi-4 14B10,883 of 14,396 right22.6 per second
  4. Qwen3 14B10,509 of 14,396 right19.1 per second
  5. Gemma 4 E4B11,056 of 14,396 right18.4 per second
  6. Mistral Small 3.210,581 of 14,396 right16.6 per second
  7. Qwen3 30B-A3B9,588 of 14,396 right14.3 per second
  8. Shisa DE-110,927 of 14,396 right12.0 per second
  9. Gemma 4 12B11,027 of 14,396 right10.5 per second
  10. Winnow-12B11,272 of 14,396 right9.9 per second
  11. Qwen3 32B10,783 of 14,396 right5.1 per second
  12. Qwen3.8 27B11,387 of 14,396 right4.6 per second

Qwen3 size ladder

Before the first run we named four Qwen3 models as representatives of the size classes. On MASSIVE the line rises with size, on fast-decisions it stays flat. The mixture of experts model (MoE) is below the smallest on both.

69.4%74.0%76.5%66.9%MASSIVE59.7%60.9%60.6%56.2%fast-decisions

Test fixed in advance: Qwen3 32B against Qwen3 4B

Grouponly Qwen3 32B rightonly Qwen3 4B rightp (McNemar)
MASSIVE scenario, German459168< 0.001
MASSIVE scenario, English368166< 0.001
MASSIVE intent, German316129< 0.001
MASSIVE intent, English229122< 0.001
fast-decisions, English1921680.225

Counted are the questions where exactly one of the two models was right. In the chart MASSIVE combines the four groups, weighted by the number of questions.

What the series shows

  • Qwen3.8 27B is the most accurate local model on both tasks. In routing it reaches 79.1%, Jev 79.4%. On MASSIVE the two are no more than half a point apart in any group. In extraction it reaches 54.3% in German and 55.6% in English. It is also the slowest model that fits fully on the card: about 16,600 decisions per hour with 18.7 GiB of GPU memory.
  • Gemma 4 E4B gives the most for little memory: 5.2 GiB, about 66,300 decisions per hour and 76.8% accuracy. That is more than Qwen3 32B with 74.9%, which takes 21.3 GiB.
  • Within Qwen3, size helps on MASSIVE. Accuracy rises from 4B to 14B to 32B, and the large against small test fixed in advance is significant on all four groups there. On fast-decisions size does not help: Qwen3 4B reaches 59.7%, Qwen3 32B 60.6%.
  • On fast-decisions no local model reaches the bar at which the test bench counts a task as usable, half the way from guessing to error free. All of them land between 54.9% and 63.6%, Jev at 63.0%.
  • Mistral Small 3.2, Qwen3 30B-A3B and Phi-4 often started with text instead of an answer letter, on 48%, 44% and 32% of the questions. Those answers are scored from the leftover letter probabilities. Their accuracies are therefore lower bounds, marked with an upward arrow in the chart.
  • Qwen3 32B does not fully fit into the 24 GB at the fixed context of 16,384 tokens, part of it runs on the CPU. That explains much of its slowness.

Read with care: speed means one request after the other, from a Mac over the local network. Batched or parallel, the same card does more, which test T10 measures. The best models are named after the run on the same data, and there is no test between them. Each model ran once. The test data is public, and whether a model saw it in training is not checked. Jev's speed is not in the chart, because Jev runs as a service in the US and its clock measures something else.

Sources: T12 findings · Chart data

All models as a table
Local models on one 24 GB card, by accuracy
ModelRouting accuracyMASSIVEfast-decisionsDecisions per hourF1 GermanF1 EnglishCommands fully rightSpoken commands per hourGPU memoryon the CPUfully on the card
Qwen3.8 27B 27.3 bn, Q4_K_M79.1%81.6%62.0%16,63254.3%55.6%990 of 2,0009,75618.7 GiB0.0 GiByes
Winnow-12B 11.9 bn, Q8_078.3%80.1%63.6%35,60445.1%48.9%900 of 2,0009,360about 13.7 GiBn/ayes
Gemma 4 E4B 7.5 bn, Q4_K_M76.8%78.8%61.1%66,34839.3%42.2%495 of 2,00016,2365.2 GiBn/ayes
Gemma 4 12B 11.9 bn, Q4_K_M76.6%78.4%61.6%37,94447.9%51.9%917 of 2,00017,7489.3 GiBn/ayes
Shisa DE-1 25.2 bn, Q4_K_M75.9%77.8%61.0%43,20046.3%49.3%752 of 2,00015,22817.7 GiBn/ayes
Phi-4 14B 14.7 bn, Q4_K_M75.6% ↑77.7%59.7%81,36037.8%43.6%665 of 2,00012,60012.0 GiB0.0 GiByes
Qwen3 32B 32.8 bn, Q4_K_M74.9%76.5%60.6%18,32439.8%44.3%588 of 2,0005,36421.3 GiB1.9 GiBno
Mistral Small 3.2 24 bn, Q4_K_M73.5% ↑74.2%62.8%59,65236.4%39.5%411 of 2,0005,61617.0 GiB0.0 GiByes
Qwen3 14B 14.8 bn, Q4_K_M73.0%74.0%60.9%68,90435.1%43.9%611 of 2,00011,05211.6 GiB0.0 GiByes
Qwen3 4B 4 bn, Q4_K_M69.2%69.4%59.7%92,88026.3%36.1%652 of 2,00027,9005.0 GiB0.0 GiByes
Qwen3 30B-A3B 30.5 bn, Q4_K_M66.6% ↑66.9%56.2%51,55233.9%39.0%536 of 2,00014,47219.2 GiB0.0 GiByes
Granite 4 Micro 3.4 bn, Q4_K_M63.8%63.6%54.9%91,26019.5%28.4%690 of 2,00031,5723.8 GiB0.0 GiByes
Jev as a service79.4%81.8%63.0%n/an/an/an/an/an/an/an/a

Accuracy pooled over all MASSIVE questions in both languages and the fast-decisions tasks with at most 26 options, 14,396 questions, for Jev by the same rule. The arrow marks a lower bound. GPU memory is the peak during the runs minus the idle value before loading, for Winnow-12B the value after loading. n/a means not available.

Capabilities

What the machine can do

The second test series runs without Jev. It asks what a workstation with one 24 GB graphics card can do in house: per capability a public data set, a test design fixed in advance and several open models. Each dot is a model. The higher, the better, the further right, the more GPU memory it needs.

  • the model the map shows
  • tied, the test does not separate the two
  • bar: 95% interval
  • baseline without a model
  • still running or did not finish
  • Dots with almost the same GPU memory sit slightly side by side, the exact values are in the table.

C1measured

German search for RAG

Which model finds the right passages for a German question? MIRACL German with 305 questions, scored as nDCG@10.

GPU memory in GB, the card has 24

qwen3-emb-8b with reranker bge-reranker-v2-m3

  • nDCG@10 0.640, 95% interval 0.605 to 0.677
  • 21.8 GB of GPU memory
  • 44 passages per second when indexing, 209.4 ms per query
  • shown on the map
  • tied with qwen3-emb-8b with reranker Qwen3-Reranker-0.6B, p 0.60

Tap a dot or reach it with the tab key to read another model.

The map shows qwen3-emb-8b with reranker bge-reranker-v2-m3, tied with qwen3-emb-8b with reranker Qwen3-Reranker-0.6B (p 0.60). Both need the same GPU memory, so the better value is on the map.

  • Qwen3-Embedding-8B is the best search model, at 0.613. The price is memory and time: 21.8 GB of GPU memory and only 44 passages per second when indexing.
  • A reranker that re-sorts the top hits lifts this to 0.640 with bge-reranker-v2-m3, established against the search model alone (p 0.036). A query then takes 209.4 ms instead of 53.7 ms. Qwen3-Reranker-0.6B is tied at 0.634 but needs 1,078.2 ms per query.
  • bge-m3 at 0.577 and e5-large at 0.564 need about 3.0 GB and index about 1,189 passages per second. For a large stock or one that changes daily they are the practical choice. That is advice, not a measured equivalence.
  • Keyword search with BM25 reaches 0.199. The data set deliberately holds many passages that share the words of the question. This is a floor for this corpus, not a verdict on BM25 over company documents.

Read on GitHub: Findings · Report with all numbers · Test design

All models as a table
German search for RAG, one 24 GB card, best first
ModelnDCG@1095% intervalGPU memorypassages per secondGap to the best
qwen3-emb-8b with reranker bge-reranker-v2-m3 (on the map)0.6400.605 to 0.67721.8 GB44best value
qwen3-emb-8b with reranker Qwen3-Reranker-0.6B0.6340.600 to 0.66721.8 GB44not established, p 0.60
qwen3-emb-8b0.6130.576 to 0.65121.8 GB44established, p 0.036
nomic-v2-moe0.5820.543 to 0.6202.6 GB840established, p < 0.01
bge-m30.5770.540 to 0.6163.0 GB1,189established, p < 0.01
e5-large0.5640.527 to 0.6033.0 GB1,194established, p < 0.01
qwen3-emb-0.6b0.5460.510 to 0.5839.3 GB335established, p < 0.01
gte-multi0.5120.473 to 0.5522.4 GB2,377established, p < 0.01
e5-large-instruct0.5100.474 to 0.5503.0 GB1,204established, p < 0.01
bm25 (baseline)0.199

305 questions from MIRACL German. GPU memory is the peak over the whole run minus idle. BM25 runs on the CPU.

C2measured

German speech to text

How accurately does a model transcribe spoken German? FLEURS German with 862 read recordings, scored as word error rate (WER).

GPU memory in GB, the card has 24

whisper-large-v3

  • word error rate 4.70%, 95% interval 3.95% to 5.50%
  • 8.9 GB of GPU memory
  • 23 times real time, one recording after the other
  • shown on the map
  • tied with canary-1b-v2, p 0.62

Tap a dot or reach it with the tab key to read another model.

The map shows whisper-large-v3, tied with canary-1b-v2 (p 0.62), because it needs less GPU memory.

  • The test does not separate canary-1b-v2 at 4.52% and whisper-large-v3 at 4.70%, p 0.62. The map shows whisper-large-v3 because it takes 8.9 GB of GPU memory instead of 22.9 GB.
  • parakeet-v3 is the model for volume: 5.16% errors at 182 times real time.
  • Recordings over 30 seconds throw Whisper off: whisper-large-v3 makes 17.58% errors there, canary-1b-v2 only 3.24%. This split was not fixed in advance, it only describes, and with 21 recordings the group is small.
  • The German fine-tunes are worse than their base: whisper-turbo-de reaches 5.85% against 5.63% for the original, distil-whisper-de 13.32%.

Read on GitHub: Findings · Report with all numbers · Test design

All models as a table
German speech to text, one 24 GB card, best first
Modelword error rate95% intervalGPU memorytimes real timeGap to the best
canary-1b-v24.52%3.95% to 5.16%22.9 GB23best value
whisper-large-v3 (on the map)4.70%3.95% to 5.50%8.9 GB23not established, p 0.62
parakeet-v35.16%4.56% to 5.75%16.3 GB182established, p < 0.01
whisper-large-v3-turbo5.63%4.55% to 6.88%4.5 GB69established, p 0.026
whisper-turbo-de5.85%5.20% to 6.51%4.5 GB64established, p < 0.01
distil-whisper-de13.32%11.83% to 14.94%4.4 GB90established, p < 0.01

862 recordings from FLEURS German. GPU memory is the peak from nvidia-smi, the NeMo models reserve more than they use.

C3measured

Invoice fields from the image

How many of twelve fields does a model read correctly from the image of an invoice? 34 public XRechnung test invoices from KoSIT, 408 cells, scored as the share of correct fields.

GPU memory in GB, the card has 24

gemma4-12b

  • correct fields 0.978, 95% interval 0.963 to 0.990
  • 9.8 GB of GPU memory
  • 2.1 s per invoice
  • shown on the map
  • tied with gemma4-26b, p 0.86

Tap a dot or reach it with the tab key to read another model.

The map shows gemma4-12b, tied with gemma4-26b (p 0.86), because it needs less GPU memory.

  • not finished qwen3-vl-32b: did not finish within the budget of 180 minutes.
  • gemma4-26b at 0.980 and gemma4-12b at 0.978 are tied, p 0.86. The map shows gemma4-12b: 2.1 s per invoice with 9.8 GB. Rules over the text layer of the PDF reach 0.270.
  • mistral-small-3.2 reaches 0.909. OCR in front of the same model hurts: docling with gemma4-12b reaches 0.875. Reading the image directly is better.
  • qwen3.8-27b returned null for every money field, even from the clean text layer. Its other fields are almost all right, scored as it answered.
  • qwen3-vl-8b reached 0.571 in its second run. For 6 of 34 invoices it gave no usable answer, and it needs 39.0 s per invoice.
  • qwen3-vl-32b did not finish within the budget of 180 minutes on one RTX 4090. Model and context do not fit into GPU memory, part of it ran on the CPU.

Read on GitHub: Findings · Report with all numbers · Test design

Invoices with gold fields on Hugging Face · Browse each model's answers per invoice

All models as a table
Invoice fields from the image, one 24 GB card, best first
Modelcorrect fields95% intervalGPU memorys per invoiceGap to the best
gemma4-26b0.9800.968 to 0.99019.8 GB3.6best value
gemma4-12b (on the map)0.9780.963 to 0.9909.8 GB2.1not established, p 0.86
mistral-small-3.20.9090.880 to 0.93619.7 GB4.6established, p < 0.01
docling+gemma4-12b0.8750.838 to 0.90718.7 GB6.5established, p < 0.01
pdftext+qwen3.8-27b (reference, text layer)0.6570.647 to 0.66419.7 GB2.1established, p < 0.01
docling+qwen3.8-27b (partly on the CPU)0.5910.561 to 0.61822.8 GB26.1established, p < 0.01
qwen3-vl-8b0.5710.466 to 0.67211.9 GB39.0established, p < 0.01
Regex over the text layer (baseline)0.270
qwen3-vl-32bqwen3-vl-32b: did not finish within the budget of 180 minutes.

34 invoices with twelve fields each. GPU memory is the peak minus idle, speed the median per invoice.

C4measured

Structured output and tool calls

Does a model call the right function with the right arguments? BFCL v3 with 1,000 tasks through Ollama's built-in tool interface, scored as the share of correct calls.

GPU memory in GB, the card has 24

qwen3-14b

  • correct calls 0.919, 95% interval 0.901 to 0.935
  • 10.3 GB of GPU memory
  • 606 ms per task
  • shown on the map
  • tied with qwen3.8-27b, p 0.22

Tap a dot or reach it with the tab key to read another model.

The map shows qwen3-14b, tied with qwen3.8-27b (p 0.22), because it needs less GPU memory.

  • The test does not separate qwen3-14b at 0.919 from qwen3.8-27b at 0.906, p 0.22. The map shows qwen3-14b, which needs 10.3 GB instead of 22.3 GB of GPU memory.
  • Small models come close: granite4-3b reaches 0.869 with 3.1 GB. The gap opens when one answer has to hold several calls at once.
  • With the functions as JSON in the system prompt and the answer read from the text, qwen3-14b reaches 0.946, gemma4-26b 0.947 and mistral-small-3.2 0.934. The built-in tool interface is not automatically the safer path.
  • gpt-oss-20b fails at the Ollama integration, not on content. It returns at most one call per task and so reaches 0.508.
  • When Ollama enforces the JSON schema during generation, every model keeps to the schema more often, granite4-3b for example 0.895 instead of 0.733. Even then not every answer passes. A check behind the model stays necessary.

Read on GitHub: Findings · Report with all numbers · Test design

All models as a table
Structured output and tool calls, one 24 GB card, best first
Modelcorrect calls95% intervalGPU memoryms per taskGap to the best
qwen3-14b (on the map)0.9190.901 to 0.93510.3 GB606best value
qwen3.8-27b (partly on the CPU)0.9060.888 to 0.92322.3 GB886not established, p 0.22
qwen3-8b0.8920.872 to 0.9116.3 GB374established, p 0.010
gemma4-26b0.8810.861 to 0.90019.0 GB252established, p < 0.01
gemma4-12b0.8770.857 to 0.8979.2 GB300established, p < 0.01
granite4-3b0.8690.849 to 0.8893.1 GB223established, p < 0.01
mistral-small-3.20.8160.791 to 0.84016.2 GB726established, p < 0.01
gpt-oss-20b0.5080.479 to 0.53912.6 GB334established, p < 0.01

1,000 tasks from BFCL v3, called through Ollama's tool interface. GPU memory is the peak over the whole run minus idle, speed the median per task.

C5measured

Translation German and English

Does a model translate single sentences between German and English, in both directions? 1,012 sentences per direction from FLORES+, scored as chrF++ (mean of both directions, 0 to 100).

GPU memory in GB, the card has 24

gemma4-12b

  • chrF++ 66.5, 95% interval 65.7 to 67.3
  • 8.7 GB of GPU memory
  • 4.2 sentences per second
  • shown on the map
  • tied with gemma4-26b, p 0.053
  • German to English 67.9, English to German 65.1

Tap a dot or reach it with the tab key to read another model.

The map shows gemma4-12b, tied with gemma4-26b (p 0.053), because it needs less GPU memory.

  • The two Gemma 4 models lead in both directions. German to English, gemma4-26b reaches 68.1 and gemma4-12b 67.9. English to German it is 65.5 and 65.1. The paired test does not separate the two (p 0.060 and 0.053). The map therefore shows gemma4-12b: 8.7 GB of GPU memory instead of 18.5 GB.
  • eurollm-9b follows with 67.3 and 64.8. The test separates it from gemma4-26b in both directions (p < 0.01).
  • opus-mt, built for translation only, sits at 64.9 and 61.1, three to four points behind gemma4-12b. It needs only 1.9 GB of GPU memory and manages 6.0 sentences per second one at a time (gemma4-12b 4.2), 84 in batches of 32. For a large stack of translations the question is whether three to four points matter.
  • The metrics do not give one ranking. The learned metric COMET separates only opus-mt from the other models and gives no ranking among the language models. We therefore do not read the chrF++ rank of qwen3.8-27b as a statement about meaning.
  • Limits: one sentence per call, German and English only, news and encyclopedic text. The test says nothing about contracts, mail threads or long documents. FLORES+ has been the usual set for this measurement since 2021 and may have been in the training data of some models. Quantisation differs between arms. 1 call of gemma4-12b failed and counts as an empty translation.

Read on GitHub: Findings · Report with all numbers · Test design

All models as a table
Translation German and English, one 24 GB card, best first
ModelchrF++95% intervalGPU memorysentences per secondGap to the best
gemma4-26b66.866.0 to 67.618.5 GB4.9best value
gemma4-12b (on the map)66.565.7 to 67.38.7 GB4.2not established, p 0.053
eurollm-9b66.065.2 to 66.99.4 GB2.5established, p < 0.01
madlad-3b65.664.7 to 66.49.4 GB0.5established, p < 0.01
mistral-small-3.265.564.7 to 66.414.8 GB1.7established, p < 0.01
qwen3.8-27b64.663.8 to 65.417.7 GB2.0established, p < 0.01
opus-mt63.062.2 to 63.81.9 GB6.0established, p < 0.01
source copied unchanged (baseline)19.8

1,012 sentences per direction from FLORES+, the value is the mean of both directions. GPU memory is the peak over the whole run minus idle, speed the sentences per second one at a time (mean of both directions).

C6measured

German reading comprehension

Does a model find the answer to a question in a German passage? 1,000 questions from GermanQuAD, scored as F1 over the words of the answer.

GPU memory in GB, the card has 24

gelectra-large

  • F1 0.872, 95% interval 0.855 to 0.888
  • 1.7 GB of GPU memory
  • 28 ms per question
  • shown on the map
  • best value, the test separates it from all others

Tap a dot or reach it with the tab key to read another model.

The map shows gelectra-large.

  • The small reader model gelectra-large beats every language model: 0.872 against 0.849 for the best language model, gemma4-12b, separated at p 0.024. It needs 1.7 GB of GPU memory and 28 ms per question, gemma4-12b 9.2 GB and 302 ms. The reader has an advantage here: it was trained on the training split of the same data set.
  • Language models answer longer and less literally. The share of exact answers drops from 0.719 for the reader to 0.656 for gemma4-12b and 0.345 for qwen3-8b.
  • Size does not help here. gemma4-26b at 0.778 is below gemma4-12b, and qwen3.8-27b wrote “keine Antwort” (no answer) 104 times although the answer is in the passage.
  • Among the language models gemma4-12b is the recommendation. Its German fine-tune winnow-12b is practically level at 0.848. We did not plan a test between the two.

Read on GitHub: Findings · Report with all numbers · Test design

All models as a table
German reading comprehension, one 24 GB card, best first
ModelF195% intervalGPU memoryms per questionGap to the best
gelectra-large (on the map)0.8720.855 to 0.8881.7 GB28best value
gemma4-12b0.8490.832 to 0.8669.2 GB302established, p 0.024
winnow-12b0.8480.832 to 0.86613.5 GB448established, p 0.026
mistral-small-3.20.8010.783 to 0.81916.0 GB416established, p < 0.01
gemma4-26b0.7780.759 to 0.79819.0 GB264established, p < 0.01
qwen3.8-27b0.7510.729 to 0.77218.1 GB541established, p < 0.01
qwen3-8b0.6760.655 to 0.6966.3 GB248established, p < 0.01
Regex over the text layer (baseline)0.221

1,000 questions from GermanQuAD. GPU memory is the peak minus idle, speed the median per question.

C7measured

Long context up to 32k tokens

Does a model find a fact hidden in a long German text? 180 tasks per length, at 4k, 16k and 32k tokens, scored as the share of correct answers at 32k.

GPU memory in GB, the card has 24

qwen3-14b

  • correct at 32k 1.000, 95% interval 1.000 to 1.000
  • 14.0 GB of GPU memory
  • 8.1 s to read 32k tokens
  • shown on the map
  • tied with qwen3.8-27b, p 1.00

Tap a dot or reach it with the tab key to read another model.

The map shows qwen3-14b, tied with qwen3.8-27b (p 1.00), because it needs less GPU memory.

  • qwen3-14b, qwen3.8-27b and gpt-oss-20b answer every task at 32k correctly. gpt-oss-20b reads 32k tokens in 2.7 s, qwen3-14b needs 8.1 s. Longer texts were not tested.
  • The Gemma and Mistral misses are cut inputs, not lost facts. When a text in the model's own tokens is longer than the configured context window, Ollama drops its start without an error. That is how gemma4-12b ends at 0.944. In operation the context window must be larger than the longest input.
  • qwen3-8b reaches 0.983 with 10.0 GB of GPU memory. The test does not separate it from the best model (p 0.11).
  • The task is synthetic: an inserted fact in text from GermanQuAD. It shows whether a model reads long texts reliably, not whether it understands them.

Read on GitHub: Findings · Report with all numbers · Test design

All models as a table
Long context up to 32k tokens, one 24 GB card, best first
Modelcorrect at 32k95% intervalGPU memorys to read 32kGap to the best
qwen3-14b (on the map)1.0001.000 to 1.00014.0 GB8.1best value
qwen3.8-27b1.0001.000 to 1.00019.7 GB13.5not established, p 1.00
gpt-oss-20b1.0001.000 to 1.00013.2 GB2.7not established, p 1.00
mistral-small-3.20.9890.972 to 1.00019.6 GB9.3not established, p 0.29
qwen3-8b0.9830.961 to 1.00010.0 GB4.1not established, p 0.11
gemma4-12b0.9440.911 to 0.97210.5 GB6.0established, p < 0.01
gemma4-26b0.9440.911 to 0.97219.2 GB4.2established, p < 0.01

180 tasks at 32k tokens, three task types at three positions in the text. GPU memory is the peak at 32k minus idle, speed the time to read the prompt (median).

C9measured

Finding personal data

Does a method find names, addresses, e-mail addresses, phone, account and ID numbers and dates of birth in German text? What counts is whether exactly the right place is hit (span F1, whichever of the seven types). The data are 476 German texts from the synthetic Gretel data set in finance. The value in the chart covers 40 of them, whose reference spans were checked.

First of all: the reference spans of this data set were generated by a language model themselves, and they are noisy. The check of the 40 texts was not done by a human but by a language model (Claude Haiku 5.5), and every objection was cross-checked by Claude Opus 5.5. The test design asks for a check by a human, which is still open. Of 149 reference spans, 102 stayed as they were, 43 were wrong (for instance “Damen”, “Herren” or “Jetzt” as a person) and 4 needed a new boundary. The precision of the reference spans is 0.685, the recall 0.981. That is below the threshold of 0.90 set beforehand, so the test counts as “reference noisy”. All numbers here only describe, there is no ranking.

GPU memory in GB, the card has 24

gemma4-12b

  • F1 0.839, 95% interval 0.706 to 0.929
  • 9.2 GB of GPU memory
  • 516.9 ms per document
  • shown on the map
  • tied with qwen3.8-27b, p 0.79
  • on all 476 texts with the original reference: 0.627, recall 0.628, precision 0.626

Tap a dot or reach it with the tab key to read another model.

The map shows gemma4-12b, tied with qwen3.8-27b (p 0.79), because it needs less GPU memory.

  • The two language models lead: gemma4-12b reaches 0.839, qwen3.8-27b 0.828. The paired test does not separate the two (p 0.79). On all 476 texts with the original reference the order is the other way round: qwen3.8-27b 0.644, gemma4-12b 0.627. The map shows gemma4-12b, which needs 9.2 GB of GPU memory instead of 18.1 GB.
  • The small gliner-pii-domains reaches 0.738 with recall 0.667 and precision 0.828, in 27.7 ms per document (gemma4-12b 516.9 ms). On the 40 texts the test does not separate it from the top (p 0.19), on all 476 texts it does: 0.564 against 0.644. gliner2.5-multi (0.649) and gliner-multi-pii (0.444) follow, separated.
  • Presidio reaches 0.300 with recall 0.407 and a precision of only 0.238: it reports far more than the reference holds. The regex (0.150) hits fixed formats such as e-mail addresses cleanly, but only a small part of all spans (recall 0.083).
  • Limits: the texts are invented and from finance, no real person stands behind any span. The data set has been public since 2024 and may have been in the training data of some models. Each method ran once. Claude Haiku 5.5 is not a contestant. The numbers say nothing about real documents and are no proof that a method meets the GDPR. When a human check comes in, we recompute.

Read on GitHub: Findings · Report with all numbers · Test design

All models as a table
Finding personal data, one 24 GB card, best first
ModelF195% intervalGPU memoryms per documentGap to the best
gemma4-12b (on the map)0.8390.706 to 0.9299.2 GB516.9best value
qwen3.8-27b0.8280.738 to 0.90218.1 GB1,141.8not established, p 0.79
gliner-pii-domains0.7380.643 to 0.8162.0 GB27.7not established, p 0.19
gliner2.5-multi0.6490.517 to 0.7427.6 GB21.9established, p 0.014
gliner-multi-pii0.4440.282 to 0.5962.0 GB27.9established, p < 0.01
presidio0.3000.213 to 0.3850.0 GB36.8established, p < 0.01
regex (baseline)0.150

40 German texts from the Gretel data set whose reference spans a language model checked (not a human). GPU memory is the peak minus idle, Presidio and regex run on the CPU, speed the median per document.

Read with care: every model ran once. Tied means the paired test does not separate two models (p of 0.05 or more). The map then shows the model with less GPU memory. The test data are public, so a model may have seen them in training. The data sets are clean and read aloud, real documents and meetings are harder.

Plan

What comes next

We fixed these tests on 9 October 2026, before any of them ran. A test counts as measured only once its dated design is in the repository and it has run.

Every family gets its home ground

A comparison that only asks what one tool is built for favours that tool. So every family on the map gets at least one test on its own ground, named in advance, next to the tests where it is a guest. For Jev these are judgments with probabilities, for GLiNER fast routing and finding text spans, for local language models open tasks with structured output. If a family loses on its own ground, this page says so.

  1. T1

    Automation rate

    What share of cases can a tool decide alone at 95 percent accuracy, and are its probabilities right? to the section →

    Data: existing runs from routing and the CV match, no new ones · new row

    measured
  2. T2

    Stability

    A second run with shuffled options: how often does the answer flip? Measured on 12 October 2026 on 5,948 questions: qwen3.8-27b 5.7%, winnow-12b 7.8%, gemma4-12b 9.1%, GLiNER Decide 0%. For the language models the flips sit mostly in the question with 18 options. Findings on GitHub

    Data: Routing · new row

    measured
  3. T3

    Classic classifier

    Multilingual sentence embeddings with logistic regression, trained on the MASSIVE train split.

    Data: MASSIVE, train and test · new column

    measured
  4. T4

    GLiNER Decide fine-tuned

    Trained with 10, 50 and all examples per class.

    Data: MASSIVE, train · new variant

    planned
  5. T5

    Hosted language model

    Dropped on 9 October 2026: the test bench concentrates on open models that run in house.

    Data: Routing, extraction · new column

    dropped
  6. T6

    E-invoice yes or no

    Yes/no questions over the free text, test design committed. to the section →

    Data: KoSIT, Apache 2.0 · new row

    planned
  7. T7

    Finding personal data

    Measured as capability test C9, on a public German data set under Apache 2.0. A language model checked the reference, not a human. to the section →

    Data: Gretel, synthetic finance data, Apache 2.0 · new row

    measured
  8. T8

    Invoice fields

    Measured as capability test C3: invoices as image, the XML fields are the reference. to the section →

    Data: KoSIT, Apache 2.0 · new row

    measured
  9. T9

    Robustness

    Typos and colloquial wording in German spoken commands. Measured on 12 October 2026 on 1,000 commands in four versions: typos cost each of the four tools 3.0 to 4.2 points of accuracy, missing umlauts and colloquial wording at most 1.8. Findings on GitHub

    Data: MASSIVE German, altered · new row

    measured
  10. T10

    Throughput with batching

    How many decisions does a tool manage when requests run in batches? Measured on 12 October 2026 for GLiNER Decide: in batches of 8 about 400 decisions per second, 5.7 times as many as one at a time. Larger batches are slower, the answers stay the same. We did not re-measure the language models for this, their throughput is in C8. Findings on GitHub

    Data: Routing · new row

    measured
  11. T11

    Long documents

    Route texts that are longer than what an encoder reads at once.

    Data: still to choose · new row

    planned
  12. T12

    Series of local open models

    Eleven open models from 3.4 to 32.8 billion parameters on routing and extraction: accuracy, response time, GPU memory, decisions per hour on a 24 GB graphics card. Measured on 9 October 2026. to the section →

    Data: Routing, extraction · new variant

    measured
  13. C1

    German search for RAG

    Open search models, two rerankers and keyword search with BM25 on MIRACL German. to the section → Findings on GitHub

    Data: MIRACL de, CC BY-SA 4.0 · capability test

    measured
  14. C2

    German speech to text

    Open models on FLEURS German: word error rate, speed and GPU memory. to the section → Findings on GitHub

    Data: FLEURS de, CC BY 4.0 · capability test

    measured
  15. C3

    Invoice fields from the image

    Vision models, OCR with a language model and a regex baseline on the KoSIT test invoices. to the section → Findings on GitHub

    Data: KoSIT, Apache 2.0 · capability test

    measured
  16. C4

    Structured output and tool calls

    Open language models call functions and fill JSON schemas, free and with the schema enforced. to the section → Findings on GitHub

    Data: BFCL v3, Apache 2.0, and JSONSchemaBench, MIT · capability test

    measured
  17. C5

    Translation German and English

    Open models translate in both directions, scored as chrF++. to the section → Findings on GitHub

    Data: FLORES+, CC BY-SA 4.0 · capability test

    measured
  18. C6

    German reading comprehension

    A trained reader model and open language models answer questions about German passages. to the section → Findings on GitHub

    Data: GermanQuAD, CC BY 4.0 · capability test

    measured
  19. C7

    Long context up to 32k tokens

    Open language models look for a hidden fact in German texts of 4k, 16k and 32k tokens. to the section → Findings on GitHub

    Data: synthetic facts in text from GermanQuAD · capability test

    measured
  20. C8

    Throughput and power

    How many tokens and decisions does the card manage with one and with 16 parallel users, and what does it cost in power? Ollama against vLLM. No quality metric, so no chart under “What the machine can do”. Findings on GitHub

    Data: synthetic requests and decisions from routing · capability test

    measured
  21. C9

    Finding personal data

    Open models mark personal data in German texts, also test T7. The reference is noisy, a language model checked it instead of a human, so the numbers only describe. to the section → Findings on GitHub

    Data: Gretel, synthetic finance data, Apache 2.0 · capability test

    measured

T1, T2, T3, T7, T8, T9, T10 and T12 are measured, T5 is dropped. Next to them sits a second family without Jev, C1 to C9: what a workstation with one graphics card can do, from retrieval for RAG and speech to text to invoices as documents, throughput and power. C1 to C9 are measured, for C9 the reference is noisy and was checked by a language model, not by a human. All designs are in the plan/ folder of the repository. Test designs in the plan/ folder

Missing a test you care about? Which test should we run next? Write to us.

Everything to recompute
Cases, scripts, references and results are in the lab-bench repository, code under MIT, data under CC BY 4.0. Jev's answers per question are there too, published with TypeSafe's permission of 10 October 2026. They come with their own usage notice: for reproducing the measurements only, not for training or distilling models and not for a competing product. The result tables, without Jev's individual answers, are also on Hugging Face.

Which tool fits your task?

Bring a task and a few examples. We assess which tool family fits and what a fair test on your own data looks like.

Schedule a conversation