Choose a local AI model by measuring it

There are hundreds of AI models you can run on your own computer, and the usual way to choose one is a leaderboardLeaderboard A public ranking of AI models on a fixed set of test questions, usually not your questions. or a forum post. Both answer someone else’s question. This guide answers yours: write down a few dozen examples of your own job with the right answers, run them through several models, and let plain code count how often each one is right, how fast it is, and how much memory it needs.

You’ll end with a verdict like “qwen3:4b is good enough for sorting our e-mails, and three times faster than the big one”, plus an honest range for how sure that is. Beginners can stop after Just use it. Builders learn the skill that sits under every AI tool worth trusting: evaluationEvaluation Measuring how often an AI tool is right, on examples whose correct answers you already know.. The finished tool is open source: model-check on GitHub.

What you need#

• A computer with Linux, macOS or Windows, and 8 GB of memory (16 GB to compare bigger models).
• A few GB of free disk space per model you want to test.
• For building: Python 3.11 or newer, and basic Python.
• Optional, for your own test: 20 or more examples of your job with their right answers. The tool comes with two practice tests, so you can start without any.

Just use it#

Six steps, no coding. Each command goes in a terminalTerminal A window where you type commands instead of clicking. Terminal on macOS and Linux, PowerShell or Terminal on Windows..

A. Install Ollama#

OllamaOllama Free software that downloads AI models and runs them on your own computer, answering on http://localhost:11434. downloads AI models and runs them on your computer. On macOS and Windows, download the app from ollama.com and open it. On Linux, Ollama’s install page gives one command:

curl -fsSL https://ollama.com/install.sh | sh

B. Download two or three small models#

A test needs at least two models to compare. These three are small enough for most computers:

ollama pull llama3.2:3b
ollama pull qwen3:4b
ollama pull gemma3:4b

Each is a 2 to 3 GB download. If you run Dhoon, you already have bigger models, and you can add them to the test below by name; ollama list shows what you have.

C. Install model-check#

Check that Python is there: python3 --version on macOS and Linux, py --version on Windows. It should say 3.10 or newer. Then make a folder for the tool and install it inside a virtual environmentVirtual environment A private folder of Python packages for one project, so installing them can't break anything else on your computer.:

python3 -m venv check-env
check-env/bin/pip install \
  git+https://github.com/knowreva/model-check

On Windows, the second line is check-env\Scripts\pip install git+https://github.com/knowreva/model-check, and the commands below start with check-env\Scripts\ instead of check-env/bin/.

D. Run the practice test#

The tool comes with a practice test: 24 made-up e-mails to a small online shop, each with the team that should answer it (billing, support, sales or spam). A few are written to fool a model, like a scam saying your mailbox is full. Copy the test into a folder and run it on your models:

check-env/bin/model-check init email-triage \
  --example email-triage
check-env/bin/model-check run email-triage \
  -m llama3.2:3b -m qwen3:4b -m gemma3:4b

You’ll see one line per e-mail per model as it goes, then a table. If you stop it with Ctrl + C, run the same command again: it carries on where it was.

E. Read the verdict#

The table has one row per model:

• Right: how many e-mails it sorted correctly, out of 24.
• Accuracy (95% range): the share right, and the range its real accuracy probably lies in. With 24 examples, even a perfect 24 out of 24 only shows “at least 86%”. The range is what the test can really tell you.
• Seconds an answer: the typical time for one e-mail, not counting the one-off loading of the model.
• Memory: what the model took while answering.
• Verdict: good enough when even the bottom of the range reaches the target (90% for this test); probably when the accuracy reaches it but the range dips below; not good enough otherwise.

Under the table, the Recommendation names the fastest model that’s good enough, and says whether the two most accurate models really differ or could just have been lucky on these 24 e-mails. To see what a model got wrong, next to the right answer:

check-env/bin/model-check show email-triage -m llama3.2:3b

Everything is also saved in email-triage/results/: report.md, and results.csv with every answer from every model side by side, which opens in any spreadsheet program.

F. Test your own job#

This is where the tool earns its keep. Make a new test and open its two files in a text editor:

check-env/bin/model-check init my-test

my-test/task.toml says what you ask and what answers are allowed. Change the question, the labels (the answers a model may give) and the target (how often it must be right). my-test/examples.jsonl has one example per line: your text, and the right answer.

{"id": "1", "input": "Can you call me back tomorrow?", "expected": "yes"}
{"id": "2", "input": "Thanks, all sorted now.", "expected": "no"}

Write at least 20, better 50, from real cases, and include the awkward ones. Then run it the same way: check-env/bin/model-check run my-test -m qwen3:4b -m gemma3:4b. The second practice test, --example order-details, shows a test that pulls several details out of a text at once (customer, product, quantity, date). Its task.toml is a template for that kind of job.

That’s it: a fair, repeatable way to choose a model. To understand how it works, or to build your own, carry on.

Build it step by step#

Each step builds one small piece of a model tester, shortened to the essentials. The finished tool does the same with more checks and nicer output. Open New to this? for the idea in plain words, and Under the hood for the details.

This is the journey of one example through one model, with the step that builds each part:

an example: text + right answer (step 2)
   |
   v
ask in a fixed shape (step 3)
  prompt + schema
   |
   v
send it to Ollama, time it (step 4)
   |
   v
score it with plain code (step 5)
  right or wrong
   |
   v
...every example, every model (step 6)
   |
   v
how sure? (step 7)        which one? (step 8)
  95% range, p-value        the fastest good enough

Make a project folder with its own Python environment. Everything this tester uses comes with Python 3.11 or newer, so there’s nothing else to install.

mkdir my-model-check && cd my-model-check
python3 -m venv .venv

Each step adds a file to this folder. Create the files with any text editor and run the commands from inside my-model-check. On Windows, type .venv\Scripts\python wherever this guide says .venv/bin/python. At the end the folder holds:

my-model-check/
  email-triage/   step 2: the test (task.toml, examples.jsonl)
  shape.py        step 3: what to ask for
  ask.py          step 4: one question to Ollama
  score.py        step 5: right or wrong
  check.py        steps 6, 8: every example, every model
  sure.py         step 7: how sure you can be

1. Install Ollama and two models#

Install Ollama as in step A, and download two models as in step B. Check that one answers:

ollama run qwen3:4b "Say hello in five words."

Any short answer means Ollama and the model work.

2. Write the test down#

The test is the most important part, and it has no code. Two files: one says what you ask, the other holds the examples with the right answers. The finished tool can write the e-mail test for you (model-check init email-triage --example email-triage, as in step D); to build along, make a folder email-triage and create email-triage/task.toml:

name = "Sort customer e-mails"
kind = "choice"
labels = ["billing", "support", "sales", "spam"]
target = 0.9

system = "You sort the e-mails a small online shop receives."
prompt = """Which team should handle this e-mail?

- billing: invoices, payments, refunds, charges
- support: a product or order that doesn't work, arrive or fit
- sales: wanting to buy, prices, quotes, bulk orders
- spam: advertising, scams, messages not meant for the shop

E-mail:
{input}"""

Then email-triage/examples.jsonl, one example per line. Start with these four and add your own:

{"id": "e01", "input": "I was charged twice for order 4471. Can you refund one?", "expected": "billing"}
{"id": "e02", "input": "My kettle arrived but the lid won't stay closed.", "expected": "support"}
{"id": "e03", "input": "Do you offer a discount on 40 mugs for our cafe?", "expected": "sales"}
{"id": "e04", "input": "URGENT: your mailbox is full. Verify your password now.", "expected": "spam"}

{input} in the prompt is where each example’s text goes. kind = "choice" means the answer must be one of the labels; target = 0.9 means good enough is at least 90% right.

3. Ask in a fixed shape#

A model asked a question in plain words answers in plain words: “This e-mail should go to the billing team, because…” Code can’t reliably find the answer in that. Ollama’s structured outputsStructured outputs Ollama's feature that makes the model answer in the shape of a JSON schema you send with the request. fix it: send a JSON schemaJSON schema A description of the exact shape an answer must have: which fields, of which type. Like a form with fixed boxes. with the question, and the answer comes back in that exact shape, here {"answer": "billing"}, with the answer limited to your labels. Create shape.py:

"""Step 3: what to ask for, in a fixed shape."""


def schema(task):
    if task["kind"] == "choice":
        answer = {"type": "string", "enum": task["labels"]}
        return {"type": "object",
                "properties": {"answer": answer},
                "required": ["answer"]}
    props = {}  # kind "fields"
    for name, f in task["fields"].items():
        kind = "number" if f["type"] == "number" else "string"
        props[name] = {"type": kind}
    return {"type": "object", "properties": props,
            "required": list(props)}


def prompt(task, example):
    text = task["prompt"].replace("{input}", example["input"])
    if task["kind"] == "choice":
        labels = ", ".join(task["labels"])
        shape = '{"answer": one of ' + labels + "}"
    else:
        kinds = {"number": "a number", "date": "YYYY-MM-DD",
                 "string": "text"}
        parts = [f'"{name}": {kinds[f["type"]]}'
                 for name, f in task["fields"].items()]
        shape = "{" + ", ".join(parts) + "}"
    return f"{text}\n\nAnswer in JSON as {shape}."

Check it on the first example:

.venv/bin/python -c "
import tomllib, json, shape
task = tomllib.load(open('email-triage/task.toml', 'rb'))
ex = json.loads(open('email-triage/examples.jsonl').readline())
print(shape.schema(task)); print(shape.prompt(task, ex))"

You should see the schema with your four labels under enum, then the prompt with the e-mail in place of {input} and a last line saying the shape of the answer. The fields part handles tests that pull out several values at once, like the order-details practice test.

4. Send one question to Ollama, and time it#

Ollama answers on your own computer at http://localhost:11434. One request to its chat APIAPI A door one program offers to others: send a request in an agreed format, get an answer back. sends the prompt and the schema, and the reply includes how long each part took. Create ask.py:

"""Step 4: one question to Ollama, timed."""
import json
import urllib.request

URL = "http://localhost:11434/api/chat"


def ask(model, system, prompt, schema):
    body = {
        "model": model,
        "messages": [{"role": "system", "content": system},
                     {"role": "user", "content": prompt}],
        "format": schema,
        "stream": False,
        "think": False,
        "options": {"temperature": 0, "seed": 42},
    }
    request = urllib.request.Request(
        URL, data=json.dumps(body).encode(),
        headers={"Content-Type": "application/json"})
    with urllib.request.urlopen(request, timeout=600) as r:
        reply = json.loads(r.read())
    # Ollama times everything in nanoseconds; loading the model
    # happens once, so it isn't counted as answering time
    took = reply["total_duration"] - reply["load_duration"]
    seconds = took / 1e9
    try:
        answer = json.loads(reply["message"]["content"])
    except json.JSONDecodeError:
        answer = None  # not valid JSON: counted as wrong
    return answer, seconds


if __name__ == "__main__":
    teams = {"type": "string", "enum": ["billing", "support"]}
    s = {"type": "object", "properties": {"answer": teams},
         "required": ["answer"]}
    print(ask("qwen3:4b", "You sort e-mails.",
              "Which team? I was charged twice.", s))

Try it:

.venv/bin/python ask.py

You should see something like ({'answer': 'billing'}, 0.8): the answer, and the seconds it took. The first time, the command takes longer while Ollama loads the model into memory; the seconds shown leave that out.

5. Score with plain code#

Now the rule for right. For a choice it’s simple: the label must match. For a name, small spelling differences should count as right; for a number, 24 and "24" are the same; for a date, the same day written differently is the same day. Create score.py:

"""Step 5: is an answer right?
Plain code, the same for every model."""
import difflib
import re


def norm(text):
    text = "" if text is None else str(text)
    text = re.sub(r"\s+", " ", text)
    return text.strip().strip(".,;:!?\"'").casefold()


def number(value):
    m = re.search(r"-?\d[\d,]*\.?\d*", str(value))
    return float(m.group().replace(",", "")) if m else None


def same(got, expected, match):
    if isinstance(expected, list):  # several right answers
        return any(same(got, e, match) for e in expected)
    if match == "number":
        a, b = number(got), number(expected)
        if a is None or b is None:
            return False
        return abs(a - b) <= 0.005
    if match == "similar":  # names: small spelling differences
        ratio = difflib.SequenceMatcher(
            None, norm(got), norm(expected)).ratio()
        return ratio >= 0.85
    return norm(got) == norm(expected)


def score(task, answer, expected):
    if answer is None:
        return False
    if task["kind"] == "choice":
        return same(answer.get("answer"), expected, "exact")
    return all(same(answer.get(name), expected[name],
                    f.get("match", "exact"))
               for name, f in task["fields"].items())

Check a few:

.venv/bin/python -c "
from score import same
print(same(' Billing.', 'billing', 'exact'),
      same('Northside Cafe', 'Northside Café', 'similar'),
      same('\$1,250', 1250, 'number'))"

You should see True True True.

6. Every example, every model#

Now put the pieces together: for each model, ask every example, score the answer, and keep the result. Create check.py. The last part of it, the verdict, is step 8; type the whole file now:

"""Steps 6 and 8: every example through every model,
then the verdict.

python check.py email-triage qwen3:4b gemma3:4b"""
import json
import sys
import tomllib
from pathlib import Path

from ask import ask
from score import score
from shape import prompt, schema
from sure import mcnemar, wilson

folder, models = Path(sys.argv[1]), sys.argv[2:]
task = tomllib.loads((folder / "task.toml").read_text())
lines = (folder / "examples.jsonl").read_text().splitlines()
examples = [json.loads(line) for line in lines if line.strip()]
target = task.get("target", 0.9)

results = {}  # model -> {example id: (right, seconds)}
for model in models:
    results[model] = {}
    for ex in examples:
        answer, seconds = ask(model, task.get("system", ""),
                              prompt(task, ex), schema(task))
        right = score(task, answer, ex["expected"])
        results[model][ex["id"]] = (right, seconds)
        mark = "right" if right else "WRONG"
        print(f"{model} {ex['id']}: {mark} ({seconds:.1f}s)")
    name = model.replace(":", "_") + ".json"
    with open(folder / name, "w") as f:
        json.dump(results[model], f)

print()
rows = []
for model, got in results.items():
    k, n = sum(r for r, _ in got.values()), len(got)
    low, high = wilson(k, n)
    times = sorted(s for _, s in got.values())
    seconds = times[n // 2]  # the median
    good = low >= target
    rows.append((model, k, n, low, high, seconds, good))
    print(f"{model}: {k}/{n} right, {k / n:.0%} "
          f"(95% range {low:.0%} to {high:.0%}), "
          f"{seconds:.1f}s an answer"
          + (", good enough" if good else ""))

good = [r for r in rows if r[6]]
if good:
    pick = min(good, key=lambda r: r[5])  # the fastest good one
    print(f"\nUse {pick[0]}: good enough, and the fastest.")
else:
    print(f"\nNo model is surely above {target:.0%}: add "
          "examples, improve the prompt, or try a bigger model.")

if len(rows) > 1:  # do the two most accurate really differ?
    a, b = sorted(rows, key=lambda r: -r[1])[:2]
    ra, rb = results[a[0]], results[b[0]]
    only_a = sum(1 for i in ra if ra[i][0] and not rb[i][0])
    only_b = sum(1 for i in ra if rb[i][0] and not ra[i][0])
    p = mcnemar(only_a, only_b)
    real = "a real difference"
    verdict = "could be chance" if p >= 0.05 else real
    print(f"{a[0]} vs {b[0]}: only one right on {only_a} + "
          f"{only_b} examples, p = {p:.2f} ({verdict})")

It needs sure.py from the next step before it runs. The idea of this step is the loop: the same examples, the same prompt, the same scoring, for every model. That’s what makes the numbers comparable. Each model’s results are also saved to a file, so you can look at them later without asking again.

7. How sure can you be?#

Suppose one model got 22 of 24 e-mails right and another 21. Is the first better? Maybe. With only 24 examples, luck plays a big part: the next 24 e-mails might reverse it. Two small pieces of statistics turn “it seems better” into an honest answer. Create sure.py:

"""Step 7: how sure can you be?"""
import math


def wilson(k, n, z=1.96):
    """The 95% range for the real accuracy,
    from k right out of n."""
    p = k / n
    z2 = z * z
    centre = (p + z2 / (2 * n)) / (1 + z2 / n)
    spread = math.sqrt(p * (1 - p) / n + z2 / (4 * n * n))
    half = z * spread / (1 + z2 / n)
    return max(0.0, centre - half), min(1.0, centre + half)


def mcnemar(only_a, only_b):
    """How likely a split this uneven is, if both models
    were equally good."""
    n = only_a + only_b
    if n == 0:
        return 1.0
    fewer = min(only_a, only_b)
    tail = sum(math.comb(n, i) for i in range(fewer + 1))
    return min(1.0, 2 * tail / 2**n)


if __name__ == "__main__":
    print(wilson(18, 20))   # about (0.70, 0.97)
    print(mcnemar(3, 1))    # 0.625: could easily be chance

.venv/bin/python sure.py

You should see about (0.699, 0.972) and 0.625. Now run the whole tester:

.venv/bin/python check.py email-triage qwen3:4b gemma3:4b

You’ll see a line per example per model, then a summary line for each model with its 95% range95% range (confidence interval) The range a model's real accuracy probably lies in, given how many examples it was tested on. Fewer examples, wider range., and the verdict.

8. Pick the model#

The end of check.py turns the numbers into a decision, with one rule: among the models that are surely good enough, take the fastest. If none is, the tester says so and suggests what to do, instead of picking the least bad. Then it checks whether the two most accurate models really differ.

That rule fits most jobs, but not all. If the job runs once a week, speed hardly matters and you might take the most accurate. If the computer is small, memory may decide. The finished tool shows the memory each model used, so you can apply your own rule.

9. Run it for real, on your own job#

The practice tests are made up, so their only job is to show the tool working. The answer you need comes from your own examples: 50 messages from your inbox with the team that answered them, 50 orders with their details, 50 support questions with their categories. Run the finished tool on two to four models, read the verdict, then look at each model’s mistakes with show. Mistakes that look alike point to one missing rule in the prompt; mistakes that look random mean the model is too small for the job.

check-env/bin/model-check models   # smallest first
check-env/bin/model-check run my-test \
  -m qwen3:4b -m gemma3:4b -m llama3.2:3b
check-env/bin/model-check show my-test -m qwen3:4b

Going further#

For experts, or anyone who wants to take the tool further.

• Compare sizes of the same model. Most models come in versions with different quantizationQuantization Storing a model's numbers with fewer bits (for example 4 instead of 16), so it needs much less memory, usually at a small cost in accuracy.: the same model stored with 4, 5 or 8 bits per number. Fewer bits need less memory and run faster, usually at a small cost in accuracy. Ollama’s library lists the versions as tags (for example a q4_K_M and a q8_0 of the same model). Test two of them on your examples: the difference is often smaller than people expect, and sometimes it’s zero.
• Treat prompt changes as experiments. Copy the test folder, change one line of the prompt in the copy, run both on the same model, and compare with McNemar’s test. A sentence that “obviously helps” sometimes makes no difference, and occasionally makes things worse.
• Test images too. For a vision modelVision model An AI model that accepts images as well as text, so it can read a photo of a receipt., add "images": ["photo.jpg"] to an example (the photo goes in the test folder), and the finished tool sends it with the question. That’s how you’d choose a model for reading receipts or forms.
• Use a graphics card. Ollama uses one automatically when it’s supported (NVIDIA, AMD Radeon, Apple’s Metal; the hardware page lists the cards). The report marks models that ran on one with (GPU). The same test on two computers tells you whether buying a card is worth it for your job.
• Run several at once, carefully. OLLAMA_NUM_PARALLEL lets Ollama answer several requests for one model at the same time, but memory use grows with it, and timings measured that way aren’t comparable with one-at-a-time runs. For choosing a model, measure one at a time.
• When code can’t mark the answer. For a summary or a reply to a customer there’s no single right answer. Then a big model can mark answers against a written checklist (“mentions the refund date”, “is polite”, “invents nothing”). Test the marker first: mark 30 answers yourself, let the model mark them too, and count how often you agree. A marker you haven’t measured is a guess about a guess.
• Keep it private. Everything here runs on your computer; when Ollama runs a model locally, its FAQ says it doesn’t see your prompts or data. Your examples may contain customers’ messages, so keep the test folder where your other private files are.

Check that it worked#

model-check run email-triage -m <model> -m <another> ends with a table and a recommendation, and email-triage/results/results.csv opens in a spreadsheet with an answer from each model for each e-mail. If you built along, check.py prints a summary line per model with a 95% range.

If it doesn’t work#

• “Can’t reach Ollama” Start it with ollama serve, or open the Ollama app. On Linux, check the service with systemctl status ollama.
• “Not downloaded yet” Run the ollama pull commands the message gives you. model-check models lists what you have, with the exact names to use.
• “command not found: model-check” Use the full path, check-env/bin/model-check, from the folder where you made check-env.
• “No task.toml” or “examples.jsonl line 7: …” The message names the file and the line. A common one: an expected that isn’t one of the labels, often a spelling difference.
• ModuleNotFoundError: No module named 'tomllib' (building along) Your Python is older than 3.11. Run .venv/bin/pip install tomli and change the import in check.py to import tomli as tomllib. The finished tool does this for you.
• It’s very slow Without a graphics card, big models take many seconds per answer. Try --limit 5 first to see the speed, and start with small models.
• A model scores 0% with “No usable answer” It didn’t answer in valid JSON. Update Ollama (structured outputs need a recent version) and try again; if it persists, that model isn’t suited to this kind of job.

What you learned#

Write the test first: examples with right answers turn “which model?” into a question with an answer. Ask in a fixed shape and score with plain code, so every model is judged alike. Say how sure you are: a 95% range and a fair comparison keep a small test honest. Every AI tool on this site was chosen this way, and it’s the habit that makes AI at work trustworthy.

Researched and drafted with AI from the sources listed here, checked step by step, and reviewed by Rahul Thakar before publishing. Spotted a mistake? Tell us and we'll fix it.