Build a receipt reader with a local AI model

Most people type receipts into a spreadsheet by hand, or pay a monthly fee for an app that uploads them somewhere. This guide does the same job on your own computer: photos and PDF invoices go in, a CSVCSV A spreadsheet saved as plain text, one row per line and commas between columns. Excel, Google Sheets and LibreOffice open it. spreadsheet with the date, shop, tax and total comes out, and nothing leaves the machine.

Pick your path above. Beginners can stop after Just use it and have a working tool. Builders learn three skills every AI document job uses: getting answers in a fixed shape, checking the model’s answer with plain code, and measuring how often it’s right. The finished tool is open source: receipt-reader on GitHub.

What you need#

• A computer with Linux, macOS or Windows.
• About 7 GB of free disk space for the model, and 16 GB of memory (8 GB works with the smaller model).
• A few receipts: photos from your phone, or PDF invoices.
• For building: Python 3.10 or newer, and basic Python.

Just use it#

Five steps, no coding. Each command goes in a terminalTerminal A window where you type commands instead of clicking. Terminal on macOS and Linux, PowerShell or Terminal on Windows..

A. Install Ollama#

OllamaOllama Free software that downloads AI models and runs them on your own computer, answering on http://localhost:11434. downloads AI models and runs them on your computer. On macOS and Windows, download the app from ollama.com and open it. On Linux, Ollama’s install page gives one command:

curl -fsSL https://ollama.com/install.sh | sh

B. Download a model that can read images#

ollama pull qwen2.5vl:7b

That’s a 6 GB download, so it takes a while on a slow connection. On a computer with 8 GB of memory, use qwen2.5vl:3b instead (3.2 GB), and add --model qwen2.5vl:3b to the commands below. If you run Dhoon, its model gemma4:26b can read images too, so you can skip this download and use --model gemma4:26b.

C. Install receipt-reader#

First check that Python is there: python3 --version on macOS and Linux, py --version on Windows. It should say 3.10 or newer; if not, install it from python.org. Then make a folder for the tool and install it inside a virtual environmentVirtual environment A private folder of Python packages for one project, so installing them can't break anything else on your computer.:

python3 -m venv receipts-env
receipts-env/bin/pip install \
  git+https://github.com/knowreva/receipt-reader

On Windows, the second line is receipts-env\Scripts\pip install git+https://github.com/knowreva/receipt-reader, and the commands below start with receipts-env\Scripts\ instead of receipts-env/bin/.

D. Read your receipts#

Put your receipts in one folder, for example Receipts in your home folder: phone photos (iPhone photos too), PDF invoices, or PDFs from a scanner app. Then run:

receipts-env/bin/receipt-reader scan ~/Receipts \
  -o receipts.csv --currency CAD

Use your own currency instead of CAD (USD, EUR, INR, AUD…). It’s only used when a receipt prints a plain “$” without saying which. You’ll see one line per receipt as it’s read.

E. Open the spreadsheet#

Open receipts.csv in Excel, Google Sheets, Numbers or LibreOffice. Look at the status column first:

• ok: the numbers on that receipt add up. It’s still worth comparing a few with the paper at first.
• check: something didn’t add up, and the problems column says what. Look at those receipts yourself.
• notes: anything the tool corrected by itself, and why. For example, a store’s tax registration number the model mistook for a tax.

That’s it: a working receipt reader. To get later improvements, run the install line from step C again with --upgrade after install.

If you run Dhoon, it can do this for you in chat: install receipt-reader into Dhoon’s own environment with ~/dhoon/runtime/venv/bin/pip install git+https://github.com/knowreva/receipt-reader, restart its tools with sudo systemctl restart ai-toolbox, put the receipts in a receipts folder in Dhoon’s workspace, and ask “Read my receipts into a spreadsheet.” For receipts that only print “$”, add RECEIPT_CURRENCY=CAD (your currency) to Dhoon’s config/models.env and run bash scripts/04_toolbox.sh. To understand how it works, or to build your own, carry on.

Build it step by step#

Each step shows the part of receipt-reader’s code that does the job, shortened to the essentials. Open New to this? for the idea in plain words, and Under the hood for the details.

This is the whole journey of one receipt, with the step that builds each part:

photo or PDF
   |
   v
prepare it (step 5)
  photos upright and smaller
   |
   v
ask the model (steps 2 to 4)
  prompt + schema -> JSON answer
   |
   v
check it (step 6)
  do the numbers add up?
   |               \
  yes       no, or a broken answer
   |                |
   |     second look (step 7),
   |     told what was wrong
   |                |
   v                v
write the row (step 8)
  "ok", or "check" + the reason

To build along, make a project folder with its own Python environment (a virtual environmentVirtual environment A private folder of Python packages for one project, so installing them can't break anything else on your computer.) and install the libraries the code uses. Everything else comes with Python.

mkdir my-receipt-reader && cd my-receipt-reader
python3 -m venv .venv
.venv/bin/pip install pillow pillow-heif pypdf pypdfium2

Pillow opens and shrinks photos, pillow-heif adds iPhone photos (HEIC) to Pillow, and pypdf and pypdfium2 read PDFs. To keep the list with your code, put those four names in a file called requirements.txt, one per line; .venv/bin/pip install -r requirements.txt installs them on another computer.

1. Install Ollama and a model that can see#

Install Ollama as in step A, then download a vision modelVision model An AI model that accepts images as well as text, so it can read a photo of a receipt., one that accepts images as well as text. This guide uses qwen2.5vl, which its Ollama page describes as strong at “understanding documents and diagrams”, with structured output for “scans of invoices, forms, tables”. It needs Ollama 0.7.0 or newer.

ollama pull qwen2.5vl:7b
ollama run qwen2.5vl:7b ./receipt.jpg "What is the total?"

If the second command answers with your receipt’s total, the model can see.

2. Decide exactly what you want back#

The most important design decision comes before any code: the exact fields you want. A model asked to “read this receipt” answers differently every time. Ollama’s structured outputsStructured outputs Ollama's feature that makes the model answer in the shape of a JSON schema you send with the request. solve that. You send a JSON schemaJSON schema A description of the exact shape an answer must have: which fields, of which type. Like a form with fixed boxes. in the request’s format field, and the answer always comes back in that shape.

MONEY = {"type": ["number", "null"]}  # null: not printed

ITEM = {"type": "object",
        "properties": {"description": {"type": "string"},
                       "quantity": {"type": "number"},
                       "amount": {"type": "number"}},
        "required": ["description", "quantity", "amount"]}
TAX = {"type": "object",
       "properties": {"name": {"type": "string"},
                      "amount": {"type": "number"}},
       "required": ["name", "amount"]}
CATEGORIES = ["groceries", "restaurant", "fuel", "travel",
              "office", "utilities", "health", "other"]

SCHEMA = {
    "type": "object",
    "properties": {
        "vendor": {"type": "string"},
        "date": {"type": "string"},      # YYYY-MM-DD
        "currency": {"type": "string"},  # CAD, USD, EUR...
        "items": {"type": "array", "items": ITEM},
        "subtotal": MONEY,
        "taxes": {"type": "array", "items": TAX},
        "tip": MONEY,
        "total": {"type": "number"},
        "payment_method": {"type": "string"},
        "category": {"type": "string", "enum": CATEGORIES},
    },
    "required": ["vendor", "date", "currency", "items",
                 "subtotal", "taxes", "tip", "total",
                 "payment_method", "category"],
}

Three choices here matter more than they look:

• Line items are part of the answer, even if you only need the total. They let code check the total in step 6.
• null for anything not printed. Without it, a model will invent a tip of 0.00 or a subtotal that isn’t there.
• Categories come from a fixed list (enum), so your spreadsheet can be sorted and totalled.

3. Write the prompt#

The schema says what shape to answer in; the promptPrompt The instructions you give the model, in plain language. says how to read. Be specific about the parts that trip models up: dates and taxes.

Read this receipt or invoice and return its details as JSON.
- date: the purchase or invoice date as YYYY-MM-DD. A
  two-digit year means 20YY: "10/17/26" is 2026-10-17. If the
  receipt writes 03/04/2026, use the country's usual order
  (day first in Europe, India and Australia; month first in
  the USA; ...).
- taxes: each tax line (GST, HST, PST, QST, VAT, CGST, SGST,
  sales tax...) with its amount. A line like "GST/HST
  137466199 RT 0001" is the store's registration number,
  not an amount.
- total: the final amount paid.
Copy numbers exactly as printed; never guess a number that
isn't on the receipt.

Ollama’s structured outputs page also advises putting the schema itself in the prompt “to ground the model’s response”. The tool adds it at the end.

4. Send the receipt to Ollama#

One request to Ollama’s chat APIAPI A door one program offers to others: send a request in an agreed format, get an answer back. does the reading. The image travels inside the message as base64Base64 A way to write any file, such as a photo, as plain letters and digits, so it can travel inside a text request. text, the schema goes in format, and a temperatureTemperature How much randomness the model may use when it picks its next words. 0 means it always takes the most likely one. of 0 makes the same receipt give the same answer each time. Python’s standard library is enough, so there is nothing extra to install.

import json
import urllib.request

class BadAnswer(Exception):
    """An answer that isn't complete JSON: worth a retry."""

def chat(messages, schema, model="qwen2.5vl:7b",
         url="http://localhost:11434"):
    body = {"model": model,
            "messages": messages,
            "format": schema,           # the answer's shape
            "stream": False,            # one complete answer
            "think": False,             # no thinking: faster
            "options": {"temperature": 0,
                        "num_ctx": 8192}}  # room to read
    request = urllib.request.Request(
        f"{url}/api/chat", data=json.dumps(body).encode(),
        headers={"Content-Type": "application/json"})
    with urllib.request.urlopen(request, timeout=600) as r:
        reply = json.loads(r.read())
    if reply.get("done_reason", "stop") != "stop":
        raise BadAnswer("the answer was cut off")
    try:
        return json.loads(reply["message"]["content"])
    except json.JSONDecodeError:
        raise BadAnswer("the answer wasn't valid JSON")

message = {"role": "user", "content": PROMPT,
           "images": [image_base64]}
answer = chat([message], SCHEMA)
print(answer["vendor"], answer["total"])

The last lines matter. Structured outputs make the model write in the schema’s shape, but an answer can still stop early, and a half-written answer is broken JSONJSON A plain-text way to write data as names and values, like {"total": 38.96}. Almost every program can read it.. Ollama’s reply says how the answer ended in done_reason: "stop" means the model finished. Anything else, or text that doesn’t parse, raises BadAnswer, and step 7 treats that like a receipt that doesn’t add up: it gets one second look.

The real tool also turns Ollama’s errors into plain advice: a missing model tells you which ollama pull to run, a stopped Ollama tells you to start it, and a timeout suggests a smaller model.

5. Prepare the photos and PDFs#

Two small steps make a big difference:

• Photos: phones often store photos sideways, with an EXIFEXIF Information a camera or phone saves inside a photo: when it was taken, which way is up, and more. flag saying which way is up. ImageOps.exif_transpose from the Pillow library applies that flag. The tool then shrinks the photo to at most 1600 pixels on its longest side, which is faster to read and no less accurate for a receipt.
• iPhone photos are saved as HEIC, which Pillow can’t open by itself. One call teaches it: register_heif_opener() from the pillow-heif plugin.
• PDFs: invoices from accounting software contain real textPDF text layer The actual letters inside a PDF made by software, which can be copied. A scanned PDF has only a picture of the page.. Reading that text with pypdf is faster and exact, so the tool sends the text instead of a picture. A scanned PDF has no text, only a picture of the page; for those, each page is drawn as an image with pypdfium2 and read like a photo.
• PDFs from phone scanner apps (Adobe Scan, Microsoft Lens) are a trap: a picture of the page plus text the app guessed from it, with mistakes. Our Walmart receipt’s text began “Walmartsa”, next to a survey advert. So when a PDF page is mostly one picture, the tool reads the picture and passes the guessed text along only as a hint.

from PIL import Image, ImageOps
from pillow_heif import register_heif_opener

register_heif_opener()   # Image.open now reads .heic too

def image_b64(img):
    img = ImageOps.exif_transpose(img).convert("RGB")
    img.thumbnail((1600, 1600))
    buffer = io.BytesIO()
    img.save(buffer, format="JPEG", quality=90)
    return base64.b64encode(buffer.getvalue()).decode()

6. Check the model’s answer with arithmetic#

This step makes the output trustworthy. Models misread numbers: a 7 becomes a 1, or two lines get swapped. But a receipt checks itself: the items add up to the subtotal, and the subtotal plus tax plus tip is the total. Plain code can check that, and it never gets tired.

def close(a, b):
    # two cents of rounding, counted in whole cents
    return round(abs(a - b), 2) <= 0.02

items_sum = sum(i["amount"] for i in r["items"])
taxes = sum(t["amount"] for t in r["taxes"])
tip = r["tip"] or 0
subtotal = r["subtotal"]
base = subtotal if subtotal is not None else items_sum

if subtotal is not None and not close(items_sum, subtotal):
    problems.append(f"items add up to {items_sum:.2f}, "
                    f"not the subtotal {subtotal:.2f}")

# tax on top (Canada, USA, India)
added = close(base + taxes + tip, r["total"])
# tax already in the prices (VAT in Europe, GST in Australia)
included = taxes > 0 and close(base + tip, r["total"])
if not (added or included):
    problems.append(f"{base:.2f} + tax {taxes:.2f} = "
                    f"{base + taxes + tip:.2f}, "
                    f"not the total {r['total']:.2f}")

Notice the second case. In Europe and Australia, prices on a receipt usually include VAT or GST, and the tax line only says how much of the total was tax. A check that only knew North American receipts would flag every one of them. The real tool also checks that the date is a real date and not in the future, and that the currency is a 3-letter code.

What one real receipt taught us#

Practice receipts are tidy. The first real one we tried, a crumpled Walmart grocery receipt from Kitchener, Ontario, found five mistakes in a row:

1. The model read the store’s tax registration numbers as tax. The bottom of the receipt prints “GST/HST 137466199 RT 0001” and “QST 1016551356 TQ 0001”. The model added them up into a “tax” of 1,154,017,555 dollars on a $38.96 bill, and kept doing it even when told. So code now removes any “tax” bigger than the receipt’s total, or any whole number of seven digits or more, and records what it removed in a notes column.
2. The included-tax rule was too loose. On an earlier reading, a made-up tax of $23.92 passed as “tax included in the prices”, because the items did add up to the total. Taxes have limits: no country’s VAT is more than about 27%, which is about 21% of a tax-included price. So the check now allows included tax of at most 22% of the total, and added tax of at most 30%.
3. “10/17/26” became 2023. A two-digit year is a guess for a model. The prompt now says that 26 means 2026, and code compares the receipt’s date with the date the photo was taken (or the file was saved). A receipt more than a year older than its photo gets a second look and, if it stays that old, is marked for you to check.
4. The shop came out as “RED CARROT”, one of the items. That was the receipt scanned with Adobe Scan, read from the app’s guessed text (see step 5). Scans are now read from the picture, and a shop name that matches an item line gets a second look.
5. A tax of $3.74 appeared on a receipt that prints no tax amount. It passed as “tax included in the prices”, because the items did add up to the total. But Canadian and US receipts add tax on top; they don’t include it. So on a CAD or USD receipt, an included tax is now flagged unless the receipt’s own tax line says “incl.”. The second look made the same mistake, so one more rule fixes it in code: when the items, the subtotal and the total all agree, nothing was added on top, and the tax is removed with a note.

# no tax is bigger than this
limit = 0.22 * total if included else 0.30 * base
if taxes > limit:
    problems.append(f"tax {taxes:.2f} is too large")

# a receipt much older than its photo
if (photo_date - receipt_date).days > 400:
    problems.append("check the year")

# Canada and the USA add tax on top of the prices
if included and currency in ("CAD", "USD"):
    if not any("incl" in t["name"].lower()
               for t in r["taxes"]):
        problems.append("tax isn't included here")

# items = subtotal = total: no room for any tax
if close(subtotal, total) and close(items_sum, total):
    notes.append(f"ignored tax {taxes:.2f}")
    r["taxes"] = []

After those fixes, the same receipt came back right in every field: Walmart, 17 October 2026, $38.96, no tax (Ontario doesn’t tax basic groceries). The lesson is the real point of this guide. Each mistake was first caught by a check and only then fixed in code, once the pattern was clear. That’s how an AI tool becomes trustworthy.

7. Give the model one second look#

When the check finds a problem, ask again, and say exactly what didn’t add up. Keep the second answer only if it has fewer problems than the first.

try:
    r = ask(content)
    found = problems(r)
except BadAnswer as e:      # cut off, or not JSON
    r, found = None, [str(e)]
if found:
    hint = ("A first reading had these problems: "
            + "; ".join(found) + ". Look again carefully.")
    again = ask(content, hint)   # may raise BadAnswer
    if r is None or len(problems(again)) < len(found):
        r, found = again, problems(again)
status = "check" if found else "ok"

ask builds the request the same way both times: one user message holding the prompt, the image and, the second time, the hint added to the end of the prompt. It isn’t a conversation. The first answer is never sent back, so messages always has exactly one entry and there’s no history to get out of order.

Whatever still doesn’t add up is marked check, with the reason, in the spreadsheet. That’s the point of the whole design: a mistake is either fixed or flagged, never silently saved.

8. Write the spreadsheet#

The finished tool puts it all together as one command. It reads files or whole folders, prints a line per receipt, and writes a CSV you can open in Excel, Google Sheets or LibreOffice.

pip install git+https://github.com/knowreva/receipt-reader
receipt-reader scan ~/Receipts -o receipts.csv
receipt-reader scan ~/Receipts --items items.csv
# --items: also every line item, one per row
receipt-reader scan ~/Receipts --currency CAD
# --currency: for receipts that only print "$"

The status column says ok or check, problems says why, and notes lists anything code corrected by itself. Sort by status and you only look at the receipts that need you.

9. Measure how good it is#

Without a number, “it seems to work” is a guess. The tool can draw 30 practice receipts with their correct answers: fictional shops in five regions, with each region’s taxes and date formats, a slight tilt, blur and JPEG noise. Then eval reads them all and compares. That’s an evaluationEvaluation Measuring how often an AI tool is right, on examples whose correct answers you already know..

receipt-reader samples samples/
receipt-reader eval samples/
receipt-reader eval samples/ --model qwen2.5vl:3b
receipt-reader eval samples/ --model gemma4:26b
# the smaller model, then Dhoon's model if you have it

The report shows how many receipts got each field right (date, total, tax, currency, shop), how many got all five right, and the time per receipt on your computer. The line that matters most is how many of the mistakes the arithmetic check flagged. Those are the mistakes that can’t reach your spreadsheet unnoticed. Run it with two models and you have a real answer to “which model should I use?” instead of a forum opinion.

Our own run, on an Intel NUC with no graphics card and Dhoon’s gemma4:26b:

That took about 100 seconds a receipt. The first run took 216, until we told Ollama not to let the model “think” before answering. The two currency misses taught something too. Those practice receipts printed only “$” with no city, and “$” with GST could be Australia, Canada or New Zealand. Arithmetic can’t catch a wrong currency, so the tool has the --currency option for your own receipts, and the practice receipts now print a city, as real ones do.

The practice receipts are still easier than crumpled real ones. To test on your own, put a photo next to a file with the same name ending in .json, holding the correct answer in the same fields, and run eval on that folder.

Going further#

For experts, or anyone who wants to take the tool further.

• Use a graphics card. Ollama uses one automatically when it supports it: NVIDIA (compute capability 5.0 or newer, driver 550 or newer), AMD Radeon, and Apple’s Metal. The hardware page lists the cards. Check with ollama ps while a receipt is being read: the PROCESSOR column should say GPU. A supported graphics card can be many times faster than the processor.
• Read many receipts at once. By default Ollama handles one request per model at a time. The OLLAMA_NUM_PARALLEL setting raises that, but memory use grows with it (the FAQ explains how). On a computer without a graphics card, one at a time is usually fastest overall.
• Send the results to your accounting software. Most accounting programs can import a CSV of expenses or transactions. Map the columns once (date, vendor, total, tax), and import only the rows marked ok.
• Teach a model your own receipts. Every receipt you correct by hand is a training example: the photo and the right answer. With a few hundred, a small vision model can be fine-tuned with an outside toolkit, and Ollama can run the result: its import guide covers models in Safetensors or GGUF format. Measure before and after with eval on the same test set; fine-tuning that isn’t measured is a guess.
• Keep it private by design. When you run a model locally, Ollama’s FAQ says it doesn’t see your prompts or data. Keep the tool that way: no step that uploads receipts, and nothing that acts on what a receipt says.

Check that it worked#

receipt-reader eval samples/ finishes with a summary, and receipt-reader scan on a folder of your own receipts writes a CSV whose totals match the paper. Open three rows marked ok and compare them with the receipts.

If it doesn’t work#

• “Can’t reach Ollama” Start it with ollama serve, or open the Ollama app. On Linux, check the service with systemctl status ollama.
• “Ollama doesn’t have the model” Run the ollama pull command the message gives you.
• “command not found: receipt-reader” Use the full path, receipts-env/bin/receipt-reader, from the folder where you made receipts-env.
• It’s very slow On a computer without a graphics card, a receipt can take a minute or two (about 100 seconds on our NUC). Try --model qwen2.5vl:3b. Time per receipt is in the eval report.
• Many receipts marked check Look at the problems column: a pattern (always the date, always the tax) means the prompt in schema.py needs one more rule for your receipts.
• iPhone photos (HEIC) aren’t read Update receipt-reader (it reads HEIC since October 2026), or in your own build install pillow-heif and call register_heif_opener() (step 5).
• “the answer was cut off” The receipt needed more room than the context window. Run with RECEIPT_CONTEXT=16384 in front of the command; it uses more memory.

What you learned#

Structured outputs turn a chatty model into a reliable data source. Code checks catch what the model gets wrong, and a second look fixes some of it. Evaluation replaces “it seems fine” with a number. The same three ideas carry over to invoices, forms, bank statements and contracts: most of the work in a real business automation is exactly this.

Researched and drafted with AI from the sources listed here, checked step by step, and reviewed by Rahul Thakar before publishing. Spotted a mistake? Tell us and we'll fix it.