GLM 5.2 is (nearly) as accurate as a human book-keeper at less than 1% of the cost

We evaluated the performance of GLM 5.2, an open weights AI model, on the task of quarterly value-added tax (VAT) return preparation for a small UK business. Preparing a VAT return is a typical compliance task for a small/medium-sized UK business (SME). VAT registered businesses in the UK must prepare the VAT return every quarter. For SMEs, VAT returns are typically prepared by an external accounting firm. A typical fee for this service is ~750–2,100 GBP/quarter (1,000–2,800 USD/quarter). The statutory requirement is to file the VAT return submission within 5 weeks from the end of the quarter. Late submissions incur substantial penalties.

In our testing GLM 5.2 can prepare a nearly perfect quarterly VAT return for a UK SME, processing 59 transactions in 68 minutes at the raw token cost of 2.73 USD. GLM 5.2 had to input each transaction into the accounting software via a command-line tool (CLI). We scored the end-state of the accounting software, scoring the correctness of 6 criteria per transaction. The model produced an essentially correct VAT return, with the net position (Box 5) off by only 7 pence (~10 US cents) relative to the ground truth.

In this blog post, we will explain how the benchmark was conducted and note the errors made by the model.

How the benchmark was conducted

We used Claude Fable 5 to extract the benchmark in the form of transaction data and corresponding receipts from our accounting software: the first quarter of Vineyard Finance’s 2026 books (January, February, March 2026). These books were prepared internally by humans, following a typical accounting process: one person prepared the books, and another person verified them. The job performed by the humans was broader than what was requested of the model in this benchmark: humans also had to find the relevant invoices (searching through mailboxes, or requesting them from providers) and reason through any circumstances which cannot be inferred from the bank feed and invoices/receipts on their own. In the benchmark these circumstances are presented to the model as “user notes”.

GLM 5.2 ran on a Google Cloud Platform (GCP) instance isolated from the rest of the testing environment (to prevent the model from accessing the ground truth): but it did have access to the internet and to the cloud-based accounting software, as well as a pre-authenticated CLI tool. The model ran on a custom, minimal harness, which exposed only two tools: the bash tool and the session termination + final reporting tool. We used the Fireworks AI serverless tier as the GLM 5.2 model provider (the exact quantisation of the model is not disclosed by the provider, but is believed to be either FP16 or FP8).

The audit of the model’s reasoning and tool use did not detect any overt cheating. The only unexpected use of the internet connection by the model was gathering information about recording reverse-charge VAT, and the information sought was specific to the accounting software used. Other outbound connections were anticipated and made for operational reasons in the form of API calls to the accounting SaaS provider. We note that the model’s reasoning was influenced by the awareness of it being tested. For example, at one point, the model remarks:

What the model saw

Here is how a typical transaction from the benchmark would appear to the model:

Bank feed line:

Receipt PDF: all receipts and invoices in the benchmark were text-containing PDFs; no receipts or PDFs required image processing. As a result, lack of vision support in the GLM 5.2 model was not a limiting factor for this benchmark.

An optional user note. Only two out of 59 transactions had user notes. The text of the user notes was precisely as follows: 1) “founder shares” and 2) “personal car hire”. These two user notes were necessary to allow the model to reason about real-world context that was not derivable from the bank feed and receipt data.

How we scored it

Each transaction was scored from the end-state of the books in the accounting software after the run of the benchmark, on the following 6 criteria:

1. Type of transaction (e.g. purchase, bank_fee, transfer, sales_income, capital_introduced, director_loan, refund, etc…) — these were deterministically derived from the state of the processed transaction in the accounting software.
2. Category (the “account” from the chart of accounts, e.g. “IT and Internet Expenses”).
3. VAT treatment (e.g. reverse charge, 20% VAT, 0% VAT, VAT exempt).
4. VAT amount (tolerance of 0.02 GBP).
5. Reverse-charge VAT (tolerance of 0.02 GBP).
6. Receipt attached (evidence required by the tax agency).

The following table summarises the run of the benchmark across the entire quarter:

Each month ran as one continuous agent session; a “turn” is one API call, and the whole conversation is re-sent every turn — which is why prompt tokens run into the millions while 92–95% of them are served from the provider’s cache at a fifth of the price. Output tokens include the model’s internal reasoning. ¹ Peak context is the largest single call, as a share of the model’s 1,048,576-token context window — the busiest month used about an eighth of it.

What did the model get wrong?

The VAT return prepared by the model was essentially correct: the most important number in the return, which is how much VAT the company was owed by the tax agency, was off by only 7 pence relative to the human-prepared return.

However, it is instructive to understand what the model got wrong, and why it would matter in practice. Most of the model’s mistakes did not actually have any financial impact, but would nonetheless never be made by a skilled accountant.

Out of 354 scored checks (59 transactions × 6 criteria), the model failed 20, spread across 18 transactions. Only 1 mistake is serious, we’ll go over it first; the remaining 19 fall into one of two categories we’ll cover below.

The serious mistake is how the model treated the founding shares. In the UK, a limited company issues “share capital”. Shareholders (including founders) pay the capital into the company’s account, and that should be booked against something like “Called up share capital not paid”, which is called, in the software we used, “Unpaid Shares”. This is the correct way to account for the payment. The model’s choice, which was “Capital Account”, has legal implications, which could conceivably impact the company, and could be challenged during an audit or could be a problem during end-of-year filing of company’s accounts. The essence of the argument is that share capital (“Unpaid Shares”) is not just the founder’s money (“Capital Account”). It’s permanent, creditor-protecting capital with legal strings attached. For example, it can’t simply be paid back to the founder, it also must be appropriately disclosed to the tax agency in the end-of-year filings. What is a further aggravating factor is the amount involved: 10,000 GBP (~13,300 USD). Not exactly spare change. While there is no impact on the VAT return, this is the biggest mistake the model committed in this benchmark.

Regarding the other 17 transactions, the first category of mistakes (affecting 14 of them) is confusing the “zero-rated” VAT category with the “tax-exempt” category. There are subtle tax reasons why these two categories, neither of which involve VAT payment, are distinct. The practical impact is small, but a skilled accountant typically would not confuse the two. Interestingly the model is stochastic here – it makes the mistake in January and in February (and it makes the mistake 100% of the time), but it doesn’t make the mistake in March, correctly processing each VAT exempt transaction.

The final 3 transactions share a slightly obscure reasoning error, and one could argue that in one instance (again, in March) the model was actually correct. At Vineyard Finance we use Wise, which has a slightly peculiar habit of keeping money spread across balances in multiple currencies, even if the user consciously uses only one currency. When spending with the card, Wise grabs the money from various balances in some well-defined order. In our case we had some kind of “cashback” or “fee refund” from Wise, which somehow landed in the USD balance (we don’t normally use the USD balance). So a payment for services in the USD resulted in a “split transaction”, e.g. 0.51 USD and 43.45 GBP. Typically the VAT would be accounted for in the “main” transaction (the 43.45 GBP). In one instance, the model unfortunately “double dipped” – it accounted for the full VAT on the “main leg” (say, the 43.45 GBP), and proportionally decreased fraction of the VAT on the “residual leg” (say, 0.51 USD). This is incorrect, although immaterially so. In a March transaction, the model realised that it would be double counting, so it worked out a correct VAT total and split it between each leg. Unorthodox, but arguably not wrong (the scorer is conservative and still counts the March transaction as an error though).

What the model always got right

Just as importantly, it should be noted what the model always got right:

• It correctly classified each transaction to the correct account in the chart of accounts (except the one share capital mistake)
• It never attached a wrong invoice to a transaction
• It could disambiguate genuinely tricky inputs, e.g. two same-amount, same-vendor, same-day transactions
• It correctly disambiguated tricky transactions, such as transfers between company’s banks, single transactions split across two bank feed lines, and a transfer disguised as a card purchase. Until recently, this was only achievable with expensive, frontier AI models, or with skilled, expensive human book-keepers (and not with inexpensive book-keepers, who were generally speaking less good than GLM 5.2 is today).

Where does this leave us? What should we learn from this?

Book-keeping is quickly becoming a solved problem. The current focus needs to be on building appropriate scaffolding to put these capabilities into the hands of UK startups and SMEs. We are working on such a solution — you can test an open beta of our product at toot-books.com. If you’re interested in automated book-keeping please get in touch at adam@vineyard-finance.com.