TagTrace: an LLM that reads UK company accounts by their tags
Every number in a UK company filing already carries a machine-readable label. I built a system that answers questions from those labels instead of from the surrounding text, and measured how much that helps.
Supervised by Dr Ali Gheitasy · University of West London · Marked 86/100, with 20/20 for the oral defence
The problem
UK accounts are filed in iXBRL (Inline XBRL). To a person the document looks like a normal web page. Underneath, every figure is wrapped in a tag that records its accounting concept, period, currency, scale and any breakdown.
I wanted to know what happens when a general-purpose LLM is asked questions about these filings. Does it make numbers up, and can anything I build around it stop that?
Three research questions
- RQ1. How often and how severely do general-purpose LLMs fail on UK iXBRL filings without retrieval?
- RQ2. Does SelfCheckGPT-style consistency sampling, proven on free-text biographies, also detect hallucination in structured numerical output?
- RQ3. Does retrieval grounded in iXBRL's own tags improve accuracy over both a no-retrieval baseline and a detection-only approach?
Method
I wrote a direct iXBRL parser with lxml and checked it against Sainsbury's FY2025 accounts: 621 facts, with revenue reconciled to £32,812m. It feeds a structured fact store of 7,106 facts across 12 filings, 8 from FTSE 350 companies reporting under IFRS and 4 under FRS 102.
I wrote a 102-question benchmark covering extraction, computation and disclosure, with gold answers and provenance for each. I ran it under three conditions on Claude Sonnet 4.6 and replicated it on GPT-5.6 Terra:
- C1: a plain-text baseline with an 8,000-token budget.
- C2: C1 plus numeric-aware SelfCheckGPT-style consistency sampling.
- C3: tag-aware retrieval from the fact store.
A demo app then runs the C3 pipeline on filings outside the 12-filing study.
Results
| Model and condition | Correct | Incorrect | Abstained |
|---|---|---|---|
| Claude, C1 baseline | 45.1% | 2.9% | 52.0% |
| Claude, C3 retrieval | 92.2% | 3.9% | 3.9% |
| GPT-5.6 Terra, C1 baseline | 43.1% | 2.9% | 53.9% |
| GPT-5.6 Terra, C3 retrieval | 84.3% | 3.9% | 11.8% |
Each row covers 102 questions. The Claude gain from C1 to C3 is +47.1 pp (95% CI [37.3, 56.9], bootstrap n = 1000).
RQ1: the models abstained instead of fabricating
Without retrieval, 52.0% of Claude's answers were an honest "not found" and only 2.9% were confidently wrong. How often it answered correctly tracked how long the filing was compared with the 8,000-token budget: 29% correct on FTSE 350 filings against 86% on the shorter FRS 102 filings.
RQ2: the detector caught nothing
The consistency check caught 0 of 3 errors on either model. There are two reasons. Many iXBRL concepts legitimately have more than one valid value, so disagreement between samples does not mean a hallucination. And the baseline's errors were systematic misreadings, not random ones, which is exactly what consistency sampling cannot see.
RQ3: retrieval works, mainly by improving coverage
Retrieval lifted Claude from 45.1% to 92.2%. An ablation separates retrieval from presentation: unstructured retrieval alone reaches 86.3%, and presenting the facts in structured form adds about 6 points more. Questions about company-specific extension concepts rose from 15% to 92%, and dimensional questions from 17% to 88%.
Under C3, Claude beat GPT by 7.8 pp (95% CI [2.9, 12.7]). That gap comes from abstention (3.9% against 11.8%), not from the error rate on answered questions (4.1% against 4.4%, not significantly different). Coverage gains carried over to the second model. Abstention behaviour did not.
The live app
The same C3 pipeline runs as a free web app. You upload a UK iXBRL filing and ask questions in plain English. The server checks every cited fact and does any arithmetic itself, so the model cannot invent a figure, and it says so when the filing has no tagged answer.
A regression suite runs in GitHub Actions: a free retrieval check on every pull request (93% recall@15), end-to-end accuracy runs (66% correct, 5% wrong) under a $0.50 spend cap, and 54 automated tests.
Limitations
- I reduced the scope from a planned 50 filings and about 500 questions to 12 and 102, with my supervisor's approval, because project time was short. Reporting framework and company size are perfectly confounded in the corpus.
- The 8,000-token baseline budget was set by cost, not by principle.
- I did not test a finance-specialised model, and GPT-5.6 Terra could not be run at a controlled temperature.
- The planned independent dual review of the gold answers was not completed because the reviewer was unavailable.
- I made three scorer corrections and one gold-answer correction during the study. All are logged.
What I would do next
- Scale the corpus back towards 50 filings to separate company size from reporting framework.
- Test a finance-specialised model such as FinMA or FinGPT.
- Build a detector that checks against all of a concept's legitimate values instead of raw consistency.
- Extend narrative retrieval past the first matching block and add explicit disambiguation for dimensions.
- Use the ablation to show what metadata AI-ready reporting standards need.