AI in Accounting: Ask Twice, Get Two Answers

AI in accounting returns different answers to the same question because large language models sample probabilistically and their serving infrastructure is not deterministic, even at temperature zero. The fix has two layers: a shared context layer holding firm rules once, and a deterministic validation step that decides routine cases in code.

Here's the problem nobody puts on the webinar slide: when you couple AI with a traditional accounting process, you will often get different results over time - from the same data, the same question, even the same prompt.

That's not a bug in one product. It's the nature of the technology. And it collides head-on with the entire point of the profession. Accounting doesn't reward brilliance; it rewards consistency. The goal of every accountant, every close, every workpaper is work that is consistent, reproducible, and auditable. An answer that's 95% likely to be right - but different every time you ask - is, for a CPA, worse than useless. It's a liability with a login screen.

I say this as someone who is deeply bullish on AI in accounting. The efficiency gains are real. But you only capture them if you understand where the variance comes from - and build for it deliberately.

Why the same question gets different answers

Large language models are probabilistic by design. They don't retrieve the answer; they generate an answer, sampling from a distribution of plausible next words. Ask an LLM to categorize the same $247 Home Depot receipt in January, March, and June, and you can plausibly get Supplies & Materials, then Cost of Goods Sold, then Repairs & Maintenance. Each one is defensible on its own. Together, they quietly corrupt your books: false spending trends, phantom category spikes, quarter-over-quarter comparisons that compare nothing. And the worst part is that this unreliability is quiet. Nothing errors out. Nothing warns you. The drift just accumulates until someone - a partner, a lender, an auditor - asks why the numbers moved.

Here's the part most people don't know: you can't fully fix this with settings. The natural instinct is "just turn the temperature to zero" - force the model to always pick the most likely answer. Researchers at Thinking Machines Lab tested this rigorously in their essay "Defeating Nondeterminism in LLM Inference." They gave a state-of-the-art model the identical prompt 1,000 times at temperature zero - the setting that's supposed to be fully deterministic - and got 80 distinct answers. The completions were identical for the first 102 tokens, then split. The cause isn't randomness in the model; it's the serving infrastructure. How your request gets batched with other people's requests on the provider's servers changes the low-level floating-point arithmetic, and tiny numerical differences cascade into different words. In other words: the load on OpenAI's or Anthropic's servers at the moment you hit enter can change your answer. That is unavoidable with essentially every general-purpose AI tool on the market today, because you don't control their inference stack.

And there's a second, more mundane source of drift that compounds the first: context. Every AI session starts fresh. The rules you carefully explained on Tuesday - "we capitalize equipment over $2,500," "job-site materials are always COGS" - evaporate by Thursday. Different staff members prompt differently. Different tools hold different fragments of firm policy. Multiply that across a team and a tax year, and variance isn't a risk. It's a certainty. If the vocabulary here is new, our decision intelligence concepts glossary defines the terms this argument leans on.

Most firms learn this the hard way. The pilot goes great. The demo dazzles. Then three months in, someone reconciles AI-assisted work against the prior quarter and finds the drift - and the cleanup costs more than the AI saved. If that story sounds familiar, you're not behind. You're exactly where most of the profession is right now.

The two-layer fix: shared context, deterministic checks

The good news is this problem has a structural solution, not a "prompt better" solution. It takes two layers.

Layer one: a shared context layer. Your firm's hard rules - chart of accounts logic, capitalization thresholds, materiality policies, client-specific treatments, entity structures - live in one governed place, and every AI tool and every AI session draws from it, every time. No re-explaining. No tribal knowledge trapped in one senior associate's prompt history. The same rules are applied on the first ask in January and the thousandth ask in December, whether it's the partner or the new hire asking, and regardless of which AI tool they're using that day. This alone eliminates the largest and most preventable source of variance: the model improvising because nobody told it the rules.

Layer two: a deterministic validation step. Every calculation and every AI response gets checked against deterministic rules that the CPA firm (or the user) defines - plain old if-then logic, executed in code, that produces the same output every single time. Home Depot + this client + under $2,500 -> Supplies, full stop. Debits equal credits, full stop. AI proposes; deterministic rules dispose. Anything the rules can verify gets verified; anything they can't gets flagged for a human, not silently guessed. The walkthrough of how SIGNLD works shows where each layer sits in the pipeline.

It's a belt-and-suspenders approach, and it changes the economics in four ways:

A) You keep the AI efficiencies. The model still does what it's uniquely good at - reading messy source documents, drafting narratives, spotting anomalies, handling the ambiguous edge cases. You're not retreating from AI; you're fencing it.

B) You get more consistent and more accurate results. Errors get caught at the validation layer instead of at review - or worse, at audit. The AI's judgment is applied only where judgment is actually required.

C) You burn dramatically fewer tokens. This is the part nobody talks about. Every time you make an LLM re-infer something a rule could have decided, you're paying for inference you didn't need. Run the back-of-envelope math: in a typical small-business ledger, somewhere between 70% and 90% of transactions are routine and repetitive - the same vendors, the same treatments, month after month. A single AI categorization call, with the receipt text and instructions in context, typically consumes on the order of 500-1,500 tokens. A firm processing 50,000 transactions a month through pure AI inference is spending roughly 25-75 million tokens monthly on decisions a rules table resolves for fractions of a cent - instantly, and identically every time. Route the routine 80% through deterministic rules and reserve inference for the genuinely ambiguous 20%, and you've cut your AI spend by roughly three-quarters while making the output more reliable. (These are estimates - your mix will vary - but the direction is not in doubt: deterministic rules cost effectively nothing to execute; inference never does.)

D) The end results are consistent and reproducible. Same inputs, same rules, same outputs - in March, in September, and when the auditor re-performs the work next year. That's not a nice-to-have. That's the definition of the job.

How we learned this (the hard way, naturally)

I'll be honest about where this thinking comes from. We didn't arrive at the two-layer architecture on a whiteboard. We arrived at it by stepping on every rake described above while building SIGNLD.

Early on, we did what everyone does: pointed powerful models at financial data and marveled at the demos. Then we started asking the same questions twice. The answers were good - they just weren't the same. And we were building for CFOs and firm leaders, people who will forgive a system for saying "I don't know" long before they forgive it for saying two different things on Tuesday and Thursday. The CFO use-case walkthrough covers the questions those leaders bring first.

So we rebuilt around the two layers. In SIGNLD, the shared context layer is a private knowledge graph - your accounts, entities, policies, and definitions from across all your systems, living in one governed place that every AI session and every connected AI tool draws from, every time. Nobody re-explains the capitalization threshold; it's just there. And the validation layer is deterministic and traceable by design: figures are computed by rules, not vibes, and every number links back to the source rows it came from - so when someone asks "where did this come from?", the answer is a click, not a shrug. The AI does the language, the reasoning, the ranking of what matters. The arithmetic and the rules never get delegated to a probability distribution. Our security and data handling details cover how that context layer is isolated and where inference runs.

And yes, that includes transaction classification - with a twist I've become genuinely attached to. You don't program the rules. You teach them. SIGNLD watches a human classify transactions the way they always have, proposes the deterministic rule it thinks it just witnessed - "Home Depot, this client, under $2,500 -> Supplies" - and the human confirms, corrects, or refines it. The confirmed rule goes into the knowledge graph as plain text. Not weights, not embeddings, not a black box: a markdown file a partner can open, read, and red-pen at any time. If you can review a workpaper, you can review the rules your AI is running on. That reviewability isn't a nice UI touch; it's the whole point. A rule you can't inspect is just a slower, more expensive probability distribution.

The part that took us longest - and that I haven't seen anyone else do - is rule versioning. Firms' policies aren't static; they change mid-year, mid-engagement, mid-argument. The naive systems handle this badly in one of two ways: either they silently rewrite history (last year's numbers quietly shift under this year's logic) or they freeze it (you can never improve a rule without breaking comparability). SIGNLD keeps every version of every rule, timestamped in the knowledge graph, so it can do both honestly. Change a classification rule today, and you can view any prior period two ways: as reported - computed under the rules in place at the time, exactly as the books stood - and as restated - the same source data run through all current rules. Then it shows you the variance between the two, decomposed and tied back to the individual rule changes that caused it. Not "Q2 supplies moved by 4,000," but "Q2 supplies moved by 4,000, of which 1,200 traces to the March change in the Home Depot rule and $2,800 to the new capitalization threshold." It's the accounting equivalent of a diff with blame annotations - a smoothed, comparable view alongside the literal one, with nothing hidden in between.

If you've ever restated comparatives by hand, or fielded the question "why doesn't this quarter tie to what we showed the bank in April?", you know why this matters. Consistency over time doesn't mean rules never change. It means that when they change, you can prove exactly what changed, when, and what it did to every number downstream.

I'm not claiming we've solved AI for the profession. I'm claiming we found the failure mode early, paid our tuition, and built the fix into the foundation instead of the marketing.

The Pepsi challenge

So here's my provocation, in two parts.

First: tell me if you've experienced this. Ask your current AI tool the same substantive question about the same data, a few days apart, and compare the answers word for word and number for number. If they match every time, I'd sincerely love to know what you're using. If they don't - you now know why, and you know it's not your prompting.

Second: if you have felt this drift, run our approach head-to-head against whatever you're doing today. Same data, same questions, same month. See which one gives you answers you'd be comfortable initialing in a workpaper. Then come tell me the result - publicly, in the comments, either way. If it works better, great, we both learned something. If we're all wet, I want to hear that too, because that's cheaper tuition than the alternative.

The honest bottom line

Thinking Machines showed that even the AI labs themselves have to re-engineer their inference stacks - accepting real performance penalties - just to make a model give the same answer twice. If achieving reproducibility takes that much work inside the model, the practical path for an accounting firm is to stop demanding it from the model at all. Let the AI be brilliant and probabilistic where brilliance helps. Let deterministic rules be boring and identical where boring is the entire point.

The firms that will win with AI aren't the ones with the cleverest prompts. They're the ones whose AI gives the same right answer every time - because they stopped leaving consistency up to a probability distribution.

Source: "Defeating Nondeterminism in LLM Inference," Thinking Machines Lab (thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference) - 1,000 temperature-zero runs of an identical prompt produced 80 distinct completions, diverging after token 102, due to batch-dependent floating-point effects in serving infrastructure. Token-savings figures are the author's estimates.

Key takeaways

The same prompt on the same accounting data can return different treatments, and nothing errors out when it does.
Temperature zero does not remove the variance, because batch-dependent floating-point effects in the provider's serving stack change the output.
A shared context layer holds firm rules in one governed place so no session has to be re-taught the capitalization threshold.
A deterministic validation step resolves the routine majority of transactions in code and reserves inference for genuinely ambiguous ones.
Rule versioning lets you show any prior period as reported and as restated, with the variance traced back to the rule change that caused it.

FAQ

Why does AI in accounting give different answers to the same question?

Because a language model generates answers by sampling from a probability distribution rather than retrieving a stored result. Two identical prompts can follow different word paths and land on different account treatments, both defensible in isolation. Serving infrastructure adds a second source of variance, so even careful prompting cannot make the output reliably identical run to run.

Does setting temperature to zero make AI output deterministic?

No. Thinking Machines Lab ran an identical prompt 1,000 times at temperature zero and got 80 distinct completions, which diverged after the first 102 tokens. The cause is batch-dependent floating-point arithmetic in the provider's serving stack, not sampling randomness. Since you do not control that stack, temperature settings alone cannot guarantee reproducible answers.

What is a shared context layer in an accounting AI workflow?

It is one governed place that holds a firm's hard rules: chart of accounts logic, capitalization thresholds, materiality policies, client-specific treatments, and entity structures. Every AI session and every connected AI tool reads from it, so nobody re-explains policy in a prompt. That removes the largest preventable source of variance, which is the model improvising because it was never told the rules.

How do deterministic rules reduce AI costs?

Most transactions in a small-business ledger are routine and repetitive. Resolving them with if-then rules executed in code costs a fraction of a cent, while a single categorization call to a model can consume roughly 500 to 1,500 tokens. Routing the routine majority through rules and reserving inference for ambiguous cases cuts spend substantially while making output more consistent.

Can classification rules change without breaking prior-period comparability?

Yes, if every version of every rule is kept and timestamped. Then any prior period can be shown two ways: as reported, computed under the rules in force at the time, and as restated, the same source data run through current rules. The variance between the two can be decomposed and attributed to the specific rule changes that caused it.

Where should a firm still let AI make the call?

On the work that is genuinely ambiguous or linguistic: reading messy source documents, drafting narratives, spotting anomalies, and handling edge cases the rules do not cover. Arithmetic, balancing checks, and repeat vendor treatments belong in deterministic code. The division of labor is the point, and anything the rules cannot verify should be flagged for a human rather than silently guessed.

Try SIGNLD free

Connect a system, ask the question you ask every month, and check whether the answer holds up twice. Try SIGNLD free or browse all articles for more on traceable answers and deterministic rules.

The Dirty Secret of AI in Accounting: Ask Twice, Get Two Answers

In this article