Why does AI get numbers wrong when analyzing data?
An error can occur before the system adds anything. That's why it helps to trace how it reached the answer.
Access and reading: The assistant may receive an old file, read the wrong sheet, or interpret a column as text. It can also confuse numbers when extracting them from a scanned PDF. The OpenAI data analysis guide warns that extracting table values from images or complex layouts may not be reliable enough.
Selection: The system may exclude valid records, use the wrong period, or duplicate rows when joining tables. A query can run without errors and still answer a different question.
Metric definition: “Overdue receivables” could mean the outstanding balance as of the cutoff date or the original value of overdue invoices. Without an agreed definition, two different results can both seem reasonable.
Calculation and interpretation: A formula may point to the wrong range, or the assistant may describe a total as an average. A clear answer doesn't prove the calculation is correct.
Even if a system recognizes a table's structure, it can cite a value incorrectly or leave one out. These failures appear in a study by Yang and coauthors on reading tables. Checking the structure alone doesn't confirm that the values used are correct.
How can you check a figure before using it?
Keep the original file and work on a copy. That way, you can repeat the test without changing the evidence of the error.
- Confirm what data the system received. Check the file name and version, sheet, and headers. Compare the number of records with the source. For a scanned PDF, compare extracted values with the visible cells. The row count can reveal missing records, but it doesn't prove every value is correct.
- Define the question before calculating. Agree on the metric, period, and exclusions with the person who uses the result. The OpenAI Academy data analysis guide recommends specifying those criteria and asking for assumptions and formulas. Check for missing data and unusual values, too.
- Inspect the selected records. Ask for a list of included and excluded rows, with the reason for each exclusion. If the system joins tables, verify that each operation keeps the expected number of records. An invoice repeated because it has several associated payments can inflate the total.
- Review the calculation it ran. If the tool runs code or a SQL query, inspect the filters, operations, and intermediate results. Check the calculation with an independent formula or query. If you can't see how it calculated the figure, ask for evidence. A written explanation doesn't prove it ran that calculation.
- Repeat the test after the fix. Save a small set of cases with results calculated and reviewed by a person. Include missing records, duplicates, and cutoff dates. Compare results before and after the change. Check both incorrect figures and correct figures the system flagged by mistake.
What does a test with known results look like?
This example is illustrative; it doesn't describe a client.
An accounts receivable team provides a file with four invoices. Their outstanding balances are 100, 200, 50, and 80. At the agreed cutoff date, the invoices with balances of 100 and 80 are overdue; the others aren't.
The assistant is asked to add the outstanding balances of invoices overdue at the cutoff date. The expected result is 180, with two invoices included. The system should show which invoices it used and what formula or query it ran.
If it answers 280, the team checks the invoice list before changing the prompt. It may have included an invoice that wasn't overdue or duplicated the invoice with a balance of 100 when joining data. The list and query help distinguish between those causes.
After fixing the filter or join, we repeat the test with the same file. We also test a case with no overdue invoices and another with missing values. We save the file version, prompt, and results so we can compare future changes.
The same approach applies to an accounts receivable aging report. First, agree on the balance and cutoff date; then check each classification.
Which option should you use to get reliable figures?
The choice depends on where the data lives, how often you calculate the metric, and what evidence you need to keep.
| Option | When to use it | What to check |
|---|---|---|
| Existing formula, SQL query, or dashboard | The metric is stable and the data is already organized. | Permissions, filters, and metric definition. |
| Your tool's analysis feature | You need to explore a file and can review the calculation. | File used, selected records, and available code. |
| Automation platform | You need to move data between tools and apply repeatable rules. | Connections, duplicates, and error logs. |
| Custom development | You need specific controls or must integrate several sources. | Data access, tests, maintenance, and ownership. |
Cost and timing depend on data quality and access, integrations, and the level of review needed. If the system is already failing, document the error. That helps you decide whether to fix it or seek AI project rescue.
