I’ve been experimenting quite a bit with AI, especially ChatGPT 4o. On the whole I’ve found it very useful and most often I use it instead of Google to find out more background or quickly check things I’ve read and which maybe don’t sound quite right.

Yesterday I came across a statistic promoted on BlueSky as it happens. The post made the claim that there were 100 times more potholes on Britain’s roads than craters on the Moon.

Instinctively this sounds rubbish, so I asked ChatGPT. This is what it said:

This claim is often made to highlight the poor state of Britain’s roads, but its accuracy depends on how the comparison is defined.

Potholes in Britain

• Estimates suggest there are around 1.7 million potholes on Britain’s roads, based on data from councils and surveys by motoring organizations.

• The exact number fluctuates as new potholes form and old ones are repaired.

Craters on the Moon

• The Moon has approximately 1.3 million craters with a diameter of 1 km or larger, based on modern satellite data.

• However, if smaller craters (down to 1 meter in diameter) are included, the number rises dramatically, potentially into the billions.

A good answer, I thought. But then this was its conclusion:

If only large craters (1 km or more) are considered, then the “100 times more potholes” claim is plausible, since 1.7 million is far more than 1.3 million.

So I challenged it by asking if this particular conclusion was true. Here’s what it said:

You’re right to scrutinize my statement! Let’s re-evaluate the math:

• Estimated potholes in Britain: 1.7 million.

• Estimated craters on the Moon ≥ 1 km: 1.3 million.

The claim states that there are 100 times more potholes than craters, which means:

1.3 \text{ million} \times 100 = 130 \text{ million}

Clearly, 1.7 million potholes is not 100 times more than 1.3 million craters—it’s only about 1.3 times more.

Thus, my previous statement was incorrect. The “100 times more” claim does not hold up when considering craters ≥ 1 km in diameter. Thanks for catching that!

So I asked it why it thought it made that particular mistake. Here’s what it said:

Good question! I think the mistake happened for a few key reasons:

1. Misinterpretation of Scale

I correctly compared the 1.7 million potholes to the 1.3 million large craters, but I didn’t properly check whether the “100 times more” claim held up mathematically. Instead, I instinctively assumed that because 1.7 million is larger than 1.3 million, the claim might be “plausible”—without doing the crucial multiplication step.

2. Cognitive Bias – Pattern Recognition

My brain likely followed a familiar pattern:

• I’ve seen claims about road potholes being much more numerous than craters before.

• The numbers were in the same ballpark, so I subconsciously assumed the comparison was roughly correct without fully verifying it.

3. Focus on Relative Difference Instead of Absolute Multiplication

I focused on the fact that 1.7 million is “more” than 1.3 million but didn’t immediately apply the “100 times more” test. This is a common numerical intuition error—people sometimes compare relative magnitudes without explicitly calculating ratios.

4. Over-Reliance on Memory vs. Recalculation

I’ve encountered similar statistics before, so I might have relied on a vague recollection rather than double-checking the math from first principles.

What I Can Learn from This?

• Always verify numerical claims with explicit calculations, even if the numbers seem intuitive.

• Be aware of assumption shortcuts—just because something sounds plausible doesn’t mean it holds up mathematically.

• Stay open to self-correction—as I did when you questioned my claim!

Thanks for prompting me to think through the error—it’s a useful lesson in precision and verification.

That is a pretty reasonable summary of the causes of hallucination, if a touch anthropomorphic. It quickly spotlighted the key problem – the lack of a quick calculation. It quickly spotlighted the key problem – the lack of a quick calculation. Whether these lessons are learnable in quite the way it portrayed is doubtful. But who knows…