People tend to underestimate just how noisy small samples are.

If a process has a known long-term probability of success, it does not mean that every short sequence will look anything like that probability. A 20% event can easily appear zero times in ten observations, or four times in the same sample, without anything unusual happening.

This creates a common analytical trap. Someone takes 20, 50, or even 100 observations, sees a result far away from the theoretical average, and immediately assumes that the underlying process has changed.

Most of the time, the sample is simply too small.

As sample size increases, observed frequencies generally move closer to their expected values, but the path is rarely smooth. There can be long periods where the data looks clearly biased in one direction before eventually correcting.

I randomly came across a SLOT STAT methodology piece explaining exactly why small samples can be misleading, and the logic applies far beyond their original dataset.

How do people here decide when a sample has become large enough to draw a practical conclusion rather than just describe noise?