How does an estimator behave across samples?

Intuition

An estimator is a rule that accepts data; an estimate is the output after one sample is observed. Repeating the data-generating experiment produces a distribution of outputs. Its center and spread describe the rule, while one run only reveals a finite collection of estimates.

Definition

The explorer uses $T=\bar X+c$ for a chosen shift $c$. Relative to fixed target $\mu$, theoretical bias is $b=E[T]-\mu$, sampling variance is $\operatorname{Var}(T)$, and mean squared error is $E[(T-\mu)^2]$. These quantities average over the specified sampling law, not over a selected favorable simulation.

Mechanics

Change the shift, sampling rule, seed, sample size and repetition count. Inspect each complete sample, its mean and resulting estimate. The empirical inspector uses all displayed estimates with denominator R. Its exact finite-run values are kept separate from independently calculated theoretical properties.

Derivation

Write $T-\mu=(T-E[T])+b$. Expanding and using $E[T-E[T]]=0$ gives $\operatorname{MSE}(T)=\operatorname{Var}(T)+b^2$. The analogous finite-list identity holds around its empirical mean when variance divides by R. Using R−1 would require a corresponding correction and would not give this displayed identity.

Worked examples

Uniform sampling at $n=4$ with shift $c=1$ gives theoretical bias 1 and variance $41/16$, so MSE is $57/16$. For the actual first sample $[2,0,6,1]$, the estimate is $13/4$. For hand estimates $[0,2]$ targeting 1, empirical bias is 0, variance is 1 and MSE is 1.

Executable reference code

Run both files. The pedagogical program exposes the arithmetic; the NumPy companion uses array operations. Literal assertions check the worked example rather than testing a simulation against a desired statistical outcome.

"""Runnable estimators reference; exact mechanics / NumPy companion."""
from fractions import Fraction as F
estimates=list(map(F,[0,2]))
target=F(1)
mean=sum(estimates)/len(estimates)
bias=mean-target
variance=sum((x-mean)**2 for x in estimates)/len(estimates)
mse=sum((x-target)**2 for x in estimates)/len(estimates)
assert (bias,variance,mse)==(F(0),F(1),F(1))
assert mse==variance+bias*bias
print(bias,variance,mse)

"""Runnable estimators reference; exact mechanics / NumPy companion."""
import numpy as np
estimates=np.array([0.,2.]); target=1.
bias=estimates.mean()-target
variance=estimates.var(ddof=0)
mse=np.mean((estimates-target)**2)
assert bias==0 and variance==1 and mse==1
assert np.isclose(mse,variance+bias**2)
print(bias,variance,mse)

Explore

Compare unbiased mean, shifted mean and biased selection. Keep the target fixed. Find a seed whose empirical signed bias disagrees with theoretical bias, then explain why this does not invalidate the theoretical formula. Increasing R refines a simulation; increasing n changes the estimator distribution.

Common mistakes and counterexamples

Finite-run agreement with a target does not prove unbiasedness. A theoretical unbiased estimator can have large sampling variance, and a biased estimator can have smaller MSE for a particular problem. Do not edit samples to make their empirical average equal the claimed expectation.

Numerical considerations

The population and selected-source moments are computed directly with rational arithmetic. Actual sample estimates and finite-run MSE identities remain exact. Theoretical standard errors are binary64 square roots. At R=1 empirical variance is zero even if theoretical sampling variance is positive; one realization cannot describe its full variability.

Engineering and ML connections

Validation scores, Monte Carlo estimates and sensor calibration are outputs of procedures. Compare error across repeated samples and distinguish systematic offset from random spread. Later regularization can trade bias against variance, but the tradeoff must be measured under a stated target and sampling law.

Exercises

Predict, calculate and explain before checking the corrective feedback. Each solution states the relevant rule and its conditions.

Summary and checkpoints

Distinguish rule from result, target from estimate, and theory from finite simulation. Derive MSE decomposition and explain why both squared bias and sampling variance matter.