Post on 23-Jan-2023
1
Errors in Experimental Measurements by tarungehlot Sources of errors Accuracy, precision, resolution A mathematical model of errors Confidence intervals
For means For proportions
How many measurements are needed for desired error?
2
Why do we need statistics?1. Noise, noise, noise, noise, noise!
OK – not really this type of noise
3
Why do we need statistics?
2. Aggregate data into meaningful information.445 446 397 226
388 3445 188 100247762 432 54 1298 345 2245 883977492 472 565 9991 34 882 545 4022827 572 597 364
...x
4
What is a statistic? “A quantity that is computed
from a sample [of data].” Merriam-Webster
→ A single number used to summarize a larger collection of values.
5
What are statistics? “A branch of mathematics dealing
with the collection, analysis, interpretation, and presentation of masses of numerical data.”
Merriam-Webster → We are most interested in analysis and interpretation here.
“Lies, damn lies, and statistics!”
6
Goals Provide intuitive conceptual
background for some standard statistical tools.
Draw meaningful conclusions in presence of noisy measurements.
Allow you to correctly and intelligently apply techniques in new situations.
→ Don’t simply plug and crank from a formula.
7
Goals Present techniques for
aggregating large quantities of data.
Obtain a big-picture view of your results.
Obtain new insights from complex measurement and simulation results.
→ E.g. How does a new feature impact the overall system?
9
Experimental errors Errors → noise in measured values Systematic errors
Result of an experimental “mistake” Typically produce constant or slowly varying bias
Controlled through skill of experimenter
Examples Temperature change causes clock drift Forget to clear cache before timing run
10
Experimental errors Random errors
Unpredictable, non-deterministic Unbiased → equal probability of increasing
or decreasing measured value Result of
Limitations of measuring tool Observer reading output of tool Random processes within system
Typically cannot be controlled Use statistical tools to characterize and
quantify
12
Quantization error Timer resolution → quantization error
Repeated measurements X ± ΔCompletely unpredictable
14
A Model of ErrorsError 1 Error 2 Measured
valueProbability
-E -E x – 2E ¼
-E +E x ¼
+E -E x ¼
+E +E x + 2E ¼
17
A Model of Errors Pr(X=xi) = Pr(measure xi)= number of paths from real value to xi
Pr(X=xi) ~ binomial distribution As number of error sources becomes large n ,→ ∞ Binomial → Gaussian (Normal)
Thus, the bell curve
18
Frequency of Measuring Specific Values
Mean of measured values
True valueResolution
Precision
Accuracy
19
Accuracy, Precision, Resolution
Systematic errors → accuracy How close mean of measured values is to true value
Random errors → precision Repeatability of measurements
Characteristics of tools → resolution Smallest increment between measured values
20
Quantifying Accuracy, Precision, Resolution Accuracy
Hard to determine true accuracy Relative to a predefined standard
E.g. definition of a “second” Resolution
Dependent on tools Precision
Quantify amount of imprecision using statistical tools
23
Confidence Interval for the Mean Normalized z follows a Student’s t distribution (n-1) degrees of freedom Area left of c2 = 1 – α/2 Tabulated values for t
c1 c2
1-α
α/2 α/2
24
Confidence Interval for the Mean As n → ∞, normalized distribution becomes Gaussian (normal)
c1 c2
1-α
α/2 α/2
28
An Example (cont.)
90% CI → 90% chance actual value in interval
90% CI → α = 0.10 1 - α /2 = 0.95
n = 8 → 7 degrees of freedom
c1 c2
1-α
α/2 α/2
29
90% Confidence Intervala
n 0.90 0.95 0.975
… … … …5 1.47
62.015
2.571
6 1.440
1.943
2.447
7 1.415
1.895
2.365
… … … …∞ 1.28
21.645
1.960
4.98
)14.2(895.194.7
5.68
)14.2(895.194.7
895.195.02/10.012/1
2
1
7;95.01;
c
c
tta
na
30
95% Confidence Intervala
n 0.90 0.95 0.975
… … … …5 1.47
62.015
2.571
6 1.440
1.943
2.447
7 1.415
1.895
2.365
… … … …∞ 1.28
21.645
1.960
7.98
)14.2(365.294.7
1.68
)14.2(365.294.7
365.2975.02/10.012/1
2
1
7;975.01;
c
c
tta
na
31
What does it mean? 90% CI = [6.5, 9.4]
90% chance real value is between 6.5, 9.4
95% CI = [6.1, 9.7] 95% chance real value is between 6.1, 9.7
Why is interval wider when we are more confident?
33
Key Assumption Measurement errors are Normally distributed.
Is this true for most measurements on real computer systems?
c1 c2
1-α
α/2
α/2
34
Key Assumption Saved by the Central Limit TheoremSum of a “large number” of values from any
distribution will be Normally (Gaussian) distributed.
What is a “large number?” Typically assumed to be >≈ 6 or 7.
35
How many measurements? Width of interval inversely proportional to √n
Want to minimize number of measurements
Find confidence interval for mean, such that: Pr(actual mean in interval) = (1 – α) xexecc )1(,)1(),( 21
37
How many measurements? But n depends on knowing mean and standard deviation!
Estimate s with small number of measurements
Use this s to find n needed for desired interval width
38
How many measurements? Mean = 7.94 s Standard deviation = 2.14 s Want 90% confidence mean is within 7% of actual mean.
39
How many measurements? Mean = 7.94 s Standard deviation = 2.14 s Want 90% confidence mean is within 7% of actual mean.
α = 0.90 (1-α/2) = 0.95 Error = ± 3.5% e = 0.035
40
How many measurements?
9.212)94.7(035.0)14.2(895.12
2/1
exszn
213 measurements→ 90% chance true mean is within ± 3.5% interval
41
Proportions p = Pr(success) in n trials of binomial experiment
Estimate proportion: p = m/n m = number of successes n = total number of trials
43
Proportions How much time does processor spend in OS?
Interrupt every 10 ms Increment counters
n = number of interrupts m = number of interrupts when PC within OS
44
Proportions How much time does processor spend in OS?
Interrupt every 10 ms Increment counters
n = number of interrupts m = number of interrupts when PC within OS
Run for 1 minute n = 6000 m = 658
45
Proportions
)1176.0,1018.0(6000)1097.01(1097.096.11097.0
)1(),( 2/121
n
ppzpcc
95% confidence interval for proportion So 95% certain processor spends 10.2-11.8% of its time in OS
48
Number of measurements for proportions How long to run OS experiment? Want 95% confidence ± 0.5% e = 0.005 p = 0.1097
49
Number of measurements for proportions
102,247,1
)1097.0(005.0)1097.01)(1097.0()960.1(
)()1(
2
2
2
22/1
peppzn
10 ms interrupts→ 3.46 hours
50
Important Points Use statistics to
Deal with noisy measurements Aggregate large amounts of data
Errors in measurements are due to: Accuracy, precision, resolution of tools
Other sources of noise→ Systematic, random errors
51
Important Points: Model errors with bell curve
True value
Precision
Mean of measured valuesResolution
Accuracy