Basics of Hypothesis testing

 

Parametric

∝ = Significance level = p(H0 is rejected | H0 is true) = Probability of making type-1 error if reject using this alpha. Before doing the statistical test, one writes down this number as: I am ok with ∝  percentage chance of rejecting the null hypothesis i.e. H0 even if H0 should not have been rejected 

p-value = p(observing the sampled parameter estimates or more extreme than it | H0 is true)

β = p(failing to reject H0 | Ha is True) = Probability of making type-2 error

1-β = Power  =  p(rejecting H0 | Ha is True)

•   Power can be influenced by increasing sample size, difference to be observed, and ∝
•   ∝%ile value as per null hypothesis, based on the effect size to be observed decide on n(shape of distribution)

Confidence Interval (say 95%)  = 95% probability, that the confidence interval contains true parameter

Bootstrap confidence Interval

1. x1, x2, x3, .......xn is a data sample drawn from distribution F
2. For each bootstrap sample δ = x_i - avg(F) 
3. For each group calculate avg(δ)
4. Find required quantile avg(δ) and make range around avg(F)
5. [avg(F) - avg(δ)_q1 ,  avg(F) - avg(δ)_q2]

Bayesian Hypothesis Testing

P(H0 | Y=y)  > P(H1 | Y=y)