The permutation test is a statistical test for outcome differences (continuous dependent variable) between groups (categorical independent variable). For example, the hypothesis may be “Do the number of minutes of exercise per week (continuous numerical values) differ between men and women (categorical groups)?”

The permutation test is an alternative to say a student t-test. The benefit of the permutation test is it requires no assumptions about the variable distribution, e.g. the outcome variables come from a normal distribution, as you generate a distribution from the data.

So how do we generate that distribution? The permutation test is essentially mixing up the group labels of the data. If there were no difference between the two groups, the observed outcome differences should be pretty likely if we randomly mix up our labels.

To get a sense of this you can play around with the app below or for a little better visibility here. (This post was partly created so I could try out embedding a shiny app in a WordPress post).

Try playing around with how the mean differences change with the number of labels swapped and how the p-values, roughly the estimate of likelihood of the outcome, change with more or less samples.