If nothing else, this succinctly describes how to vet decision processes based on statistical tests.
I ran into a situation within my own organization where costly decisions were being made on variations in statistical test results that I suspected were local minima/maxima in natural variance. Replaying the decision process against the same target multiple times provided strong evidence towards changing how we did that.
It wasn't enough to just calculate the estimated population variance, etc., because explaining the significance of that was essentially a TL;DR problem to anyone whose opinion counted. It also required significantly better stats skills than I had, since adequately explaining the issue generally requires more expertise than recognizing it in the first place.
However, empirically demonstrating that the decision process was flipflopping on the same available information was extremely compelling.
I ran into a situation within my own organization where costly decisions were being made on variations in statistical test results that I suspected were local minima/maxima in natural variance. Replaying the decision process against the same target multiple times provided strong evidence towards changing how we did that.
It wasn't enough to just calculate the estimated population variance, etc., because explaining the significance of that was essentially a TL;DR problem to anyone whose opinion counted. It also required significantly better stats skills than I had, since adequately explaining the issue generally requires more expertise than recognizing it in the first place.
However, empirically demonstrating that the decision process was flipflopping on the same available information was extremely compelling.