Predictions are big business. Joe Lunardi makes a pretty decent living predicting the NCAA bracket each year. August BCS predictions often influence December bowl selections. In fact, it seems like every year, any Joe Schmo with a computer and some numbers will predict something and ask you to believe him. But with any predictive algorithm, the question is always the same: how accurate is it?

We regularly test the accuracy of the BeastScore algorithm, and the CrossFit Games Open presented us with another opportunity to do so. We randomly selected 42 athletes (see tables below) and ranked them according to BeastScore as well as their finish at this year’s Open. If BeastScore was perfectly accurate at predicting relative competition placement, then these athletes’ BeastScore rank would perfectly match their Open rank.

Of course no predictive algorithm is perfect, but BeastScore’s accuracy is pretty impressive. For the 42 women that we evaluated, the average difference between their BeastScore rank and Open rank was about 2.8. In other words, for these 42 athletes, on average BeastScore predicted their relative Open ranking within 3 places. For three men and two women, their BeastScore rank exactly matched their Open rank.

One key result is how well BeastScore would have properly segmented the athletes into division levels for a competition. Keep in mind BeastScore was invented for precisely this purpose (matching athletes of similar ability for fairer competition), so one would expect it to do this well. Indeed it does, and the Open data confirms it. Let’s assume this year’s Open consisted of only the 23 men shown below. And let’s also assume Dave Castro, Director of the CrossFit Games, wanted to have two division levels (with the top 12 athletes by BeastScore placed in Division I, and the bottom 11 athletes placed in Division II). Had this been the case, BeastScore would have divided the athletes almost perfectly. For the 12 athletes in Division I, all but one finished in the top 12. For the 11 athletes in Division II, all but one finished in the bottom 11. (The ‘Law of Conservation of Athletes’ dictates that number of athletes crossing over is the same…) If the same scenario is repeated for the 19 women looked at, all but two perform in line with the division they were placed. That’s really good! Even coaches who have known their athletes for years couldn’t split them into divisions that well.