Here at FantasyRef.ca I have created a ranking system to evaluate and compare NHL Fantasy projections. The algorithm is straight forward and can be viewed here. In coming up with the ranking system a few decisions were made in the process about;
In this article I explain my reasoning and thought process behind the decisions that were made.
The goal of any fantasy hockey pool is to get the most points possible. The players which are selected first are the players that are going to get the most points for your team. Thus, the consideration of the importance and impact of each individual player projection is required. We have created a points system that factors in this importance by weighting the points for each player range based on actual point total of players in that range. It is more important for poolers that fantasy projections are the most accurate for the players that will give the most points overall. As point totals decrease, so does the importance of the accuracy of the projections. With the weighting system that I have implemented in the calculations, more value is given to the projections of players that will get the most points.
This is something that has been a struggle to pin down from the initial conception of FantasyRef.ca. In year one, the average ranking of forecasters RANK projections (for Pts, G, A, Wins, and Shutouts) determined the overall winner. In year two, the average ranking of forecasters TOTAL and RANK projections (for Pts, G, A, Wins, and Shutouts) determined the overall winner. In year three, I am still debating whether or not to determine an overall best forecaster. There is a lot of variation in the amount and type projections that forecasters release. Some just release Pts and Wins and not all the general categories we have evaluated in the past. Also, this season we are going to be evaluating other categories like PIM, Save%, and more! This will make it harder to determine which forecaster has the overall best projections. Currently, I believe that the only fair (and best) thing to do is just to evaluate and determine a winner from each statistical category. I will leave it up to each individual pooler to select the categories that are relevant to their fantasy hockey league.
All forecasters evaluated have a different total number of projections. To fairly compare and rank the projections of forecasters, a common number of projections is needed to be used. The gold standard for comparison over the past three seasons has been 300 projections. This number of projections will cover most fantasy hockey leagues and almost all forecasters produce projections up until this point. The number of forecasters with projections over 300 drops significantly. In the 2017/18 season I also released rankings for 200 and 100 projections. This was done to best suit poolers who participate in the shallower leagues.
For skater projections the total error is summed in groups of 25 before comparison. For Goalie projections the total error is summed in groups of 10 before comparison. Ultimately the group size does not have that much impact on the overall ranking, it only effects the point output range that forecasters can achieve. With small comparison groups, there is larger relative error in projections. The larger relative error leads to forecasters obtaining negative points in some player range comparisons, and negative overall totals. Grouping in 25 and 10, produces a point output that ranges from 100 (max points possible) to usually 0 (it is possible to have negative points). Ignoring the convenience of the point system output being limited to positive numbers, I also feel that groups of 25 skaters and 10 goalies are ranges of players that poolers will pick players from during each round of the draft. Poolers do not always pick the highest ranked player available, and sometimes go deeper to choose a player. Having moderate player ranges to compare has been the best fit for this calculation system.
Many fantasy projectors include a rank beside a player which runs 1 through the number of players in their projection. Sometimes the forecasters will give players with the same totals a different ranking. For example two players with a point total of 45 could have the ranking 100, and 101. The way that this is determined varies. Sometimes it is due to alphabetical order, and in other cases it is due to the number of goals the player scored.
I believe that the projections of players that have the same forecasted point totals should have the same rank. Therefore, before determining the error in a set of rank projections, each players rank is adjusted. So as highlighted in the in the previous example, the two players with the same Pts total would have their rank adjusted to be the same for that stat (Each player would have a rank of 100).
Remember, it is up to the viewer to determine how the rankings are interpreted for their pool scoring system.