Seems like ranking scheme could still be improved. Right now it uses a 'bonus' based on coins graded, plus base percentage. But the real question we should ask is: How good has someone PROVEN they can be, to a certain confidence level. There is an equation that already answers this question, the confidence bound of a proportion. The normal form is pretty easy:
https://towardsdatascience.com/wp-content/uploads/2020/08/1pz3MJ46nliH4sBgMrgwSHw.png
We only care about the lower form (with the minus). So all we need is proportion correct, number of guesses, and the z table, which is a standard part of a ton of libraries in python and javascript.
The current leaderboard looks like this:
Crepidodera 75.1%
BigNubNumismatics 69.5%
YN_Evan2010 66.9%
KSCOIN 70.5%
9002913 67.1%
TW-Numis 67.5%
90sch 65.7%
Lafaa 65.1%
DavidV_Numismatics 66.2%
2eth 66.1%
With the confidence bound equation, using LCB, it would look like this:
BigNubNumismatics 68.5%
Crepidodera 68.2%
YN_Evan2010 65.8%
KSCOIN 64.3%
9002913 63.9%
90sch 62.9%
Lafaa 62.4%
TW-Numis 61.7%
DavidV_Numismatics 61.5%
2eth 61.5%
This result intuitively feels 'more correct'. It prevents frequent resets because it automatically penalizes small sample sizes. The other nice feature is we can play with confidence interval to get the desired penalty for small sample sizes. It works very similarly to the current solution but scales better, especially at higher sample sizes. the other nice thing is that it will never go OVER the observed percentage. It's not possible to get over 100% either.
Curious to hear what everyone thinks about this approach.
Seems like ranking scheme could still be improved. Right now it uses a 'bonus' based on coins graded, plus base percentage. But the real question we should ask is: How good has someone PROVEN they can be, to a certain confidence level. There is an equation that already answers this question, the confidence bound of a proportion. The normal form is pretty easy:
https://towardsdatascience.com/wp-content/uploads/2020/08/1pz3MJ46nliH4sBgMrgwSHw.png
We only care about the lower form (with the minus). So all we need is proportion correct, number of guesses, and the z table, which is a standard part of a ton of libraries in python and javascript.
The current leaderboard looks like this:
Crepidodera 75.1%
BigNubNumismatics 69.5%
YN_Evan2010 66.9%
KSCOIN 70.5%
9002913 67.1%
TW-Numis 67.5%
90sch 65.7%
Lafaa 65.1%
DavidV_Numismatics 66.2%
2eth 66.1%
With the confidence bound equation, using LCB, it would look like this:
BigNubNumismatics 68.5%
Crepidodera 68.2%
YN_Evan2010 65.8%
KSCOIN 64.3%
9002913 63.9%
90sch 62.9%
Lafaa 62.4%
TW-Numis 61.7%
DavidV_Numismatics 61.5%
2eth 61.5%
This result intuitively feels 'more correct'. It prevents frequent resets because it automatically penalizes small sample sizes. The other nice feature is we can play with confidence interval to get the desired penalty for small sample sizes. It works very similarly to the current solution but scales better, especially at higher sample sizes. the other nice thing is that it will never go OVER the observed percentage. It's not possible to get over 100% either.
Curious to hear what everyone thinks about this approach.