Can you make plots of how well calibrated the market is vs how long until it resolves? I.e. if the market becomes well calibrated only in the last 3 days prior to the match and previously it was only amateur creating an uncalibrated market?
This is a great analysis Mike! My intuition about whether to use Brier vs log scoring for evaluating prediction market forecasts is that it's not that important since they're both proper scoring rules which mean that, in both cases, the error minimizing strategy is to always attempt to forecast the ground truth probability. However if someone is considering gambling with their life savings (or anything where the cost for overconfidence is much higher than the cost for underconfidence), then it makes sense to use log scoring.
Can you make plots of how well calibrated the market is vs how long until it resolves? I.e. if the market becomes well calibrated only in the last 3 days prior to the match and previously it was only amateur creating an uncalibrated market?
This is a great analysis Mike! My intuition about whether to use Brier vs log scoring for evaluating prediction market forecasts is that it's not that important since they're both proper scoring rules which mean that, in both cases, the error minimizing strategy is to always attempt to forecast the ground truth probability. However if someone is considering gambling with their life savings (or anything where the cost for overconfidence is much higher than the cost for underconfidence), then it makes sense to use log scoring.