Performance metrics for PredictIt, FiveThirtyEight, and Manifold Markets
Thanks Mike, I would be interested to hear why you don't think ROC curves are an appropriate metric for assessing predictive competency. I watched the Philip Tetlock lecture and came out thinking that they would be.
Another analysis of midterm forecasts from First Sigma, which includes more sites (including Polymarket and Metaculus), but fewer forecast questions:
Very interesting post to check out!
Thanks for writing this. PredictIt skewed republican across the board, at least relative to 538.
For the election, I just went through the PredictIt markets and bet on those showing high discrepancies relative to 538 (e.g., 538 gave around 15% greater odds to democrats winning the NV senate relative to PredictIt). Made a tidy profit.
I think PredictIt has issues with very unlikely outcomes, inflating things in the 1-3% range to 5-10%. This is roughly evident in your spreadsheet. There is some strategy to this. I occasionally buys shares even if I believe they are correctly priced, reasoning: "This 4 c share is at its floor. It can't go lower than this, and even if it doesn't pan out, I can probably sell it at the same price in a few weeks.
In either case, even if you only look at markets where Predictit had them priced 11c to 89c, 538 would've certainly done better given that it overall leaned relatively Democrat.