What is a “cult,” exactly? What does it mean for a group of people to be a cult?

Usually when we call something a cult, we're talking about a group of people who were deliberately indoctrinated into a shared group identity. Cults are typically based on ideas, and the group identity formation happens when people have their pre-cult identities broken down, and a new identity formed based on the idea. This idea-based identity supplants other, more shallow forms of identity, so often a cult will include members who were originally from very different walks of life, but who bonded together over their shared commitment to the idea.

Also, cults are always bad. You never hear about a good cult. The members have always been duped into believing something wrong. The idea that they’re bonding over is always something crazy and obviously false, like that they need to commit suicide so that they can be taken to heaven by aliens. It’s automatically understood that referring to a group as a cult is an insult, since the term implies all these negative aspects.

Is there a word that means something similar to “cult,” except without the implied negative aspects? I can’t think of one. What would you call a group that practices deliberate group identity formation around an idea, where the idea is not something wrong or crazy? It seems like the same techniques used to build identities around bad ideas could, in principle, be applied to good or neutral ideas. At the very least, it seems like it would be interesting to study the identity-formation process from a morally neutral standpoint, without automatic moral judgement.

I’ve actually been through something like this myself. When I was 23, shortly after graduating from college, I enlisted in the US Coast Guard and went through the service’s two-month long basic training course – colloquially called “boot camp.” Despite all the jokes (some of which are pretty funny), the Coast Guard is in fact a real branch of the US military. Its boot camp training is pretty similar to the other branches, with the same goal: to transform civilian recruits into military personnel, by breaking down the recruits' sense of individual identity and building up a sense of group identity in its place.

And interestingly, the military is pretty open about what they’re doing – they even use the word “indoctrination” unironically and without negative connotation, as simply a noun to describe what’s happening. Normally if you ask a cult leader about this kind of thing, they deny it: “What?! Break down the members' sense of individual identity to replace it with a group identity?! Of course not, that’s ridiculous!” But the military owns it. This is a straightforward goal that they pursue, not an accusation of something negative.

The identity-formation process is also remarkably effective. Consider the starting point: a bus full of strangers, chatting and joking around at first, but getting quieter and more nervous as the bus makes its way to a training center in southern New Jersey. Most are young men, though some are young women, and they’re roughly representative of the US population in terms of race/ethnicity, religion, geographic-point-of-origin, and socioeconomic background – as if you’d taken a cross section of American society. After two months of boot camp, this group will feel an extremely high level of cohesion and camaraderie with each other (really the highest level of camaraderie I’ve ever felt in my life), and with the Coast Guard as a whole. All of them will have a new sense of identity as United States Coast Guardsmen, and for many of them it will be the most important aspect of their identity, at least for a time.

The goal of this essay is to give a first-person account of how this happens. We’ll explore how a person’s sense of individual identity can be deliberately broken down and replaced with a sense of group identity, and approach this not as something good or bad, but simply as an interesting phenomenon worth understanding.

Separateness

The first key element in the process of identity formation is the removal of recruits from the outside world for the duration of boot camp training. Many of the recruits are only 18-19 years old, so for a lot of them it’s the first time away from home and their family.

Part of this is the physical separateness. Coast Guard boot camp takes place on a base in Cape May, New Jersey, and recruits are physically separated from the outside world during this time. The location is actually somewhat interesting, since Cape May is a beach town and popular vacation spot. I went through boot camp during the summer months, and it occurred to me at one point that only a couple miles away families were hanging out on the beach, enjoying a relaxing vacation, but that might as well have been in a different universe.

Perhaps even more important than the physical separateness, there’s also the social and informational separateness. Recruits have practically zero contact with the outside world for the duration of boot camp, other than periodically writing and receiving letters from friends and family. But no phones, no visits, no internet, no social media, no watching the news or reading the newspaper, so it’s a near-total information cutoff from the outside world.

Getting back to the cult analogy, it’s easy to see some similarity here. A key part of cult indoctrination typically involves cutting members off from their previous lives, both in terms of physical location (often some kind of remote compound) and by limiting contact with friends and family. An important difference though, is that for cults the cutoff is indefinite – they try to keep you cut off from the outside world for as long as possible. The military, on the other hand, has about two months to try to transform you.

Disorientation

One of the most striking things about boot camp is how utterly different it feels from normal life. It’s hard to explain, and it’s hard to even remember the exact feeling, but I remember thinking that it feels like a strange, dystopian nightmare – a different reality from the outside world.

Rather than being gradually eased into this new reality, recruits are thrown in headfirst, in a deliberate attempt to disorient them and induce culture shock. After a day of traveling from various parts of the country to Cape May, the recruits arrive by bus at the training center. Upon arrival, a drill instructor1 steps on the bus and it begins.

The recruits are introduced to several strange things about the new environment immediately. One is the volume. In boot camp, you’re usually required to yell everything at the top of your lungs, as loud as you can. This is called “sounding off.” It begins before the recruits have even gotten off the bus and continues for the duration of boot camp, pretty much all the time (except when you’re in a classroom, medical clinic, etc). And typically, your loudest isn’t loud enough to satisfy the drill instructors, who are also constantly yelling at you at the top of their lungs. This is actually pretty physically and mentally strenuous to do all the time, and many recruits end up losing their voices after a couple weeks of it.

Another unusual thing about boot camp is the altered sense of time. Much of what the recruits do will be in a hurry (even things like showering, shaving, and getting dressed), and will often be explicitly timed with a punishment if the time limit is not met. This feeling of constantly being rushed has the effect of locking you into the present. Whatever you’re trying to do, you’re doing it as fast as you can, and it still probably isn’t fast enough. This begins immediately, with the first order the recruits are given: to get off the bus, grab their bag, and line up on a set of markings on the pavement – all to be done as fast as possible, while the drill instructor screams at them.

At this point, the drill instructor introduces the recruits to one of the strangest things about this new environment: the language. In boot camp, you’re required to speak a strange new dialect of English. The floor is called a “deck.” The wall is a “bulkhead.” Bathrooms are “heads.” Left and right are “port” and “starboard.” The cafeteria is the “galley” and your fellow recruits are your “shipmates.” These are all nautical terms, but some other terms seem to come out of nowhere – pens are “ink-sticks” and sneakers are “go-fasters.” There are also all kinds of rules about what you’re supposed to say, and how and when you’re supposed to say it – a kind of new grammar to go along with the new vocabulary. For example, if you ever want to say something to a drill instructor besides “yes,” “no,” or “aye-aye2,” you need to yell: “Chief [your drill instructor’s last name]. Seaman recruit [your last name]. I request to inform you that [whatever you’re trying to say]3.” So the beginning of boot camp is a bit like a language immersion program, where you’re under extreme pressure to learn this new dialect and will get screamed at if you say the wrong thing.

Before the real training begins, the first couple days are spent on administrative in-processing steps. You get issued your uniforms and have your hair buzzed completely off, so pretty soon everyone’s starting to look the same. There’s also gear issue, paperwork, a drug test, etc. But this is all done within this loud, chaotic new environment, where everything is rushed and you learn what the rules are by getting screamed at for accidentally breaking them. Come to think of it, I could probably write an entire essay about how hard it is to pee in a cup with a maniac screaming at you and counting down on a timer!

So even before the real difficulty starts, you’re already having a completely different sensory and psychological experience, rushing to follow commands as quickly as possible, literally speaking a strange new language that must be shouted at the top of your lungs. It hit me the hardest the first night there, lying in the bunk bed (called a “rack”) and looking at the ceiling. I remember thinking “What did I get myself into?” It’s just very different. Importantly, you’re experiencing this differentness with a group of people who you don’t really know yet, but who are in the same boat as you.

Extreme Difficulty

After the first couple days of administrative processing, the real training begins. The most notable thing about the boot camp training is the sheer difficulty. It’s extremely difficult, both physically and mentally – the hardest thing I’ve ever done in my life, by far.

The difficulty comes mostly from the totalitarian nature of the training. Boot camp is totalitarian in the sense that there’s a rule for practically every single thing you do: how you talk (as loudly as you can and with the new language mentioned before), how you walk (marching in formation), how you stand, how you hold your tray in the cafeteria, how you make your bed, how you organize your things (measured down to the inch), how you wear your uniform. Plus a million other little things. You’re not allowed to touch your face4. When you’re done using a sink, you can’t leave any “standing water” left in the sink. Random stuff like that. Two of the most common rule violations are simply being too slow (again, lots of orders are timed) and not being loud enough.

There’s also basically zero free time. Every second of every day is controlled. No breaks. Not even a minute to relax. You’re scrutinized from the moment you wake up in the morning, throughout the day, until the moment you go to bed at night, trying to navigate this totalitarian environment. Even the meals offer no relief from the pressure – best case scenario is you sit and eat in complete silence, and worst-case scenario is this.

All this pressure is backed up with the threat of physical punishment (called “beatings” or “getting smoked”). To be clear, this doesn’t involve the drill instructors actually hitting the recruits. Rather, the drill instructors simply order the recruits to do physically exhausting and painful things. I remember having a philosophical musing during one of these beatings: Hmm so the drill instructor can’t hit me, but he can order me to do physically painful things. Is there actually a difference between the two in utilitarian terms?

Sometimes these punishments take the form of a punitive calisthenic exercise – the stereotypical “DROP AND GIVE ME TWENTY!” These could be pushups, squats, or things like that, and they push you beyond your physical limit to make sure it really is very difficult and painful. The recruits are also issued rifles (with no ammunition, obviously), and these rifles are often used as props for punitive calisthenics. An especially tough one is “rifle squats” – doing a squat while holding the rifle (which is pretty heavy) straight out in front of you. The drill instructor calls these out with “DOWN! UP! DOWN! UP!” and often leaves the recruits in the “DOWN” position for a while to make it as hard as possible.

Besides punitive calisthenics, there’s also the use of stress positions. In my experience, the worst punishment (and also one of the most common ones) was being ordered to hold a full canteen of water over your head for about 20 minutes. This might sound easy, but after 10 minutes or so it becomes very painful. There’s also the mental agony of not knowing how much longer it would go on for – again, the altered sense of time locking you into the present moment.

When someone is caught breaking a rule, they incur the punishment. Sometimes they’ll be punished individually, but usually the whole recruit company will be punished collectively. The punishment will occur either on the spot, or they’ll build up a punishment “debt” to be paid all at once later, often for several hours, with only short water breaks interspersed in a long series of “beatings.” In theory, if the recruits behave perfectly all the time, they’ll never be punished. However, since there are so many rules and so many recruits, someone is caught accidentally breaking a rule constantly. So, for the first 4-5 weeks of boot camp, at least a couple hours per day are spent simply getting punished with calisthenics and stress positions. Of course, there are other things you do throughout the day, various classes and chores, but those parts aren’t the highlight, and they aren’t what I remember. The part of boot camp I remember most distinctly is the time spent holding a canteen over my head.

Besides all the rules, there’s also the “required knowledge” – various facts the recruits must memorize, word-for-word, and be able to recite at any time, under threat of the same punishments. Some of this is pretty important and will be useful for their time in the service. For example, they must memorize all of the ranks and their insignia (including word-for-word descriptions), so that they can properly identify a person’s rank from their uniform and know how to address them. They also memorize the NATO phonetic alphabet (“Alpha, Bravo, Charlie, Delta…”), and their chain of command starting from their drill instructors and going all the way up to the president.

But some of the required knowledge is useless outside of boot camp and exists only to add to the stress and difficulty. For example, there’s something called the eleven general orders. These are simply eleven sentences that recruits need to memorize word for word, and be able to recite on command. The orders are loosely related to watchstanding, but their meaning is never really discussed or explained. They’re just sentences that need to be memorized, and will be forgotten almost immediately after boot camp.

Recruits can be called out and asked about required knowledge at any time. You could be silently eating a meal in the cafeteria and suddenly hear: “RECRUIT SO-AND-SO, WHAT IS YOUR FOURTH GENERAL ORDER?!?” Then you have to stand up, while everyone else continues silently eating, and yell the correct word-for-word answer as loud as you can, or else everyone will get punished later. As with the rules, there’s simply so much required knowledge that for the first month or so people are constantly getting things wrong and the group is constantly getting collectively punished for it.

This is the extreme difficulty of boot camp: the totalitarian set of rules to be followed and arbitrary knowledge to be memorized, backed by the threat of physically painful collective punishment that often takes up several hours per day. This adversity is one of the most important elements in the group identity formation process. Again, you’re in this with a group of people who are also going through it, and that makes for a powerful bonding experience. An unexpected takeaway from my experience in boot camp was that (and I know this sounds bad, please don’t cancel me) I became more sympathetic to fraternity hazing rituals. Before I just thought they were a dumb and pointless form of bullying. Now, I at least kind of understand the point of them, and would guess that they probably are effective at bonding together new members through shared hardship.

The Other

Importantly, the source of this hardship has a name, a face, and… a Smokey Bear hat. This is the drill instructor. They are in control of every aspect of this totalitarian environment. Typically a recruit company will have three drill instructors (a lead instructor and two assistants) and at least one of them will be with the recruits most of the time, screaming orders and dealing out punishments.

A common misconception is that the scary thing about drill instructors is just the yelling or the saying mean and insulting things. Sometimes you hear a self-styled tough guy say “I could never join the military. If some drill instructor started yelling at me, I’d just get angry and punch him in the face!” No, they wouldn’t. That’s because the scary thing about the drill instructors isn’t the yelling – it’s the total control they have over you, and the power they have to make your life hell, which they frequently exercise.

We already talked about the punitive exercises, but there’s actually an even worse punishment than that: getting rolled back a week in training, which drill instructors can do at their own discretion. And this is no empty threat – probably something like 25% of a given recruit class will get rolled back at least a week, often as the result of failing one of the periodic inspections. These inspections are especially stressful because you can be rolled back on the spot over tiny things, like having a piece of clothing folded the wrong way or in the wrong place, or having a loose thread on your uniform. Of course it’s impossible to be perfect all the time, so these inspections often come down to the drill instructor's discretion and how closely they decide to look.

Besides that, you can also be rolled back simply for screwing up too much. As I’ve mentioned before, there are so many rules, so much required knowledge, and so many hurried orders to be followed that it’s impossible for anyone to be perfect all the time. So the real threat from the drill instructors isn’t the yelling or insults like you see in the movies. Rather, it’s the possibility of a scrutiny-spiral, where if you run afoul of the drill instructor in any way, this will cause them to notice you more, which will lead to them catching you screwing up, which will cause even more scrutiny – and this spiral could very easily end up with you getting rolled back in training and having to spend even more time suffering with a canteen held over your head.

As you’ve probably already guessed, the drill instructors play a key role in the process of group identity formation for the recruits. They’re the antagonist, the Other, the Outgroup that the recruits must band together in opposition to. And it works. Facing this extreme difficulty and adversity personified by the drill instructors, the recruits quickly learn to work together and have each others’ backs, helping out and cheering on the struggling members.

Value and Habit Formation

So a lot of boot camp is basically hell. When you’re holding your full canteen over your head, or stuck in the “DOWN” position of a rifle-squat, you feel like you’re in hell.

A side-effect of this is that any time not spent in hell is a relief. For example, towards the beginning of training recruits have to go to the base dentist to have their teeth inspected, and some recruits will be scheduled to have their wisdom teeth removed during boot camp. Amazingly, this is considered to be a lucky, fortunate thing! When you’re in hell, getting to temporarily take a break for a day and go get your wisdom teeth removed is a stroke of good luck.

This is the context in which the Coast Guard attempts to teach values like cleanliness, grooming standards, and fitness. Towards the middle and end of training, when the recruits are getting better at following the rules and not getting punished all the time, a decent amount of time is spent on “breaks” like scrubbing the bathroom floor, cleaning the barracks, ironing uniforms, polishing boots, organizing your things, and silently studying the required knowledge. Compared to the alternative of getting screamed at while holding a canteen over your head, these things seem like a relief.

Likewise, physical exercise isn’t just used as a punishment – it’s also used neutrally and even as a reward. We’ve covered the punishments already: the punitive exercises and stress positions. But often a neutral, non-punitive workout will just be on the schedule as a thing to do, like one of the other classes.

Other workouts are treated as rewards and must be earned. For example, sometimes there will be an off-base run, where recruit companies that have been well-behaved enough are taken on a ~4ish mile run in formation, through the streets of Cape May, while singing cadences. This is a lot of fun, especially during the summer when Cape May is full of tourists cheering you on and shouting encouragement. Besides that, one of the best rewards we ever got was towards the end of training, after a day of particularly good behavior, our drill instructor just took us to the base gym for an hour and let us do whatever workouts we felt like. Anyway, with physical exercise being used as a punishment, a reward, and as something neutral that’s just on the schedule to do, you spend quite a lot of the time at bootcamp working out.

Culture Formation

At this point, you might have noticed something peculiar: I haven’t mentioned any job-skills training – just pain, suffering, stress, working out, cleaning, etc. That’s because the “training” at boot camp really isn’t about learning job skills. Sure, you take classes on various maritime topics, tying knots, firefighting, and learning how to shoot a gun, but if your eventual post-boot-camp job requires any of those things you’ll need to redo the training at your new unit anyway. The real purpose of boot camp is cultural indoctrination, and much of the time not spent getting screamed at or doing punitive exercises is devoted to this goal.

For example, a lot of time in boot camp is spent on something called “close order drill” – basically marching and doing ceremonial rifle movements in formation. Hours and hours of grueling practice are spent learning these movements, practicing them as a group, and getting inspected on them by drill instructors. It’s actually very difficult to perfect these movements as a group. A strange side effect of going through boot camp myself is that (again, please don’t cancel me) I now have a kind of respect for the North Korean army’s ability to march in formation.

After boot camp, the recruits will almost never do close order drill again, other than maybe for the occasional parade or retirement ceremony. So, what’s the point of it?

Well, one answer is that it has to do with teaching attention to detail and teamwork. It isn’t about the marching or the “RIGHT SHOULDERRRRRR ARMS!” being important for its own sake. No, it’s about learning precision and group coordination. This is basically what drill instructors tell you towards the very end of boot camp when they start to drop the mask a little bit and explain some of the rationale behind things.

Another possible answer, though, has to do with culture creation through shared arbitrary practices. One of the most straightforward ways to create a distinct culture is to just have some unusual, arbitrary practices that distinguish you from other cultures. Orthodox Jews, for example, follow a very strict set of rules on the Sabbath. For example, many will not press elevator buttons on Saturdays because of a prohibition against “creating sparks or fires,” which is interpreted to include operating electrical switches (like elevator buttons). There’s actually a special type of elevator designed to get around this problem.

Taken literally as a matter of theology or moral philosophy, this just seems silly and is an easy thing for atheists to criticize and score points on. But as a matter of culture creation, these arbitrary rules and practices work well. Of course they seem silly to me because I’m not an Orthodox Jew. That’s the point! They allow for commonality among members of the culture, and differentness from the external culture.

I think close order drill also serves this purpose, in addition to teaching attention to detail and teamwork. It gives the recruits something in common – a shared experience. Importantly, the commonality persists even after boot camp is over and they will probably never have to do a “RIGHT SHOULDERRRRRR ARMS!” again. It’s like how I, as a lapsed-Catholic, can still find commonality and bond with other lapsed-Catholics over the strictness of Catholic school. In the military, thinking back on all the marching and close order drill has the same feeling: “Pssh, what a waste of time that was.” But the joking cynicism can itself be used as a bonding mechanism between two people who are both feeling it.

Besides the total arbitrariness of close order drill, some parts of boot camp are more recognizable as culture in a straightforward sense. One of the most obvious examples is the uniforms. Although we didn’t earn the right to be called Coast Guardsmen until we graduated, we began wearing the uniforms almost immediately. This, combined with the shaved heads, contributes to the loss of individuality and the feeling of collectiveness. This loss of individuality might sound pretty dystopian, but it can actually feel pretty great to be a part of. Personally, I’ve always enjoyed wearing uniforms, even at minimum-wage jobs I had as a teenager. It was cool to feel like part of a team. Wearing the uniforms in boot camp felt the same way. It was a source of strength, like wow, I’m really part of this thing.

Of course, the uniforms also contribute to the stress of boot camp, since they come with a ton of rules, and frequent inspections. I knew one guy who got rolled back a week in training because he was caught with a wrinkly shirt (called a “blouse”) that he apparently hadn’t ironed enough. While a lot of the rules at boot camp are completely arbitrary and exist only to create a high-stress environment, the recruits will continue to follow the rules around uniforms and grooming standards (shaving, etc) throughout their time in the military. The extreme pressure of boot camp basically serves to deeply ingrain these rules so that the recruits get used to following them and it becomes a habit. For example, one of the rules is that when you’re outdoors in uniform you must be wearing your hat (called a “cover”). Being outdoors in uniform without your hat is one of the things that can bring down the wrath of the drill instructor on the entire recruit company. To this day, even though I’m no longer in the Coast Guard, the thought of being outdoors in my uniform without my hat gives me a physical cringe sensation, like if I had my fly down in public or something.

Some other things were learned were also recognizable as regular culture: Coast Guard history, famous heroic Coast Guardsmen, how to sing the Coast Guard hymn, and a variety of practices called “customs and courtesies” that include saluting officers when outdoors, how to address superiors, how and when to stand at attention, etc. Like the uniform regulations, the customs and courtesies will continue to be followed even after boot camp is over.

Getting Used to Another Plane of Existence

The sum total of all of this is that boot camp feels like an alternate universe. Your days are spent navigating a strange, totalitarian environment under extreme pressure from the Smokey-Bear-hat-wearing Outgroup, who are eager to inflict punishment on you. You navigate this environment with your fellow recruits, who you practice your new culture with and endure the suffering with.

And after like 5-6 weeks of extreme mental and physical stress, you start to get used to it. Things also get easier, as everyone gets more familiar with all the rules and the punitive beatings become less frequent (maybe ~40 minutes per day rather than a couple hours). You get used to living in hell, and hell also gets a bit easier, and then it doesn’t feel like hell anymore. It just feels like an alternate universe that is now your existence. And your previous life feels very distant.

I’ve actually never heard anyone else describe this feeling before after going through boot camp, and I think that’s because it’s easy to feel without consciously noticing, and then easy to forget once it’s all over. But there was a specific moment that caused me to consciously put my finger on this feeling and promise myself that I would remember it.

About 6 weeks into the training there was a blood drive on the base, and recruits who wanted to were allowed to go donate blood. When I was there, I saw that one of the nurses collecting the blood had a coffee cup from Wawa, a regional chain of convenience stores in the Pennsylvania / New Jersey area. And when I saw that, I thought “Oh, yeah…”

In that moment, I remembered that there was another universe out there that had this thing called Wawa. And I remembered that in a previous life, I used to like to go to Wawa when I wanted some time alone with my thoughts. I’d especially like to go late at night, often after a shift at work, and get a snack or a coffee, and just sit in the parking lot and relax. I remembered that in this parallel universe, you had the freedom to just decide to go to the store, get a snack, and sit in the parking lot if you wanted to.

Because of the altered experience of time in boot camp, where everything is rushed and you’re locked into the present while feeling constantly stressed, you don’t really get a chance to reflect on what’s happening. So when I saw that nurse’s Wawa cup, that was really the first time I reflected on things and thought “wait a minute – something pretty wild is going on here.” And I told myself that I needed to purposely try to remember this feeling, because it would probably be easy to forget.

And that’s basically what happens. After about two months of boot camp, you eventually graduate, and are back in the regular world just as quickly as you left it. After graduation, you get about a week of leave before you need to actually check into your new unit, so the very same day that I graduated, I was back in my hometown in Pennsylvania that night, having a drink at the bar with a couple of my high school friends. And that was a weird feeling, because I could already feel myself starting to forget that alternate plane of existence. It was like time had skipped over the past two months (a bit like how time skipped over Covid), and I was just left with a feeling like: “Woah… that was weird.”

Epilogue: Yeah, It Works

This essay was about deliberate group identity formation – whether or not it’s possible to take a diverse group of people from all kinds of different backgrounds with very little in common, break down their individual identities, and create a group identity in their place. Well, it is. The boot camp training process works very well. Towards the end of training, everyone is extremely hyped up about the Coast Guard, and the feeling of group camaraderie is really amazing – the highest level of camaraderie I’ve ever felt. The day I graduated and could officially call myself a United States Coast Guardsman was the proudest day of my life.

Of course, the sense of identity fades over time, and its staying-power varies from person to person. Some people get cynical pretty quickly and forget the feeling entirely. If you ask them, they might deny ever having felt it.

I think for most people though, the feeling fades but doesn’t disappear. That’s how I feel at least. I still like to wear my old unit sweatshirt because it sometimes leads to interesting conversations. For example, one night I was picking up some groceries at the supermarket and an older guy asked me about the sweatshirt. It turned out he had been stationed at the same unit back in the 1980s, and we spent about 20 minutes talking about it. Maybe that’s what it feels like to be a scientologist and run into another scientologist out in public. Anyway, it was pretty cool.

In the post, I tried to be a bit fancy and high-tech, so for the Brier score tables rather than just pasting in a PNG image, I created the tables with an online data visualization tool called Datawrapper and embedded them in the post.

This worked ok for the actual post on Substack, so if you actually click on the link and look at the post, the numbers in those tables are all correct.

But it came to my attention today (thank you to the vigilant reader who noticed this) that in the email version of the post that got sent out to everyone, the numbers in these tables are completely wrong. For example, here is what the first Brier score table looks like in the actual post on Substack (correct version):

But then here is that same table, in the email version (incorrect numbers):

I think what happened was that when I was drafting the post (before the election), I made these tables on Datawrapper with some hypothetical results just to get a draft and see what the table would look like. Then, once the results were in, I plugged in the real numbers and re-saved/re-published on Datawrapper. At that point things looked fine on my end, with the embedding in the Substack post showing the updated, correct numbers, but the version of the post in the email version that got sent out apparently showed the old version of the Datawrapper table rather than the current version.

Here’s some actual proof of this so you guys don’t think I’m just making up an excuse:

The other Brier score table also had incorrect numbers in the email version for the same reason. The contest winners table had the correct numbers in the email version, but the third-place winner was incorrectly referred to as “Anonymous Person” because the first saved draft of that Datawrapper table was before I had gotten official permission from that guy (Frederik Holzer) to publish his name.

Anyway I’m very sorry about that! In the future I’m going to avoid using these Datawrapper embeddings for tables, and just go back to the old-fashion method of typing the results into Excel and saving them as an image.

Thanks for reading!

(Sorry it took so long to publish this, but I was waiting for the CA-13 House race, which just got called yesterday)

Introduction

The goal of this post is to report evaluation metrics for some forecasts of the 2024 US election, similar to the forecast evaluation I did for the 2022 midterms. Importantly, this is an apples-to-apples comparison, judging forecasts on the same set of 50 questions (including presidential winner and individual state results, House/Senate control, specific House/Senate races, and specific governor races) with forecast data recorded at the same time. This was the list of questions.

The forecasts are from:

• Manifold Markets — a play-money prediction market site. Lately they’ve also had some real-money markets, but I only collected data from the play-money ones, since I’m interested in studying whether the play-money reputation-based incentive leads to a similar level of accuracy as real-money prediction markets.

• Polymarket — a real-money prediction market site.

• Crowdsourced forecast — generated largely by you guys, as part of the forecasting contest (winners announced later in this post btw).

• My own forecast — which turned out to be completely wrong and terrible.

• Nate Silver’s forecast — NOTE: Nate Silver only gave a forecast of the presidential race, not the House, Senate, and governor races, so his forecast will be judged in a separate results section, considering only questions relating to the presidential race. ANOTHER NOTE: I collected the Nate Silver forecasts at the same time as the Polymarket and Manifold forecasts (around 8am). This meant that each day that I collected, I was actually recording Nate Silver’s forecast from the previous day, since he updates them later in the day. So if you’re looking at the code and see his forecasts labeled one day earlier, that’s why.

For Manifold, Polymarket, and Nate Silver, I collected data at 3 timepoints: 2 weeks before the election, 1 week before, and 1 day before. The crowdsourced forecast and my own forecast were only 1 day before.

The results reported here are Brier scores (remember — lower is better!). If you don’t know what a Brier score is, there’s an explanation in the Methodology section at the end. I’ve also included calibration plots, although I think the sample size is too small for them to really be meaningful.

The forecast data, as well as the code I used in the analysis, are available on my Github. Let me know if you have any questions!

Results 1: Full Set of 50 Questions

The results are pretty messy here and it’s tough to really tell a story about them — but that’s science! The goal of this project was just to collect and report data, not to try to tell any particular story.

One thing that’s somewhat notable though is that for Polymarket, the forecasts were notably worse the day before the election, compared to 1-2 weeks before. For a reminder of what the markets looked like, here’s their market on whether Trump would win or not:

1-2 weeks out, Trump was at >60% chance of winning (and generally the Polymarket forecasts were more optimistic for the Republicans, across many races). But then in the days just before the election, things swung back closer to 50/50. By contrast, the play-money markets on Manifold were more stable throughout this period.

Are Manifold play-money prices more stable than Polymarket prices in general? It’s hard to say based on such a small sample size, so I’d caution against drawing any strong conclusions like that. But I’m going to continue doing analyses like this in the future, so I’ll be interesting to see if that pattern keeps coming up.

Another result — you guys (the crowdsourced average forecast) did pretty bad, but I did even worse! I guess I won’t quit my day job, lol.

Results 2: Subset of Nate Silver Questions

The previous results were for all 50 questions (president, congress, governors). But Nate Silver only gave a forecast for the presidential race, so the results in this section are for a subset of 20 questions for which there was an overlap between Nate Silver’s forecast and the others.

Again, tough to draw any solid conclusions here, especially because of the tiny sample size. But Nate Silver’s forecast wasn’t terrible, like a lot of internet-haters have suggested. He did about as well as Manifold, and only slightly worse than Polymarket (which everyone has been congratulating for being especially accurate).

Anyway, sorry that the results are kinda boring this election cycle. But I’m committed to publishing boring results on this blog, because when people only publish exciting results that creates a bias and skews the distribution of reported results. This is called the “filedrawer effect,” and is one of the causes behind the ongoing replication crisis in academia. Lately I’ve been thinking that one of the most important things in science is just the non-glamorous collection and reporting of data, with the goal of contributing to a large, accurate distribution that you can use to say something about the thing you’re studying. So one of the goals of this blog is basically to do that for prediction market accuracy.

Contest Winners!

Here are the results of the election forecast contest. Details on the scoring methodology here (basically time-weighted Brier, with a slight reward for submitting early). Congrats to the winners, especially Jonny, who won both First Prize and the early submission prize. Also, for those interested, the third-place winner Frederik Holzer published his forecast on this website.

Thank you to everyone who participated!

Calibration Plots

I’m including calibration plots here in case people want to see them. If you don’t know what a calibration plot is, please see the Methodology section at the very end for an explanation. Anyway, I don’t think they’re very informative because the sample size is just too small — only 50 questions, and only 20 for the Nate Silver subset. So the plots ended up looking pretty bad from a visual standpoint, even though the forecast Brier scores were mostly decent.

Another thing to note is that many of the elections that were forecasts with around 0.5 probability (like the swing states) ended up swinging in the same direction this cycle (toward the Republicans), so that’s why many of the plots have a dip in the middle.

For comparison I’ve also included an example of a calibration plot for election market odds from ElectionBettingOdds.com in the conclusion section, with 807 events going all the way back to 2016. With that, you can get an idea of how well election markets are calibrated when the sample size is large enough and when there are many election cycles with some swinging towards the Republicans and some towards the Democrats.

Conclusion

Like I’ve been saying, I would caution against drawing any strong conclusions from the results in this post. My goal with this project (and all of these prediction market analysis projects) was to collect and report data, not to attempt to tell any particular story. So I would be cautious about extrapolating.

And this is a bit tough to say, because I’ve been a huge fan of prediction markets ever since the old PredictIt days, so it’s tempting to go along with the media in congratulating them and declaring victory for Polymarket. But I’m still not sure if what we saw from Polymarket was real insight, or just a bias toward the Republicans, combined with the election happening to tilt to the Republicans (see also Scott Alexander’s recent post on this). So we’ll have to wait and see if they’re able to keep up this level of accuracy in the future, including with election cycles that happen to tilt in favor of the Democrats. It could also be that they are biased in favor of the Republicans, but each cycle more and more people notice this and exploit it by taking the other side of these bets, and the bias gets a little smaller each time.

Anyway, with all that being said, I think the long-run performance of election markets overall has been pretty good. I mentioned before that the calibration plots for this project look pretty bad because of the small sample size, but as a point of comparison here’s a calibration plot of the ElectionBettingOdds.com track record, with 807 events going all the way back to 2016.

Not bad! Events predicted to happen with 0.25 probability happened about 25% of the time, 0.75 probability happen about 75% of the time, and so on. I’ve also written before about sports betting markets, which have similarly impressive calibration, as a proof of concept for the ability of betting markets to make accurate probabilistic forecasts. So yeah, I’m pretty bullish on prediction markets, even though I’m hesitant to draw too strong of a conclusion from this election cycle alone.

Anyway thanks for reading, and I’ll see you guys in 2026 for the midterms.

Further Reading

• Superforecasting: The Art and Science of Prediction by Phillip Tetlock and Dan Gardner. AMAZING BOOK!! Really the best introduction to forecasting. 

• The Signal and the Noise by Nate Silver

• Gwern on prediction markets

• “A Bet Is a Tax on Bullshit” – Marginal Revolution post by Alex Tabarrok

• Idea Futures by Robin Hanson

• Introduction to Prediction Markets, by Jorge I. Velez 

• Scott Alexander’s Prediction Market FAQ

Methodology

(copied from my 2022 forecast evaluation)

I kinda feel weird trying to explain something when someone else has already done a better job of explaining it than me – I’d rather just refer people to the best possible explanation of the thing. In this case, this lecture from Phillip Tetlock does a better job of explaining the problem formulation, calibration plots, and Brier scores than I can – he also has a great book on the topic.

With that being said, I’ll still provide some explanations here in case you don’t feel like watching a lecture or reading an entire book.

Data Format: Probabilities and Outcomes

In forecasting, the prediction of an event happening or not happening is usually given in the form of a probability. For prediction markets, these are typically prices (like cents on the dollar) that can also be viewed as probabilities ($0.20 corresponds to a probability of 0.2). The outcomes can be labeled with 1 and 0 depending on whether the event happens or not.

So, whether we’re talking about sports, elections, current events, or something else, most forecast or prediction market datasets with binary yes/no outcomes can be represented in this format:

The data spreadsheet includes other information, like the actual question being forecast, but in order to calculate the evaluation metrics all you really need is the forecast probability and 0/1 outcome.

Brier Score

The most commonly used metric for scoring probabilistic forecasts is the Brier score. Here is the formula:

Here, N is the total number of predictions, f_i is the prediction probability, and o_i is the true outcome of the event (either 1 or 0). (Sorry about the awkward notation, I haven’t really been able to find a good way to write math notation like subscripts and stuff on Substack.)

Basically, this is just a measure of the forecast’s average squared deviation from the true outcomes of the events being predicted. The lower the Brier score, the better the forecast, and a perfect score is 0.

Calibration Plots

A common objection people first have to probabilistic forecasts is that it seems impossible to say if they’re correct or not. For example, let's say a forecaster estimates a 40% chance that a candidate will win an election, and they end up winning. Does that mean the forecaster got it wrong? Well, no. 40% chance isn't 0%. But we can't exactly say they got it right either.

But let's say the forecaster has made hundreds of probability estimates for different events. Then, we can look at all of the events they estimated 40% for. If those events ended up happening about 40% of the time, that means the forecaster is well-calibrated.

A calibration plot is a visual representation of this. The horizontal axis shows the binned forecast probabilities for different events, and the vertical axis shows the fraction of the time that those events actually ended up happening. The diagonal line up the middle is what a perfect forecast calibration would look like (events predicted with 0.3 probability happen 30% of the time, 0.4 probability happen 40% of the time, and so on…).

The binning of the probabilities just means that, for example, events forecast with probabilities anywhere between 0.2 and 0.3 would all be put in the 0.25 bin, so that we have enough samples to calculate the outcome fraction within that bin.

Here are two examples of calibration plots (from my 2022 midterms evaluation):

I hope this explanation is clear and makes sense, but just in case it doesn’t, I’m going to again refer you to this lecture by Philip Tetlock, who I think does a better job of explaining these concepts than me.

In the meantime, I used the forecasts from the contest to calculate a time-weighted average, basically averaging all the contest entries for each question, and giving more weight to the more recent ones, and less weight to the older ones. These predictions are listed in the crowdsourced_forecast column of the table below (shoutout to Datawrapper for the cool visualization).

I’m also publishing my own predictions here, in the mike_forecast column. I don’t have any fancy model or methodology here, they’re just guesses based on vibes. For what it’s worth, I live in Pennsylvania, and I’m also a libertarian and not a huge fan of either major party, so maybe that means I don’t have as much of a bias.

Anyway, once the results come out, I’ll compare these forecasts to the implied probabilities from Manifold (play-money) and Polymarket (real-money) prediction markets, as well as Nate Silver’s predictions for a subset of questions, and report how everyone did in terms of accuracy. Stay tuned!

Deadline: Monday November 4, before 12 noon US Eastern Time

Prizes:

• 1st Place: $500

• 2nd Place: $250

• 3rd Place: $100

Link to the original post with details on the scoring methodology (basically time-weighted Brier) and rationale for doing this contest (collecting data to compare with Manifold, Polymarket, and Nate Silver forecasts).

Here’s the link to sign up and enter your predictions!

Deadline: Monday November 4, before 12 noon US Eastern Time

My first ever post on this blog was an analysis of forecasts for the 2022 midterm elections, comparing the accuracy of PredictIt, Manifold, and Nate Silver. I’m planning a similar comparison of 2024 election forecasts, except with Polymarket rather than PredictIt (RIP).

This time there will also be a forecasting contest that regular people can enter, and an analysis of the aggregated regular-people forecasts compared to the market predictions. The idea here is basically to see how well a non-market “wisdom of crowds” approach does compared to real-money and play-money prediction markets, in an apples-to-apples comparison on the same set of questions.

Also, there will be prizes! The prizes will be (at a minimum, I might increase them if I need to get more participants):

• 1st Place: $500

• 2nd Place: $250

• 3rd Place: $100

• Best Early Forecast: $100 (deadline: Tuesday October 22, 12 noon US Eastern Time — two weeks out from election)

And of course, the bragging rights — I’ll probably publish the names of the top 5-10 or so most accurate forecasters overall, and the top 3-5 or so that submit before the early deadline.

Scoring Methodology

Forecasts will be evaluated based on time-weighted Brier scores. Here’s the raw Brier score formula:

In this formula, N is the total number of predictions, p_i is the prediction probability, and o_i is the true outcome of the event (either 1 if it happened or 0 if it didn’t). (Sorry about the awkward notation, I haven’t really been able to find a good way to write math notation like subscripts and stuff on Substack.) 

Basically, this is just a measure of the forecast’s average squared deviation from the true outcomes of the events being predicted. The lower the Brier score, the better the forecast, and a perfect score is 0.

But I also want there to be a slight reward for earlier forecasts. So the formula for the final score will be:

where D is the number of days before the election the forecast is made. So, for example, a forecast with a raw Brier score of 0.082 made 20 days before the election would slightly beat a forecast with a raw Brier score of 0.08 made the day before the election. So just a slight reward for being early, but remember there’s also a separate prize for early predictions made made more than two weeks out.

If you have any questions you can reach me by email at mikest@udel.edu, or by leaving a comment on this post. Thanks and good luck!

PS - you don’t need to be a subscriber of this blog to win the contest, but if you would like to subscribe (it’s free and has lots of articles about forecasting and prediction markets) then just click this button:

Subscribe now

Your country has two industries: puppy-sitting and angry-customer-handling. The puppy-sitters spend the day playing with cute puppies. The angry-customer-handlers are sent to other countries on year-long assignments, posted in stores, and their sole job is to get screamed at by angry customers while maintaining perfect composure like a British royal guard, until the customers tire themselves out. Each citizen is randomly assigned to one of these jobs based on a lottery.

As it stands, the angry-customer-handlers get paid more than the puppy-sitters, $60k vs $40k. One day the puppy-sitters notice this and complain. “This is unfair!” they say. “Why should they get paid so much more than us? We demand equal salaries!”

You agree and decide to issue a new decree from the office of central planning: from now on, puppy-sitters and angry-customer-handlers will be paid an equal salary of $50k.

Does this make things more fair? I don’t think think so. It makes things more equal in terms of monetary compensation, but this actually seems quite unfair when we consider non-monetary factors (in this case, one job is obviously much more fun than the other). Equalizing salaries for puppy-sitters and angry-customer-handlers has almost certainly made things more unequal in terms of overall happiness and job satisfaction.

This thought experiment is exaggerated to make a point, but here’s a real example: a court in the UK recently ruled that Next (a retail company) must have equal pay for store staff and warehouse employees. The ruling is based on a claim of gender discrimination, since the store staff (who were paid less) are predominantly women, while warehouse employees (who were paid more) are about half men and half women.

Some more info from the article:

More than 3,500 current and former workers at Next have won the final stage of a six-year legal battle for equal pay.

An employment tribunal said store staff, who are predominantly women, should not have been paid at lower rates than employees in warehouses, where just over half the staff are male.

Lawyers for the shop staff described the judgement as "hugely significant" and the amount of back-pay owed could amount to more than £30m.

Importantly, the claim in this case was not that men and women were paid unequally for doing the same jobs. The claim was that two different jobs paid differently, and this was a problem because they had different gender-compositions. So the court ruled that the two jobs must have the same salaries, ostensibly to make things more fair and equal.

Now, we could oppose this on the usual free-market grounds (infringement on personal freedom, inefficiency due to misallocation of labor, etc, etc). But those points aren’t even really necessary. This ruling still seems pretty bad, even if we’re only concerned with fairness and equality, once non-monetary factors are accounted for.

Alex Tabarrok at Marginal Revolution writes:

In fact, the court case reveals that Next was struggling to fill the warehouse positions and offered any retail employee—including the plaintiffs—the opportunity to switch to warehouse work. On cross-examination, one of the plaintiffs admitted that, given the unpleasant conditions in the warehouse—described by the court as “the drone of machinery,…vibration, alarm sirens and the screeching of machinery, wheels and rollers, continuously present in all areas”—the warehouse job “did not seem particularly attractive” compared to the greater autonomy and more appealing environment of the retail job. The plaintiff added that she would only have considered the warehouse job if it paid “a lot more money.”

This is the key point. Even the plaintiffs — the store staff demanding to be paid the same as the warehouse workers — acknowledge that working in the warehouse is generally less enjoyable than working in the store, and that they would not want to switch jobs to work in the warehouse unless it paid a lot more money.

I can confirm this myself. Unlike most British employment tribunal judges, I’ve actually worked in both a retail setting (supermarket) and a warehouse before.

In my opinion, the warehouse work is not only more difficult and generally unpleasant, but also more physically exhausting, and dangerous. So in this recent case in the UK, like the puppies and angry-customer-handlers thought experiment, requiring equal salaries for warehouse workers and store staff probably makes things more unequal in terms of overall job satisfaction and happiness of the workers — one of the jobs is still less enjoyable than the other, but no longer has any extra monetary benefit to make up for it.

Remarkably, the original BBC article seems to make this same case, but unwittingly (I think, based on the tone of the article):

There have already been cases in the public sector over lower pay for workers, including teaching assistants and dinner ladies paid less than men employed in refuse collection and similar roles.

Yes, garbage collectors get paid more than teaching assistants. Again, forgetting about labor market dynamics, supply/demand, etc, this seems right just from the standpoint of fairness and overall equality of wellbeing — being a garbage collector is a much more unpleasant job than being a teaching assistant!1 If you mandate that garbage collectors and teaching assistants get paid the same, that will actually make things more unequal in terms of their overall wellbeing and job satisfaction!

To give yet another example, I currently work as a research scientist in academia and sometimes hear the idea that academic research scientists are paid unfairly because we make much less than people doing similar work in the private sector. So sometimes I have this conversation with myself:

Me: Ah man, I wish I was paid more, this is unfair. There are people working in finance doing similar work to me, making twice the salary.

Also me: Um ok. Then why don’t you go work in finance?

Me: …what?

Also me: You could literally quit your job right now and get a different job doing very similar work, except modeling financial data rather than biological data, and it would probably pay a lot more.

Me: Well, I don’t want to. Finance is boring and academic research is more fun.

Also me: So it sounds like you’ve navigated a tradeoff between monetary (salary) and non-monetary benefits (fun, interesting work). And since you don’t want to switch when given the chance, it seems like in terms of overall fulfillment you’re getting more at your current job than you would at a finance job. In fact, if the government mandated that you had to be paid the same as the finance people, that would actually be kinda unfair because you’d still get all the non-monetary benefits you’re enjoying and they wouldn’t.

I want to emphasize that in this post I haven’t appealed to the usual arguments against central planning: personal freedom (people should be free to engage in voluntary contracts with each other without government coercion) and economic efficiency (society is generally less prosperous and people are generally worse off when the government interferes with market dynamics).

I largely agree with those points, but I’m trying to respond to the argument that central planning proponents usually make: “we need government involvement to ensure fairness and equality, and this is more important than personal freedom and economic efficiency.”

But even accepting this as the goal, we face the same information problem2 that central planners usually face: while ensuring equal happiness and wellbeing of people is arguably a justified goal for the government to pursue, this involves a lot of information that is inherently decentralized, subjective, and unknown to the central planners — what is it actually like to work in a warehouse? Is being a garbage collector more or less enjoyable than being a teaching assistant? How much more fun is playing with puppies all day compared to getting yelled at by angry customers?

Without this subjective information, if central planners try to equalize things based on the limited information they have access to — like salaries — they could very well end up making things more unequal in terms of overall job satisfaction.

Is there a better way to do things? Let’s go back to the original example — you’re the head of central planning, and you’re only goal is to maximize equality and fairness among citizens, subject to the constraint that 50% of citizens must be puppy-sitters and 50% must be angry-customer-handlers.

You’re beginning to suspect that the puppy-sitting job might be more fun than angry-customer-handling, so maybe the angry-customer-handlers should be paid more to make up for it. But how much more? 10%? 50%? Twice as much?

First, you eliminate the job lottery system. Citizens will be allowed to choose which of the two jobs they want (again, this is not about personal freedom, still strictly a plan to maximize equality and fairness). If both jobs pay the same, then every single citizen will choose to become a puppy-sitter. So you adjust the salaries, lowering the puppy-sitter salary and raising the angry-customer-handler salary. Then maybe you get a 90/10 split instead of 100/0. So you keep gradually adjusting them until you get to 50/50 for each job — at this point, the median citizen is roughly indifferent between the two of them.

Here’s the court order, vacating the CFTC’s ban on election markets:

For those not familiar with the legal jargon, “summary judgement” means the court has decided to issue a ruling without having a full trial — often done when there’s no real dispute about the key facts of the case.

When asked for an official statement, Kalshi CEO Tarek Mansour said, and I quote: :)

This is huge! Great news and congrats to Kalshi!! Of course, the CFTC could appeal the ruling, potentially leading to another legal battle, but at least for now Kalshi seems to be in the clear to legally operate prediction markets on US elections — and just in time for the 2024 election!

By the way, I’m assuming that most subscribers to this blog are already pretty supportive of prediction markets, but just in case anyone is thinking “Wait, why is this a good thing? Isn’t this basically just gambling?” please check out my other article The Case for Legalizing Prediction Markets, where I talk about how prediction markets differ from other forms of gambling, most importantly in their use as information-aggregators and the positive externalities this creates for society.

Take Ben Franklin, for example. On the 4th of July, we usually credit Ben Franklin and the other founders with establishing the United States as a political entity. And sure, that’s important. But Franklin was also a founding father of the American entrepreneurial spirit. He was a scientist, an inventor, a businessman, and a philanthropist — and yes, a political leader, but that was the least cool thing about him.

And Ben Franklin just did these things. Unlike many of the other founders who were highly-educated (Adams at Harvard, Madison at Princeton, Hamilton at Columbia), Franklin only had two years of formal schooling, from age 8-10. That was it. Beyond that he was self-taught, driven by curiosity to read and learn about topics he was interested in. Just like you can just do things, you can also just learn about topics you’re interested in.

And then there was Thomas Edison. In a lot of ways, he had a similar upbringing to Franklin — very limited formal education (only a few months in school), but he was curious about the world and loved to read, learn, and experiment. Like Franklin, he started working at a very young age and learned how to be an entrepreneur by doing it (same job as Franklin too — a printing press!).

Then, while still a teenager, Edison worked as a telegraph operator, specifically requesting the night shift so that he could tinker around with the machines and do experiments while on the clock. This had mixed results. On the one hand, consider this quote from the Thomas Edison House website:

Edison requested the night shift at work which allowed him plenty of time to spend at his two favorite pastimes -- reading and experimenting. However, it was the latter that eventually cost him his job. One night in 1867, he was working with a battery when he spilled sulphuric acid onto the floor. It ran between the floorboards and onto his boss' desk below. The next morning he was fired.

Yikes! But on the other hand, Edison’s tinkering and experimenting did pay off with iterative improvements on the telegraph machine, culminating in the invention of the quadruplex telegraph, which allowed four separate signals to be transmitted and received simultaneously over a single wire. This basically set the tone for Edison’s entire life — curiosity about how something works, obsessive study, and iterative attempts at improvement (the “99% perspiration” grind).

Usually when we see a photo portrait of Edison, it’s him as an old man after a successful life. But I think this is a better portrait:

This is Edison as a telegraph operator around the age of 18. At this point, the adventure is still ahead.

And of course, there were the Wright brothers. Similar background: very little formal education (one finished high school, the other didn’t), a printing business (!!), and eventually working as bicycle repairmen. Then curiosity, self-education, hard work, iteration, iteration, iteration, and a world-changing breakthrough.

These are only a couple examples. There are many other famous American tinkerers and inventors, as well as many more who should be famous. If I had more time, I could write a whole section on how many early Silicon Valley tech entrepreneurs were similar to Franklin, Edison, and the Wright brothers — clever tinkerers, lacking in formal education but driven by curiosity to learn about the world, and extremely hard-working. And maybe I’d write about this printing press pattern, and how this seems similar to the internet. But I’m about to go to the beach, so that’ll have to wait for another day.

Anyway, these are the people that I want to celebrate on the 4th of July. And this is the American spirit that I want to celebrate: this idea that you can just do things. You can just learn about something that you’re curious about. You can just build something new. And while this post was about some cool people who did this in the past, I’m even more excited about the people who are currently doing, and those who will do it in the future. Will there still be a political entity called the United States a thousand years from now? Who knows. But there will almost certainly still be people who embody this American spirit.

Alright, I’m kinda joking here, but I’m also kinda serious. Libertarians are extremely bad at this. Take the 2016 election for example. With the US facing two wildly unpopular candidates from the two major parties, many voters were looking for an alternative — the perfect opportunity for libertarians to market themselves! But rather than seizing this rare opportunity, the Libertarian Party focused instead on debating whether drivers licenses should exist and, um…… this.

That’s why Build, Baby, Build is so refreshing. Rather than focusing on some abstract philosophy about how to apply the non-aggression principle, Caplan instead focuses on a real, concrete issue that impacts most people’s lives: the high cost of housing. He presents a straightforward case: housing is expensive because its supply is artificially restricted by excessive government regulations, and the best way to make it cheaper is to remove these regulations and allow more housing to be built.

Also, it’s a graphic novel, so the pages look like this:

I’m a big fan of this format (which Caplan has used before). I think it’s great for communicating ideas in a way that’s entertaining and more accessible than a textbook or research paper. For example:

On this page, Caplan rips up a plot of supply and demand curves, and instead explains in intuitive terms why increasing supply leads to lower prices — not because the developers (who are portrayed as greedy fat cats) are just being nice and lowering prices out of the goodness of their hearts, but because they want to make as much money as possible, and this means they have to compete with each other to attract customers by lowering prices.

So yeah, I like the comic book style. Bryan Caplan’s pretty funny and this is a good format for his humor. I also think the illustrations are pretty good, even though they’ve gotten some criticism.

At the same time (and maybe fittingly for an economics book), the use of the comic book format involves a tradeoff between entertainment/accessibility and rigorous literature review. For example, Chapter 3 of the book is called “The Panacea Policy,” and makes the case that a wide variety of problems in society could be solved or improved with housing deregulation — inequality, social mobility, crime, falling birthrates, homelessness, etc. I found this part of the book to be a bit lacking in terms of supporting evidence. Every case Caplan presents here sounds plausible but there isn’t enough evidence presented to reach a solid conclusion.

Take homelessness, for example. I’ve written before about the relationship between homelessness and land-use/housing regulations, and there appears to be a correlation between them, at least at the state level.

But there are so many other relevant variables that it’s tough to say if this is a causal relationship or not. For example, it looks like the blue states tend to have more homelessness than the red states, so this is potentially a confounding variable — I probably could have plotted the homelessness rate against a lot of other variables that differ between red and blue states (gun ownership, church attendance, BBQ restaurant quality, etc), and shown a non-causal correlation between them.

Anyway, I think it’s plausible that excessive housing regulation causes homelessness, but to be fully convinced of this (and several other minor claims throughout the book) I’d need to see something like a Scott-Alexander-style deep dive on the literature or formal academic review of all the relevant studies, and that isn’t really compatible with the comic book format1.

So that’s probably my biggest criticism. And again, it isn’t even really a criticism — it’s just an acknowledgement that by using the fun comic book format, Caplan is navigating a tradeoff between entertainment/accessibility and rigorous literature review, and prioritizing the former at the necessary expense of the latter. And that’s probably a good thing that we need more of!

Anyway, I think it’s great book overall. Despite my nitpicking and skepticism around some minor claims, the fundamental argument that we should deregulate housing in order to reduce housing prices is solid and convincing, and presented in a fun and entertaining format. Caplan does a good job of making this case, and responding to all the usual counter-arguments. I definitely recommend the book, and you can buy it here if interested.

Lastly, I’d like to end this review on a note of cautious optimism. At one point, Caplan points out YIMBYism isn’t a strictly partisan issue with one party in favor and the other against. Rather, it’s a common sense solution to high housing costs that either party could pick up and run with. And in other countries, this is actually happening! In the UK, the left-wing Labour Party is promising YIMBY solutions to bring down housing costs… and they’re on track to win the next election and oust the incumbent Conservatives. In Canada it’s the opposite — the Conservatives are promising YIMBY solutions to bring down housing costs… and they’re on track to win the next election and oust the incumbent Liberals! Maybe more American politicians will start to notice that this is a winning issue, and I think Bryan Caplan’s advocacy could help us towards that goal.

Introduction

The goal of this post is to report performance metrics for the Oscars 2024 predictions from a couple different prediction market platforms. Importantly, this is a fair apples-to-apples comparison between the platforms, considering only questions for which all of them gave a prediction, and with predictions recorded at the same time (give or take a couple minutes).

The sites included in this analysis are: Manifold, Kalshi, Polymarket, and DraftKings (a traditional sports betting company).

Brief note on data collection:

• Probabilities were recorded on Friday night March 8, 2024 (2 days before the Oscars). Video documentation here.

• Odds from DraftKings were collected via SportsBookReview, which at the time of collection had most recently been updated on March 7. So the DraftKings predictions are possibly at a slight disadvantage compared to the others, since they’re a day older.

• DraftKings odds were only recorded for markets where at least one of the other sites gave a prediction.

• Full dataset and code are available on my GitHub.

Brier Scores

Very Short Explanation of Brier Scores:

In this formula, N is the total number of predictions, f_i is the prediction probability, and o_i is the true outcome of the event (either 1 if it happened or 0 if it didn’t). (Sorry about the awkward notation, I haven’t really been able to find a good way to write math notation like subscripts and stuff on Substack.) 

Basically, this is just a measure of the forecast’s average squared deviation from the true outcomes of the events being predicted. The lower the Brier score, the better the forecast, and a perfect score is 0.

Overlap of All Four

First we consider the questions for which all four platforms gave a prediction. This overlap gives us 24 events across 11 different awards. The Brier scores for this comparison are shown in the table below.

Overlap of Manifold, Kalshi, and DraftKings

Taking the overlap of all four is quite limiting, and we can get a larger sample of questions by taking the overlap of only 3. The table below shows a comparison using the overlap of Manifold, Kalshi, and DraftKings, which gives us 35 events across 14 different awards.

Overlap of Manifold, Polymarket, and DraftKings

The table below shows the comparison between Manifold, Polymarket, and DraftKings. This overlap gives us 32 events across 11 different awards.

Calibration Plots

Very Short Explanation of Calibration Plots: Let’s say that I predict a 40% chance of rain tomorrow, and then it ends up raining. Does that mean that I got it wrong? Well, no… 40% isn’t zero. Things that have 40% probability happen sometimes. But I didn’t get it right either. For a single probabilistic forecast, you can never really say if it was right or wrong, unless you forecast something with 0% or 100% chance.

But with a large collection of predictions, we can say if someone is well-calibrated or not. Being well-calibrated means that things I forecast with about 40% chance tend to happen about 40% of the time, things I forecast with about 80% chance tend to happen about 80% of the time, and so on.

Calibration plots allow us to visualize this. The horizontal axis shows binned prediction probabilities for events, and the vertical axis shows the fraction of those events that ended up happening. In the plots below, I’ve also scaled the size of the points to reflect how many events fell into each bin.

The calibration plots below show the results for each platform separately, considering all questions for which that platform gave a prediction, without considering overlap between platforms. So they should not be used for cross-platform comparison.

“Mike this is a boring result, why the heck are you even making this post?”

I agree, this result was pretty boring! There wasn’t any huge difference in platform performance, and all the platforms did about as well as each other. This isn’t surprising, because if you look at the prediction probabilities (available on my GitHub), all four platforms mostly gave very similar predictions. Maybe this is still an interesting result though, since Manifold (which uses play-money) did almost as well as the real money platforms. Besides that, pretty boring results though.

But I’m committed to publishing boring results on this blog, because when people only publish exciting results that creates a bias and skews the distribution of reported results. This is called the “filedrawer effect,” and is one of the causes behind the ongoing replication crisis in academia.

Lately I’ve been thinking that one of the most important things in science is just the non-glamorous collection and reporting of data, with the goal of contributing to a large, accurate distribution that you can use to say something about the thing you’re studying. This is one of the goals of this blog — to simply collect forecast data, evaluate it for accuracy, and report the results, with the goal of producing correct information about forecasting and prediction market performance, whether that information is exciting or not.

Thanks for reading!

Further Reading

• Scott Alexander’s Prediction Market FAQ

• Superforecasting: The Art and Science of Prediction by Phillip Tetlock and Dan Gardner

• The Signal and the Noise by Nate Silver

• Gwern on prediction markets

• Idea Futures by Robin Hanson

• Thinking in Bets by Annie Duke

What will happen to Bloomberg for getting it so wrong? Well, probably nothing. In fact, I’m not even sure they made a bad decision by publishing that article. After all, that over-the-top headline probably got more clicks and generated more advertising revenue than a headline with a more realistic (and boring) forecast would have. Bloomberg, like most media companies, is facing a complex mix of incentives: publishing correct information, getting clicks, maximizing ad revenue, getting new subscribers, retaining current subscribers, etc, and these goals don’t always completely align with each other. Sometimes appealing to the reader’s emotions (especially fear, anger, or political tribalism) might get more clicks and generate more revenue than simply conveying correct-but-boring information.

By contrast, the incentives of a betting market are completely straightforward. Correctly predicting an outcome is financially rewarded, and getting it wrong is financially punished. Or as economist Alex Tabarrok puts it: “A bet is a tax on bullshit.” But it doesn’t take an economist to understand this concept –  schoolkids understand it too: “Oh yeah, wanna bet?” 

Putting something of value on the line can quickly untangle someone’s actual beliefs from their stated beliefs.The Bloomberg editorial board were happy to print a headline predicting 100% chance of a recession, but would they have bet money on a recession happening, given 99:1 odds? Of course not, that would be throwing money away! What about 75:25 odds? 50:50? 

Well, some people were betting on whether or not there would be a recession. At the beginning of this year, the odds-implied probability of a recession occurring in 2023 was around 40-50% on the betting site Kalshi, and gradually drifted towards 0% throughout the year as it became clear that a recession wouldn’t happen. Importantly, the people betting on this market weren’t concerned with getting clicks, ad revenue, or pandering to either political side. Rather, they were actually trying to get it right so they could make money. 

This is what’s called a “prediction market.” Sites like Kalshi allow users to bet on the outcomes of events. You can bet on things like fed interest rate decisions, when GPT-5 will be released, whether Joe Manchin will leave the Democratic Party, the Oscars, and more. The markets involve assets that pay out $1 if an event occurs and $0 if it doesn’t. Traders can buy and sell these assets at different prices, which correspond to implied probabilities that an event will occur. For example, at this writing the movie Oppenheimer is trading at $0.42 to win the Oscar for Best Picture – an implied probability of 42%.

A lot of people have a negative gut reaction to this. I mean, it’s gambling! Right? Isn’t it just like a casino or the lottery or something? Regulatory agencies have had mixed reactions so far. Recently, the CFTC denied a request by Kalshi to operate election markets, and Kalshi is now suing the CFTC in response. The global platform Polymarket is illegal for Americans to bet on, though it can still be interesting to follow the markets and see how people around the world are betting. The New Zealand-based platform PredictIt is currently in a state of legal limbo, temporarily allowed to continue operating while the courts decide its fate. Manifold Markets is an experimental platform that dodges the legal issues by using play-money, with the hope that a reputation-based incentive system can take the place of the financial incentives in real-money markets. Actually, there are currently markets on Manifold about what will happen with PredictIt and whether or not Kalshi will win their lawsuit against the CFTC!

I think putting prediction markets in the same legal category as casinos and lotteries is a mistake. For one thing, these traditional forms of gambling are usually based entirely on luck, with no skill involved – and a negative expected payout for the player. By contrast, success in prediction markets is based on the ability to make accurate probabilistic forecasts, a valuable skill. In the book Superforecasting, Professor Philip Tetlock summarizes years of academic research on this topic. He finds that probabilistic forecasting is a skill that can be learned and honed through practice. It doesn’t require psychic powers or genius-level IQ – rather, it’s a matter of doing careful research and following good epistemic practices, like reasoning from base-rates and being willing to update your predictions based on new information. 

Prediction markets are a good way for participants to practice this skill, and for society to identify people who are good at it. For example, if a media network wants to put together an expert panel on some complex geopolitical issue, they might consider picking someone from the Manifold Markets leaderboard who has a solid track record of correct predictions on geopolitics, rather than simply choosing the person with the fanciest-sounding degree or job title.

Prediction markets also aggregate information in a way that’s useful for outside observers. For example, the website Election Betting Odds is currently tracking odds for the 2024 elections. Rather than trying to keep up with a constant flurry of information – new polls, something outrageous Trump said, some gaffe Biden made, biased commentary in either direction – it’s a lot easier to just check this website once in a while and see how the odds have changed. At the very least, following the markets this way allows you to track the opinions of people who are actually trying to get it right, rather than maximize clicks, pander to either political side, etc.

A best case scenario is that, under the right conditions, the Efficient Market Hypothesis might apply to prediction market prices in the same way it applies to stock prices. That is, with enough trading volume, there is a pressure on prices to reflect all publicly available information about the value of an asset, because if somebody knew some information about its underlying value that wasn’t reflected by the current price, they would have a strong financial incentive to either buy or sell/short the asset to profit from this information. The act of buying/selling an asset changes the market price of the asset (through increased/decreased demand), so the new information gets “priced in” by the trade. 

Of course, this is all theoretical. How accurate are prediction markets in real life? Well, an academic study (Berg et al. 2008) found that prediction markets often outperform polls for predicting election outcomes. Manifold Markets, the play-money platform mentioned previously, has an impressive calibration record according to an analysis posted on their website – events forecast with 20% probability tended to happen about 20% of the time, 80% probability tended to happen about 80% of the time, and so on. Kalshi also boasts an impressive record on predicting inflation, outperforming many economists. Sports betting markets, which operate according to similar fundamental principles as current event prediction markets, are also remarkably well-calibrated.

There’s also some evidence in the other direction. An analysis I did myself and posted on my own blog found that forecasts on PredictIt were slightly worse than Nate Silver’s forecasts for the 2022 US midterm elections. So prediction markets aren’t perfect and can underperform experts sometimes. But even in these cases, there will at least be some cosmic justice in the end, as bettors who get it wrong will lose money to bettors who get it right, and maybe everyone will learn from it and be a little more accurate the next time.

To conclude, while prediction markets might be considered “gambling” in a technical sense, their usefulness – both in creating good epistemic incentives for participants and in aggregating information for observers – ought to put them in a different legal category than other forms of gambling, like casinos and lotteries. 

Further Reading

• Scott Alexander’s Prediction Market FAQ

• Superforecasting: The Art and Science of Prediction by Phillip Tetlock and Dan Gardner.

• Gwern on prediction markets

• Idea Futures by Robin Hanson

• New Yorker Article about PredictIt

Introduction

This is a continuation of a previous analysis I did using some data from sports betting markets. I’m not a huge sports person, but I’m interested in prediction markets and my idea was that maybe we can learn about prediction markets by analyzing sports betting markets. After all, they’re based on the same fundamental principles as political prediction markets (like PredictIt and Kalshi), but they’ve been around much longer, are less controversial, more legal, and have tons of publicly available data.

The previous analysis was about comparing sports betting market implied probabilities to Nate Silver’s forecasts. In that post, I only included Brier scores, since I think that’s the best metric to use for forecast comparisons (I’m open to being convinced to switch to log-odds, but have not been persuaded yet). But I wish I had included regular calibration plots too, since those are more intuitive for people who want to get a sense of how accurate forecasts are.

Anyway, this post has the calibration plots and tables for the same sports betting datasets I analyzed in the previous post. I’m partly posting this because I think some of my subscribers might be interested to see them, and I’m also partly posting it for myself to use as a reference in future writing. As I mentioned before, there isn’t a lot of publicly available data yet on political / current-event prediction markets, so when advocating in favor of prediction markets I think it can be useful to show how well calibrated sports betting markets are as a general proof of concept.

Very Short Explanation of Calibration Plots: Let’s say that I predict a 40% chance of rain tomorrow, and then it ends up raining. Does that mean that I got it wrong? Well, no… 40% isn’t zero. Things that have 40% probability happen sometimes. But I didn’t get it right either. For a single probabilistic forecast, you can never really say if it was right or wrong, unless you forecast something with 0% or 100% chance.1

But with a large collection of predictions, we can say if someone is well-calibrated or not. Being well-calibrated means that things I forecast with about 40% chance tend to happen about 40% of the time, things I forecast with about 80% chance tend to happen about 80% of the time, and so on.

Results

Here are the results for MLB (baseball), NFL (football), NBA (basketball), NHL (hockey), and NCAA (college football), for the 2016 - 2020 seasons. Each event is a game, and each forecast is whether the home team will win (as in the previous post), and the data is from Sportsbook Reviews Online (same source as before).

The plots are just standard calibration plots, with the outcome fraction plotted against the binned probabilities (the center of the bin, to be specific). The dots are scaled based on how many forecasts fell into that bin.

[QUICK METHODOLOGY NOTE – in sports betting, the odds are purposely set up in a way that the raw implied probabilities sum to greater than 1 (usually like 1.02 or so) [credit to u/wstewartXYZ on reddit for correcting me on this], so that the bookies are guaranteed to make money regardless of the outcome. I decided to correct this for my analysis, and normalize the implied probabilities so that they sum to 1. I think this makes sense in terms of comparing prediction accuracy, but if anyone has a good counter-argument for why I should not do this, please let me know and I’ll consider it.]

As you can see, the betting markets are remarkably well-calibrated.

[posting on August 3, 2023. The markets I’m comparing here will probably be active for awhile, so if people are interested I’ll keep collecting data and updating the results.]

Intro

Like everyone else, I’ve been following the LK-99 room temperature superconductor news out of South Korea, wondering if it’s legit and if the results will replicate. I don’t know enough about physics to actually read the pre-prints coming out, so I’ve mostly just been watching the up and down swings of the prediction markets on Manifold and Polymarket.

While we all wait for more information to come out, I figured this could be a good time to test out a hypothesis I had about Manifold and Polymarket, and see if there’s any evidence to support or challenge it.

For those who don’t already know, Polymarket is a real-money prediction market site (illegal to bet on in the US, but legal in other countries). Manifold is a fake-money prediction market site, where users bet a fake currency called “Mana,” with the hope that their reputation will still serve as an incentive to make accurate predictions, even with no real money involved.

Hypothesis / Results

For awhile, I’ve been wondering if Manifold is more volatile than Polymarket, since there’s no real money involved and people might be willing to take bigger risks. The past couple days there have been some pretty big swings in the LK-99 prediction markets as new information comes out. So, I thought it would be interesting to compare the volatility of the Manifold and Polymarket prices for LK-99 being legit. I was basically wondering if the swings look something like this:

Like maybe both markets have similar long run averages, but maybe Manifold has more intense swings in response to news, since there’s no real money involved and people are less scrupulous and more eager to take risks. This isn’t really a 100% apples-to-apples comparison looking at real-money / fake-money, because there could also differences in trading volume, number of traders, etc between the markets, but I thought it would at least be interesting to look and see if the prices look like this.

Also, I was apparently not the only one to think this. I saw a lot of comments on the internet like:

suggesting that the fake-money nature of Manifold could be leading to people making wild bets they wouldn’t otherwise make, potentially leading to high market volatility (in fact — I myself made a reckless bet of YES on Manifold that I probably wouldn’t have made with real money! ).

Anyway, this is NOT what I found when I looked at the actual data. Here’s a plot of hourly prices on these markets over the past couple days:

At least looking at it visually, it doesn’t appear that Manifold has wilder, more extreme swings than Polymarket.

Next I looked at an actual measure of volatility: rolling coefficient of variation. Coefficient of variation (CV) is just a fancy term for the standard deviation divided by the mean. “Rolling” means that for each datapoint, I looked at that datapoint, the datapoints from the 5 previous hours, and the datapoints from the next 5 hours, and calculated the CV for this 11-hour span. So basically just think of this as a plot of volatility over time. Here’s the plot:

Again, nothing too exciting, and no obvious differences, other than some anomalous bumps in volatility for Polymarket between 7-29 and 7-31 (also visible on the price vs time plot).

Lastly, I wanted to get a single numerical measure of overall volatility for each market, so I calculated the hour-to-hour percent changes for the data, and looked at the variation in these distributions of percent changes:

Polymarket SD of Percent Changes: 13.084

Manifold SD of Percent Changes: 9.019

Here’s also the CV (again, just the above standard deviations divided by the means of the hour-to-hour percent changes):

Polymarket CV of Percent Changes: 12.947

Manifold CV of Percent Changes: 10.217

So yeah, this does NOT support my earlier hypothesis that Manifold would be more volatile, due to people being more risk-seeking when there’s no real money involved. In fact, it looks like Polymarket is slightly more volatile in this case, although there’s no enough evidence yet to say if this is a general trend that will replicate across markets. (Actually I might start making my own prediction markets about whether results I publish on this blog will replicate or not lol)

Feature Request

Nevermind about this part! Turns out I was wrong and this data IS available on the API (at least for Manifold, will need to look again for Polymarket). If anyone is having the same problem as me and can’t find it, it’s in the /v0/bets part of the API, not /v0/market.

I’m going to end this post with a feature request from Manifold and Polymarket (I think some of their employees actually read this blog, so maybe it’ll get some traction). First of all, I just want to say that I’m a huge fan of both sites. I think the work they’re doing is extremely valuable, and that prediction markets are very useful as a serious epistemological tool (as we’re seeing now with the LK-99 news).

I would like to keep doing these types of comparative analysis projects about different prediction markets. However, one of the biggest limiting factors I’m facing is that it’s actually pretty difficult and time-consuming to collect the price vs time data. Even though it’s all available to look at on the specific market pages, I can’t find any easy way to download it, and as far as I can tell there isn’t any way to get this data from the APIs (at least for an active market). (If I’m wrong about this and someone knows how to get it from the API, please let me know!)

So to write this post, I actually had to go through and manually collect the hourly price data, by tracing my mouse across the plot and typing the numbers into a spreadsheet.

This is my request to any Manifold / Polymarket employees reading this — is it possible to add time-series price data to the API for active markets, so that users can easy download the data that’s being plotted on the market information pages? That would be awesome, and would make it easier for hobbyists like me to do these kinds of informal research projects. Thanks!

For example, recently a law was passed with the goal of protecting people with sesame allergies, and the result is that now sesame is being put in many more products.

Seriously, I’m not kidding. The Food Allergy Safety, Treatment, Education and Research (Faster) Act, which was passed with bipartisan support, basically said:

• If you’re selling a food that contains sesame, it has to be labelled.

• If you’re selling a food that doesn’t contain sesame, you need to follow a bunch of burdensome rules, including careful cleaning of manufacturing equipment, to be absolutely 100% sure there is no contamination.

The result is that now a lot of companies are choosing to add small amounts of sesame to products that had previously been sesame-free, and labeling it. According to the article, this is causing some pretty serious hardship for people who are allergic to sesame:

The article also quotes some consumer protection advocates and politicians who had lobbied for the passage of the law. None of them apologized or took responsibility for the situation.

This is a general problem in public policy. Sometimes political conflicts happen when different groups have competing interests, public choice theory, etc, and advocate for policies accordingly. But other times there are these alignment problems where we simply fail to correctly predict what the results of a policy will be, so the results don’t match our desired outcome even when we get the policy we’re advocating for.

Anyway, this is just something to think about. As far as I understand, AI alignment is a technical problem about predicting the behavior of a complex system given a set of rules that the system is following (the objective function / loss function). I think these examples from public policy make good metaphors, and help explain the problem of unpredictability of complex systems even when you get to set the rules for them.

They also might be more believable as examples because they’re actually real — the famous Paperclip Maximizer sounds like science fiction to a lot of people, but a bunch of US lawmakers really did just try to program a sesame-minimizer and got a sesame-maximizer instead.

Like I said, I don’t know if I really believe in the AI doomerism stuff or not. Eliezer Yudkowsy was interviewed on the Lunar Society Podcast (great podcast btw) recently and made a compelling case that if an AGI were developed, with our current lack of knowledge about alignment, and it had the ability to physically interact with the world, that it would be very likely to destroy the human race.

I don’t really have counterarguments to what he said, except that I’m not sure if the current trajectory we’re on will lead to an AGI with agency and the ability to physically interact with the world. At least, it doesn’t seem inevitable to me that this will happen if we keep making more and more advanced ChatGPTs.

But if we do end up in a situation where we have some extremely complex system, much smarter than us, with agency and the ability to interact with the world… then yeah, I’m not very optimistic that we’ll be able to predict and constrain its behavior even if we have the power to set the rules for it.

Sidenote — I was mostly just using the sesame story as a metaphor here, but while I’m talking about it I’ll also mention a related issue. Apparently there have been some attempts by vegans to insist that any vegan options a restaurant provides must be cooked on separate equipment that isn’t used to cook meat.

In 2019, someone sued Burger King (though the lawsuit was dismissed) after finding out that their Impossible Whopper was cooked on a grill that meat had also been cooked on.

PETA has actually chimed in with a bit of sanity, urging vegans not to do this, since making it more difficult, expensive, and legally-risky for restaurants to offer vegan options will probably lead to fewer vegan options being offered. Article about it here.

Anyway, I’m a vegetarian myself, and I’d like it if we could get this alignment problem right and not discourage restaurants from offering vegetarian options by making it more difficult.

I previously wrote a post about sports betting markets, where I compared the accuracy of betting market odds and forecasts from FiveThirtyEight on predicting game outcomes for the NBA (basketball), NFL (football), and MLB (baseball). Overall, the betting markets tended to slightly outperform FiveThirtyEight across all three sports. This result was consistent with the efficient market hypothesis. However, I cautioned that we shouldn’t draw any definitive conclusions, but rather we should update our beliefs in favor of sports betting markets being efficient, while also collecting and analyzing more data to see if we keep getting the same results.

Last month I decided to try to collect some more data on this question, specifically from the March Madness college basketball tournament. Unfortunately I’m also completely swamped lately (trying to finish up my dissertation this semester), so I didn’t end up recording data on all of the games, only 16 of them. Sorry about that! This is a much smaller dataset than the previous post, which included data on thousands of games over 5 years and for 3 different sports. At the same time, it has an advantage that the previous post didn’t have: I collected the data myself, so I was able to ensure that the forecasts for each game were recorded at the same time for both the betting market and FiveThirtyEight, allowing for a fair apples-to-apples comparison between the two.

Anyway I’m still going to report the results because 1.) some new data is better than no new data, and 2.) based on these results I want to put a hypothesis on the table to test with future datasets, including hopefully a similar analysis on the NBA playoffs that are coming up.

Here’s the data I collected:

Methodology Note: Betting odds data is from FanDuel. In my previous post I had much larger datasets, but a disadvantage was that I was downloading them from other sources and didn’t know the exact time they were recorded, so it wasn’t a completely 100% apples-to-apples comparison. For the dataset in this post, I recorded the data myself by actually going to the websites (FanDuel and FiveThirtyEight), checking the odds, and manually typing them into a spreadsheet, while also documenting this on video (I’ll upload the video to Youtube probably this weekend). So, this analysis is an apples-to-apples comparison, since I was able to ensure that the forecasts for both were recorded at the same time, for the same events being predicted.

Here are the resulting Brier scores (remember, lower is better for Brier scores, and a perfect score is 0):

So on these 16 games (again, a small and incomplete sample), FiveThirtyEight very slightly outperformed the betting market odds, but both did about the same as the control (which simply forecast 0.5 every time).

“Mike this is a boring result, why the heck are you even making this post?”

I agree, this is a boring result! And not only that, but it’s a boring result that’s very slightly in the opposite direction of my hypothesis and a previous result that I got on a much larger dataset (that betting markets would outperform experts due to the Efficient Market Hypothesis). So yeah, from my standpoint this result is both boring and also slightly inconvenient.

The reason I’m writing this post is that it’s important to publish the boring and inconvenient results! If people only publish the exciting results that support their hypotheses, that skews the distribution of reported data. This type of publication bias is called the “filedrawer effect,” and is one of the causes behind the ongoing replication crisis in academia.

Lately I’ve been thinking that one of the most important things in science is just the non-glamorous collection and reporting of data, with the goal of contributing to a large, accurate distribution that you can use to say something about the thing you’re studying. This is going to be one of the goals of this blog — to simply collect forecast data, evaluate it for accuracy, and report the results, with the goal of producing correct information about forecasting and prediction market performance, whether that information is exciting or not.

Another reason I’m making this post is that I want to put a hypothesis on the table to test with future analyses. Hypothesis: both betting markets and expert forecasters like Nate Silver / FiveThirtyEight will tend to have worse accuracy on playoffs and tournaments compared to regular season sports games.

At first I was surprised by how badly both the betting markets and FiveThirtyEight did in this small dataset that I collected — they performed about as well as the control of simply forecasting 0.5 probability every time. This is substantially worse than their performance in my previous post, looking at mostly regular season games. But after thinking about it, this makes sense. In regular season games, you often have a matchup between a good team and a bad team, and it’s easier to predict the outcome. But in tournaments and playoffs, all of the teams are pretty good because you need to already be one of the best to make it into the tournament/playoffs. So that probably makes the forecasting harder, since there aren’t as many lopsided matchups.

Anyway this is just a hypothesis I’m putting on the table. I’d like to do a similar analysis in the NBA playoffs coming up in a couple weeks to see if I get similar results with those.

Thanks for reading!

Imagine two people – I’ll call them Partially-Psychic Pete and Just-A-Regular-Guy John – are playing a game where they take turns attempting to make predictions about a fair coin toss by forecasting the probability that it will come up heads.

Pete is both partially-psychic and overconfident. When it’s his turn, he always forecasts 1 or 0 for the probability of heads, and he gets it right 60% of the time. So Pete’s predictions look like this (an outcome of 1 means heads, and 0 means tails):

Except imagine a thousand or so predictions with 60% of them being correct.

John, on the other hand, is not a psychic, and correctly judges his own uncertainty about the coin toss. When it’s his turn, he simply gives a forecast of 0.5 probability of heads each time. His predictions look like:

Again, imagine a thousand or so predictions like this.

So, who’s the better forecaster? Who’s better “calibrated”? 

In one sense Pete’s the better forecaster. He’s actually partly psychic and his predictions add new information about what we should expect for the outcome. In another sense, John’s the better forecaster, since he’s judging his own certainty correctly — he predicts 0.5 probability of heads, and the coin does indeed come up heads 50% of the time.

This is a minor pet peeve I have when it comes to forecast evaluation. When we talk about accuracy/calibration, we’re actually talking about two different things that are kinda related, but can also be separated from each other. Something about the imprecise language around this bothers me.

Has anyone written about this before? I couldn’t find anything, but if so please let me know so that I can cite them and use their terminology. But for now, I’m going to refer to Pete’s type of accuracy as Correct Prediction Accuracy and to John’s type of accuracy as Uncertainty Judgement Accuracy.

As we just saw in the example, it’s possible to have a good — even perfect — Uncertainty Judgement Accuracy, while having terrible Correct Prediction Accuracy (John correctly judged the fact that he didn’t have any information about the outcome of the coin flip beyond random chance). You can also have very impressive Correct Prediction Accuracy (literally being psychic with 60% accuracy on a fair coin flip) and still have terrible Uncertainty Judgement Accuracy.

How do Brier scores handle this?

Brier scores are the most commonly used (and best, in my opinion) metric for forecast evaluation. The formula is:

Here, N is the total number of predictions, y_i is the prediction probability, and o_i is the true outcome of the event (either 1 or 0). (Sorry about the awkward notation, I haven’t found a way to write the subscripts on Substack.) 

Basically, this is just a measure of the forecast’s average squared deviation from the true outcomes of the events being predicted. The lower the Brier score, the better the forecast, and a perfect score is 0.

Brier scores incorporate both Correct Prediction Accuracy and Uncertainty Judgement accuracy — you’re rewarded for predicting the outcome correctly and correctly judging your own uncertainty won’t get you a perfect score, but you’re also punished for overconfidence. This combination is kinda good in the sense that it makes Brier scores a useful measure of overall forecast performance. But it’s also kinda bad in the sense that it includes an implicit tradeoff between the two that is somewhat arbitrary.

Let’s go back to the coin flip example. Remember that Pete is overconfident and gives probability 0 or 1 every time, and gets it right 60% of the time. John correctly judges his own uncertainty, and gives a probability of 0.5 every time.

Under these conditions, Pete’s Brier score is 0.4 and John’s Brier score is 0.25 (just trust me for now, in the next sections we’ll calculate through computer simulations and on paper). Remember, lower is better, so John has the better Brier score — his correct judgement of his own certainty is enough to beat Pete’s correct predictions of the outcome.

But what if instead of getting it right 60% of the time, Pete was getting it right 80% of the time? Under those conditions, Pete’s Brier score would be 0.2 — better than John’s score of 0.25! Apparently 80% correct, even when overconfident, is enough to outweigh John’s perfect judgement of his own uncertainty.

What if Pete was getting it right 75% of the time? Then his Brier score would be 0.25, equal to John’s score. Here’s all of that in a nice table:

Anyway, I still think Brier scores are probably the best method we currently have for evaluating forecasts, but it’s important to be aware of these two types of accuracy, and the implicit tradeoff that Brier scores make between them.

Python Simulation of Coin Flip Game

Here’s a Python program you can use to simulate the coin clip game, and compute Brier scores for the players, setting any correctness probability you want for the partial-psychic Pete. I think you should be able to copy and paste this directly into a .py file and run it, but if anyone’s having trouble let me know and I’ll just put it on Github instead.

import random


def brier_score(y_true,y_score):

    # check to make sure predictions and outcomes are the same size
    assert len(y_true) == len(y_score)

    # N = total number of coin tosses
    N = len(y_score)

    # summing up squared difference between prediction prob and outcome
    total = 0
    for i in range(N):
        total += (y_score[i] - y_true[i])**2

    # dividing by total number of coin tosses
    total = total/N

    return total

# 1 for heads, 0 for tails
y_true = []

# forecasts 0.5 every time (John in the story)
control = []

# forecasts 0 or 1 every time, but getting it right with some prob
# (Pete in the story)
partial_psychic = []


## How often does the overconfident partial-psychic get it right?
psychic_correct_prob = 0.6
# psychic_correct_prob = 0.75
# psychic_correct_prob = 0.8


num_coinflips = 1000000

for i in range(num_coinflips):

    # random draw, 1 is heads, 0 is tails
    draw = random.randint(0,1)
    y_true.append(draw)

    # control forecasts 0.5 probability of heads every time
    control.append(0.5)

    # psychic forecasts 0 or 1, and gets it right with some prob
    if random.random() <= psychic_correct_prob:
        partial_psychic.append(draw)
    else:
        partial_psychic.append(abs(1-draw))





print(f"Control Brier Score: {brier_score(y_true,control)}")
print()
print(f"Psychic Brier Score: {brier_score(y_true,partial_psychic)}")

Computing Brier Scores on Paper

[WARNING!! MATH AHEAD!! Seriously though, this section is just some optional extra information and can be skipped.]

This is a brief explanation of how to use some basic probability theory to compute Brier scores for the coin flip game on paper. We’re going to use a concept known as expected value. The expected value of a random variable is the average of all of the possible values of the random variable, weighted according to their respective probabilities:

Here, X is the random variable, x_1, x_2, …, x_n (sorry again, can’t write subscripts on Substack) are the possible values that X can take, and p_1, p_2, …, p_n are the respective probabilities of X taking those values. By convention, the angle brackets <> are used to denote the expected value of a random variable.

Let’s take a concrete example. Let’s say you’re playing a game where you can win $2 with probability 0.5, $10 with probability 0.3, and $20 with probability 0.2. Your expected winnings are $8:

Alright, now let’s take another look at that Brier score formula:

Basically, the formula is taking the average of (y_i - o_i)^2 (again sorry for the awkward subscript notation here). Remember, y_i is the forecast probability of the coin coming up heads, and o_i is the outcome, either 0 (tails) or 1 (heads), for each flip of the coin, with N flips total.

Let’s define a new random variable now, to represent the difference between y (forecast probability) and o (outcome):

Remember how we said the Brier score is just the average of (y_i - o_i)^2? Well now we can write the Brier score as the expected value (average of the outcomes weighted according to their probabilities) of our new variable X squared:

Ok, let’s try using this approach to actually calculate the Brier scores for Pete and John. Remember, John forecasts 0.5 chance of heads every time. So if the coin comes up heads, then X = (y_i - o_i) = (0.5 - 1) = -0.5. If it comes up tails, then X = (y_i - o_i) = (0.5 - 0) = 0.5. So,

We can then compute the expected value of X squared by simply squaring both of the outcomes and taking the average, weighted according to their respective probabilities:

And so we’ve confirmed John’s Brier score of 0.25.

What about Pete? Remember Pete is an overconfident partial-psychic who forecasts 0 or 1 every time, and gets it right some percentage of the time. Let’s use a constant parameter “f” for Pete’s probability of getting it right. Let’s think again about our random variable X, which measures the difference between the forecast probability and the outcome. In Pete’s case, X will be 0 with probability f — those are the times he’s getting it right. He’s getting it wrong with probability (1-f). Half of these times he’s forecasting 1 and it comes up tails (0), so X will be 1 with probability (1-f)/2. The other half of the his incorrect guesses will be a forecast of 0 and it comes up heads (1), so X will be -1 with probability (1-f)/2. So,

Remember, to find the Brier score we need the expected value of X^2. Again, we can find this by simply squaring all of the outcomes and taking their average, weighted according to their respective probabilities:

So if Pete is forecasting 0 or 1 every time, and getting it right with probability f, then his Brier score is 1-f. We can use this formula to recover the Brier scores listed in the table previously. If his probability of getting it right is 0.6, then his Brier score is 0.4. If his probability of getting it right is 0.75, then his Brier score is 0.25. If his probability of getting it right is 0.8, then his Brier score is 0.2.

Further Reading

• Superforecasting: The Art and Science of Prediction by Phillip Tetlock and Dan Gardner. AMAZING BOOK!! Really the best introduction to forecasting. 

• The Signal and the Noise by Nate Silver

• Gwern on prediction markets

• “A Bet Is a Tax on Bullshit” – Marginal Revolution post by Alex Tabarrok

• Idea Futures by Robin Hanson

• Introduction to Prediction Markets, by Jorge I. Velez 

• New Yorker Article about PredictIt

• Scott Alexander’s Prediction Market FAQ

Credit to Gary Winslett for noticing this pattern:

I saw this tweet from him earlier today, and decided to do a quick analysis to put some numbers on this observation and visualize the relationship. If you want to read more, here’s a longer article from Gary related to this topic.

The Land Use Freedom rank here comes from the Cato Institute, and is based on scores across several categories including local rent control laws (which count negatively), Wharton Residential Land Use Regulatory Index (a measure of housing market regulation), and more. Details here.

Below is a plot of state homelessness rate per 10,000 people (data here) against land use freedom score (from this Cato Institute report, data available here, calculated with relative weights listed here). This is using only the 2019 data from the Cato report.

And yes I watermarked this plot in case anyone tries to use it as definitive evidence for causation lol. But the correlation is pretty interesting. R-squared of 0.3822 means that 38.22% of the variance in the homeless rate can be explained by the land use freedom score.

I also colored the dots red and blue based on whether they voted Republican or Democrat in the 2020 presidential election. It looks like the blue states tend to have more homelessness than the red states, so this is potentially a confounding variable — I probably could have plotted the homelessness rate against a lot of other variables that differ between red and blue states (gun ownership, church attendance, BBQ restaurant quality, etc), and shown a non-causal correlation between them.

Edit 2-7-23: When I first wrote this, I mentioned that the difference between red states and blue states could be a confounding factor, but I didn’t expand on why. u/GymmNTonic on the SSC subreddit listed some specific reasons why blue states might have more homelessness than red states. Here is their comment:

These are big, densely populated cities that are liberal. I’m not saying the author is 100% wrong, there’s obviously regulation issues driving housing inflation that can result in homelessness, but I think bigger factors are that it’s easier to be homeless in big liberal cities that:

a) don’t bus you out to other cities and are generally more tolerant of the population existing

b) where you have more opportunities to panhandle from a very dense population

c) have more social services for the homeless

d) for those who are not mentally ill and have the capacity to work, more job opportunities

e) in many cases, more opportunities for shelter against winter in big city infrastructure, particularly Los Angeles

So, it’s more that it’s much much more difficult to be mentally ill in rural areas with low cost of housing, because to someone in full on psychosis, any housing cost other than a free awning on a doorstep is still too much cost.

Anyway, it could be that blue states tend to have more homelessness and blue states tend to have more restrictive housing markets and expensive housing — but that the market regulations and expensive housing are not causing the homelessness.

Still, the relationship holds even when we analyze the red and blue states separately, although it’s a bit weaker.

Anyway, these simple linear regressions aren’t solid, definitive proof of anything, but I think they’re enough to put a hypothesis on the table for further investigation. Correlation doesn’t always mean causation, but in this case there’s a clear potential mechanism of causation: NIMBY regulations → lower housing supply→ more expensive housing → more homelessness.

Here are plots of the relationship between the land use freedom score and median home value (data here).

Then, lastly, Alyssa Vance showed in her LessWrong post that housing costs are correlated with the homelessness rate (R-squared ≈ 0.69).

Again, none of this is solid proof that overregulation of housing markets is causing homelessness. But I think it’s enough to warrant further investigation of this hypothesis, and maybe look for causation in natural experiments. For example, Oregon recently passed a statewide YIMBY policy to reform some zoning regulations and allow more housing to be built — it will be interesting to see if this is followed by a drop in the homelessness rate in Oregon in the coming years.

Whenever there’s some big event happening like the World Cup, Olympics, or a presidential election, I often see headlines about psychic animals:

Sometimes the psychic animal seems to have a pretty impressive record. For example, Paul the Octopus, who predicted soccer games from 2008-2010, had a record of 12 correct predictions out of 14 (87.5% correct). This included a perfect record of 2010 World Cup predictions (image source):

Unfortunately, as you might have guessed, these animals aren’t really psychic. What’s probably happening is that for all of these events, many animals make some type of prediction. Then after the event is over and the results are known, the animals that happened to get it right by chance make the news. If you were to select some animals to watch before the results were known, or simply keep following the animals from these articles into the future, their performance would be consistent with regular probability from random guessing (unless there’s some kind of fraud or Clever Hans thing going on).

Eh, whatever. These stories are still cute and fun to read about, and probably most people aren’t taking them too seriously anyway.

Ok that’s enough about these cute psychic animal stories. Now on to a serious topic: financial investing. When you’re researching how to invest your savings, you might come across some articles like these, recommending mutual funds:

A mutual fund is a type of investment that contains a portfolio of stocks, bonds, and other securities. They’re usually1 actively managed by financial professionals, who make decisions about the composition of the fund, and charge a fee of ~0.5% per year of the total investment. 

The way these articles from sites like Forbes, US News, and Nasdaq.com usually choose which mutual funds to recommend is by looking at past performance and picking the funds that have beat the market over the past year, 5 years, 10 years, or whatever time period. 

But wait… according to the Efficient Market Hypothesis it should be impossible for any investor to consistently outperform the market, other than due to luck. So what’s going on? Are these mutual funds really able to beat the market because they’re more skillfully managed and have better portfolios?

My guess is no. I’m willing to bet2 that if you keep tracking the mutual funds recommended by these articles into the future, they will not outperform the market on average over the long run.

Can we test this prediction? Well for the articles recommending funds to invest in for 2023, we’ll just have to wait and see. But what we can do is go back and take a look at similar articles from the past. For example, Nasdaq.com’s list of the top performing mutual funds for 2016 is still up on their website. Here’s the top of the list:

It goes on to list 60 mutual funds total, chosen based on their 2016 return percentage. The numbers here look insanely good: 70.51% return, 55.65% return, etc. 

But how did these funds do in the year after the article was published?

I used the Python yfinance library to calculate the percent returns of these mutual funds in the year before the article was published, and the year after. I then compared these returns to the returns from SPY (a stand-in for overall market performance) over the same time period. The plots below show the distributions of fund-to-SPY return ratios. I put a vertical dashed line at 1, since a ratio of 1 means that a mutual fund had the same return on investment as SPY. Return ratios above 1 mean the fund beat the market, and return ratios below 1 mean the fund underperformed the market.

The top plot shows the return ratios in the year before the article was published. During this time period, the mutual fund returns substantially outperformed the market. The bottom plot shows the return ratios for the same mutual funds in the year after the article was published. For this time period, the vast majority underperformed the market.

A couple technical notes:

• Code for analysis available here

• The funds JHSVX, WTSVX, and ACRTX are excluded because I couldn’t find the data for them with yfinance.

• I used Adjusted Close here to calculate returns rather than Raw Close. I think this is the appropriate measure because Adjusted Close accounts for things like dividends and splits, but if anyone has a case for why I should have used Raw Close please let me know (I’m really not an expert with this stuff). 

That was the only article I found that had a long list of top-performing funds, enough to plot a distribution. But I also found a couple articles with short lists of top-performing funds, so I did a similar analysis on those.

Article

Publication date: February 6, 2014

Fund selection criteria: 1-year performance

Article

Publication date: October 15, 2015

Fund selection criteria: 10-year performance

Again, the numbers here are the fund/SPY return ratios, so a number above 1 means the fund beat the market, and a number below 1 means the fund underperformed the market.

So, what do psychic animals and mutual funds have in common? Both look pretty impressive if you select the ones with the best performance after the results are already known. But when you do this, you’re really just selecting the ones that got lucky, and if you continue following them into the future, the impressive results evaporate into the underlying probability distribution.

In this post I decided to focus on mutual funds, but this type of probabilistic fallacy, where results from random chance are misinterpreted due to a selection effect, is really a general problem found in many different fields besides just finance. For example, a closely related problem in academic contexts is called HARKing, an acronym coined by Norbert Kerr that stands for “Hypothesizing After Results are Known.” This is thought to be one of the causes of the replication crisis, and is why preregistration of experimental studies is so important.

Further Reading

• NYT Article by Jeff Sommer: “Mutual Funds That Consistently Beat the Market? Not One of 2,132.”

• A Random Walk Down Wall Street by Burton Malkiel – This book is basically the opposite of the “get-rich-quick” investing books. It advocates for a passive investing strategy (index funds and index-ETFs) rather than trying to beat the market through stocks or actively-managed funds, which he argues is impossible in the long run. Here’s a lecture the author gave if you don’t feel like reading the book.

• Academic paper on HARKing

Introduction

I’ve been interested in prediction markets for a while, and have a lot of questions about how they work that I think could be answered through some pretty simple data analysis. Unfortunately, there aren’t a lot of publicly available datasets for sites like PredictIt, Manifold Markets, etc. I previously did a project where I collected data from these sites myself and analyzed accuracy for the 2022 midterms, but this approach has some limitations – the data collection is very tedious and can only be done in the present, and would have to be done for several years to see if a trend repeats itself. 

Something that occurred to me, though, is that maybe some of my questions could be answered by looking at data from sports betting markets. They’re basically the original prediction markets – based on the same fundamental principles as political prediction markets, except they’ve been around much longer, are less controversial, more legal, and have tons of publicly available data.

In this post, I’ll try to answer a question about prediction market efficiency by comparing data from sports betting markets to forecasts made by Nate Silver (of the famous forecasting site FiveThirtyEight).

Are Prediction Markets Efficient?

When I say “efficient” here, I’m not using it as a synonym for “good,” “fair,” or some other value judgment. I’m using the word in the sense of the Efficient Market Hypothesis (EMH). According to the EMH, when financial assets are traded under the right conditions (enough traders, low barriers to entry, etc), their prices will reflect all available information about the underlying financial value of the assets.

The idea is that if there was some piece of information that wasn’t already reflected in the asset price, there would be a financial incentive for anyone who knows this information to make the appropriate trades based on it, thus pricing in the information.

There’s a famous but inaccurate economics joke about the EMH, which has since been corrected by an extremely unfunny but more accurate version of the joke - reading these might be helpful in understanding the concept.

How do we know if a market is efficient? Well, we can never really be sure. But there are ways to spot if a market is inefficient. For example, a corollary of the EMH is that it should be impossible for an expert’s predictions to consistently outperform the market in the long run (other than due to luck), because if this were happening, then there would be an incentive for people to start trading based on this expert’s predictions, thus pricing in the information and eliminating the market inefficiency. So if we see an expert making predictions that consistently outperform the market again and again, that probably means the market is inefficient – unless we’re making a probabilistic fallacy that causes us to interpret luck as impressive skill in hindsight.

So, are prediction markets efficient? Scott Alexander addressed this in his recent Prediction Market FAQ – and he says yes, at least when the market is operating under the right conditions. The problem, according to Scott, is that a lot of the prediction markets are not operating under these ideal conditions. For example, it’s somewhat well known that PredictIt tends to have a slight conservative bias and consistently underprices the Democrats’ chances of winning. Scott mentions that he typically makes a couple hundred bucks per election cycle by betting counter to this inefficiency (and not to brag, but so do I actually).

Also, I recently did an analysis of forecast accuracy for the 2022 midterm elections, and found that Nate Silver outperformed PredictIt overall. This is not a solid confirmation of the market inefficiency, since it was only based on one year, but it’s a piece of evidence that supports what Scott and many others have anecdotally observed. 

But according to Scott, this inefficiency only exists because PredictIt has a $850 cap on how much you can bet – if it weren’t for this cap, someone would take advantage of knowing about this inefficiency to make a much larger bet, enough to price in the information. 

I find this to be a convincing theoretical argument, but it would be nice to have some data to test it. This is where the sports betting data comes in. 

Results

In addition to election forecasts, FiveThirtyEight (Nate Silver’s site) also makes forecasts about sports games. One great thing about FiveThirtyEight is that they make all of their forecast data available online, for anyone to analyze, so I was able to get ahold of their predictions for the 2016-2020 MLB, NFL, and NBA seasons. These datasets included forecast probabilities for every game in each season – forecasts for several thousand games total.

I was able to get ahold of some sports betting odds datasets for the 2016-2020 MLB, NFL, and NBA seasons from Sportsbook Reviews Online, which has an impressive archive of data for a variety of sports. Again, I’m looking at the odds for each game in each season.

[QUICK METHODOLOGY NOTE – in sports betting, the odds are purposely set up in a way that the raw implied probabilities sum to greater than 1 (usually like 1.02 or so) [credit to u/wstewartXYZ on reddit for correcting me on this], so that the bookies are guaranteed to make money regardless of the outcome. I decided to correct this for my analysis, and normalize the implied probabilities so that they sum to 1. I think this makes sense in terms of comparing prediction accuracy, but if anyone has a good counter-argument for why I should not do this, please let me know and I’ll consider it. 

(By the way, do you guys know what I mean when I’m talking about odds and their implied probabilities in this context? Right now I haven’t written a methods section for this post, but if people are confused about this let me know and I’ll write up a methods section explaining it.)]

[ANOTHER METHODOLOGY NOTE – when I did my previous analysis of the midterm forecasts, I made sure to collect the data from all sites at the same time (give or take 15 minutes or so), in order to have a solid apples-to-apples comparison between sites. Unfortunately with this analysis of sports betting and forecasts, I don’t know the exact time that these odds and forecasts were recorded, so there is some possibility of bias in timing. With that being said, I think the results are informative even when we factor this consideration in.]

Anyway, here are the results comparing FiveThirtyEight to the betting market data, for the 2016-2020 seasons of the MLB (baseball), NFL (football), and NBA (basketball). Please note that Brier scores are a measure of error, so lower scores are better here.

As you can see, Nate Silver does nearly as well as the betting markets, but is not able to consistently outperform them in a way that would suggest a market inefficiency.

Here are the results in a bar graph, with error bars showing one standard deviation:

Do you guys remember that lesson from 7th-grade science class about how we never actually “accept” a hypothesis, but rather we “fail to reject” it? I think that kind of language is appropriate here. Based on this analysis, we can’t accept that sports betting markets are efficient, but so far we have failed to reject this hypothesis. Or to put it in Bayesian terms, these results warrant some update in favor of the efficient market hypothesis being true for sports betting markets.

Reasons Why This Might Not Extrapolate to Political Prediction Markets

Even though the whole point of this post is to attempt to learn about prediction markets by studying sports betting, I think it’s important to be cautious here. It’s appropriate to update our beliefs in the direction of thinking prediction markets are efficient in general (when operating under the right conditions, like no bet size limits), but this is far from being a rock-solid confirmation. 

For one thing, it’s possible that a prediction market could be biased because of a selection effect related to the type of people who trade in the market (this could be true even in the absence of bet size limits and legal barriers to entry) – and it seems like this kind of selection effect could affect political prediction markets like PredictIt more than it affects sports betting.

For example, probably most traders in both sports betting markets and political prediction markets are men. In political prediction markets, this could cause the markets to skew slightly conservative in a consistent way – especially if the prediction markets attract men who are interested in finance and economics. On the other hand, it’s more difficult to see how sports betting markets being primarily made up of men would bias the market in favor of some teams over others.

This is just speculation. Anyway, the point is there could still be some differences between sports betting markets and political prediction markets (other than just betting caps, market size, and legal barriers to entry), that could lead to sports betting markets being efficient in a way that sites like PredictIt aren’t – so we need to be very cautious about extrapolating here.

[SIDE NOTE – one possible bias I can think of for sports betting markets is that teams with larger fanbases may have more people betting on them to win compared to teams with smaller fanbases, biasing the forecast probabilities in favor of the more popular team. For example, when the NY Yankees are playing the Milwaukee Brewers, it could be that the Yankees’ chances of winning are consistently overestimated simply because they have a much larger fanbase and more fans betting on them to win. I’d like to do an analysis to see if this is true, and either write up the results in a report or (if it is true) maybe just keep it to myself and try to make some money lol.]

Another Interesting Result

Unrelated to the efficient markets question, but during this analysis I also noticed that some sports seem to be inherently more predictable than others. Here’s a plot of Brier scores for the betting markets from 2016-2020, with error bars showing one standard deviation. This plot includes the sports I mentioned before, as well as the NHL (hockey) and NCAA football (this is a college football league, in case anyone reading this is unfamiliar).

I think this result is somewhat interesting. Some sports like hockey and baseball are apparently hard to predict, while NCAA football is pretty predictable by comparison.

Conclusion

This analysis has failed to disprove the efficient market hypothesis for sports betting. We need to be very cautious about extrapolating results from sports betting to political prediction markets, but I think it does warrant some update in favor of the EMH being true for prediction markets in general, under the right conditions. 

Data and code for this analysis are available on my Github.

I’m probably going to keep writing about forecasting and prediction markets in my free time, so if you’re interested in reading more about this please subscribe (for free)! And if anyone has any specific analyses they’d like to see or questions they’re interested in, please let me know and I’ll work on answering them.

Thanks for reading!

Subscribe now

Further Reading

• Superforecasting: The Art and Science of Prediction by Phillip Tetlock and Dan Gardner. AMAZING BOOK!! Really the best introduction to forecasting. 

• The Signal and the Noise by Nate Silver

• Gwern on prediction markets

• “A Bet Is a Tax on Bullshit” – Marginal Revolution post by Alex Tabarrok

• Idea Futures by Robin Hanson

• Introduction to Prediction Markets, by Jorge I. Velez