Nate Cohn

Nate Cohn @Nate_Cohn

I think B is a very scary world for poll aggregators. A lot of the underlying samples can be biased the same way. A lot of resulting choices will be similar (try past vote!). And a lot of the choices are calibrated to expectations (aka herding, which risks systemic bias). — PolitiTweet.org

Nate Cohn @Nate_Cohn

One last way to think about it and then I'll shut up. Polling averages can work for one of two reasons: a) poll samples, industry wide, are more-or-less unbiased after basic weighting; b) pollsters, industry wide, can eliminate bias in ways that cancel out in the aggregate — PolitiTweet.org

Nate Cohn @Nate_Cohn

@davidshor idk doesn't look that weird to me — PolitiTweet.org

Nate Cohn @Nate_Cohn

But this has some long term consequences for the FiveThirtyEight scheme as a whole, since a world where live phones can be systematically biased is one where FiveThirtyEight can't reduce bias by calibrating its House Effects to a particular kind of survey — PolitiTweet.org

Nate Cohn @Nate_Cohn

FiveThirtyEight is more or less acknowledging that this is happening, indirectly. Whatever qualitative advantage live phone samples still enjoy is no longer making up for their lighter weighting schemes, at least in an electoral context — PolitiTweet.org

Nate Cohn @Nate_Cohn

Whether a pollster is prone to systemic bias really boils down to two things: --does the poll make heavy bias-variance trade offs? lvie very rarely; ivr/online all the time --is the underlying sample unbiased? this was live's case a few years ago; now it's at least in question — PolitiTweet.org

Nate Cohn @Nate_Cohn

That's great for the world of 2006, when you could assume the industry-wide sample/choices are more-or-less nonbiased. Today, you'd rather have a rating based on whether a firm is prone to systemic bias--though it's basically impossible to construct with the available data — PolitiTweet.org

Nate Cohn @Nate_Cohn

The FiveThirtyEight pollster ratings, on the other hand, are almost entirely constructed around pollster average error. The ratings basically don't care whether you're biased; the assumption is that they can get rid of the bias with House Effects — PolitiTweet.org

Nate Cohn @Nate_Cohn

As a result, I don't think the fairly typical average error in 2020 is really worth looking at tbh. I'm not saying it's irrelevant, but it kind of is! If there's no systemic error, the polling averages will be great and the big picture story will be sound, even if error is high — PolitiTweet.org

Nate Cohn @Nate_Cohn

If we get to the point where the typical systemic error is greater than 3 pts or so, I think that's a big challenge. The whole range of presidential election results in my lifetime is, what, R+2 to D+7? 3 pts, either way, includes most natl outcomes — PolitiTweet.org

Nate Cohn @Nate_Cohn

Virtually every non-phone pollster is making choices like this. It does keep their error down and it can also reduce bias. But it also basically makes it inevitable that every election includes some systemic bias, especially since these choices are similar — PolitiTweet.org

Nate Cohn @Nate_Cohn

Take party ID. Let's say you get back that's D+10 and Clinton+6. You can weight it to D+5 and Clinton+2 probably reduce error. But if I always weight to D+5/Clinton+2, and the electorate is D+2/Trump+1, then *on average* my polls are biased by Dems to 3 points — PolitiTweet.org

Nate Cohn @Nate_Cohn

What's a bias-variance trade off? Well, often times you can make a choice between reducing error at the expense of inducing possible bias. — PolitiTweet.org

Nate Cohn @Nate_Cohn

There are fundamental, underlying reasons why the risk of bias is increasing, while error stays low. The quality of the survey data is decreasing, as response rates decline and polls go online. Pollsters compensate by weighting, forcing industry-wide bias-variance tradeoffs — PolitiTweet.org

Nate Cohn @Nate_Cohn

Now, should you care more about average error or systemic bias? I think you should care way more about systemic bias. Poll averages can be fine with high avg error, so long as bias is low. But even the lowest avg error on record--4.8 points--would be horrible iff it was all bias — PolitiTweet.org

Nate Cohn @Nate_Cohn

You can see both the magnitude of systemic bias in this chart, along with the case that there's a trend toward greater systemic bias. And fwiw, I think the D+5 bias is probably mitigated by some 'nonpartisan' firms that, tbh, aren't so nonpartisan or above the board https://t.co/SIYr8lXMop — PolitiTweet.org

Nate Cohn @Nate_Cohn

I'd boil down the disagreement to one fundamental thing: I think low systemic bias is far, far more important than important for thinking about the polls than average error, while I think @NateSilver538 looks a lot more at average error — PolitiTweet.org

Nate Cohn @Nate_Cohn

I'd boil down the disagreement to one fundamental thing: I think low systemic bias is far, far more important than important for thinking about the polls average error, while I think @NateSilver538 looks a lot more at average error — PolitiTweet.org

Nate Cohn @Nate_Cohn

I don't really agree with @NateSilver538's glass-half full perspective on the 2020 polling miss https://t.co/1lwJLknbt3 — PolitiTweet.org

Nate Cohn @Nate_Cohn

Yeah, I don't think we have any reason to believe, based on the history of midterm elections, that these early generic ballot figures (or presidential approval ratings) have a lot of value for gaming out what the national environment will look like in 2022 https://t.co/QMyKLjzTtn — PolitiTweet.org

Nate Cohn @Nate_Cohn

@BrendanNyhan take it or leave it, but in similar circumstances i've used the general population weight as a first stage weight and then raked to known subgroup demographics — PolitiTweet.org

Nate Cohn @Nate_Cohn

RT @Reuters: Name of vaccine Russia's Vladimir Putin to take will be a secret, says Kremlin https://t.co/HrOpc1a4f3 https://t.co/OZRucPGXej — PolitiTweet.org

Nate Cohn @Nate_Cohn

RT @LeusderListicle: The last heir to the Ottoman Empire lived in this Lexington Avenue walk up, which his wife still occupies and which is… — PolitiTweet.org

Nate Cohn @Nate_Cohn

@nataliemj10 @jon_m_rob @gelliottmorris @davidshor @_Jon_Green advocacy groups are exactly the kinds of political actors that i'm talking about, and i think @jon_m_rob issues with the gun control polling would certainly apply in this case — PolitiTweet.org

Nate Cohn @Nate_Cohn

@jon_m_rob @gelliottmorris @nataliemj10 @davidshor @_Jon_Green i happen to think this isn't nearly as easily severed from utility as they do: the value of elevating it is so that political actors can act on it! if they can't, then there's a problem — PolitiTweet.org

Nate Cohn @Nate_Cohn

@jon_m_rob @gelliottmorris @nataliemj10 @davidshor @_Jon_Green in my experience, the issue polling advocates don't even really think it needs to have utility--they think there's inherent, lower-case 'd' democratic value to elevating the opinion of the public in a democracy — PolitiTweet.org

Nate Cohn @Nate_Cohn

@ElectProject @jonmladd @davidshor @kimballdc (trump+1.4 points is the average result across those eight states) — PolitiTweet.org

Nate Cohn @Nate_Cohn

@ElectProject @jonmladd @davidshor @kimballdc If you kept all other states in place, and all of nv/pa/mi/wi/nc/fl/az/ga/tx voted for trump by 1.4 points, you'd have an EC-PV gap of 6 — PolitiTweet.org

Nate Cohn @Nate_Cohn

@ElectProject @jonmladd @davidshor @kimballdc depends on the magnitude of the noise, i suppose. i do think the theoretical GOP EC-PV maximum is probably when all of these tipping-point plausible states vote the exact same way (conditional on the base states remaining as they are) — PolitiTweet.org

Nate Cohn @Nate_Cohn

@ElectProject @jonmladd @davidshor @kimballdc depends on the magnitude of the noise, i suppose. it's certainly true that theoretical GOP EC-PV maximum is probably when all of these tipping-point plausible states vote the exact same way (conditional on the base states remaining as they are) — PolitiTweet.org