Interdisciplinary arbitrage and an AI-assisted research paper (paper 2 of n)

Another quick peak into a project I can lay to rest (but still report)

In this post, I test the impacts of a small insight into the rural v urban increase in polarization, describe a key advantage of interdisciplinary research, and highlight another RA vs. AI project task.

The brief backstory—I was listening to an episode of Ezra Klein with Suzanne Mettler about her new book on increasing polarization and the rural-urban divide. And she provided some stark and interesting facts that, before 1990 or so, there basically was no urban-rural (by county) partisan divide, but now there is ~20 point difference. That sounds (and is) quite big and seemed interesting enough to dig into more.

The insight I had while listening to this interview was—some places become urban over time, and not a random set of places. Obviously it’s the ones that grow, and these places likely differ from those that stay rural or stay urban.

So how much of a role does this play in changes in rural vs urban voting patterns?

In earlier times, I’d consider handing this project idea over to an RA as an “exercise” in building up a paper, especially an RA that cares a little about either rurality or voting. But many of these project ideas do not have a good RA match and then accumulate in a massive and growing “Future Projects” list. But now…AI to the rescue?

So, a pretty simple idea. And one that lets me introduce another fun concept: an interdisciplinary arbitrage opportunity.

That is—discipline A (demography) understands X (rurality/population change) more than discipline B (political science), who understand Y (voting/partisanship) more than A. So when B combines X and Y in an analysis, they open up an opportunity for A to show something about Y without really understanding Y that much, aka “interdisciplinary arbitrage”.

For the non economists: arbitrage opportunities are about informational asymmetries—one side knows more than the other. The typical examples would be stocks or currency.
Here, for interdisciplinary arbitrage: demography knows more about population change and political science knows more about polarization. But the book shows what political science knows about polarization—that is, the demographer is now “informed”. But the political scientist (likely) didn’t collaborate with a demographer in order to be on the frontier of knowledge about population change, so there is an asymmetry to be exploited by the demographer—build on the political science frontier in its expertise in polarization but add-on the frontier in expertise on population change.

Image note: this gets the gist but is not perfect—I liked the AI’s idea of the payoff though ($$).

The other way around would be a demography paper that is trying to say something interesting about population change and polarization but without really understanding polarization that well. The demographer published a paper (thus, informing the political scientists about the frontier in pollution change) that the political scientist can build on by strengthening the polarization analysis.

Before moving on, I wanted to credit this “arbitrage” label to Ichiro Kawachi. I very vividly recall him talking about these arbitrage opportunities being THE KEY advantage of pursuing interdisciplinary scholarship— in the context of the (sadly now defunct) Robert Wood Johnson Foundation Health & Society Program (RIP), including at Columbia where I was in cohort ~10.

******
So, back to the project. I started it in late October using ChatGPT and got stuck with the available tools at the time. I orchestrated some agents and they helped create a report linked here (note it was completed Oct 25, the podcast was October 21).

It was fine, but not as automated as I wanted it to be, and the process included a lot of fighting with AI to get it to show what I wanted and how I wanted it shown. And it clearly was not going to be easy to create a publishable version of it without me being very involved in the minutia. So, I moved on and archived it (put it back on the long list).

But then, as I got more interested in new skills packages (and using Cursor/Claude), I dipped back into the project to try to get more end-to-end AI assistance. Pedro’s Claude skills are really great! The final version is here as a preprint. I could have done more iterating (e.g. appendix was ~complete but still a mess), but it was getting expensive to fine-tune further: MC>MB.

I decided to not submit it to a journal because the results were not as stark as I thought they might be. The changing nature of whether places are defined as rural/urban contributes maybe 10% of what we see in polarization over time. Indeed, it contributes to (masked) higher polarization than typical accounts would show because places that grow and are no longer rural (and classified as non-rural) are ‘in the middle’ in republican vote share between rural and urban places. So, as they shift to be non-rural, they pull down (‘mask’) measures of growing polarization.

My guess is there is a publishable paper here that would be a little better than the pre-print but would require a lift to create by verifying the analysis and digging in further. But I can now cross this project off the “to do eventually” list and move on.

Lots of projects forthcoming….stay tuned.