Size up was too big!
So in American bike jerseys I'm typically size L. In Castelli I usually go XL. From the comments here, and my past experience, I ordered XL, and it is too big! I exchanged for size L, and while the jersey fit better in general, now the problem of the arm holes being too high is REALLY obvious. Man, what a good-looking jersey too! Just doesn't fit right. The parts I don't love about this jersey are the sleeves - that they're set in (instead of raglan) and that they're floppy at the ends like a T-shirt.
I had to change the star rating down to three because the fit problem isn't in the sizing, apparently.