I am 5'7" and am always asking myself the same question as I sit right between a small and a medium. I am pretty certain that moving forward I will be sizing down. I like the more upright and neutral position as I like bikes to be a bit more playful with less horsepower. I can see if I was an endurance athlete appreciating the longer top tube, but that ain't my jam.
Body proportions, riding style, and overall cycling fitness play a role in the way you want your bike to fit. And the lengths and angles between bikes vary, so every once in a while I find myself on a small that makes me wish for a set back seat post, but not very often.
To sum it up, I support your notion to size down. The bits and pieces can always change, but the frame is fairly permanent.