Sirgod, Ravok and Max...
Based on what I know about evolution (kind of like a hobby of mine) high protein diets (meat) are needed mostly for intelligence to fuel large brains...not for overall size. At least in mammals and it appears in dinosaurs.
The majority of "super sized" beasts were veggies...but really dumb. Baleen style Whales are omnivorous since all that goop they suck in is both plant and animal. -Size in super predators is directly related to the size of the game (availability) they use for food, modified by their style of hunting (as in hunting in social groups). In general primates have rarely attained anything large mostly because we are social and omniverous.
I am surprised about the size of Homo Erectus. It is actually still unproven that we came from their stock at all since many human-like hominids coexisted and we may not have found them all.
I don't know why the Romans were so small. Maybe it was the lead. Maybe not. Jeeze if the average was 5'1 then there must have been a whole mess of dudes under 5 feet...sorta like an army of dwarves or 8th graders.
I too wonder about size differentials in peoples especially over the generations. It seems to me size can change rapidly from one generation to the next and I don't know whether it is purely diet based (scarcity vs post scarcity) or some genetic "spring" that might be a function of adaptability. Another words were there is greater availability of food the species might ramp up the height just to consume it as did super predators would do to consume their larger prey. In this scenario genes would be the controlling factor in size, not diet. Hence food availability does not directly effect size; adaptability (and the genetic sub works) does. A subtle difference perhaps.
Just a theory though.
Also that might not obviate diet directly effecting sub size. Stunting and dwarfism but on the scale we are talking about it would seem unlikely. Roman civilization was pretty advanced so it couldn't be scarcity. maybe they simply didn't eat a lot ( a cultural thing perhaps) so they didn't trigger the "spring".
But our society is based on "consumerism". We eat a lot.
hmmmmmm