I would level Vigor plenty early on as well. It was a big meme around release, loads of people complaining on the difficulty because they were getting one shot by everything and hadnt levelled Vigor much or at all.
So far I haven't been one-shot by anything yet, but the heavy armour probably helps too.
