Realism is what they did with TLOU's aiming. In addition to using a PS3 controller the devs decided to screw with my aiming just because Joel is a p#ssy. I worked hard to become a decent shot and I don't appreciated devs f#cking with that to make the game seem more realistic. Its not my fault Joel can't shoot for sh!t.
And thats one of the many reasons why I hate realism.
No, weapon swaying is not realism, quite the opposite. If you've ever aimed a gun you know your aim does not sway like that and that the lack of accuracy is due to recoil rather than breathing and heartbeat. For someone's aim to sway like at the beginning of TLoU, they'd have to be drunk, high, or have serious neurological damage.
Also not realistic is becoming more accurate by swallowing purple flowers.
In short, what frustrated you so much is not the realism, but rather the lack of realism, which is a byproduct of the artificial character progression mechanics implemented in many games today.