At WorldCon I went to a panel about unlikeable heroes and how far can you push their behaviour to the detestable. The panelists had some trouble keeping focus on the heroes and kept slipping into discussing the villains, which I think goes a long way to illustrating our natural tendency to want heroes to be the good guys!
A hero who is a shining paragon of all virtues usually doesn’t thrill adult readers these days. We’ve come to want the complexity of the real world in our imagined ones so we want flaws at the very least.
Rogues have long been an acceptable type of not-so-shiny hero. They have often represented freedom from societies forms (as opposed to just lacking morals).
But what about those characters in whose behaviour morals are twisted or even lacking? Rogues who act out of nothing but self-interest are hard to like and can be hard to write believably because they can’t suddenly develop morals.
Somewhere between the morally faulty and the true villain you hit a two-sided quandary: Your anti-hero must act in a consistent way, but where do you stop? An evil act is enough to make many readers put a book down.
So what would make you hate a protagonist? They kill a person? The torture a person? Kill a child? Rape someone? Kill an animal? Knowingly send an army to their deaths?
Obviously the line will be different for everyone and the circumstances in the book will affect it too, but there is one unavoidable truth in writing… The reader must care about the protagonist, and that means the ‘hero’ can’t be too unlikeable.