With the former, you have to keep the reader's sympathy for someone who is not not a good person. That is hard to do, most writers soften the 'bad stuff' a villainous protagonist does or goes so far into the weepy and whiny inner life of the villain that you feel like your at a My Chemical Romance* concert.
Someone who does this well is Aaron Dembski-Bowden. If you haven't heard of him yet, it's because he writers for the Black Library's Warhammer 40k imprint. For better or for worse, 'media property' writers like ADB get looked down on. In Aaron's case, that's a mistake. The guy writes very bad people** but get you unapologetically on their side.
Here's some ways he does it (and stuff I intend to steal):
1. Show what or who your villains care about.
--Your villains should not be robots, who go about their wicked way with no regard for themselves or others. The Terminator only works for a limited number of stories and frankly makes a terrible protagonist (yes, it's been tried). Show who they care about: teammates, minions, wives. Maybe they just care about being professional, which ties into my next point.
2. Show your villain being good at what they do. Part of this applies to any protagonist. I in particular, worship competence. Have your villain play his role with style, panache and daring. Above all, make them competent. That doesn't mean perfect, though. Which leads me to...
3. Make your villains flawed. Part of this is to make sure you're not making cardboard cutout or 'Mary Sue' villains that are perfect and powerful and everyone fears them, etc... But more importantly, if the villain suffers, we will identify with them. Look at Joe Abercrombie's First Law series. His whole schtick is terrible people who suffer in some way, so we're supposed to feel sorry for them.
4. Have things go wrong from them. Not just internal failures, like the above point, but have their plans go awry. Have people frustrate them, pursue them, wound them. Hitchcock did this very well, showing a cat burglar inside a house. Then we see headlights. Suddenly, we the audience, WANT the burglar to get away, to not get caught. Because we put ourselves in that person's place. Even if what they're doing is bad. Which is a reminder to...
5. Make sure your bad guy stays bad. After a period of introspection or suffering, ADB has his Night Lord evil Space Marine protagonist kill someone and flay their skin off. Seriously. I'm not saying you have to do that but keep your villain bad. Don't soften them. Readers want consistency in their characters, even the bad characters.***
6. Make whoever they're opposing, worse. This is sometimes done very clumsily by bad writers. This is not a way overtly justifying the actions of the villain or to inject 'grey' morality. That's a whole 'nother bag, often best left closed. But, if your villain's opponents aren't 'the good guys' but are people even worse than he/she is, that can be powerful stuff. See, we as readers want bad people to be punished, that impulse goes back thousands of years. Bad people having 'justice' administered to them by your villainous character will resonate with readers.
Keep these things in mind, not just for villainous protagonists but also for your traditional main character's antagonist. Now go be
*Just kidding, mostly. "The Black Parade" is a great album.
** And they are seriously bad people, especially his Night Lords novels
*** This also applies if your writing a 'redemption' character arc. Keep them villainous right up until the moment of revelation. Signal regret and the 'turn' early but keep it subtle. I could go on but this is already getting long for a footnote.