We know that prolonged exposure to a horcrux can bring out the worst in you and it affects some people more than others. So do you think that the Dursley's abusive treatment of Harry was a side effect of the horcrux or were they just horrid people that were good at hiding it?
shareHarry wasn't a proper Horcrux, so odds are being around him didn't do anything.
As for the Dursleys...the problem with them is that they're written to be these hideous 1-Dimensional caricatures because apparently the "Good-hearted Orphan Boy" with bad guardians is a British Kid's Literature staple, going back to Charles Dickens and Dumas.
Everything about them is silly and over-the-top. From their physical descriptions to their attitudes to their actions...it's all like a loony tunes show.