Allen is also pretty douchey in this film. As in real life.
Just look at all his behavior. He refuses to earn money to support himself. No wonder his wife cuts him off sexually. He then deliberately sabatoges his own paying job. While trying to cheat on his wife.
He's basically a childish, self-indulgent, selfish infant. Like the actual man who chose to sleep with his (essentially) adopted daughter.
So he's not really an innocent hero either.