Billie is Just Not that Likable
Don't get me wrong, Broderick Crawford is a complete and utter jerk in this movie and treats Billie with contempt and dismissiveness, but one wonders why she is with him in the first place. They aren't married and it's not like he saved her from a life on the streets - evidently she was working in the chorus and even had some lines!
So being that there is not one nice word that comes out of his mouth, we know the only reason Billie is with him is because of his money - the clothes, jewels and travel he provides her. She's using him just like he's using her.
And then even as she becomes marginally more cultured and educated, she really doesn't become more likable. She lords it over everyone that she knows things now.
I'm betting she and Holden get divorced within a year. :)