Has anybody else noticed how he had an occasional habit of referring to shared possessions as "my" rather than "our"?
E.g. "This was my house!" and "If they've lost my car, that's it!" both from the second series.
That's not surprising, considering that Victor always speaks HIS mind and delivers HIS opinions to everyone and sundry. He never speaks for Margaret, only for himself. She can look after herself very well, thank you.
The man is self-centered and we have to accept him as such. It's in his character.