LOL at the responses in this thread. Of course this was a gay reference. Hitchcock had tons of gay references in his movies, and many of the men were gay or implied to have problems with their masculinity. For example, in North by Northwest, Martin Landau's character talks about having "women's intuition," to suggest he is gay. There is also no question that the crazy stalker in Strangers on a Train was gay, or the men in Rope. In Frenzy, the murderer is a foppish dandy, and Norman Bates a cross dresser. People are being really obtuse about this.
So, yes, the joke is what it is. Back in the day, a common joke was that gay men were effeminate and would do things like carry purses. That is why the inspector says that if the guy walks around with a purse, he will be arrested. Law enforcement will assume he is gay and then arrest him.
Good points above about LGBTQ+ characters being hinted at to various levels in Hitchcock's movies.
I found the scene in Dial M really funny, really caught me off guard and I burst out laughing ...like a Carry On Movie or 1970s sitcom type of UK joke. Yes it was clearly a joke about looking gay with a little handbag hanging off his wrist.... But also some truth in that homosexuality was illegal at that time in the UK, to me Hitchcock was actually making a joke about the stupidity of that law, that's how it felt to me.
I had not thought about the Frenzy guy possibly being gay ('foppish dandy') before, as he seemed more of a cockney Jack the Lad type to me but it's made me think about him using Ties as his weapon... Women barely wore ties, Men did.... Maybe it was him turning the women into men in his fantasy, etc