Job opportunity declining?

So I had a business call me up recently that rather surprised me. The reason, because I hadn't actually applied there. I'm assuming they got my resume from an employment agency I signed up with. I was going to apply about a year ago, until I heard this asshole I had a previous bad experience with was the new manager. So when he called, he asked if I'm still looking for work, even though I'm happily employed now, I said "yes". Then he asked if I could come for an interview in a few days. This is when I stole a line from a "Naked Gun" film, and told him "I would rather swim in raw sewage" than work for him and this company. He immediately rebutted something, but in the midst of those words, I ended the call. Did I decline his offer well?