This old interview with Vincent Kartheiser (Connor) shows us that Angel's season 4 set was not full of enthusiast people
https://ohnotheydidnt.livejournal.com/15242440.html
"The Scanner: Speaking of Angel, you played a fairly divisive character on that series as well—Angel’s sullen offspring Connor. Were you aware of the strong feelings that character generated?
Kartheiser: I knew people were reacting really strongly to Connor, but I didn’t know why because I didn’t know the show’s history. To me, the character lost its thrill about four episodes in. From there on out I felt like I was doing the same scene over and over and over. Every week I’d show up and have a scene with Cordelia, then Angel would show up and I’d have some sort of conflict with him. There’d be a couple of fight scenes where I’d fight with them even though I didn’t want to and then I would sulk and leave. That to me was every episode. Ultimately, they wrote him into a corner. There was nowhere for him to go. I think the majority of the fans really hated Conner and really hated me and getting me off the show was the highest priority. And I don’t blame them.
The Scanner: What’s been the biggest difference between making Angel and Mad Men?
Kartheiser: There was a real sense on Angel that people were just doing a job. The grips, the DP, even the directors would kind of just show up, do their job and go home. On Mad Men, people really love the show and it means something to them. Every DP we’ve had has worked their ass off, we see directors we won’t be working with for two weeks walking around the set setting up shots and all of the actors come in prepared. People really care about this project. Not to say we didn’t care about Angel, but there was a sense we could do no wrong. We had our audience and regardless of what we did, we were going to keep that audience. On Mad Men we also have Matthew Weiner on set all the time whereas Joss [Whedon] was hardly ever on Angel. I think Joss was doing Firefly at that point and was in love with his next project. I had a friend who filmed a few episodes in the first season of Angel and said everyone was invested and there was crazy energy, so maybe I just came into it late. I let them know right off the bat that some of the choices they were making [about Conner] were wrong. I showed up to play that character and I had a lot of ideas. And they didn’t like any of those ideas. That’s okay, I’m in the business of having my ideas rejected. But after about 7 or 8 episodes of coming in with ideas and realizing it just didn’t matter, I became really complacent and jaded and angry at the project. I felt like it wasn’t a collaboration, that the people I was working with didn’t care to take risks. Some actors were able to find risks in there. If you look at Amy Acker, she just did an incredible job; she came onto that show with a developed character and stayed with it. She took a big risk and succeeded and I admired that.
The Scanner: Was it in part because of your experience on Angel that you chose Mad Men over another sci-fi series like Heroes or Jericho?
Kartheiser: That was more with Heroes because I don’t know if Jericho is that sci-fi. I was never a fan of Buffy, I’ll say it straight out. I was never a fan of Angel. I always found it hard to say Joss’ words. And that’s not a knock on Joss. I just don’t think I ever understood the show and my performance suffered because of that. They did the right thing by removing me. Then Spike came in the next year and he gave the show the shot in the arm it needed. That’s what they hired me for and I failed. So I didn’t blame that at all for wanting to try something new. I think it was a relief for me and for them."