Over the Top Comedy Killed This Show
Don't get me wrong, I really enjoyed this show and the premise from the movie is excellent, I just think they squandered it by changing the direction of the show from being mostly serious to mostly comedic, and not just funny but lampoonish in the '60s Batman TV series way. Much of the comedy, while goofy, was genuinely funny, but it needed to be toned down and less frequent for the show to be taken seriously.
Had they kept the tone of the first few episodes for the rest of the season, we'd probably be watching season two now. I suspect many viewers began tuning out when they went the Barney the dinosaur route, which was Brian's imaginary friend during the serial killer episode. It was too much, too fast. Maybe 3-4 years down the line they could've gotten away with it but not in the initial season when they're just trying to establish their fundamentals and their fan-base. Also, episodes like that need to be one off, not the norm, that makes it too weird and quirky for most folks.
It's really too bad because the cast was solid and story-line was well developed. Also I think they failed to develop the potential of the pill somewhat, I think that's what drew people in to begin with but tuned out when it didn't really go in that direction.