A scene that might have helped.
What I think might have cleared up a few things would have been a flashback or two of Frank Miller in Hadleyville. We would have gotten a sense of just how bad a man Miller was. As good as the movie is, we just don't really get a glimpse of how much Miller influenced the town. For both the good and bad people.
"I'm in such bad shape, I'm wearing prescription underwear." Phyllis Diller 1917-2012