Was it to symbolize that he had broken his marriage and relationship with his wife?
They show it twice and it's made to be significant but there's no pay-off with it.
1. His dummy brother lived there once.
2. He hit a baseball through it
3. He likes to stare at broken windows
4. Reason: probably on the cutting room floor
I saw it as a metaphor for his broken dream to become a professional baseball player.
Probably symbolism. Also, the house is broken down and abandoned.
shareIt was the house next door, right? If it was, it suggests that no matter how much he has accomplished and how hard he's worked, that he's never been able to escape living in the midst of poverty and decay. He's building a fence to separate his house from the abandoned one next door, but that house's decrepitude and abandonment is a constant reminder that despite all the grand poses he strikes, he's still living in a neighborhood on it's way to becoming a ghetto.
sharegood analogy CineasteWest