Why doesn't anybody notice the building is empty?

This building is located in the downtown district, surrounded by other buildings and pedestrians.

If this building jumps every 36 hours, doesn't anybody ever hear it shake and rumble with its lights going off and on as they pass by? What if someone is just leaning against the walls or door, would they jump as well? Isn't that how Tinker jumped when he planned to blow up the building?

Also, wouldn't the city notice its empty and you know, lock it up or even demolish it, or better yet renovate it and make it a functional office building?

Don't get me wrong, I love the movie and its premise. But I just can't help notice why on some Earths the building is in the middle of a metropolis, yet it can safely and discreetly operate as a parallel universe portal without anybody noticing.