The photograph he had been looking at had "Look here" written on it in the area in the basement next to the sled. Since there was nothing there it must have occurred to him that the clue had been painted over. Maybe there was evidence that part of the wall had been painted over, but I couldn't tell that from my TV. Or maybe we just take a leap of faith in believing that removing paint on the wall made sense. It never would have occurred to me but I wasn't in the movie. :)
Just be truthful and if you can fake that you've got it made. ;)
I'm watching the Blu-Ray, and I was confused by the whole scene at first, but having read your questions, I took another look.
When he first looks at the photo, I couldn't figure out what I was looking at, because it didn't match what we were being shown of that corner of the room... because there's a big scribble scratched into the photo in the middle of that section of wall where the word "Azazel" had been...