The only thing I can think of is that the timeline doesn't match up. It's not as though he was born 9 months after their encounter in 1955.
Marty is the youngest, so the infidelity would have had to come later on. Lorraine would have had to sneak away to reconnect with the guy they had never seen again, all for their third child.
The "new" family seemed to be well adjusted with strong relationships, so an extramarital affair may not have been a consideration or at least one that was easily dismissed had the suspicion arisen.
Just to add that a long time had passed since George saw Calvin, and he only knew him for two weeks. I think back to some friends I had in high school for four years and haven't seen since, and I realize I don't remember them all that well (I'm 64 so older than George).
Further, Marty wasn't born with Calvin's face, he grew into it over the years so Marty's face was his own as he grew into it, not Calvin's. If Marty popped out looking exactly like teenage Calvin, that might have triggered George's memory.
Perhaps but I did mean at a later point in life since Marty was born several years later and it was more along the lines of George looking at a teenage Marty and it triggering the memory ... Not at the time of birth. (Hence the 'ends up looking')
Seamus McFly (Georges ancestor) looks like Marty, so maybe there was an old family photo of him and George thought nothing about it other than Marty looks like a McFly.