Couldn't he have just told her they were his wife's kids
...from a previous marriage?
As far as we know he never told Palmer how long he and his wife had been married exactly, he could have just told her those kids weren't his, just his wife's. It would have solved the problem quickly right then and there.