Nicholas Farrell could be 'Lionel's' son.
At one point Jean says to Lionel that not all romances need to be great ones. Something along those lines anyway and Lionel quietly agrees, making Jean wonder what he meant. What was Lionel up to between his divorce in Kenya and returning to publish his book and lecture? I bet that's what she was thinking.
Well, what if unknown to Lionel one of these romances produced a son, and this son finally finds Lionel. I think Nicholas Farrell already shares a good resemblance to Geoffrey Palmer. And would play Lionels son very well. It would add a new dynamic for a season of stories. Though I doubt Geoffrey Palmer and Judi Dench are interested in doing anymore. Plus as others have stated the scripts need to be really good.
These stories would take place almost exclusively at Lionels house in the country, as in my mind Lionel and Jean would be spending more and more time there.