Elsie and Sir William
How does their affair actually work? Elsie is surely aware of Sir William's past randy habits with young girls and knows it won't work out.
Sylvia is just bitter enough with Sir William to have no problem insulting and humiliating him at his own dinner table that surely she would have taken glee in submarining his affair with Elsie.
Mrs. Parks and Mrs. Croft would have certainly had their own reasons for disrupting this affair and dismissed Elsie for such a violation as having sex with Sir William.
And such an affair would surely have been prime fodder for staff gossip if not somewhat divisive.
Now, the scullery maid seems to get a pass for her own sexual escapades, even with Mrs. Croft, but it seems like some kind of weird thing where its tolerated. Perhaps being the fat girl in the kitchen makes her promiscuity somewhat acceptable or non-threatening, a convenient tart. But Elsie is good looking and more socially capable, which would be more likely to raise the scandal factor.
So how does Elsie get away with carrying on with Sir William? The rules of the game and the multiple people with multiple motivations to derail it seem to make it at best a one-off.