Lol... believe it or not this very scenario has happened in real life. I've heard of a few stories about couples that hooked up only to find out later on that they're related.
I think it might have added a certain dynamic to the movie. I mean, Luke and Leia kissed in that one scene, and this wouldn't be the only film to have incestuous overtones. I would bet anything you like that movie Back To The Future.
But then again, the R-rating would mean fewer kids could see it. Are you a kid?
Well it's been done many times in one form or another. Off the top of my head...Blue lagoon had first cousins....probably others that I'm happy to forget...
Wasn't going to happen. Lucas never intended for Luke & Leia to be a couple. When fans kept asking for it, even after she said "I love you" to Han, he changed the story and made them brother and sister.
Well, I think it's obvious that the creative team hadn't conjured up the Luke and Leia twins relationship until after Empire, otherwise she wouldn't be smooching him in Hoth. In other words: if they had gone with a sex scene and/or pregnancy with Luke and Leia, they would have gone a different route with the sister plotline.
Star Wars operates in a sort of children's adventure film world where sexuality is a little subdued. We get some kisses, but no passionate embracing. We see some scintillating costumes, but no actual nudity. We have relationships, but not really sex. It's a franchise that walks the line of adolescence between childhood and adulthood, and the approach that the films have towards sexuality is sort of part of that line-walking.
Just realised I technically never answered the main question. Yes, a sex scene between two character who are later revealed to be brother and sister would doom Star Wars to being a very uncomfortable film to watch, I think. Some franchises can get away with jarring shifts, but I think Star Wars would not have weathered that storm particularly well.
You might be right. A twist like this might not have worked. But I do wonder what a more adult oriented Star Wars would have been like. Guess we'll never know now that it's basically a Disney product.