What exactly did Sy do that was so terrible???

I haven't seen the movie for a long time, but I do know it well. I think Sy, as a child, was forced to pose for child pornography pictures, which both affected him deeply and also led to his fascination with photography later on.

Tell you what, Foster, I'll let Sy explain in his own words what I mean:
https://www.youtube.com/watch?v=OdU3vEh3qfs

That sounds like being an extremely unwilling participant in child pornography to me. Poor Sy.

Holding people at knife point and making demands is highly illegal. Now, while the man and woman were going to engage in sexual conduct, they didn't want Sy to be a participant, nor did they want photographs being taken, which would make the situation sexual assault. Finally, he smacked the woman in the head with the door as he came in, so he had a decent amount of crimes.

I thought the whole take would be that they can't prove anything, he didn't take pictures of them, he only took pictures of his room and toilet. All would be circumstantial and you say vs I say.

They never really said what happened to Sy at the end, just showed him being interrogated and giving the reasons why he did the things he did because he was abused as a boy, but did he go to prison or a mental hospital like you said??

It's much more than that. Everything he did to them is a crime; assault, battery, kidnapping.. he'd go to jail and then an institution most likely.

He dropped a deuce in their toilet in their house and didn't flush.