What did his friend see that made him leave?

He just got the vibe that something is really off with Jeff. He's alone in that house, barely 18, obviously self medicating. Not the weird attention getting Dahmer anymore but more like he can tell the wheels have come off the wagon. He knew something was really wrong with him and that he needed to not be alone with him.

In addition to Jimmy's answer, the whole bad vibe was added to by the fact that the home was partly emptied out by the mom taking most of her stuff when she moved out with the younger brother. The house just looked all abandoned and there was Jeff living in it anyway. It just looked bad enough that the friend realized something is totally wrong with Jeff.

His friends were unreasonable

What Jimmy said is accurate. To add to that though, there was an indicator even before the two went inside the home.

When Derf and Dahmer arrive at the home and park outside, Derf notices blood on Dahmer’s fingers. Derf asks him if it’s blood (which it obviously is) and Dahmer says “it’s paint”. Derf knows Dahmer is lying (you can tell by his reaction). He gets seriously freaked out. Dahmer invited Derf inside and Derf refuses twice. Derf senses something is wrong and I believe he agreed to Dahmer’s invitation after that as a way to appease him and lighten the mood. Because Derf sensed something wasn’t right and might have not wanted to make Dahmer mad.

Once inside the home, Derf looks around and noticed the home is partially empty, it’s dark, it’s quiet and something feels very off. Derf can sense it. He decides to get out of there because he listens to his instinct.