Why was Alex grinning?

I don't know, because she was a sick bitch?

Because he didn't hurt/kill her. In the bathroom scene, she told Beth "he tried to say goodbye to me last night, but he couldn't." That was her warped interpretation of him putting the knife down instead of killing her (he let her live, therefore he still wanted her in his life).

I think this has something to do with the original ending. In the original ending, Alex committed suicide following this confrontation with Dan, using the same knife that Dan was holding, with the intention of framing Dan for "murder" as his fingerprints were on the knife. I think the grin she gives represents this idea entering her mind, but the ending was unpopular with test audiences, so they changed it to the ending we have now, but the grinning scene remained.

Now the grin can be interpreted however you want it.