What was the significance of Joel wishing...*spoilers*

...that he had stayed with Clem at the abandoned house on the day they met, instead of abruptly walking out on her? During the final memory-erasing scene at the abandoned house (the day they met), Joel tells Clem, "I wished I had stayed." The way he says it is very profound, yet they still ended up in a two-year relationship.

Is it being implied that the reason for their initial break-up can be traced to the day they met? If so, why would Joel staying with her at the abandoned house upon their very first meeting necessarily change their fate together? If it was his relationship with Naomi that made him hesitant, we know that this wasn't an issue for Clem as long as she didn't have to "tip-toe" around it.