I don't understand why the Vietnamese guards were telling the family to turn back and that they don't have permission to enter the country, but then protect and welcome them the second their boat drifted over the border...
hcheatle answered the question just fine. When the Vietnamese soldiers saw Jack's family paddling towards them, they warned them to stop. This is something they would do to anyone paddling towards the border. However, once Jack's family was officially on the Vietnam side, the country had a duty to provide them asylum.
I'm assuming that once the soldiers saw a bunch of rebels shooting at Jack's family, they figured out what was going on and let them inside the gate without issue.
What stopped them from actually shooting pre refugee status
Nothing really. Vietnam was aware their neighbor was in the middle of a nasty civil war, so if the soldiers felt threatened enough they could technically shoot Jack and his family. Jack's hands were up, though. They held their fire.
If the Vietnamese guards were to shoot at the armed mob on the other side of the border it would be an act of war. At the very least it would create a very serious international incident between the two countries.