I don't understand why the priest helped kill the guests for Mary. I know he's her uncle but I don't know why he's a sociopath. Also, I don't know where that dog come from, it seems to come out of left field.
shareThe rottweiler was owned and trained by the priest (since Mary wouldn't be able to own one in the asylum). I suspect he was the biological father of the twin girls and took Mary's side in the eventual split of the two daughters. He was angry at Julia for forsaking her family, as observed in his sermon. Being a Roman priest and therefore sworn to celibacy, he had to hide the fact that he was their true father, but the duplicity drove him mad, apparently.
Another reason he was on crazy Mary's side, is because Julia's dream at the beginning suggests that she was the one who permanently disfigured Mary's face when they were kids to get back at her for being an abusive "master" (remember, she described her relationship with Mary as that of slave and master respectively). Mary was subsequently taken to the institution and Julia blocked out that she disfigured her face in a rage. So, when Father James (the girls' literal father) meets Julia at the hospital to finally see Mary after like fifteen years, he lies to her about why her face is disfigured.