Bringing Rutlers to justce issue plot hole

One thing I could never understand is the scene where they catch the three rustlers who have just killed the man's brother and father. They all refuse to ride with Clint's character even though they know it to be a virtually impossible task to take three men a three days ride to the fort. Why would they do this. The man's brother and father were brutally murdered by the men in front of him and though their swift justice is thwarted for obvious reasons, they are fine with him going alone. That just doesn't make sense. At least two of them would have gone along. Anyway there were plenty of them remaining to take the herd back. Why risk sending him off alone when you want justice knowing that in all likelihood they will get away. It just seems like a plot hole to me.

My name's Pitt. And your ass ain't talkin' your way out of this *beep*