People's arguments about time travel

My theory on why we don't travel to the past is because of the example given in the second movie. Marty gets the idea to buy the almanac so he can bet on races that he would already know the results to so he could become wealthy. Doc is against it because he didn't build the machine for personal gain. He throws it away unaware that Biff is overhearing them. Biff steals Marty's idea along with the machine and returns to the past to give it to his younger self so that his life would turn out to his advantage. It not only turns out to Biff's advantage but to the disadvantage of the town and Marty's family. The Doc now wants to destroy the time machine because they see the effect on world events in the wrong hands. Even in this original installment of the movie, Marty inadvertently almost wiped out his own existence and that of his siblings. I believe this is why we aren't allowed to travel backwards in time, not that it's impossible to do so.