This movie is NOT set over a period of some 20 years
Many people I know have said that. But between the time he first arrives at French Guiana and the first 2 year stint in solitary, it's only a few weeks. Then as soon as he's out of solitary and the infirmary, it's only a week or two before he makes his second escape. Between the time he escapes a second time and his recapture, it's only maybe a few months. Then he's put in 5 years. It's just that the prison and solitary life aged him prematurely.
BTW, I could never find out why the native people just got up and left one day and left him the pearls!