Why do they keep referring to Mattie as ugly?

I read the book years ago after watching the John Wayne version, and I do think they refer to her as plain and homely. And you do get a couple mentions of that in the original film version.

My guess is the Coen Bros. simply wrote that into the script before casting any actress. Also, keep in mind, sometimes people do not fully believe the insult they're using. They just want to use something that they think will be hurtful -- to shoo a pest away -- and so rush to the most brazen and extreme word they believe will do the job. Here, they imagined that saying "ugly" to a young lady would immediately get Mattie off their backs.

What I like in the John Wayne version is the girl didn't even care that she was called unattractive. She just kept on smiling, with her eyes completely set on her objective.

It's a tradition in American movies going back to silent films. You usually have to suspend your disbelief when you are told any leading female character is unattractive.

she's not pretty, amazing actress though

She is not pretty but she's not ugly either just plain. I guess I just don't find her attractive.