The TV announcers said it. The radio announcers said it. What they said was that kicking off the dirt infield is difficult or precarious. And kickers often have trouble with it.
I thought about that for a bit, and realized that I have not seen any kickers have "trouble" with it, with one exception. In fact, I don't believe that they are statistically less likely to miss from the dirt than they are from a comparable distance from the other end.
The one exception would be Olindo Mare, who *always* had trouble on the dirt. And on the grass. And with shoes on.