A man says he has become anxious and dislikes driving after he was punched in the face for driving at 20mph.