Amber always asserted to him that he was the only guy she...

Amber always asserted to him that he was the only guy she was sleeping with during that time period so Anthony's never had any reason to question it. Anthony's big thing is that his reputation has been harmed. This poor guy can't go anywhere without people pointing at him.