We do not worry about being respected in towns through which...

We do not worry about being respected in towns through which we pass. But if we are going to remain in one for a certain time, we do worry. How long does this time have to be?