I might say: if the place I want to get to could only be...

I might say: if the place I want to get to could only be reached by way of a ladder, I would give up trying to get there. For the place I really have to get to is a place I must already be at now. Anything that I might reach by climbing a ladder does not interest me.