If you pay points up front, it's harder to get your money...

If you pay points up front, it's harder to get your money back. When rates are high, borrowers have to pay points to trim rates any way they can, but with rates so low there is really no need to pay those points.