"We can't afford to live here" is exactly right. Normal folks know exactly what he's talking about. They feel it, even if they don't express it--other than with their U-haul vans.
It's an issue that either Patrick or Healy is going to seize upon before the campaign is over. The smart one will do so first.