The issue is that it becomes a question of wealth. If you're incredibly wealthy, there will always be opportunities to live elsewhere relatively easily. What the EU has done is allowed a lot of people who would not otherwise have been able to to live and work in Europe, and Spain in particular has become an attractive destination for retirees specifically because the cost of living is relatively cheap and because your healthcare is taken ff by reciprocal agreements amongst EU countries.
If these people are well off enough that they'd be immune to the consequences of leaving the EU then they're mean spirited people who would deny the lifestyle they have chosen to others, and if they are not then they're idiots. It's quite possible that they are both.