If you stop and think about it, for some it has already happened. some of them cannot even go on a "tour" if word of their arrival is announced. It has to be totally hush-hush and/or spontaneous, and even then they leave quickly knowing the posse is forming up to confront them.
Or, at the very least, the only way they've been able to walk down the street is after their gestapo has cleared the area, and surrounds them with a wall of bodies.
Is being a billionaire worth it if your own behavior has caused you to be restricted to your (well fortified) home all the time?