Sunan Ibn Majah, Book 25, Hadith 137 (Ibn Majah 3020)
It was narrated that ‘Aishah said:
“The Quraish said: ‘We are the neighbors of the House and we do not leave the sanctuary.’ Allah said: ‘Then depart from the place whence all the people depart.’”