Jami` at-Tirmidhi, Book 8, Hadith 108 (Tirmidhi 789)

Aishah narrated that:

The Messenger of Allah said: "Whoever stays with a people, then he is not to fast without their permission."