Sunan Abi Dawud, Book 2, Hadith 74 (Abu Dawud 464)

Nafi said :

‘Umar b. al-Khattab used to prohibit (men) to enter through the door reserved for women.