The prime minister's remarks come amid ongoing protests by thousands of farmers, mostly from Haryana, Punjab and Uttar Pradesh, at several border points in Delhi.
Television visuals showed that while a protest meeting was being held in front of the office, some of the activists had climbed the building walls and rolled down the flag.