Elderly farmer threatened by council after knocking down ‘historic’ wall he built himself