Chalavadi Narayanaswamy should also return site allotted to trust headed by him, says Congress