Referring to the conspiracy to break the society, PM Modi said, 'We have to be careful'