Local leaders in Redgranite say people from out of town are causing problems at the village's quarry.