A passenger bus skidded off a cliff in Sri Lanka killing at least 21 people and injuring 35 others, a police spokesman said.