Is freedom of speech bad?

Whilst people can misuse free speech, if you restrict it then who gets the power to decide what speech is acceptable and what isn't?