Understanding hate speech remains a significant challenge for both creating reliable datasets and automated hate speech detection. We hypothesize that being part of the targeted group or personally agreeing with an assertion substantially effects hate speech perception. To test these hypotheses, we create FEMHATE – a dataset containing 400 assertions that target women. These assertion are judged by female and male subjects for (i) how hateful these assertions are and (ii) for whether they agree with the assertions. We find that women and men consistently evaluate extreme cases of hate speech. We also find a strong relationship between hate speech and agreement judgments, showing that a low agreement score is a prerequisite for hate speech. We show how this relationship can be used for automatic hate speech detection. Our best system based on agreement judgments outperforms a baseline SVM classifier (equipped with ngrams) by a wide margin.