Perceivers use visual information to categorize others into social groups. That said, anecdotal reports suggest that perceivers are more comfortable making some categorizations (race, sex) than others (sexual orientation) on the basis of such limited information, perhaps because they hold differing beliefs about the diagnosticity of visual cues to those categories. The current studies tested this hypothesis empirically. We first developed a new measure—the Diagnosticity Scale—to assess beliefs about the diagnosticity of visual cues to diverse social categories. Next, we demonstrated that diagnosticity beliefs explain response tendencies in social perception, such that weak beliefs in the diagnosticity of visual cues to a given category predict biases toward the non-stigmatized, default response option. Collectively, these studies introduce the Diagnosticity Scale as a valid measure of perceivers’ beliefs in visual cues to social categories, which help to explain some noteworthy biases in social perception.