It is well established that low-level visual features affect person categorization in a bottom-up fashion. Few studies have examined top-down influences, however, and have largely focused on how information recalled from memory or from motivation influences categorization. Here, we investigated how race categorizations are affected by the context in which targets are perceived by manipulating semantic information associated with the faces being categorized. We found that presenting faces that systematically varied in racial ambiguity with race-congruent (vs. incongruent) semantic labels shifted the threshold at which perceivers distinguished between racial groups. The semantic information offered by the labels therefore appeared to influence the categorization of race. These findings suggest that semantic information creates a context for the interpretation of perceptual cues during social categorization, highlighting an active role of top-down information in race perception.