Matt Yglesias pulls out some interesting poll results: Both Black and White Americans significantly overestimate the number of Black people there are in America.
As Matt says, segregation can explain half of this phenomenon: Blacks who are disproportionately housed in segregated locales would understandably overstate the proportion of the Black population.
But what about Whites? Residential segregation would predict that they would understate the Black population. Yet they go over as well. What gives?
Unfortunately, my first thought by way of explanation is the "horde of brown people" hypothesis: Whites view Blacks as threatening, and build that threat into to something sufficiently large and scary to mobilize against. This is actually made easier by the fact of segregation: one can construct the image of a vast, teeming army of Black folks "over there", in "that part of town", which one never actually visits or hears about outside the crime section of the local paper.
Slightly more benignly, the psychological view that Blacks are everywhere makes it easier to dismiss claims that they're a vulnerable minority. In this view, the need to justify continued racial inequality is the tail wagging the dog: the more Blacks can be said to be mainstreamed, the less that needs to be done to account for their actual differential social standing.
Finally, because White people are the norm in American society, Blacks standout and are likely to be more memorable as a result; Whites misinterpret that to think that there are more Black people than there are, rather than correctly surmising that they're just less likely to notice all the White folks floating around their lives.