The reason this smacks of racism to so many minorities is because it's the same thing that every minority goes through when they succeed. If you're young and black, you're sent the message that you have to pull yourself up by your own bootstraps and succeed, and stop whining about how you're such a victim. If you do succeed, however, you've inherently victimized some cohort of white person, probably because someone gave you a leg up that they didn't have. You can never win and never succeed on your own merits, unless you happen to pull a Bill Cosby and savage your brethren for not having your success. Merit, particularly for blacks, is not a matter of what you've done, but a matter of what you're willing to let your success say about the unwashed hordes who trail behind you.
Barack Obama is suspect in the eyes of birthers because he is a prominent minority who refuses to burn the bridge behind him. Whatever his failures as a President and as a leader, he is not willing to turn his race into a badge of shame in order to curry favor.
This double bind reminds me of something I wrote awhile back on another blog (devil of time tracking that comment down), regarding the way Blacks are treated when they adopt "Washingtonian" (self-uplift) versus "Du Boisian" (moral case for equality) approaches to racial politics.
[W]hen [Du Bois] is the primary voice of the Black community, people criticize them for being insufficiently Washingtonian (why are you always demanding stuff out of the White community? Why don't you get your own house in order first -- try doing something for yourself rather than getting stuck in this dependency loop!). What we see now is a classic double bind: if Blacks are Du Boisian (trumpeting the moral case for equality), they need to be Washingtonian (solve your own problems -- stop asking so much out of Whites!); when they're Washingtonian (fine -- we'll stop looking to Whites and concentrate on self-improvement), they need to be more Du Boisian (what, you won't talk to White people anymore? Racists!).
Ta-Nehisi Coates raised similar themes with respect to why Booker T. Washington's program was such a massive failure.
The dominant logic of the post-Reconstruction era held that the real problem wasn't white racists, but carpetbaggers and meddlers from up North who'd elevated illiterate blacks above their station. The white Southerner, presumably, had no existential objection to blacks, they just didn't want to live next door to them or have an illiterate and morally degenerate population electing their politicians. To this Washington, and much of black America, said Fine. Cease fire. You let us be, we'll let you be.
In retrospect, this was a grievous error. In point of fact, whites actually did have an existential objection to black people. Their beef wasn't that illiterates and moral degenerates might get too much power. Quite the opposite. Their beef was that blacks would prove to not be illiterates and moral degenerates, and thus fully able to compete with them. To see this point illustrated, one need only look at the history of race riots in the South. When white mobs set upon black communities they didn't simply burn down the "morally degenerate" portions--they attacked the South's burgeoning black middle and working class and its institutions. They went for the churches, the schools and the businesses. It's one thing to be opposed to black amorality. It's quite another to be opposed to black progress. The lesson blacks took post-Atlanta Compromise was that whites had used the former to cover for the latter. These days, it's popular to bemoan the fact that Washington has fallen into disfavor. But it wasn't blacks who proved the Atlanta Compromise fraudulent--it was the whites of that era.
We see this outward hostility towards minority accomplishment all the time. Being a Black magna cum laude graduate of Harvard Law School (or a Latina summa cum laude graduate of Princeton) just means you were affirmative action baby. Writing a best-selling book means a White guy ghost-wrote it for you. White folks complain about Black dependency, and then complain about teaching people of color to take control of their own destiny as promoting "racial hostility" (even though the programs in question empirically raise the performance of the enrolled students).
And no, this can't be attributed to affirmative action -- it predates that program by quite some time. Marcus Garvey sarcastically defined a "negro" as "a person of dark complexion or race, who has not accomplished anything and to whom others are not obligated for any useful service." He was riffing off the anthropological trend whereby any African culture discovered to have made any substantial contributions to art, culture, politics, whatever, was suddenly not "really" Black. Du Bois made a similar observation. The problem here is exactly what Taylor lays out -- "merit", for Blacks, has nothing to do with the normal qualities of attainment (educational, political, or otherwise). That can always be attributed to someone else. "Merit" occurs when you're willing to turn on everyone else -- which is to say, it has nothing to do with merit at all and everything to do with confirming certain White ideological predispositions.