An oft-heard argument against affirmative action is that the use of racialized preferences ends up hurting the beneficiaries because other people then question whether they're "really" qualified for the positions they've received, or are just "affirmative action hires." Even people who might have earned it "on their own" get caught up in this mistrustful sentiment, and by ultimately impugning the abilities of Black people everywhere, it shows that affirmative action, despite its good intentions, ultimately is harmful to the interests of Blacks.
This argument can come in several forms, and some would argue that even though the sentiment isn't just, it's there, and hence pragmatically affirmative action causes more problems than it solves. Unfair as that might be, it is a sentiment that needs to be grappled with in reality, not wished away by utopian hopes. This argument I think is at least a reasonable one, and though I don't think it ultimately shows that we should abolish affirmative action, it is one that needs to be addressed on its terms.
Another form, however, is to argue that the view really is just, and not racist. Blacks really are receiving a preference, this really does raise legitimate questions about their "actual" qualifications, which are masked by the use of preferences, and thus it is perfectly rational to have suspicions about the true ability of Black people hired in affirmative action practicing institutions.
This argument, to be blunt, is bogus. Even if we accept some rather shaky underlying assumptions that it rests on: that there is a stable, objective entity known as "merit" which can be measured, and affirmative action (or any sort of preference that is not explicitly "merit" based) means giving less meritorious applicants positions over more-qualified individuals, it would still be true that the way this argument plays out it racist.
How do I know? Simple. Nobody mistrusts the qualifications of White people even when there is significant evidence that they received their positions through non-merit based channels -- even and including race-based preference. The argument is only applied against Black people, and when a facially neutral argument only is applied in a manner harmful to Blacks, that's racism.
Now, I could argue that Whites today often gain significant advantages by virtue of White privilege that undoubtedly have some bearing on their current position, job, or class level. And I think that's true, and I think it's pretty clear that nobody wonders aloud about whether all White folks are "really" qualified for the positions they hold. But I don't even need to focus on the present, because we have a far more slam-dunk case: how White achievement was treated at the cusp of the civil rights movement.
Nobody, presumably, would disagree that prior to the civil rights movement, Whites had race-based "preferences" that gave them advantages in the workplace and society. And few would disagree (hopefully) that those preferences were immoral. But even during the civil rights era, there was not a broad based acceptance that individual Whites might not "deserve" to be where they are -- in their job, in their schools, in the neighborhoods, in their class, in their seniority rankings -- whatever. Insofar as the average White was referenced, it was to posit their "innocence"; that they hadn't done anything wrong, so all they had accomplished was perfectly meritorious. This belief was then used to block many integration efforts during the civil rights movement, as Blacks, paradoxically, were seen as taking positions from Whites who had "earned" them despite the fact that they clearly had been treated preferentially. That's why, from a personnel standpoint, very little changed after the civil rights "revolution". We often act like America had the equivalent of an oil change" after 1965, taking out our racist citizens and replacing them with purely color-blind people. But obviously, that didn't happen -- the same Whites who were benefiting from preferential treatment in 1960 had, by and large, their same jobs and same positions in 1970, and nobody considered asking whether or not they truly earned it.
So even though we concede those people got preferential treatment, and even though (unlike affirmative action) we universally agree that treatment was wrong, we still don't tell ourselves that their qualifications are suspect. Even retrospectively, we don't look back and wonder about whether they truly were the best men and women for their jobs. We don't ask ourselves whether our first 35 or so Presidents really were "qualified" for office. We don't ask whether or not the entire White middle class "earned" its place there. Clearly, preferential treatment does not have to entail mistrust about qualifications -- and it never has for any White person in the history of America. In fact, despite the theoretical neutrality of the argument, it is only applied against Blacks. That's racism, no matter how you slice it.