I'm sure it's a failing on my part, but I've never understood the appeal of the argument that poor children should be denied opportunities to escape (via vouchers, home-schooling, etc.) failing and failure-generating public schools because the departure of some kids would make things even worse for those kids who stayed behind. The argument is particularly tough to take (for me) coming from persons who would never send (and are not forced by circumstance to send) their own children to the schools in question.
With regards to vouchers specifically, I too share an aversion to certain liberals who seem to oppose them reflexively because they'll "weaken" public schools. Even the church/state argument I don't find compelling, and I consider myself pretty zealous on the issue. I think vouchers should be evaluated along the very simple paradigm of "will they fix the problem"? And I'm not sure they will. Awhile back, US News ran a feature article on school vouchers, and they raised a number of significant points that I think have been under-represented in the debate. First, even with voucher money, many poor families still wouldn't be able to afford many private schools. Second, and more importantly though, private schools are under no obligation to accept kids in the voucher program. Where are all the slots coming from to accommodate all these children? I don't think that increased demand would necessarily create more private schools, since the market price is effectively capped at the voucher level plus some (presumably small) amount of discretionary income the poor can spend on education. So the solution, to me, lacks cohesiveness.
But on to the meta-point of Garnett's question--which I think really is an attack on those who would prevent individuals from escaping poor schools because the overall community would suffer. Basically, an individualist critique. I think that an excellent engagement with that line of thinking is done by Charles Lawrence III in his article Forbidden Conversations: On Race, Privacy, and Community (A Continuing Conversation with John Ely on Racism and Democracy), 114 Yale L.J. 1353 (2005) [PDF]. Like Garnett, I am uncomfortable when wealthy upper-class whites say that minority families should stay in inner-city schools, so it's worth noting that Lawrence is black and his children attend D.C. public schools (as he reveals in the article). I think that he raises many points which would be of interest to Garnett. Based on my vague recollections of the article (I don't have time to re-read it now--so some of this may just be my own musings), he thinks that we should view schools as a community issue, rather than just a collection of individuals acting as education consumers. In a school choice model, a community that has (say) 10 school age children might see them all attend different schools (or be home-schooled). This may be somewhat appealing because we like a breadth of choice. But I think we also lose something in such a situation. Education isn't just textbooks and word problems. I do believe it is some way intricately connected in a community of learning, an environment conducive to intellectual and personal development, both inside and outside school walls. When what was a cohesive community splinters of into dozens of fragmented individuals, those bonds are lost, and I think that students will suffer for it.
Lawrence says that instead of individualist solutions, we should look toward collective proposals that will both strengthen the community and rebuild the schools themselves. For example, he proposes that we extend affirmative action benefits to students of any race who attend integrated schools, to discourage white flight (one of the primary causes of inner-city school attrition). Presumably, he would also support endeavors like local tutoring organizations, daycare, and community-based academic resources, to cultivate a healthy academic community rather than focusing on particular persons. I am not hostile to individualism by any stretch, but I think schools are a perfect example of where its better to build bridges rather than break bonds. I highly recommend Lawrence's article, and think that it raises some interesting positions for our education debate.
UPDATE: Shavar Jefferies also takes up the issue in favor of "school choice" and more explicitly within a racial perspective. My answer remains the same. At the end of the day, even when they work as planned vouchers do little to aid community-building endeavors and more often actively harm them. That our system has not been integrated, that Brown's promise has been breached, is indeed an overlooked and severe problem. However, is not the answer a proactive effort to incorporate these communal values into the schools, rather than engaging in the fracturing and atomizing extreme individualism that only drives people further apart? We can't fix this problem alone, and I fear that a "school choice" paradigm erodes the bonds of trust that are our only hope of overcoming our segregated system.