The problem is that Wieseltier's argument is not "baseless", at least in the sense that it isn't supported by clearly-stated warrants. He's making pretty specific claims about what demands Sullivan makes of Jews, and how they fit into a broader history and structure of power that Christians possess over Jews (and majorities tend to possess over minorities). One could disagree with those claims on the substance, but what Yglesias is relying on is a particular definition of when something is legitimately labeled anti-Semitism:
If you call anti-semites anti-semites, then people who aren’t motivated by anti-Jewish racism will figure “hey, since my political opinions aren’t motivated by anti-Jewish racism, then I’m safe.” The idea is to put everyone on notice that mere innocence will be no defense.
And y'all know what my objection here is. Intent is not a magical ward guarding its wielders from doing racist or bigoted things. You can mean perfectly well, consider oneself to be a totally fair and egalitarian person, and still hold positions or beliefs that are marginalizing or discriminatory towards particular groups. Being "motivated by anti-Jewish racism" is not the totality of what makes something potentially anti-Semitic, and if the threshold for non-abusive discussion of anti-Semitism is getting the defendant to admit "I'm an evil bigot who hates Jews", then we're just trying to suppress any critical discussion of anti-Semitism under the guise of preventing its "baseless" deployment. It is not a good thing when everyone who isn't a Klan member gets to claim immunity from grappling with ingrained racism and anti-Semitism. It simply allows people to say that nothing is ever anti-Semitic.
What does Sullivan feel about Jews? Like Wieseltier, who writes "it is impossible to know what is in a man’s heart", I have no way of knowing. But I suspect that he considers himself a friend of the Jewish people, committed to them flourishing and living in peace and security. That's great. It's also entirely consistent with holding policy prescriptions, stereotypes, prejudices, and other attitudes that act to keep Jews in an unequal status and deny them the ability to flourish, live as equals, and enjoy peace and security. The measure of a true friend and ally isn't what's in their heart, because racism isn't fundamentally "in the heart".* It's whether they are willing to go beyond that bare assertion of friendship and dig deep when they are told "you're hurting me".
* See Tommie Shelby, "Is Racism in the 'Heart'?" 33 Journal of Social Philosophy 411 (2002).
UPDATE: I think there is a fair tactical review of the upshot of Wieseltier's piece. I still maintain, though, that restricting the scope of anti-Semitism to evil Klansman cackling in their robes is wrong on essentially every level, and buying into that framework guarantees an increasingly impoverished discourse on the subject that will continue to marginalize Jewish lives.