I can agree with this argument by "liberal multilateralist" Kenneth Cain in the Wall Street Journal (behind a firewall, but helpfully summarized by Powerline). Cain argues that Annan's fatal flaw is not corruption, but cowardice. Specifically, the UN's inaction in the face of clear genocide (not that they're alone) is morally shocking and ethically depraved. These failures in leadership are damning, and they necessitate Annan's ouster.
Speaking of Powerline (though this might not be the best segway), they've been named blog of the year by Time Magazine. A hearty congratulations to a truly superb blog.