Most developing countries start with textiles, just as England did 200 years ago. But Cambodia's garment trade is incredibly dependent on special treatment from America, where it sells almost all its wares. Since the expiration of the Multi-Fiber Agreement in 2004, which imposed quotas on textiles in the developing world, countries like China and Vietnam have been subject to special, stopgap measures to dampen down the flood of textiles they can pour into Western markets. In exchange for enacting higher labor standards (and also for being really small and poor), Cambodia has been exempted from this treatment.
In theory, this is a bad thing; trade should go where the market dictates. In practice, it's hard to criticize something which is pulling a lot of very poor people into decent-paying jobs. And Cambodia's 14 million population can hardly be said to be doing serious damage to China, or even Vietnam.
I think Megan is right -- Cambodia needs a few more years still to get on its feet, build up its infrastructure, and diversify its economy (like Vietnam is beginning to do). Pull the rug out from underneath it now, and it will get completely swamped by the greater economic heft of its neighbors, and might never recover.