Mr Karzai says the special forces would have to withdraw from Wardak within two weeks because armed Afghan groups they had set up were fuelling "insecurity and instability".
It is another blow to US-led forces, coming the same month Mr Karzai issued a decree ordering an end to local security forces calling in NATO air strikes.
That came amid new tensions over civilian casualties.
Mr Karzai made the decision about Wardak in a meeting of the national security council, said presidential spokesman Aimal Faizi.
"The US special forces and illegal armed groups created by them are causing insecurity, instability, and harass local people in this province," he told a press conference.
He said armed individuals "named as US special forces" in Wardak "engage in harassing, annoying, torturing and even murdering innocent people," according to a statement from the presidential office.
The statement cited a recent example in which which nine people disappeared in an operation conducted by this "suspicious force".
In another incident, a student was taken away at night from his home, and his tortured body was found with its throat cut two days later under a bridge, the statement said.