About 132 protesters were arrested after the group removed razor wire and attempted to remove concrete barriers near government house.
Rally organisers asked the crowd to calm down, but said the rally can't end because of the arrest of innocent people.
Thai police fired tear gas at the protesters as tensions flared in the Thai capital, the scene of several outbreaks of violent unrest in recent years.
About 17,000 police were deployed for the rally, organised by the royalist group Pitak Siam, which wants Prime Minister Yingluck Shinawatra's government to step down.
The authorities expected tens of thousands of people to attend the demonstration, the first major street protest against Yingluck's 16-month-old administration.
By early afternoon police estimated that about 12,000 people had turned out.
"In the name of Pitak Siam and its allies I promise that we will topple this government," the movement's head, retired general Boonlert Kaewprasit, told demonstrators from the rally stage.
In the name of Pitak Siam and its allies I promise that we will topple this government.
Boonlert Kaewprasit, movement leader and retired general
Police fired 10 tear gas canisters at a group of protesters who removed barbed wire and barriers blocking their route in front of a UN building close to the main rally site, police said.
"Tear gas was used in one area because protesters did not comply with the rules," national police spokesman, Major General Piya Uthayo, said.
About 100 protesters were detained while knives and bullets were confiscated, he said.
Seventeen people, including seven police officers, were treated for cuts and other injuries.
Yingluck on Thursday voiced fears the protesters aimed to use violence and to "overthrow an elected government and democratic rule", in a televised address to the nation.
The government has invoked a special security law, the Internal Security Act (ISA), in three districts of the capital to cope with possible unrest.
"We will evaluate the situation daily and if it escalates we are ready to invoke emergency rule, but so far I think the ISA will be sufficient," Thai police chief, General Adul Sangsingkaew, said on national television.
Politically turbulent Thailand has been rocked by a series of sometimes violent rival street protests in recent years, although an uneasy calm has returned after national elections in 2011.
Two months of mass opposition protests in 2010 by "Red Shirt" supporters of ousted premier Thaksin Shinawatra sparked a deadly military crackdown that left about 90 people dead and nearly 1900 wounded.
Mr Thaksin's sister Yingluck is now prime minister after his political allies won a landslide election victory last year.
Mr Thaksin, who made billions as a telecoms tycoon, is adored by many poor Thais for his populist policies while in power, but reviled by many in elite, military and palace circles who see him as authoritarian and a threat to the monarchy.
"This government ignores widespread disrespect of the monarchy and even supports the perpetrators. It is a puppet of Thaksin," Pitak Siam spokesman, Vachara Riddhagni, said ahead of the demo.
Observers say prosecutions for insulting the monarchy have surged since royalist generals toppled Thaksin in a coup in 2006. Many of those targeted are linked to the Red Shirt movement.