The UN agency had been invited by the interim government to investigate claims by local trade unions that workers' rights were not being respected.
Felix Anthony, the general-secretary of the Fiji Trades Union Congress, told Pacific Beat the group had not been allowed to speak to anyone since arriving on Sunday.
The group did meet with the Labour Minister, he said, but that meeting was cut short.
"I’m told that around 11am (local time) the Prime Minister’s office wrote to the head of the mission and advised the mission that it ought to leave Fiji as soon as possible," Mr Anthony said.
"There cannot be any valid reason for the expulsion, more particularly after they were invited to Fiji. And Fiji, being a member of the International Labour Organisation, ought to have respected the delegation’s visit and participated."
Fiji's interim government has so far not given a reason for the expulsion.
The move has prompted calls for Australian holiday-makers to boycott Fiji.
"I would certainly be telling all Australians considering going to Fiji to reconsider that decision," said Ged Kearney, president of the Australian Council of Trade Unions.
Ms Kearney said she was "absolutely outraged" and called on the Australian Government to condemn the decision.
"The decision by the regime to expel the ILO does not fit with the picture they are trying to give that this is a government trying to engage with its people, trying to engage in transparency, accountability and indeed democracy."
'Not the first time'
Mr Anthony said it was not the first time the interim government had "behaved in this manner", referring to Fiji's decision in December not to allow members of Australian and New Zealand trade unions to visit the country.
That group had wanted to investigate the alleged abuse of workers' rights in Fiji.
But they were turned back at Nadi airport after the government accused them of bias.