There is opposition among Manus Islanders to the plan to relocate men from the Australia-funded detention centre when it shuts down at the end of the month.
Immigration authorities want to move 600 men to three sites around the main town Lorengau.
But residents in Ward One on Lorengau's outskirts are angry about the construction of one of the accommodation facilities in their neighbourhood.
They say there was no warning or consultation before construction started earlier this month.
"We want our safety. What guarantee do we have?" said one man who spoke at a community meeting attended by about 50 people.
"If there is no guarantee, close this down. It must stop."
"This project, I want the community's support, we must all say no to it," said another male resident.
At the meeting people signed a petition calling on the Federal Government to take the men from the detention centre back to Australia and resettle them to third countries from there.
Others suggested taking more drastic action.
An elderly man said, "I'm going to get the youths. We'll get spear guns, knives, axes, spades, crowbars and we will block the road".
The new accommodation site sits in the middle of a residential area and is surrounded by several houses.
A chain strung across the access road restricts entry into the site.
Photos taken by locals show heavy machinery at work, roughly graded roads and a several demountable accommodation blocks.
A Manus provincial government representative told the community meeting the government was also in the dark, and that it had not been consulted or advised of the project before construction began.