Murorua e Tatou says the issue was detailed in a leaked report from the Ministry of Defence to the French government dated March 2010.
The Nuclear Association's president, Roland Oldham, told Radio Australia's Pacific Beat program that radioactive material could be released into the Pacific Ocean if the atoll were to collapse.
"Just in that little area there is over maybe twelve underground tests in that area and we have to remember that France have done altogether 193 nuclear test explosions in Murorua," he said.
"In the soil of Muroroa, if something happens there is about 150 holes containing very dangerous radioactivity."
The association says if the atoll were to collapse it could also trigger a 15 metre tsunami.
Mr Oldham is concerned the government didn't make the report available to the public earlier.
"This information was very discrete, I mean we only got this information now," he said.
"I mean the report is from 2010, why wait so long?
"So the public is not very aware of this situation."
Mr Oldham says the report doesn't properly emphasise the serious threat posed by the buried radioactive material.
"In this report that we got not too long ago, they're not even talking about radioactivity," he said.
"The way they present it it's like it's not very dangerous."
Raising public awareness
Mr Oldham says the association has been trying to raise the issue with the government and public.
"We've been trying to raise the consciousness of the people - our own people and our government and all the rest about this really frightening thing that could happen if actually one part of Murorua would collapse," he said.
The association want independent experts to be allowed to conduct a study to provide more information about the danger of the atoll collapsing.
Mr Oldham says if the atoll collapses there could be international ramifications.
"We have to warn everybody because the problem will not only concern some of the atolls that are only 100 kilometres from Murorua," he said.
"But I think it will be a really big problem to the environment if this nuclear radioactivity is to be diluted in the ocean and from there we have no control over what would happen."