Several sources have told the ABC the Immigration Department was formally notified of the move a week ago.
The Scherger detention centre in far-north Queensland as well as the empty Pontville facility in Tasmania are among the sites to shut down.
The other two facilities to close are at Leonora in Western Australia and Port Augusta in South Australia.
Meanwhile, PM has been told that a fifth centre, the Inverbrackie facility in the Adelaide Hills, will also be closed.
Contractors working for the department have been anticipating the announcement for several days.
In a statement, Mr Morrison says the four closures will save the Government $88.8 million a year.
The Scherger detention centre, Port Augusta residential housing and Leonora alternative place of detention are all set to close by the end of February.
The Pontville facility will be handed back to the Department of Defence at an unspecified date, the statement says.
"These sites are remote, relatively small and expensive," Mr Morrison said.
"While I acknowledge there will be an impact on some local business and service provider staff, these closures bring significant financial savings for the Government and the Australian taxpayer.
"The savings, which amount to at least $7.4 million a month, comprise the costs payable to the detention services provider, infrastructure and leasing arrangements and detention contractual arrangements."
Mr Morrison says the four facilities were never meant to be permanent.
He has indicated there may be more closures in the future, saying the centres are no longer needed because the Government's border protection policies are slowing the flow of asylum seeker boats to Australia.
"This is the first down payment on closures of detention centres," he said.
"And as time goes on and we're able to further draw down that detention population - because we've been able to address the border protection challenge - then I expect there'll be further savings down the track."
In April last year the Coalition also promised if it was elected it would shut the Inverbrackie centre "as soon as practicable".
Today, a spokesman for Jamie Briggs, the Assistant Minister for Infrastructure and Regional Development, told PM this promise will be kept, and there will be more said on the matter soon.
Serco, which operates nine detention centres and four alternative places of detention, has contracts with the Immigration Department which are due to expire later this year, according to the company's website.
The Government has been reviewing the future of Australia's various detention centres as it aims to divert all new boat arrivals offshore.
Labor claims credit for closure
Federal Labor MP Nick Champion says it is because of the actions of the previous Government that a number of immigration detention centres are now able to be closed.
"That's a result of the actions of the previous government - the Rudd government - in adopting the New Guinea solution which saw a massive decline in the numbers seeking asylum through irregular means in Australia," he said.
The Greens' Immigration spokeswoman Sarah Hanson-Young says she is concerned about where the asylum seekers will be moved to.
"Christmas Island is already crowded; Manus Island and Nauru are not fit to hold anybody, let alone more people," she said.
"Mr Morrison needs to be upfront with the Australian people about exactly where people in those camps are going to go."
Closure to have economic impact on communities
The Mayor of Brighton, in Tasmania, says the closure of the Pontville detention centre will be an economic blow to the community.
Mayor Tony Foster says he is also angry that the Federal Government did not inform him of the decision before announcing it.
"Somewhere between 45 and 48 [staff] were able to service the centre while it was open, so that business is no longer there," he said.
"Very disappointing, disheartening ... there's been no consultation.
"I think it is absolutely just the way that this Government is treating the whole of Australia at the moment."
Tanya Browning, the acting chief executive of the Western Australia's shire of Leonora, says the decision to close the detention centre in her town is disappointing.
But she says it has not come as a complete surprise.
"We certainly will miss some of the staff that are here, especially those that have chosen to make Leonora their home," she said.
"That's a loss that certainly will be felt.
"Fortunately our community is very resilient.
"We are prepared for those types of events, [like] when mines close, and we might have a loss of a number of people within the community."