The remark was made during a reception in honour of the Prime Minister's XI cricket match.
Mr Mathieson is a men's health ambassador and was encouraging the crowd to get prostate checks.
He then suggested men should look for a small, female, Asian doctor to perform the examination.
"We can get a blood test for it, but the digital examination is the only true way to get a correct reading on your prostate, so make sure you go and do that, and perhaps look for a small, Asian, female doctor is probably the best way," he said.
In a statement released this morning, Mr Mathieson admitted the joke was in poor taste.
"My comments last night were trying to raise awareness about prostate cancer and the need for men to have regular checks and the importance of early detection," he said.
"It was meant as a joke and on reflection I accept it was in poor taste. I apologise for any offence caused."
The Prime Minister said Mr Mathieson worked hard to raise awareness of prostate cancer, but he did the right thing by apologising.
"Obviously there are various ways of getting that message across, but he's acknowledged that the joke cracked last night was in poor taste," she said.
Earlier, the Opposition Leader Tony Abbott called on the Prime Minister to respond to Mr Mathieson's remark.
"After everything the Prime Minister has said on these sorts of subjects, I do think that she does have to deal with this personally," he said.
But shadow attorney-general George Brandis told Sky News that Australia did not need a culture of finger wagging.
"The fact that the first item in your program should be ... the Prime Minister's partner [making] a slightly unfortunate reference in remarks that were meant to be in good faith, just goes to show how stupid the culture of political correctness, which this Government has deliberately fostered, has become," he said.
The head of the Men's Shed movement, David Helmers, said Mr Mathieson's joke was "inappropriate", but paid tribute to his work for the organisation.
"Tim's done wonderful job promoting men's health and Men's Shed over the years, and it (the joke) would definitely not affect his position with us at all," he said.
The chief executive of the Prostate Cancer Foundation, Dr Anthony Lowe, described Mr Mathieson's joke as "a bit unfortunate".
"Tim, by the way, has been a great supporter of men's health - both the Men's Shed movement and also [for] prostate cancer for a long period of time - and we're grateful for his support," Dr Lowe said.
"But we'd prefer that there's not too much humour around getting tested for prostate cancer because it's such a serious subject.
"We would recommend that men over 50 have both the blood test and the digital rectal examination, because neither of them is perfect."