An ABC investigation has found children as young as 12 are being exploited by the men.
Some of the children have been transported between towns and even interstate, while the ABC has also been told that some abuse has been filmed by paedophiles.
It is understood that police have identified between 30 and 40 children who are living in Department of Human Services residential care that may have been exploited in this fashion in the past 18 months, but the number could be higher.
The suburbs around Dandenong are believed to be an area of particular concern to police and the department.
Police have found it difficult to charge the perpetrators, as the victims, many of whom come from traumatic backgrounds, are reluctant to give statements implicating their abusers whether through fear, a mistrust of authority or a belief that they are not actually being abused.
The Department of Human Services and Victoria Police have spent two years working on a new approach to tackling the issue, which revolves around better sharing of information and educating workers on how to recognise the signs of sexual exploitation.
Police are also practising disruptive policing.
This can mean charging the abusers with drug or property offences to bring them into the criminal justice system and keep them away from their victims.
Staff from the department have also been embedded in the police sex crimes unit.
"What we would say is that these young people are not competent to make a decision, and using that word 'prostitution' implies that they are making a lifestyle choice, working in the sex industry," the department's chief practitioner, Robyn Miller, told the ABC.
"They are children, they are young people, they have this history of trauma. They are not competent to make that decision.
"It is abuse and we name it that and we have a zero tolerance.
"Our whole approach has been to raise consciousness within police and with carers that this is not adolescent behaviour, it is not experimentation, it is sexual exploitation and it is a form of abuse."
Men would choose a girl to have sex with
In one case that did go to court, a number of men in Shepparton were charged with child sex and prostitution offences after having sex with girls in return for cigarettes, cash and shopping trips.
At the plea hearing of the ringleader, Emran Dad, in 2012, the court heard that he gave a 13-year-old girl a pack of cigarettes in exchange for sex.
Another girl, 16, said Dad would contact her via Facebook, then drive her to the house of a friend of his.
The girl said other girls were at the house and that men would then turn up and choose a girl to have sex with.
She said they would be paid about $200 each time this happened, depending on the sex acts they would perform.
She said Dad would receive a separate $150 payment for organising it.
"The young people that we work with have experienced such loss and grief and pain that they are really vulnerable to people giving them attention, and these offenders are expert manipulators," Ms Miller said.
"They are disgusting in the way that they entrap our young people, so they will use social media in an expert way and will befriend, so it's like they will pretend to be something they are not."
In a statement, Victoria Police said there had been a number of successful prosecutions for exploitation offences of this nature and that there were ongoing investigations.
The State Opposition's Jenny Mikakos says there must be an investigation.
"There needs to be a thorough inquiry into how this has happened," she said.
But Community Services Minister Mary Wooldridge says the exploitation precedes the current Government.
"It's only the Coalition Government that has responded comprehensively," she said.
She says 2,000 police, child protection and care workers have been trained disrupt the exploitation of children.
Do you know more? Email firstname.lastname@example.org