Devinder Pal Singh Bhullar from northwestern Punjab was convicted over a New Delhi car bombing that killed nine people in 1993.
The Supreme Court has ruled that his lengthy wait in prison and his pleaded mental problems are not reasons to set aside the death sentence.
"The petitioner has not been able to make out a case for commuting his sentence," Justice GS Singhvi said.
Mr Singh had appealed for his sentence to be commuted to life imprisonment on the grounds he had spent two decades on death row.
He had also challenged the right of the state to execute mentally ill convicts, claiming he had developed psychological problems while languishing in prison, including schizophrenia.
The ruling could have implications for more than a dozen other prisoners whose executions have also been held up for decades, including three men convicted over the assassination of former prime minister Rajiv Gandhi in 1991.
Suhas Chakma of the New Delhi-based Asian Centre for Human Rights says after this ruling India will have to resume executions, including of 17 death row convicts who had filed petitions before the courts on the grounds of delay.
"India must decide whether it wants the tag of the top five executioners of the world along with China, Iran, Saudi Arabia and Iraq," Mr Chakma said.
India had not carried out an execution for eight years until last November when it put to death the only surviving gunman from the 2008 Mumbai attacks.
Another militant convicted for his role in an attack on the Indian parliament in 2001 was executed in February after his appeal for clemency was rejected by President Pranab Mukherjee.
Mr Mukherjee, who took office in July last year, has signalled a hard line on the death penalty by regularly dismissing mercy pleas, unlike his three predecessors.
More than 400 people are on death row in India.
'We waited for so long'
Navneet Kaur, the 48-year-old wife of Mr Bhullar who was present in the courtroom, said she was disappointed by the order.
"We waited for so long... we have suffered so much," she told India's NDTV news channel. "The court did not consider our point, it has not been fair with us."
Mr Bhullar was awarded the death penalty in 2001 for carrying out a bomb blast in Delhi in 1993 that killed nine people and injured several others, including prominent youth Congress leader MS Bitta.
An ideologue of the Sikh separatist militant group Khalistan Liberation Force, Mr Bhullar was also suspected of being involved in a 1991 bomb blast in the northern Indian city of Amritsar.
He fled to Germany in 1994 but was deported a year later to India to face charges of terrorism.
The main political parties welcomed the court's judgment with the ruling Congress saying the "entire country would accept this order" and the Bharatiya Janata Party calling it a just verdict.