"Where were these people when David was struggling in the orphanage?" Yohane Banda asked, according to the AP news agency, and he added: "These so-called human rights groups should leave my baby alone."
The little boy is currently in London with Madonna but officials from Malawi will monitor him over a period of 18 months before the adoption gets approved.
Mr Banda explained he was forced to give his baby up to the Home of Hope Orphanage in a Malawian village after his wife had died a week after giving birth to him. The couple had another two sons who died in infancy from malaria.
"I was alone with a baby, I had no money. I couldn't buy him milk. That's why I surrendered him to the orphanage," Yohane Banda said.
"Orphanage life is no good. We leave kids there because we can't look after them properly ourselves. Now my son has been taken by a kind-hearted woman, these people want to bring him back to the orphanage," Yohane Banda added.
The child's grandmother, Asianati Mwale, 56, was in support of Mr Banda and Madonna's actions.
