Union minister says he met Jeffrey Epstein only a few times in official capacity, denies any link to criminal activities