On Twitter, Grant put across "my side of the story", saying he declined the handshake because he felt Mr Javid was dismissive to press abuse victims.
Grant, 59, has been one of the leading voices in the campaign for stricter press regulation, after being one of the victims of the phone-hacking scandal.
He said his exact words to Mr Javid were: "If you don't mind, I won't shake your hand because you were rude and dismissive to the victims of press abuse when you met them as culture secretary."
A spokesman for the actor said he was referring to a meeting when victims of the press said they felt Mr Javid's "attitude in the meeting was 'borderline contemptuous'".
"Hugh would like to point out that the victims in question were not celebrities," the spokesman added.
"They were people with personal family tragedies who had been abused by sections of the press."