Not giving a beggar anything due to not having any change

Sometimes beggars come and ask us for money when we are outside the office with colleagues. Sometimes, I don't have small currency to give them and neither can I say anything to them. Last time I went to the city and we were outside the office and one beggar was asking for money. He came to me but I didn't have small currency at that time to give to him, but he stood infront of me. After a few minutes a colleague who was with me gave him some money. I sometimes think what they might think of me. Genuniely if I had small currency I would have given it to him. Can we deny if we don't have small currency notes.