Husband not allowing children to visit their non-Muslim grand-mother

I am a revert alhumdulilah. I am in a dilemma regarding visiting my mother who is Hindu. My husband does not allow my kids to visit her because she has a temple in her yard. It's not like the kids play outside. My family has a lot of respect that I am a Muslim and they have never forced us into doing anything unlawful or haram. My mother is now heartbroken as she can't be with her grandchildren and I feel bad visiting her without the kids. Her tears are haunting me. I know that a mother holds Jannat under her feet and to make a mother cry is the biggest sin. I can't take this anymore I'm miserable and how do I earn Jannat one day in sha Allah if my mother is hurt. So please advise as I can't hear or see my mother cry anymore.