Wearing the niqaab in a home with non-mahram males

I have a question about purdah. I live at my mother in-laws house and I wear niqab. There are always males in the home who are my husband's cousins and uncles. Is it permissible to not wear the niqab when they are present? I try avoid taking it off as much as possible but sometimes I am unaware that they are in the home and am not wearing it which makes me feel guilty because there have been times where my husband says I should leave it off or he doesn't encourage to wear it when those particular people are around.