Marital problems

I have been married for couple of years now and ever since my marriage my husband keeps talking bad about my family and how he did not get anything as dowry from my family. I have stayed with him for 1 year after marriage since he is unable to provide me visa in order to stay together. His job is not giving him visa so he buys business visa for himself and the only reason he is sticking to his job is because it's easy where he does not have to work. The one year I stayed was when I found a job on my own. I want to your advise please since am not sure what to do. My husband does not make any effort in keeping us together, does not like my family, takes help from them but talks bad behind, he uses bad language with me, tells me I am not fit to be a wife, does not like the food I prepare, and asks me to stay in his father's house since I am married to that house. He married me thinking my family is from US and thought he will get everything and it is not necessary to work. Now I have a baby as well and he does not show any interest in keeping us together. All the time he cries about how he has no money and has to send to his family and how we have become an expense on him. I keep thinking why am I with such a person who does not like any of my qualities. I want to divorce him but worried about my child. Please advise is this the right ground to take a divorce or should I keep trying.