Am i wrong for letting my dad go??? its my life and i'm not gonna apologize for it?

When i became pregnant my dad basically cussed me out, told me my life would amount to nothing and told me that i need to apologize to him and his wife or never come over again (dont even live with him). i did nothing to him so i didnt come over. O and my stepdad had a heart attack that weekend. so i basically havent spoken to my father since, my daughter will will 2 this year. he calls me occasionally but always just ends up cussin at me so i just hang up the phone i dont like drama. but now the other side of my family is mad at me because i am not coming around anymore, but they live out of state and when they do come here of course i dont feel welcome over my dads house. im just tired of being made to feel like **** when im taking care of my business and my daughter. im not gonna apologize for livin my life, **** happens sometimes. am i wrong for just letting my dad go??? i have tried to talk to him but his temper just gets in the way and i am not gonna sit there and get cussed out for nothing