This is probably the healthiest way to eat while you're pregnant! Congratulations!
I can't speak to your medical professionals' concerns. They may be valid or they may just expect everyone to fall into the expected range and are raising unnecessary alarms. Were you or your husband considered small at birth (or in utero)?
When my son was young, I knew a woman in LLL with a very petite daughter. Her doctor was always trying to get her to feed her toddler junk food (butter, ice cream, etc) to help her to gain weight.
She and her husband were both petite. I happened to run into them 10 years later and the little girl is a perfect size! Did I mention the mother refused the junk food regimen that had been prescribed?