Hi, Scott. Welcome to the board.
It sounds like you're doing great with staying away from dairy and sugar. Congratulations on already being way ahead of most of the population in that regard.
Have you considered beans? It sounds like your body may be accustomed to a higher level of protein in your diet than you would get from sweet potatos and brown rice.
From your note, it's not clear how much of a transition you've given yourself. Perhaps in the past you've simply dropped your accustomed diet and replaced it in one swell foop with sweet potatos, brown rice, fruit, and veggies? I wonder how it might work to reduce the meat and replace it gradually with your chosen plant foods over time. For example...
week 0: 100% meat
week 1: 75% meat, 25% plants
week 2: 50%/50%
week 3: 25%/75%
week 4: 100% plants
Also, you might want to post this question in Jeff Novick's forum. He's a registered dietician and very knowledgeable about the practical issues connected with eating this way.
Whether you think you can or whether you think you can't, you're probably right.
Weight Loss Through the Magic of Calorie Density: http://wp.me/p1utH8-v