From what I've read, it is best to buy what organic you can afford for multiple reasons including your health and the health of the planet.
That being said, I don't buy all organic, however I buy as much as possible. If your supermarket has an organic produce section, don't hesitate to check for sales, often times (especially if it is very ripe) they'll sell the organic at the same or less cost than non-organic produce. A good majority of food at the health food stores is organic as well, yay!
Also, if your grocery stores don't have much in the way of organic foods, don't hesitate to ask!! I was asking about agave nectar at my local health food store, and (believe it or not) the lady working had never even heard of it. Low and behold, I stopped in today and they had added it to their inventory

I used to write letters consistantly to large chains and fastfood restaurants asking for more meatless alternatives; I was very proud the day Burger King added a veggie burger to their menu. If enough consumers state
exactly what they want, eventually the providers catch on and make it available.