I am hoping that by not growing them a year maybe the bugs will travel on. Is that a good idea? Nothing seems to get rid of them.
That most likely won't stop them since many other plants in your area will be harboring them. They come in so easily. In my area syrphid fly larvae (little green 'worms', about 1/4 inch long, with pointed ends)(see photo below) and lady bugs are the best natural remedy, but they also come in naturally.
Aphids are why I stopped trying to grow broccoli - they would get up in the buds and no amount of soaking could get htem out. I assume I eat some here and there, but knowing there are so many in home-grown broccoli makes it too icky to eat.
Our weather here is very nice right now. High 60s when the sun is out. I've spent much time in the garden today working the beds, planting and mulching. Over the past week I put out the kale, spigarello, parsley and onions. Still have to plant the chard seedlings. They all should be ok since we are in 'the avocado belt' microclimate, and rarely get frost here - maybe one light one in 20 years.
Syrphid fly larva with sucked out aphid bodies- ours here are about 1/4 inch long, but this one looks larger: