I shop mostly at a local pet supply store. They end up being cheaper than pretty much every other store in the city for Oxbow stuff. They also offer a frequent buyer program for pellets and hay that gives the 7th bag free, this is through oxbow, but I don't think available at every store.
For hay I do buy from a local farmer. The hay is quite nice and much cheaper that buying from a pet store. I go through about a bale a month.
Litter I usually get at a hardware store since I use wood pellets. The store can vary depending on if they have it in stock and the price. There are some feed stores I can go to as well, but the hardware stores are much closer.
If there is anything else I need, I just go to a store that will have it. This really depends on the item and how much I am willing to pay for it.