Peeing next to the box can sometimes just be them hanging their butt over the edge if there's a low edge, and missing peeing in the box. Solution is to get a higher edged litter box. I prefer the rectangle ones too. They are more accommodating for bun to hang out in and eat, especially as they get older and bigger.
If he seemed comfortable being around people and not scared, I would probably just invade his space and sit in his pen with him. I wouldn't bug him but would give him a chance to come check me out and get comfortable with me. If you don't want to sit in his area, you could pen off an area around his cage for him to come out and explore with you in there. If you are concerned about accidents on the carpet, put a blanket down or I've even used a tarp. Sitting quietly with your bun in a smaller area, and not trying to pick up or force contact, is the best way to form a bond with your bun and let them learn they can trust you.
http://flashsplace.webs.com/bondingwithyourbunny.htm
If the pellets are a different kind(brand, type) than he is used to from his previous home, then that could be why he's not eating them, different smell and taste. It's always best to keep a new bun on the same food and then gradually over a couple weeks at least, switch them onto a different pellet if you intend to change the brand/type of pellet. This minimizes digestive problems occurring from the change, something that rabbits, baby rabbits in particular, are prone to. It also gives them a chance to get used to the taste of the new pellet.
If he's on the same pellet as he was getting in his previous home, then maybe he just really likes the taste of the hay you are feeding. You aren't feeding alfalfa are you? If you are feeding a grass hay, then it's perfectly fine for him to be eating lots of it. Rabbits should always be free fed grass hay. Eventually he should take an interest in the pellets and start eating them, unless they are spoiled in some way, then you don't want him to eat them and should throw them out and get a new bag/batch of pellets.
I store my hay in an enclosed storage closet where rodents and other critters can't get to it. Though if we had roaches here they would be able to. If you live in a humid environment, the most important thing with hay storage is making sure there is good air circulation so the hay doesn't mold, which can be dangerous for rabbits to eat(moldy hay). It's also good to store it somewhere where critters can't have access and contaminate the hay(indoor closet in a cardboard box or old duvet).