Baby buns, just like any babies, need lots of rest. Of course, if you hold them while you're watching tv or something,theymight take their nap on your lap.
I would think one of the big issues would be going back to the cage to go potty! Baby bunnies don't have a lot of control over their bladders, especially. It might be best to hold your bun multiple times a day. Rabbits are crepuscular (sp?), which means they are most active early in the day and during the evening. Most of ours prefer not to be bothered much during the day. You might want to plan on giving your rabbit attention during those times that he's more active, because he'll be most receptive to it.