I would say that when you found him he was about 5-6 weeks old. So that would make him about 11 weeks old now, or 2 1/2 months. In his current pictures, I agree that he does look to be about 4 lbs. At 2 months old that is still quite big. Just as perspective, my 7.5 lb full grown rabbit was probably around no more than 2-3 lbs at that age. So your bun will likely be close to 10 lbs full grown, though could be bigger. Which would match if he was a flemish giant mix. He probably won't be as big as a full flemish, but will still be quite big as rabbits go.
Small to medium size rabbits are usually fully mature around 6 months, though do continue to grow and fill out a bit more in the next few months. Large breed rabbits like flemish, take about a year plus to fully mature.
So yeah, you do need to wait at least 2 months and for his testicles to descend, before you could get him neutered, if that is what you plan to do. Though if he is a flemish mix, some vets like to wait a bit longer to neuter for large breed rabbits. Usually around 8 months old. Neutering isn't a necessity though, if you don't plan on having other rabbits and he doesn't start displaying hormonal behaviors like humping and spraying. If 'he' turns out to be a girl though, I would suggest a spay, as uterine cancer is an issue for pet rabbits especially.
It can be very difficult to sex young rabbits, even for very experienced rabbit people. Sometimes you just have to wait til they mature enough for the 'plums' to drop(in the case of boys). Once that happens they are old enough to get neutered.
Do keep us updated on 'his' growth. I'd be very interested to see how big he gets.