It depends on how much was consumed, and possibly the variety of the plant, on how much it could affect your rabbit. Consumption of woolly pod milkweed can cause something called head down syndrome in rabbits. It can also be fatal when a certain percentage is consumed. But I've also read of wild rabbits eating common milkweed without problems. So maybe it's just the woolly pod variety that's more of a problem, or maybe it's just the amount that is consumed. Best thing to do would be to consult with a knowledgeable rabbit vet.
We should also be concerned about Dogbane, which might be confused with milkweed since it a similar seed pod. Those in the know would not mistake them for each other. Nonetheless, dogbane is poisonous for human ingestion and likely so for rabbits.