Some of my does naturally refuse to nurse their kits around 4 weeks. I have seen no difference in the health and development of my kits when they were weaned at 4 weeks, 6 weeks, or 8 weeks. I think your kits will be fine as long as they are eating hay and pellets with no issues.