Apparently so. I mean, if seiyuu can be that young, why not idols?
I guess because of the more physical part of being an idol, such as the photoshoots and things. Since seiyuu-work is behind the scenes and out of the public eye, rather than the idols, who's entire career pretty much revolves around pushing your image and personality to a (predominantly older male) fanbase, it's a little more acceptable.
It would be like a child actress compared to a child pop sensation.
Even AKB48 and its sister groups stick to above the age of prepubescance. Mostly. The youngest one I know of is Matsui Jurina, and she's 15 now, though I think there are younger ones in NMB48.
Then again, this is fiction. Some disparity is allowed for the sake of creative license. Edited For: A spelling mistake. And Jurina is 15 now.