One concern I've heard is the notion that he's a prime product of the general healthiness of Mormon culture. Mormons try to set up their lives to have a lot of good influences from other Mormons around them. But it gets pretty lonely in the White House, and it's a new set of challenges. (Also, despite his looks, he's not young anymore.)
By way of analogy, think of the late Neil Armstrong. He was a prime product of the general healthiness of mid-20th Century American culture (which Mormons continue continue to espouse, which is why they are considered so weird and creepy today). American culture systems in place that produced a lot of competent, brave, altruistic, and modest people, few more so than Armstrong. And part of that modesty was that he didn't much mind being viewed less as a unique superman and more as proof that the systems worked. He didn't run for President.
Today, we have a sense that our society's general systems don't work that well, so we are more invested in longshot hopes placed upon space oddities like Obama.