I certainly see it killing the casual sector of handheld gaming on dedicated consoles. It'd be convenient for people to have games like Brain Training or Nintendogs on their phones as opposed to having to buy another system and carrying it around with them all day. So now that other devices like smartphones can offer those experiences (or similar), I don't expect you'll see the same non-gamers buying either the 3DS or the Vita. At least not in any significant number.
Of course for traditional games, like Mario, Zelda or Uncharted and Resistance the limited controls of smartphones (touch-screen or nothing) means that the 3DS and Vita will be unquestionably the way to go. So they'll sell to gamers the same they always have.
I think this new generation will still be successful, I just think we'll be seeing (most notably on Nintendo's side) less consoles sold, but much bigger attach rates compared to last gen. Maybe it'll be less profitable than last gen, but for us gamers I don't think there'll be much effect because traditional games will, more so than ever before, be a key selling point.