Definitely PSP, but GBA comes in at a close second. The PS Vita is awesome too, it can do so much more if it's modded, but that is so much harder to do than modding a PSP.
The PSP unmodded is still great. Ratchet and Clank, Daxter, Lego Star Wars, Star Wars Battlefront, ATV Offroad Fury, Socom Fireteam Bravo, GTA Liberty City Stories... It just has everything. Plus you can buy an adapter for micro SD, get like 128gb of storage, and fill it full of music and movies. It pre-dates the iPhone, but can do everything! There are even some fan made games that dont even require modding, like MineCraft PSP.
GBA because I can pick up cheap but interesting games for it all over the place. It's a shame that the system has gotten so expensive. I use to find them for 5-15$ and now they're over 100$ 0.o