Portofino is, like its name suggests, built around a large harbor. So, if you have a yacht, this is where you want to park it. If you are a mere peasant like me, though, this is where you go to look at the people with the yachts.
What I'm trying to say is that Portofino is the posher version of the Cinque Terre, with the expensive shops to show for it. That said, it is perfectly easy to visit Portofino without spending money, as Daniele and I did (except for the ice cream we got on our way out). There's a perfectly good reason that rich folk flock here, and it is that Portofino is absolutely beautiful and worth stopping by. When people ask me for places to visit around the Cinque Terre, Portofino is usually the first place I suggest.
Like most Italian towns, your day is best served just wandering around. If you have the time and energy, though, I highly recommend taking the hike up to Castello Brown, an old house-museum located on a large hill directly opposite of the inner-harbor. Even if you don't feel like going in the museum itself (which, it's cool, and you should), the hike has the most incredible views of Portofino. If you're looking for a most romantic place to experience a sunrise or sunset, this is it.