If only the recipient can see their rep, then what's the point?
There is a rep icon under your post count, little green boxes with a phrase like "xxx is on a distinguished road". More good rep = more little green boxes. Most folks have one for now. I think there are black boxes for negative rep (most of these systems have both), but I'll have to ask Ryan to be sure. Not a lot of negative rep on this board.
To the original question "what's the point?" - The point in my opinion, is to have a simple tool to help foster communication. We can't read facial expressions or body language on this contraption - the reputation doohicky helps us communicate our likes an dislikes without the usual people reading skills of everyday human interaction.
Over time, it also serves to aggregate the general social feeling towards you - with that little green box trick. It means that eventually, a new member can see at a glance those who the community generally feel have something worth saying, or not. Coming from several motorcycle boards where being an ass is a sport, knowing where all the old-timers stand with the majority is a very good thing.
And yes, because humans are involved, it can be used to be a pain in the ass. It DOES have some basic safeguards (you can't give too much rep to the same person for example, without spreading it around) to keep it relatively free of abuse, but some folks are jerks no matter what the medium of jackassery delivery is. They'd be jerks without the rep feature too.
The anonymity feature is there to make certain that there is no retaliation for giving negative rep. The side effect is that jerks can use it too - but over time, it should average out. and in my experience, it makes the jerks that much easier to spot for new members.
And of course there is always the perfect solution - if you don't care for the feature, ignore it. It's just an extra, no need to worry about it too much.