Instant Karma (or not)

I think I should add this note to the discussion on post ratings before going any further:

Re:Poste is not intended to be a threaded discussion, like some BBSes or Slashdot. While criticial review of other posters is certainly important, the main focus is on a critical review of the original story. I don't really want a mechanism that encourages people to get into extended flame wars, which is what I've often seen from threaded discussions. The intent is to have textual replies only to the original article, and validity reviews on those textual replies...but I know that isn't realistic. I think a compromise is to make the discussion unthreaded, but give a simple mechanism to allow quoting of previous posts when inputting a new textual reply...so there is no real 'reply to this post' button, but you can review a post and then quote it in your own reply later.

This is important because I want to try to maintain focus on the original article, but it is also important for the mathematics of ratings. If you start getting into flamewars, then it's pretty likely two people will start rating each other's posts as invalid, which means both of their credibilities take a hit, which means their ratings become less valid and their credibilities go back up, and it becomes very circular. Circular is bad. The best way to prevent this situation is to just discourage flame wars in the first place, and I'm trying to do that in a number of ways:

• Non-threaded discussion
• Hidden names (though they're viewable, they're at least a click away
• Community makeup (though academics are certainly capable of a flamewar, at least it's more likely to be productive than the typical "ZOMFG YOU SUXX0RS!!!!11!1")
• Credibility ratings, which themselves discourage prolonged flame wars

However, all of these are soft measures of prevention, not hard rules, so the credibility calculations need to be able to deal with these sort of circular variables.

Credibility

The easiest way to avoid a recursive formula is to define it as something that can only run once. In this context, that means that a person's credibility is not updated dynamically whenever somebody reviews them. I think this also makes sense from a philosophical perspective. The ideal of "academic thinking" is not one that turns on instant analysis...it is intended to be more thoughtful. Tanking somebody's credibility just because the first person that came along happened to disagree doesn't really follow that philosophy. Waiting gives the entire community the chance to weigh in before somebody's fate is decided.

However, there needs to be a way to rank things quickly, so just delaying the entire process isn't workable. The pace of media is too fast for that. As a compromise, I think that posts should have their credibility calculated in real time, but people should have their credibility calculations delayed. Over the long term, personal credibility is more important...but in the short term, the individual posts are what count. And even though personal credibility is not updated in real time, a person's existing credibility is used in the calculations for post credibility.

So, how do you reflect a longer period of contemplation in personal credibility calculations? I think I can use the pace of media to my advantage here. It's unlikely that any given story will remain a hot topic of discussion for an extended period, just because new stories come out so quickly. During the time a story is in the spotlight, the credibility of posts about that story are calculated, but personal credibility is not updated to reflect these calculations. After a sufficient cooling-off period, maybe a week or so, post credibility is frozen. Those post credibilities are then used to update personal credibilities. The updated personal credibilities are never used to update the post credibilities in that thread, so the circular calculation is broken.

Updating personal credibility is a simple weighted average of all the credibilities of posts the person has written. Claims of expertise are given the same exponential weight as in the post credibility calculation: 2^E, where E is the expertise level the person claimed when making their post. The expertise of the reviewers are already reflected in the post credibility calculation. In effect, this means that the stakes are higher when you claim to be an expert on something. If you claim expertise, but then the community says you don't know what you're talking about, it will cause a much bigger drop (8x) in your credibility than if you just made the post but didn't claim to know anything special about the subject. This works as a check to keep people from claiming expertise they don't have.