The influence of credibility
Graphing the influence of credibility and expertise on a rating:
I have removed the actual rating variable from the equation here, since what's really important is how the other data modifies whatever rating a person puts in. From top to bottom, the curves are a claim of specific expertise, expertise in the area, expertise in a related area, and no expertise. The red lines are just reference points--they are the influence a person with neutral credibility would have at the four levels of expertise. A few interesting outcomes in this graph:
- A person who claims specific expertise can have about -65% credibility and have as much influence as somebody with no specific knowledge and neutral credibility.
- Somebody with no expertise can only max out as having as much influence as a person with expertise in the area and neutral credibility. The same is true of max related expertise and neutral specific expertise.
- No expertise and perfect credibility is the same as specific expertise and -30% credibility.
When viewed by itself, I think this seems to show credibility as being somewhat undervalued. However, this is only a small part of the system. The credibility value itself is based on a person's claims of expertise. So while a person might be able to game the system once and claim specific expertise when they don't know what they're talking about, the resulting reviews would cause their credibility to tank in the future to an extent that it would take quite a bit to climb back to that -30% figure. Somebody with consistently bad reviews on claims of high expertise isn't going to be able to climb back out of the hole without some good reviews at the same expertise level. The system is pretty harsh to people who claim expertise only to be shot down by others, so I doubt many people are going to be riding that upper curve...particularly people with poor credibility.
posted by John @ 3:07 AM
2 comments
2 Comments:
John,
Judging from your analysis of credibility versus expertise, I'd say this graph was useful to you. I'm not sure I always agree with your decisions; for example, maybe Alan Sokol's substantial expertise in physics should translate into credibility when he's talking about cultural studies. But that's up to you, and no one solution is going to be best.
To make the metric more transparent, you could use a weighting system like RatingPlot, which allows the user to adjust the value of the various inputs to accord to their own preferences. More on that later.
BTW, as a mathematical purist, I would argue for using the exponential function e rather than powers of 2. The latter to me suggests a binary relation that isn't implied by the natural log.
2^E is in there because I was going to do some fancy bitwise calculations...but, now I'm not as sure I want to do that. I'm somewhat afraid that e would put too much weight on the expertise factor though...it changes the range from 1-8 to 1-~20.
Post a Comment
<< Home