Since the system is basicly a way to assign ‘experts’ on things it might be usefull to do credibility scoring per subject or maybe news-source.

I might consider reviews by stephen hawking very ‘credible’ for articles on physics but maybe less ‘credible’ for articles on cooking(as I have no idea how extended his cooking knowledge is).

This would make the system a little more complex, as you would need set terms to define subjects and it needs to be clear to the user in what context someone is reviewing an article. A article on string-theory for instance could be reviewed by a mathematician or someone in physics, which is both valid. But you might not have a physics score for the mathematician or vice versa. You do need to prevent someone with a high credibility-score on politics getting a high rating on his review on an article on math, even though he might claim the article is political.

Food for thought.

Goal

To provide a ‘network’ of peer review in order to calculate the credibility or authors and articles based on the credibility and reliability of ones peers, without comprosing personal scoring information of users

Terms

• user: someone participating in the project
• review (score/text): the review of an article or author in the form of a score and/or text
• reviewer: a user who reviews articles or authors
• credibility (score): the indication of how credible someones ‘reviews’ are.
• reliability (score): the indication of how reliable someones reliability or credibility scoring is.
• review directory: any site listing reviews by reviewers

Structure

a user stores the following under his site(a webhosting place under his control, a blog for instance):

• any review he has made on any author or article
• a ’static’ table with the reliability score of his ‘peers’ (anyone he wants to score)
• a ‘cache’ with the reliability scores of ones peers peers.
• a ’static’ table with the credibility score of his ‘peers’ (anyone he wants to score)
• a ‘cache’ with the credibility scores of ones peers peers.

user needs to be able to answer the following questions:
(no question can result in another request and ‘I do not know’ is a very valid answer)

what is your ‘review’ on an author/article

• If you do not know any thin about the author/article, you should answer ‘i do not know’.
• If you have made a review of the author/article you provide your review data.

How reliable do you think a user is

This result is computed as follows:

• You take the scores of your peers from the user
• each score is scaled down with the ‘relability of the peer’
  example:
you take the score of user A from peer 1.
peer 1 has assigned a reliability of 70% to user A
you have a reliability of 60% assigned to peer 1
now the reliability of user A @ peer 1 is 60% of 70% == 42%

• you add your own score to the list
• you return the average scores of the list
• if neither you nor your peers have any scores of the user, you answer that you have no scores.

How credible do you think a user is.

This result is computed as follows:

• You take the credibility scores of your peers
• Each credibility score is scaled down with the ‘relability of the peer’
  example:
you take the credibility score of user B from peer 2
peer 2 has assigned a credibility of 80% to user B
you have a reliability of 70% assigned to peer 2
now the credibility of user B at peer 2 if 70% of 80% == 56%

• you add your own score to the list
• your return the average score of the list
• if neither you nor your peers have any scores of the user, you answer that you have no scores.

Cache maintance

• Caches needed to be updated frequently but can -NEVER- be request based. This in order to prevent network loops.
• Caches are updated by asking questions from peers.

Directory maintanance

• users push reviews to directory sites.
• directory sites should demand authentication (for instance through openid) or do source checking.
• directiry sites are recommended to not pre-filter reviews

Filtering/Score checking

Credibility Score requesting by users

or: How to find out the credibility of an author or artcicle

• Lookup the author/article at a directory service and/or your peers.
• take the reviewer and request the score of the reviewer at your peers.

Filtering

• request an article list at an directory service
• take the score (see above) of each reviewer
• filter out all articles with score below oyour personally chosen threshold