Kappa:

The Kappa is the ratio of the proportion of times the appraisers did agree to the proportion of times the appraisers could agree.

Kappa Statistics: If you have a known standard for each rating, you can assess the correctness of all appraisers' ratings compared to the known standard. If Kappa = 1, then there is perfect agreement. If Kappa = 0, then there is no agreement. The higher the value of Kappa, the stronger the agreement. Negative values occur when agreement is weaker than expected by chance, but this rarely happens. Depending on the application, Kappa less than 0.7 indicates that your measurement system needs improvement. Kappa values greater than 0.9 are considered excellent.