For expert evaluation, for example, evaluating the competitiveness of products, it is necessary, as in any scientific work, to carry out statistical data processing. The latter begins with determining the consistency of expert opinions, the numerical expression of which is the coefficient of concordance.
Why do we need an assessment of the consistency of expert opinions?
This assessment is necessary, first of all, because the opinions of experts can diverge greatly on the evaluated parameters. Initially, the assessment is carried out by ranking the indicators and assigning them a certain coefficient of significance (weight). Inconsistent ranking leads to the fact that these coefficients will be statistically unreliable. The opinions of experts, if necessary (more than 7-10), should be distributed according to normal law.
The concept of the coefficient of concordance
So. Consistency is a concordance. Coefficient is a dimensionless quantity showing the ratio in the general case of variance to maximum variance. Summarize these concepts.
The concordance coefficient is a number from 0 to 1, showing the consistency of expert opinions when ranking some properties. The closer this value is to 0, the consistency is considered lower. With the value of this coefficient less than 0.3, expert opinions are considered inconsistent. When the coefficient value is in the range from 0.3 to 0.7, the consistency is considered average. For values greater than 0.7, consistency is accepted as high.
Use cases
When conducting statistical studies, situations may arise in which an object can be characterized not by two sequences that are statistically processed using the concordance coefficient, but by several that are ranked accordingly with the help of experts having the same level of professionalism in a certain field.
The consistency of the ranking carried out by experts must be determined to confirm the hypothesis that the experts make relatively accurate measurements, which allows the formation of various groups in expert groups, which are largely determined by human factors, primarily such as differences in views, concepts, and various scientific schools, the nature of professional activities, etc.
Brief description of the rank method. Its advantages and disadvantages
When ranking, the rank method is used. Its essence lies in the fact that each property of an object is assigned its own specific rank. Moreover, each expert included in the expert group, this rank is assigned independently, as a result of which there is a need to process this data in order to identify consistency of expert opinions. This process is carried out by calculating the coefficient of concordance.
The main advantage of the rank method is ease of implementation.
The main disadvantages of the method are:
- a small number of ranking objects, since when their number is exceeded 15-20, it becomes difficult to assign objective rank estimates;
- Based on the use of this method, the question remains of how far in importance the objects under study are from each other.
When using this method, it is necessary to take into account that the ratings are based on some probabilistic model, therefore they must be used with caution, given the scope.
Kendall's Concordance Rank Rank
It is used to determine the relationship between quantitative and qualitative characteristics characterizing homogeneous objects and ranked according to one principle.
The determination of this coefficient is carried out according to the formula:
t = 2S / (n (n-1)), where
S is the sum of the differences between the number of sequences and the number of inversions by the second feature;
n is the number of observations.
Calculation Algorithm:
- The values of x are ranked in the order of either decreasing or increasing.
- The y values are arranged in the order in which they correspond to the x values.
- For each subsequent rank y, it is determined how many rank values exceeding it goes after it. They are added up and a measure of the correspondence of sequences of ranks in x and y is calculated .
- Similarly, calculate the number of ranks of y with lower values, which also add up.
- Add up the number of ranks with higher values and the number of ranks with lower values, as a result, obtain the value of S.
This coefficient shows the relationship between the two variables, and in most cases is called the Kendall rank correlation coefficient. Such a relationship can be graphically depicted.
Coefficient determination
How it's done? If the number of ranked signs or factors exceeds 2, use the coefficient of concordance, which, in essence, is a multiple version of rank correlation.
Be careful. The calculation of the concordance coefficient is based on the ratio of the deviation of the sum of squares of ranks from the average sum of squares of ranks times 12 to the square of experts multiplied by the difference between the cube of the number of objects and the number of objects.
Calculation algorithm
In order to understand where the number 12 in the numerator of the calculation formula comes from, let's look at the determination algorithm.
For each row with the ranks of a certain expert, the sum of ranks is calculated, which is a random value.
The concordance coefficient in general is defined as the ratio of the variance estimate (D) to the maximum value of the variance estimate (D max ). We give successively formulas for the determination of these quantities.
where r cf is the estimate of the mathematical expectation;
m is the number of objects.
Substituting the obtained formulas in the ratio of D to D max, we obtain the final formula for the coefficient of concordance:
Here m is the number of experts, n is the number of objects.
The first formula is used to determine the concordance coefficient if there are no related ranks. The second formula is used if related ranks are available.
So, the calculation of the coefficient of concordance is completed. What's next? The resulting value is evaluated for significance using the Pearson coefficient by multiplying this coefficient by the number of experts and the number of degrees of freedom (m-1). The obtained criterion is compared with the tabular value, and when the value of the former exceeds the latter, they speak of the significance of the coefficient under study.
If there are related ranks, the calculation of the Pearson criterion is somewhat more complicated and is performed by the following relation: (12S) / (d (m 2 + m) - (1 / (m-1)) x (T s1 + T s2 + T sn )
Example
Suppose that the expert method evaluates the competitiveness of butter sold in a retail network. Here is an example of calculating the coefficient of concordance. Before assessing competitiveness, it is necessary to rank the consumer properties of the product that are involved in the assessment. Suppose that such properties will be the following: taste and smell, texture and appearance, color, packaging and labeling, fat content, trade name, manufacturer, price.
We assume that the expert group consists of 7 experts. The figure shows the results of ranking these properties.
The average value of r is calculated as the arithmetic mean and will be 31.5. To find S, we summarize the squares of the differences between r is and r average, according to the above formula, and determine that the value of S is 1718.
We calculate the concordance coefficient using the formula without using related ranks (ranks would have been linked if the same expert on different properties had the same ranks).
The value of this coefficient will be 0.83. This indicates a strong agreement of experts.
Let's check its significance by the Pearson criterion:
7 x 0.83 x (8-1) = 40.7.
The Pearson tabular criterion at a 1% significance level is 18.5, and at 5% - 14.1. Both of these numbers are less than the calculated value, therefore, at a significance level of 1%, the calculated coefficient of concordance is taken as significant.
The example demonstrates the simplicity and accessibility of the calculation for any person who knows the basics of mathematical calculations. To facilitate them, you can use the forms of spreadsheets.
Finally
Thus, the coefficient of concordance shows the consistency of the opinions of several experts. The farther it is from 0 and closer to 1, the more consensus opinions. These coefficients must be confirmed by the calculation of the Pearson criterion.