The AGREE II evaluates the quality of CPGs, addresses how and what to present in published guidelines, and provides a methodology for the development of CPGs (6
). It is an important tool for education regarding the development of CPGs; it also allows developers to understand the strengths and weaknesses of their guidelines when evaluating their own or others' guidelines and to update their guidelines accordingly.
Although the history of CPGs in Korea is short, demand for the development of CPGs has increased. Indeed, nearly 120 guidelines are being planned by professional member societies of the KAMS (10
). Presently, CPG developers in Korea typically use an "adaptation process." Differences in the interpretations of evaluators during the process of applying the AGREE instrument in evaluations of the quality of existing CPGs is among the most difficult problems faced by CPG developers. For example, an AGREE II evaluation was performed on the 10 draft guidelines developed with regard to the CPGs for stomach cancer in Korea in 2011. Significant deviations were observed during this process, such as differences of 2 points or more on 6.6±3.5 (mean±SD) items (unpublished data), which reflect problems in the use of the AGREE instrument in a Korean situation. The reasons that Korean CPG developers and evaluators experience relatively greater difficulty using the AGREE tool include the absence of consensus among medical professionals regarding the relative levels of importance of the elements listed in the "how-to-rate" guidelines in the AGREE II manual, differences among medical environments, socio-cultural differences between Korea and Western nations, and subjective interpretations of each of the questions. Although AGREE is a widely accepted tool in the field of CPGs evaluation, it has the disadvantage that it can be affected by the subjective perceptions of evaluators (12
). In 2010, Dans and Dans (15
) noted that AGREE II items demand that activities be "described well" rather than be "be performed well," which causes confusion about the purpose of the evaluation and, ultimately, about the grading. As the Korean medical community does not yet have sufficient experience with CPGs, differing interpretations and understandings among evaluators constitute major obstacles. In Korea, almost every participant in the development and evaluation of CPGs is a medical doctor who has majored in medicine and has experience in clinical practice. Thus, majority of evaluators considered the quality of expected performance of recommendations in addition to the quality of description in the evaluation process. This may be another reason for the major differences among evaluators using the AGREE II instrument. Thus, it is expected that provision of an AGREE II scoring guide will facilitate the achievement of consensus about the purpose of and the approach to the development of CPGs.
ECC attempted to provide anchor points for scores of 1, 3, 5, and 7 based on importance and a quantitative measure of satisfaction after agreeing on the standards for a seven point scale. These were based on the "How to Rate" instructions for the AGREE II instrument because these provided a good description of the purpose and the content of each item. In the first round of the Delphi consensus process, however, we could not reach consensus on the anchor point 3 for AGREE II item numbers 2, 5, 11, and 12 (). Thus, it was difficult to identify four phased anchor points based on the "How to Rate" guidelines alone. In these cases, we tried to provide standards that considered being "be performed well."
To identify problems, the guide was applied to several recently developed domestic CPGs. Although no major differences among evaluators were observed, the data are not reported here because too few guidelines were involved. This scoring guide will be further organized and modified in the process of actually applying it to diverse CPGs, and it will be revised to reflect further developments in CPGs and medical environments in Korea. The scoring guide for the AGREE II instrument proposed herein can be used to evaluate previously developed CPGs in Korea and is a useful foundation for the creation of new CPGs in the future.