Outcome measures play an extremely important role in clinical trials and observational research. Outcome measures for rheumatoid arthritis cover a whole array of domains, ranging from measures describing the inflammatory process to measures describing the ultimate consequences of long-term disease, such as joint damage, physical function and quality of life. There is a scientific need to be able to quantify what is called the 'severity of rheumatoid arthritis', so that patients with rheumatoid arthritis can be clustered according to their propensity to develop an unfavourable outcome. It is a challenge to find an appropriate measure for severity. One attempt has been the development of the Rheumatoid Arthritis Medical Record-Based Index of Severity. This commentary elaborates on how such a measure of severity should be validated to determine whether it is appropriate for practical use.