Bland-Altman Agreement Analysis in Laboratory Research

Use of Bland-Altman agreement analysis in laboratory research: A survey of current reporting standards.

Introduction-

Advances in technology have led to development of new instruments and measurement devices in field of medicine. The clinicians and researchers often need to compare a newer method of measurement with an established one, to check for interchangeability. While assessing for interchangeability the emphasis should be on testing how well two methods agree with each other. Earlier Pearson’s product-moment correlation coefficient was used as a measure of agreement[R]. However the approach was inappropriate as this coefficient merely indicated association rather than agreement [R]. Hence Bland and Altman in their series of publications[R] stressed on quantification of bias. They provided a simpler and visually attractive plot for agreement analysis of continuous variables measured on the same scale.[R]

After its introduction to medical literature in 1983, the Bland-Altman’s (B-A) method [R] is one of most commonly used statistical method for agreement analysis. The method is extensively used in evaluating the agreement of laboratory analytes, physiological variables, newer instruments and other devices.

B-A method[R] advocates the construction of a scatter plot, where the absolute difference between the paired measurements is plotted on y-axis against the mean of two methods on x-axis. The SD of differences between paired measurements is then used to construct 95% limits of agreement (as ± 1.96 SD). The 95% of differences between paired measurements are expected to lie between these upper and lower LOA. The conclusions on agreement and interchangeability of two methods are then made based upon the width of these LOA in comparison to a priori defined clinical criteria[R]. The plot also enables the researcher to visually assess the bias, data scatter and the relationship between magnitude of difference and size of measurement. Often in biologic systems data scatter and the magnitude of differences increases proportionally to the size of the measurement (hetero-scedastic distribution). Bland and Altman recommended the logarithmic or percentage transformation of data in case of hetero-scedastic distribution and then constructing B-A plot with transformed data[R] instead of classical absolute difference plot.

Contrary to conventional statistical hypothesis testing, the output of B-A analysis consists of bias and LOA, both of which are estimates[R]. The estimates have inherent risk of sampling error and hence the authors suggested calculation of confidence interval (CI) of bias and LOA. The method also advocated the collection of data in replicates. Replicates are defined as two or more measurements on the same individual by the same method, taken in identical conditions. Replicates enable the comparison of the agreement between the two methods with the agreement each method has to itself (repeatability) [R] B-A also advocated for sample size calculations on in method comparison studies[R].

Despite its simplicity and frequent use in clinical laboratory research, the B-A method is not properly interpreted and reported in medical literature. Studies [R]conducted a decade ago highlighted poor reporting standards of B-A method, however there is paucity of current information on the same. Furthermore, uniform statistical reporting of results not only increases the generalizability of results, but also facilitates the inclusion of studies in systemic reviews and meta-analysis. Hence the aim of study was to review the current reporting standards of B-A method in laboratory research in medical literature.

Material and methods-

Three researchers (VC, RB, and SK) participated in this study. All researchers were qualified health professionals. VC and SK had previous experience of publishing laboratory research [R] with use of B- A agreement analysis.

Eligibility criteria- Studies which tested agreement of laboratory analytes with continuous measurements, as per B-A methodology were included.

Literature search- A thorough search of PUBMED, MEDLINE and GOOGLE SCHOLAR was conducted for studies published in years 2012 and 2013. The search strings used to search potential studies were “Agreement analysis” AND/OR “Bland Altman analysis” (MeSH) and “Laboratory analytes” and “clinical biochemistry” (MeSH). Included studies were evaluated according to Bland and Altman methodology on a predesigned checklist. The studies were evaluated for following 8 items: (1.) Measures of repeatability (2.) Representation and correct definition of LOA (3.) Correct representation of x-axis on BA plot (4.) Reporting of CI of LOA (5.) Comparison of limits of agreement with a priori defined clinical criteria (6.) Evaluation of pattern of relationship between difference (y-axis) and average (x-axis) (7.) Use of logarithmic or percentage conversion of data in case of heteroscedastic relationship between the difference and average (8.) Sample size calculations. Each item on the checklist was rated as ‘Yes’or ‘No’. We also recorded the data on use of other statistical methods for testing of agreement. However, we did not perform detailed evaluation of included studies for other statistical methods of agreement.

To ensure accurate data retrieval, each included study was evaluated twice by one author (VC) and data recorded on predesigned checklist. Opinion was taken from second author (SKK) in case of confusion arising during data extraction. We compared the results of our study with 3 similar surveys done earlier.

Results-

A total of 156 studies were screened for potential inclusion in the study. A total of 50 studies, were retrieved and included in the final study. The 38% of included studies were published in journals of various streams of internal medicine, while 30%, 26% and 6% were published in journals of laboratory medicine, emergency medicine, anaesthesia respectively. Results of survey and its comparison to three previous studies are as shown in Table-1.The other statistical methods used in addition to B-A plot in included studies were correlation coefficient (70%), Deming Regression(14%), Passing Bablok regression (14%), linear regression (24%), Lin’s Concordance (8%), Sensitivity specificity analysis (16%), Interclass correlation coefficient (6%), Grid error plot (10%), Critchley polar plots (2%).

Discussion-

Use of B-A for method comparison has increased in recent years with most of authors using it for analysing agreement. The original paper on agreement analysis by B-A[R] is among one of most cited statistical publication, with more than 34000 citations. Although claimed as a method which is simpler to perform and interpret, the method is often used and interpreted without proper understanding. Review by Berthelsen et al[R] in 2006 and earlier studies[R] demonstrated unsatisfactory reporting of B-A analyses, in anaesthesiology literature. Williamson et al[R] proposed a method of meta-analysis of method comparison studies, however authors also highlighted the problem of non-uniform reporting of studies. [R]

Twomey et al [R] suggested use of method hierarchy for selection of x-axis and advocated use of gold standard method as x-axis in B-A plot. However Bland and Altman statistically proved that use of any single method instead of average of two methods as x-axis is misguided and leads to misinterpretation[R]. Results of our study suggest that 94% of studies reported x-axis correctly, which is almost similar to results of earlier studies conducted by Mantha et al (94%) [R] and Dewitt et al (87%) [R]. although most method comparison computer softwares (analyse it, Graphpad Prism, EP evaluator) automatically select x-axis as mean of two methods, errors in selection of x-axis are still noticed.

The 95% LOA were correctly defined and drawn in 94 %( 47) of included studies. Further among 47 studies with correct definition of LOA, the 3 studies interpreted LOA wrongly concluding good agreement because 95 % of differences were present in-between upper and lower LOA. The 95 % LOA are in-fact drawn so as to contain 95% of differences between them. It is not LOA per se, but width of these LOA in comparison to a priori defined clinical criteria that conclusions regarding agreements can be made. The decision on acceptable differences between two methods is primarily clinical rather than statistical. Earlier studies by Dewitte et al [R]and Mantha et al[R] had shown that comparison with pre-defined clinical criteria was missing in >90% of studies. Total 74 % of authors in our study commented on agreement on basis of predefined clinical criteria which represents a significant improvement in reporting standards. The specifications for clinical acceptance criteria of laboratory analytes have been provided as by Ricos et al[R], CLSI[R], and West guard QC[R]. Alternatively a Delphi survey (expert opinion) can be done to determine acceptable limits before instituting study.

The CI limits of LOA were reported in only 6% of included studies in our study. The LOA are estimates and reporting LOA without CI is equivalent to reporting a sample mean without its CI. The CI limits [Ludbrook et al] represent the range within which a single, new, observation taken from the same population would be expected to lie. Although strongly recommended by B-A[R], and subsequently proved by a simulation study conducted by Hamilton et al[R], the statistical reporting of CI of LOA has remained poor (Mantha et al-2.6%) [R].

Although recommended by B-A method, the pattern of relationship between difference and wider concentration range is rarely evaluated[R]. Drawing difference plot with parallel LOA in datasets with heteroscedastic scatter makes LOA wider in lower concentration range and narrower in higher concentration range thus affecting validity of interpretation. [R] Bland and Altman [R]proposed logarithmic transformation of data with heteroscedasticy and then constructing difference plot against average of two methods using log transformed data. For meaningful understanding of LOA, they suggested back-transformation (antilog) of the log transformed data. Alternatively[R] plot of ratios of two methods or percent difference can be plotted against average of two methods for simpler interpretation. Transformation of data usually renders the scatter of differences as uniform (Homoscedastic). Twomey et al[R] recommended the drawing up of funnel shaped or V shaped LOA instead of classical parallel LOA in data sets with heteroscedastic scatter. Another option is breaking the data into smaller subsets and then analysing these subsets with absolute difference plot to make conclusions. [Twomey et al] We observed that only 28 % of studies made an attempt at evaluation of pattern of scatter. Rest of authors did not comment on pattern thus affecting the validity of results.

Another important problem noticed was lack of assessment of repeatability (38%), a practice that has not shown any substantial improvement Table-1. Conclusions drawn from studies without repeatability assessment are likely to be uncertain. Assessment of errors of the two methods (repeatability) enables the construction of the worst-case acceptable LOA. [R] With poor repeatability of one or both methods, the agreement between the two methods is bound to be unacceptable. [R]

Sample size calculations were done in only 15 studies. Lack of power and sample size analysis reduces validity of results. Different researchers have proposed sample size calculation for method comparison studies using Bayesian[R], regression[R], or concordance [R] approach. However Stockl et al[R] proposed an approach incorporating CI of LOA and predefined error limits in B-A plot. The approach is simple and allows for visual interpretation of appropriate sample size, as the classical B-A plot provides for agreement.

Despite a lot of research on B-A method in field of statistics, the uptake of the method in medical research has been slow. While efforts are on in statistical community for use of modifications of B-A plot in special situations like repeated measure studies[R] or using bar charts in B-A plots with limited value ranges[R], unfortunately reporting standards of classical B-A method among medical community are unacceptable. Guidelines “Reporting reliability and Agreement Studies (GRRAS)” were published as a guide to appropriate reporting of reliability and agreement studies. We found unsatisfactory reporting of B-A analysis in our study.