Reproducibility in Research

Reproducibility is a pillar of research. Studies must be reproducible to have merit. This is for good reason, as research is foundational, one study builds upon another and so on with each group contributing their piece to the overall puzzle. At least this is the way it is supposed to work. However, when research is reported erroneously, it creates a tremendous waste of time and resources for the entire research community working on the same application. While concerns about reproducibility have been discussed for years, the problem more recently has escalated to what is frequently called a “reproducibility crisis”.

The Reproducibility Crisis

While most people have certainly read of the journal publication that is later found to be not reproducible, several articles in recent years have drawn attention to how pervasive the lack of reproducibility is. In Monya Baker’s, 2016 Nature article, 1,500 scientists lift the lid on reproducibility, she reveals the results of a Nature survey of 1,576 researchers who filled out an online questionnaire on reproducibility in research¹. The results shared by Baker show that “more than 70% of researchers surveyed have tried and failed to reproduce another scientist’s experiments and more than half have failed to reproduce their own experiments. The article is very enlightening about how widespread the problem is and some key areas that researchers identify as problems for reproducibility.

While the survey in Baker’s article included researchers from many disciplines, a 2015 analysis of past studies in PLoS Biology, The Economics of Reproducibility in Preclinical Research, focused on issues of reproducibility in Life Science research². In the article the authors” analysis of past studies indicates that “the cumulative (total) prevalence of irreproducible preclinical research exceeds 50%, resulting in approximately US$28,000,000,000 (US$28B)/year spent on preclinical research that is not reproducible”. In the analysis they also identify four main categories of errors that cause preclinical irreproducibility:

• Biological reagents and reference materials (36.1%)
• Study design (27.1%)
• Data analysis and reporting (25.5%)
• Laboratory protocols (10.8%)

From this analysis the largest cause of irreproducibility is problems with the reagents and antibodies used in the studies.

Low reproducibility delays or prevents development of therapeutic drugs, wastes precious research funding and the time of countless researchers relying on this information to inform future work. Below are some highlights of the challenges and possible solutions to improve antibody quality and study reproducibility.

Reagent and Antibody Articles Quality

There are several factors that make evaluating research antibodies difficult, it is important to consider reagent quality, specificity and consistency. Most researchers get their antibodies from a vendor; however what they are finding is that antibodies vary widely on quality and validation methods. Antibodies against the same target can produce different results depending on the supplier and lot-to-lot.

The primary problems with antibodies are:

• Variability – both among suppliers and lot-to-lot.
• Non-specific – cross reactivity/off target binding.
• Incorrect application use – antibody is being used in an application it wasn’t intended for.

Antibody Validation

Antibody validation and testing should be done prior to using an antibody in a study. It is designed to validate that an antibody is consistent, specific, and selective for the application in which it will be used. There are a number of methods available to qualify an antibody including western blotting, immunoprecipitation, ELISAs, and immunochemistry. In addition there are a number of new technologies now available including knockout, knockdown, peptide array and mass spectrometry. There is unfortunately no “one size fits all” test for antibody validation and a thorough validation often requires a combination of techniques.

In my research on this topic, I discovered a very useful white paper, Raising antibody standards. The paper, published by Abcam, included a very informative table that concisely describes the benefits and limitations of several key antibody validation methods. I have included it as Table 1 with Abcam’s permission.

Why is antibody validation difficult?

First, it can be extremely time consuming for a lab to conduct thorough validation and often students or lab technicians aren’t properly trained on the validation protocols to conduct the work. Also, some journal publications are missing important information in the methods and materials section. Without information about protocols and which specific reagents were used (including the vendor), it is often difficult for another lab to reproduce the results.

What can be done?

Researcher Responsibilities

Researchers must remember, “buyer beware,” when sourcing their antibodies. It is important that they consider the methods that vendors are using to validate their antibodies. It is also important to understand the attributes of the product and the specific application it was designed for.

Researchers must also take the time to validate the antibody internally for their own specific application. Study design was the second highest issue in the reasons for irreproducibility. It is important to always include the proper controls to address experimental variables but also to know where to go when there are errors. Lastly, share detailed information about your methods in your publications including the antibody, supplier and clone used so that others may be able to more easily reproduce your results.

Vendor Responsibilities

Vendors need to properly characterize their antibodies to ensure quality, consistency and specificity. They also need to clearly label the antibody’s intended application.

Some scientists are asking that vendors do even more. In a 2015 Nature article, “Reproducibility: Articlesize antibodies used in research,³” the authors, Andrew Bradbury and Andreas Plückthun and their 110 co-signatories stated that protein-binding reagents must be defined by their sequences and produced as recombinant proteins. While this kind of a change in antibody validation and manufacturing would reduce variability greatly, it would also impact the range of antibodies available, could impact cost of the antibodies and would require the support and buy-in of antibody suppliers.

One Vendor’s Approach

In reaching this article, I was able to interview Bruce Hamilton, Head, Antibody Characterization at Abcam. We discussed the issue of reproducibility and what Abcam is doing to address the quality of their antibodies. He shared with me some key approaches.

First Dr. Hamilton talked about the importance of validating antibodies and confirming specificity. Abcam has a knock out validation program that uses human knockout cell lines generated from haploid cellular models using CRISPR/Cas9 technology from Horizon Discovery. He explained that knockout models provide a true negative control, which makes this model a great choice for antibody validation. He shared that Abcam has knockout validated over 1,000 antibodies to ensure they recognize specifically the protein of interest.

Bruce went on to explain that despite the strengths of knockout validation, it is still an indirect method for determining antibody specificity, which is why it is important to use multiple validation methods. In addition to knockout validation, Abcam confirms quality using a variety of methods including: ICC/IF, IHC, flow cytometry, ELISA, IP, chromatin IP (ChIP), and peptide array. Abcam also follows up on feedback from customers and if any issues arise they submit those products for further testing. If the product fails further testing, it is removed from the catalog and customers are contacted.

The aspect of Abcam’s approach that I was most interested in was their commitment to transparency. In our interview, Bruce was enthusiastic about ensuring that all relevant information and data about Abcam products was available to customers. They share data generated with clones and provide literature citations for that particular antibody to permit researchers to see the history of the antibody and the applications it has been used for.

They also encourage an open review policy on their website called Abreviews where researchers can post positive and negative reviews about a product. This review program allows scientists to share information about how the products worked in specific applications and species. Abcam can also use the reviews as a tool to identify products that may have issues. This is something we have all used in our everyday life (I rarely buy anything electronic these days without a thorough investigation of Amazon reviews) and it was nice to see it being used in an area where feedback is so important to successful studies.

Abcam also partners with academia to help validate antibodies for different applications and Bruce said that academia has been enthusiastic in their support of the program. The program fosters collaboration to ensure that these antibodies are working as expected and encourages discussion of reagents that aren’t working, which is equally important.

Lastly, Bruce discussed the importance of implementing new manufacturing technologies in generating antibodies. Abcam has embraced recombinant technology and many of their antibodies are recombinant, which avoids hybridoma cell drift and batch-to-batch variation. They currently offer over 10,000 recombinant monoclonal antibodies. Their recombinant antibody technology is also applied to custom projects and for diagnostic use.

Summary

It is clear that reproducibility is a major issue in drug discovery and disease research today. It is also clear that it is a complicated problem without a clear, simple solution. I believe that researchers and vendors can work together to create collaborations that further antibody validation and that transparency is a key part of the solution. Scientists need to be transparent about their methods so that their results may be reliably reproduced. They also need to report what doesn’t work, which can be just as important as reporting what does. Vendors need to be transparent in their manufacturing and validation methods and need to be willing to work with the research community to better meet their needs.