Dr Justin Zook - Developing Standards for Benchmarking Genetic Variant Detection -Genome In A Bottle

In Episode 6 of the Rising Tide Podcast, Dr. Julie M. Eggington (Co-Founder of the Center for Genomic Interpretation) interviews Dr. Justin Zook, the Co-Leader of the Biomarker and Genomic Sciences Group at the National Institute of Standards and Technology (NIST) in the USA. This 7/7/2022 interview is a conversation about the development of standards for benchmarking genetic variant detection, focusing on Genome in a Bottle (GIAB). The GIAB Consortium was organized to deeply characterize human genome samples for benchmarking mutation detection. The group is working hard to further characterize structural variation and challenging regions of the genome to further assist laboratories and academic facilities utilizing GIAB. WHAT HEALTH INSURANCE PAYERS SHOULD KNOW About The differences Between Labs "Not all genetic tests are the same and part of that is by their design. For example, there are tests that focus on a very narrow set of variants or genes. You'd miss any variants that are in other genes that might be important for that condition. That's one of the things to understand - that there are trade offs there in what you're going to detect and what you're not going to detect based on how targeted your sequencing or method is." "Genetic tests may have different degrees of accuracy within these genes. So even if they cover the same genes, one test might cover pretty much any type of variant in that gene, another test might cover most of the gene but miss a really hard part of the gene due to limitations of the method that they're using." [Note from CGI Co-Founder, Julie Eggington, Ph.D. "Labs can have CLIA, CAP, or NY State approvals, and their tests can be marketed the same, yet test accuracy can drastically vary between the labs in hidden ways."] About Labs using old Human Genome References "Anytime you change any part of your process, you have to do re-validation. Many clinical laboratories have not seen sufficient value to change to previous new versions of the reference. As part of the Telomere-to-Telomere (T2T) Consortium, we are working on a few ways to enable labs to improve results for genes that are fixed in the new T2T reference." Standard performance metrics can be misleading for Next Generation Sequencing "Essentially what specificity is trying to measure is your false positive rate. In this context, false positive means an incorrect variant that you detect." Traditional measures of specificity can be misleading in the genetic testing space: "The primary limitation of specificity is that there are so many true negatives in the genome because only 1 out of 1000 bases has a variant in it. If you count every base that doesn't have a variant in it as a negative, then your specificity usually ends up being 99.999%. It looks really good and sometimes it is really good but it's hard to tell at face value whether that's good or not. An easier to understand metric might be the number of false positives per million bases tested." WHAT CLINICAL LABS SHOULD DO TO IMPROVE GENETIC TEST ACCURACY Utilize T2T Reference Genome and Resources The T2T reference is the first complete human genome, and is publicly available for use by clinical laboratories. It can improve accuracy in medically relevant genes: "If a clinical laboratory has their list of genes that they're testing, one of the things that they can do is look at a companion T2T paper that examined where variant detection is improved with this new reference. You can look at the lists of genes in the supplementary tables and see if any of those are ones that you're currently including." Use GIAB Samples, but Understand The Limitations "Labs should not only be using GIAB samples for their validation. Pre-analytical steps are really important as well, which are not assessed by NIST's Reference Material DNA, but can be partly assessed by other products using GIAB samples (e.g., in FFPE). Note that GIAB samples will not entirely capture the variability that is common across hospital pathology labs and therefore should not be the only source of validating pre-analytical steps." "We stop at the analytical performance, or how well you are detecting these variants. Rather than assessing interpretation of specific clinical variants, GIAB samples have a large number of variants that help assess analytical performance for different variant types and difficult genome contexts." "GIAB evaluates each benchmark to minimize errors using a variety of high-performing assays to confirm variant calls. We don't call our benchmarks truth sets because we know they are not truth with a capital T, in the sense that they are absolutely perfect across all of the regions that we have defined. That is part of why we release new versions of benchmarks over time, so that as new methods are developed or as clinical laboratories or others report potential errors to us, we can improve those over time." ALSO, learn about what GIAB consortium is working on.