Kipoi Seminar - Abdul Muntakim Rafi (University of British Columbia)

Title: Detecting and avoiding homology-based data leakage in genome-trained sequence models Abstract: Models that predict function from sequence have become critical tools in deciphering the functional roles of genomic sequences and genetic variation within them. However, traditional approaches for dividing the genomic sequences into training data, used to create the model, and test data, used to determine the model’s performance on unseen data, fail to account for the widespread homology that permeates the genome. Using models that predict human gene expression from DNA sequence, we demonstrate that model performance on test sequences varies by their homology with training sequences, consistent with a form of ‘data leakage’ that inflates model performance by rewarding overfitting of sequences that are also present in the test data. We also show that for test sequences that share high homology with training data, predictions of gene expression can be accurately made simply by averaging the outputs from the most similar sequences in the training set, underscoring the issue of having homologous sequences across train-test sets. Furthermore, we observe that neural networks fail to generalize when predicting the effects of mutations, with larger expression changes predicted for unseen sequences compared with seen sequences. This issue is particularly concerning because many GWAS SNPs have doppelgangers of alternate alleles present elsewhere in the genome, often multiple times, and may be inadvertently included in the training data, compromising the reliability of model-predicted effects of genetic variation. To prevent leakage in genome-trained models, we introduce ‘hashFrag,' a scalable solution for partitioning data with minimal leakage. Altogether, we address a fundamental challenge in creating appropriate train-test set splits for sequence-based models on genomes, and highlight the consequences of failing to do so.