Adversarial Sequence Mutations in AlphaFold and ESMFold Reveal Nonphysical Structural Invariance, Confidence Failures, and Concerns for Protein Design
Source: PubMed Central Open Access, NCBI / U.S. National Library of Medicine
AlphaFold has transformed structural biology and spawned an ecosystem of derivative tools for protein design, binding prediction, and drug discovery. However, whether AlphaFold has learned generalizable biophysical principles as opposed to template-based pattern matching remains unclear—a distinction critical for applications beyond its training context. Here, we perform a systematic adversarial evaluation of AlphaFold 3 using point and deletion mutations across 200 proteins. Remarkably, predicted structures remain invariant to mutations of up to 40% of residues—including deliberately destabilizing substitutions—and to deletions of 10%. Notably, this invariance holds even for experimentally validated fold-switching proteins that are known to adopt alternative conformations in response to such mutations, despite the fact that these proteins are small and monomeric—precisely the category where AlphaFold is expected to perform best. Confidence metrics prove unreliable, as they select the most accurate structure at most 35% of the time and consistently correlate with the structural quality of the best available training-set template. ESMFold exhibits greater, though still imperfect, mutational sensitivity, suggesting a tighter coupling between sequence identity and predicted structure that may reflect differences in training objective rather than overall model quality. These findings indicate that AlphaFold may rely heavily on memorized templates rather than biophysical reasoning
Abstract
AlphaFold has transformed structural biology and spawned an ecosystem of derivative tools for protein design, binding prediction, and drug discovery. However, whether AlphaFold has learned generalizable biophysical principles as opposed to template-based pattern matching remains unclear—a distinction critical for applications beyond its training context. Here, we perform a systematic adversarial evaluation of AlphaFold 3 using point and deletion mutations across 200 proteins. Remarkably, predicted structures remain invariant to mutations of up to 40% of residues—including deliberately destabilizing substitutions—and to deletions of 10%. Notably, this invariance holds even for experimentally validated fold-switching proteins that are known to adopt alternative conformations in response to such mutations, despite the fact that these proteins are small and monomeric—precisely the category where AlphaFold is expected to perform best. Confidence metrics prove unreliable, as they select the most accurate structure at most 35% of the time and consistently correlate with the structural quality of the best available training-set template. ESMFold exhibits greater, though still imperfect, mutational sensitivity, suggesting a tighter coupling between sequence identity and predicted structure that may reflect differences in training objective rather than overall model quality. These findings indicate that AlphaFold may rely heavily on memorized templates rather than biophysical reasoning, with direct implications for mutation-effect interpretation, confidence-guided model selection, and sequence optimization workflows.