Seasonal influenza viruses evolve to evade host antibody responses, a process known as antigenic drift. However, we have limited knowledge of how epitope-specific antibodies drive antigenic drift and how these mutations impact viral fitness. We previously identified first exposure to the 2009 pandemic H1N1 virus robustly induced antibodies against the receptor binding site (RBS) of the hemagglutinin (HA) head domain. Now, we identified that RBS-specific antibodies specificities drove antigenic drift. We identified RBS-specific monoclonal antibodies (mAbs) drove mutations at residues 186 and 190, which mediate sialic acid binding. We identified that S186P has been fixed into H1N1 viruses since 2019, whereas D190A emerged around 2020 and has since disappeared. We found that S186P increased the binding breadth of HA to a2,6 sialylated glycans, whereas D190A reduced binding breadth. Molecular dynamic simulations show that S186P increases the flexibility of RBS, which may allow for expanded binding breadth. As D190A ablates a critical sialic acid contact, we propose this mutation was selected to evade RBS-specific antibodies but due to reduced receptor binding, D190A variants were counter-selected. Together, our studies define how RBS-specific antibodies can drive antigenic drift and how mutations of sialic acid contacts can alter receptor binding.