Influenza B viruses (IBV) were first identified in humans over 80 years ago and contribute to annual epidemics. Although IBV evolves at a lower rate than IAV, antigenic changes in the IBV haemagglutinin (HA) impede vaccine efficacy. Despite considerable phylogenetic analyses, experimental mapping of the IBV HA antigenic space and its molecular basis remain limited. Such information could enhance surveillance and vaccine design and improve our fundamental understanding of IBV evolution. We applied antigenic cartography using haemagglutination inhibition data from 50 ferret anti-sera and 253 IBV isolates from 1940-2021 and combined it with phylogenetic analyses of >400 IBV HA sequences. We paired amino acid sequence and antigenic data to identify antigenic cluster-defining mutations on the HA, which were experimentally tested using site-directed mutagenesis and reverse genetics. We identified 13 antigenic clusters spanning the Ancestral, B/Victoria and B/Yamagata lineages. Cluster-defining amino acid substitutions were limited to a small number of positions proximal to the receptor binding site, including previously uncharacterised positions and epistatic interactions. The effects of cluster-defining mutations on viral fitness were assessed in air-liquid-interface differentiated human airway cell cultures. Antigenic mutations on their own often had a fitness cost, but compensatory mutations were able to rescue viral fitness, suggesting the presence of genetic barriers in IBV HA antigenic evolution. The recycling of antigenic mutations and phenotypes may indicate limits of the IBV HA antigenic space. Our results also suggest the existence of genetic barriers that may explain the lower antigenic drift rate of the IBV HA. Further deciphering the molecular basis of IBV antigenic evolution and the limits of the IBV antigenic space could help predict the emergence of new antigenic variants and inform future vaccine design.