Successively emerging SARS-CoV-2 variants lead to repeated epidemic surges through escalated transmissibility (i.e., fitness) in the human population. Modeling genotype-fitness relationship enables us to pinpoint the mutations boosting viral fitness and flag high-risk variants immediately after their detection. Here, we introduce CoVFit, a protein language model able to predict the fitness of variants based solely on their spike protein sequences. CoVFit was trained with genotype-fitness data derived from viral genome surveillance and functional mutation data related to immune evasion. CoVFit identified 549 fitness elevation events throughout SARS-CoV-2 evolution until late 2023. CoVFit successfully predicted the higher fitness of variants that emerged after training data creation through sequential or non-sequential evolution. Furthermore, a CoVFit-based simulation successfully predicted which mutations would be acquired next by the exisiting variants. Our study provides both insight into the SARS-CoV-2 fitness landscape and a novel tool potentially transforming viral genome surveillance.