The effects of apolipoprotein E (APOE) and Klotho genes, both implicated in aging, on human cognition as a function of sex and age are yet to be definitively established. Here, we showed in the largest cohort studied to date (N = 320,861) that APOE homozygous ε4 carriers had a greater decline in cognition with aging compared to ε3 carriers (ε3/ε4 and ε3/ε3) as well as smaller hippocampi and amygdala (N = 29,510). Critically, sex and age differentially affected the decline in cognition. Younger (40 to 50 y) female homozygous ε4 carriers showed a cognitive advantage over female ε3 carriers, but this advantage was not present in males. By contrast, Klotho-VS heterozygosity did not affect cognition or brain volume, regardless of APOE genotype, sex, or age. These cognitive trajectories with aging demonstrate clear sex-dependent antagonistic pleiotropy effects of APOE ε4, but no effects of Klotho genotype on cognition and brain volume.