The apolipoprotein E (ApOE) ε4 allele is associated with neuropathological buildup of amyloid in the brain, and with lower performance on some laboratory measures of memory in some populations. In two studies, we tested whether ApOE genotype affects memory for everyday activities. In Study 1, participants aged 20–79 years old (n = 188) watched movies of actors engaged in daily activities and completed memory tests for the activities in the movies. In Study 2, cognitively healthy and demented older adults (n = 97) watched and remembered similar movies, and also underwent structural MRI scanning. All participants provided saliva samples for genetic analysis. In both samples we found that, in older adults, ApOE ε4 carriers demonstrated worse everyday memory performance than did ε4 noncarriers. In Study 2, ApOE ε4 carriers had smaller medial temporal lobes (MTL) volumes, and MTL volume mediated the relationship between ApOE genotype and everyday memory performance. These everyday memory tasks measure genetically determined cognitive decline that can occur prior to a clinical diagnosis of dementia. Further, these tasks are easily administered and may be a useful clinical tool in identifying ε4 carriers who may be at risk for MTL atrophy and further cognitive decline that is a common characteristic of the earliest stages of Alzheimer’s disease.