This study tested whether listeners adapt to speaker-specific prosody in anticipatory processing, and, if so, whether adaptation is modulated by belief about the speaker’s intention. In three visual-world eye-tracking experiments, we compared how listeners responded to proper uses of contrastive accent on an adjective-noun pair (e.g., First, find the red cat. Next, find the PURPLEL+H* cat) in Experiment 1 to deviant or improper uses of contrastive accent (e.g., First, find the red cat. Next, find the PURPLEL+H* pig) in Experiments 2 and 3. Experiment 1 confirmed that proper uses of contrastive accent lead anticipatory looks to the target object. Experiment 2 showed no anticipatory effects of contrastive accent when the adjective accent was deviant. Experiment 3, where participants were informed before the experiment that the speaker was not trustworthy, showed participants learned to anticipate the upcoming referent (pig) with improper contrastive accent over the course of the experiment.