Three visual-world eye-tracking experiments investigated how listeners responded to proper conventional (Exp.1) and unconventional (Exp.2-3) uses of contrastive accent. Exp.1 confirmed that conventional uses of contrastive accent lead anticipatory looks to a target object. Exp.2 showed no anticipation when contrastive accent was consistently misleading. When participants were told before the experiment that the speaker was not trustworthy in Exp.3, they learned to predict the upcoming referent with improper contrastive accent. The results suggest that listener’s beliefs about the speaker’s intentions modulates the degree of prosodic adaptation.