Objective: Despite a generally good prognosis, many prostate cancer survivors have poor quality of life (QOL). A greater understanding of how psychological appraisals influence QOL is merited given their potentially modifiable nature. In this study, we considered how elements of survivors' retrospective and prospective appraisals relate to QOL. Methods: A total of 1229 prostate cancer survivors between 2 and 5 years post‐diagnosis, identified from a population‐based National Cancer Registry, were asked questions on their socio‐demographics, health, treatment received, and adverse‐effects using a cross‐sectional design. QOL was assessed using the EORTC QLQ‐C30. Retrospective appraisals were assessed by asking survivors to reflect on their experience of treatment‐related adverse‐effects compared with their prior expectations. A fear of recurrence scale assessed prospective appraisals of future disease course. A multiple regression model explored the impact of psychological appraisals on QOL, after controlling for socio‐demographic, treatment, and health‐related factors. Results: The model was significant explaining 37% of variance in QOL. The strongest associate with QOL was fear of recurrence (β = −.29; P < .001). Survivors who experienced side effects that were worse than expected had significantly lower QOL (β = −.10; P = .002). Other significant correlates of lower QOL were presence of comorbidities, having undergone a less invasive treatment, and having more advanced disease. Working at diagnosis and having a higher level of education were significantly associated with higher QOL. Conclusions: Results suggest both retrospective and prospective appraisals are independently related to QOL in prostate cancer. Providing survivors with more information about possible adverse effects of treatment, as well as providing appropriate information regarding future disease progression, may improve QOL.