Real-time prediction of free-surface elevation or wave excitation force is necessary for a variety of applications, including noncausal optimal control of wave energy converters and detection of quiescent periods for offshore operations. Prediction methods can use past measurements at the point of interest, treating the wave elevation as a time series (looking backward). Alternatively, wave elevation values can be recorded at a set of locations about the point of interest and propagated in time and space through physical or statistical models (looking forward). In this paper, assuming Gaussian waves, a unified framework is proposed, which treats any combination of wave elevation values at various points in time and space as a Gaussian vector and covers both “backward” and “forward” approaches. It is shown that, using any given combination of measurement points in time and space, the optimal predictor, in a least mean square sense, is linear for any time horizon and can be directly derived from the wave spectrum, provided that the latter is known. The associated error can be readily calculated, thus providing, based on the wave spectrum, an upper bound on the predictability of the wave elevation process. In addition, the applicability of the spectrum-based predictor to real-time wave elevation forecasting is discussed.