A fully non-linear numerical wave tank (NWT), based on Computational Fluid Dynamics (CFD), provides a useful tool for the analysis of coastal and offshore engineering problems. To generate and absorb free surface waves within a NWT, a variety of numerical wave maker (NWM) methodologies have been suggested in the literature. Therefore, when setting up a CFD-based NWT, the user is faced with the task of selecting the most appropriate NWM, which should be driven by a rigorous assessment of the available methods. To provide a consistent framework for the quantitative assessment of different NWMs, this paper presents a suite of metrics and methodologies, considering three key performance parameters: accuracy, computational requirements and available features. An illustrative example is presented to exemplify the proposed evaluation metrics, applied to the main NWMs available for the open source CFD software, OpenFOAM. The considered NWMs are found to reproduce waves with an accuracy comparable to real wave makers in physical wave tank experiments. However, the paper shows that significant differences are found between the various NWMs, and no single method performed best in all aspects of the assessment across the different test cases.