All areas of research are increasingly confronted with demands for ‘accountability’, for the implementation of performance measures or other means of ensuring ‘value for money’. In particular for research funding, a researcher’s demonstration of ‘high quality’, if not ‘excellence’ – of his or her individual achievements and working environment – are preconditions sine qua non. The European Educational Research Quality Indicators (EERQI) project was motivated by this development and the assumption that it may cause undesired side effects – not least, because many methods and instruments which are applied in order to detect ‘quality’ seem to lack in quality themselves. Within the framework of the EERQI-project, this assumption was examined from different perspectives. Our contribution presents one of the EERQI-approaches, namely the attempt to develop and evaluate an instrument for peer review purposes. We describe the process from the first attempts to develop a set of criteria, which most likely refer to the quality of an educational research publication, to the final evaluation of a peer review questionnaire with criteria which are widely accepted and shared in the educational research community.