Context: Mining patterns is one of the most attractive topics in the field of software design. Knowledge about the number, type, and location of pattern instances is crucial to understand the original design decisions. Several techniques and tools have been presented in the literature for mining patterns in a software system. However, evaluating the quality of the detection results is usually done manually or subjectively. This can significantly affect the evaluation results. Therefore, a fair comparison of the quality of the various mining methods is not possible. Objective: This paper describes a new benchmark to evaluate pattern mining methods in source code or design. Our work aims at overcoming the challenges faced in benchmarking in pattern detection. The proposed benchmark is comprehensive, fair, and objective, with a repeatable evaluation process. Method: Our proposed benchmark is based on automatic generation of testbeds using graph theory. The generated testbeds are the codes and their corresponding class diagrams in which various types of patterns and their variants are inserted in different locations. The generated testbeds differ in their levels of complexity and full information is available on the utilized patterns. Results: The results show that our proposed benchmark is able to evaluate the pattern mining methods quantitatively and objectively. Also, it can be used to compare pattern mining methods in a fair and repeatable manner. Conclusions: Based on our findings, it can be argued that benchmarking in the pattern mining field is significantly less mature than topics such as presenting a new detection method. Therefore, special attention is needed in the pattern evaluation topic. Our proposed benchmark is a step towards achieving a comparative understanding of the effectiveness of detection methods and demonstrating their strengths and weaknesses.

Keywords