Detection of design pattern occurrences is part of several solutions to software engineering problems, and high accuracy ofdetection is important to help solve the actual problems. The improvement in accuracy of design pattern occurrence detection requiressome way of evaluating various approaches. Currently, there are several different methods used in the community to evaluateaccuracy. We show that these differences may greatly influence the accuracy results, which makes it nearly impossible to compare thequality of different techniques. We propose a benchmark suite to improve the situation and a community effort to contribute to, andevolve, the benchmark suite. Also, we propose fine-grained metrics assessing the accuracy of various approaches in the benchmarksuite. This allows comparing the detection techniques and helps improve the accuracy of detecting design pattern occurrences.

We present a dependency-driven parser that parses both dependency structures and constituent structures. Constituency representations are automatically transformed into dependency representations with complex arc labels, which makes it possible to recover the constituent structure with both constituent labels and grammatical functions. We report a labeled attachment score close to 90% for dependency versions of the TIGER and TüBa-D/Z treebanks. Moreover, the parser is able to recover both constituent labels and grammatical functions with an F-Score over 75% for TüBa-D/Z and over 65% for TIGER.

This study presents new language and treebank independent

graph transformations that improve accuracy in data-driven dependency parsing. We show that individual generic graph transformations can increase accuracy across treebanks, but especially when they are combined using established parser combination techniques. The combination experiments also indicate that the presumed best way to combine parsers, using the highest scoring parsers, is not necessarily the best approach.

This paper presents a freely available evaluation tool for dependency parsing, MaltEval (http://w3.msi.vxu.se/users/jni/malteval). It is flexible and extensible, and provides functionality for both quantitative evaluation and visualization of dependency structure. The quantitative evaluation is compatible with other standard evaluation software for dependency structure which does not produce visualization of dependency structure, and can output more details as well as new types of evaluation metrics. In addition, MaltEval has generic support for confusion matrices. It can also produce statistical significance tests when more than one parsed file is specified. The visualization

module also has the ability to highlight discrepancies between the gold-standard files and the parsed files, and it comes with an easy to use GUI functionality to search in the dependency structure of the input files.

This paper presents a novel technique for parsing discontinuous phrase structure representations, labeled with both phrase labels and grammatical functions. Phrase structure representations are transformed into dependency representations with complex edge labels, which makes it possible to induce a dependency parser model that recovers the phrase structure with both phrase labels and grammatical functions. We perform an evaluation on the German TIGER treebank and the Swedish Talbanken05 treebank and report competitive results for both data sets.