This paper aptly summarizes the various challenges the authors faced, and the lessons they learned, “while applying various empirical research methods [...] to assess software architecture technologies.” It also provides good insight into some of the common definitions of software architecture and the software architecture design process as a whole. In addition, it provides a detailed literature review of empirical assessments in other software engineering areas.

The main objective of this paper, however, is to promote the application of structured empirical studies to software architecture. Although there are numerous methods, techniques, and tools for supporting architecture evaluation processes, a majority of these do not undergo a strict and well-developed empirical assessment process. This paper suggests that architectural researchers should welcome the rigorous assessment of existing and new techniques, along with supporting their development. The authors feel that empirical theory provides a substantive means to evaluate the claims of efficiency and effectiveness of particular technologies.

The empirical methods used in the research include “controlled experiments, replicated experiments, expert opinion, literature reviews, and surveys.” The entire study involved 360 practitioners and 600 students. Out of the 11 empirical studies, the authors describe two of them (“The Impact of Group Size on Evaluation” and “The Impact of Design Decision Rationale Documentation”) in some detail to demonstrate the methodology that they followed. Specifically, the authors provide some details on 13 different challenges they faced in the areas of measuring software architecture goodness, cost of investigation, and object representativeness, and challenges faced during their empirical investigation. Finally, they summarize ten lessons learned while addressing these challenges, and indicate that this provides valuable input for future empirical assessments.

Additionally, this paper presents some interesting thoughts on how to motivate software practitioners to participate in research studies, since securing the effective involvement of practitioners is seen as a major challenge. It concludes by stressing the importance of “synergy between the empirical and software architecture communities,” which would eventually support the emergence of a widely accepted body of knowledge of software architecture based on well-formed theory, empirical evidence, and experiences.

In summary, this is a well-written tutorial that links these two widely important areas of software engineering research. It is good reading for those interested in either empirical software engineering research or software architecture. The only concern is that, although the authors mention that they are quite confident about the unbiased nature of the results obtained, factors such as the experience level of subjects included or the ability of the team assembled to conduct such empirical studies could influence the results significantly.