Regression modeling of data for 209 outsourced software development projects found that the influence of tools on project effectiveness and efficiency varied according to the degree of conceptual and process novelty present. Input variables comprised project managers’ perceptual assessments of how well the following types of tools were used: requirements managers, architectural modelers, test automation tools, test case development tools, configuration management tools, and defect and change request trackers. Outcome variables were budget overrun, rework following the testing stage, and defect numbers recorded during the installation and warranty period of the software.

Many will regard this study as fatally flawed. The questions in the survey regarding tool use involve answer keys with words such as “somewhat” and “extensive”--such words cannot be unambiguously interpreted by respondents. Also, many factors contributing to the outcome of a software development are ignored. Why was available data about vendors’ Capability Maturity Model (CMM) level not part of the analysis? What about the contribution of static analysis tools and formal technical reviews to software quality? Many will also regard a regression model with 25 coefficients to be overfitting. Was there really no prior search for a much simpler model (Occam’s razor)?

Astonishingly, 87 of the 209 projects listed in Appendix A share the average values for all three outcome variables. Was this a consequence of treating considerable levels of missing data? Projects missing values for all three outcome variables should have been eliminated.

This paper will be of interest only to those researching the software process.