Conducting empirical studies in industry: balancing rigor and relevance

Abstract

Empirical Software Engineering research has achieved considerable results in building our knowledge about selecting and applying appropriate empirical methods for technology evaluation. Empirical studies in general and empirical studies in industrial settings in particular have played an important role in successful transition of many Software Engineering technologies to industry, for example, defect detection techniques and automated test cases. However, conducting empirical research in industrial settings remains a challenging undertaking for a variety of reasons. There is no substantial literature reporting on the challenges and complexities involved in conducting empirical studies in an industry in general and in settings whose business models are built around global sourcing. This paper reports some of our experiences and lessons learned from conducting empirical research in industry. Some of the observed challenges include short time horizon for research, high expectations, limited research skills, and the 'acceptable' research rigor. The paper discusses some of these issues with relevant examples and provides some strategies for overcoming these issues. We also stress that researchers and practitioners should share their experiences of conducting empirical research in order to help build a body of knowledge to guide the future efforts.