Software is ubiquitous, affecting the life of everyone from the most senior researcher in a major software development company to a retired octogenarian trying to make a purchase at the local Safeway in Orlando. However, not all software is created equal. Software can be excellent, even spectacular. But just as often, we run across software that, quite frankly, just isn’t all that good. As a result, users experience frustration with software all the time. Most people don't have to think too hard to identify the last time that the technology they were attempting to use didn't perform as expected.

Students in the SEQuOIA Laboratory at Brigham Young University — led by Dr. Charles Knutson — are tackling this problem head-on as they attempt to discover what a truly exceptional software development organization (one which produces truly exceptional software) looks like.

SEQuOIA is an acronym which describes what the lab is all about: Software Engineering Quality: Observation, Insight, Analysis.

Here are some of the research areas currently underway in the SEQuOIA Lab.

Artifact-Based Software Engineering Research

As organizations construct software, they naturally and inevitably generate artifacts (including source code, defect reports, email discussions, and a number of associated attributes). Artifact-based software engineering researchers are akin to archaeologists, sifting through the remnants of a project looking for software pottery shards or searching for ancient software development burial grounds. In the artifacts, we find a wealth of information about the software product itself, the organization that built the product, and the process that was followed in order to construct it. Further, we gain the ability to view artifacts not only as static snapshots, but also from an evolutionary perspective, as a function of time.

Empirical Software Engineering Research

Empiricism is an important key to understanding fundamental principles of software engineering and their application in specific contexts. By applying appropriate statistical methods and employing effective experimental design we gain confidence in our research results as well as a deeper understanding of the contexts in which software development takes place.

Replication in Empirical Software Engineering Research

Many fundamental results in Software Engineering suffer from threats to validity that can be addressed by replication studies. These threats include: 1) Lack of independent validation of empirical results; 2) Contextual shifts in Software Engineering practices or environments since the time of the original research studies; and 3) Limited data sets at the time of the original research studies. However, certain factors discourage replication studies: 1) A perception persists that replication studies are less valuable than the presentation of original studies; 2) Data sets are often not made publicly available; 3) Reports of empirical studies are often not sufficiently detailed to foster replication; and 4) Research tools are either not available or not usable, so precise replication is impractical.

Dr. Knutson and PhD candidate Jonathan Krein were founding organizers of the International Workshop on Replication in Empirical Software Engineering (RESER), co-located with ICSE.

Bokeo — An Extensible Framework for Replicable Empirical Research Tools

Bokeo is a plugin-based framework that facilitates rapid development of research tools for replicable empirical research in software engineering. Bokeo adopts an artifact-based approach to software engineering research, extracting, cataloguing, and analyzing artifacts from software development projects of all types (including both open source and industrial projects). The Bokeo system provides an objective view into the artifacts that result from software development projects without biasing or tainting the organizations being studied. Bokeo provides a unified framework with which users may obtain, store, analyze, manipulate, organize and present software data and metrics.