Validation of the Defect Classification Scheme (Jan 06) by Survey

From HPCBugBase

[edit] Background

To get the feedback to Defect Classification Scheme (Jan 06), we have conducted a survey. The participants of this survey were programmers, scientists, and vendors of HPC technologies who were attending an HPC-related meeting. For the survey, we first explained our classification scheme and defect examples in an oral presentation. Then we distributed paper survey forms. We also prepared a web version of the same survey so that the participants who preferred entering the answers online could do so. The survey was anonymous, and 14 people returned the form. Their experience in HPC development varied from 5 to 22 years. The survey form consisted of several questions. The survey question intended to validate the defect analysis results at the classification level was as follows: "Do the defect types (in Defect Classification Scheme (Jan 06)) look reasonable? Can you think of other defect types, better wording or modification in definitions? Can you suggest a different classification scheme?” The answers were given in free text form.

[edit] Survey Results

For the proposed defect types, most participants agreed on our classification scheme and suggested no modifications. The comments provided were as follows:

• One participant commented that in his projects, Algorithm, algorithmic defects are eliminated by peer review before any code is written.
• One participant commented that he did not understand the Serial Constructs defect. He noted that he did not listen to the presentation given at the meeting.

For additional defect types, the following comments were provided.

• Three participants suggested defects related to I/O: I/O Defects. The proposed defect types include I/O data format/conversion errors, I/O performance issues, resource mismanagement in file/socket open, and generic I/O problems.
• Four participants suggested defects related to memory management: memory mismanagement, invalid memory operation errors, memory placement performance issues, and memory contention.