|
|
|
Research MethodThe research questions investigated in this thesis are both broad and complex. In order to lend rigor to the scientific investigation of these questions, three different methods of data collection were applied and the results compared. Original research for this thesis was carried out using surveys, interviews and case studies. As discussed by Patton (1990), a combination of research methods including both quantitative and qualitative can be used to gain a more complete understanding of the phenomenon under investigation and increase the confidence in the results. Methods to reduce the dangers of bias and data corruption associated with each method are also discussed. The principle of triangulation is explained in further detail in Section 2. 1 The research questions and metricsThe questions under investigation:
The research started with these top level questions. Questions were then developed to investigate a number of relevant issues. 1.1 QuestionsQuestions were formulated that investigated:The approaches currently used by researchers:
The problems that might be solved or have their impact reduced. The level of software engineering use and knowledge among computer science researchers:
What, if any, benefits (or costs) have been realised by those currently using software engineering techniques? Which, if any, of the tools from software engineering have the best potential for more widespread use in computer science research? 1.2 MetricsAfter the questions had been formulated a number of metrics for evaluating research and the approach to research were developed...
The full list of original questions can viewed. The list of metrics considered, along with the proposed method(s) of obtaining these metrics, can be found here. 1.3 SurveyTwo surveys were carried out as part of this research, the first was directed at computer science academics in the United States and the second at computer science academics in Australia. As will be shown in the results section, the surveys both found their target audience. The surveys were web based.The US survey was released significantly before the Australian one, and the Australian survey was delayed until a sufficient number of US response had been gathered and analysed. A number of participants included feedback on the survey form, or e-mailed me suggesting minor changes to the wording of questions or closed answer choices. These suggestions were incorporated in the Australian survey before it went on-line. 1.4 Technical DetailThe surveys were based on a Perl script with a database interface. Participants were anonymous, unless they chose to answer optional questions (asking for a name and e-mail address) and effected to receive a copy of the survey results. The code was adapted from a script by Dr David Squire produced for another purpose.Initially the code was re-commented (notably a larger header block explaining options was included) and the questions were hard coded into the script. A number of new question types were later added, some by Dr Squire and some by myself. In the Australian survey a large sections of the questions (those dealing with a particular project) could be repeated, this involved a second script and a second data file. Participants in the Australian survey were given the option of ending the survey or continuing after filling in the details for a single project. Survey participants were invited via e-mail. An E-mail was sent to 255 US universities. The letter was sent to the head of the computer science (or similar) department with an explanation of the research and a request to forward it on to their staff. In Australia a similar e-mail was sent to 34 universities, again addressed to the head of the computer science department. A notice about the survey was also sent out via IT announce 1 1.5 Software ArchitectureInviting participantsThe mailer code (a Perl script), sample outputs and the list of Australian universities mailed can be seen in Appendix C of the thesis.The online survey formsA non interactive version of the US survey can be viewed. It was online at: http://www.csse.monash.edu.au/cgi-bin/cgiwrap/andre/questionnaireA frequency count of results obtained can also be viewed. The frequencies and related graphs are auto generated and may require a little interpretation. The non interactive version of the Australia survey can also be seen. It was online at: http://www.csse.monash.edu.au/cgi-bin/cgiwrap/andre/AUSquestionnaire A frequency count of results obtained can also be viewed. The frequencies and related graphs are again auto generated and may require a little interpretation. Analysing resultsUsing a combination of Perl, AWK, C, GNU-PLOT, HTML and shell scripting, a system was created that produced summarised tables and bar graphs of the responses to each question, as well as the joint distribution over any two requested fields. The output of this system was:
Files to allow manual alteration and fine tuning were also included. These included:
These files could be manually altered to change the look of a graph or to correct data entry errors and merge groups of responses (e.g. Australia and australia were different responses according to the automatic summarising code). The system was later extended using a spread-sheet program which took the table of all detailed results mentioned above as input and returned the correlations between survey questions and the significance of these correlations. Part way through the analysis it was realised the degrees of freedom varied with the set of questions being examined, this in turn effects the significance. The spread-sheet was extended again to now include calculations for the degrees of freedom. C code was created to convert the spread-sheet results into a tabular format and match field number back up to names. It was found that the code for tabularising and matching variables up (originally coded for the correlations only) was reusable for significance values. 1.6 Statistical AnalysisCorrelationsThe Pearson product moment correlation was calculated between fields of real values, for example the number of publications and amount of funding.The Spearman rank order correlation was calculated for all combinations of fields where it least one was a ranked value, for example satisfaction level and use of software engineering. The significance of the correlations was calculated using a Student t test, as both surveys had an effective sample size of under 30 for almost all correlations. The Pearson product moment correlation is used to determine if there is a relationship between two sets of paired values. It is the correct correlation function to use for correlations of actual values, such as time in planning and time in development. It is calculated as: 
The related t value was calculated as: 
The probability of H(0), the null hypothesis, was looked up on a t-table using the t value calculated and the relevant degrees of freedom. A two tailed t test was used, the degrees of freedom was two less than the number of people who gave non blank responses to both questions. The Spearman ranked order correlation is used to determine if there is a relationship between two sets of paired ranked values. It is the correct correlation function to use for correlations of ranked answers, such as a participants level of use of a technique (e.g. 1 = know but don't use, 2 = use rarely, 3 = use sometimes, 4= use often, 5 = use always) and their level of satisfaction (1 = much less then other projects, 2 = less than other projects, 3 = about the same, 4 = more than other projects, 5 = much more than other projects). It is calculated as: 
The related t value was calculated as: 
1.7 InterviewsAll interviews were tape-recorded. Interviews were conducted privately to lower the risk of one interview influencing another. For the same reason they were scheduled to occur in a one week period.Questions were created and provided to the interviewees prior to (or at the start of) the interview. Interviewees were informed that the questions were a rough guide only and the interview could diverge from them at their instigation or my own. Once the interview was started, candidates were told that they should feel free to decline to answer any questions and that, if they would prefer, the interview tapes would be kept private (specifically from my supervisors with whom some have a working relationship). Interviewees were also informed that they would be able to view any part of the interview used in the thesis prior to the thesis being submitted. Interviews typically lasted for 1 to 2 hours. For some interviewees this was divided between a case study interview and more general discussion. There were 12 interviews in all. The tapes were to be transcribed and entered into the Nvivo software package, however due to a back-log at the transcription service this was not possible. Interviews were instead listened to with only those points considered of interest transcribed and quotes entered into Nvivo. While the full transcriptions would have been better, it is felt more than enough information was captured from the interviews and transcribed. 1.8 Case StudiesCase studies involved:CaMML
CDMS
GIFT
Interviews for case study analysis were conducted as per the interview section above. 2 TriangulationTriangulation has two benefits for research (Patton, 1990): breadth of view and confirmation. \Triangulation" originally comes from land surveying, where it was shown that given the directions from a location to two land marks, the location can be accurately pin pointed at the intersection. Additional directions to other land marks confirm the location and can be used to remove, or at least evaluate, measurement inaccuracies. This is the confirmation benefit. The depth benefit comes from the use of unstructured and open investigation techniques. It is possible for factors not taken account of in the other approaches to be discovered with methods such as open interviews and field studies. In this research, the findings from the surveys were compared to the experiences of researchers as discussed in interviews. The case studies were compared to the general trends of the survey and compared to general experiences from other projects as discovered in the interviews. Answers from case study interviews were, where possible, compared to observed results (e.g. a person's perception about their level of commenting and an objective judgment based on looking at their code). Finally general themes emerging from the three research methods were considered. A comparison of some of the findings from the three research methods is found Section 6. |
