The research in this thesis was motivated by an apparent lack of Software Engineering in Computer Science research as conducted at Australian universities. While striving to be impartial in our collection and analysis of results, our initial hypothesis was that the lack of observable scientific discipline and engineering was detrimental to the maturing of computer science as field. During the course of this research it was necessary to further refine our premise. We defined what sort of Computer Science we were examining, reverted to an older and more general definition of Software Engineering, and investigated various types of `usefulness'. As no previous work addressing these questions appears to exists, each refinement involved hard choices and has had significant impact on this thesis.

In this chapter we review the refined premise and elaborate and qualify the conclusion that `some form of discipline and engineering is certainly of benefit (in some way) to some aspects of computer science research'. The form and benefit of software engineering, as well as the aspects of computer science most likely to benefit were other than anticipated at the start of this research. We conclude that Software Engineering as it stands in industry is not an appropriate option and may have significant negative effects on research. As this area of research is largely unexplored we conclude with a number of general directions for future research and a brief summary¹ of what we have examined.

1 Review

Computer Science has been defined as that field of science that is concerned with solving problems through the use of computers and study of that field. Our research concentrates on that part of computer science that uses computers as a means to solve problems. Software Engineering has been interpreted in its broader sense as the dicipline of making this part of computer science more `Engineering like'. A number of potential problems and uses for Software Engineering were considered. Many outcome metrics were proposed and examined. It is apparent from the results that the metrics are sometimes uncorrelated or in fact negatively correlated with each other. Whether Software Engineering, or various Software Engineering methods are of use to Computer Science, depends upon which outcomes one considers important. We will now recap a number of Software Engineering approaches and when they may be useful.

External documentation
The benefits of external documentation are mostly in the area of knowledge transfer, that is communicating knowledge about the project between the research group developing it and other researchers. For a small highly focused research team there appears to be little if any benefit in maintaining up-to-date design documents of the system while it is being developed. This is due to the nature of research. External documentation may be valuable to supervisors who have a number of students working on a project over time and to research students starting out on a project.

Readable Code
While external documentation may not be useful during the intense development phase of a research project, readability of code becomes highly important. Without clear and well commented code the paired programming approach which is of great benefit does not work. Paired programming usually involves two research students. The use of well structured code with a reasonable level of commenting was shown to be important to the reuse of that code. Badly documented code leads to frustration. Reasonable commenting pitched to a reader other than oneself appears to be the right approach.

Configuration Management
Configuration management is always useful. Projects that do not use it run the risk of version inconsistence. This was shown to potentially cost large amounts of time and possibly lead to authenticity issues.

Frameworks
The use of a well designed framework allows more ideas to be tried in a shorted period of time. This allows research students to make a more significant contribution to the field. Although they have a high initial cost, our research suggests frameworks based on plug-ins provide significant long term advantages for the institution and research group. It is unclear if the time PhD students would need to invest in developing a framework would results in a similar saving of time in implementing their PhD project. That a there is both a large initial cost and a large time saving later on is clear. In order to increase reuse (saving time) and increase publications (increasing funding and prestige) the expense of hiring staff to work full time to develop the framework core may be justified. Such work should ideally be stabilised prior to any projects that rely on it commencing.

2 Future work

The work in this thesis is based on relatively small survey samples and three case studies. A more extensive examination of the issues discussed is needed. This should include a larger sample for the survey and many more case studies and interviews.

While data was collected from both the US and Australian surveys, this research focused on the Australian results. It is apparent from a brief analysis of the US data that some differences exist in the approach and tools favored in the different countries. It is thought they may be due to a different of emphasis in the funding models. These differences should be examined in detail.

The RAISER / RESET approach need further development. It meets the key requirement of agility and complements research rather than imposing restrictions on it. The finer details of what form of documentation to use in the RESET part of the project, and how best to use configuration management are open questions. One idea is that of literate programming. This could provide commenting in layers allowing the `resetters' to create the internal comments and documentation of large modules (created by combining a number of smaller modules) while retaining and including the documentation (possibly left their by the researchers) of smaller modules. It would also allow research to document design decisions in code. These could we reviewed by the resetters to ensure key structure is not lost. The RAISER / RESET methodology is yet to be field tested.

Computer science is a vast field and the approach to research and the development of research code differs greatly. There is still much work to be done in examining and documenting the approaches. Researchers across the globe may have good ideas they are applying successfully, but at present their is no forum to examine and exchange these ideas. Forums on both Computer Science and Software Engineering education exist. The lack of a forum for research education, that is discuss on methods and approaches to developing research software is a noticeable gap in the field. The organisation of such a forum to exchange knowledge and ideas constitutes a significant amount of future work in this area.

3 Brevis epitome

During his interview Meyer (2002) said "If you can show that some of the ideas from software engineering are directly applicable, then you will have achieved something". We hope with an objective view we have done this and more. Not only have we examined `what' software engineering may be useful for computer science research, but we have addressed the important question of whom it may assist. We have explained `why' and `when' these methods should be applied. The detailed question of `how' to best apply them will require significant experimentation and is largely left for future research. I hope this research has laid a useful foundation and provided an incentive for a more detailed philosophical examination of computer science research and our approach to it.