Approximately 13 hours of recorded interviews were conducted with postgraduate students and staff at Monash University. The interviews were conducted with five PhD candidates and seven staff. Additional interview material was collected via e-mail from experts overseas. This section is divided into themes and contains quotes and summaries on those themes.

1 A historical perspective

"Programming languages don't help a huge amount. At most there has been a factor of four improved productivity [from machine code to higher level languages]" (Wallace, 2002).

2 The nature of research

"I've very rarely been involved in any development work where there was a clear aim. The nearest I can think of was...the development of an operating system. A great deal of thought went into what should be put in and left out. Several man years went into the specification, the implementation wasn't so difficult" (Wallace, 2002).

"By its nature, research software evolves over time and is tweaked as the researcher learns more about the area of interest. By its nature research software is to be used by one person (although this tends to change if the research bears fruit)...

The implication (to me) is that any SE approach for research software (that would have any chance of adoption) would have to be agile and evolutionary in nature" (Pressman, 2002).

"From the point of view of research methodology, I don't know what research methodology (in computer science) is. You get an idea, you want to check it out, you get another idea, and if the first idea didn't work you change your mind. That's what research is about and what makes it fun" (Zuckerman, 2002).

"One of the people who worked for me kept complaining `well [she] keeps changing her mind'. The changes are not about changing what I want, but rather we tried some things and the results aren't very good, so I suggest we try it in a different way" (Zuckerman, 2002).

3 Paired Programming

4 Approach to Research

"For a large project there has to be a design. Its just nonsense to go into it without a design. If there are three people involved you better design it or you'll end up in a mess" (Korb, 2002). Dr Korb lists the essential elements such as a clear specification of things like classes and object and types and methods and procedures and inputs find outputs and functions implemented.

"I need to prove the feasibility of the idea, rather than flesh it out for third party users. This eliminates a lot of frills in a user interface. I also have very, very little time. I plan on making mistakes and program very defensively { hopefully others do, too, but I have my doubts there" (Schreiner, 2002).

"I don't believe in the currently fashionable idea that you produce some junk and you remanufacture it so its slightly less bad junk, and then you do it again. I don't think that works. You should aim to do a good design the first time"(Meyer, 2002)

"I'll probably develop a software design of some sort. I prefer to insist on code reviews as code gets written. Time is a problem though so sometimes I don't have that" (Korb, 2002).

"I think there's a bit of a trend to turn research into a process that can be managed like an industrial process and I think that is really antithetical to the nature of research...we still need to recognise that it is at its heart an creative and somewhat unpredictable process" (Farr, 2002).

Wallace suggests that the approach to a typical research project without clear aims is to "write a fairly crude very limited prototype and try it out on a couple of really straight forward problems, like shooting fish in a barrel. Then refine it and add to it" (Wallace, 2002). According to Prof Wallace, useful diagnostic information that does not require an intimate knowledge of the code is only added when the program has a user base. Once people start using at a lot more time is spent on development. Then "you step back and probably realise there is only one way of improving it, and that is to throw the whole thing out and start again. You just have to do this several times" (Wallace, 2002).

"The primary aim is to get a flaky prototype working sufficiently to get a few statistics out. There is absolutely zero [incentive] for producing a robust, flexible, extendable piece of software" (Alison, 2002). Prof. Wallace agreed "you get more brownie points if you leave it in a half finished barley usable state and go and do something else, read new papers and new ideas, and pretend you've done the job, and of course you haven't" (Wallace, 2002).

There is an initial penalty for writing a once off program. If you make a more general program the penalty is bigger. You just hope that there will be a pay off and it will make up for the initial penalty. "There is a positive disincentive [to write a more general program], every thing we do discourages people from doing it" (Alison, 2002).

5 Quality Software

"To get a research paper in a conference you need a flaky prototype that produces just enough statistics to give you tables and graphs to put in the paper. Once that's done the method of is old hat in a sense. You want to move on to the next flaky prototype and the next paper" (Alison, 2002).

"[Snob] did have quite a nice documentation file on how to use it [and] I suppose I put the best part of a man year into being a help desk. You get nothing back for it."

"A system built as part of a Ph.D. project is intended to prove feasibility, and it would almost always be a mistake to spend the time and effort during initial development to build it to product-quality standards. Unless, of course, the university research were precisely on engineering methods for software" (Brooks, 2002).

6 Funding

7 Management of people and time

"I feel there is a great deal of time wasted in recoding other people's software...people are recoding everything under the sun" (Zuckerman, 2002).

8 Authenticity

One (non case study) project discussed a research student who failed to provided source code. A paper was published and later had to be amended when it became clear that the software was not implementing the correct algorithm. "It was an incredible embarrassment when we found out it wasn't implemented in code. We had publications out there which were false" one participant stated. The code was fixed and new results submitted for the paper (which was published). Source code to go with the paper was never provided. It is thought the lack of code was partly due to versioning problems and partly due to insufficient management experience. The case was discussed in two interviews and the accounts agreed.

"There have been at least three cases of people implementing MML and other methods for doing the same thing and comparing them. In each of those cases they made an absolute dog's breakfast of implementing it" (Wallace, 2002). In one case the results were shown at a conference and the fault was pointed out. The same paper was then submitted to "Machine Learning" and sent to Prof. Wallace for review. Although the review pointed out the error, due to the reputation of the author the editor published the paper regardless. A follow up paper by Prof. Wallace that implemented the method correctly was turned down for publication.

"You may or may not take the bother of explaining how the code works, it doesn't matter. They should at least be able to verify that the claims to performance you have made are indeed true. A lot of people publish and when you ask for at the code doesn't exist, or it doesn't compile."

9 Readability

"I think its an unreasonable demand that the code has got to be intelligent. A lot of people complain about my code because I use gotos. Well tough. That's the way my brain works" (Wallace, 2002).

Wilkin (2002) said of code he received for his PhD project "It was a nightmare working with other people's code. It was badly designed and coded in obscure ways. It was difficult to understand what it did. Once it was understood it became easy". He said the code had very little internal commenting, and what was there were regularly wrong. There was no external documentation. "I lost two months because of undocumented preconditions" (Wilkin, 2002).

"[In the original CaMML] comments said what a chunk of code was supposed to do, they didn't say how it did it, but then that's what the code is for. Some of the code is very optimised, it runs very fast but isn't so umm...easy to understand" (O'Donnell, 2002)

"[Chris Wallace] is incredibly clever with coding...but finding the fastest way to produce the most compact result, which Chris is absolutely brilliant at, is not the best way to write code" (Korb, 2002).

"Javadoc is the best thing since sliced bread if the authors put the content in" (Alison, 2002).

10 Design decision