The problems in Computer Science

Academic software development suffers from many of the same problems, such as running over time, and budget and being under resourced, as commercial development. This was clearly pointed out in a workshop at Carnegie Mellon University, a leading academic institution in Software Engineering and home of the Software Engineering Institute. The 1993 workshop (Steier et al., 1993), field by the Engineering Design Research Center (EDRC) of Carnegie Mellon University, discussed the role of software in disseminating new engineering design methods. Those who attended included industrial affiliates as well as staff and students. The workshop looked at numerous case studies of software development and suggested that "the understanding and knowledge now exist, and the time is ripe, for adopting a more comprehensive approach to software development | even within a research setting | and for establishing a better infrastructure for software design, maintenance and reuse" (Steier et al., 1993).

While very little work has been done to investigate the benefit of applying software engineering principles in computer science research at academic institutions, much work has been done on the application of this knowledge in industry. A number of problem areas have been found in the process of software development (in industry) and, over time, the field of software engineering has produced solutions to many of these problems. Industry problems, such as the difficulty of coming to grips with large system, the difficulty of explaining how software works to experts outside the design team, the difficulties of staff knowledge becoming out-dated, the difficulty of coding more efficiently and others have been at least partially solved through software engineering. These problems and solutions are discussed below. While the development environments in a university research setting and a commercial software development company are vastly different, in many cases either the same problems exist, although possibly on a different scale, or different problems exist but these problems may be addressed using existing methods found in industry. This will also be discussed below, under the sub-headings "The SE solution".

The sections below each relate to a problem I have discovered through observation of past or on-going projects, or discussion with computer science researchers about their work. Following each problem statement and explanation is a suggested method for addressing it that has been used or recommended in the software engineering literature to solve industry-based problems of a similar nature. In some cases solutions from other fields of science, that have engaged in problem-solving with the aid of computers, will also be mentioned.

1 Difficulties in communication and understanding research

The difficulties in understanding other academics' research methods, concepts, software and how they all relate, may hinder or prevent research being understood, expanded upon and practically applied.

1.1 The CS (Computer Science) problem

Scientists in a university department (outside of computer science) generally use scientific languages such as mathematics, chemical reaction diagrams, Feynman Diagrams etc to help them express and explain their theories and ideas. In the computer science department, computer scientists working in support of such scientific research may use these common symbolic languages when discussing the scientific concepts behind their software with these other scientists. While this may be adequate for other sciences who are merely using the computer as a tool, for research in the field of computer science itself one needs to know more. From a scientific point of view, it must be explained why some code is actually an implementation that experimentally proves and validates a proposed theory. As shown by their use of other notations for ideas outside the field of computer science, computer scientists are apt at explaining concepts in precise notation if given the right symbolic languages. Concepts in the field of computer science itself however, are often `proven' with software that remains a black box. Many papers get published in the field explaining some concepts said to be embodied in software, yet the code itself (one of many possible forms of notation) is often poorly documented and no attempt is made to symbolically explain how and why it works.

1.2 The SE (Software Engineering) solution

The solution provided by SE to industry's difficulties with large software projects was the creation of formalised system analysis and design languages and methods. While this has clearly not solved all the problems, it has enabled improvement and allowed professional developers to discuss design and development at a level above the code. It allowed outside experts to come in and rapidly obtain an overall picture of the system (Fowler and Scott, 1997). It improved communication of processes between experienced and more junior developers. While there is still a failure rate, it has dropped over time. Where good design documents exist, future developers (be they new members of the team or contractors) do find it much easier to continue development and correctly understand and integrate with exist work (Nutt, 1995).

1.3 Application of a solution

Unlike a counterpart in industry, a new researcher may have access to software that implements the theory and access to scientific papers discussing the theory, but he will often not have access to design documents that show why and how the software is an implementation of the theory (and through the software how the theory is applied). The researcher may have to spend a long time reconciling the theory and application through experiment, or painstaking examination of the code. This is very similar to the situation that is faced by an outside expert, or new member of a design team, working on an industry project that lacks design documentation.

The need for design documentation has been recognised and used in some university projects. The COMIC Project, an international multi-disciplinary research project highlighted the importance of appropriate symbolic language by making it one of the project's four major themes (Cooperation Technologies Laboratory, 2002). The CaMML (Causal Minimum Message Length) (Wallace and Korb, 1999), GIFT (GNU Image-Finding Tool) (The GIFT Team, 2002) and CDMS (Core Data Mining Software) (Fitzgibbon, Comley and Allison, 2002) projects have also used design languages such as the UML (Unified Modeling Language) (Object Management Group, 2001) at various stages of development.

As shown in industry, the availability of design documents that explain why the software does what it does in a symbolic language understood by all working on the project can make the task of understanding much simpler. it is however fair to suggest that when taken to a research environment not all design notation might be useful in this way, further, unlike industry with one large project, research tends to be broken into smaller research projects, each using the ideas that came before. If the goal is only to communicate the finished ideas and how they were implemented, there is no reason the "design" should not be done after the theory has been applied and proven to work, i.e. after the product is completed.

Making design documents public will make it easier for others to replicate the experiments or incorporate the research into their own work. Unlike commercial work, which aims to protect its software, this is often the very idea behind university research. A desire to share knowledge is the reason why algorithms are published.

1.4 Results

3.2 Knowledge Transfer

This is the problem of getting leading edge concepts understood well enough that they are appreciated and eventually widely used.

2.1 The CS problem

It is plausible that, given the design documents (if easy to follow design documents existed) for programs such as CaMML and Snob that use MML, others would be able to better understand not only how MML works, but also how it has been implemented in the past. This would lower the barriers to its future use. This increased reuse has been shown in the past in other areas (Steier et al., 1993).

The EDRC workshop explained this phenomenon by suggesting that the characteristics of the development process, and specifically the adherence to software engineering standards, have a large influence in enabling a knowledge transfer from the university setting to industry or other academic institutions (Steier et al., 1993). In other words, good design may not only make research clearer, but may also increase the likelihood that people in industry will take the time to look.

2.2 The SE solution

Others modelling tools such as DFDs (Data Flow Diagrams), ER (Entity Relationship) diagrams, Data Dictionaries and many others may be used when working in the structured analysis and design paradigm (Yourdon, 1989). The Structured and OO paradigms are "fundamentally different" and require a "different way of thinking about decomposition" (Booch, 1989). For communication to work, the same meaning must be attached to common terms. The choice of design language is a means of specifying the methodology that is being used and adopting a formalised set of definitions for certain programming, architectural and abstract concepts, clarifying communication of ideas much like the formal scientific symbolisms mentioned in Section 1 above.

There are many other methodologies and languages. Just within the OO paradigm there are the Demeter Method, SCRUM Development Process, MOSES (Methodology for Object-oriented Software Engineering of Systems), Shlaer-Mellor Method, Fusion Method, Open Modelling Language, and many others (Suzuki, 2002).

Extreme Programming (XP), is perhaps the newest methodology that some have begun to embrace (Wells, 2002). It explicitly allows for change and aims to reduce the level of management (and hence documentation) required for a project (Wells, 2002). Though this methodology may be of particular use in an academic environment, the approach is still very new and further experience with it and research is needed.

3.2.3 Application of a solution

While many may not understand the technical implications of the research, given a blueprint in a language they understand, they will still be able to implement the work from the documentation. It is not the first version of the software (the proof of concept) but the second that is likely to be used. As Fred Brooks suggested "plan to throw one away; you will anyhow" (Brooks, 1995). I will suggest later that an incremental development should not involve throwing everything away. Nevertheless, the development of a commercial version of software properly belongs in a commercial environment (be that private industry or a company attached to a university department). Allowing industry to build this second version will ensure that more ideas make the transition from proven theory to incorporation in usable products.

3 Loss of Knowledge

An inability to communicate ideas effectively due to differing language and culture can lead to a loss of detail and misunderstanding between student and supervisor or between members of a research team.

3.1 The CS problem

In order for a project team to work well together, either every one must speak the same technical language, or the group must resort to avoiding technical language. In the case of a design language, avoiding the language is most easily achieved by avoiding design work altogether. In too many cases this simple approach is used.

Many researchers I have spoken with admit to having most of the details of their work in their head and very little, if any, on paper, the exception being a published paper once the work is complete. The problem with this approach is that the knowledge requires the person. If the researcher loses interest, moves to a new institution, or is hit by a bus, the result for their research students, institution, and potentially the computer science community as a whole, is a disaster. While many computer scientists believe strongly that information should be public, their own actions prevent a full discussion of their research.

3.2 The SE solution

Like AT&T Bell in their shift to software, the IT industry (including the education sector of the industry) has also undergone a rapid change: from mainly procedural to mostly OO programming. Industry has been working to adapt to the "paradigm shift" (Fowler and Scott, 1997) and to retrain staff, while in the universities some have still not made the transition or taken the time to learn OO and add the new paradigm to their own knowledge.

3.3 Application of a solution

4 Inefficient and ineffective reinvention

The problem of re-inventing the wheel one wants to use.

4.1 The CS problem

Casual observation will show research students learning how to code up basic components like linked lists, and then coding them again year after year (even after they leave the university) from scratch each time they need to use one. This is inefficient when the code for a doubly linked list is part of standard template library in C++, a standard class in Java and some languages, such as Perl, just don't need linked lists (the Perl array has all the functionality one might want a linked list for) (Stroustrup, 1997; Flanagan, 1999; Wall, 2000). This re-coding becomes ineffective and indeed dangerous when programmers forget subtleties of the data structures they are coding, or make logical or typing errors. Now not only have they wasted time re-coding, but they will have to spend a lot more time debugging code that wasn't needed in the first place. While the pedagogical motivation of re-coding fundamental data structures is acknowledged, like assembler, there is a time and place for this in a computer science course. For most students, postgraduate research is neither the time, nor the place.

While scientists, and particularly computer scientists, have benefited from these advantages of modern high-level languages, at the macro level this inefficiency and ineffectiveness often still holds when whole projects need to be re-coding from scratch just to add what could otherwise be a small plug-in. The reason for re-coding is often an inability to read the last person's poorly documented code. Unfortunately there is a habit of passing the problem on when the new person also fails to document what they've done. This can be seen in early version of CaMML, Snob and other projects.

Another common problem is the lack of a framework. Often many projects would be able to share components if someone had built the framework to allow it to happen. Unfortunately as this is often not research in and of itself, but rather the coding of a tool, no one is willing to invest the time. The issue of frameworks has in the past been addressed. Brooks suggests that a dedicated tool-maker be attached to a programming team to provide a common framework (Brooks, 1995), and in a university environment this was in fact done in the past (including at Monash). For many years now computer science has been funded (at least in the UK) on a similar model to mathematics rather than that of other sciences (where technicians to set up experiments are regarded as a standard requirement), this is cited as one of the problems with research in the Computer Science field by Mike Holcombe, Dean of Faculty of Engineering and a past head of the Computer Science Department at the University of Shefield (Holcombe, 2000).

4.2 The SE solution

At the implementation level in industry, programming languages such as Java are being pushed, and many argue succeeding, due (as mentioned before) to their vast array of well-maintained libraries containing common data structures (Neighbors, 1998). The Microsoft Foundation Class was similarly invented to decrease the amount of work needed to do common tasks and was similarly accepted by a large part of the industry. The idea of frameworks have also been explicitly used, successfully, in industry. An example of a successful framework implementation in industry is that of the GENVIS project, a report visualisation framework developed for a leading Swiss Bank (Bredenfeld, Ihler and Vogel, 2000).

These are all specific (limited) methods of reuse. The larger and more general question has been shown to be much more elusive. Research literature in the area of reuse has mostly taken the form of developers in ongoing projects tracing the cause of their success back to reuse, rather than publication of new techniques for reuse developed from first principles (Krueger, 1992).

4.3 Application of a solution

CDMS (Core Data Mining Software) (Fitzgibbon et al., 2002), is a framework for a collection of programs related to modeling data. CDMS makes use of Java libraries and then extends them into its own language, while still allowing plug-ins written in Java to be integrated. A number of projects currently plan to make use of CDMS either as a platform to build on, or as a simple method for managing input find output with the user (i.e. reuse of the interface, parsing and sorting abilities).

5 Repeated framework rebuilding

There is a loss of time "re-inventing the clock in order to adjust the time".

5.1 The CS problem

5.2 The SE solution

Another concept in use in industry is that of design patterns. Although not mainstream, it has become a core approach of many companies and given them tangible benefits (Seen, Taylor and Dick, 2000). The concept of design patterns is used to give developers a basic solution to common problems that their software may be trying to solve (Larman, 1998). The effect is such that "one person's pattern is another person's primitive building block" (Gamma, Helm, Johnson and Vlissides, 1995). In short, once a solution is found and recognised as a pattern, the next time this class of problem comes up the same solution can be reapplied rather than reinvented. The result is improved communication and knowledge sharing resulting in increased reusability, flexibility and compatibility (Seen et al., 2000).

The explicit concept of frameworks is also used. Two such example are the KOM and SCAF frameworks (Rakotonirainy et al., 2000; Weis and Geihs, 2000). KOM used a common framework to manage components in the K desktop environment. It allowed communication between these components, and also the use of one component by another. KOM was based on CORBA and as a result allowed communication and reuse of components regardless of the languages used (Weis and Geihs, 2000). The creation of KOM also introduced new design patterns, the "signals and actions pattern" (Weis and Geihs, 2000). SCAF (Simple Component Architecture Framework) was also based on CORBA and aimed to provide a simple general purpose component based framework (Rakotonirainy et al., 2000).

5.3 Application of a solution

The concept of patterns can also be adapted to academic research. Providing (perhaps publishing) patterns for such concepts as model finding using MML might make it easier for other researchers to implement the MML concept in their own work. Solving the framework problem could potentially also solve the knowledge transfer problem by effectively lowering the barriers against new or different techniques.

6 Guaranteeing Authenticity

The authenticity of research results issue

6.1 The CS problem

There have in the past been cases where software intending to prove a thesis has been knowingly written using a method other than that purported in the related paper (Korb, 2002). Possibly more common is the case of simple bugs. The algorithm may be implemented incorrectly and as a result give a better or worse result than the actual. The outcome could be an incorrect analysis either suggesting against an improved method, or advocating a poor one.

By creating suitable documentation of how the software actually works, the author can more easily test their program for correctness and the scientific community can more easily check that the algorithms used are those stated in research papers.

7 Intermediate software

Dangers with the use of "intermediate software" that has not yet been stabilised.

7.1 The CS problem

The EDRC workshop noted that the first generation of the software had to be re-written and in some cases many times before it was accepted by industry. They also noted that industry was aware of this and was as a result reluctant to use university produced software | even if it was free (Steier et al., 1993).

7.2 The SE solution

Requirements in industry change constantly and often a developer is simply not able to decide to put off changes until later, or to leave out what the client considers useful features. The solution in industry was Boehm's Spiral Model which allowed for rapid improvements using an incremental model and prototyping while still retaining the systematic nature of development that had been gain through development of the waterfall model (Boehm, 1988; Pressman, 2001).

7.3 Application of a solution

8 Creative Problem

8.1 The CS problem

8.2 The SE solution

The most valuable part, is often not the part that takes the most time. "With any creative activity comes dreary hours of tedious, painstaking labor, and programming is no exception" Brooks acknowledges (Brooks, 1995). The claim that research is pure inspiration and therefore the research process cannot be improved is one that we reject. The question is not if the process can be improved, but rather how and at what cost.

8.3 Application of a solution

Another potential indirect gain to the creative process may occur by reducing the hours of "painstaking labor" that Brooks referred to. Freeing up more time for leading researchers to come up with their ideas will effectively increase overall scientific output. It remains to be seen if the initial investment in recording the knowledge would eventually be recouped. Even if the system is documented exceptionally well, a certain amount of clarification will always be unavoidable and invaluable.

While side-effects of an improved process may remove many of the impediments to creativity, the initial creative process is something that can not be artificially improved at this time.

References above can be seen in the Bibliography.