|
Teaching Aspects Staff Students CUTSD Publications Feedback |
Project Title: An Internet Environment for Learning Software Testing Processes Project Summary: This project will develop an Internet based interactive multimedia environment for students to learn the process of software testing in an object-oriented (O-O) system. O-O is considered as the most promising software development technology by both industry and academia. This developed Internet environment will include programs that provide facilities to:
Commercially software testing attributes around 28% to 50% of the cost of software development. O-O languages are gaining increasing acceptance in the industry at 35% per annum [Age, 9/4/96]. Hence, there is an obvious need for students to be taught systematic approaches to software testing, especially in the development of complex systems. The system must be shown to satisfy the specification through testing. Our experience has shown that traditional techniques of lectures and tutorials are not sufficiently effective to learn the complexity of comprehensive software testing. The use of multimedia facilities incorporating colour, sound and animation, can enhance the learning process by providing an audio-visual environment. The proposed system will highlight the behaviour changes of the components that make up the software-under-test. It will allow visualisation of complex software testing process. Students can interact with the system and gain a better understanding of how to test software systems. The project in addition will:
1. Project and project context Traditionally, computing graduates have been taught techniques to design and develop relatively small programs, with little emphasis on software testing and maintenance. However, testing is an important activity both at the development and maintenance phases of software development. Industry data show that around 60% of software costs go into the maintenance phase of the software life-cycle. As an extreme example, the explosion of the first Ariane-5 rocket 30 seconds after lift off in June 1996 was found to be due to hidden assumptions in software elements from a previous rocket having been reused in Ariane-5 without proper understanding of the testing required (7). Clearly, it is important that Computing students learn systematic testing of software systems. Recently, a new technology called Object-oriented (O-O) software development has shown a 35% per annum increase in use in the industry [Age, page C1, 9/4/96]. This OO technology encourages modular and incremental development of large systems. We teach O-O software development for building complex systems. Testing OO software is an integral part of the course. However a lack of adequate teaching tools support has made it difficult for students to master the complexity of software testing. 2. Expected Outcomes for target group : In this project, we intend to develop a suite of multimedia programs based on WWW (World Wide Web) technology. This approach will provide students with a readily accessible software system through which they can interact and learn how object-oriented testing really works in a step-wise fashion. The system will draw on underlying principles to develop interactive simulation programs and help students’ comprehension of O-O software testing through dynamic displays and animation of the software elements.
The system developed will be accessible through web browsers such as Netscape from a variety of hardware platforms such as PC, Macintosh and Unix machines. The interactive multimedia code and the supporting instructions will be programmed using the widely used internet languages of JAVA and HTML. On-line forms will be implemented for feedback from students and used in monitoring their understanding and progress. The result will be a web based computer assisted learning system which will include both instructions on object-oriented testing and the use of the latest WWW technology with multimedia facilities to enable students to visualise the way object-oriented testing really works. It is expected that the students will be motivated to interact with and learn about the topic area by using this system as it will provide an opportunity to explore the material at their pace and have on-line monitoring and feedback features. 2.1 Learning outcomes: Students will recognise the need to test the software systems that they implement. Students will undertake this task of software testing systematically by visualising execution paths and state changes in the tested program. The students will produce better test data (for their assessable components) that takes the possible states of the objects and their interaction during system execution by simulated and animated examples using techniques such as finite-state machines and event interaction diagrams. This proposed project encourages a student-centered approach to learning, enabling students to work at their own pace and providing them feedback . As a consequence it is expected that the students will be motivated to use the system. 2.2 Educational principles addressed: Several educational principles are involved in the project. First, the project shifts the learning process from the traditional teacher-centred approach to student-centred approach. Since software testing is a complex process, the learning pace of students may not be uniform and therefore this shift should help the students. Second, visual images and animation are used in the education process. Computer based animation and visualisation are powerful and motivationally attractive education techniques which can simplify the presentation of the conceptual frameworks and schematic overviews thereby enhancing learning. Object-oriented software testing has the advantage of being easily 'visualizable' in terms of state changes and data-flows. By making use of this advantage, a complex process can be taught in a better way. The use of Internet technology to make the tool available on the students' computer in the University and elsewhere obviously increases the accessibility of the tool. A significant number of students have home computers with connections to the Internet. By providing the tool on the Internet the students have the flexibility of learning at their own place. Thus we provide both time and space flexibility in the learning process. It is vital for students to internalise the notion of testing so that they can produce systems which are reliable and of good quality. 3. Evidence of literature and database review: Our project is unique in its combination of multimedia, Internet, and student-centered approach in the learning of software testing. Projects that use Internet based multimedia for computer mediated education have been reported in the literature, however we are not aware of any package for teaching software testing. Proprietary tools like Statemate, Quid and Protob are used in research establishments; these however are not developed for educational purposes, but are visual O-O software modelling tools and do not focus on testing. Furthermore, the licensing requirements for such tools would work against using them as a widely available student resource. Some animations of algorithms (e.g. SORT) are available on the Internet as JAVA examples but our tool will be tackling a different problem of testing. Several tools to aid OO-testing, some as research prototypes, have been developed. Examples are ASTOOT [11], ACE and OZTest [4]. The major difference between the current project and systems like ASTOOT is that our intention is to develop a system to learn OO-testing, where testing itself is only one component; the more important aspects are visualisations of changes to the system under test, instructional material on preparing test plans, and provisions for evaluation and feedback. The European consortium "Conceptual Learning of Sciences" (CoLoS) [9], comprising a number of European universities, encourages teaching methods that promote students’ understanding and skills of scientific concepts through experimentation with simulation environments. The text book titled "Web-based instruction" (WBT) [10] reports on various aspects of the design, deployment and evaluation of instructions using the WWW. But, neither CoLoS nor WBT have projects in the area of teaching O-O software testing. Our project aims to provide interactive simulation programs of varying degrees of complexity on the internet to teach O-O software testing. 4. Anticipated practical advances and integration into department: The preparation of proper test plans and test cases is a difficult but important activity in the production of software systems. Currently, students develop object-oriented software systems by following a methodology involves deriving software designs from software models that satisfy certain conditions. These requirements (conditions or rules) are included in the implementation using an OO language and become part of the test data definition. Although this link in software design, coding and testing is covered during lectures, it does not explicitly cover the difficult area of visualizing the complex processes involved in developing large software systems. Students will benefit from a tool which can assist them to visualize the paths followed during testing. The tool will provide an alternate delivery system which addresses the issue of catering for individual learning styles. It will improve student motivation and understanding as it uses a student-centered approach to learning. The proposed tool will improve the quality of test plans, test cases and the object-oriented test strategy employed by the students. It will directly benefit the students studying these third year offerings. Further, it could also benefit those second year students studying software engineering practice and JAVA programming (total enrolments about 700 per year). A number of our postgraduate courses also teach various aspects of object-oriented system development, and the issue of testing and its relationship to quality is a recurring theme in all of them. Hence, this project will be of general benefit to a number of courses within the department and may even be customised to the specific needs of the various courses once the success of the present project has been evaluated. 5. Needs analysis and project feasibility outline: Background: "Object-oriented Programming Systems" and "Systems Quality, Verification and Validation" are third year level single semester undergraduate subjects at Monash University. Students are required to learn to develop software systems using an object-oriented approach in these two units with an emphasis on testing. Object-oriented software testing strategies are covered using the traditional lecture presentations and followed up by laboratory assignments. Our conventional approach can only teach software testing as static observational models on paper or on electronic medium. This project is to develop an interactive multimedia tool for teaching software testing based on Internet programming. The project leader has been conducting ongoing software engineering measurement experiments to monitor and evaluate student projects on a progressive basis for a number of years. The objectives of these experiments have been to improve the quality of the software process and product by measuring the testing and defect tracking activities of an O-O software system of various student teams through manual log sheets. Some of the learning outcomes of these experiments so far have been:
Currently, students studying the third year unit that focuses on testing participate in the ongoing software engineering experiment. We cover the principles of O-O design, programming and testing in conventional teaching mode in lecture rooms. The assessment of students’ work on O-O software development involves the tracking of their software testing process. Students receive feedback on their work and work with an improved understanding of especially the testing phase of the project for the next iteration of the project. The experimental results that measure students’ understanding correlated well against other assessment components for the subject [2]. But, it also highlighted the benefits of developing interactive software testing as part of our ongoing experiment to teach a deeper understanding how testing really works. A prototype case study for teaching software testing is being built this year. It is hoped that this prototype can be used as the first case study for this project. Our proposed project fits nicely within the ongoing software engineering measurement experimental framework. It will provide an additional alternative self-paced teaching environment for software testing. Students can then be required to complete these self-study teaching modules of varied complexity before attempting the major assignments. The project is feasible provided resources are made available. Both investigators are active in teaching and researching in the area of object oriented testing [1,2,3,4,5] and have relevant expertise to conduct this project. The associate investigator has supervised Masters theses in O-O testing and computer-mediated education. A research software test system, OZTEST [4], was constructed under his supervision. The investigators have received funding in 1996 for a small ARC grant application titled "A method and a Tool for Object-Oriented Testing using Attribute Grammars". Jason Lowder, one of our PhD students is an expert in incorporating multimedia technology into improvements in teaching, monitoring and feedback and his work (with Dianne Hagan) received an ASCILITE (Australian Society for Computers in Learning in Tertiary Education) award in 1996 for developing "SFT1102NET" for one of our first year subjects. We will be seeking his expertise in implementing internet feedback mechanisms for this project. Both investigators have used the traditional teaching methods of lecturing and programming exercises in teaching the subjects, and are convinced that an alternative approach based on computer-based visual interaction and student-centered learning is needed for students to master the complexity of software testing. 5.1 Needs Analysis: i. Target Audience
Characteristics: Students undertaking these subjects are involved in writing and testing object-oriented software using different O-O languages, and at various levels of complexity. ii. Target Computers
iii. Results of Review of Existing Resources As detailed in Section 3, our literature and database review revealed that none of the available resources (example references are [4 , 11]) address the content described in this project. 5.2 Project feasibility i. Use of Various Media Video and audio clips will be used to present an overview of the various case studies to the students. Still images are required to present the internal workings of the systems, i.e. the OO dynamic models of state-transition diagrams and event interaction diagrams. Animations of the diagrams will show the test paths traversed and the conditions under which the transitions are valid. These diagrams will be used to teach OO testing by developing an understanding of how test plans can be derived from these diagrams. ii. Interactivity The interactive system will allow users to either choose test-cases from a list of inputs provided or key in test-input values. It will check the validity of user inputs and report any errors. The system will respond by animating the relevant paths of the diagrams for a given set of inputs. These interactive systems will be constructed for case studies with varying degrees of complexity to teach students how to derive test plans for testing these systems by visualising the paths taken by the software during execution. iii. Expected Deliverables The outcome of this project will be an Internet based multimedia program with accompanying case-study information available on the WWW. Printed user manuals will also be produced. iv. Proof of Concept Appendix 1 gives the electronic version of two dynamic model diagrams for the prototype that is currently being developed. Three more case studies are planned. v. Project Development Timeline
vi. Plan for User Testing, Formative and Summative Evaluations Formative evaluation Stage 1: Content evaluation by project reference group members and the project investigators The prototype case study that is being developed will be evaluated by: · students · project reference group members · project investigators Stage 2: As part of the formative evaluation of the prototype case study 1, the system will incorporate quizzes and questionnaires in the on-line forms (as described in Section 6). Evaluations will be conducted in the Multimedia Windows NT lab of the Department of Software Development, Monash University. Stage 3: Prototypes of case studies 2, 3 and 4. Content evaluation by project reference group members and the project investigators. Feedback from previous prototype evaluations will be taken into account in developing these prototypes. Prototype evaluations of case studies 2, 3 and 4 by: · students · project reference group members · project investigators Stage 4: Learning evaluation of the entire program: monitoring the learning outcomes to check what is being learnt and how the students are learning and gaining this knowledge while using the program? Users will be required to respond to the case studies used in our project and other related situations both prior to and also after using the interactive learning tool for a subjective measure of the learning tool and the learning outcomes achieved. Stage 5: Summative evaluation The knowledge that has been gained on OO testing will be evaluated using focus groups and student interviews. Lecturers of the students will also be interviewed to check how well the students develop test plans and carry out OO testing as compared to previous student groups. vii. Detailed Breakdown of Resources and Personnel Required Resources required: Available in the Department of Software Development · Multimedia Windows 95 Pentiums or Windows NT machine for development · Scanner for scanning images required for the case studies · Multimedia Windows NT lab for evaluating students (once the system is operational, students can run these programs from remote sites via the WWW) · JAVA JDK 1.1.1 ( downloadable via the WWW for free) · Rendering package - Web 3D · Video Blaster - video card for encoding video files · Real Audio and Video Viewers (needed for stream video applications) - ( downloadable via the WWW for free) Needs to be budgeted · Web Editor - Microsoft Front Page 97 - $170/- · Graphics - Corel Version 7.0 - $150/- · Real Audio and Video servers (http daemons) for 20 stream video applications - $700 · Animation - Macromedia Director 5.0 - $895/- (Total - $1915)
Personnel required: As detailed in section 8, this project requires · The two investigators to provide content expertise, supervise and coordinate the project · A competent Internet programmer for a year · An evaluation expert for 360 hours 6 Evaluation (formative and summative): This project will be evaluated for its effectiveness in teaching this topic and will also monitor the amount of usage of the various modules. The purpose of our formative evaluation will be to identify any problems in the forming material and improve the teaching modules developed as part of this project before it is finalised. Summative evaluation will compare the new internet environment for teaching software testing modules with previous software testing teaching material. We will adopt the evaluation process suggested in the chapter on "Formative evaluation of web based training" [10]. As students are expected to work on these teaching modules as independent internet based study, away from peers and the lecturer, on-line forms will be incorporated in each teaching module for collecting open-ended textual feedback from participants. Web-based questionnaires are also planned to get students to complete multiple choice questions to rate the interactive multimedia modules for courseware pace, screen design and information content. Monitoring and evaluation will include:
7. Dissemination plans: Initially, the effectiveness of the project will be evaluated across the student population within Monash University and any improvements that are required will be introduced. Then the software system will be made available to the wider academic community via the Internet. The information will also be disseminated through the CUTSD clearing house, and through publications in educational and technical conferences. 8. Detailed budget with justification and explanation of project’s cost effectiveness: The project will require a competent programmer with a good working knowledge of various software such as Unix, scripting languages, Latex, HTML and JAVA. She/he needs to be guided closely by the applicants as experts in object-oriented testing. We also require an evaluation expert on a part-time basis. The resources requested is in line with what is suggested by multimedia course developers, educationalists and researchers such as Malcolm Eley, and Heinz Schmidt. <Resource details deleted>
The programming environment which will be used for this project is the Internet programming language, JAVA, available for free and can be downloaded from the Internet. Both investigators have teaching, research, committee and postgraduate coordination commitments and are not able to complete the project themselves. Therefore, we will each require 4 hours a week teaching relief from tutorials so that we can supervise and coordinate the development of the proposed teaching tool. The development will be carried out by a competent Internet programmer under our supervision. The project will require one year of full-time work to develop a suite of web-based programs in an Internet Environment for learning software testing processes. The evaluation expert will implement the on-line forms to gather feedback as part of the formative and summative evaluation of the teaching tool. We will require the project team to spend:
9. Evidence of unit and institutional support: The Department of Software Development received an ARC Infrastructure grant in 1996 for a Multimedia laboratory. The Department will also provide financial and in-kind assistance which include:
10. Previous project and financial management experience: The primary investigator received infra-structure funding of $25,000 in 1991 to set up the Object-Oriented stream of study in our department. The money was utilised to meet the hardware and software requirements. In 1995, she received faculty funding of $14,000 for her project titled "Assessing the components of O-O systems using k-th connectivity between program elements". This funding was used to allocate scholarship to a Masters student who has submitted her work on time under Ms S Ramakrishnan’s supervision. The second investigator has received ARC small grants in 1992, 1993 ($20,000) for a project titled Persistent Reusable Object Environments (PRObE) which was completed successfully. In 1995, he received another ARC small grant for a Parallel Software Engineering project which is nearing completion. Dr Sajeev is also an associate investigator of a joint research project with CSIRO and software industrial partners on Distributed Object-Oriented systems with a funding of nearly $150,000. In 1997, both investigators have obtained funding for a small ARC grant in the area of software testing research. We have a Masters student working on this project from March 97. Professional Background Sita Ramakrishnan is a Lecturer in the Department of Software Development. She has a major interest in quality in teaching and has published in Software Engineering Education Conferences. Ms Ramakrishnan is an ex-member of the Faculty Education Committee; in this capacity she has played a key role in introducing software engineering subjects in the Bachelor and Master of Computing curriculum. She teaches an undergraduate level course titled OO Programming Systems with a focus on OO testing. She teaches a Masters level course titled "Multimedia Web Applications using JAVA". She has been the Organising Chair for the Annual International Conference on Technology of Object-Oriented Languages & Systems, Melbourne for several years. She has published research papers on quality, testing, metrics and object-oriented reuse. A. S. M. Sajeev is a Senior Lecturer and Deputy Head of the Department of Software Development. He has supervised research students in the areas of object-oriented systems, software testing, software reuse and computer mediated education and has published papers in these areas. Dr Sajeev has been given the Best Lecturer award by Computer Science students at James Cook University in 1992. He has been instrumental in introducing Advanced Software Engineering in the Computer Science curriculum at James Cook University. References 1. Ramakrishnan S. Quality factors for resource allocation problems: Linking domain analysis and object-oriented software engineering. In the First International Conference on Software Testing, Reliability and Quality Assurance (STRQA 1994), pages 68--72, IEEE Publ., Dec. 1994 2. Ramakrishnan S. An Experimental approach to object-oriented software process and product measures. In Technology of Object-oriented Languages & Systems, pages 91-97, Prentice-Hall, Nov. 1995. 3. Ramakrishnan S. and Menzies T. An ongoing object-oriented software engineering measurement experiment. In Software Engineering Education & Practice Conference (SEEP 96), pages 345--350, IEEE Publ., Jan. 1996 4. Fletcher, R. and Sajeev, A. S. M. A framework for testing object-oriented software using formal specifications. In Reliable Software Technologies, Lecture Notes in Computer Science, Springer-Verlag, pp. 159-170, 1996 5. Trathen, C and Sajeev, A. S. M. A protocol for computer mediated education across the Internet. British Journal of Education Technology, 27(3), pp. 204-213, 1996. 6. Ramakrishnan, S., Menzies T., Hasslinger M., Bok P., McCarthy H., Devakadacham B. and Moulder D. On building an effective measurement system for OO software process, product and resource tracking. In Technology of Object-oriented Languages & Systems, pages 239--247, Nov. 1996. 7. Jezequel, J-M., Engineering Telecom Systems with Eiffel. In Technology of Object-Oriented Languages & Systems (TOOLS Pacific 96), Tutorial session, Melbourne, Nov. 1996 8. ESA Launchers , European Space Agency and Space Transport Systems in http://www.esrin.esa.it/htdocs/esa/progs/launch.html 9. Conceptual Learning of Sciences (CoLoS) in http://hpwww.ec-lyon.fr/colosHp/ 10. Web-based Instruction, Ed. B.H.Khan, Feb. 1997, Englewood Cliffs, NJ 07632-0564 11. R. K. Doong and P. G. Frankl, The ASTOOT Approach to Testing Object-Oriented Programs, ACM Transactions on Software Engineering and Methodology, 3(2):101-130, 1994. |
Funded by Committee of University Teaching And Staff Development (CUTSD) through DEETYA, 1998