February 2001

The analysis of variables related to subject evaluation has traditionally focused on the mechanics of delivery and formal performance measures. Our concern is to take a different approach to subject evaluation by investigating student satisfaction with the subject. Evaluation through student satisfaction, if it focuses upon student behaviour, provides a basis for modeling the interactions between what students expect, what is delivered, performance and outcomes. We present a student satisfaction model, giving an application example and the results of that evaluation. The results from the study indicate the potential complexity of a behavioural approach to subject evaluation.

The analysis of variables related to subject evaluation - teacher performance and student outcomes - has, traditionally, focused upon the mechanics of delivery and formal performance measures. There is a wealth of research which is concerned with the way the teacher delivers his/her material, as there is on the factors which effect student performance [7]. Interestingly enough there is not a substantial body of work, which relates the two.

Our concern is to take a different approach to subject evaluation by investigating student satisfaction with the subject. The rationale behind this approach is that even though a subject may exist as an entity in a course handbook and may represent the academic interests of its creator, it is the student reaction to the subject, which determines the effectiveness of the course. Gone from most subject areas are the conditions where the academic determines input and disregards student reaction to that input. Most educational institutions operate in a market economy where competition is rampant.

Evaluation through student satisfaction, if it focuses upon student behaviour, provides a basis for modeling the interactions between what students expect, what is delivered, performance and outcomes. Through such modeling, it is possible to isolate which factors are most important to students and which relate most to direct outcomes such as the recommendation of the subject and/or course to others or an interest in doing another of this lecturer's subjects. Not only is the evaluator able to look at functional elements within his/her subject, but he/she is able to position the subject within the competitive market place.

Malley [4] has carried out one of the few detailed surveys of research into student satisfaction, although his emphasis is upon the technical education sector. He concluded that a general model which takes into account student expectations of subjects, the student experience of the subject and outcome behaviours of those students, is the most appropriate methodology to be able to obtain a realistic picture of satisfaction behaviours.

It appears that there is, in fact, no operational model, which has been applied to student satisfaction behaviour. There is a limited history of attempting to develop models of student behaviour and to relate them to student outcome behaviour [5] but none have been found which work, specifically, with satisfaction. Consequently, we have adapted a model which has been used in customer satisfaction research [8] but which has an underlying behavioural model based upon the general area of expectancy-value theory [1].

The key to this approach is that it assumes that student perceptions of their educational environment are critical in determining their level of satisfaction and their outcome behaviours. Few university lecturers would deny that even when they are teaching a very dry subject, the prerogative lies with them to involve the student. Sitting at the feet of the master (no matter how boring the master is) is no longer seen as a viable pedagogical model in higher education.

The research reported gives mainly the qualitative data from the application of this type of approach to subject evaluation. The quantitative data will be reported later.

The general design for researching satisfaction, within the framework of the previous section, has these steps:

1. Collect information from the population under study on what is seen as the main parameters associated with being satisfied.
2. Develop a questionnaire based upon this information and include any other questions associated with satisfaction. Include at least a general open-ended section to obtain user comments.
3. Administer the questionnaire.
4. Analyse the quantitative data using regression modeling.
5. Check the quantitative results against the qualitative ones. See if anything was missed from the questionnaire and note it for further studies.

This process was implemented in Information Technology Project Management (CSE2203) by first asking a small sample of students to write down the important factors in being satisfied and dissatisfied with the subject. Then the questionnaire was developed which incorporated their comments and included other material of interest to the lecturer. It also included two open-ended questions, which will be the focus of this paper.

The questionnaire was administered on-line through a Web form. All subject tutors were asked to inform students of the URL for the survey and to allocate some time for its completion.

CSE2203 introduces the fundamental principles, tools and techniques of software project management. The conceptual material presented in lectures is reinforced by practical application within the context of a software development project. Students work in project teams with roles allocated to each group member. The project is defined against a set process model. Project definition, estimation, and tracking and reporting techniques presented in lectures are employed during the course of the project. Real-life problems are injected into the project in the form of changes to user requirements, budget and time-lines. Emphasis is placed on the ability to provide up-to-date management information on the actual state of the project against established milestones: reports are requested on an ad-hoc basis. A project review phase is used to analyze and report on project estimates against actual time, cost and resource expenditure.

This subject presumes a knowledge background gained in the two other core Software subjects in the Bachelor of Computing: CSE2200 Systems Design and Implementation and CSE2201 Software Engineering Practice. Hence students are expected to use their software engineering and systems analysis and design skills in the context of project management. Students are expected to have learned and used the Personal Software Process [3], and are introduced to the Team Software Process [2] in CSE2203.

Due to intake from other courses (e.g. higher diploma), the students enrolled in CSE2203 do not always have the necessary pre-requisites. Some undertake the pre-requisite subjects simultaneously. To accommodate this factor, templates and examples are provided for components such as project, quality and test plans.

Students’ teams are assigned a small software development project that encumbers a graphical user interface. This semester, it comprised an ExpenseManager Application. Teams are expected to build their application in an object-oriented language and development environment that they are familiar with, and should choose to do their development in C++ Builder, Eiffel or JBuilder. Pre-requisite knowledge assumes teams are experienced in the chosen language, and CSE2203 does not focus on new programming language acquisition. There are no tutorials or scheduled help for programming problems. This reflects a real life simulation of projects where developers need to source their own solutions to problems. With this focus, student assessment is higher on project management skills than on product produced. To encourage well-designed and quality products, a competitive element is introduced. At the conclusion of the semester, best products are peer selected from each tutorial. These are presented at the final lecture for students to view other ideas and see what their peers have produced. They then vote on the best product for the semester, and the winners are presented with certificates and a small prize. This competition is well received and encourages effort to produce a good product.

This semester had a large number of student intake (200), and it was difficult to find enough suitably qualified tutoring staff. Many lecturers can appreciate this caused some problems in the efficient administering of the subject. This was evident during the semester and in the information collected from the population to design the questionnaire (section 3).

Of the possible 200 students there were 110 useable responses. This is far below what was expected but it was noteworthy that a number of tutorial groups had very poor responses. This issue will be discussed later.

Male 68%

Female 32%

Yes 79%

No 21%

No 90%

Part time 8%

Full time 2%

Median lectures attended 10

Median tutorials 13

i.e. this was an extremely skewed distribution with 60% claiming to have attended 13.

The scales used to assess overall satisfaction with the subject produced the following results:

Both of these results suggest that students are, on the average, satisfied with the subject. What is important at the behavioural level is the statement of satisfaction is higher than that for enjoy. There is a correlation 0f 0.75 between them indicating that students rate satisfaction and enjoyment in a similar way but it may be the case that satisfaction is not premised upon enjoyment - as some cynics might say of the modern student.

Two open-ended questions were included in the questionnaire. Of the 110 survey responses, 73 included comments to these. The most common response to "What was the best thing about this subject" was the simulation of project management with deadlines in a business atmosphere, preparing students for industry (51%). This was a positive response to the subject aim. The second most common response was the knowledge gained of working as a team (19%). A few students viewed the best product competition as the best thing (8%), whilst a couple commented that high programming skills were not required (3%). A small percent commented that their tutor was the best thing about the subject (11%).

In response to "What was the worst thing about this subject", students most commonly responded with "too many management tasks, too much documentation, too much work, and takes too much time" (67%). The second most common response was the dissatisfaction with the development environment and no programming assistance (16%). The tool used for configuration management (WinCVS [9]) was also found to be difficult to use, exasperated by tutors having insufficient knowledge (11%).

Some comments (2%) indicated students viewed tutors as the worst thing in the subject. Other comments that were interesting to compare are that the subject should have more templates and sample documents (3%) whilst one student commented that too much work was similar in other subjects. There seems to be some conflicts here consistent with the fact that students have varying levels of background knowledge.

Specific comments (3%) on the lecture indicate that students would enjoy hearing more industrial examples. Whilst they experience an industry presentation in the second last week of semester (which was very well received), more examples and case studies could be included at all the lectures.

Individual comments included "marks should only be allocated for project management, i.e. the product should not be assessed" and the subject "needs more management emphasis, make the students do the work". These 2 comments are difficult to address. As mentioned earlier, the assessment emphasis is on project management, and the team development experience, with some assessment allocated to the product. Given that project management is about being self-motivated and a leader, it would be difficult for the subject leader to make students do the work. This is currently addressed by including penalty marks for documentation submitted late or done at an unsatisfactory level. Students are also required to submit peer evaluation of their team members with each product delivery. This ensures individual student marks are adjusted for those that do not contribute satisfactorily.

With large student numbers and intakes from different courses, it is imperative to have suitably experienced qualified tutoring staff. As was mentioned earlier, some tutorial groups had very poor responses, indicative of the problems experienced with these tutors during the semester. These tutors didn’t seem to have the experience for the subject and didn’t appear to provide sufficient subject support. The instructions to allocate time for students to participate in the subject questionnaire were also not carried out.

The diversity of student background (79% full fee paying implying they have come into the course and don’t have the background knowledge expected) is difficult to address.

Some students complained that the material covered in CSE2200 and CSE2201 was repeated in CSE2203. A perception seems the use of knowledge (e.g. project, quality, test plans, PSP) learned in other subjects as duplication in the course. Other student without this background knowledge complained that not enough instruction, templates and examples were provided. This may be difficult to resolve, however more specific templates with less detail will be supplied.

As mentioned earlier, students should already have been familiar with the development environment that they have chosen; however it appeared many did not. A number of students mentioned that they only knew Visual Basic, as they came into the course from a different background. These students were encouraged to join in with other teams who had the pre-requisite knowledge, but they usually chose to stay within their own social circle. As students appear to require more support for the development language and environment, this will be restricted to one (e.g. JBuilder), and students will be given the instructional exercises provided in CSE2200 as a refresher to the tool.

The analysis of CSE2203 is a part of the development of a wider program within Monash University's Faculty of Technology, through its Computing Education Research Group, to investigate structured approaches to subject evaluation. This has led to ideas and implications for improvements to the subject. Some work has been carried out on another subject [6] while there are other studies yet to be reported on comparisons of subjects and on student motivation and satisfaction.

The results from the CSE2203 study indicate the potential complexity of a behavioural approach to subject evaluation. We believe that this is appropriate because the teaching/learning milieu is behaviourally complex.