Selby Markham

Dianne Hagan

The analysis of students' satisfaction with their course of study is an important research area within educational evaluation. With the growing concern for accountability in educational outcomes, the need for meaningful and stable measures has grown.

The conventional analysis of satisfaction has been based on the assumption that satisfaction is best seen in terms of student response to course components and the methods used by teaching staff. Much of this analysis has been focussed upon comparing mean trends in these components.

Some work has been done to define fitted models for student satisfaction. Malley (1998) has extensively reviewed this area and has shown that there is a need to carry out more research into structural models which can help explain the complexities of student satisfaction. The research which is being reported here is the application of an alternative approach to satisfaction which is derived from work based on customer satisfaction with products and services as developed by Fornell and others at the University of Michigan and extended into commercial applications through the work of the CFI Group.

Satisfaction is defined as being a consequence of the expectations and experiences of the subject and/or course. The general schematic of the approach is shown in Figure 1.

Teacher performance, in this model, is seen as only one of a number of antecedents of satisfaction. In fact, it is seen as contributing only when students perceive that teacher performance has dropped below a critical level or when teacher performance surpasses student expectations. That is, the performance of the teacher will reduce satisfaction when student feel that they are not being given enough information on how to pass the subject but will only increase satisfaction when his/her performance stimulates students well beyond personal, arbitrary standards of "interesting teaching". The complexity of this relationship is shown in the likelihood that where a teaching performance brilliantly but fails to give students a sense of what is formally needed, then the effect will be overall negative on satisfaction.

An important point about this approach is that it is not a simple linear model running from expectations to outcomes. It assumes that, along with most expectancy-value models of behaviour, that the outcome perceptions have an implicit feedback loop back to expectations.

The research to be reported here is a part of a wider study evaluating the impact of an educational support tool (called BlueJ) in a first year programming subject, CSE1202, in the Bachelor of Computing at Monash University. A longitudinal study was designed to collect data from students at various points during the academic year. Students were asked to identify themselves so that a continuous record could be created. The Monash University Ethics Committee approved the design under the conditions that there was voluntary participation with informed consent and that identifiable data was not available to teaching staff.

Web-based surveys were carried out at weeks 3, 6, 8 and 12 of Semester 1. The first survey collected a range of demographic data as well as data on the expectations of students about the subject and its outcomes. The second and third surveys were aimed at monitoring the impact of the educational support tool with both quantitative and qualitative data while the final survey was concerned with experiences and outcome evaluation for both the course and BlueJ, including the satisfaction measures.

From the definition of satisfaction it was hypothesised that satisfaction would significantly related to the experiences of the course including the experience in using BlueJ. Furthermore, satisfaction would be related to the outcome measures such as recommendation of the course and the university.

Satisfaction was measured by three questions:

Of the 345 students enrolled in CSE1202, 121 agreed to participate in the study under the conditions prescribed by the Monash University Ethics Committee. Of that 121, there were 101 responses to the first survey and 76 to the final survey. Due to the vagaries of student response patterns, the final data file had only 32 students who had responded to all surveys and 53 who completed the first and last surveys.

From comparisons with previous data collection exercises in CSE1202, the respondents do not differ in any marked way from what is assumed to be the underlying characteristics of student intakes into the course.

No significant t-tests between problem faced, their summation and satisfaction - this is the final survey problems data.

There was no consistency between survey 1 and survey 2 responses for quantity of problems.

All satisfaction measures were significantly correlated as is shown in Table x. It is noteworthy that satisfaction with the subject is less strongly related to course and university satisfaction than the latter are to each other. An inspection of the distributions of the three measures shows that subject satisfaction has an uneven distribution (Table x) with a tendency to multi-modality at scale points 1, 3 and 5. The distributions for the other satisfaction variables are more evenly distributed with modes at about scale point 5.

Satisfaction with the subject was significantly correlated with initial expectations of passing the course for the 53 respondents who had completed both the first and last surveys (Table ). The correlation between satisfaction and initial expectation of passing the year and degree are not significant.

Expectations about the type of teaching (lectures, tutorials, discussion groups, problem solving groups) students expected to receive were not significantly related to satisfaction.

The data in Table indicates that there is only a low level of relationship between prior programming experience and/or training and satisfaction with the subject. There is no significant relationship with the other satisfaction variables.

* p<0.05 n=53

In the final survey students were asked to give performance ratings on

Having kept up with the work

Confidence in passing the exams

They were also asked to rate CSE1202 in terms of its performance requirements by comparing it with other subjects

The pace of the subject

The level of content of the subject

Overall comparison of difficulty against other subjects

These two areas proved to be marginally distinct when the five items were factor analysed, where the eigenvalue to extract a second factor was bordering on the standard cut-off value of 1.0.

Two regression were carried out against satisfaction with the subject. The first used the personal performance variables and it produced R² of 0.629 and a significant ANOVA for regression versus residuals (F=72.97 p<=0.05 Df 2,83). Both independent variables produced significant beta coefficients but the primary contribution to change in satisfaction came from "confidence in passing the exams" (b=0.605 t=4.674 p<=0.05).

The second regression using the subject-oriented performance variables gave an R² of 0.230 with the ANOVA being significant (F=9.473 p<0.05 Df 3,82) and the only significant contribution to the dependant variable coming from "Comparison with other courses".

The three satisfaction measures were regressed on the two recommendation measures. The model for satisfaction and recommending the course had an R² of 0.691 and an ANOVA with F-ratio of 59.693 (p<0.05 3/72). Table x gives the model coefficients and from there it can be seen that three sources of satisfaction make significant contributions although it is satisfaction with the course which has the largest Beta coefficient.

The regression of satisfaction on the recommendation of Monash had an R² of 0.748 and an F-ratio of 73.139 (p<0.05 3/74). It is clear from the coefficients from the model that only satisfaction with Monash makes a significant contribution.

The relationship between satisfaction and the BlueJ programming environment was rather complex because there were three stages in the data collection where students were asked to make evaluations of the software. Table gives the coefficients for satisfaction and the final evaluation of BlueJ and this was based on a model with an R² of 0.555 with and F-ratio of 15.99 (p<0.05 6/77). It can be seen that the user interface makes the only significant contribution to the model. Neither of the other satisfaction measures generated a useable regression model.

Regression analyses were carried out on the two intermediate evaluations of BlueJ against satisfaction but none of the models was significant.

The students who participated on the project had significantly better performance on all aspects of the assessment in the subject (see Table x) and the students who completed the final survey performed better (t=2.814 p<0.05 Df 113) than those who did not complete it (Table x).

Satisfaction ratings were regressed onto the final assessment. The model produced an R² of 0.387 and the F ratio for the model was 12.22 which was significant at p<0.05 (3/58df). The model coefficients (table x) show that the primary contribution comes from the satisfaction with the subject.

It cannot be assumed from these results that satisfaction has a direct causal relationship to performance. It is much more likely that continuing acceptable performance over the semester will produce a greater sense of satisfaction with the course and the performance and satisfaction jointly change.

The results from the research provide continuing data on the complexity of student satisfaction. Of particular interest is the differences between the satisfaction with the subject and the satisfaction with the course and the university where there is a stronger relationship between the more global measures than with the specific subject measure. Within this particular subject environment, it is clear that the subject is viewed differently from the overall course and the university within the context of student satisfaction. This is in agreement with the basic structure of the satisfaction model (Figure 1) because there will be different saliences applied to different components of the higher education experience.

The differential nature of the students' expressed satisfaction is also reflected in the outcome measures. In the first place, it is only the satisfaction with the subject with contributes significantly to total assessment suggesting a specific behavioural link in the model. In the second, the recommendation of the course and of the university are linked to their appropriate specific source of staisfaction.

The nature of the distribution for subject satisfaction (Table ) raises some questions about the interpretation of the analyses. Those respondents giving a rating of 1 might be seen as outliers but this was not shown by an inspection of the Box-Plot for the variable. It could be argued that their responses probably reflect the reality of the experience of the programming support tool. A number of students had difficulty in installing and running the software and this might be assumed to impact on satisfaction.