Previous: Factor analysis  Main page  Next: Methodological problems

Cluster analysis It was hoped that the clustering would provide an insight into the structure of the course, ideally a competency-based decomposition that would be able to be used as the basis for assessment design. Unfortunately, such decomposition can not be read from the results that we obtained. However, some valuable and unexpected insights were gained in the process of the analysis. The most obvious result is that prac tasks are clustered with prac tasks and exam tasks with exam tasks. This tendency seems more marked among the less able students and less marked among the better-performing students. Compare the clustering in the maps for BMAB, AMAB and TMAB. In the first of these, which shows the results from the bottom third of students, all but one of the prac questions cluster together. In the second, showing the results for the whole student body, the pracs cluster with exam tasks about digital logic and test data. In the third, representing the top third of students, the pracs (excluding prac 11) cluster with exam tasks on digital logic, data structures, test data, and the two smaller programming questions. Clearly, the better a student performs at computer science, the more strongly correlated her prac and exam results are. This could be interpreted as simply showing that a student in the top third of the class gets good marks at everything, but the reality is more complex than that: if the observed tendency is simply the result of selecting students on the basis of ability, then the bottom third of students (which is likely to be as homogeneous in terms of ability as the top third) should also show stronger correlations. In fact, the observed correlations are weaker for the bottom third of students. This may indicate that the weaker students are not applying theory to practical situations, or are not allowing lessons learnt in practice to illuminate their understanding of theory, or it could indicate some overriding factor that affects one kind of task but not the other: for example, difficulty using the computer system or difficulty with English. This second possibility mirrors the idea of the "modal competency" that was introduced in Section 3.3. It is significant that the exam task for which students were asked to develop sets of data to test a function is clustered with most of the prac questions for the whole-group and top-group data. The correlation between the test-data task and the prac marks could indicate that the ability to test code fully is a marker for the ability to program; alternatively, it might only mean that only students who finish the pracs get practice at testing. The former hypothesis is easy to test: give intensive lessons in software testing to a group of students, and see whether their programming ability improves as a result. It is rather surprising that the programming questions from the exam do not cluster with the prac questions for the whole-group data. It is informative that the two smaller programming questions are clustered with the prac questions for the top third of students. This is further evidence that the majority of students do not apply theory to practice. Students' programming practices need to be investigated. The program design principles that students are told to apply would serve them well enough in an exam situation, but if they are actually applying a rapid-prototyping code-test-debug cycle in pracs, in which the compiler rather than the designer forms the first line of defence against bugs, then they will not perform well in any context where a computer is not present. In most cases, the harder questions cluster together. For example, for set AMAB, the exam questions about linked lists and bubblesort, the largest programming question on the exam, and prac 12 cluster together. For set ALL, the equivalent cluster contains the exam questions on linked lists and bubblesort, the largest programming question on the exam, and four of the prac bonus questions. None of these tasks had an average mark of more than 33%, and for prac bonus 8 and prac 12 the average mark was under 7%.

Concept mapping for introductory programming Thesis main page Introduction and background     Background: education     Assessment     Background: concept maps   Aims     Competency mapping     Benefits   Method and results     Data sets     Method     Results     Random data   Analysis and conclusions     Factor analysis     Cluster analysis     Methodological problems     A better test     Conclusion   Appendix I: Datasets Appendix II: Activities Appendix III: MDS coordinates Appendix IV: Data generation scripts   Bibliography

Previous: Factor analysis  Main page  Next: Methodological problems