Previous: Cluster analysis  Main page  Next: A better test

Methodological problems This research has not been without problems. The most obvious problem is the sample size: although many students were included in the study, only one semester's data was used. Competency mapping cannot be considered to have been thoroughly tested until the study has been repeated. When the tasks were examined closely, more problems became apparent. The exam was not well laid out for the purposes of competency mapping. As demonstrated in section 3.3, competency mapping works best with orthogonal data; however, the exam had up to six questions to a page. Even if each question only assessed two competencies, the mark for that page would reflect up to twelve competencies: this is too many to be able to extract reliably. This effect would have been ameliorated if we had used the marks for each question, rather than the page mark, but the page marks were written on the front of the exam and the per-question marks were much less accessible. A similar problem applied to the pracs. There was a three-week project carried out across pracs 10--12, marked by demonstrators during the prac class. If a student finished an assessable component of this project a week early, it is possible that some demonstrators marked it and entered it in the column for the previous prac. That would have distorted that student's contribution to the correlation matrix. Furthermore, it is not clear what effects, if any, the groupwork had on the final mark. Prac 12 combined searching, sorting and recursion with the final assessment for the group project. The low average mark of 5.33% for P12 could indicate some irregularity in entering that mark --- perhaps not all demonstrators keyed the mark for the assignment in the column for that week, or perhaps students simply did not have time to finish. Because Prac 12 was the last for the semester, it is also possible that not all demonstrators had entered their marks at the time the database was copied. It may be wisest to consider P12 an outlier. The methods used for data analysis need some improving. Neither the scree plot nor the Kaiser criterion seem to be adequate to give the right number of clusters. In some cases these two criteria differed by up to 10! A better method for assessing the number of clusters is needed. It is possible that minimum message length encoding (MML), which rewards goodness of fit while penalising complex models, might give a better estimate. Although this test of competency mapping was not as successful as had been hoped, the technique should not be discarded until these methodological problems have been addressed and the test has been repeated.

Concept mapping for introductory programming Thesis main page Introduction and background     Background: education     Assessment     Background: concept maps   Aims     Competency mapping     Benefits   Method and results     Data sets     Method     Results     Random data   Analysis and conclusions     Factor analysis     Cluster analysis     Methodological problems     A better test     Conclusion   Appendix I: Datasets Appendix II: Activities Appendix III: MDS coordinates Appendix IV: Data generation scripts   Bibliography

Previous: Cluster analysis  Main page  Next: A better test