Previous: Method and results  Main page  Next: Method

Datasets Students sitting CSE1301 in 2001 were required to undertake 14 distinct summative assessment tasks: A practical component comprising 12 weekly pracs, which were marked in situ by the students' demonstrators (lab tutors). This component was worth 30% of the final mark. Six of the pracs also contained some bonus questions, whose marks were recorded separately. A mid-semester test, worth 10% of the final mark. A 28-page final exam, worth 60% of the final mark. Choice of activities Activities were chosen for inclusion in the study on the basis of availability. The aim was to maximise the number of activities included, taking ease of availability and likely usefulness into account. The prac and bonus marks were easy to get, as they were entered into a database at the time of marking. The exam and the mid-semester test were delivered on paper, and although it would have been easy to acquire an electronic list of total marks, it is impossible to infer much conceptual structure from a single numeric mark. The finer the granularity of measurement, the better the chances of being able to infer conceptual structure. Therefore, the physical exam papers were acquired, and the page totals were transcribed from the front of each exam. This was a time-consuming task, so it was decided not to repeat the process with the mid-semester test and therefore the test results are not included in this study. This still left 12 prac marks, 6 prac bonus marks, and 28 exam pages --- a total of 46 marks per student, which was felt to be ample. The prac and exam marks were combined into a single file, one line per student and one column per activity. At this time, the data was stripped of identfying features in accordance with Ethics Committee guidelines. Therefore, no breakdown on student demographics (for example, gender or ethnicity) was possible using this data. It was, however, possible to break students down according to their final mark. The activities that were used comprise 12 pracs, ranging from an introduction to using the computer to an advanced assignment; six bonus prac questions, which were not compulsory and were not attempted by many students; and 28 exam pages. These activities are described in more detail in Appendix II. Data validation Before analysis began, the data was vetted to ensure that the marks values were sane. This entailed: checking that no student was recorded as achieving more than the maximum mark for any assessment task. Such records were considered to be erroneous, and the whole line was removed before analysis. This only affected two students, and all other marks were assumed (perhaps rather optimistically) to be correct. removal of records for students who had not sat the exam. Most of these students had dropped out during the year and had not sat at least some of the pracs, and it was felt that including these students in the data would skew the results. The remaining students had sat a supplementary exam, but the questions on that exam were different. replacing "absent" or "sick" marks in the prac section with zeroes. This does not accurately reflect the student's summative mark: a student who is absent from a prac does indeed score zero, but a student who is sick receives his or her average prac mark. However, for the purposes of this study, the summative mark is less important than the student's personal experience: regardless of earned mark, a student who is absent from a prac has not experienced it. This replacement affected 205 of the 454 students who were recorded in the prac database. It is worth noting that only 362 students sat the exam: many of the students who had been absent from pracs during the year had in fact dropped out of the course and had therefore been recorded as absent from pracs, and had not sat the exam. Most of the students who did sit the exam were only absent once or twice. After this modification, there were still 350 student records left in the main dataset, each comprising 46 separate activity marks: a total of 15,916 marks. Choice of datasets Several subsets of the data were chosen for analysis. The first dataset chosen for analysis was, of course, the full set ALL: all students and all activities. However, this dataset produced a graph with so many points on it that it was difficult to interpret it, so ways were investigated to reduce its visual complexity. A correlation matrix was calculated for the data, using Pearson's r. The first look at the matrix showed some surprising results: correlation coefficients between exam questions were relatively high across the board, mostly in the neighbourhood of 0.7, whereas correlation coefficients between pracs tended to be around 0.35. Prac bonus questions correlated poorly with almost everything. Examining the raw data showed that very few students had attempted them, and they were eventually removed from consideration from most datasets, leaving 40 activities per student. Note that the sets ALL and TMA include the bonus questions. When the exam paper was analysed, it was found that the first ten pages consisted of multiple-choice questions, four to a page. Another four pages contained short-answer questions, six to a page. If the questions on each page had been related, this would not have been a problem; however, the questions were heterogeneous. So many factors contribute to the marks for each page that it is unlikely that cluster analysis would be able to do much to draw them out, so datasets were generated with totals, rather than page marks, for the multiple-choice and short-answer questions. This left 28 unique activities per student. Note that set ALL is the only dataset that includes separate marks for each page of multiple-choice and short-answer questions.

Concept mapping for introductory programming Thesis main page Introduction and background     Background: education     Assessment     Background: concept maps   Aims     Competency mapping     Benefits   Method and results     Data sets     Method     Results     Random data   Analysis and conclusions     Factor analysis     Cluster analysis     Methodological problems     A better test     Conclusion   Appendix I: Datasets Appendix II: Activities Appendix III: MDS coordinates Appendix IV: Data generation scripts   Bibliography

Previous: Method and results  Main page  Next: Method