An analysis of large-Scale numeracy assessment data in Australia

This thesis analysed large-scale assessments in Australia from a perspective synthesised from two disciplines: Systems Engineering and Education Research. Large-scale assessments are used internationally for the purpose of improving education through the provision of evidence of educational attainment to interested parties such as teachers, schools, parents, researchers and government entities. The assessments’ purposes include increased accountability and the provision of diagnostic information. However, scant research exists on how the data obtained from such assessments might validly and effectively be used to achieve these purposes. This thesis addresses the extent to which large-scale assessments achieve their accountability and diagnostic purposes. With a focus on Numeracy assessments in Australia, the thesis addressed the interpretation and use of assessment data, and includes a detailed analysis of Victorian data from the 2006 Achievement Improvement Monitor (AIM) assessment and the 2008 National Assessment Program - Literacy and Numeracy (NAPLAN) assessment. Exploratory Data Analysis techniques were used to investigate and illuminate the data. The data analysis presented contributes to the ability to use the existing data for diagnostic purposes, thus partially compensating for the effects of the constraints on test design as a consequence of the multiple purposes that the tests are expected to fulfil. The results of the analysis suggest that children’s success with numeracy assessments is mainly dependent on factors that underpin performance as a whole. Such factors include a strong sense of number and an analytical approach. The implication is that attempts to improve scores by narrowing the curriculum and teaching procedures for solving particular item types may be counter-productive compared with approaches which develop mathematical understanding and developing children’s ability to approach problems that are unfamiliar. The results of large-scale assessments are valuable in providing an opportunity to reflect on what additional experiences might develop children’s abilities. The analysis suggests that narrowing the curriculum is likely to be counter productive. However, an increased emphasis on sense-making in mathematics may be an effective course of action for teachers to take to increase scores. The thesis argues that the large amount of data collected is a resource which has the potential to provide teachers with diagnostic information for planning purposes but is under-utilised. This situation arises because the relationship between student outcomes and teacher practices is not direct, and therefore the process of drawing valid inferences from the data requires careful consideration. The systems analysis presented raises questions about what outcomes can reasonably be expected from large-scale assessments. The work has implications for the use of large-scale assessments as feedback for teaching practices, teacher educators and curriculum and policy development.