20161123-Heffernan-Thesis.pdf (10.27 MB)
The measurement of understanding
thesis
posted on 2017-01-13, 02:40 authored by Michael William HeffernanThe project reported in this thesis has two major concerns - the development of a rating form to measure cognitive understanding and a review of the nature of understanding. The development of the rating form hinges on the evolution of a scale of operationalised items that would be ordinal, hierarchical and function independently of a given content area. The manner in which these objectives are addressed is as follows.
Chapter 1 presents a brief review of the project objectives and process. It reviews the events during the project that brought about changes to the project protocol, resulting in a major review of the nature of understanding. Chapters 2, 3 and 4 are given over to this review and its results.
The initial portion of Chapter 2 formulates a view of the scientific and philosophical dilemma arising from the lack of definition inherent in the current use of the concept of understanding. In this formulation the characteristics of understanding (as culled from the literature) are listed and discussed. Issues such as the non-dichotomous nature of understanding and whether or not recall of memories constitutes a kind of understanding are the substance of this section.
The final section of the chapter is given over to the presentation of a model that conceptualises the author's view of understanding and distinguishes between understanding and cognition(s).
'Cognitive understanding is the name given to that reorganised portion of an individual's cognitive structure produced by the interaction of the pre-existing structure with a particular stimulus.
Understanding is transitory in the sense that it may disappear, or be committed to memory, where it becomes part of the cognitive structure, and then exists as a cognition'.
Chapter 3 examines the implications of the model and definition. Some of these issues are philosophical e.g. the relationship between an individual's cognitive structure and his various understandings, and an explanation of how the concepts of quality and quantity may apply to understanding. Other implications deal with the conduct of teaching which aims at creating and measuring understanding. For example, it is suggested that understanding can only be measured by a test which creates the understanding which is to be assessed, and a protocol for developing such a test is outlined.
Chapter 4 summarises briefly what has gone before and leads logically to a consideration of how the rating form was developed. Chapter 5 reviews all of the major elements of the project - the objectives - developmental principles - the process of conceptualisation - the development of the interview schedules which guided the rank-order task - and like matters. It also discusses the major design questions (e.g. how to control for anticipated interactions) and the logic behind the decisions that were taken regarding the project's design.
Chapter 6 is an extension of Chapter 5 as it deals with the development of the items for the rating form's scale. It reviews the theoretical data base and the practical considerations upon which the items were based.
The data generated by the project, follows in Chapter 7, as do the conclusions based upon its analysis. The first part reviews the data relevant to the major research questions such as those dealing with the rating form's validity and the scale's hierarchiality and reliability.
The second part explores the responses to the tasks contained within each interview, presenting the collated views of respondents regarding the meaning of the items, the utility of the rating form and the like. It also explores unexpected issues generated by the interviews, such as the question as to what is the importance of being able to 'see' the relationship(s) between a problem's variables within the problem solving process.
Part three explores outcomes such as the implications for the construct validity of the items in the way respondents clustered them during the ranking task. It also presents the discussion relating to the final rank-order of the items.
The final chapter summarises the project's conclusions, outlines uses for the rating form and considers future developments. It is concluded, unexpectedly from the author's initial point of view, that the rating form, though a reliable and valid instrument, is generally not able to be used to measure cognitive understanding, as he now defines it. However it can be used to measure the extent of an individual's cognitions. Equally unexpectedly it is concluded that items within the scale may be useful to assess cognitive functioning and to teach people how to think.
The suggestions for future development fall into two categories, those related to the refinement of the rating form (e.g. a large scale rank-ordering of refined items) and those ideas suggested by the project, but not directly related to the rating form. Amongst the most interesting of these is the suggestion to re-evaluate the role which 'seeing relationships between variables' plays in problem solving in the practice of schooling and higher education.
Finally the appendices include such elements as the raw data, interview schedules and summaries of small projects undertaken within the larger protocol.
Chapter 1 presents a brief review of the project objectives and process. It reviews the events during the project that brought about changes to the project protocol, resulting in a major review of the nature of understanding. Chapters 2, 3 and 4 are given over to this review and its results.
The initial portion of Chapter 2 formulates a view of the scientific and philosophical dilemma arising from the lack of definition inherent in the current use of the concept of understanding. In this formulation the characteristics of understanding (as culled from the literature) are listed and discussed. Issues such as the non-dichotomous nature of understanding and whether or not recall of memories constitutes a kind of understanding are the substance of this section.
The final section of the chapter is given over to the presentation of a model that conceptualises the author's view of understanding and distinguishes between understanding and cognition(s).
'Cognitive understanding is the name given to that reorganised portion of an individual's cognitive structure produced by the interaction of the pre-existing structure with a particular stimulus.
Understanding is transitory in the sense that it may disappear, or be committed to memory, where it becomes part of the cognitive structure, and then exists as a cognition'.
Chapter 3 examines the implications of the model and definition. Some of these issues are philosophical e.g. the relationship between an individual's cognitive structure and his various understandings, and an explanation of how the concepts of quality and quantity may apply to understanding. Other implications deal with the conduct of teaching which aims at creating and measuring understanding. For example, it is suggested that understanding can only be measured by a test which creates the understanding which is to be assessed, and a protocol for developing such a test is outlined.
Chapter 4 summarises briefly what has gone before and leads logically to a consideration of how the rating form was developed. Chapter 5 reviews all of the major elements of the project - the objectives - developmental principles - the process of conceptualisation - the development of the interview schedules which guided the rank-order task - and like matters. It also discusses the major design questions (e.g. how to control for anticipated interactions) and the logic behind the decisions that were taken regarding the project's design.
Chapter 6 is an extension of Chapter 5 as it deals with the development of the items for the rating form's scale. It reviews the theoretical data base and the practical considerations upon which the items were based.
The data generated by the project, follows in Chapter 7, as do the conclusions based upon its analysis. The first part reviews the data relevant to the major research questions such as those dealing with the rating form's validity and the scale's hierarchiality and reliability.
The second part explores the responses to the tasks contained within each interview, presenting the collated views of respondents regarding the meaning of the items, the utility of the rating form and the like. It also explores unexpected issues generated by the interviews, such as the question as to what is the importance of being able to 'see' the relationship(s) between a problem's variables within the problem solving process.
Part three explores outcomes such as the implications for the construct validity of the items in the way respondents clustered them during the ranking task. It also presents the discussion relating to the final rank-order of the items.
The final chapter summarises the project's conclusions, outlines uses for the rating form and considers future developments. It is concluded, unexpectedly from the author's initial point of view, that the rating form, though a reliable and valid instrument, is generally not able to be used to measure cognitive understanding, as he now defines it. However it can be used to measure the extent of an individual's cognitions. Equally unexpectedly it is concluded that items within the scale may be useful to assess cognitive functioning and to teach people how to think.
The suggestions for future development fall into two categories, those related to the refinement of the rating form (e.g. a large scale rank-ordering of refined items) and those ideas suggested by the project, but not directly related to the rating form. Amongst the most interesting of these is the suggestion to re-evaluate the role which 'seeing relationships between variables' plays in problem solving in the practice of schooling and higher education.
Finally the appendices include such elements as the raw data, interview schedules and summaries of small projects undertaken within the larger protocol.
