Category Archives: curriculum evaluation

glass chess board set

Postpositivist Paradigm and Program Evaluation

Leave a reply
Advertisements

Within the context of program evaluation, different schools of thought or paradigms affect how evaluators do evaluation. In this post, we will look specifically at the postpositivist paradigm.

Postpositivism

The postpositivist paradigm grew out of the positivist paradigm. Both paradigms believe in using the scientific method to uncover laws of human behavior. There is also a focus on experiments whether they are true or quasi with the use of surveys and or observation. However, postpositivism will also take a mixed method (combining quantitative with qualitative) approach when it makes sense.

ad

The main differences between positivism and postpositivism are the level of certainty and their contrasting positions on metaphysics. Positivists focus on absolute certainty of results while postpositivists are more focused on the probability of certainty. In addition, Positivists believe in one objective reality that is independent of the distant observer while postpositivists tend to have a more nuanced view of reality.

The typical academic research article follows the positivist/postpositivist paradigm. Such an article will contain a problem, purpose, hypotheses, methods, results, and conclusion. This structure is not unique to postpositivism, but it is important to note how ubiquitous this format is. The example above is primarily for quantitative research, but qualitative and mixed methods follow this format more loosely.

Within evaluation, postpositivism has influenced theory-based evaluation and program theory. Theory-based evaluation is focused on theories or ideas about what makes a great program, which are realized in the traits and tools used in the program.

Program theory is a closely related idea focused on the elements needed for achieving results and showing how these elements relate to each other. The natural outgrowth of this is the logic model which identifies what is needed (inputs) for the program, what will be done with these resources (output), and what is the impact of the use of these resources among stakeholders (outcomes). The logic model is the bedrock of program evaluation in many contexts such as within the government.

The reason for the success of the logic model is how incredibly structured and clear it is. Anybody can understand the results even if they may not be useful. In addition, the logic model was developed earlier than other approaches to program evaluation and it may be popular because it’s one of the first approaches most students learn in graduate school.

The emphasis on theory with postpositivism can often be at the expense of what is taking place in the actual world. While the use of theory is critical for grounding a study scientifically this can be alienating to the stakeholders who are tasked with using the results of a postpositivist program evaluation. As such, other schools of thought have looked to address this.

Conclusion

Postpositivism is one of many ways to view program evaluation. The steps are highly clear and sequential, and generally, everybody knows what to do. However, the appearance of clarity does not imply that it exists. Other paradigms have challenged the usefulness of the results of a program evaluation inspired by postpositivism.

Program Evaluation Paradigms

Leave a reply
Advertisements

Program evaluation plays a critical role in assessing program performance. However, as with most disciplines of knowledge, there are different views or paradigms for how to assess a program.

The word paradigm, in this context, means a collection of assumptions or beliefs that shape an individual’s worldview. For example, creationists have assumptions about how life came to be that are different from those of people who believe in evolution. Just as paradigms influence science, they also play a role in how evaluators view the structure and purpose of program evaluation.

In this post, we will briefly go over four schools of thought or paradigms of program evaluation, along with a description of each and how they approach program evaluation. These four paradigms are

  • Postpositive
  • Pragmatic
  • Constructivist
  • Transformative

Postpositive

The postpositive paradigm grew out of the positive paradigm. Both paradigms are focused on the use of the scientific method to investigate a phenomenon. They also both support the idea of a single reality that is observable. However, postpositivists believe in a level of probability that accounts for human behavior. This assumption may have given rise to statistics which focuses heavily on probability.

Postpositivism is heavily focused on methods that involve quantitative data. Therefore, any program evaluator who is eager to gather numerical data is probably highly supportive of postpositivism.

Pragmatic

A pragmatic paradigm is one in which there is a strong emphasis on the actual use of the results. A pragmatist wants to collect data that they are sure will be used to make a difference in the program. In terms of data and methods, anything goes as long as it leads to implementation.

Since pragmatism is so flexible it is supportive of mixed methods which can include quantitative or qualitative data. While a postpositivist might be happy once the report is completed, a pragmatist is only happy if their research is used by stakeholders.

Constructivist

The constructivist paradigm is focused on how people create knowledge. Therefore, constructivists are focused on the values of people because values shape ideas and the construction of knowledge. As such, constructivists want to use methods that focus on the interaction of people.

With the focus on people, constructivists want to create a story using narrative approaches that are often associated with qualitative methods. It is possible but unusual to use quantitative methods with constructivists because such an approach does help to identify what makes a person tick in the same way as an interview would.

Transformative

The transformative paradigm is focused on social justice. Therefore, adherents to this paradigm want to bring about social change. This approach constantly investigates injustice and oppression. The world and the system need to be radically changed for the benefit of those who are oppressed.

People who support the transformative paradigm are focused on the viewpoints of others and the development of more rights for minority groups. When the transformative paradigm is the view of a program evaluation the evaluators will look for inequity, inequality, and injustice. Generally, with this approach, the outcome is already determined in that there is some sort of oppression and injustice that is happening, and the purpose of the evaluation is to determine where it is so that it can be stamped out.

Conclusion

The paradigm that someone adheres to has a powerful influence on how they would approach program evaluation. The point is not to say that one approach is better than the other. Instead, the point is that being aware of the various positions can help people to better understand those around them.

Evaluation Models Part III: Eisner’s Criticism Model

1 Reply
Advertisements

In the last few posts, we have looked at evaluation models that come from a scientific approach. In this post, we look at evaluation from a humanistic perspective.

Curriculum evaluation models that come from a humanistic approach see evaluation as more qualitative than scientific models. Humanistic evaluators believe that the process of assessing curriculum is too complex and messy to rely so heavily on quantitative methods. There should be fewer numbers and more written descriptions  The purpose is not to answer questions completely but to add to the conversation by exposing additional questions.

One example of an evaluation model that adheres to the principles of humanistic approaches is Eisner’s Criticism Model.

Eisner’s Criticism Model

Eisner’s Criticism Model is used for evaluating a new curriculum and not necessarily to examine a curriculum that is established. This model has four steps as follows

  1. Describe
  2. Interpret
  3. Evaluate
  4. Identify themes

The steps are mainly self-explanatory. First, evaluators describe the setting and curriculum of the study. Second,  the evaluators explain the reasons for the new curriculum by explaining the need. Third, The evaluators present their understanding of the value of the new curriculum. Fourth, various themes are identified within the curriculum that are meaningful.

This four-step process involves qualitative activities for data collection. Evaluators might participate in classes, observe classes or other activities, analyze student work,  use video, photos, interviews, of teacher and students in action. The goal is to notate what is happening but also what may not be happening in the data. This involves written narrative much more the numerical summaries.

In order to complete such an evaluation, an individual needs to have an expert knowledge in education. Eisner refers to such individuals as connoisseurs. These are people who know what to look for as well as how to value and appreciate what is happening.

Conclusion

Although not nearly as common as scientific models such as Stufflebeam or Stake humanistic models such as Eisner’s Criticism model has its place in education.  For smaller schools or smaller programs, this model is an excellent way to go beyond just numbers and to describe the effectiveness of a curriculum with words. However, for larger schools or programs it is challenging to use such an approach when there is so much happening that needs to be summarized succinctly.

Evaluation Models Part II: Stufflebeam’s Model

3 Replies
Advertisements

In the last post, we began a discussion on evaluation models of the scientific variety. In this post, we conclude are look at scientific evaluation models by examining Stufflebeam’s CIPP model. Daniel Stufflebeam is a famous educator and evaluator of curriculum.

Stufflebeam’s model is called the CIPP model which stands for

  • Context
  • Input
  • Process
  • Product

Context

Context is about studying the context in which the curriculum is used. The purpose is to assess what is happening and examine why needs of the stakeholders may not be met. Common activities at this stage include the following.

  • Physical environment
  • Philosophical foundations of the curriculum
  • Determine stakeholders
  • Background of the context

Of course, there are other activities that could take place but these just provide some examples

Input

Input is about determining whether there are adequate resources to conduct an evaluation. Evaluators look at the goals and strategies of the evaluation. If there are concerns, other approaches may be suggested in order to evaluate the curriculum.

In many ways, this is when the methodology of the evaluation is checked rigorously. Common questions assessed at this level include the following

  • Do goals and objectives align?
  • Are teaching strategies suitable?

Process

Whereas context and process take place before implementation,  process evaluation takes place during implementation. At this stage, evaluators examine if the plan is actually happening in the classroom. Evaluators look for defects in the implementation or use of the curriculum. Whatever problems are identified, strategies are developed to address them.

Normally, process happens at a pilot stage before a larger implementation. For example, the algebra teachers might experiment with a new teaching approach before using it in the entire math department at a school. In many ways, process evaluation is formative evaluation.

Product

The product stage involves collecting data to make a decision about the curriculum. Information is gathered to see how well the curriculum is meeting objectives. From this decision are made. The product stage is really a type of summative evaluation.

Conclusion

Stufflebeam’s model involves analyzing the context and the appropriateness of the curriculum to this context. From there, a formative and summative evaluation is used to assess the impact of the curriculum among the stakeholders. This practical model is useful for educators who are seeking ways to examine the various programs under their care.

Evaluation Models Part I: Stake’s Congruence-Contingency Model

15 Replies
Advertisements

Evaluation models are used in curriculum as a process for assessing the appropriateness of a curriculum for a context. As with approaches to curriculum evaluation, evaluation models can be divided into scientific and humanistic models. For the next few post, we will look at scientific models of curriculum evaluation. Our first example is Robert Stake’s Congruence-Contingency Model.

Congruence-Contingency Model

Stake’s model of curriculum evaluation is more than just an evaluation process. Stake’s model also looks at the development of the curriculum. When using this model, it is necessary to compare the developed curriculum with what actually happens in the classroom.

There are six key terms, broken down into two groups of three, that we need to know in order to understand Stake’s model and they are as follows.

Development Stage

  • Potential prerequisites
  • Potential Curriculum
  • Potential results

Evaluation Stage

  • Prerequisites applied in context
  • Evaluation of operational curriculum
  • Actual results

Prerequisites

The prerequisites is another way of saying “before” or the state of the context before the intervention of teaching. This includes student’s attitude, motivation, prior academic performance, teacher characteristics, and more. In the development stage, the teachers need to identify what are some potential prerequisites that may impact learning. In the evaluation stage, the evaluators determine what prerequisites actually impact the curriculum. In other words, there is a comparison of what was anticipated and what actually was the case in terms of the prerequisites.

Potential & Operational Curriculum

Potential curriculum is the “dream” curriculum that is developed. It includes everything that the teachers want to do. The Operational curriculum is what was actually used. There is normally a discrepancy between the two as it is difficult to cover all of the material and use all of the activities. The evaluation will examine the difference between these two aspects of curriculum as another criterion for assessing the quality of the curriculum.

Potential vs. Actual Results

Potential results are what the teachers hope to see as a result of the use of the curriculum. Actual results are the real performance of the students. The difference between the potential or desired results and actual results is another indicator of the quality of the curriculum in Stake’s model.

Conclusion

Stake’s Model provides evaluators with an opportunity to compare the desired outcome with the actual outcome. The benefit of this is that it is the curriculum developers that set the criteria of evaluation. All the evaluators do is determine if the curriculum performed in a manner that is consistent with the ideas of the developers.

Approaches to Curriculum Evaluation Part II: Intrinsic vs Payoff Approach

Leave a reply
Advertisements

In the last post, we looked at scientific vs humanistic approach to curriculum evaluation. In this post, we continue the discussion by examining the intrinsic vs payoff approach to curriculum evaluation

Intrinsic Approach

The intrinsic approach is used to assess the overall quality of a curriculum. This involves looking at the various components of curriculum design such as the scope, sequence, articulation, balance and other aspects. Aspects of curriculum development are also assessed which means examining the teaching methods, content, and learning experiences. However, initially, at least, the criteria of evaluation is not determined but emerges after the process begins.

The intrinsic approach not only examines the value of the curriculum but also how well a given curriculum reaches its goals and objectives. This involves collecting some form of data whether quantitative or qualitative. As such, most evaluators normally approach evaluation with some of the characteristics of the intrinsic approach.

Payoff Approach

In the payoff approach, clear evaluation criteria are set from the outset. Normally, evaluators look at the impact of the curriculum on its stakeholders, which often includes, students, teachers, parents, and administrators. This approach to evaluation is among the most popular in education because of the clear criteria which makes data collection smooth and efficient.

Which Approach to Use?

The intrinsic approach may be most useful when it is unclear exactly what the stakeholders want to know. In other words, it is useful for exploratory purposes. Nobody is sure where they are going and the intrinsic approach helps to setup a map of strengths and weaknesses within a program. From there, other approaches can be used to refine the evaluation if necessary.

The payoff approach is best when the evaluation team knows exactly what it wants to know. Clear evaluation questions/criteria has been set and it is only a matter of answering the questions or assessing the level at which the curriculum meets the criteria by collecting data.

As such, the purpose is not to declare one approach superior to the other but to keep in mind the context when deciding which tool to use.

Approaches to Curriculum Evaluation Part I: Scientific vs Humanistic Approach

5 Replies
Advertisements

Curriculum evaluation is the process of collecting data in order to make decisions about the curriculum in question. Curriculum can mean a host of things. It could refer to a particular subject such as 7th grade reading, it could refer to a particular grade such as 8th grade in general, it can also refer to an entire school such as elementary or secondary school. As such, one aspect of curriculum evaluation to consider is the scope or what is being evaluated.

There are meaning different approaches or ways of seeing curriculum evaluation. For whatever reason, approaches to curriculum evaluation are always explained in extremes. In this post we will look at the following approach to curriculum.

  • Scientific vs Humanistic Approach to Curriculum Evaluation

Scientific Approach

The scientific approach is probably the oldest approach to curriculum evaluation as it dates from modernism and the emphasis on the scientific method of the 19th to 20th century. This approach to curriculum evaluation focuses on using quantitative data generate by the learners. This allows for statistical analysis. Furthermore, the results are compared in order to determine the level of success. This comparison is at the heart of decision-making when this approach is employed.

There are natural issues with such a heavy emphasis on numerical data. For one, the students narrative is missing. A lickert scale analysis is not as rich in content as an interview. Another issue is the assumption of similar circumstances. The diversity in student ability and even in teaching ability makes it difficult to assume that students are facing similar challenges and circumstances.

Humanistic Approach

In a more post-modern worldview the Humanistic approach looks at the individual rather than the numbers. Data is much more qualitative in nature. The rationale behind this is that life has multiple perspectives to it and quantitative data only provides one perspective.

Humanistic evaluators want to understand the complexities of the environment they are assessing. This involves capturing narratives through interviews and focus groups. Observation is used not to count frequencies but to take notes of what is happening in the classroom.

The major issues with this approach is the smaller sample size that is required. It is not feasible to interview 400 students but perhaps 20 is doable. In contrast, conducting a survey with 400 students should not be a challenge for a scientific evaluator. Furthermore, there are questions as to the objectivity of the results.

Since qualitative data is processed by the researcher their own perspective can filter what they report when they share the perspective of the respondents, In contrast, scientific approaches are more objective in that computer processes and reports the results.

Conclusion

Instead of having a bias towards scientific or humanistic approaches to curriculum evaluation. It is better to look at the context of what needs to be evaluated and determine the most appropriate approach. It should be the context and not the preference of the evaluators that should decide which direction to take. In many situations, a mixture of both approaches may be appropriate but this involves much more work and complexity.

The Purpose of Curriculum Evaluation

1 Reply
Advertisements

Curriculum evaluation is a critical aspect of the educational process. Most schools have to evaluate their curriculum at one point or another. This post will share insights into defining evaluation as well as the distinction between measurement and evaluation.

Defining Evaluation

Evaluation of a curriculum happens in order to decide whether to accept, change, or eliminate various aspects of a curriculum. The overall goal is to understand if the curriculum is producing the desired results. This implies that the evaluators know what to expect prior to the evaluation and are looking for these predetermined results.

Evaluation is about gathering data. This data can be collected in many different ways. The various data collection approaches are the same as any used in research. They include observation, interviews, surveys, and more. The data is often aggregated and used to determine if the goals of a program are being met.

Measurement and Evaluation

Evaluation is not only about measuring a phenomenon. Instead, evaluation assigns value and meaning to the results of a study. It assesses quality through quantitative or qualitative means. This is in contrast to measurement which describes a phenomenon but does not interpret the quality of it.

It is common for there to be confusion over measurement and evaluation. To remember the difference, measurement describes something numerical. Evaluation, on the other hand, judges something qualitatively.

Conclusion 

It is important to remember that evaluation can happen at many different levels. The teacher can assess their unit plans. A department can assess their reading program. A school can assist its entire curriculum. Regardless of the level. Curriculum evaluation is often focused on determining how the curriculum is doing in terms of achieving the goals set for it.