Category Archives: Uncategorized

WHERE, LIKE and IN Commands for SQL

This post will explore the use and application of the WHERE, LIKE, and IN commands in SQL.

WHERE

The WHERE command is generally used as a way to filter data in SQL. The database we are using in this post contains data on basketball players from 1950-2017. What we want to do is filter the data so that we only see data from players who played in 2017. In the example below, we will filter our basketball players by year be set to 2017

SELECT *
FROM Seasons_Stats
WHERE year = 2017

If you look at the year column you can clearly see that the value is set to 2017 just as we wanted it.

The WHERE command is not limited to numbers as it can also be used with text. In the example below, we filter our data with the WHERE command so that we only see players who played for the Golden State Warriors (GSW).

SELECT *
FROM Seasons_Stats
WHERE Tm = 'GSW'

If you look closely at the “Tm” column you will only see the initials for the Golden State Warriors.

LIKE

The WHERE command is often teamed with the LIKE command when you are looking for a text but are not sure of an exact match. You can specify a pattern you are looking for similar to regular expressions. In the code below we use the LIKE command with the WHERE command searching for any player whose name begins with L.

SELECT *
FROM Seasons_Stats
WHERE Player LIKE 'L%'

The percentage sign (%) after the letter L tells SQL that anything can be after the letter L in the search and meet the criteria. We can also put the % at the end of or text or put one on both sides are some other combination as shown in the examples below.

In the example below, the last letter must be an L. After this example, is one in which an L must appear anywhere in the name.

SELECT *
FROM Seasons_Stats
WHERE Player LIKE '%L'

SELECT *
FROM Seasons_Stats
WHERE Player LIKE '%L%'

The IN command allows you to filter your data based on several values. This command is also combined with the WHERE command. In the example below, we filter our data so that we only see players whose position is small forward or shooting guard.

SELECT *
FROM Seasons_Stats
WHERE Pos in  ('SG', 'SF')

You could also insert numerical values in the parentheses when using the IN command.

Conclusion

The command shared in this post provide more information on basic tools you can use and apply in SQL. Filtering using the WHERE, LIKE, and IN commands get helpo you to focus the breath of your research to find answers to your questions.

Kotter’s Change Model and Schools

1 Reply

This post will look at Kotter’s Change model, another model of change for an organization. In all, there are eight steps for this process, as shared below. There will also be several brief examples of how one educational leader has perhaps unknowingly used this model, at least in part.

Establish Urgency

Step one of bringing change, according to Kotter, is to develop a sense of urgency. Since most people are emotional by nature, they may need an emotional push to accept and work for change. Urgency can be developed through creating a narrative about why change is necessary and sharing some of the prophesied consequences if change does not happen.

I once worked with an administrator, and we will call him Jim, who was a complete master of establishing urgency. Whenever he wanted something done, Jim was sure to mention how the entire school was endanger if what he wanted was not done. However, because everything he wanted was a “do or die” scenario, people started to ignore him. What happened to him is similar to “the boy who cried wolf” or chicken little and the sky is falling.” Urgency can be a great tool, but it must be used sparingly; otherwise, it will lose its power to mobilize.

Form Coalition

Once urgency is established, a leader needs to build a team of influential people within the organization to shape the change. The people involved in this colation should be influencers who are affected by the change. These people serve as local go-to contacts to influence the masses within the institution to support and make change happen.

Jim was also an expert at building coalitions. When he wanted to change, he knew he needed help and had the political acumen to build complex alliances. One mistake I think he made was that sometimes he would team with people who were established and influential but maybe not popular. When unpopular people are pushing change, it is often rejected because people often value relationships over performance.

Make a Vision and Communicate it

Once the team is in place, they work together with the leader to develop the scope and rationale for change. By scope, it is meant the breadth and depth of the change and rationale are the motives behind the change. Creating this vision helps the team determine what they are focusing on for a change and how they will know when they are successful.

Once the vision is set, it needs to be communicated with the institution. People need to know where they are going and how they will get there. Communication can help people to buy in and accept the change.

Jim was always good and sharing the vision even if nobody else contributed to it. Once his colation was aware of the plan, it was shared with the institution. One thing to be careful of is how much of the vision to share. If your plans are overly ambitious, people may be intimidated by what you want to do. Tell people enough to get things started and slowly reveal more details as small goals are achieved.

Remove Obstacles and Strive for Small Wins

Removing obstacles is about problem-solving. Whenever people try to accomplish anything, some surprises try to disrupt the process. The leader must solve these problems by providing training, resources, encouragement, supplies, etc., so that the vision can be achieved.

Small wins relate to sharing the vision. Many people struggle with the big picture as they are more detail-oriented. If you tell some people all the work they have to do, they will become discouraged. Breaking the large vision into small wins or goals is critical for managing people psychologically.

Small wins are created when the leader develops milestones that help to achieve the vision. These milestones are shorter and less complex in nature compared to whatever the final vision is. Therefore, they are easier to attain. When people begin to have success completing small goals for wins, it helps motivate many individuals.

Consolidate Improvements and Anchor Changes

Consolidation involves reinforcing what has worked well so far and removing what has not worked well. As success is experienced and momentum develops, people begin to get excited about the changes they have been a part of making happen. In other words, focus on sharing the success to help push people to finish the changes.

The final step is anchoring changes. Anchoring changes involves making what changes were made permanent. Doing this requires discipline to support change long-term. It is common for people to get so excited about the change that they do not make an effort to maintain the new normal. The same energy that was brought to bringing change must be used to maintain it.

Conclusion

Change is part of the journey of any institution. Having a process to guide the change process can help leaders who need to push for change. Kotter’s model for change is one tool for walking through change and making it a reality.

Types of Change and Schools

Leave a reply

Change is a part of life, and one thing most people have in common is a dislike of change. This post will look at change and its relationship with the organization of schools.

Types of Change in an Organization

There are at least three ways that an organization, such as a school, can change. These three ways are structural, technological, and cultural.

Structural change relates to redesigning how the school is organized. For example, a school might add or remove departments, change job responsibilities, and or create new positions within the institution.

Technological change refers to having to make adjustments to the use of various electronics. It is common for there to be resistance to changing technology because people generally do not want to waste time learning new things. Technology can also, at times, lead to downsizing, which is something people do not like as well.

The final form of change is cultural change. This form of change has to deal with how people think about the organization. In other words, cultural change causes a shift in the beliefs and assumptions about the company and how things are done. Each school has its unique way of seeing the world and teaching and helping students—cultural change involves modifying these views.

Points to Ponder

The scope of change can affect people’s willingness to accept it. For example, suppose a school hires an additional teacher because of the overload of the current teachers. In that case, there will probably be little resistance to this form of change because the current system was so intolerable. However, if the change calls removing teachers, it is safe to assume strong resistance.

This same line of thought applies to the other forms of change, technological and cultural. Minor changes will be tolerated, and significant changes will be tolerated if they relieve a significant problem. However, if the changes are unpalatable due to their size or inability to solve a problem, resistance is more likely.

It is also important to realize that all of these types of change can happen simultaneously in a school. For example, a technological change such as incorporating e-learning could lead to a need to change things in terms of the organization. For example, it may be necessary to restructure the IT department by splitting responsibilities and hiring additional people. In addition, cultural changes may also be affected by e-learning adoption through the need for the organization to be more receptive to the rapid changes of the IT world.

The point being made here is to remember that change cannot happen in a vacuum. Unfortunately, when change comes, it will affect things that the leadership did not want to be changed. This has led in part to disdain by many leaders of change. It is not so much the change that is the problem but the unforeseen consequences of the change that bothers many educational leaders.

Conclusion

Change will always be a threat to a school. However, when it is time to make a change, leaders need to know how change can impact an organization.

Organizational Culture and Schools

Leave a reply

The culture of an organization is one of the main factors in motivating the actions and attitudes of employees. The culture of an organization is what brings people together for a common purpose. As such, since these ideas on culture come from business, this may be something that administrators and teachers need to be aware of as they set up the institutional culture or classroom culture.

Therefore, this post will look at several common types of organizational cultures and their relationship or similarity to what happens in a school context. The ideas discussed below come from the Competing Values Framework and include four main quadrants in which cultures can be found, and these are.

Clan
Adhocracy
Hierarchy
Market

Clan

An organization that has a clan-style culture is perhaps the one most similar to most schools. A clan organization emphasizes relationships, mentoring, development, and other personal growth characteristics. Most teachers want to see their students develop into responsible young adults and take satisfaction from this. The same can be said of many administrators regarding seeing their teachers and their students grow and develop healthy relationships.

Adhocracy

An adhocracy culture is one in which there is an emphasis on innovation, experimentation, and risk-taking. This style of culture may not be the most common in schools. Schools often tend to focus on preserving the social structure rather than pushing the edges of the envelope. However, this is not to say that no innovation and experimentation is happening in schools. The real point relative to the industry and companies like Google and Facebook is that schools are not highly innovative.

Hierarchy

Efficiency is the name of the game for hierarchy culture. In this culture, there is a focus on precision, expertise, cautiousness, and conservatism. A hierarchical culture has found a system that works and does not want to disturb said system. Like the clan culture, the hierarchy culture is widespread in the educational setting.

Market

Last but not least is the market culture. This culture focuses on delivering value, fast decision-making, and a general sense of getting things done. Educational institutions are not generally known for their speed and decision-making. However, this may be because of the focus on relationships and a preference for a clan-like culture.

Conclusion

The main benefit of this information is reflection. Every teacher and leader needs to ask themselves what kind of culture do I want to develop. Having insights into what types of cultures are common can help any leader develop their unique approach. The culture of a school can be firmly in one style or the other or be a mixture of various techniques to facilitate success.

Interactions & Power in the Classroom

Leave a reply

Power is the ability to influence others. Several things affect power in the classroom. In organizational behavior, these factors are called power dependencies. Some common power dependencies in the classroom include student values, the relationship between the teacher and student, and counterpower.

Power Dependencies

Student values can play a significant role in whether or not the teacher’s power influences a student. In other words, if the student cares about what the teacher wants them to do, it is more likely they will be affected by the teacher. For example, it is common for students to love PE. If the teacher wants the students to complete specific assignments to have PE, students will comply because they care about PE. However, the converse is true that if students do not care or value PE, they may not comply.

The second dependency of power is the relationship between the student and teacher. If the two parties hate each other, there will be little hope of compliance except through coercion. A bitter truth of teaching is that sometimes a teacher can foster good relationships with students, and sometimes they cannot. It is essential to realize that there could be student resistance to the teacher’s power if there is tension between a student and teacher.

Counterpower is essentially the power the student has to influence the teacher. If a student possesses a high amount of power, it is possible to expect a high resistance level. For example, there is a common stereotype of the student-athlete not complying with completing academic assignments. The athlete can do this because they possess some counterpower due to their athletic status. This stops the teacher from holding the athlete accountable for not completing assignments.

Use of Power

There are also several ways that a teacher can use power. A teacher can control the flow of information to students. This is common when the teacher is still making decisions about something or withhold information to elicit a desired behavior. For example, a teacher may not share the details of the amount of PE time students will get if they complete assignments. This may be because the teacher is unsure how much they can give at that moment and need to work it out. Information can also be shared to encourage behavior, such as a teacher being honest about why the students can not play outside due to unforeseen circumstances.

Teachers can also control access to people. For example, it is common for teachers to separate from students who might be talking too much when together. This is a classical power move to encourage compliance with on-task behavior. A teacher can also award good behavior by allowing students to work together.

Another everyday use of power in the classroom is controlling the choices that are available to students. A teacher may want to give the students a specific range of options for various activities. By shaping the choices, the teacher exercises power while also allowing the students a say in what happens in the classroom.

A final exercise of power is the students’ perception of the cooperation between the teacher and the administration. If students know that they will not get in trouble if they are ever sent out of class to the higher administration, this can seriously hamper the power of a teacher. Therefore, the teacher and administration must show that they have a strong alliance and work together to address students’ misbehavior.

Conclusion

Power is a an important aspect of the teaching experience. Teacher need to be consciously aware of how different factors can affect their power. Without this knowledge a teacher can struggle with determining the best way to handle a particular situation in their classroom.

Leadership Substiutes and Neutralizers in the Classroom

Leave a reply

Leading in the classroom is a serious challenge for even experienced teachers. However, teachers can take actions to enhance their leadership in ways that do not require more work. This post will look at leadership substitutes and leadership neutralizers and how these ideas help and hurt a teacher in the classroom.

Substitutes

Substitutes for leadership are things that are in place in the classroom that do not require leadership from the teacher. In other words, substitutes replace the teacher so that certain things run by themselves. The more substitutes a teacher can put in place, the less active management they have to do because the students already know what to thanks to the substitutes that are put in place by the teacher.

One example of leadership substitutes would be to have routine or procedures in the classroom. When students know what to do in various situations based on the training they received in the past, it is unnecessary for the teacher to actively control these situations, such as procedures for coming into the classroom or going to lunch.

Developing student leaders is another way to create substitutes for the teacher’s leadership. How this is done varies from school to school and from teacher to teacher. However, the point is that if students can assist in the supervision of the students, it can serve as another form of substitution of the teacher’s leadership.

There is a term in education call withitness, which means knowing almost subconsciously how to respond to a problem in the classroom or having “eyes in the back of one’s head.” If students understand that a teacher is “withit,” it can serve as another form of substitution of leadership in the classroom because students know they cannot get away with whatever they are thinking of doing.

There are also more intangible ways in which leadership substitutes can be established. If a teacher has a strong reputation for expertise and leadership among the students and the school, this reputation alone can serve as a substitute for leadership. The students know that this teacher is good and will sometimes modify their behavior because of the teacher’s leadership ability.

Neutralizers

Neutralizers are the opposite of substitutes in that these are things that block leadership and lead the teacher to spend time trying to manage instead of leading. An example of a neutralizer would be the absence of any of the ideas presented in the substitute section of this post. When these ideas discussed above are missing from a classroom, a teacher cannot get many other things done because the focus of their work is on managing behavior.

Another neutralizer is a poor or a lack of communication. This is related to the previous paragraph. If students do not know what the teacher wants them to do, they will find something to do themselves. Again this takes away from the learning experience and leads to chaos in the classroom.

Some neutralizers are outside the teacher’s control. One example would be family problems in the homes of students. In this day and age of broken families, students often have unstable home situations that often bleed over into the classroom. There is little a teacher can do about the home setting, and if home problems impact student behavior, it will also neutralize leadership.

Conclusion

When there are a lot of neutralizers, this means that there will be little leadership. The teacher is not able to set aside management challenges and has to focus on controlling students. People generally do not like to be controlled but would instead manage themselves. If there is no system in place to allow this, the teacher has to be the one to control students. Rather the goal should be for the students to follow the example of the classroom through the expectation of the teacher and the standard of peers, which serve as substitutes to overt control of behavior.

Identifying Leadership in the Classroom

Leave a reply

It is always hard to predict who will make a great leader. Some students do not seem to show any potential for this but eventually become highly influential. Other students who show great promise never seem to reach the level that many anticipate. Despite this, there has been a great deal of research that tries to predicate who will become a great leader and who will not.

One overarching theory of leadership is called the “Great Man Theory of Leadership.” This view holds that some people are born with the traits of leadership. Essentially, this view holds that nature and not nurture are the primary factors in leadership development. Within this paradigm, scholars have wanted to know what these traits were, and we will look at some of them right now.

Leadership Research

One researcher in this field was Stogdill. This research found that leaders often exhibit such traits as a strong drive, problem-solving skills, persistence, initiative, self-confidence, tolerance of interpersonal stress and general frustration, a sense of personal responsibility, and are influential in others’ behavior.

The real question is whether these skills are skills students are born with or can be developed. This is a difficult question to answer. The teacher’s job is to put students in a situation in which these traits can be developed. Some students may grow in such situations, while others may not. In other words, it’s more important that students are allowed to develop leadership skills rather than that they become leaders. Everybody is not interested in influencing others, whether formally or informally.

Another researcher named Locke found results similar to Stogdill. Locke found that leaders are often driven, motivated to lead, display honesty, self-confidence, demonstrate expertise and cognitive ability. Lesser skills that leaders show are charisma and creativity.

What is essential for students regarding Locke’s research is that there are different ways to lead. Some students may be traditional leaders who are often people who always stand at the front and are at the center of the action. However, another way to lead is through expertise. For these types of leaders, the maybe in charge during certain situations are serve as advisors for the main leader. This is a way for people who don’t want the constant stress of leadership to have their moment in influencing the team. If students are not aware of this, they may believe that they are not cut out for leadership, which is rarely the case. Some people lead all the time, but everyone should lead some of the time.

Other traits that leaders often possess are high energy and enthusiasm. Energy is contagious, and enthusiasm helps people to keep pushing through discouragement because of the emotional boost. This implies that the cheerleader type personal can be advantageous. However, a leader cannot only be enthusiastic as they must show that they can work and have skills to offer the team besides encouragement.

There is also this idea of self-monitors. These are people who observe verbal and nonverbal cues and adjust their behavior to influence others. People who are highly sensitive to monitoring themselves are often better leaders because they are worried about influence. People who don’t care usually lack the popularity and social capital to be in leadership positions. Students tend to be highly sensitive to what others think, but only those who are the best at monitoring their actions will achieve the leadership positions in many situations.

What leaders do

So far, the focus has been on what leaders are. Now we will look at what leaders do. Leaders often show a willingness to trust others, which is difficult to do these days. Leaders also have a vision of what they want and either know how to make it happen or find someone else who can. Leaders also show a willingness to take risks and encourage others to do so. Failure is where learning begins, and this is something that many people do not like.

Leaders help teams focus on tasks and even encourage dissent or disagreement because challenging ideas help determine what works and doesn’t work. Lastly, many leaders can stay calm in the face of adversity, at least outwardly. This strengthens the team that may be experiencing strong emotions during a problem or crisis.

As teachers, we must show these actions in our classroom. Showing students that we know what we want and how to get there and that we want students to take risks in their learning is essential. Furthermore, teachers need to encourage discussion and dissent to develop critical thinking skills.

Conclusion

Perhaps the best way to develop leaders is for students to see excellent leadership. The real problem may be that it is so hard to see examples of leadership. If students can witness leadership rather than hear theories about it, this may lead to more leaders who can make a difference. The primary purpose is to provide students with the tools they need for success. However, it is always the students’ decision if they want to develop and use these tools to benefit themselves and others.

Roles of the Teacher

Leave a reply

All teachers are called to a variety of responsibilities in their position. This post will look at the significant roles teachers play in their position as instructional leaders in and outside the classroom.

Interpersonal Role

The interpersonal roles of a teacher can be broken down into two main categories, and these are interpersonal roles within the classroom and outside the classroom. The primary interpersonal relationships a teacher has within the classroom involves their role with students. The teacher must find ways to balance being the classroom’s authority and disciplinary leader while also maintaining warm relations. This is generally difficult for even the most experienced teacher to do.

A teacher also has interpersonal relationships with people outside the classroom. This can include dealing with parents, school leadership, staff, the local community, and other teachers. Each of these unique relationships has slightly different rules for engagement and success regarding communication and interaction.

The dangers and pitfalls of dealing with any of these people are numerous, and a teacher much show caution. For example, how a teacher would communicate with a teacher is different from how they would speak with leadership or a parent. The context is influenced by the role of the person the teacher is talking to.

Informational Role

Teachers also have a role in conveying and obtaining information. A teacher can share and receive information in such context as the classroom, meetings, over the phone, through email, etc. Information can be formal or informal, or it can be announced or gossip. All these various forms of communication are challenges through which a teacher shares and receives information.

AS a conduit of information, teachers often serve as liaisons to several parties to transfer information between groups. For example, the leadership might have the teachers share something with students or parents. A community member may want the teacher to share something with the administration. The point is that information flows from and through the teacher to people in their immediate social network.

Decisional Role

One of the primary roles of a teacher is making decisions. Decision-making may be a primary role of the teacher. Teachers have to decide about policies, assignments, how and what to teach, classroom management, resource allocation, etc. Making these decisions involves communication and interacting with others.

Teachers must also make decisions about negotiating matters. This can involve gathering information and working with others to develop an agreeable plan for both sides. Decision-making is critical because a wrong decision can cause a lot of problems for a teacher and students. However, sound decisions usually are not noticed as it seems to be human nature to see negative situations over positive ones.

Conclusion

Versatility is a critical skill that a teacher needs to develop in order to help the people they come into contact with. Awareness of the roles a teacher plays can help anyone who finds themself in a position where teaching plays an important role.

Goal Theory and the Classroom

Leave a reply

Goal theory is almost a self-explanatory term. Essentially, goal theory states that people are motivated when they have goals. This seems obvious, yet many people do not have goals and thus often lack motivation. As such, goal theory can be useful for people who lack motivation or who perhaps need help in clarifying the goals they have but cannot achieve.

Principles of Goal Theory

There are several principles related to goal theory. First, as has already been stated, is that people perform better when they have goals. Second, and this one needs explanation, the goals must be personal goals that the person wants to achieve. It is hard to be motivated by someone else’s goals. Goals must come from the individual. Many students struggle in school because all the goals come from the curriculum or teacher and rarely from the student. When all actions are coming from the top down, and it could lead to a loss of motivation.

A third principle similar and related to the second is that people have to commit to the goal(s). In other words, it cannot only be in the person’s head but must be followed with action. Procrastination is a sign of a lack of commitment. Such behavior is seen by everyone when they make a goal, maybe a reasonable goal, but never actually do anything to make it happen.

The fourth principle of goal theory is that challenging goals encourage better performance than easy goals. A struggling helps people to perform better, whether adult or child. In the classroom, goals need to elicit moderate to hard struggle because this motivates a student to push themselves. Of course, it is possible to have too much of a good thing, and this means that goals should be challenging but attainable; otherwise, people will give up and be even less motivated.

Fifth, goals need to specific rather than broad. This is a good point. However, different people have different views on broad and specific. Determining whether a goal is broad or specific can be done by assessing a person’s ability to achieve the goal if it is not apparent to the person what they need to succeed. This means that the goal may require refinement in the form of breaking a goal into several subgoals, defining what it means to complete the goal, or setting boundaries such as a timeframe in which the goal is pursued.

Consequences

The consequences of setting goals are not necessarily negative. When adults or students achieve goals, there is a sense of satisfaction in achieving them. Achieving goals brings a sense of autonomy and even self-actualization as a person sees that they can do something and have an impact, no matter how small, on their environment.

There can also be rewards when involving goals. Students can be given various privileges fr achieving goals. This is a more extrinsic matter, but providing external rewards can be beneficial for students on occasion.

There are also problems with goal setting. When goals are set in one area, other areas may be ignored. For example, a student set a goal of doing their math homework at the exclusion of other homework. To achieve this one goal meant to create problems in another area.

Another problem is when goal setting is abuse. An example of this is when a child sets goals that are easy to achieve their real goal of being lazy. It takes experience on the part of a teacher to know when the students’ goals are reasonable and not too hard or too easy.

Conclusion

Children need goals. It breaks the learning experience of school into small measurable steps that they can achieve little by little. These goals must be negotiated at least partially so that students have ownership in the process. When this is done, cooperation may be achieved.

Types of Experiments

Leave a reply

This post will provide some basic ideas for developing experiments. The process of doing valid experiments is rather challenging as one misstep can make your results invalid. Therefore, care is needed when attempting to set up an experiment

Definition

An experiment is a process in which changes are made to input variables to see how they affect the output variable(s). The inputs are called controllable variables, while the outputs are called response variables. Other variables that cannot be controlled are called uncontrollable variables.

When developing an experiment, the experimenter’s approach or plan for experimenting is called the strategy of experimentation. Extensive planning is necessary to conduct an experiment, while the actual data collection is often not that difficult.

Best Guess Approach

There are several different strategies for experimentation. The best-guess approach involves manipulating input variables based on prior results from the output variable. For example, if you are teaching a math class and notice that students score better when they work in groups in the morning compared to working in the afternoon. You may switch to group work in the morning and see if lectures may further increase performance.

This guesswork can be highly efficient if you are familiar with the domain in which you are doing the experiments. However, if the guess is wrong, you have to continue guessing, and this can go on for a long time.

One-Factor-At-A-Time

Another strategy of experimentation is the one-factor-at-a-time (OFAT) approach. You begin by having a baseline for each factor (variable) and then vary each variable to see how it affects the output. For example, you can switch whether students study in the morning or even and see how it affects performance. Then you might test whether group work and individual work affect scores.

The biggest weakness with this is that you can see interactions between variables. Interactions are an instance in which one factor does not produce the same results at a different level of another factor. Interactions can be hard to understand, but sometimes when two factors are mapped at the same time with the response variable, the lines cross to indicate that there is an interaction.

Factorial Experiments

Factorial experiments involve varying factors together. For example, a 2^2 factorial design means four combinations of experiments with two variables are varied, and one response variable with four possible combinations of experiments. Often these types of experiments are drawn as a square, as shown below.

Each point represents a different combination of the two factors. The calculation of this involves subtracting the means of the variable or factor on the x-axis. If we run each combination twice, we would calculate the difference, as shown below.

The more significant this difference, the more likely there is a strong effect based on the independent variables in the model.

When the number of combinations becomes large and complicated to manage, it may not be practical to run all possible combinations. In this situation, an experimenter will use a fractional factorial experiment in which only some of the combinations are used. For example, if 32 experiments are possible (2^5), maybe only 12 of them are conducted. The calculation is the same as above, just with more groups to compare.

Conclusion

Experiments are a practical way to determine the best combination of factors or variables for a given output variable(s). The majority of the time is spent planning and designing the experiment, with the actual data collection being straightforward.

Probability & the Union of Two Events VIDEO

Leave a reply

Probability & the Union of Two Events

E-Learning & Support Staff

Leave a reply

Many think e-learning is only about teaching. In reality, there is an entire army of people working behind the scenes so the teacher and student can shine. In this post, we will look at support services and how they can be made ready to support e-learning

Defining Support

The support staff is all of the people involved in the online learning experience whose primary function does not involve teaching or leadership. Examples of support staff can include information technology, academic services, finance, student services, and marketing among other possible individuals.

A common problem with support staff is that like many teachers they have never studied or taught online. This means that they may not be familiar or comfortable with supporting students virtually such as in the context of an online experience. In addition, even though they do not teach support staff needs to be familiar with the LMS and other online tools so they can at least communicate basic information to students when it is necessary.

To put things as simply as possible, even the secretaries should be familiar with e-learning ideas an concepts not for the sake of teaching but for the sake of being able to understand students and even teachers’ questions and concerns as they provide support for them. This can only be done through training and some experience.

Information Technology

The IT team has the advantage of being thoroughly comfortable with technology. Therefore, learning about the technical aspects of e-learning is not much of a challenge for them. The problem with IT often is that they constantly want to add more and more technology to solve various problems with the existing technology that you are using. When this happens the new technology clashes with old technology causing bugs and the learning curve to use the LMS grows. This discourages students and teachers from using the system employed by the institution. Therefore, it is important to limit the expansion of technological tools employed for e-learning for the sake of simplicity.

Other problems facing IT is the selection of the LMS. Generally, you want them to be a part of this process to win their future cooperation. If you pick it without them and they know what you picked was bad they will resist supporting the lousy system with 100% commitment. However, if you pick a bad system together you will get full cooperation because of the IT’s ownership in the bad decision.

When deciding on what LMS to pick the choices are essentially free (moodle) and commercial (Blackboard). Free is not free as you have to provide extensive support to get the LMS working which may require hiring additional people. Commercial cost more but is already fully functional.

The choice of LMS also has to consider supporting the system. This includes standard procedures for updates/archives, the appearance of website, security, and how to address technical support. These are all basic IT concepts but they need to be worked out for the e-learning context.

Academic Services

Academic services will need to determine if the Student Information System is going to integrate with the LMS. This is primarily so that information in one system is simultaneously available in the other. Examples of information that may need to be in both systems include grades, student ID, student names, majors, faculties, etc. Doing this is simple for IT (most of the time) but there are security issues that need to be addressed.

It is also important to have all the standard academic policies adapted for the online experience. For example, addressing issues involving plagiarism and cheating need to be adapted for e-learning. Lastly, the academic office needs to think of how tutoring and remedial support can take place in the online context. For example, proficiency exams for entrance, writing support, and additional forms of tutoring needs to be addressed in the online context.

Library Service, Finance & Student Services

The library needs to establish a strong online presence by acquiring ebooks and electronic databases. The staff of the library must also becoming comfortable helping students online rather than face-to-face. For example, library staff may need to make a video recording of how to use online library services which means the staff has to be familiar with technology.

Finance needs to determine the cost of learning online. Students expect the fees to be lower than on-campus because there is no room and board involved. Therefore, there is intense pressure to keep costs down in e-learning.

In addition, contrary to what we see with MOOCs, online classes should be the size of traditional classes at your institution. Teaching 500 students at once cannot be done with 1 person even with all the technology. Students need the individual attention that comes with moderately small classes. With a large class, the institution is forced to higher TAs that quickly start to eat up the budget. Therefore, finance needs to be sure to treat online classes like regular classes financially.

Student services can involve such things as counseling and social activities. Therefore, it is left to Student Services to develop these tools in an online context. Students require emotional support like anybody else.

Another project is for the student service team to develop ways for online students to socialize and get to know each other. This may be done by breaking students into social cohorts who have to interact synchronously at times.

When thinking of all these services, they can be shared by the online program with the university or they can be separate. For example, the online program can share financial services or have its own financial services. There is no right or wrong but what works best for you.

Conclusion

There is a lot involved in support staff to have a quality online experience. Everyone needs to work together for the sake of the students to learn. If anything is neglected the online experience will be negative for many.

Arithmetic Sequences VIDEO

Leave a reply

Arithmetic Sequences

Paraphrasing Tips for ESL Students VIDEO

2 Replies

Paraphrasing Tips for ESL Students

Different Views of Research

Leave a reply

People have been doing research formally or informally since the beginning of time. We are always trying to figure out how to do this or why something is the way that it is. In this post, we will look at different ways to view and or conduct research. These perspectives are empirical, theoretical, and analytical.

Empirical

Perhaps the most common form or approach to doing research is the empirical approach. This approach involves observing reality and developing hypotheses and theories based on what was observed. This is an inductive approach to doing research because the researcher starts with their observations to make a theory. In other words, you start with examples and abstract them to theories.

An example of this is found in the work of Charles Darwin and evolution. Darwin collected a lot of examples and observations of birds during his travels. Based on what he saw he inferred that animals evolved over time. This was his conclusion based on his interpretation of the data. Later, other researchers tried to further bolster Darwin’s theory by finding mathematical support for his claims.

The order in which empirical research is conducted is as follows…

Identify the phenomenon
Collect data
Abstraction/model development
Hypothesis
Test

You can see that hypotheses and theory are derived from data which is similar to qualitative research. However, steps 4 and 5 are were the equation developing and or statistical tools are used. As such the empirical view of research is valuable when there is a large amount of data available and can include many variables, which is again often common for quantitative methods.

To summarize this, empirical research is focused on what happened, which is one way in which scientific laws are derived.

Theoretical

The theoretical perspective is essentially the same process as empirical but moving in the opposite direction. For theorists, the will start with what they think about the phenomenon and how things should be. This approach starts with a general principle and then the researcher goes and looks for evidence that supports their general principle. Another way of stating this is that the theoretical approach is deductive in nature.

A classic example of this is Einstein’s theory of relativity. Apparently, he deduced this theory through logic and left it to others to determine if the theory was correct. To put it simply, he knew without knowing, if this makes sense. In this approach, the steps are as follows

Theory
Hypotheses
model abstraction
data collection
Phenomenon

You collect data to confirm the hypotheses. Common statistical tools can include simulations or any other method that is suitable in situations in which there is little data available. The caveat is that the data must match the phenomenon to have meaning. For example, if I am trying to understand some sort of phenomenon about women I cannot collect data from as this does not match the phenomenon.

In general, theoretical research is focused on why something happens which is the goal of most theories, explaining why.

Analytical

Analytical research is probably the hardest to explain and understand. Essentially, analytical research is trying to understand how people develop their empirical or theoretical research. How did Darwin make this collection or how did Einstein develop his ideas.

In other words, analytical research is commonly used to judge the research of others. Examples of this can be people who spend a lot of time criticizing the works of others. An analytical approach is looking for the strengths and weaknesses of various research. Therefore, this approach is focused on how research is done and can use tools both from empirical and theoretical research.

Conclusion

The point here was to explain different views om conducting research. The goal was not to state that one is superior to the other. Rather, the goal was to show how different tools can be used in different ways

Medieval Universities Costs

1 Reply

The post will talk about some of the characteristics and costs of university studies during the Medieval time period. Naturally, there are a lot of similarities to modern times. However, many aspects of university life took time to grow and develop as we will see.

University

Universities during the Middle Ages were distinct from what we see today. There were essentially no buildings that made up the university. This means that initially in many situations in Europe there ewer no libraries, no laboratories, no halls, no endowments or money, and even no sports. Today, we often think of universities in terms of there physical presence. In the past, universities were thought of in terms of the students and teachers who learned and taught regardless of the physical location.

A university was defined as the totality of students and teachers in a particular location. Both the teachers and the students organized themselves into groups for bargaining power. The university of students would work together to control rent, book price, and tuition. If local businesses tried to abuse them the students would threaten to leave. The students also placed expectations on the teachers such as no absences without permission, no leaving the city without leaving a deposit (this prevented crooks from taking tuition and running), maintain a regular schedule.

College

Professors formed their own guild called the college and set expectations for people who wanted to become professors. In addition to colleges, teachers would form themselves into faculty, which is several teaches from the same discipline. Faculties were allowed to confer degrees and promote students to the academic rank of masters. In addition, it was common for teachers to be celibate

The term “college” was also used to refer to the hospice or residence hall where students live. This is similar to the modern-day dormitories. Originally, colleges were for religious students and not for secular. To this day, institutions of higher learning are referred to as colleges and or universities. The success of universities put the cathedral, monastic and provincial schools a=out of business.

Textbooks

Textbooks were hard to find during the Middle Ages. This was before the printing press which means that books were copied by hand. this was highly time-consuming and kept the price of books high. To get around this, it was common for students to rent books rather than purchase them. This is a strategy that is stilled being used today, especially with ebooks.

Books were so valuable that they were not even supposed to leave the city. In addition, professors were expected to turn over their lecture notes occasionally so that they could be converted into books. Famous textbooks from this time include Peter Lombard’s “Sentences” a theological book and the jurist Gratian’s text “Decretum.” With the rental system, it actually postponed the need for libraries

Degree

Completing the degrees involved 3-4 for the BA which included completing an examination before 4 teachers. Since nobody owned books, memorizing was heavy. For many students, the BA was the end of their academic career but for those who wanted to continued they were often expected to teach for two years before taking the masters.

The masters was often focused on obtaining the license to teach. This process involved attending lectures until a student believed he was ready for the examinations. This varied by disciplined but after the BA a student could take 2-4 years after completing the BA for a total of 5-8 years

Conclusion

University life was different yet somewhat similar to the modern era. The features of the modern university crept in gradually as the schools adjusted to the demands of modern life. As such, we can be sure that higher education will continue to change as it continues to adapt to the needs of the students

The Shocks of Teaching

Leave a reply

New teachers often experience the shock of being a teacher. In this post, we will look at three common shocks new teachers face. These shocks are the shock of the classroom, administration, and peers.

Shock of the Classroom

A new teacher has to deal with the reality of the classroom. The problem here is that teachers are highly familiar with the classroom experience as students. This warps their perception of the classroom as they are no longer a student but a teacher. In other words, the student is now on the other side of the desk as a teacher.

This change can be difficult to adjust to. For example, it is common for new teachers to struggle with developing appropriate relationships with students. By appropriate it is meant avoiding the pitfall of trying to be buddies with the students. Cordial relationships are good as a teacher but the teacher is still an authority figure who needs to respected and obeyed by the students. This balance is difficult for many teachers to find as many new teachers want to be liked.

Another major challenge is the implementation of all the various teaching strategies that were acquired as a student-teacher. All teaching styles work but all teaching styles do not work for all teachers. It takes time to develop a personal style of teaching and this is learned mainly through trial and error. Unfortunately, the students are the guinea pigs in this process of instructional mastery.

Shock of Administration

Working with the principal also demands a shift in perspective. All teachers were students who interacted with principals before but at a larger social distance. Now as a teacher, the social distance is smaller but this can actually make things more confusing in terms of how to relate.

The principal is a colleague but also superior. They can support a teacher’s teaching with advice and counsel but could also, and even simultaneously, believe that a teacher is unfit for their school. This dual role of supporter and judge can be uncomfortable for many.

Some principals have an open door policy while others say they have an open-door policy because that is what they are supposed to say. Some will help while others will say they will help because that is what they are supposed to say. The mixed messages can be frustrating. However, if there are any significant problems at a school it is the principal who is the first to pay the price. Therefore, many leaders are not only looking at the teachers but also trying to watch their own back.

Shock of Peers

Another shift in perspective needed is dealing with peers. Again, a new teacher brings their viewpoint of being a former student with them when interacting with fellow teachers. Now as a teacher, a new teacher gets to see what teachers are really like. The gossip in the breakroom, politic intrigue with the administration, complaints about parents and students, and more. Sometimes the atmosphere can be somewhat negative, to say the least.

Dealing with other teachers is not always negative. There are opportunities for collaboration and learning from more experienced teachers. However, it is important to know both sides of the experience so that a new teacher is not disappointed with what they see.

Conclusion

The main problem here is that a new teacher has to deal with changing their perspective on how they see education. Going forward, a new teacher is an authority figure and not a friend and a colleague/employee and not a student. With this transition comes confusion that can be overcome with time.

Paraphrasing Tips for ESL Students

1 Reply

Paraphrasing is an absolute skill in a professional setting. By paraphrasing, it is meant to have the ability to take someone else’s words and rephrase them while giving credit for the original source. Whenever a student fails to do this it is called plagiarism which is a major problem in academia. In this post, we will look at several tips on how to paraphrase.

The ability to paraphrase academically takes almost near-native writing ability. This is because you have to be able to play with the language in a highly complex manner. To be able to do this after a few semesters of ESL is difficult for the typical student. Despite this, there are several ways to try to make paraphrase work. Below are just some ideas.

Use synonyms
Change the syntax
Make several sentences
Condense/summarize

One tip not mentioned is reading. Next, to actually writing, nothing will improve writing skills like reading. Being exposed to different texts helps you develop an intuitive understanding of the second language in a way that copying and pasting never will.

Use Synonyms

Using synonyms is a first step in paraphrasing an idea but this approach cannot be used by itself as that is considered to be plagiarism by many people. With synonyms, you replace some words with others. The easiest words to replace are adjectives and verbs, followed by nouns. Below is an example. The first sentence is the original one and the second is the paraphrase.

The man loves to play guitar
The man likes to play guitar

In the example above all we did was change the word “loves” to “like”. This is a superficial change that is often still considered plagiarism because of how easy it is to do. We can take this a step further by modifying the infinitive verb “to play.”

The man loves to play guitar
The man likes to play guitar
The man likes playing guitar

Again this is superficial but a step above the first example. In addition, most word processors will provide synonyms if you right-click on the word and off course there are online options as well. Remember that this is a beginning and is a tool you use in addition to more complex approaches.

Change the Syntax

Changing the syntax has to do with the word order of the sentence or sentences. Below is an example

The man loves to play guitar
Playing the guitar is something the man loves

In this example, we move the infinitive phrase “to play” to the front and change it to a present participle. There were other adjustments that needed to be made to maintain the flow of the sentence. This example is a more advanced form of paraphrasing and it may be enough to only do this to avoid plagiarism. However, you can combine synonyms and syntax as shown in the example below

The man loves to play guitar
Playing the guitar is something the man likes

Make Several Sentences

Another approach is to convert a sentence(s) into several more sentences. As shown below

The man loves to play guitar
This man has a hobby. He likes playing guitar.

You can see that there are two sentences now. The first sentence indicates the man has a hobby and the second explains what the hobby is and how much he likes it. In addition, in the second sentence, the verb “to play” was changed to the present participle of “playing.”

Condense/Summarize

Condensing or summarizing is not considered by everyone to be paraphrasing. The separation between paraphrasing and summarizing is fuzzy and it is more of a continuum than black and white. With this technique, you try to reduce the length of the statement you are paraphrasing as shown below.

The man loves to play guitar
He likes guitar

This was a difficult sentence to summarizes because it was already so short. However, we were able to shrink it from six to three words by removing what it was about the guitar he liked.

Academic Examples

We will now look at several academic examples to show the applications of these rules in a real context. The passage below is some academic text

There is also a push within Southeast Asia for college graduates to have
interpersonal skills. For example, Malaysia is calling for graduates to
have soft skills and that these need to be part of the curriculum of tertiary schools.
In addition, a lack of these skills has been found to limit graduates’ employability.

Example 1: Paraphrase with synonyms and syntax changes

There are several skills graduates need for employability in Southeast Asia. For example, people skills are needed. The ability to relate to others is being pushed for inclusion in higher education from parts of Southeast Asia (Thomas, 2018).

You can see how difficult this can be. We rearranged several concepts and changed several verbs to try and make this our own sentence. Below is an example of condensing.

Example 2: Condensing

There is demand in Southeast Asia for higher education to develop the interpersonal skills of their students as this is limiting the employability of graduates (Thomas, 2018).

With this example, we reduced the paragraph to one sentence.

Culture and Plagiarism

There are majors differences in terms of how plagiarism is viewed based on culture. In the West, plagiarism is universally condemned both in and out of academia as essentially stealing ideas from other people. However, in other places, the idea of plagiarism is much more nuanced or even okay.

In some cultures, one way to honor what someone has said or taught is to literally repeat it verbatim. The thought process goes something like this

This person is a great teacher/elder
What they said is insightful
As a student or lower person, I cannot improve what they said
Therefore, I should copy these perfects words into my own paper.

Of course, everyone does not think like this but I have experienced enough to know that it does happen.

Whether the West likes it or not plagiarism is a cultural position rather than an ethical one. To reduce plagiarism requires to show students how it is culturally unacceptable in an academic/professional setting to do this. The tips in this post will at least provide tools for how to support students to overcome this habit

Whole Language vs Phonics

1 Reply

Among educators who specialized in reading instruction there has been a long controversy over how to teach students to read. Generally, the two main schools of thought are phonics on one side and the whole language approach on the other side. In this post, we will look at both of these approaches as well as a compromise position.

Phonics

Phonics is an approach that has the students decode the words that they see by sounding out individual letters and letter combinations. By blending the individual sounds of a word together the students is able to read the word. This requires that the student know what sounds different letters make. Without this phonemic awareness there is no hope for reading.

The benefits of this is that it is clear if a student can do this or not. This makes it easy to provide the needed support in order to help the students. This means that it is easy to assess the students development. Another benefit of this approach is that it focuses on the smallest aspects of speech sound. This helps a child to keep track of one thing at a time.
Problems with a phonic-based approach is that the importance of the context is lost because students only focus on sounding out the words rather than developing reading comprehension. This can lead to students who can read and sound out well but have no idea what they read nor the meaning of the text. The idea of seeing the passage as a whole is lost.

Whole Language

Whole language is a literature based approach that emphasizes the relevancy for the student and culture. Activities used include oral reading, silent reading, journal writing, group activities, etc. Students do not focus on sounded out words but rather on knowing the whole word through a knowledge of the context. There is even allowance made for inventive spelling in which students make for up their own words for spelling to avoid discouraging them through frequent correction of misspelled words.

An extreme example of whole language approach is when students are allowed to use substitute words in a text they are reading rather than the word the author wrote in the book. For example, if in the story the author mentions the word “pony” and the student does not understand this word. The student can substitute the word “horse with “pony” in the author’s story and this is considered okay by whole language approach standards.

Some benefits of this approach is that it is much more enjoyable in comparison to the phonics approach. Students begin reading immediately content that is relevant to their lives and interesting and their prior knowledge supposedly helps with understanding.

The drawbacks of whole language is that at times students struggle to generalize their reading skills to new contexts. In addition, the replacement of unknown to known words of the student with their own words can make it difficult for the teacher to understand where the students are struggling. If all students are doing this, it becomes difficult for them to communicate with each other about a commonly read text. This may be one reason why whole language has been reject over the pass 30 years with an emphasis on phonics.

Balance Approach

Currently, there is more of a push for a mixture of both methods. Phonics can be taught to enhance a bottom up approach while whole language is more of a use for bottom down approach to reading. By blending the two method it is possible to capture the strengths of both approaches without the corresponding weaknesses.

How this may look in the classroom may be relevant literature for the student with reading teaching that matches the needs of the students. If the student can reading without extensive phonemic awareness, training whole word might be more appropriate. When the student cannot read a word, phonics may be beneficial;.

Conclusion

It is better to match the system to the student than to match the student to the system. Whenever extreme positions are taken it helps some while hurting others. A teacher needs to have the flexibility to find the best tool for the context they are working in rather than based on what they were taught as students.

School as a Socializing Agent: Cultural Preservation

1 Reply

Many would agree that education, as found in schools, as an obligation to socialize students to help them fit into society. With this goal in mind, it is logical to conclude that there will be different views on how to socialize students. The two main extreme positions on this continuum of socialization would be

Socializing through the preservation of cultural form on generation to the next
Socializing through the questioning of prior norms and pushing for social change

As I have already mentioned, these may be the two extremes on a continuum going from complete and total cultural preservation to complete and total anarchy. In this post, we will focus the discussion on schools as agents of cultural preservation.

School as Cultural Preserver

In the view of the school as a cultural preserver, the responsibility of the school to society is to support the dominant ideas and views of the culture. This is done through teaching and explaining things from a dominant group’s perspective and excluding or censoring other viewpoints to some degree. In other words, American schools should produce Americans who support and live American values, Chinese schools should produce Chinese who support and live Chinese values, etc.

This approach to schooling has been used throughout history to compel people from minority groups to conform to the views of the dominant group. In the US, there were boarding schools for Native Americans to try to “civilize” them. This was also seen in many parts of Asia in which ethnic tribes were sent to government schools, forbidden to use their mother tongue in place of the national language, and pledge devotion and loyalty to the dominant culture. Through the process of weakening local identities, it is believed by many that it will help to strengthen the state or at least maintain the status quo. If you are in a position of dominance either of these would benefit you.

What this view lacks in diversity, due to minority views being absent, it makes up for it through stability. Schools that support cultural preservation show students their place in society and how to interact with those around them. Through the limits of a specific predefined worldview, it lowers but does not reduce internal social strife.

Problems and Pushback

A natural consequence of schools as cultural preservers is a strong sense of pride in those who belong to the culture that is being preserved. This can lead, at times, to a sense of superiority and pride. Of course, if you are not from the dominant culture, it can be suffocating to constantly have other people’s values and beliefs push upon you.

This sense of exclusion can lead to serious challenges from minority groups. There are countless examples of this in the United States where it seems everyone is pushing back against the establish dominant culture. There are those who are pushing for Black, Latino, Asian, feminist, and other worldviews to be a part of the education of the school. This is not inherently a problem, however, if everyone has an equal voice and everyone is talking at the same time this means that nobody is listening. In other words, a voice needs an ear as much as an ear needs a voice.

Conclusion

It is convenient to take an extreme position and say that using school to preserve culture is wrong. The problem with this is that the people who say this want to preserve the belief that using school to preserve culture is wrong. In other words, it is not the preservation of culture that is the problem. The real battle is over what culture is going to be preserved. Whether it is the current dominant view or the view of a challenger.

Special Angle 45-45-90 VIDEO

Leave a reply

Special Angle 45-45-90

Intro to Angles VIDEO

Leave a reply

Understanding angles as an introduction trigonometry

Exponential Equations VIDEO

Leave a reply

Exponential equations

Understanding Variables

Leave a reply

In research, there are many terms that have the same underlying meaning which can be confusing for researchers as they try to complete a project. The problem is that people have different backgrounds and learn different terms during their studies and when they try to work with others there is often confusion over what is what.

In this post, we will try to clarify as much as possible various terms that are used when referring to variables. We will look at the following during this discussion

Definition of a variable
Minimum knowledge of the characteristics of a variable in research
Various synonyms of variable

Definition

The word variable has the root of “vary” and the suffix “able”. This literally means that a variable is something that is able to change. Examples include such concepts as height, weigh, salary, etc. All of these concepts change as you gather data from different people. Statistics is primarily about trying to explain and or understand the variability of variables.

However, to make things more confusing there are times in research when a variable dies not change or remains constant. This will be explained in greater detail in a moment.

Minimum You Need to Know

Two broad concepts that you need to understand regardless of the specific variable terms you encounter are the following

Whether the variable(s) are independent or dependent
Whether the variable(s) are categorical or continuous

When we speak of independent and dependent variables we are looking at the relationship(s) between variables. Dependent variables are explained by independent variables. Therefore, one dimension of variables is understanding how they relate to each other and the most basic way to see this is independent vs dependent.

The second dimension to consider when thinking about variables is how they are measured which is captured with the terms categorical or continuous. A categorical variable has a finite number of values that can be used. Examples in clue gender, hair color, or cellphone brand. A person can only be male or female, have blue or brown eyes, and can only have one brand of cellphone.

Continuous variables are variables that can take on an infinite number of values. Salary, temperature, etc are all continuous in nature. It is possible to limit a continuous variable to categorical variable by creating intervals in which to place values. This is commonly done when creating bins for histograms. In sum, here are the four possible general variable types below

Independent categorical
Independent continuous
Dependent categorical
Dependent continuous

Natural, most models have one dependent categorical or continuous variable, however you can have any combination of continuous and categorical variables as independents. Remember that all variables have the above characteristics despite whatever terms is used for them.

Variable Synonyms

Below is a list of various names that variables go by in different disciplines. This is by no means an exhaustive list.

Experimental variable

A variable whose values are independent of any changes in the values of other variables. In other words, an experimental variable is just another term for independent variable.

Manipulated Variable

A variable that is independent in an experiment but whose value/behavior the researcher is able to control or manipulate. This is also another term for an independent variable.

Control Variable

A variable whose value does not change. Controlling a variable helps to explain the relationship between the independent and dependent variable in an experiment by making sure the control variable has not influenced in the model

Responding Variable

The dependent variable in an experiment. It responds to the experimental variable.

Intervening Variable

This is a hypothetical variable. It is used to explain the causal links between variables. Since they are hypothetical, they are observed in an actual experiment. For example, if you are looking at a strong relationship between income and life expectancy and find a positive relationship. The intervening variable for this may be access to healthcare. People who make more money have more access to health care and this contributes to them often living longer.

Mediating Variable

This is the same thing as an intervening variable. The difference being often that the mediating variable is not always hypothetical in nature and is often measured it’s self.

Confounding Variable

A confounder is a variable that influences both the independent and dependent variable, causing a spurious or false association. Often a confounding variable is a causal idea and cannot be described in terms of correlations or associations with other variables. In other words, it is often the same thing as an intervening variable.

Explanatory Variable

This variable is the same as an independent variable. The difference being that an independent variable is not influenced by any other variables. However, when independence is not for sure, than the variable is called an explanatory variable.

Predictor Variable

A predictor variable is an independent variable. This term is commonly used for regression analysis.

Outcome Variable

An outcome variable is a dependent variable in the context of regression analysis.

Observed Variable

This is a variable that is measured directly. An example would be gender or height. There is no psychology construct to infer the meaning of such variables.

Unobserved Variable

Unobserved variables are constructs that cannot be measured directly. In such situations, observe variables are used to try to determine the characteristic of the unobserved variable. For example, it is hard to measure addiction directly. Instead, other things will be measure to infer addiction such as health, drug use, performance, etc. The measures of this observed variables will indicate the level of the unobserved variable of addiction

Features

A feature is an independent variable in the context of machine learning and data science.

Target Variable

A target variable is the dependent variable in the context f machine learning and data science.

To conclude this, below is a summary of the different variables discussed and whether they are independent, dependent, or neither.

Independent	Dependent	Neither
Experimental	Responding	Control
Manipulated	Target	Explanatory
Predictor	Outcome	Intervening
Feature		Mediating
		Observed
		Unobserved
		Confounding

You can see how confusing this can be. Even though variables are mostly independent or dependent, there is a class of variables that do not fall into either category. However, for most purposes, the first to columns cover the majority of needs in simple research.

Conclusion

The confusion over variables is mainly due to an inconsistency in terms across variables. There is nothing right or wrong about the different terms. They all developed in different places to address the same common problem. However, for students or those new to research, this can be confusing and this post hopefully helps to clarify this.

T-SNE Visualization and R

Leave a reply

It is common in research to want to visualize data in order to search for patterns. When the number of features increases, this can often become even more important. Common tools for visualizing numerous features include principal component analysis and linear discriminant analysis. Not only do these tools work for visualization they can also be beneficial in dimension reduction.

However, the available tools for us are not limited to these two options. Another option for achieving either of these goals is t-Distributed Stochastic Embedding. This relative young algorithm (2008) is the focus of the post. We will explain what it is and provide an example using a simple dataset from the Ecdat package in R.

t-sne Defined

t-sne is a nonlinear dimension reduction visualization tool. Essentially what it does is identify observed clusters. However, it is not a clustering algorithm because it reduces the dimensions (normally to 2) for visualizing. This means that the input features are not longer present in their original form and this limits the ability to make inference. Therefore, t-sne is often used for exploratory purposes.

T-sne non-linear characteristic is what makes it often appear to be superior to PCA, which is linear. Without getting too technical t-sne takes simultaneously a global and local approach to mapping points while PCA can only use a global approach.

The downside to t-sne approach is that it requires a large amount of calculations. The calculations are often pairwise comparisons which can grow exponential in large datasets.

Initial Packages

We will use the “Rtsne” package for the analysis, and we will use the “Fair” dataset from the “Ecdat” package. The “Fair” dataset is data collected from people who had cheated on their spouse. We want to see if we can find patterns among the unfaithful people based on their occupation. Below is some initial code.

library(Rtsne)
library(Ecdat)

Dataset Preparation

To prepare the data, we first remove in rows with missing data using the “na.omit” function. This is saved in a new object called “train”. Next, we change or outcome variable into a factor variable. The categories range from 1 to 9

Farm laborer, day laborer,
Unskilled worker, service worker,
Machine operator, semiskilled worker,
Skilled manual worker, craftsman, police,
Clerical/sales, small farm owner,
Technician, semiprofessional, supervisor,
Small business owner, farm owner, teacher,
Mid-level manager or professional,
Senior manager or professional.

Below is the code.

train<-na.omit(Fair)
train$occupation<-as.factor(train$occupation)

Visualization Preparation

Before we do the analysis we need to set the colors for the different categories. This is done with the code below.

colors<-rainbow(length(unique(train$occupation)))
names(colors)<-unique(train$occupation)

We can now do are analysis. We will use the “Rtsne” function. When you input the dataset you must exclude the dependent variable as well as any other factor variables. You also set the dimensions and the perplexity. Perplexity determines how many neighbors are used to determine the location of the datapoint after the calculations. Verbose just provides information during the calculation. This is useful if you want to know what progress is being made. max_iter is the number of iterations to take to complete the analysis and check_duplicates checks for duplicates which could be a problem in the analysis. Below is the code.

tsne<-Rtsne(train[,-c(1,4,7)],dims=2,perplexity=30,verbose=T,max_iter=1500,check_duplicates=F)

## Performing PCA
## Read the 601 x 6 data matrix successfully!
## OpenMP is working. 1 threads.
## Using no_dims = 2, perplexity = 30.000000, and theta = 0.500000
## Computing input similarities...
## Building tree...
## Done in 0.05 seconds (sparsity = 0.190597)!
## Learning embedding...
## Iteration 1450: error is 0.280471 (50 iterations in 0.07 seconds)
## Iteration 1500: error is 0.279962 (50 iterations in 0.07 seconds)
## Fitting performed in 2.21 seconds.

Below is the code for making the visual.

plot(tsne$Y,t='n',main='tsne',xlim=c(-30,30),ylim=c(-30,30))
text(tsne$Y,labels=train$occupation,col = colors[train$occupation])
legend(25,5,legend=unique(train$occupation),col = colors,,pch=c(1))

You can see that there are clusters however, the clusters are all mixed with the different occupations. What this indicates is that the features we used to make the two dimensions do not discriminant between the different occupations.

Conclusion

T-SNE is an improved way to visualize data. This is not to say that there is no place for PCA anymore. Rather, this newer approach provides a different way of quickly visualizing complex data without the limitations of PCA.

Checkout ERT online courses

Drag, Pan, & Zoom Elements with D3.js

Leave a reply

Mouse events can be combined in order to create some rather complex interactions using d3.js. Some examples of these complex actions includes dragging, panning, and zooming. These events are handle with tools called behaviors. The behaviors deal with dragging, panning, and zooming.

In this post, we will look at these three behaviors in two examples.

Dragging
Panning and zooming

Dragging

Dragging allows the user to move an element around on the screen. What we are going to do is make three circles that are different colors that we can move around as we desire within the element. We start by setting the width, height of the svg element as well as the radius of the circles we will make (line 7). Next, we create our svg by appending it to the body element. We also set a black line around the element so that the user knows where the borders are (lines 8-14).

The next part involves setting the colors for the circles and then creating the circles and setting all of their attributes (lines 21 – 30). Setting the drag behavior comes later, and we use the .drag() and the .on() methods t create this behavior and the .call() method connects the information in this section to our circles variable.

The last part is the use of the onDrag function. This function retrieves the position of the moving element and transform the element within the svg element (lines 36-46). This involves using an if statement as well as setting attributes. If this sounds confusing, below is the code followed by a visual of what the code does.

If you look carefully you will notice I can never move the circles beyond the border. This is because the border represents the edge of the element. This is important because you can limit how far an element can travel by determining the size of the elements space.

Panning and Zooming

Panning allows you to move all visuals around at once inside an element. Zooming allows you to expand or contract what you see. Most of this code is a extension of the what we did in the previous example. The new additions are explained below.

A variable called zoomAction sets the zoom behavior by determining the scale of the zoom and setting the .on() method (Lines 9-11)
We add the .call() method to the svg variable as well as the .append(‘g’) so that this behavior can be used (Lines 20-21).
The dragAction variable is created to allow us to pan or move the entire element around. This same variable is placed inside a .call() method for the circles variable that was created earlier (Lines 40-46).
Lines 48-60 update the position of the element by making two functions. The onDrag function deals with panning and the onZzoom function deal with zooming.

Below is the code and a visual of what it does.

You can clearly see that we can move the circles individually or as a group. In addition, you also were able to see how we could zoom in and out. Unlike the first example this example allows you to leave the border. This is probably due to the zoom capability.

Conclusion

The behaviors shared here provide additional tools that you can use as you design visuals using D3.js. There are other more practical ways to use these tools as we shall see.

Factoring by Grouping VIDEO

Leave a reply

Factoring a polynomial using grouping

Intro to Interactivity with D3.js

Leave a reply

The D3.js provides many ways in which the user can interact with visual data. Interaction with a visual can help the user to better understand the nature and characteristics of the data, which can lead to insights. In this post, we will look at three basic examples of interactivity involving mouse events.

Mouse events are actions taken by the browser in response to some action by the mouse. The handler for mouse events is primarily the .on() method. The three examples of mouse events in this post are listed below.

Tracking the mouse’s position
Highlighting an element based on mouse position
Communicating to the user when they have clicked on an element

Tracking the Mouses’s Position

The code for tracking the mouse’s position is rather simple. What is new is Is that we need to create a variable that appends a text element to the svg element. When we do this we need to indicate the position and size of the text as well.

Next, we need to use the .on() method on the svg variable we created. Inside this method is the type of behavior to monitor which in this case is the movement of the mouse. We then create a simple way for the browser to display the x, y coordinates. Below is the code followed by the actual visual.

You can see that as the mouse moves the x,y coordinates move as well. The browser is watching the movement of the mouse and communicating this through the changes in the coordinates in the clip above.

Highlighting an Element Based on Mouse Position

This example allows an element to change color when the mouse comes in contact with it. To do this we need to create some data that will contain the radius of four circles with their x,y position (line 13).

Next we use the .selectAll() method to select all circles, load the data, enter the data, append the circles, set the color of the circles to green, and create a function that sets the position of the circles (lines 15-26).

Lastly, we will use the .on() function twice. Once will be for when the mouse touches the circle and the second time for when the mouse leaves the circle. When the mouse touches a circle the circle will turn black. When the mouse leaves a circle the circle will return to the original color of green (lines 27-32). Below is the code followed by the visual.

Indicating when a User Clicks on an Element

This example is an extension of the previous one. All the code is the same except you add the following at the bottom of the code right before the close of the script element.

.on('click', function (d, i) {

alert(d + ' ' + i);

});

This .on() method has an alert inside the function. When this is used it will tell the user when they have clicked on an element and will also tell the user the radius of the circle as well what position in the array the data comes from. Below is the visual of this code.

Conclusion

You can perhaps see the fun that is possible with interaction when using D3.js. There is much more that can be done in ways that are much more practical than what was shown here.

Tweening with D3.js

Leave a reply

Tweening is a tool that allows you to tell D3.js how to calculate attributes during transitions without keyframes tracking. The problem with keyframes tracking is that it can develop performance issues if there is a lot of animation.

We are going to look at three examples of the use of tweening in this post. The examples are as follows.

Counting numbers animation
Changing font size animation
Spinning shape animation

Counting Numbers Animation

This simple animation involves using the .tween() method to count from 0 to 25. The other information in the code determines the position of the element, the font-size, and the length of the animation.

In order to use the .tween() method you must make a function. You first give the function a name followed by providing the arguments to be used. Inside the function we indicate what it should do using the .interpolateRound() method which indicates to d3.js to count from 0 to 25. Below is the code followed by the animation.

You can see that the speed of the numbers is not constant. This is because we did not control for this.

Changing Font-Size Animation

The next example is more of the same. This time we simply make the size of a text change. TO do this you use the .text() method in your svg element. In addition, you now use the .styleTween() method. Inside this method we use the .interpolate method and set arguments for the font and font-size at the beginning and the end of the animation. Below is the code and the animation

Spinning Shape Animation

The last example is somewhat more complicated. It involves create a shape that spins in place. To achieve this we do the following.

Set the width and height of the element
Set the svg element to the width and height
Append a group element to the svg element.
Transform and translate the g element in order to move it
Append a path to the g element
Set the shape to a diamond using the .symbol(), .type(), and .size() methods.
Set the color of the shape using .style()
Set the .each() method to follow the cycle function
Create the cycle function
Set the .transition(), and .duration() methods
Use the .attrTween() and use the .interpolateString() method to set the rotation of the spinning.
Finish with the .each() method

Below is the code followed by the animation.

This animation never stops because we are using a cycle.

Conclusion

Animations can be a lot of fun when using d3.js. The examples here may not be the most practical, but they provide you with an opportunity to look at the code and decide how you will want to use d3.js in the future.

Adding labels to Graphs D3.js

Leave a reply

In this post, we will look at how to add the following to a bar graph using d3.js.

Labels
Margins
Axes

Before we begin, you need the initial code that has a bar graph already created. This is shown below follow by what it should look like before we make any changes.

The first change is in line 16-19. Here, we change the name of the variable and modify the type of element it creates.

Our next change begins at line 27 and continues until line 38. Here we make two changes. First, we make a variable called barGroup, which selects all the group elements of the variable g. We also use the data, enter, append and attr methods. Starting in line 33 and continuing until line 38 we use the append method on our new variable barGroup to add rect elements as well as the color and size of each bar. Below is the code.

The last step for adding text appears in lines 42-50. First, we make a variable called textTranslator to move our text. Then we append the text to the bargroup variable. The color, font type, and font size are all set in the code below followed by a visual of what our graph looks like now.

Margin

Margins serve to provide spacing in a graph. This is especially useful if you want to add axes. The changes in the code take place in lines 16-39 and include an extensive reworking of the code. In lines 16-20 we create several variables that are used for calculating the margins and the size and shape of the svg element. In lines 22-30 we set the attributes for the svg variable. In line 32-34 we add a group element to hold the main parts of the graph. Lastly, in lines 36-40 we add a gray background for effect. Below is the code followed by our new graph.

Axes

In order for this to work, we have to change the value for the variable maxValue to 150. This would give a little more space at the top of the graph. The code for the axis goes form line 74 to line 98.

Line 74-77 we create variables to set up the axis so that it is on the left
Lines 78-85 we create two more variables that set the scale and the orientation of the axis
Lines 87-99 sets the visual characteristics of the axis.

Below is the code followed by the updated graph

You can see the scale off to the left as planned..

Conclusion

Make bar graphs is a basic task for d3.js. Although the code can seem cumbersome to people who do not use JavaScript. The ability to design visuals like this often outweighs the challenges.

Defining Terms in Debates

Leave a reply

Defining terms in debates is an important part of the process that can be tricky at times. In this post, we will look at three criteria to consider when dealing with terms in debates. Below are the three criteria

When to define
What to define
How to define

When to Define

Definitions are almost always giving at the beginning of the debate. This is cause it helps to set up limits about what is discussed. It also makes it clear what the issue and potential propositions are.

Some debates focus exclusively on just defining terms. For example, highly controversial ideas such as abortion, non-traditional marriage, etc. Often the focus is just on such definitions as when does life beginning, or what is marriage? Defining terms helps to remove the fuzziness of the controversy and to focus on the exchange of ideas.

What to Define

It is not always clear what needs to be defined when staring a debate. Consider the following proposition of value

Resolved: That playing videos games is detrimental to the development of children

Here are just a few things that may need to be defined.

Video games: Does this refer to online, mobile, or console games? What about violent vs non-violent? Do educational games also fall into this category as well?
Development: What kind of development? Is this referring to emotional, physical, social or some other form of development
Children: Is this referring only to small children (0-6), young children (7-12) or teenagers?

These are just some of the questions to consider when trying to determine what to define. Again this is important because the affirmative may be arguing that videos are bad for small children but not for teenagers while the negative may be preparing a debate for the opposite.

How to Define

There are several ways to define a term below are just a few examples of how to do this.

Common Usage

Common usage is the everyday meaning of the term. For example,

We define children as individuals who are under the age of 18

This is clear and simple

Example

Example definitions give an example of the term to illustrate it as shown below.

An example of a video game would be PlayerUnknwon’s Battleground

This provides a context of the type of video games the debate may focus one

Operation

An operational definition is a working definition limited to the specific context. For example,

Video games for us is any game that is played on an electronic device

Fex define video games like this but this is an example.

Authority

Authority is a term that is defined by an expert.

According to technopedia, a video game is…..

Authority uses their experiences and knowledge to set what a term means and this can be used by debaters.

Negation

Negation is defining a word by what it is not. For example,

When we speak of video games we are not talking about educational games such as Oregon Trail. Rather, we are speaking of violent games such as Grand Theft Auto

The contrast between the types of games here is what the debater is using to define their term.

Conclusion

Defining terms is part of debating. Debaters need to be trained to understand the importance of this so that they can enhance their communication and persuasion.

Making Bar Graphs with D3.js

Leave a reply

This post will provide an example of how to make a basic bar graph using d3.js. Visualizing data is important and developing bar graphs in one way to communicate information efficiently.

This post has the following steps

Initial Template
Enter the data
Setup for the bar graphs
Svg element
Positioning
Make the bar graph

Initial Template

Below is what the initial code should look like.

Entering the Data

For the data we will hard code it into the script using an array. This is not the most optimal way of doing this but it is the simplest for a first time experience. This code is placed inside the second script element. Below is a picture.

The new code is in lines 10-11 save as the variable data.

Setup for the Bars in the Graph

We are now going to create three variables. Each is explained below

The barWidth variable will indicate ho wide the bars should be for the graph
barPadding variable will put space between the bars in the graph. If this is set to 0 it would make a histogram
The variable maxValue scales the height of the bars relative to the largest observation in the array. This variable uses the method .max() to find the largest value.

Below is the code for these three variables

The new information was added in lines 13-14

SVG Element

We can now begin to work with the svg element. We are going to create another variable called mainGroup. This will assign the svg element inside the body element using the .select() method. We will append the svg using .append and will set the width and height using .attr. Lastly, we will append a group element inside the svg so that all of our bars are inside the group that is inside the svg element.

The code is getting longer, so we will only show the new additions in the pictures with a reference to older code. Below is the new code in lines 16-19 directly under the maxValue variable.

Positioning

New=x we need to make three functions.

The first function will calculate the x location of the bar graph
The second function will calculate the y location of the bar graph
The last function will combine the work of the first two functions to place the bar in the proper x,y coordinate in the svg element.

Below is the code for the three functions. These are added in lines 21-25

The xloc function starts in the bottom left of the mainGroup element and adds the barWidth plus the barPadding to make the next bar. The yloc function starts in the top left and subtracts the maxValue from the given data point to calculate the y position. Lastly, the translator combines the output of both the xloc and the yloc functions to position bar using the translate method.

Making the Graph

We can now make our graph. We will use our mainGroup variable with the .selectAll method with the rect argument inside. Next, we use .data(data) to add the data, .enter() to update the element, .append(“rect”) to add the rectangles. Lastly, we use .attr() to set the color, transformation, and height of the bars. Below is the code in lines 27-36 followed by actual bar graph.

The graph is complete but you can see that there is a lot of work that needs to be done in order to improve it. However, that will be done in a future post.

Quadratic Discriminant Analysis with Python

Leave a reply

Quadratic discriminant analysis allows for the classifier to assess non -linear relationships. This of course something that linear discriminant analysis is not able to do. This post will go through the steps necessary to complete a qda analysis using Python. The steps that will be conducted are as follows

Data preparation
Model training
Model testing

Our goal will be to predict the gender of examples in the “Wages1” dataset using the available independent variables.

Data Preparation

We will begin by first loading the libraries we will need

import pandas as pd
from pydataset import data
import matplotlib.pyplot as plt
from sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis as QDA
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report
from sklearn.metrics import (confusion_matrix,accuracy_score)
import seaborn as sns
from matplotlib.colors import ListedColormap

Next, we will load our data “Wages1” it comes from the “pydataset” library. After loading the data, we will use the .head() method to look at it briefly.

We need to transform the variable ‘sex’, our dependent variable, into a dummy variable using numbers instead of text. We will use the .getdummies() method to make the dummy variables and then add them to the dataset using the .concat() method. The code for this is below.

In the code below we have the histogram for the continuous independent variables. We are using the .distplot() method from seaborn to make the histograms.

fig = plt.figure()
fig, axs = plt.subplots(figsize=(15, 10),ncols=3)
sns.set(font_scale=1.4)
sns.distplot(df['exper'],color='black',ax=axs[0])
sns.distplot(df['school'],color='black',ax=axs[1])
sns.distplot(df['wage'],color='black',ax=axs[2])

The variables look reasonable normal. Below is the proportions of the categorical dependent variable.

round(df.groupby('sex').count()/3294,2)
Out[247]: 
exper school wage female male
sex 
female 0.48 0.48 0.48 0.48 0.48
male 0.52 0.52 0.52 0.52 0.52

About half male and half female.

We will now make the correlational matrix

corrmat=df.corr(method='pearson')
f,ax=plt.subplots(figsize=(12,12))
sns.set(font_scale=1.2)
sns.heatmap(round(corrmat,2),
vmax=1.,square=True,
cmap="gist_gray",annot=True)

There appears to be no major problems with correlations. The last thing we will do is set up our train and test datasets.

X=df[['exper','school','wage']]
y=df['male']
X_train,X_test,y_train,y_test=train_test_split(X,y,
test_size=.2, random_state=50)

We can now move to model development

Model Development

To create our model we will instantiate an instance of the quadratic discriminant analysis function and use the .fit() method.

qda_model=QDA()
qda_model.fit(X_train,y_train)

There are some descriptive statistics that we can pull from our model. For our purposes, we will look at the group means Below are the group means.

	exper	school	wage
Female	7.73	11.84	5.14
Male	8.28	11.49	6.38

You can see from the table that mean generally have more experience, higher wages, but slightly less education.

We will now use the qda_model we create to predict the classifications for the training set. This information will be used to make a confusion matrix.

cm = confusion_matrix(y_train, y_pred)
ax= plt.subplots(figsize=(10,10))
sns.set(font_scale=3.4)
with sns.axes_style('white'):
sns.heatmap(cm, cbar=False, square=True, annot=True, fmt='g',
cmap=ListedColormap(['gray']), linewidths=2.5)

The information in the upper-left corner are the number of people who were female and correctly classified as female. The lower-right corner is for the men who were correctly classified as men. The upper-right corner is females who were classified as male. Lastly, the lower-left corner is males who were classified as females. Below is the actually accuracy of our model

round(accuracy_score(y_train, y_pred),2)
Out[256]: 0.6

Sixty percent accuracy is not that great. However, we will now move to model testing.

Model Testing

Model testing involves using the .predict() method again but this time with the testing data. Below is the prediction with the confusion matrix.

 y_pred=qda_model.predict(X_test)
cm = confusion_matrix(y_test, y_pred)
from matplotlib.colors import ListedColormap
ax= plt.subplots(figsize=(10,10))
sns.set(font_scale=3.4)
with sns.axes_style('white'):
sns.heatmap(cm, cbar=False, square=True,annot=True,fmt='g',
cmap=ListedColormap(['gray']),linewidths=2.5)

The results seem similar. Below is the accuracy.

round(accuracy_score(y_test, y_pred),2)
Out[259]: 0.62

About the same, our model generalizes even though it performs somewhat poorly.

Conclusion

This post provided an explanation of how to do a quadratic discriminant analysis using python. This is just another potential tool that may be useful for the data scientist.

Cramer’s Rule for Two Variable System of Equations VIDEO

Leave a reply

Cramer’s ruling for solving a system of equations with two variables

Double Inequalities VIDEO

Leave a reply

Solving double inequalities

Data Exploration Case Study: Credit Default

Leave a reply

Exploratory data analysis is the main task of a Data Scientist with as much as 60% of their time being devoted to this task. As such, the majority of their time is spent on something that is rather boring compared to building models.

This post will provide a simple example of how to analyze a dataset from the website called Kaggle. This dataset is looking at how is likely to default on their credit. The following steps will be conducted in this analysis.

Load the libraries and dataset
Deal with missing data
Some descriptive stats
Normality check
Model development

This is not an exhaustive analysis but rather a simple one for demonstration purposes. The dataset is available here

Load Libraries and Data

Here are some packages we will need

import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from scipy.stats import norm
from sklearn import tree
from scipy import stats
from sklearn import metrics

You can load the data with the code below

df_train=pd.read_csv('/application_train.csv')

You can examine what variables are available with the code below. This is not displayed here because it is rather long

df_train.columns
df_train.head()

Missing Data

I prefer to deal with missing data first because missing values can cause errors throughout the analysis if they are not dealt with immediately. The code below calculates the percentage of missing data in each column.

total=df_train.isnull().sum().sort_values(ascending=False)
percent=(df_train.isnull().sum()/df_train.isnull().count()).sort_values(ascending=False)
missing_data=pd.concat([total,percent],axis=1,keys=['Total','Percent'])
missing_data.head()
 
                           Total   Percent
COMMONAREA_MEDI           214865  0.698723
COMMONAREA_AVG            214865  0.698723
COMMONAREA_MODE           214865  0.698723
NONLIVINGAPARTMENTS_MODE  213514  0.694330
NONLIVINGAPARTMENTS_MEDI  213514  0.694330

Only the first five values are printed. You can see that some variables have a large amount of missing data. As such, they are probably worthless for inclusion in additional analysis. The code below removes all variables with any missing data.

pct_null = df_train.isnull().sum() / len(df_train)
missing_features = pct_null[pct_null > 0.0].index
df_train.drop(missing_features, axis=1, inplace=True)

You can use the .head() function if you want to see how many variables are left.

Data Description & Visualization

For demonstration purposes, we will print descriptive stats and make visualizations of a few of the variables that are remaining.

round(df_train['AMT_CREDIT'].describe())
Out[8]: 
count     307511.0
mean      599026.0
std       402491.0
min        45000.0
25%       270000.0
50%       513531.0
75%       808650.0
max      4050000.0

sns.distplot(df_train['AMT_CREDIT']

round(df_train['AMT_INCOME_TOTAL'].describe())
Out[10]: 
count       307511.0
mean        168798.0
std         237123.0
min          25650.0
25%         112500.0
50%         147150.0
75%         202500.0
max      117000000.0
sns.distplot(df_train['AMT_INCOME_TOTAL']

I think you are getting the point. You can also look at categorical variables using the groupby() function.

We also need to address categorical variables in terms of creating dummy variables. This is so that we can develop a model in the future. Below is the code for dealing with all the categorical variables and converting them to dummy variable’s

df_train.groupby('NAME_CONTRACT_TYPE').count()
dummy=pd.get_dummies(df_train['NAME_CONTRACT_TYPE'])
df_train=pd.concat([df_train,dummy],axis=1)
df_train=df_train.drop(['NAME_CONTRACT_TYPE'],axis=1)

df_train.groupby('CODE_GENDER').count()
dummy=pd.get_dummies(df_train['CODE_GENDER'])
df_train=pd.concat([df_train,dummy],axis=1)
df_train=df_train.drop(['CODE_GENDER'],axis=1)

df_train.groupby('FLAG_OWN_CAR').count()
dummy=pd.get_dummies(df_train['FLAG_OWN_CAR'])
df_train=pd.concat([df_train,dummy],axis=1)
df_train=df_train.drop(['FLAG_OWN_CAR'],axis=1)

df_train.groupby('FLAG_OWN_REALTY').count()
dummy=pd.get_dummies(df_train['FLAG_OWN_REALTY'])
df_train=pd.concat([df_train,dummy],axis=1)
df_train=df_train.drop(['FLAG_OWN_REALTY'],axis=1)

df_train.groupby('NAME_INCOME_TYPE').count()
dummy=pd.get_dummies(df_train['NAME_INCOME_TYPE'])
df_train=pd.concat([df_train,dummy],axis=1)
df_train=df_train.drop(['NAME_INCOME_TYPE'],axis=1)

df_train.groupby('NAME_EDUCATION_TYPE').count()
dummy=pd.get_dummies(df_train['NAME_EDUCATION_TYPE'])
df_train=pd.concat([df_train,dummy],axis=1)
df_train=df_train.drop(['NAME_EDUCATION_TYPE'],axis=1)

df_train.groupby('NAME_FAMILY_STATUS').count()
dummy=pd.get_dummies(df_train['NAME_FAMILY_STATUS'])
df_train=pd.concat([df_train,dummy],axis=1)
df_train=df_train.drop(['NAME_FAMILY_STATUS'],axis=1)

df_train.groupby('NAME_HOUSING_TYPE').count()
dummy=pd.get_dummies(df_train['NAME_HOUSING_TYPE'])
df_train=pd.concat([df_train,dummy],axis=1)
df_train=df_train.drop(['NAME_HOUSING_TYPE'],axis=1)

df_train.groupby('ORGANIZATION_TYPE').count()
dummy=pd.get_dummies(df_train['ORGANIZATION_TYPE'])
df_train=pd.concat([df_train,dummy],axis=1)
df_train=df_train.drop(['ORGANIZATION_TYPE'],axis=1)

You have to be careful with this because now you have many variables that are not necessary. For every categorical variable you must remove at least one category in order for the model to work properly. Below we did this manually.

df_train=df_train.drop(['Revolving loans','F','XNA','N','Y','SK_ID_CURR,''Student','Emergency','Lower secondary','Civil marriage','Municipal apartment'],axis=1)

Below are some boxplots with the target variable and other variables in the dataset.

f,ax=plt.subplots(figsize=(8,6))
fig=sns.boxplot(x=df_train['TARGET'],y=df_train['AMT_INCOME_TOTAL'])

There is a clear outlier there. Below is another boxplot with a different variable

f,ax=plt.subplots(figsize=(8,6))
fig=sns.boxplot(x=df_train['TARGET'],y=df_train['CNT_CHILDREN'])

It appears several people have more than 10 children. This is probably a typo.

Below is a correlation matrix using a heatmap technique

corrmat=df_train.corr()
f,ax=plt.subplots(figsize=(12,9))
sns.heatmap(corrmat,vmax=.8,square=True)

The heatmap is nice but it is hard to really appreciate what is happening. The code below will sort the correlations from least to strongest, so we can remove high correlations.

c = df_train.corr().abs()

s = c.unstack()
so = s.sort_values(kind="quicksort")
print(so.head())

FLAG_DOCUMENT_12 FLAG_MOBIL 0.000005
FLAG_MOBIL FLAG_DOCUMENT_12 0.000005
Unknown FLAG_MOBIL 0.000005
FLAG_MOBIL Unknown 0.000005
Cash loans FLAG_DOCUMENT_14 0.000005

The list is to long to show here but the following variables were removed for having a high correlation with other variables.

df_train=df_train.drop(['WEEKDAY_APPR_PROCESS_START','FLAG_EMP_PHONE','REG_CITY_NOT_WORK_CITY','REGION_RATING_CLIENT','REG_REGION_NOT_WORK_REGION'],axis=1)

Below we check a few variables for homoscedasticity, linearity, and normality using plots and histograms

sns.distplot(df_train['AMT_INCOME_TOTAL'],fit=norm)
fig=plt.figure()
res=stats.probplot(df_train['AMT_INCOME_TOTAL'],plot=plt)

This is not normal

sns.distplot(df_train['AMT_CREDIT'],fit=norm)
fig=plt.figure()
res=stats.probplot(df_train['AMT_CREDIT'],plot=plt)

This is not normal either. We could do transformations, or we can make a non-linear model instead.

Model Development

Now comes the easy part. We will make a decision tree using only some variables to predict the target. In the code below we make are X and y dataset.

X=df_train[['Cash loans','DAYS_EMPLOYED','AMT_CREDIT','AMT_INCOME_TOTAL','CNT_CHILDREN','REGION_POPULATION_RELATIVE']]
y=df_train['TARGET']

The code below fits are model and makes the predictions

clf=tree.DecisionTreeClassifier(min_samples_split=20)
clf=clf.fit(X,y)
y_pred=clf.predict(X)

Below is the confusion matrix followed by the accuracy

print (pd.crosstab(y_pred,df_train['TARGET']))
TARGET       0      1
row_0                
0       280873  18493
1         1813   6332
accuracy_score(y_pred,df_train['TARGET'])
Out[47]: 0.933966589813047

Lastly, we can look at the precision, recall, and f1 score

print(metrics.classification_report(y_pred,df_train['TARGET']))
              precision    recall  f1-score   support

           0       0.99      0.94      0.97    299366
           1       0.26      0.78      0.38      8145

   micro avg       0.93      0.93      0.93    307511
   macro avg       0.62      0.86      0.67    307511
weighted avg       0.97      0.93      0.95    307511

This model looks rather good in terms of accuracy of the training set. It actually impressive that we could use so few variables from such a large dataset and achieve such a high degree of accuracy.

Conclusion

Data exploration and analysis is the primary task of a data scientist. This post was just an example of how this can be approached. Of course, there are many other creative ways to do this but the simplistic nature of this analysis yielded strong results

Confidence Intervals for Proportions VIDEO

Leave a reply

Calculating Confidence Intervals for Proportions

Teaching Materials

3 Replies

Regardless of what level a teacher teaches at you are always looking or materials and activities to do in your class. It is always a challenge to have new ideas and activities to support and help students because the world changes. This leads to an constant need to remove old inefficient activities and bring in new fresh one. The primary purpose of activities is to provide practical application of the skills taught and experience in school.

For the math teacher you can naturally make your own math problems. However, this can quickly become quietly. One solution to this is to employed other worksheets that provide growth opportunities for the students with stressing out the teacher.

There are many great websites for this. For example, education.com provides many different types of worksheets to help students. They have some great simple math worksheets like the ones below

addition_outer space_answers

addition_outer space

There are many more resources available at education.com as well as other sites. There is no purpose or benefit to reinventing the wheel. The incorporation of the assignments of others is a great way to expand the resources you have available without the stress of developing them yourself.

Data Science Pipeline

Leave a reply

One of the challenges of conducting a data analysis or any form of research is making decisions. You have to decide primarily two things

What to do
When to do it

People who are familiar with statistics may know what to do but may struggle with timing or when to do it. Others who are weaker when it comes to numbers may not know what to do or when to do it. Generally, it is rare for someone to know when to do something but not know how to do it.

In this post, we will look at a process that that can be used to perform an analysis in the context of data science. Keep in mind that this is just an example and there are naturally many ways to perform an analysis. The purpose here is to provide some basic structure for people who are not sure of what to do and when. One caveat, this process is focused primarily on supervised learning which has a clearer beginning, middle, and end in terms of the process.

Generally, there are three steps that probably always take place when conducting a data analysis and they are as follows.

Data preparation (data mugging)
Model training
Model testing

Off course, it is much more complicated than this but this is the minimum. Within each of these steps there are several substeps, However, depending on the context, the substeps can be optional.

There is one pre-step that you have to consider. How you approach these three steps depends a great deal on the algorithm(s) you have in mind to use for developing different models. The assumptions and characteristics of one algorithm are different from another and shape how you prepare the data and develop models. With this in mind, we will go through each of these three steps.

Data Preparation

Data preparation involves several substeps. Some of these steps are necessary but general not all of them happen ever analysis. Below is a list of steps at this level

Data mugging
Scaling
Normality
Dimension reduction/feature extraction/feature selection
Train, test, validation split

Data mugging is often the first step in data preparation and involves making sure your data is in a readable structure for your algorithm. This can involve changing the format of dates, removing punctuation/text, changing text into dummy variables or factors, combining tables, splitting tables, etc. This is probably the hardest and most unclear aspect of data science because the problems you will face will be highly unique to the dataset you are working with.

Scaling involves making sure all the variables/features are on the same scale. This is important because most algorithms are sensitive to the scale of the variables/features. Scaling can be done through normalization or standardization. Normalization reduces the variables to a range of 0 – 1. Standardization involves converting the examples in the variable to their respective z-score. Which one you use depends on the situation but normally it is expected to do this.

Normality is often an optional step because there are so many variables that can be involved with big data and data science in a given project. However, when fewer variables are involved checking for normality is doable with a few tests and some visualizations. If normality is violated various transformations can be used to deal with this problem. Keep mind that many machine learning algorithms are robust against the influence of non-normal data.

Dimension reduction involves reduce the number of variables that will be included in the final analysis. This is done through factor analysis or principal component analysis. This reduction in the number of variables is also an example of feature extraction. In some context, feature extraction is the in goal in itself. Some algorithms make their own features such as neural networks through the use of hidden layer(s)

Feature selection is the process of determining which variables to keep for future analysis. This can be done through the use of regularization such or in smaller datasets with subset regression. Whether you extract or select features depends on the context.

After all this is accomplished, it is necessary to split the dataset. Traditionally, the data was split in two. This led to the development of a training set and a testing set. You trained the model on the training set and tested the performance on the test set.

However, now many analyst split the data into three parts to avoid overfitting the data to the test set. There is now a training a set, a validation set, and a testing set. The validation set allows you to check the model performance several times. Once you are satisfied you use the test set once at the end.

Once the data is prepared, which again is perhaps the most difficult part, it is time to train the model.

Model training

Model training involves several substeps as well

Determine the metric(s) for success
Creating a grid of several hyperparameter values
Cross-validation
Selection of the most appropriate hyperparameter values

The first thing you have to do and this is probably required is determined how you will know if your model is performing well. This involves selecting a metric. It can be accuracy for classification or mean squared error for a regression model or something else. What you pick depends on your goals. You use these metrics to determine the best algorithm and hyperparameters settings.

Most algorithms have some sort of hyperparameter(s). A hyperparameter is a value or estimate that the algorithm cannot learn and must be set by you. Since there is no way of knowing what values to select it is common practice to have several values tested and see which one is the best.

Cross-validation is another consideration. Using cross-validation always you to stabilize the results through averaging the results of the model over several folds of the data if you are using k-folds cross-validation. This also helps to improve the results of the hyperparameters as well. There are several types of cross-validation but k-folds is probably best initially.

The information for the metric, hyperparameters, and cross-validation are usually put into a grid that then runs the model. Whether you are using R or Python the printout will tell you which combination of hyperparameters is the best based on the metric you determined.

Validation test

When you know what your hyperparameters are you can now move your model to validation or straight to testing. If you are using a validation set you asses your models performance by using this new data. If the results are satisfying based on your metric you can move to testing. If not, you may move back and forth between training and the validation set making the necessary adjustments.

Test set

The final step is testing the model. You want to use the testing dataset as little as possible. The purpose here is to see how your model generalizes to data it has not seen before. There is little turning back after this point as there is an intense danger of overfitting now. Therefore, make sure you are ready before playing with the test data.

Conclusion

This is just one approach to conducting data analysis. Keep in mind the need to prepare data, train your model, and test it. This is the big picture for a somewhat complex process

Solving Linear Equation VIDEO

Leave a reply

Solving a linear equation

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

t-sne Defined

Initial Packages

Dataset Preparation

Visualization Preparation

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this: