Adam's research idea

Let's assume, for the moment, that some method exists for generating assessments of some kind. These may or may not be facet assessments, and they may or may not be automatically generated. What can we learn about students and their interactions by mining the temporal patterns among activities and assessments. There are many ways this could be focused, but I'll concentrate on two major areas, just to give some examples. The basic idea is to merge information from different sources (e.g. facet profiles, facet acquisition records and activity logs) to learn something about the process of learning.

The first thing you could do with activity logs and some sort of student model is evaluate the benefits of educational activities. One problem with pre-tests and post-tests is that they are fairly broad and present a huge credit-assignment problem. But if you have timestamped evidence that a student has achieved a certain level of mastery or understood a concept, and you also have logs of the students activities, then you might be able to do a much better job of determining which activities (or possibly even specific actions within the activities) are most responsible for the learning. This is, of course, still not perfect. Not all activities create a log. Maybe the student talked to his neighbor, who explained the concept to him. Maybe she had an aha! moment earlier in the day and is just testing it out. However, some claims could reasonably be made that if, in the course of doing some online activity, evidence for a conceptual leap is found. And if that evidence is then supported by subsequent corroboration, it is reasonable to credit the activity for contributing to that learning. Furthermore, if this happened a number of times with different students, then the argument would be made even stronger.

A more interesting use of this idea would be to add more features to the "model" of how educational activities aid in learning. For example, if a students with a particular facet profile tend to learn a concept from some activity, but students with a different facet profile don't, then that might speak to the learning style of those students, or possibly some larger world-view conceptualization that the two facet profiles represent, one of which admits of extension in a particular way while the other does not.

The second application of this idea is to evaluate how well students teach each other, rather than how well software teaches them. This makes the most sense in the context of the INFACT-forum, but could well be augmented by other sources of facet assessments. Again, the idea is to look for those moments when a student seems to make a breakthrough and then try to trace the source of that learning, but this time, by looking at the INFACT discussions preceding that breakthrough. If several students in a group all learn a new concept in response to a post from a single student, and this happens several times, then that student might be labelled a "knowledge catalyst." This is a student who's good at explaining concepts to her peers. This could be useful information in group formation. Another way to get at this is if we had automatic labels for posts according to speech acts (question, answer, acknowledgement, reinforcement, etc.) Then we could find students who tend to ask a lot of questions and student who tend to try and answer questions frequently. Again, this might be helpful for group formation.


Adam Carlson
Last modified: Mon Mar 8 15:08:33 PST 2004