Competency-Based Grading

Abstract

We have created a browser extension which will allow for Gradescope to more expediently be used for competency-based grading. Competency-based grading is an approach to academic grading which ties learning outcomes to specific competencies. Competencies can be evaluated in a variety of assignments and later grades on a specific competency supersede earlier grades automatically, allowing for students to be rewarded for demonstrating growth. Current native support for competency-based grading in enterprise-level learning management software (LMS) is slim, and any LMS extensions would need to be specifically approved by technical administrators, adding a barrier to adoption. Thus, we propose the usage of browser extension which scrapes the Gradescope page for data and uses existing site functionality to run competency-based grading as an instructor. As the extension doesn’t transmit any data to remote servers, but instead only acts as an intermediate layer between the user and Gradescope, the extension could be used in a class without violating FERPA.

A class dashboard for Gradescope. The extension is open in the top right and presents a menu that shows vertically-stacked buttons to view individual competencies. An assignment creation page on Gradescope. The extension is open in the top right and shows assignment creation options, such as a rubric and the assignment's percentage of the final grade.

Basic Functions

  • The ability for instructors to grade assignments through the interface, scoring students on each included competency, and writing comments where relevant.

  • The ability for students to view their grades through the interface. This must support both viewing the student’s calculated competency scores, as well as a conversion from the competency-based scale to the standard four-point scale.

  • The ability for students to fill out assignments through the interface, submitting text or other media. The interface must clearly indicate which parts or aspects of the assignment are relevant to which competencies.

Background

Research

Heather

Link: http://www.nutritiousminds.org/surviving-college-as-a-student-with-disabilities.html

The paper describes the author’s experiences with undergraduate classes as a disabled person. It points out issues in the academic system for people with disabilities. It focuses on the struggle to gain accommodations by highlighting points of conflict with the author’s professors. Some of the examples given include direct bullying or saying she did not need accommodations. The most shocking thing is that all of this took place at a university known for its excellent Disability Support Services program. This shows that the issues still exist in places that are considered more disability-friendly. This paper relates to our project indirectly. A key point raised in this paper is that professors have the final say on whether or not an accommodation will be given. By making competency-based grading much more automated it is harder to say no to using. This is because it would not require much more work than typical grading. Competency-based grading as a whole would also help solve some of these issues. This is because it would encourage professors to create assignments that test specific skills rather than looking for specific answers. It’s always shocking to hear about how closely these issues are attached to the system. However, people inside of the system itself are fighting these issues. Silber specifically talks about actions she has taken with the university’s Disability Support Services program that have helped improve it. Seeing that the system has been changed makes me feel much more confident about our idea’s chances of working. Seeing the author’s problems with getting accommodations makes me feel like our project would be actually helpful. This has been a very encouraging paper.

Ananya

Link: https://arxiv.org/abs/2510.25964
Summary: Computer courses must be accessible for disabled students. This proves difficult in large courses with heavy content and massive staff teams. We share our attempts to make introductory computer courses more accessible. These courses enroll more than 3,500 students, and the university hires 100 teaching assistants (TAs) every year. First, we search course materials for accessibility problems and fix them. However, fixing content after publication is difficult. To solve this, we introduce two ways to find accessibility problems in advance. Then, we develop a system to manage these efforts. We make course content easier to access on the web, provide alternative ways to view content, and write automated checks to find problems. In addition, we run accessibility trainings for TAs so that accessibility becomes a course-wide norm. Feedback from staff and students shows that their engagement with accessible work and technologies increases. Finally, we discuss lessons learned and offer advice on how to develop similar systems.

Frost

Link: http://www.nutritiousminds.org/surviving-college-as-a-student-with-disabilities.html
Silber speaks about her experiences as a college student studying math and computer science. She mentions her struggles to obtain the help she needs to succeed from her past universities. This quote sums up the barriers she faced: “The problem was that I’m differently-abled, so how I learn, how long it takes me to achieve comprehension, what methods of inputting information work for me are all factors that reflect how much extra work and how much extra time I would need to devote.”
The author writes about specific problems with teachers:

  • Teachers who came from countries that do not have disability assistance in schools did not expect to have to provide special assistance
  • Some teachers did not think that the author needed special assistance because the author did not appear to have a disability
  • Teachers did not have enough experience teaching disabled students
  • Some teachers thought that the author was getting unfair help from the special assistance that they asked for

This article describes a person trying to succeed at multiple colleges, and mentions the issues that they had to deal with. The article doesn’t describe any specific accessibility technology. The author describes the specific aid that they were given, and the ways that this aid helped. The article also mentions the problems that the author had while trying to access special aid for disabled students. The problems that the author describes are examples of how hard it can be to get help as a disabled student, even when the government requires your college to support you. The author refuses to go to any college without a Disability Services center, which limits the schools that they could choose from. These issues can make it hard for students with disabilities to succeed in school.

First-Person Account

Link: http://www.nutritiousminds.org/surviving-college-as-a-student-with-disabilities.html

This first person account meets the requirements of the assignment because it’s not a teaching video or an advertisement. It also gives us a perspective outside of personal experiences/the experiences of people we know like peers and course staff.

Silber speaks about her experiences as an undergraduate studying mathematics and a graduate studying computer science, as well as her struggle to obtain the appropriate accommodations she would need to succeed in those pursuits. One specific quote sums up the barriers she faced very succinctly: “The problem was that I’m differently-abled, so how I learn, how long it takes me to achieve comprehension, what methods of inputting information work for me are all factors that reflect how much extra work and how much extra time I would need to devote.” Some of the specific hurdles with professors she mentions while studying include those who were from countries without disability accommodations, who didn’t understand why she needed them when she wasn’t visibly disabled, who lacked experience teaching disabled students, and those who thought she was privileged for needing federally mandated accommodations in the first place.

While not strictly about an assistive technology, this first-person account pertains to a disabled person attempting to navigate undergrad and later grad school, and describing the issues faced along the way. The author references specific accommodations that were provided (such as time-and-a-half on exams), and the way these accommodations helped, as well as issues that were experienced when attempting to access accommodations.

The issues raised in the first-person account are an example of the broader issue of a lack of available accommodations for students who require them, and the resistance of faculty and administration to provide those accommodations (even when required by law). The author specifically mentions that their choice in grad school was limited because they only considered universities with an active Disability Services center— “If your school does not have disability services, don’t bother applying there”. These issues can make it difficult for even the most self-advocating of students to succeed in school (or academia in general).

Goals

Storyboards

Storyboard panel 1: Creating an assignment and selecting competencies. Storyboard panel 2: Publishing the assignment and collecting submissions. Storyboard panel 3: Closing the assignment and grading submissions. Storyboard panel 4: Handling regrades

Storyboard panel 1: Student views default assignment page. Storyboard panel 2: Student opens competency filter and selects "Assessment" from dropdown. Storyboard panel 3: Assignments page shows only assignments with the given competency.

Core

These are our core goals for the project, roughly ordered by importance:

1) Include a tool which allows for the management of competency-based grades and assignments when used with Gradescope. The tool should support basic functions such as posting assignments and checking grades.

2) Include a tool which is accessible, which can be used with a variety of ATs, and which complies with WCAG guidelines to at least an AA level.

3) Include a tool which can be easily used with little to no instruction. The tool should not have noticeable delays or inconsistencies.

4) Develop a tool which can be used in an enterprise educational setting without violating FERPA or otherwise requiring an audit. Functionally, this means no transmitting of data to remote servers.

Stretch

Features and actions we would like to add or take in the future, given time:

1) Test the tool with users outside of our project group to evaluate usability for end users.

2) Make our tool more visually appealing.

3) Refine our tool such that it conforms to AAA WCAG standards.

4) Improve compatibility such that it works across a variety of browsers.

5) Improve robustness of the tool such that minor changes to Gradescope do not break its functionality.

Usability

In order to meet our third goal, to be usable with little to no instruction, our tool should:

  • Include an about page with instructions on how to use the tool

  • Use standard symbology for navigation e.g. back arrow for back navigation, hamburger menu symbol for menu

  • Include tooltips on buttons

  • Surface all menus and navigation in expected places such as at the top or the bottom of the page

FERPA Compliance

In order to meet our fourth goal, FERPA compliance, our tool will:

  • Never send grades to a remote server as part of its operation.

    • The tool can achieve this by reading grades from the Canvas and Gradescope pages each time it is activated in the user’s browser, so that grades never need to be stored by our tool across sessions.

Validation and Assessment

Validation

We minimally validated the functionality of our project by three methods: navigation by a typical user without use of any ATs, navigation by a typical user while using VoiceOver, and navigation by a typical user while using switch controls.

Assessment

Our project is still very much under construction and does not meet all WCAG guidelines. Some are met and some are not. For each of the following criteria, we have determined that our project falls into one of the following categories:

  • Supports: The functionality of the product has at least one method that meets the criterion without known defects or meets with equivalent facilitation.
  • Partially Supports: Some functionality of the product does not meet the criterion (explain which)
  • Does Not Support: The majority of product functionality does not meet the criterion (explain which)

Success Criterion 4.1.2 - Name, Role, Value

Partially Supports: The names of the buttons all clearly convey what actions they lead to, and there are additional descriptions beneath the text inputs that provide context.

Success Criterion 2.4.3 - Focus Order

Supports: When using NVDA, all the elements are read aloud in logical order

Success Criterion 1.4.10 - Reflow

Does Not Support: As of now, there is no way to change the font size, so this criterion has not been met.

Success Criterion 1.4.11 - Non textual Contrast

Supports: Most colors used are either black or white, so most elements are maximally contrasted with one another.

Success Criterion 1.3.1 - Info and Relationships

Partially Supports: For now, the relationships and information can be ascertained programatically, but they are conveyed primarily through visuals.

Success Criterion 1.1.1 - Non-text Content

Supports: There is no content without text.

Success Criterion 1.3.4 - Orientation

Does Not Support: Our extension does not respond to re-orienting the viewport.

Success Criterion 4.1.3 - Status Messages

Partially Supports: Any UI changes made by this extension will no doubt show up on an assistive technology, but we could implement more robust messaging.

Success Criterion 1.4.3 - Contrast (Minimum)

Supports: See 1.4.11 above.

Disability Justice Principles

RECOGNIZING WHOLENESS

The DJ principle recognizing wholeness is about the inherent worth people have outside of all the systems we have created that claim to measure it. Sins Invalid specifically mentions “commodity relations and capitalist notions of productivity”. A traditional 0-100 grading system very much falls into that trap. It allows for little recognition of a student’s ability beyond their performance at very rigid checkpoints, like homework assignments or quizzes. While students can demonstrate understanding through projects and homework sets, there is little room for improvement, and no incentive to encourage continued learning. By creating a tool to increase the ease with which competency-based grading can be implemented in the classroom, we can hopefully allow for students’ skills to be measured in a more holistic way. According to “A Narrative Review of Selected Studies in Competency”, a report by Dr. Paul Wright of Marzano Academies, competency-based grading measures demonstration and understanding rather than rote learning, which on the whole serves as a more meaningful measure of a student’s mastery of a subject. With this in mind, a tool that encourages the use of competency-based grading relates to the DJ Principle of recognizing wholeness.

SUSTAINABILITY

The DJ principle sustainability is about prioritizing ongoing change by pacing ourselves. In a traditional 100-point grading scale, you have only one chance to succeed, and one chance to fail. This encourages a very troubling mindset of “if this is not on the test, I will not need to know it”. As students who are in college specifically to build skills that will sustain us once we leave, this isn’t exactly a mindset that is conducive to truly mastering anything. Competency based grading encourages sustained learning because it has chances for multiple redos; there is an emphasis on showing mastery instead of relying on just one limited metric. Another way we embody sustainability is because building tools like this encourages long-term behavior change in our target audience. If implementing competency based grading becomes simpler—and we also have other tools to do this, like some kind of research-backed guide—then our tool could potentially create long-term change in the Allen School.

Going Forward

This was quite an ambitious undertaking, and we did not fully implement everything we had been hoping to; the difference between what we intended and what we have can be pretty clearly seen when you compare the functions described here against what we’ve described in our design doc. If we had more time, then our priorities in order would be:

1) Finish implementing our other core features (particularly instructor-side items like adding assignments and competencies)

2) Bring our tool up to WCAG AA standards, because right now it simply isn’t compliant to the degree it should be.

3) Handle backend storage better, because right now I’ve heard it described as a “rat’s nest” and that’s… not really wrong.

4) Repeat 1-3 above as necessary for our stretch goals!

In terms of things we learned, we knew going into this that competency-based grading would be hard to implement in pretty much any context – in a class, on a website, wherever; pretty much no system is built for it. But we didn’t realize exactly how hostile the actual website architecture of Gradescope would be. To quote our main backend dev, ‘wow this really wasn’t built to support this at all.’ At a baseline converting from competency-based grade to 4.0 scale is kinda annoying and unwieldy, but at least it’s not too bad within the code. The hard part is that Gradescope doesn’t even seem to support grades beyond ‘not submitted’, ‘not graded’, and number scores. That means even in the best case we end up storing the competency grades as something like x/3.0, which is both not entirely accurate and also not very pretty. On another note, the grouping of assignments by certain parameters just isn’t supported at all. The grading interface and structure are not very extensible at all and aren’t well equipped to support any alternative grading system – which is pretty reflective of grading systems in academics as a whole.

The system isn’t built to change, which isn’t great, because the system is flawed. We’ve put forward this project mostly as proof of concept that it can be changed, though as of right now it’s still rather clunky and unfinished. It would take a lot more work to get a proper version up and going, but we do think it’s possible.