Engaging with Children’s Artwork in Mixed-Ability Families

Introduction

Creating art is a fundamental part of childhood, cherished by both children and parents as a means to connect and bond over their children’s artistic expressions. For children, creating art is a way to express their emotions, imagination, and experiences. For parents, viewing and discussing their children’s artwork is a way to understand their child’s world, encourage their creativity, and foster emotional connections.

However, for parents who are blind or have low vision (BLV), engaging with their sighted children’s artwork poses unique challenges. Traditional methods of interacting with visual art rely heavily on sight, which can make it difficult for BLV parents to fully appreciate and discuss their children’s creations. This gap in engagement can impact the emotional connection and understanding between parents and their children, potentially affecting the child’s confidence and the parent-child relationship.

Our project aims to address this challenge by developing an iOS app that uses AI-generated descriptions to help BLV parents engage with their children’s artwork. The app leverages advanced GPT-APIs to create detailed, vivid descriptions from pictures of children’s artwork, which parents can then explore using voiceover, high-contrast, and magnification settings. This tool not only facilitates self-exploration of the artwork but also enables BLV parents to have meaningful conversations with their children about their creations, thus bridging the gap caused by visual impairment. By incorporating feedback from formative research with BLV parents and their children, our app is designed to meet the specific needs of mixed visual ability families. The inclusion of a human-in-the-loop system allows children and parents to edit the AI-generated descriptions, ensuring accuracy and personal relevance. This collaborative approach not only improves the quality of the descriptions but also empowers both parents and children to contribute to the understanding and appreciation of the artwork. By ensuring that BLV parents can actively engage with and discuss their children’s artistic expressions, we aim to enhance the emotional bonds within mixed-ability families and support the developmental growth of children through positive reinforcement and shared experiences.

In addition to addressing the immediate needs of BLV parents, our app also contributes to the broader goals of accessibility and inclusion. By demonstrating how technology can be used to create inclusive experiences, we hope to inspire further innovations that support diverse needs and promote equity in family interactions and beyond. With this project, we envision a future where all parents, regardless of their visual abilities, can fully participate in their children’s creative journeys, fostering a more inclusive and understanding world for everyone.

An image with 5 smaller images of different children's artwork. The images (from top left to bottom right) are labeled A, B, C, E, F. The first image (A) consists of 6 figures sketched using pencil in a notebook. Each figure has a different expression and haircut, and is only shown from the top up. The second image (B) is of a black silhouette with white lines and a red streak on a white paper, surrounded by swirls and dots in different colors. The third image (C) has large blue and green brushstrokes, seemingly abstract in nature. The fourth image (E) is of a colorful fish with rectangular cutouts of colored construction paper set against a black background. The last image (F) is of a pink and green mandala artwork, which is cut out in a circular shape and held up against a white background. Figure 1: Images of different children’s artworks

Positive Disability Principles

1. Our project is not ableist and aims to make children’s artwork more accessible to BLV parents. The formative work guiding this project ensures that our technology solution is not a “disability dongle”.

2. BLV family members and their children have guided this work, primarily through the findings from formative studies as previously mentioned. In the future, we hope to co-design and evaluate any technology outputs of this work with mixed visual ability families.

3. Our project aims to improve the agency of people with disabilities by providing BLV parents a supplemental understanding of their children’s artwork which they can use to further engagements with their children.

4. While we are not addressing multiple disabled people (such as deafblind individuals) through this work, as that would require more formative understanding, we have consulted with multiple different BLV individuals and their children. Additionally, the formative work has looked at a wide range of relative-child relationships: parent-child, grandparent-grandchild, great-grandparent-great-grandchild, aunt/uncle-niece/nephew. This ensures that our solutions work not only for BLV parents and their children but also for other BLV-sighted family relationships.

5. All technology produced through this work is accessible to BLV individuals. We tested our app with VoiceOver and other accessibility settings like magnification and high-contrast. We have also avoided cognitively overloaded designs, and used plain language wherever possible in our application. We used standard digital design tools such as PowerPoint or Figma, which have been audited and tested for accessibility, during our design phase.

Several projects and studies have informed our approach to making children’s artwork accessible to BLV parents.

User-Centered Design in Accessibility

User-centered design principles are crucial in accessibility projects. Previous studies emphasize the importance of involving end-users in the design process to create solutions that truly meet their needs. Hence, the research mentioned above has been conducted with BLV parents and children, ensuring that the app is both usable and valuable in the mixed-ability settings.

AI-Generated Descriptions

Previous work has explored the use of AI to generate descriptions for visual content. Projects like Microsoft’s Seeing AI and Google’s Lookout provide insights into the capabilities and limitations of AI in generating useful descriptions for visually impaired users.

Technology in Mixed-Ability Family Interactions

Prior research in mixed-ability family settings includes work by Park, S., Cassidy, C.T., & Branham, S.M. (2023). “It’s All About the Pictures:” Understanding How Parents/Guardians With Visual Impairments Co-Read With Their Child(ren). Proceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility. and Storer, K.M., & Branham, S.M. (2019). “That’s the Way Sighted People Do It”: What Blind Parents Can Teach Technology Designers About Co-Reading with Children. Proceedings of the 2019 on Designing Interactive Systems Conference., who explore the design of technology for co-reading interactions between BLV parents and their children, approaching the BLV parent’s perspective through observational and interview studies like we do in our work. However, prior research has not investigated technology facilitating a mixed visual-ability family’s interaction regarding children’s artwork.

Formative Work [In Submission]

A member of our team led formative work that is in submission Anonymous Authors, “Engaging with Children’s Artwork in Mixed Visual Ability Families,” in Proceedings of the ACM SIGACCESS, 2024. to investigate what motivates BLV relatives to engage with their children’s artwork, current practices by BLV relatives and their children to discuss and adapt the child’s artwork, BLV adults and sighted children’s reactions to AI interpretations of the child’s artwork, and the families’ collective responses to the technology design probes. These research questions were explored through two studies: the first was with 14 BLV individuals, and the second study was with 5 BLV relative and sighted children groups. These studies consisted of semi-structured interviews, using two different AI tools to describe children’s artwork (provided by the BLV family members), and two in-person design probes to evaluate preferred technology representation mediums for nonvisual understanding of children’s artwork. Through these studies, researchers found that BLV family members are motivated by a desire to emotionally connect and bond with their child, as well as a need to understand their child’s developmental growth and skills. While many BLV individuals found some benefits to AI descriptions, saying that some information was better than none and it enabled them to have a conversation or ask questions to their children, some parents did not like it when the AI used simplistic or reductive language to explain their child’s art. These findings specifically around the role and language of AI have heavily informed this present work.

Methodology and Evaluation

Project Design

We designed an iOS app that uses ChatGPT to generate detailed descriptions from pictures of children’s artwork. The app includes essential accessibility features such as voiceover compatibility, high-contrast visual settings, and an intuitive navigation system to ensure ease of use for BLV users. The design process was iterative, involving multiple rounds of feedback from BLV parents and their children to ensure the app met their specific needs.

Implementation

Our major contributions in the final project include:

Salient Features of the iOS app

Metrics and Validation

These are some metrics for success that we considered:

Success Criteria for AI Descriptions

We evaluated the AI descriptions of 10 different artworks created by children using GPT-4o and our custom GPT (ArtInsight). We measured the number of elements identified, usage of reductive language, simplistic descriptions, and presumptive statements.

Performance Comparison:

These metrics and validation steps ensured that our app met the necessary accessibility standards and provided meaningful, accurate descriptions of children’s artwork.

These are some other metrics for success (Out of Scope):

A series of screens indicating different states of the Children's Artwork app, from left to right. The first screen (far left) shows a sign in screen to access the app, where you can use your email or Gmail. The second screen shows the option to upload an image. The third screen shows the Upload flow triggering a photo gallery, the following screens are organized as two stacked screens where the top one shows the photo gallery from the camera roll, and the other shows the uploaded photo. The next two screens to the right first show a confirmation screen to use the selected photo, and then show a child's artwork of a pink flower with its AI-generated description along with 'Save' and 'Edit' buttons. The last screen (far right) shows how you can use the 'Edit' button to change a part of the AI description, which is highlighted in yellow. Figure 2: Different UI screens indicating different states of the Children’s artwork app

Two screenshots of the app interface screens. The first screen has a white progress spinner with the label: 'Waiting for description'. Behind the spinner is a photo of a child's colorful drawing of a family standing in front of a house and tree with a sun in the sky. Above the photo and progress spinner is a 'Cancel' button.The showing an artwork description screen. The description reads 'A colorful drawing of a house with a sun and flowers.' The interface includes options for voiceover and high-contrast settings. The second screen has a ‘ Cancel’ button in the top-left corner and a ‘Save’ button in the top-right corner. Beneath is a photo of a child's colorful drawing of a family standing in front of a house and tree with a sun in the sky. Below the photo is a ‘Description’ label and an editable text box with a textual description of the image. At the bottom of the screen is a button to ‘Record a voice memo’. Figure 3: Screenshots of the app interface showing screens to capture an artwork and show the AI-generated description

Results

The project has made significant strides in developing a tool that addresses a critical need for BLV parents, enabling them to engage more deeply with their children’s artwork. The initial technical achievements and validation of core accessibility features provide a solid foundation for further refinement and user testing.

Technical Achievements

Disability Model Analysis

Leadership of Those Most Impacted

Our project is guided by findings from formative research consisting of interviews with BLV family members and their sighted children. As such, our project places the perspective of BLV relatives at the center of our work.

Recognizing Wholeness

While there is a large body of valuable research dedicated to making productivity tools more accessible to people with visual impairments, it is important to recognize that all of our lives encompass so much more than work. With that in mind, this project focuses on aspects of life outside of work and productivity.

Interdependence

The formative work for this project observed interactions between both the BLV relative and their sighted children, recognizing that artwork engagement in families is an interdependent practice where the relative and child work in support of each other.

Learnings and Future Work

Learnings

Through this project, we learned the importance of involving end-users in the design process. Feedback from BLV parents and their children was invaluable in shaping the app’s features and ensuring its relevance. We learnt how to build an accessible iOS app while integrating GPT APIs. We defined various success metrics for validating our work, and also recognized the potential of AI in generating meaningful descriptions. However, we noted the necessity of human oversight to ensure accuracy and personal connection.

Future Work

Future work will focus on expanding the app’s capabilities to include more interactive features, such as voice memos from children describing their artwork. We also plan to explore integrations with other accessibility tools and conduct long-term studies to assess the app’s impact on parent-child relationships. Additionally, we aim to refine the prompts to improve the accuracy and emotional depth of the descriptions.

We will focus primarily on implementing the human-in-the-loop system, conducting extensive user testing, and continuously improving the app based on user feedback. This iterative process will ensure that the app remains relevant, effective, and truly beneficial for BLV parents and their children. Specific next steps include:

References

Park, S., Cassidy, C.T., & Branham, S.M. (2023). “It’s All About the Pictures:” Understanding How Parents/Guardians With Visual Impairments Co-Read With Their Child(ren). Proceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility.

Storer, K.M., & Branham, S.M. (2019). “That’s the Way Sighted People Do It”: What Blind Parents Can Teach Technology Designers About Co-Reading with Children. Proceedings of the 2019 on Designing Interactive Systems Conference.

Anonymous Authors, “Engaging with Children’s Artwork in Mixed Visual Ability Families,” in Proceedings of the ACM SIGACCESS, 2024.

Arnavi Chheda-Kothary, Melanie Kneitmix, and Anukriti Kumar