Access App Summative Evaluation


June-August 2017

Executive Summary

Audience Focus conducted a summative study for the Access App development process with sighted and blind or low vision visitors. The evaluation focused on the effect of the App on visitors’ experience, their response to the crowdsourced content, and their likelihood of contributing content to the App. The Access App tested in this study was a beta version with limited crowdsourced content and uneven audio quality.

General visitors were invited to use the App with a selection of objects in the Peabody Essex Museum American art gallery. Evaluators observed how the visitor used the app and then conducted a semi-structured open-end interview with the visitor. Demographic and psychographic data were collected after the interview with a brief survey. In addition, focus groups were held at both the Peabody Essex Museum and the John F. Kennedy Center for the Performing Arts. The focus groups consisted of blind or low vision participants, many accompanied by their sighted partners. These visitors listened to the App audio in front of the artworks in the gallery and then moved to a conference room to discuss the experience.

Project partners from the John F. Kennedy Center for Performing Arts and Plimoth Plantation also collected a small amount of observations and interviews. While the sample sizes are too small to identify any patterns or trends or to allow for any conclusions, the findings are reported.

How the App affects visitors’ experience
  • Time Spent: Visitors spent an average of less than two minutes interacting with the App. During the interview and focus group conversations visitors noted that they likely spent more time listening to the App than they would have if they it on their own.
  • Aspects that Enhanced Visitors’ Experience:
    • Deepened Experience: Visitors appreciated that the App caused them to pay more attention to the work of art, attending to details they might have missed
    • Varied Perspectives: Visitors enjoyed hearing speaker’s different points of view when they were not repetitious.
    • Universal Design: Blind or low vision visitors like the on-demand, universally designed aspect of the App as they do not always want to schedule a special tour on a particular day.
  • What Troubled Visitors:
    • Lack of or Inaccurate Information: Most visitors felt that critical information about the artworks was missing and were bothered when they heard inaccurate or highly subjective comments.
    • Background Music Distracting or Poor Audio Quality: Many visitors mentioned issues with the background music. Some found it was too loud and competed with the crowdsourced content while others felt any music should enhance the specific object directly.
    • Randomness of Audio Content: Visitors often were confused by the different crowdsourced descriptions woven together in what felt like a random way.
  • Who the App is For: There were no strong patterns in the data on who visitors thought the App was for.
  • How Visitors Respond to Crowdsourced Approach: Most visitors found the crowdsourced content troublesome or confusing. The content of the information was considered uneven and most visitors wanted to hear from knowledgeable experts about the work. Blind or low vision visitors particularly wanted more robust visual description, similar to what they are used to getting in other settings. They recommended having professionals who are experienced in visual description.
  • Likelihood that Visitors will Contribute Content to the App: Most sighted visitors were unlikely to contibute content to the app. They admitted to rarely contributing content online in most any form.
  • Interface Design Issue: The most frequent interface suggestion was to create better menus, guides, or some sort of filters that gave visitors more choice and control. In addition, visitors wanted better production quality.
Implications & Recommendations
  • Clarify the purpose of the App for users.
  • Re-think the reliance on crowdsourced content and consider a mix of scripted and visitor-contributed content.
  • Differentiate between content experts as well as crowdsourced visitor contributions.
  • Improve production quality of audio and interface controls that allow users to know how many entries are available, how long each entry is, and how they can move through and select the content.

Access App Summative Evaluation_Aug 2017