Focus Group with Blind/Low Vision Testers

Timeframe

April 2017

Evaluation Goals

  • How do the crowdsourced recordings affect blind or low vision testers’ experience with an object/exhibition/experience? Do they find the audio to be engaging?
  • To what degree did the crowdsourced recordings help blind or low vision testers feel more connected with object/exhibition/experience?

Methodology

Jessica Swanson served as facilitator and hosted a focus group of blind or low vision testers following a performance of Chicago at the Kennedy Center for the Performing Arts. The testers ranged in age from 17 to 66+ and there were 5 women and 4 men.  Testers were asked to listen to three separate audio recordings, each a sample clip of visitors describing an object (a chandelier) from the Peabody Essex Museum’s American Art galleries.  After playing each recording, the focus group facilitator asked these follow-up questions:

 

1) On a scale of 1 to 4, how engaging did you find this crowdsourced description?

 

  1. I was so engaged that I’d listen to crowdsourced description similar to this again.
  2. I was engaged, but I wouldn’t use crowdsourced description similar to this again.
  3. I wasn’t really engaged.
  4. I didn’t find it engaging at all.

Follow-up question:

Which part of the description did you find most engaging?

Which part of the description did you find least engaging?

2) To what degree did accessing this information help you engage with or feel more connected with the [object/exhibit/experience]? [I will read through all options first and then get a “by show of hands” count.]

Audio Recording Transcriptions

Audio Clip 1

This goes on the ceiling, it gives light. The light it’s giving that I think is electric but it could have been with candles or with gas in its time because I see that it was between 1850 and 1855, but the person who created was…I think it was sculpted because it has some metal and I think it might have some wooden parts. No it says no, gilded bronze and glass. So it’s sculpted. But the bronze is not like the statue of liberty—green bronze—it’s all still…it might have been painted. It’s not black and it’s not green. It looks like gold and it’s even shiny, so it looks like it’s either taken care of…but it’s indescribable because it’s lacy, delicate, yet it’s bronze so it is strong…WOW…it’s beautiful isn’t it? And the artist…I don’t know who that is…Henry Hooper and Company. It must have stood in a beautiful mansion, must have been on the ceiling of a main room…and these are definitely gas lamps…look at them, the structure…they are from the same house.

Audio Clip 2

Um, hmm…it’s a chandelier. But throw out any ideas you ever had about what a chandelier looks like. This one got dragged through the forest when it was being created. It has loaded with all sorts of, um, vegetation and little ‘putie’ angels. Uh, and is it a gasolier? Or I don’t know if it’s a lamp or a gas light because there are, uh, chimneys around the illumination part. But it’s very baroque in design. Um, rather over the top.

Audio Clip 3

Very ornate chandelier…sort of a floral motif with small cherubs or children decorating it in a gold or a brass colored finish, and the cherubs are more… look like traditional bronze. It may have once been candle fired or gas, but it appears to have been, perhaps, electrified, with sort of frosted glass globes around where the candles or the gas might have been.

Key Findings

Note from Jessica: We did have one person who had to leave so there are 9 replies to the first clip and 8 for the other 2.  The testers ranged in age from 17 to 66+ and there were 5 women and 4 men.  

Audio Clip 1

Follow up questions:

1) On a scale of 1 to 4, how engaging did you find this crowdsourced description? [I will read through all options first and then get a “by show of hands” count.]

□ I was so engaged that I’d listlen to crowdsourced description similar to this again. – 2

□ I was engaged, but I wouldn’t use crowdsourced description similar to this again. – 3

□ I wasn’t really engaged. – 4

□ I didn’t find it engaging at al.  – 0

Follow-up question:

Which part of the description did you find most engaging?

>> I like when she was trying to puzzle out —

>> It was more fun trying to figure out what she was talking about.

>> Like guessing.

>> Yeah.

>> The description of the bronze was pretty nice.

>> I like that she said the date and all that, because I can never read the date by the painting, so I dig that information. When it was made. The description of it, like, It might have been gas or whatever in its time. So those details were cool. (Reading the label)

 

Which part of the description did you find least engaging?

 

>> When she was fumbling, like, Well, I don’t know, this might have been this or might have been that. If you’re going to tell me, don’t — don’t put me through all that.

(Laughter.)

>> So yeah.

>> Maybe think it through before you start the file.

(Laughter.)

>> That kind of thing is most helpful when you know what’s going to come out of your mouth.

I already don’t want to be in a museum probably. I probably don’t want to hear you say maybe it was this, maybe it was that, oh, I see a sign. Maybe look for the sign before you turn on the app.

>> Absolutely.

(Laughter.)

 

2) To what degree did accessing this information help you engage with or feel more connected with the [object/exhibit/experience]? [I will read through all options first and then get a “by show of hands” count.]

 

□ Not at all – 3                

□ Somewhat – 4

1 in between              

□ Very much

 

Follow-up question:

Why or why not? Can you tell me a bit more about that?

>> It’s kind of the difference between looking at something with a friend and having the official description.

>> Absolutely.

>> Like, You’re now looking at a 19th century chandelier that’s from the period of —

(Laughter.)

>> I like that more. Particularly with a British accent like you just did.

(Laughter.)

>> But I kind of like the puzzling it out together approach. It’s fun. But I’m kind of weird.

>> Especially if you’re there by yourself and you’re sight-impaired.  That would be a fun way to go! (Laughter.)

Audio Clip 2

Follow up questions:

1) On a scale of 1 to 4, how engaging did you find this crowdsourced description? [I will read through all options first and then get a “by show of hands” count.]

 

□ I was so engaged that I’d listen to crowdsourced description similar to this again._2_

□ I was engaged, but I wouldn’t use crowdsourced description similar to this again._2_

□ I wasn’t really engaged._2_

□ I didn’t find it engaging at all. _3_

 

Follow-up question:

Which part of the description did you find most engaging?

Which part of the description did you find least engaging?

 

>> I liked the commentary, like, it looked like it had been dragged through the forest when they made it. I’ll hold the rest for the not so much.

And he seemed to describe — the other gal didn’t describe the angels or anything. So it went into more detail.

>> I thought it was a much better description. She didn’t say anything — was it the same chandelier?  Do you know?

>> Jessica: It was the same.

>> Okay. She didn’t say anything about the chimneys, vegetation, or angels. Those are big things. I thought it was much better throughout. I thought the hmmm at the beginning was really funny.

(Laughter.)

Like, hmmm, I don’t know what I’m looking at, but here you go. It was much better throughout. I didn’t have to sit there while he wrestled with his own brain about what he was seeing. So I thought it was better.

>> It was a little more technical. He threw in more technical terms there about the actual illumination part. It was interesting.

>> It was. But I’ll be honest. That’s where I got lost. It wasn’t as whimsical anymore. You tell me it’s baroque. I know there was a baroque period. But that doesn’t resonate with me. I don’t think, okay, that’s what it looks like then. So when he used technical terms, I couldn’t envision what he was talking about.

>> Jessica: So that’s when it was least engaging?

So what made it least or less engaging?  Just the technical terms?

>> For me. But he liked that. So I think we go back to that, do I want description during a musical or not?  So much of it is preference.

>> It sounded like they were talking about two different things.

>> That’s why I thought it was a different chandelier.

>> I think it’s going to be like Wikipedia where you don’t know what you’re going to get, but if you’re going alone, you might know more than what’s.

>> Part of crowdsourcing is you get different information, different approaches, so it’s an opportunity to look at something through different eyes. Some are more technical. Some are more artistic.

>> You should be able to upvote the commenters you like the best.

>> Yeah. Thumbs up. Thumbs down.

>> Maybe five stars!  Or, it was hysterical!

>> Or when people finish the description, they should tag it as more technical or more whimsical. Seriously!  Because I would tend to go to the more technical and Brian would go to the more whimsical or whatever you call it.

>> That’s a good idea.

>> So if you have somebody who does art history, and goes to a museum and wants to do this, that’s great for someone who actually wants to know what it looks like, what the terminology is. So they should tag themselves as more technical.

>> Educational, technical. Yup. That’s a good idea.

>> British accent.

(Laughter.)

 

 2) To what degree did accessing this information help you engage with or feel more connected with the [object/exhibit/experience]? [I will read through all options first and then get a “by show of hands” count.]

 

□ Not at all _4_               

□ Somewhat __

2 in between              

□ Very much_1_

 

AUDIO CLIP 3

Follow up questions:

1) On a scale of 1 to 4, how engaging did you find this crowdsourced description? [I will read through all options first and then get a “by show of hands” count.]

 

□ I was so engaged that I’d listen to crowdsourced description similar to this again._8_

□ I was engaged, but I wouldn’t use crowdsourced description similar to this again.__

□ I wasn’t really engaged.__

□ I didn’t find it engaging at all. __

 

Follow-up question:

Which part of the description did you find most engaging?

Which part of the description did you find least engaging?

>> I could even see, and I was engaged.

(Laughter.)

I haven’t been voting up until now.

>> It was very descriptive.

>> Detail-oriented.

>> It’s interesting because, well, baroque does mean to be ornate. It was a characteristic of the baroque period. But because he said ornate instead of baroque, people found it resonated more.

>> Even though it was very descriptive, and so was the other man, none of them mentioned dates except the lady [first audio clip]. So what am I missing if they didn’t tell me the dates?

>> So maybe for folks who will be describing, there could be a quick job-aid bullet list to think about these things during the description.

>> Jessica: For instance, to read the label…

>> Yeah.  So there’s some sort of standard. We don’t want it to be too standardized, because look how fun and different they were. But if you could at least give them some idea about how to be thorough, that would be good.

>> He got a lot of description in a very short period.

>> He did.

>> It was great.

>> He thought about his before he turned his app on!

 

 2) To what degree did accessing this information help you engage with or feel more connected with the [object/exhibit/experience]? [I will read through all options first and then get a “by show of hands” count.]

 

□ Not at all __               

□ Somewhat __

□ Very much_8_

 

General Comments about App Usability for People with Vision Loss

  • Speaking to the technology bit, since that’s what you’re doing with this, I loved having my own phone and my own earbuds. I really appreciate you — being able to access the audio description that way as opposed to having a separate device, separate phones, not being sure what to hit.
  • Have a tutorial built into the opening screen – a getting started guide, so it will walk you step by step, like here’s the demo thing. Here’s what you do. I think it would be helpful, because it would instruct people in context rather than reading this 20 step thing, and before you even open the app, people think, I don’t know, it’s going to be really complicated. So I think that would help the mindset a little bit.

06 Access App Summative Evaluation_Aug 2017