W4A2016 Session 7: Accessible Learning, Web-based Testing and Assessment

W4A2016 Session 7: Accessible Learning, Web-based Testing and Assessment


– Did my job. So, the next speaker will be Erin Buehler from the University of Maryland. – Hi, sorry for all of our
exciting technical difficulties. So, as was mentioned,
my name is Erin Buehler. I am a PhD student at the University of Maryland, Baltimore County. I’m actually pretty close
to being a PhD candidate, so keep your fingers crossed
for me, couple months here. And I wanted to mention, this is my first time with W4A and I’m very excited to be here. After dinosaurs and science-fiction films, probably my two favorite
topics to discuss are education and accessibility. So, I’m pretty delighted
to be able to attend. And I am presenting this paper, Accessibility Barriers in Online Education for Young Adults with
Intellectual Disabilities We did not win longest
title, but we tried. And I’m presenting on behalf of myself, my colleagues, William
Easley and Amy Poole, who are also grad students at UMBC in Human-Centered Computing program and of course, my advisor, Dr. Amy Hurst and we’re all part of the UMBC Interactive Systems Research Center. When to pay attention to my talk, you’re welcome to nod off
for the first 7-10 minutes and then I’ll actually
discuss my findings, so go check your email,
whatever you wanna do. I’m gonna talk a little bit
about intellectual disability, the motivation, the methods,
the context of our study and then dive into the– (cough) the findings and my
delightful recommendations. So, I noticed a lot of
folks at this conference maybe are more versed in
perhaps visual impairments, not necessarily cognitive impairments or intellectual disabilities,
so I wanted to take just a minute to talk about
what intellectual disability is and what kinds of support students with intellectual
disability actually need. So, intellectual disability is a type of developmental disability. As the name suggests, it is a disability that develops during childhood rather than onset later in life. And within intellectual disability, underneath this higher umbrella
of developmental disability, there’s actually a wide array of diagnoses that qualify as intellectual disability. Probably the most
familiar is Down syndrome, but you can also have persons
with fetal alcohol syndrome, Fragile X syndrome,
and a few other things. But the two factors that are
important to keep in mind when looking at support for these students are the impacts that
intellectual disability has on intellectual functioning
and adaptive behavior. I’m gonna take a drink
from my sippy cup here. So, intellectual
functioning is essentially reasoning and learning skills, things like scheduling,
planning, strategizing. And then adaptive behavior has to do with social and practical skills, understanding etiquette,
understanding exchanges in everyday life, things of that nature. So, are we all square on
intellectual disability? I got a couple of nods, I’ll take that. I’m also lucky to be presenting
later in the conference because I feel like the
motivation is taken care for me like our past presenters,
our wonderful keynote, have all said education is
becoming technologically-driven, it’s really important for
persons with disabilities to have access to education, to have accessible technologies
in the context of education, and then we have this messy
beast that is the internet where some people follow the
rules and some people don’t and we have so much information,
how do we filter it? So, I’m really interested in how students with intellectual disability
get access to education and education through technology tools. I also think it’s important
to support diverse learners because I’m personally afraid
of this dystopian future where we standardize learning and exclude a bunch of
people early on, so. The brave new world of
education, watch out. Nobody laughed, it’s okay. (laughter) I also care a lot about employment for persons with disabilities and if you wanna have a
really long irate conversation with me later over some beer, I’ll talk about the difference between gainful employment and
meaningful employment in terms of self-determination. So, my methods. Let me take a moment and apologize to the people who reviewed my paper, you were like “What are you doing? “Why are you appropriating
methodologies from anthropology? “What is that?” But they still let me come, thanks. So, if you are not big on anthropology or you don’t know about
the origin of Greek words, in emic ethnography, the word emic just implies from within. So, myself, my colleague
William, Amy Poole, we all worked really
closely students who have intellectual disabilities at my school. We work as teachers,
we work as assistants, and we have a lot of one-on-one
activity with these folks. So, we would be lying to ourselves, if we said we are anything,
but within this culture and so, we just wanna be upfront about it and say it’s emic and
it’s more of ethnography than a qualitative study,
than a usability testing. We just hung out and we thought. And they’re letting me
present that as science. And some of the methods, just personal observation,
journaling by the students, looking at artifacts, and
collecting a bunch of data over the course of a couple of semesters. The context for this study
is post-secondary education. So, not elementary
school, not middle school, not high school, but post-high school. In the United States and Canada, there are at least 240
post-secondary programs for young adults with
intellectual disabilities and those programs can be vocational, they can be certificate programs, they can be integrated
at a college campus, they can be segregated
at a college campus. They can be a lot of things, but primarily the goal is
independence and employability and I’m lucky to work at a
school that has such a program. So, I have access to students with intellectual
disabilities all the time. And in the paper, we talk
through two specific contexts. The first one is an inclusive
3D printing classroom and the second one are student internships with students with
intellectual disabilities. More water. So, in the inclusive classroom, I have half students with
intellectual disabilities and half neurotypical undergraduates from various degree programs. Students work together
in a workshop-type space of which I have a picture
of up on my slide right now. Students are gathered
around a big worktable with markers and crayons
and paper and laptops and they’re working through design ideas for their 3D printed objects. And the premise of the
class is that it functions as like an entrepreneurship minor. So, the students create an
outward facing print shop for the campus where people can come and ask for 3D modeling
and 3D printing services. And within the classroom,
the students work in pairs. One student with an
intellectual disability and one student without
and the idea is that throughout the semester, as
they’re learning 3D modeling, as they’re learning
about entrepreneurship, they gain expertise in
all these different areas, work together, teach other. Lots of motivation for
this, but I can explain some other time, education folks, you know what I’m talking about. So, this picture that I
have up right now is just a couple of groups of
students huddled around a computer screen, they work on Macs. And what you can’t see is the tears while they try to figure out why their 3D prints aren’t printing. And then that secondary setting, one-on-one job coaching for interns. Our research lab hires
an intern every semester who works on everything from 3D printing to digitizing documents
for our research efforts. All kinds of stuff, very
hands-on, very interactive. And those students
usually have a job coach who works on them in a
one-to-one ratio every day. In the beginning, they help with training, and then as the student transitions to being more independent,
they become more supervisory, reminders, prompts, that type of thing. And here’s a picture
of two students working up on the screen right now. Again, just sort of at the computer with the student with intellectual
disability driving the mouse. Okay, so, we had a time deciding how to present the findings from this because it sounds a little loosey-goosey. Like we did observations,
we have the classroom, we have the student interns. We spent weeks whiteboarding
this and thinking, “How do we present this? “What’s the right framework?” And what we landed on was talking about this idea of meta skills. So, a metaskill is a combination
of fundamental skills like memory recall, typing, vocabulary, combined with the task
necessary to complete some kind of assignment
or some kind of goal in an education setting. We came up with about four
that we wanted to discuss that were kind of problematic. I’m gonna step through and
then I’m gonna talk about what worked well for us
in mitigating these issues and what we think the next steps are. The first of the four, we
called information retrieval. You can imagine the classroom. You have assignments
were you have to go out into the scary world of the
internet and find information. You need to know where
to look for information, how to locate it, how to validate it, and then return it back in
the form of an assignment or a write-up or a presentation and our students with
intellectual disability struggle through this sometimes. Lots of different factors
between their own abilities and the interface and the
nature of the internet. The internet impacted this interaction. One thing that we did in
the classroom each week was the students had to find
article relevant to 3D printing and then, digest it, write a summary, and spit it back out to us. And the students had a rough time picking the right keywords, using filters and search engines. And then, also differentiating
between advertisements, click here to read more, and “Oh, you were
interested in 3D printing, “would you also like to know 15 tips “for weight loss strategies?” You know, the internet. Next up, navigation
information and architecture. This had more to do with
the structure of the webpage and being able to understand
hierarchical menus, being able to navigate
and understanding icons, had some difficulty with students with intellectual disability, in terms of really abstract icons and then more concrete ones. For anyone who’s seen
the finder icon for Mac, it’s the like home base
for getting all your files, it’s a smiley face, kinda like “I’m really happy about files.” Files don’t look like smiley
faces, you guys, what is that? So, students struggled here. And of course, in the classroom, lots of different places
where this might come up, in a university portal,
on an online website, global resources and all that jazz. Okay, more water, and
then file management. File management is important
in digital education. We need to be able to get documents, access things for course
materials, submit assignments. You might submit assignments via email, you might submit them to
a portal like Blackboard or to some kind of other delivery system. You might have to maintain a blog. So, being able to understand files, file extensions, where they are, where they get saved to, where you’re keeping ’em, also important. We had two kind of
high-level problems with this for our students with ID, both the interns and in the classroom. Understanding where files that
are downloaded locally end up which, to be honest, confuses
me a lot of the time, like why isn’t it downloads? Why did it go to desktop? Why is it in this folder
labelled Funfetti? Oh, who knows. So, identifying where things
are downloaded locally and then also abstract concepts like permissions and sharing. So, in the classroom, when
the students are working with their partners, they have a lot of shared documentation, shared project files and we uploaded those to Google Drive and for the students with
intellectual disability, it was hard to conceptualize
that when they access that shared folder, they’re
making changes to content that their partner also has access to. So, we also talked a
little bit, in the paper, about password management. Here’s how I feel. Passwords are hard, we get it. There’s a whole field
of science with that. If you have online education, odds are you have some kind
of an account-driven service. So that means more login credentials, more passwords, access issues, logouts, redundant logins after
trying to reset passwords. It was a super mess and
this can be a super mess for anybody, it was just a
little bit more frustrating for our students because
they struggle, sometimes, with memory recall and also,
interfaces across different services that require a
login are sometimes different and it was confusing,
a little disorienting. If a login interface changed, sometimes students would get locked out, typing in caps versus not caps, some stuff that was
covered in yesterdays paper about how much it was fielding. So, what’s good? My favorite solution, even
though I’m a computer scientist is a solution where you
don’t actually need to make a new technology to solve a problem. In some cases, the
problems that we identified were actually things that you can solve through education and training. And education and training for the students with disabilities, but also, the kindly folks
who make these interfaces and can design web portals and platforms. So, issues of file naming, for example. The intellectually disabled students sometimes didn’t know that they should give a file a name that was
conducive to find it later and so, they would just name it whatever and they would download it and then we would struggle
to find it later on when they needed that file. Nobody had explained
that it was a good idea to give stuff a name for future action sort of those scheduling
and printing tasks that these students need more support for and just nobody told
them, that’s a good idea. Easy fix, doesn’t cost anything. It cost a little bit of time, let’s not talk about
how much time is worth. Other things, there are
some existing considerations that are out there that would
actually work really well to mitigate some of these problems. So, just customized
dictionaries to catch typing and vocabulary errors or to help address issues of literacy levels. Within intellectual disability, you have a really wide range of ability. You have a really independent folks who don’t need a lot of support. You have some folks who are very dependent on a lot of support and in
between, there’s so much room for customization of existing
stuff that we already do to catch like, everytime
I type entrepreneurship into Google, I spell it wrong
and Google helps me out. Entrepreneurship has so
many e’s and u’s in it. Okay, so we talked a lot
more about that in the paper. I got 20 minutes, I’d
love to talk forever, but other people wanna go
and we started late, so. Ideas that we think
would be really valuable for ourselves and other
researchers to look into that might help address a lot of these obstacles, improving usability. So, a lot of the issues that we had were related to this navigation structure, things that are hidden or obscured in favor of looking more
aesthetically pleasing and if we could use augmentation
or adapted interfaces to kind of highlight things
in a context-aware nature that would probably help out some of these students a great deal. Another thing we thought was interesting was maybe exploring an
affect of computing. So, for our students,
there’s a little bit higher of a chance that you’ll experience some real upsets and emotional frustration. The second or third time that you try and login to your account
and you get locked out, it’s rough and derails
class and it wastes time. So, when tasks are not getting completed, when they sense that the
students are getting frustrated, even just alerting a teacher or a peer to come over and say, “Hey, what’s up? “Do you need some help?”
would be fantastic. And sometimes the teacher doesn’t have the resources to be watching everybody who’s trying to login to get stuff going. (mumbles) And a little bit more water. We are super strong advocates of including persons with disabilities and
designing their own solutions and so, something that I
really wanted to reiterate that we’ve actually
already heard a little bit, like in our keynote talk,
is involving these students with intellectual disabilities in research and encouraging them to help with development, to become developers. These students have students
have so many opinions on what can help them,
what kinda needs they have, what their purposes are
and they feel like there’s a little bit of intimidation factor, if you’re not used to working with persons with intellectual disability
or with cognitive impairments. Ask them, bring them into the loop, they have so much to say and they’re fantastic designers. Now, in my second semester of teaching this 3D design course and they’re great. So, it’s just something that
folks know is out there, but I’m saying extra do that thing. You’re champs, this is the summary slide, pat on your back, you didn’t fall asleep. I seen you making eye contact with me, you listened the whole time. – [Voiceover] Thank you. – So, the summary slide,
what I talked through. I presented this
appropriated anthropological concept of an emic ethnography that I probably did really poorly. If you know a lot about anthropology, like wait to slap me until afterward. I talked through the
different types of information that we have pulled from,
how we decided to present it, those four meta skills
that were kind of issues and then talked through some
stuff that already exists as a good solution, offered scaffolding using tools that we already have and reappropriating them. And then there’s a few advanced solutions that might really help. This is the point where I thank all of the miraculous funding that gets me through my
graduate school life. The NSF for the research, the Alexander Brown Center
for Entrepreneurship at UMBC for funding my
course that I designed, and then of course, Canvas and Intuit for funding my student
travel grant to come here. And I’ll listen to any questions and probably not answer them well. (clapping) – We probably have time for one question, we’re way behind, sorry. – That wasn’t my fault. – Well, if there are no
questions, I have one. Just a quick one, you
sort of rolled up to it. Meaningful employment, maybe just a quick, brief summary of what you… – I think that there is
a perception or a goal that procedural work is the end-all for a person with intellectual disability because they can complete steps that are given to them
in a procedural format. You know, complete this and
then follow this next step and follow this next step. There’s this perception
that that’s the ceiling for what these individuals can do and absolutely not the case
and it’s really restrictive for their ability to be
creative and to contribute. And so, I feel like there’s
a big difference between “Oh, well we can make
a system that gives you “the ability to do a vocational build “which if that’s what you wanna do “or think you really like vocational “skills or building, fantastic,” but for the students who are creative and who are really interested
in becoming designers, there should be more options for them. So, I just have a lot
of feelings about that. – Thank you, Erin. Maybe you can see Erin
afterward, if you have questions. – I’ll come find you all. (clapping) – Next up is Luz Rello from
Carnegie Mellon University. She’s been kind enough to join her two presentations into one, so if you could your questions to the second presentation from her, save a bit of time that way. – Thank you, Peter. So, yes, two presentations in one. So, first, I’m going to
start with the first one, with an online game designed
for people with dyslexia. This is joint work with Sergi Subirats from University Pompeu Fabra and Jeff Bigham from Carnegie University. He’s not here. I think he had to run – [Voiceover] Yes, back to– – Yeah. (mumbles) So, dyslexia, you have heard already some things about dyslexia,
so I will go very fast. Dyslexia is frequent, it’s
about 10% of the population. Depending on the language, you have different manifestations, at least about 10% of the population. It’s universal. I mean, it has biological
basis in the brains of people with dyslexia. And it causes school failure. Right now, in Spain, I’m citing a news presentation, 40% of the school failures
are students with dyslexia. So, what happens? Why does it cause conflict? So, what happens at
school is that normally, when you have to learn
something, you have to read, and when you have to prove
that you know something, you have to write it. Dyslexia, since dyslexia
affects reading and writing, you have a problem when you’re at school. So that is why it causes school failure. And normally, all the exercises and all the treatments for dyslexia, it are related to reading and writing. So, if you have dyslexia and you have trouble reading and writing, you have plus do a lot of
exercises with reading and writing which is super, super, boring. So, it was like, “Hey,
why don’t we do chess?” We could support dyslexia or we can do something without
using reading and writing, so let’s try something
different and see what happens. And then, if we go deeper
into what Chess involves, we know that that also involves
visual-spatial abilities, it involves kind of a
calculation abilities and also executive functions
like attention, planning, and all these abilities are
actually related to dyslexia, we say, “Let’s see what’s going on.” Plus, our second author, Sergi Subirats, as you can see in there in the picture, he’s actually a professional chess player. He’s among the 5% top in
the world like chess player and he was my undergraduate student and “Hey, you know,
you’re so good in chess, “I mean, why don’t you
do something for chess, “related to dyslexia.” And then we were like, “So, yes, “learning chess is fine, “but let’s put it in the form of a game.” So, games and dyslexia, you may have noticed is a
hot topic at this moment. There is even research being done using action games to support dyslexia. So, it has been proven, for example, this is research that was
done in Italy in Padua where they measured children
playing action games and they found out that
after playing action games, before they were improving, what’s going on? – Sorry, it does that, just– – Ah, okay. Because they improved
their attention abilities, they found out that
like a secondary effect of improving their attention abilities, they were improving
their reading abilities. We say, “Let’s mix chess, computer game “and see what happens.” And then we go through literature, and of course, yes, people have thought about this before, as always. So, but not with dyslexia. So, there’s a brain training study that was done with 80, sorry,
with 92 teenagers in Zaire and all these studies are longitudinal studies like long studies. And they were measuring
different spatial abilities, pro-social abilities, reasoning creative, general intelligence. And they divided students in two groups. One group were taking chess lessons, another group were not
taking chess lessons and at the end of the
study, they found out, very important, that chess
improve their numerical aptitude, yes, aptitude, and their verbal ability. So, it was like “Ha! It
can preserve our ability.” But then we see other studies and there was another
study done in Belgium with 20 fifth grade students and no results were found, very unfortunately, although, they also measured regular ability, only there were some effects
on the scores the students had and there was another
study done in Brooklyn in the USA and again, no
differences were found between the group who were playing chess and the group who were not playing chess. So that is more or less
the state (mumbling) So, then Sergi decided to create a game, so he made nine lessons. He followed the World Chess Federation, let’s say, ways of learning chess which is actually the standard
of how to learn chess. And he designed nine lessons and in each of the lessons,
you could learn how to and practice how to move the different chess pieces. They are called pieces,
he insisted, it’s pieces. It’s not other words that people from outside chess were calling it. So, there you got it. So, pawns, are you familiar with chess? I don’t have to explain this, no? Pawns, bishop, rook, queen,
king, knights, you all know. I have no idea about chess. That is another thing. That is another thing why,
again, we decided to choose this because I mean, I know so
many people with dyslexia that kind of play chess, you know? Maybe there is something there. The structure of each
lesson is the following. So, first there’s a theory
part where they explain how to move each of the pieces. Then you have exercises to
learn how to learn this. And then you have like a playing part. Each lesson has a challenge
where you have to play. And here you have some screen, I think I know why it does that. And you have some screenshots over there. The methodology. We implemented this, we
recruited 64 participants, 32 of them with diagnosed dyslexia because attention-deficit
disorder has an impact on how people interact with the computer especially with the mouse
and we were actually doing mouse tracking. We made sure that none of the participants had attention-deficit disorder and we also were very careful. As you can see here in these slides that chess knowledge were mapped among the two groups of the participants because that could have an effect on the data that we were gathering. We measured their performance using objective measures and
subjective measures. The objective measures. Do you remember that
each lesson was divided into three parts, theory,
exercise, and playing. Here you have the objective
measures for theory. We measured time that they had spend, the accuracy of the answers
that they had in the exercises. We also measured the time, we measured the number of squares that the mouse goes over because in chess, this is a performance measure. And the number of tries for the exercises and for instance, time they
had spent to the first try. So, the time that they spent thinking before they moved the mouse. And how many times the first answer they gave to the questions
was the correct ones. And then in the playing
when they have to… to play, to exercise, how do you say, (mumbling to self) in playing moment like in
the last part of the lesson. The time that they spent,
the number of movements, the time to the first
movement, and the score that they have at the end. So, each of the lessons have like a score. As objective measures, we used indicator scales, how do you feel? I mean, how was your
performance using Likert scales and for the people who had dyslexia at the very end of the
study, so this didn’t have an effect on how do they feel. We asked them, “Do you think that dyslexia “has an impact in your performance?” And we also used Likert
scales for measuring this. So, our results. They could be more exciting. So, yeah, they could be more exciting. We hardly found significant differences. Something’s happened. So, we found some differences
and always in times. So, they spend more time in theory. They spend more time doing the exercises. They spend more time playing the game. And in the only performance measure that we found differences were
in the number of their scores that they went through with their mouse. Maybe they were counting,
we don’t know exactly what. Most of the participants
thought that dyslexia didn’t have an impact
on their performance. So, why, I mean, discussion. Why this happened? So, when we designed the theory, I mean, the part where people learned how to play chess, we used frequent words. I mean, we designed it in a way that it shouldn’t had an
impact on people with dyslexia, but it seems that it had. So, because there was a reading
component in the theory, we believe that it was what caused the differences between people
with or without dyslexia because they have to read even
if the sentences were short and in a way that we were not expecting that difference. Second, in terms of the performance, what could have happened here is, and you will see in the next presentation, dyslexia is related to many
common different skills, it’s not only reading and writing. I mean, everything is involved. You have working memory, you have visual-spatial attention, you have many, many, many common
different skills involved. And people normally overcome dyslexia, putting more strengths in these other common different skills. So, maybe what would have
happened there is that when it comes to chess, many
other common different skills are involved there and we
could not find differences or maybe there is no difference
at all, that is you know. And then in playing, the only difference we found was in time, but in performance, there were no differences. To sum up, we did not find any effects in the performance measures, probably because of the differences that we find in timing and
in number of the scores, dyslexia could have some impact in chess, but we are very cautious to say this because of the results. And right now, for his
undergraduate project, Sergi, in the picture,
he’s adapting the game to be released in Android market. We are now integrating some of the exercises that might cause more trouble for people with dyslexia
to predict dyslexia. And this is were it comes
the next presentation, Yeah! And we’re saving a lot of time. The results are much more
exciting, I promise you. which is Dytective, it is towards detecting dyslexia across languages using an online game. So, this is a joint work
with Kristin Williams and Jeff Bigham from CMU, Abdullah Ali from University
of Maryland, Baltimore County, UMBC, and Nancy Cushen White from University of California, San Francisco. So, let’s go. So, not to repeat myself, I’m going now to tell you
other things about dyslexia. It’s fascinating, no? So, here you have a study where they have braining imaging with people
with and without dylsexia and this is a brain of
people with dyslexia reading. So, you can see like the
differences are a pattern there, but normally language is in the… Oh my god! It’s in the right or the left hemisphere? Okay, yes, normally language
is in the left hemisphere, but in the case of dyslexia, when people with dyslexia read and write, both hemispheres, I mean, you know what I mean, right? So, people without
dyslexia, when they read, there’s some parts of the left hemisphere that get activated. When people with dyslexia read, it’s both hemispheres
and there are some parts that are supposed to be activated that are not activated. Okay, there we go. And now, I’m going to tell you something I think I’ve never shared in the research community. It’s a personal story just to explain you how difficult
it is to detect dyslexia. What we’re doing here is we’re
trying to detect dyslexia and this is to try to explain you how difficult it is to detect dyslexia. So, I was five years old, I was in kindergarten,
the year before school, and we were all sitting around, in the class, we were
all sitting in the floor and we all had like a little booklet and here, I remember as if it this was yesterday. And we have different images and the name on the back. So, the teacher was going around the class and saying, “You read the next word, “you read the next word,
you read the next word.” And I was like, “What? Oh my god. “I’m going to read the next word.” You know how this feel. You need to prepare
yourself, not screwing up, and I’m going to make it
work in front of the class. So, I was trying to prepare myself and I just couldn’t do it. I started like cross, I don’t
know, it’s very difficult. I have a PhD on dyslexia, I still don’t know how to explain this. But it was very, very
difficult, to you know. So, I got very, very
nervous and I was like, “Okay, I have to focus, focus, focus.” So then, I was like, “Okay,
the four images, good. “Oh, good, so I’m going to do, I think “one, two, three, four, one, okay. “I think that my word is going to be “the one that Peter did
and is going to read, “so I’m going to pay a lot of attention “on what he is going to read. “I’m going to remember and
then when it comes my turn, “the teacher is not going
to notice, hopefully.” And it happened, so the
girl who read the word, who now is my favorite school friend, I look for her so many years afterwards, said “pata” which means duck. And I was like, “Okay, pata, pata, pata” and when it comes to my turn, I said, “Pata” and the
teacher said, “Very good.” It was strange, hey, I got it right. So, imagine the teacher didn’t notice that I had a problem, I didn’t
notice that I had a problem. My parents, of course, didn’t
notice that I had a problem because I was super careful
that they never noticed. I mean, that I was like fine
like the rest of the students. And this is how dyslexia is. People with dyslexia, we
don’t notice we have a problem because this is how we have
accessed knowledge all our life, so we have no idea, we only assume that we are less intelligent than the rest because, of course, I kind
of thought that I was stupid because all the students could
do it and I couldn’t do it. But we keep on with the rest,
trying not to be noticed, trying to be normal until school becomes more and more complicated
and then you have a problem because you cannot keep up
doing this stuff all your life. So that is why dyslexia is
called hidden disability and that is why it is
so difficult to detect by everyone, by parents, by teachers and by people with dyslexia. And then, we go to how
do we detect dyslexia? Well, normally, it’s super late. We detect dyslexia when
students are failing at school. This is how you detect dyslexia. You have a student that
fails out of school who has normal intelligence
and they have dyslexia. Well, you’re already
failing, it’s too late. So, what we have our
paper-based diagnosis. There are both of them. There’s a very interesting
study done at MIT, McGovern Brain Institute
about neuroimaging, so they can predict
risk of having dyslexia, even when you have four
or eight months of birth, after eight months of birth,
they do brain scanning and they can predict, but not
many people can afford this. (mumbling) so more people can afford this. There are some computer
games to detect dyslexia. This has been done in
Finnish, in English, in Dutch, but they are language-dependent. And if you were, last year, in W4A and saw my presentation,
you know that we can also predict dyslexia using
eye-tracking measures. But what happens is this is not scalable. These solutions are not
scalable, are not affordable, are too late, they’re not moved anyone, and they are not fun. So, do you remember W4A 2015, yes? That was last year. That was in Florence. We were in the coffee break. That was right after my presentation of using eye-tracking
measures to detect dyslexia and it was with Jeff and Abdullah Ali, we met in the coffee break and I was like, “To help, we’re going to use games “to predict dyslexia, let’s
forget about eye-tracking. “Let’s forget about everything
that can be expensive. “Let’s do something scalable.” And we were so excited about this like “Oh my god, oh my god” and we took a picture
of us and yes, that is. So, now we are presenting the results of an idea that came in query from last year that we framed. So, what happens here? There are so many behaviors
that are different between people with and without dyslexia, but what we started doing
after trying a lot of stuff, actually, was like, “Oh my god, “we have this in front of us, “all this time, all these years” and that was using errors, so the input that people with dyslexia do as a source of knowledge. So, people with dyslexia when they write, when they read, they make a lot of errors and these errors are actual indicators of the difficulties that they have, so why don’t we use the
errors that they make, the inputs that they do, to create games based on this because
probably their behaviors are going to be different
to predict dyslexia. We have a game made of, I
mean, we use calculators and it was like let’s use all
the information that we have from this game, all the
performance that we have from children playing
with this game, since 2012 we have this game, to try to get the most interesting
exercises to be able to see. I mean, the exercises are going to have more differences between people
with and without dyslexia and see what happens. So, we extracted the
knowledge from this game and this is what we get. Here we got cut. Okay, very good. This is how we did. Oh, there you are. So, you’re going to see it
a little bit smaller, okay? So that is how Dytective looks. And just a second. (fun music) So, first there is an
anonymous questionnaire where we have the independent variables. (foreign language) And they have 25 seconds
to click as many times as possible a target
letter or a target syllable or an error or other things that we found from the exercises from before. So, this is composed of 32 games and these games target specific
indicators of dyslexia. So, now I go again to click that, yes. Specific indicators of dyslexia. All these indicators here
are related to dyslexia, so visual memory, visual attention, working memory, reading speed,
orthographic processing, phonological awareness,
phonological memory, phonemic segmentation,
syllabic segmentation, word recognition, non-word recognition, syntactic awareness, semantic awareness, error detection and
correction, word writing, and non-word writing and working memory. So, all these are related to dyslexia and each of the exercises that we designed were focusing one or two or three, sorry, these dyslexia indicators. So, here, for example,
you have another example of visual attention because
we knew for computer games that visual attention
has a lot to do with it. So, here you have to find
the different letters in a record amount of time. This is another example of exercises. Here we have to find the
errors in the sentences and we need to find as
many errors as possible. So, for example, smoking is prohibit on the entire aircraft. Oh, yes, and it says
off the entire aircraft, instead of on the entire aircraft. So, they need to find the errors. And then, went back from the experiment. We recruited 60 participants,
30 English-native speakers, 30 Spanish-native speakers, 15 and 15 with diagnosed dyslexia from 7 to 12 years old and
they had 15 minutes of play. Some of them were
supervised in their schools, some of them were supervised online by us, because some of them were from families with dyslexia and we had to
make a connection via online. So, per participant we have 192 measures, but basically it is like 32 games and per game, sorry, yes, 32 games and
per game we have 6 measures. And here are the measures,
they are super simple. So, it’s number of clicks, number of hits, so what exercises were
done, number of missing, score, accuracy, and miss rate. Super simple dependent measures. So, yes, this is more exciting
than the one, the one before. So then, if we put everything together, we find that for most of
the dependent measures, we could find significant
differences between groups when they play games, linguistic games, based off the errors that
people with dyslexia have. Yes? Okay. So, this is a new method and
it is potentially multilingual. Multilingual in terms of this can be used for languages that have alphabets, for Chinese and this, no. But for language with alphabets, we believe that this can be multilingual. And actually, Maria
Rauschenberger is doing this for German, we have another person called (inaudible) and finally, probably this
is cheap, what is going on? This is cheap and hopefully scalable. So that we can really,
really go to schools and instead of waiting
for the children to fail to predict, sorry not to
predict, to diagnose dyslexia, this could be before this happens. I’m sorry about that, I don’t know what’s going on with this. – It’s not you, it’s the– – Yeah, it’s been happening. – technical problems. – This is an example of how… This is the first time
that we show Dytective. This is an actual session
of people, of families with dyslexia, this is in
the start of the Spring and we were here doing a little informal focus group to know
that their disability was fine and how to improve the… basically the interface of the game. This is the best, look at this, look at… That is the best, no? Oh, I have to tell you
something about this game. There are three brothers,
two of them with dyslexia, one without dyslexia, and then
I went to one of the brothers and I say, “Hi, how are you doing?” And then this kid came
in front of his brother and told me, “Excuse me, the
one with dyslexia is me.” (laughter) It’s actually to say out
loud that you have dyslexia, so I was like, “Wow, okay.” We’re doing a good job because they are, probably not proud, but at
least they can say they have it. So, future work that we are doing, more online work that
we are doing right now. This was like the first approach for English and for Spanish. Since we had more contact
of schools in Spain and South America, we already put this, we made a large scale study with now it’s almost 7,000,
but these results are with around 4,300 students and
we used matching learning, we used record neural adult networks so, bidirectional literacy
and we can predict dyslexia, only for Spanish with almost 90% accuracy. We’re planning to do this multilingual and hopefully, we will
put this in the web, in the social media, you know, like a little button, so everyone can share this around facebook and maybe in one year time, you could say, “Hey, 10% of your Facebook
friends have dyslexia.” It’s like, “Hey, you already knew.” Probably one day dyslexia is
not hidden disability anymore, you know, this can be so cool. Now, we want to bring this
into the school for real, so if you really like this idea because the support of all this cannot be financed by
Carnegie Mellon University, only the research was financed, but we need to put this into schools. If you like it, share this. This is a kickstarter campaign
to put this into schools, to develop this as a real tool, to pay a developer to do this, and to put this into schools. So, please don’t hesitate
to share this all over because we have fourteen days left and we only have raised
like 20% of the money, yes, it sucks, yes. We are researchers, we
don’t know how to do that. Anyway, next time I think
we will have to contract like an IT guy in our
communication, I don’t know. But anyway, please share this all over. I mean, seriously, because
if we get this, this summer, after summer we will implement it and we will put it in the schools. And we actually have agreements with some of Spain’s communities
to put them in the schools. So, people want this in their schools, so I’m not only saying this. So, that’s it, so thank you very much. Here are the pictures of
my wonderful co-authors and now, I think we’re on time. No? Fine, okay, good. (clapping) – Okay, questions? Anyone? From the two presentations. The first one was on chess-based games, the second one was on Dytective. We have a question. – [Voiceover] I should be like Jennison, stand next to the mic. It sounds to me when I was
listening to your presentation, the key was that the games
work off of the child’s innate dyslexia problem,
they’re not able to see the e between the o,
but they don’t feel like they’re failing which is the opposite of what you thought as a child, where you felt you didn’t wanna fail, you wanted to fit in,
so you were hiding it. Is that true that the game is basically, the child is just having a fun time. And they don’t know that
they’re missing this or not. – That is the goal. The goal is to make
dyslexia detection fun. So, in the experiments with the schools, actually children wanted to play and maybe it’s just because they wanted to be on the computer, I don’t know. But they wanted to take the test. So, the idea is to make
dyslexia detection available because at this moment it’s not available, it’s super expensive. Actually, also diagnosis,
I forgot to mention this, are super expensive and to make it fun. This is just a screener,
it’s just detection. So, if the teacher or the parents know that this kid has risk of having dyslexia, not dyslexic, but has risk of having any reading or writing issue, they can put solution earlier. It’s not a boring test,
we made a lot of effort of not making it boring. Although, it’s 15 minutes,
language-related so, but we’re trying not to make it boring. – Samuel? – [Voiceover] Yes, we’re
going to chess-playing one – Yes, go on. – [Voiceover] If I
understand it right, it’s… Okay, let me start over. So, as far as I’m aware, the chess players grades are different, but we don’t know whether
those differences are… – Yeah, we don’t know this. – [Voiceover] So, we don’t
know whether they’re different based on whether they start out different or whether the chess-playing
makes them different as they learn it over the 20 (mumbles). So, do you think there’s any… What the views are and
whether there might be benefit in playing chess? Would there be a benefit for dyslexia kind of the other way around (inaudible). – Yes. – [Voiceover] Resources,
it’s due to lots of different people play with dyslexia there. It’s like playing chess with dyslexia. What about where you just
got through playing chess and just talk about people
with dyslexia who play chess feel the effects of dyslexia lessening? – Really. So, that was the idea behind this because we didn’t find strong differences. I haven’t done further experiments. So that’s very interesting. So difficult to save time on this, but there are other
studies that I haven’t put in the slides, but I’ve read them which they use chess for people with intellectual disabilities
not with dyslexia, okay? And they found improvements. It’s in the paper, in the related works. I didn’t know that the brain
of chess players are different. I should have. This is something I should have known. – [Voiceover] Yeah, ’cause
it seems interesting that that could be something, I
don’t know if you looked at adult with dyslexia who play chess, how their dyslexia– – We found one. There’s a professional adult player who didn’t take the test
because he was a professional who has dyslexia, he’s really
good, but we only found one and Sergi, well he’s a player, so he know many, many people. And asked me for the study, around in his matches, competitions, and we only found one. So, at the beginning, they idea was to do, how you say, professional chess players with and without dyslexia, we do… – [Voiceover] Lots of chess players just play for fun though. They’re not professionals,
they play for fun. They’re very good, but they’re not gonna start coming over to. – If we think people can not
only produce intellectual… Hypothetically, it has to be beneficial. In our experiment, we
didn’t find differences. Because we didn’t find
differences a priori. It’s difficult to say whether it’s going to have an impact long-term. We don’t know because our study was not longitudinal. It was like the nine lessons, boom! (mumbles) Only that, so yes. In an ideal world, it’s easy to review and it will be beneficial,
it could be beneficial, but we don’t know. – Okay, well thank you. – Okay, thank you. (clapping) – Next up is Claudia Buzzi. Is that right, great. From CNR-IIT – Yes, good morning. The work is a blackboard to deliver cognitive games for people with intellectual disabilities. It’s well known that ICT can
be beneficial for students support in learning, but also stimulating their individual potential
and can be more beneficial also for people with special needs. But accessibility issues in the tools for proper learning that can be often detected. So, our platform was specifically designed for people with down syndrome together with the condition
that train these people, but features of the platform that allow to make games customizable and make it usable, also with a wider audience. Why people with down syndrome? Because in last year or so, the main age of these people has increased and in this way, it was seen that adults have a high susceptibility to develop dementia and so on. So, to contrast this
trend, it’s important to make training, brain training on specific abilities such as attention of memory and the possibility to have a tool for make this kind of training can be beneficial. And an important aspect
is that personalization ’cause these kind of people are very special, you can read the unique characteristics. The possibility to create a learning path specific for each individual is crucial. So, we tried to do that. We looked for a platform with games in order to make plenty activities for the users. Hoping that computer enhanced tools can provide a tool more effective that
replace classical ones. For instance, I think
to capture attention. The specific abilities that we are trying to train are the classical ones, so attention, memory, but also motricity. The main important
characteristic of the platform is that it is a rich internet application with classical internet technologies. It’s web-based. It allow to make the student work everywhere. The tutors of the kids to check the performance result and eventually to modify
the learning part. The main accessibility aspect
are related to learnabililty because the learning path is performing with the
increasing level difficulties and in order to avoid implementations techniques that applied. Mainly focused on the visual channel, using augmentative
alternative communication. It use a reinforcement system, little visual reinforcement, but it can be personalized. The award system try to make the teacher or therapist to create the specific games usable for each individuals. The platform was designed with the help of the (mumbling) So, initials, but also
people with down syndrome. There is a specific
tool in order to extract the learning trends of the training. This is the main page. It is possible to access the platform in two modalities. The first is free and it is possible to access the different types of games at the moment avaible. So, sequence, here’s an
example of sequence or puzzles that can be set for the number and so, they go up in the difficulties. The classical memory games. This is an example with animals, all the families complete. There are all the results in a positive video to add example of action. So, to train a particular function. The second modalities is the… possible after registration. So, the tutor or instructor can register itself and the child or the student or the adult The additional features of the platform are the recording of that performance. And so, the learning analytics tool, but also the possibility to personalize the learning path for the student. So, with the personalization of content, but also level of difficulty and the creation of games. This is the user interface for the tutor and it is possible to add the new student, but also to refine the content at the moment, present in the platform or to upload content, images or videos. And to use them to create new games for instance, it means
a new part of games. But also, a sequence, and this is the view for the student. It is possible to extract
data on the performance in the form of graphs but also interactive graphs. but also, downloading all the data. In order to refine the software and the user interface, we
performed the pilot test. The first pilot test with the subject. The result of the subject were interesting because both the users seemed to appreciate the
platform and the user interfeace. Particularly, they loved the challenge. The fact that they were able to perform something challenges for them. The success was sort of reinforce from themselves. The pleasant aspect of the platform, the users were interested. One of the users that was with the hire query, QI, tried to suggest us a lot
of interesting things. For instance, content to use, how to perform some action, and all the feedback were evaluated in order to improve the platform. And she was very autonomous, but the same she had problem, some of the main problems were related to the idea to perform, in the best way, the task. Even if the user was informed that there is an attractive region, so it’s not necessary to perform in drag and drop tasks in the perfect way because of the attractive region could complete the task, but she wanted to perform as exactly, and finally I adapt the time. The problem was the connection. The responsiveness of
the internet connection was not always acceptable causing high frustration in the users. Another possible issue is that the internet connection
is not always available. As a future work, we obviously want to enrich our platform, both for content, but
also game collection. We started less than an hour ago, so we are always in the development phase. We are performing another pilot test for the usability aspect, but we are trying to design clean kind of an evaluation. Design a user study with clinicians and psychologists in order to define if
this kind of platform can be useful and effective to train people with down syndrome and in general, people
with cognitive impairments. Project has been funded by the Italian government and I’m finished. (clapping) – Questions? – [Voiceover] You thought
you could keep me away from the microphone? Erin Buehler, UMBC. Thank you so much for your talk, Claudia. I have a question/suggestion
in terms of custom content. I don’t know if you’re familiar with the training
technique of self-modeling where you give a person
with intellectual disability prompts and break up task into this step is actually this step and this step and the self-modeling
one is where they use pictures or videos of
themselves completing the tasks. And I was wondering if
you were considering including that future
generations of the game, having the people using it posing for a different sort of task, as part of the customization. – Yes, we are also performing on this platform to develop another part related to other procedures and the step of the possible actions that can be performed by the students are also using a tablet to make a picture or
videos of the activities and enter this content in the platform, in order to have a more direct content to train… I completely agree it’s
one of the new features. Thank you. – Anymore questions? I have a, oh. – [Voiceover] From the
preliminary objective, what kind of data have you collected from the learning analytics testing? What kind of data are you collecting? – For each game we tried to individuate the relevant data. For instance, the number
of correct answer, number of attempts, number of errors because even if the platform… It depends on the game because data recorded for
each game are different because the metrics to evaluate the important things in the performance of
the data are different. But at the moment, we have designed for each game how that data will capture. But are merely related to the interaction and the failed and correct trials. – Okay, well, thank you. (clapping) Okay, next up is David
Sloan and Sarah Horton from The Paciello Group. – I know what you’re thinking, the people who talk far too
long are on just before lunch. We’ll be ask quick as we can. So, Sarah and I are talking
to you this morning, as people who used to work for higher education institutions in slightly different roles,
but related to accessibility, and we now work for an
accessibility consultancy that is very interested in supporting educational organizations
providing the most inclusive learning environment possible And that’s why we’re really interested in figuring out the best possible way to support educational organizations in dealing with accessibility
which is, let’s face it, a pretty fiendishly
complex challenge to solve, but also one with enormous opportunities, And today, we’re introducing the idea of the master plan as a way to think more strategically about
managing accessibility. And we also have some questions
for you folks, as well. This talk is going to
morph from us presenting some ideas to you to a
discussion that we have either here or in lunch. So, master planning is an approach that really has been present in the physical space for quite some time. A master plan is a strategic approach to decision-making in a
complex space such as a city. And the slide we’re showing
just now is a map of Montreal and at dinner last night,
Chris LaRoche kept talking about the island that Montreal lies on and I couldn’t understand that and the map here clearly shows Montreal as an island in the middle
of this large river. A map as a visual artifact or– (woman coughs) A master plan can help you figure out the space that you’re working in. The Montreal master plan
is a way for the city to bring all the different
stakeholders together, a way for a clear
articulation of the objectives of the city as a space
where people live and work and make money and have fun, to figure out what can
happen in a particular area of the city in a way that
a decision has been made understanding what else is important and what else is going on. We’re also seeing the master plan idea also move into the digital space. I discovered that Dublin in Ireland has a digital master plan which is used to help define and implement a strategy for connecting
the citizens of Dublin to a digital world and
providing access to broadband and these other support
features like that. And we saw the master plan as something that helps or is potentially
useful in helping tackle accessibility as this very complex, multi-stakeholder challenge, but with enormous
opportunities for inclusion and improvement in the way
that we support learning and the way that we use
technology to do that. So, we thought as we’re
spending a lot of time working with organizations to find
and implement strategy, perhaps a master plan is an opportunity to move things forward, but there are lots of
challenges in doing that. Over the course of the
last couple of days, we’ve seen a lot of great
resources and insights that you shared to support
accessible learning and we’re here to talk
about how much that work is part of an accessibility master plan. And for that, we bring
our mascot, the Lorax. The Lorax is a Dr. Seuss character and he has very valuable insights into important things having
to do with trees and air and his insights aren’t valued very much and they’re not part of the master plan. And accessibility work can often feel as though it is marginal and that it’s not a part
of an organizational or even global master plan. So, Dave and I work a
lot of with organizations that are often in education and we’re trying to create
accessibility master plans and one thing we see quite a lot, especially within research
and scholarship is a separation between the organizations that are doing scholarship and research and the institution in
which they’re doing it. in a lot of cases, for
example, the research institute that is essentially, what we
call, a tub on its own bottom. It brings in its own money,
it has its own IT department, it has servers under desks. Does this sound like anyone’s world that I’m describing here? Is a tubs on their own bottoms? Or are you individually
a tub on your own bottom? So, when we’re thinking about that, there’s a value to autonomy, right? And that’s why that happens. But what happens is the
work that you’re doing here and that we’re talking about today isn’t part of that master plan. And so, the Lorax with all
of these great insights isn’t getting integrated into
some larger context and the work that you’re
doing isn’t, for example, helping your institution
teach better MOOCs for a global audience. So, what we wanted to talk
about today with you is ways to look at master
planning and to find your place within your
organization or global master plan. So, we’re helping you from seeing success as having a paper accepted at W4A, talking about some aspect of supporting accessible learning to
actually making a difference at your own institution
or other institutions and integrating that knowledge
that you have gathered and tested and typed peer-approved at a conference like this. And we have identified four key aspects, foundations of accessible online learning that could form the
basis of a master plan. Very conveniently, these
four key foundations all begin with a P, so
they’re easy to remember. We’ll start with policy. Policy is where you
identify the priorities for achieving some kind of success and the obligations that you’re under. Whether it’s legal obligations
or some other objectives relating to the success or achievement of the educational institution that you’re working for or helping. So, this is about defining benchmarks and standards, looking at
whether you’re referencing standards like web content
accessibility guidelines, we’re all familiar with
accessibility guidelines, and integrating them into
some formal, written policy that everyone understands, this is what we’re trying to achieve, these are some basic requirements
that we want to meet. And then this process has a formal sort of
ordered way in doing things and particularly, when it
comes to digital landscape where that landscape formed through people making purchasing decisions,
people developing stuff, borrowing things from somewhere else, taking something that
they’ve seen at a conference and start introducing it
to their own teaching. That process can sometimes
be very haphazard. There may well be no process. Particularly, when it
comes to academic freedom to construct your own
digital learning experiences. If there’s no defined process, then how can accessibility
possibly be controlled in there? Then there are programs. The way that the master
plan identifies area for central services or programs can help individuals and
groups achieve their objectives and meet the obligations
identified in the policy. For example, if you say that we want to use online video to support learning, to provide a richer learning experience, we also want to make sure
that video was captioned, its audio described when required. But rather than place that responsibility in every individual who’s
acquiring or creating that video, let’s provide an in-house
captioning services or let’s provide a clear way for people to request and receive captioned videos. So that aspect of the task of making things accessible is centralized. And then there’s practice,
the things that people do. And that’s relevant to an individuals role and responsibility,
so that practice is not accessibility in its weighted sense, but the aspect of accessibility
specific to your role. Whether you’re an instructor
or an educational designer or somebody working in procurement or you work in disability support, that accessibility practice
is specific to your role. And everyone works together
and everyone understands what everyone else’s role. So, the master plan is
this holistic view of what the organization is trying to achieve. Also, allows people to see where others fit into this role and so
that there’s no duplication or overlap or gaps in support where everyone thinks that somebody else is doing something about it. So, to finish up, in
thinking about your own work in accessible online
learning and activities that we’ve been talking about over the course of these last days, what do you need personally
and organizationally to be part of an
accessibility master plan? It’s a bit of a rhetorical
question right now because we’re out of time, but the good news is that we’re working on finding
answers to this question and we’d love to have you participate. We have an accessible online
learning community group through the W3C community groups. We started it up a few months ago. Dave and I are co-chairs. We’re still trying to
kinda get our heads around who we are and what
we’re going to be doing. But we’ve identified to major work streams and we’d love to have you join in on them. And the first work stream is
looking at a gap analysis. Sorry about all the flashing. The screen keeps flashing for some reason. I kinda feel like unplugging it. Everybody in voting for unplug,? Yeah? So, we have to work streams. One is looking at the existing resources that are available through the W3C and web accessibility initiative and looking at how do these support accessible online learning
and where are their gaps that we need to fill? And then the second work stream is looking at online learning
more broadly to say, “What could the W3C bring to that “conversation of online learning?” that would support
accessible best practices and get accessibility into the
online learning master plan that we’ve all been talking about today. And it just occurred to me when
we were planning this talk, how valuable an opportunity
this conference is to bring in the research perspective. Now, the research and
development working group that quite of few of us, who regularly attend W4A were involved in, that’s no longer in existence as an entity where the research community can inform the accessibility standards
development process. But there is an opportunity through the online learning community group to provide a pipelines,
particularly for those of us who are researching in the area of accessible online
learning, have an opportunity to figure out where can
our research contribute to the suite of documents
that the W3C provides to support accessibility,
probably the informative documents from the normative documents, but it is an opportunity
here to have a conversation and we’d love you to join us for lunch or tea afterwards, whatever. – Yeah, we were thinking about
if people wanna talk more about this community
group, we’ll be at a table and having lunch and we
could gather together and talk about it or maybe we’ll take over the whole dining room and talk about it. Thank you very much. – Thank you. (clapping) – Since we’re out of time, if it’s okay, would it be okay if we moved the questions to lunch and after? – Yes. – Okay, great. So, sorry about the dim
lights, let’s go eat.

Leave a Reply

Your email address will not be published. Required fields are marked *