Welcome you all to the second panel
of the first day and
this panel
will aim to discuss
the current status of natural language processing techniques.
And during the context of the the web, we can think
or we know that they can be used
to generate textual descriptions for images,
but also for other visual media that's presented on web pages
and we will focus today our discussion on
or start to consider aspects such as text adaptation
and how to provide understandable text to better meet the web user needs.
And they're in different contexts of of use and also what
our future perspective for natural language
processing on web accessibility
are to support web accessibility.
So I'm glad to welcome back
Michael, Shivam and Amy.
And when
you are Amy and also
to welcome Shaomei Wu from AImpower
who agreed to join
us on the second panel of the day.
Welcome you all back and welcome Shaomei.
For your first intervention
I ask you just to briefly introduce yourself
and your three other co panelists
already done that on the previous panel.
So no need to to reintroduce yourselves.
But I will start by
thinking about once again the quality.
So we go back to the to the quality topic,
but now the quality of machine generated descriptions
now no longer from the perspective of image processing,
but from the perspective of the natural language generation, right?
So how can we improve the quality
of this machine generated descriptions, especially taking into account
the personalized preferences from users?
And I will start with you Shaomei
Hello. So
thank you all for
having me here
today.
And my name is Shaomei Wu and right now
I'm the founder and CEO of AImpower.org
Tech non profit that researches
and co-create
in-powering technology for
other ways marginalized users.
And first of all, I want to
and also share,
that I do have a stutter so that you may hear
there'll be more pauses and ... when I talk
and before AImpower.
I was a research scientist at Facebook
leading a lot of research and
product work on
accessibility, inclusion and equity.
So one of those product that I shipped is
automatic alt texts
and these you're allowed to we will provide
short and
machine generated
description of
images on Facebook and Instagram
to screen reader users in real time.
So when it comes to
quality of automated alt text
or other similar
systems,
we saw two kind of biggest
area of development that we wanted to do.
And the first one is accuracy,
which I think we talked a lot about
in the last panel as well.
But I want to talk
it'll be more about the second one, which is the
richness of those
descriptions.
So to be honest,
like the alt texts that we generated
was quite
limited, you know, a lot of
users they will say that it's more like
teasers, kind of, you know,
and telling you like, oh yeah, five
people smiling, pizza, indoor.
But no more than that.
No more than what kind of pizza and what kind of like
indoor environment. Is it at home?
Is it like a restaurant?
So I think our users really, when they kind of gathered all of the
richness of what someone who has eyesight can see,
you know, can handle access.
So one particular kind
an area that users do really want to know more
is it's about people who they are, how do they
look?
Look like... race, gender and
even how
attractive they are
because that is something really kind of socially salient.
So that was kind of big challenge for us.
One way was designing our
our system
because, you know, like, how can we share
those kind of attribute in a, like,
you know, both accurate and kind of socially
conscious way.
So we actually opt to not
kind of
showing like the race and gender of the,
the people being photographed,
but which I you know,
we got actually a lot of complaint on
but how to kind of yeah convey those in a kind of socially
respectful way I think it's something we
we should really work like work on
and now I can see kind of like a few ways that we can make that better.
For example, like, considering the like...
relationship
between kind of people in the photo and
yours, for example, like if they are friends
and then we can just tell them the name, and, you know, other things about those people
and then another thing is to kind of, give people
progress details.
So then you know, I have some kind of like option to kind of allow the consumer
of those alt texts to kind of request more
details that we just cannot provide
by our systems.
So I will be done here and let
others talk.
Thank you Shaomei.
Shivam, your thoughts on
how can we improve the quality of machine generated descriptions?
Okay, so this is a two part thing.
So when you come to technically implementing more, say
how you have design your model, how you train them,
and what the whoever the stakeholders of designing a particular model
is very much necessary in how and when to get the quality machine
generated description.
Now when you when we take into account users personalized
preferences, this is a two part...
So first, let's take an example.
I am a person who knows Spanish and my model,
a very famous model gives descriptions in English right, now
that model or whatever the consumption of that model
is like let's say you're using an API to consume the model.
So that should take into account the user's personalized preferences
of his language and write the output based on that as well.
So this diversity of a model to prepare output in multiple
formats, multiple languages is something that can be looked into.
This is how the quality of the machine generated description increases.
Now, you did not train the complete model separately.
What you can do
is just a post-processing scripts for your models and that can help.
And you it's not much of an effort when we say as a model training input,
but it's very simple solution to what can be a best solution.
The other thing is how you prepare your quality data.
Now you should fully carefully categorize it is strictly if needed.
And let's say you have input data that are blurred images and all sorts of thing.
So you need to carefully prepare your model, creating data and
based on that data, your description would be
a bit more clearer and the population will also be factored in.
When you
look into how you can both process your data for a certain group of people.
So that's how I see it.
Thank you.
And Amy, want to share your experiences?
Sure.
So a couple of ways that I've seen
that I think are sort of promising maybe to use NLP to improve quality.
One thing I started seeing recently is people,
you know, starting to consider context around the
the image that's going to be described, to maybe
create a description that's more that's more helpful.
So imagine, you know,
someone writes a post on Twitter and they couple that post with an image.
So considering the the post and the image together, maybe might inform
models on how to create something that's more informative.
So for instance, if I posted a picture of myself
snowboarding and I said I learned a new trick,
then it might be important to tell me what trick you learned.
Whereas on the other hand, I said, I just went on vacation.
You know, the exact trick might not matter as much.
And so I think that the idea of like using language understanding
to get more information about the context before making a prediction is promising
in another way.
I've sort of seen it used to maybe improve the quality kind of
goes back to the other
the other answers that were given so maybe you can use question
answering about the image to gain more information when you need it.
Oh, one thing
I've also thought about is seeing if maybe users could give examples
or their preferences about descriptions in in natural language.
So this is an example of a description maybe we can copy the style
of this description when we're applying it to other descriptions.
So maybe I like to hear about the costumes someone wears in a
in in a video, and I wish that future descriptions
might include more information about that rather than summarizing them.
And then finally, one other way I've seen,
I've used NLP to improve
quality, is also based on summarization.
So there can be times when there's more to describe than time
you have to describe it.
So especially in videos, there's often a really small amount
of time to describe without overlapping the other audio.
So one way you can use use NLP to improve the quality
is by trying to summarize those descriptions so they fit
within the time you have and they don't decrease the experience of people
trying to watch the video and hear the audio at the same time.
Yeah, yeah.
That's that's definitely a good use for NLP.
Michael, still in this topic
and I would like to have your perspective
on initiatives from WAI that
might assist users in
providing their preferences
so that eventually
models can use those
or anything that might be ongoing in that regard.
Okay.
So first of all, just give the disclaimer for anybody new to this session
that I'm not a machine learning professional, I'm speaking from the
of my work and the Web accessibility initiative.
I do want to talk briefly.
I think the other panelists covered almost anything
I would have said
one thing that based on my knowledge of how machine learning works generally
today, our models tend to be focused
on, you know, of our particular ability,
and it's not universal.
And the future models will have more abilities combined.
But, you know, so there may be one model that can recognize this is a human.
And here are those attributes, another one that you can say this is this human,
and yet another one that can say this human plus that human
equals this relationship.
So all of
that information, I believe is separate right now.
So the ability for models to share contexts,
I think is going to be a part of the solution that we need.
So what I can speak up for in the Web accessibility initiative.
So we are only beginning to explore
what AI and accessibility means.
And so this symposium is a part of that process.
We have a practice of doing
research papers, sort of literature reviews,
and then proposing accessibility user requirements.
So that would be something that, you know, we could be working on to
start gathering this information.
And from there we decide what to do, whether the content goes into guidelines
or into new technologies or whatever.
But I think most of the resources around AI are
would fit into new resources for those categories.
Okay, great. Thanks. And
I would like now to move on to addressing something
that was basically the core of Jutta’s keynote.
So it's
discrimination bias or any other type of bias.
And here
also looking at something that Antonio Gambabari
has entered in the Q&A for the previous panel.
But I think it's also a
very well fit.
It fits very well into this topic and it brought out
the use
large language models, LLMs
and which are currently
getting a lot of traction and a lot of spotlight.
And and do you think these LLMs
can open up
new avenues, as Antonio Gambabari was mentioning, for reducing
the different type of bias that we see
as a result of the use of AI trained models?
And Shivam you want to go first, this time?
Yeah, sure, sure.
So this is quite a question which is
and has been close to my heart as well, how can we address social
bias, in largely any model. As part of industry
have seen a lot of ML models trainings how the output comes.
So social ML model results of data that they have
and how the social attitudes are represented within that model.
And most of that available data is used between models publicly sold,
which continuously degree of bias that you can see,
because most of the data that are generated on Internet is basically
those people who can consume it, right?
It it's not that everybody once would who doesn't even know what it is.
They cannot create data over there.
So most of the data that is available to train the model, it's built out of that.
So that's how you see a bias in one way. The other
instance I can give an example is you will see a lot of street violence,
homelessness, drug overdose and, all those things overrepresented
in the text that discuss mental illness, although these both are not similar,
but you will find this kind of representation
in ML outputs like
how can we address this?
Now there there is a novel way of human in the loop.
Our human feedback loop on an existing models where you can provide some feedback
to the already existing model that this is the sort of output is not correct.
This can be a correct version and this can be a good version some some human
interface
is needed or
what that now the other aspect of it is the representational
training of ML models.
Now, the underlying data is models.
It's the main source of the issue here.
So you need to correctly source your data and at least up to date,
you're not all representing one section of data.
For example, let's say you have a bigger society.
This society can be underprivileged, overprivileged and maybe some rigid persons.
They just account society.
Now, you cannot just take the data from one section of society and train
the model and say that I can give you a complete picture of the area.
There's much a separate section which are underrepresented. That's
what is happening with all the models right from the start of ML.
You can see. Now what we can also do to mitigate is you can create inclusive buckets
where the developer of ML models or designer of ML models
you can give the inclusive bucket training to them.
You can
get them
aware that what is happening and what can how we can mitigate this.
So all the person who are included in ML generation
or there are a lot of you still going on, I mean a lot of data extraction goes on.
So all those people can be trained for inclusiveness.
There are multiple tools that help us do that.
Like if you are creating a model, you can test in
Google helps us in
analyzing the
models like Google has a lot of tools, AI fairness,
So how your models are performing when talk about
a lot of including inclusive outputs of your data.
Also you need to do a thorough testing of your models
whenever you go ahead to include that all the outputs are properly
aligned, properly represented, all the sections of your model
which it is intended to be used should be represented well.
Your testing should be that model in case of any models you're creating,
because now we're not at that stage that
AI and ML is
in the starting off state, it's quite mature
right now. We are seeing
a lot of breakthrough technology so we can do this going forward.
I guess this can be a solution.
Okay, thank you.
Shivam. Shaomei, can we have your input on how
can we address that social bias or other types of bias?
Yeah. So
yeah.
So then on these, I want to kind of go back to
I just kind of talk about before in
particular on
sensitive social
identities, you know, about people
on the photos.
I, I don't see I kind of the
way for the
a current
and machine learning
system to kind of accurately
come out with those labels.
I think the key kind of issue here is a lot of those
systems will
kind of really assume these like fixed
and definite need
these kind of social categorizations
such as race and gender.
So I think maybe we should think
be an kind of
a machine learning systems
and kind of find a way to
to kind of attribute people
respective race back fully
through by doing this kind of like
having to like agencies
of those being kind of
photographed and being
described.
For example, I think now a lot of people has been kind of
specifying their
pronouns, for example in their
social media bios
and I think those kind of
information should be
made use of or could be kind of
made use of one way
of assigning on, you know,
one way to describing
the gender of somebody in the photo.
And also another kind of
interactions that we have been kind of exploring
is to just kind of describing
the appearances
instead of identities.
For example, what kind of
describe
skin tones or hair style
and outfit
instead of
assigning a kind of race or gender label of somebody.
But I don't think any of those solutions can really address
the kind of the real cause of the problem.
So I don't really have a very good answer on this
I think maybe we should,
you know, like maybe the alternative is to kind of think of the way
to come away and kind of share who we are.
We saw so much relying on the kind of
images like we are today.
So, you know, like, how can we convey
the kind of
information that we want to share online,
not so visual centric way.
I think that's a kind of
bigger
question, the way I saw it, too.
Thank you.
Thank you, Shaomei.
And Amy next to you.
I think the prior the prior answer is mostly covered.
The things I was going to mention I loved.
Shaomei’s answer about,
you know, describing ourselves in ways that are like figuring out ways
that don't rely on the visual information and giving agency to people
to just to add their own identities that they want to be shared.
I will say that I think that that depends in different contexts.
You might want to share
different parts of your identity if it's important to you and you might.
And so I think that even things that give like end users agency
might have a lot of subtlety and how they would be applied in different cases.
And I like the idea, you know, of describing,
you know, aspects of appearance.
I think you're missing one one challenge with that is
you might be sort of trading off
between these like aspects of appearance that you're describing and
and the efficiency with which someone can like
maybe they're not going to get
the information as quickly as a sighted person would perceiving that person.
And just because, you know, audio occurs over time.
So so I think there's it's an extremely difficult, difficult challenge.
And and in some cases it can matter.
Like I can imagine, you know, seeing a photograph of the leadership
of a company, you might want to know some some quick
details about about the demographics of who's who's leading it.
For instance, one one thing that I've noticed that
is is sort of related to this is that, you know,
when I'm when I am asking.
So I sometimes,
you know, have people describe videos and there can be a lot of differences
in which aspects, even if they're going to describe the aspects of someone's appearance
the way they describe those based on who is in front of them
can also differ based on biases that people have.
So if people see a woman, they might describe her differently
than they would describe a man.
They might focus on different aspects of of appearance.
And so I think even things that go towards describing aspects of appearance
will have to be, you know, very carefully, very carefully designed.
And it really feels like a challenging a challenging problem.
Yeah.
Thank you so much, Amy.
Michael, any thoughts on this?
And I would
add something
here, especially for you, ...
do you see any future role in accessibility guidelines
in contributing to
preventing bias in machine
learning, generated descriptions or whatever
that results from these models?
I know my answer to that question.
It could be longer than my prepared answers.
So let's see where we go.
I just want to add a couple of thoughts to what the others have been saying.
I want to first to categorize bias
as we're talking so far mainly about bias and recognition.
You know, this... are there biases of how machine learning recognizes
objects, people, etc., contexts
in that, one thing that magnifies the challenge and accessibility context
is that the sample size of people with disabilities
can be smaller in various training sets.
And so there is a risk
that images of people with
disabilities on a training set or contexts that are important for them, like wheelchair ramps
or something will be excluded as outliers
or will be less well recognizable by the AI
than, you know, images of other people are.
So, you know, that's just another
another dimension to the aspects of bias that we need to look at.
But then we also need to look at the own bias in the application of this.
You know, we've talked a few times during the session
about the risk of relying on machine generated
descriptions, captions as being good enough,
whereas content that has more of a mainstream
audience might also have captions as descriptions.
But get more curated, you know, quality assurance.
So, you know, that kind of bias could creep in and
that can
magnify the impact on disability bias,
you know, because it can cause people to be excluded from the fora
that, often, which people are recruited to be part of training sets, etc..
So, you know, again, the ethical principles from where machine learning
speaks to that.
And I think that we may by identifying some content
that we need to add to that.
So moving on to what we can do about that, you know,
I do believe that it's within the scope of the Web accessibility initiative
or the W3C to provide guidance
in some form about how AI and accessibility
should work together, addressing many of these things.
You know, typically this sort of thing
would be a working group node, which means that it's a
it is a a formal document
published by the W3C that's had a certain level of review.
There's even opportunities for versions
that have had more review and signoff.
So I think that's one thing we might like to do.
I'll also talk briefly about the work that we're doing
on the Web content accessibility guidelines
3.0 sorry, the W3C accessibility
guidelines 3
or WCAG 3.
We it's a it's a substantial re-envisioning
and it's been a core requirement
from the beginning that we wanted to address, you know,
addressed equity in the approach, in the guidelines.
How are we going to make sure that they're equitable to people with disabilities?
We've been exploring that in specific ways and within the working group,
really unpacking that to understand,
you know, the relationship of equity and accessibility and bias.
And in the other dimension.
So that's turning, you know, we're connecting that with other work
W3C has been doing to make itself a more equitable organization.
And so, you know, this is to say that I believe WCAG 3,
will also have some structure built in and support resources,
addressing issues of bias specifically.
Now, these are
hopes, not promises, but you know,
that's the direction from activities like this.
Thank you so much.
And yes, those are really some exciting avenues that we
we hope that can come to fruition in the near future.
So I guess final question
for everyone, and it is
I would like to know a bit about your future perspectives on the use
of natural language processing for the field or in the field of accessibility.
And I'll start with you this time, Amy.
Yeah, So I think this is a really exciting area.
And one thing, one one shift I've found recently among people in NLP
who I talked to is that, you know, as the models are getting better
at just creating like fluent, fluent text that looks reasonable,
that lot of people
are becoming more interested in what are the actual applications of this
and how can we build tools that actually support those applications
rather than relying on, you know, automated metrics for,
but that might not, you know, capture people's experiences.
So I wanted to to note that that's a direction I found exciting.
So I guess a couple a couple of things I think could be promising are
and I've kind of mentioned them before in my other in my other responses, But,
you know, as gain the ability to describe more and more about the image, I,
I think that NLP can provide a really good opportunity to personalize
those descriptions based on the person and what they want as well as the context
there is.
You know, if you think about walking in a room,
there's like so much you could possibly describe.
If we can make it easier for people to get the information that they're
looking for quickly from their media, that that would be a
a great improvement.
You know, combining computer vision to recognize things
in the underlying image and using something like NLP to
to summarize that description I think is is promising and exciting.
And one other way I think I'm excited about
it is in its opportunities to maybe help people with their own description tasks.
So when we have humans working on descriptions, it's really hard.
So, you know, novices sometimes have a hard time remembering
and applying the guidelines that exist.
You know, maybe we could rewrite people's descriptions of videos to be more in line
with how an expert would write them
by making them more concise or changing the grammar a bit
so that it fits what people are expecting from their guidelines.
Or we might alert people to aspects of their own descriptions
that that might need to that could be changed a little bit
to perhaps reduce something like bias that they have in the description.
So I think there's there's really lots of exciting opportunities
in terms of authoring descriptions as well as making those end descriptions
a little bit better. Yeah.
Great.
Yeah.
Thanks a lot. Shivam?
Yeah, so
I see it a bit of more it now rather than earlier,
because now the models, the engines are quite advanced
so I see a good context
aware solution that gives you faster processing some efficient data.
Right.
And that works on text, video and as well as audio.
So I see this happening to be a reality.
A good use case I would have been following up
also is how we can make the academic textbooks.
And we have academic assignments, right?
There are multiple charts, bar chart graphs, all associated data.
If some of these models or technologies can create better
understanding of those things, it would help a lot of
people in understanding that we have difficulty just by reading it.
Or maybe in absence of good quality
descriptions of these charts, bars and all those things.
I see this happening in the next few years.
A better description of the generation.
And as a closing comment, I would say there are different types of consumers
of media that some can easily read but not comprehend.
Some comprehend easily, but have difficulty consuming,
consuming it visually.
Now in that sense, the coming NLP technologies would help designers
have contextual description of outputs and that I would see in simple terms.
If you give me a simple, efficient faster output of a media and it's correct,
then it will be the pinnacle of what I see as the NLP.
And these are for natural language processing understanding
as well as generation for all key technologies.
Thank you so much.
It's exciting times ahead. Definitely.
Michael, you want to share your vision.
So based on my knowledge of
how machine learning in the present day works,
you know, the tools tend to be more
focused on specific abilities,
which means that the context is a bit isolated.
So I think I'm speaking as a as a person working the field,
identifying a need rather than something that may necessarily be a technological
potential.
But the Internet of Things used as APIs
to exchange data between different types of devices.
And if we could model some structure like that so that these tools
could share contexts with each other and negotiate a better group description,
I think that that might be an opportunity for an early
evolution of this field.
You know, the long term, of course,
I think tools will emerge with greater sense of context built in,
but that'll probably be, you know, another
tier slash singularity or whatever.
So yeah, that's my view on that near term future based on my knowledge.
Yeah.
Good, good suggestions too to look at
also. And Shaomei?
Yeah.
So yeah. So
looking into the future
I can see kind of two
areas
that I think will have a lot of potentials.
And the first one it's from the
technology
perspective which I agree with Michael that I can see
a lot of gain in kind of
incorporating the context
surrounding photos
and by you know like taking advantage
of the reason and,
and progressing
and deep learning models that kind of have
what kind of math models are
representations space So you know like we can embed both
the kind of image as well as the kind of tags
surrounding it and then
and then add a kind of
metadata such as the author or the time
when, you know,
when the photo was taken, all kind of posted.
So, you know, a lot of those can be kind of drawing in a kind of big
represent patient space that kind of that
provides a lot more than just kind of
visual
information alone.
So I think that's a kind of big
technology break
through that we can see in the kind of near-term future.
But the kind of second thing I think and more important to me
is the kind of use case
perspectives.
I think right now when we think about all kind of talk
about
the media
accessibility
we are mostly kind of think about our consumption case,
like how do I help some people who can not see to kind of
to kind of consume photos that posted by
others and mostly by kind of sighted folks.
But I think it's equally important but largely kind of
overlook is,
is these kind of media
creation
use cases, you know, like how can we support
people with visual
impairment to kind of
create and kind of share photos and videos
in my own work into, you know,
these use cases, which is why you know
there's like such a gap in what the kind of
current technology can do.
For example one like all a kind of modern
AI models really failed
when it came to processing photos
taken by people with visual impairments
because they are just not the same kind of photo that are used to
train those
models.
So, you know, there's a huge gap in what kind of current like the kind of
fundamentals of
those models and then what they can do.
And then second is there is a lot need for more
personalized and
and aesthetic needs.
Right after I take ten
selfies, I wanna find out why
I wanna post that kind of share who I am
and that it's or it's like we cannot do at all.
We can, you know we can kind of tell you're like, okay, you'll have ten
photos and are kind of containing your face,
but you know but, but,
but then how like, how can we change kind of the models
that can really represent somebody's space and then, you know,
somebody’s kind of aesthetics and I think that's another interesting
future
development that I want to see.
So that's all.
Thank you so much, Shaomei.
And I think we we only have 4 minutes more.
So I won’t risk another question
because we need to to to end at the top of the hour.
And so I will take the opportunity to once again, thanks
thank our our panelists and I hope everyone enjoyed it as much as I did.
And it was
really interesting
and very, very optimistic perspectives so that
we can see that's not just the, uh, the,
um, risky, uh,
or risk enabling, uh, outputs that A.I.
can have.
So it's nice to, to have these perspectives.
So thank you once again.
So Shaomei, Shivam, Amy and Michael, it was brilliant to have you here,
and thanks
who attended.