The topic for for this panel
will be computer vision for media accessibility.
So here we aim to foster
a discussion on the current state of computer vision techniques
and focus on image recognition and identification
and recognition of elements and text in web images and media.
And considering all the different usage
scenarios that
that that emerge on the web.
And so we'll be looking here at aspects
like how can we improve quality and
and how do we define quality for this
quality and accuracy of current computer vision techniques
and what are the opportunities and what are the future directions for this
in the in this domain? So
we'll be joined
by three panelists for this first panel
Amy Pavel from the University of Texas
and Shivam Singh from mavQ
and Michael Cooper from the W3C.
And Okay, great.
Everyone's online and sharing their videos.
So thank you all for agreeing to join.
I will ask you to before your first intervention.
Just give a brief introduction to yourself.
So let people know who you are and what you're doing.
And I would like to start on
one of the issues of quality.
And as I was saying, so how do we define quality
here? And here
I was looking at aspects such as how do we
or how can we train AI models
that are able to identify aspects in an image
such as identity, emotion and appearance,
which are particular relevant for personal images.
So how can we
get API to do that that we humans can do?
So and I'll start with you, Amy.
Excellent.
Thank you so much. So my name is Amy Pavel.
I am an assistant professor at UT Austin
in the computer science department, and I'm super excited to be here
because a big part of my research is exploring how to
create better descriptions for online media.
And so I work everywhere from social media, like describing
images on Twitter and as well as new forms of online media like GIFs and Means.
And I've also worked a little bit on videos.
So both educational videos like making the descriptions for lectures
as well as entertainment videos.
So improving the accessibility of user generated like YouTube videos,
for instance.
So I think this question you bring up is really important,
and I typically think about it in two ways.
So I think about what what does our computer understand
about an image and then how do we express
what the computer understands about an image and or other form of media?
And so I think that we're getting better and better at
computers
that can understand more of the underlying image.
For instance, we've gotten, if we think about something like emotion,
we've gotten a lot better at determining exact landmarks on the face
and how they how they move, for instance,
or we might be able to describe something specific about a person
So if you look at me in this image,
I have brown hair tied back into a bun and a black turtleneck on and
this is the type of thing we might be able to understand using automated systems.
However, the second question is kind of how do we describe what we know
about an image?
And if I give you all of the information about my facial landmarks
and what I'm wearing for every context, that might not be super useful.
And so a lot of what I think about is sort of how we can best describe
and like what people might want to know about an image
given its context and and the background of the user.
So just briefly on that point, I usually think about
who is viewing this image and what might they want to get out of it
and also who's creating it and what did they intend to communicate.
So there's these two questions, I think give us interesting ideas about what
data we could use to train to create better descriptions based on the context.
So example, for example, we might use descriptions
that are actually given by people to describe their their own images
or their identities or aspects that they've shown in videos in the past.
On the other hand, we might improve,
so we might use a bunch of different methods
and improve our ability to select a method based on the context of the image.
So for instance, when I worked on Twitter images, we would run things
like captioning to describe the image, like an image of a a note.
It might just say note, but we also ran like OCR to automatically
extract the text and tried to pick the best strategy to give people.
You know what we thought might be the best amount of information given the image.
So that's my initial.
I'm sure more aspects of this will come up as as we have a conversation,
but I just wanted to give that as my first part of my answer.
Yeah.
Okay.
Thank you so much.
So Shivam you want to go next?
Yeah, sure. Hi everyone, I am Shivam.
I lead the document based products at mavQ India
and I'm super excited to be here in front of all of you.
So the question here is how should we train models that are capable
of identifying aspects like identity, emotion and appearance in personal images?
So this is a two part answers. So
I'm more in a technical background, so I would go bit of technical
diverse figures. So preparing a data on diverse dataset.
So that is the first point.
So most of our available quality data, it's sourced from a publicly available
data. That's right.
So we can carefully plan and prepare the data before training our models
to include the rates for peripheral peripheral data surrounding environment.
Like in an image, there can a subject and there can be a lot of peripheral data.
So if if we train, if we do the algorithm that take care of that data as well,
that will be helpful in getting a better output.
For example, you have
like subjects gesture, it's relation with the environment
and it's linking emotion to its external manifestation on a subject area.
Now this will give us a more inclusive output
if and you have a subject of the user, a person,
then it will give you a better emotion identity in appearance.
And there should be a thought where we can have a diverse dataset.
Not but it's only depends on availability of data.
Now the second part of it would be fine-tuning your model based on
personal preferences.
Let's say you have a better, bigger model, right?
And you can use that as a general model and then you can fine tune that on
based on a small little on small scale trainings and smaller datasets.
And you can continuously find you need to get a better result.
Now the the fine tuning is kind of a human in the loop feature
where every time you get a data, you can expect some feedback on your data
and then perform a better output of it.
So that's something which is a bit of... includes
a bit of human intervention that yeah, that's how I see.
How can we train models?
Hey, thank you.
Shivam uh, Michael. So,
Michael Cooper, I
work with the web accessibility initiative and I'm speaking
specifically from my role there, I'm not a machine learning professional,
so I'm not speaking about technology so much as some considerations
for accessibility that I'm aware of for that.
So in terms of improving quality of descriptions,
the other two speakers spoke about, you know, technically how we do it.
I think we might be able to give advice on that.
Some of what needs to be done, for instance, machine learning
should... it’s output
should be able to conform to the media accessibility user requirements
and the cognitive accessibility guidance.
For instance, as as sources of of
of information about what would be useful to users.
And I'm also thinking of
machine learning more broadly in terms
of what tools might be used in these different circumstances
and in particular contexts as a potential assistive technology.
And so
the question for accessibility there is not just
what is the description of this image, What is the description of this image
in this page for me, for for the purpose I'm seeking.
So tools can get context from HTML semantics,
accessibility, semantics like ARIA and adapt to technology.
They can also generate their own context from machine learning algorithms.
But I think there is going to be a need
to have a way to communicate
user preferences to machine learning, whether that is added
to the semantics or something and
let's see.
So just a couple closing notes on that.
Users need to be involved in the design and training process.
That's a sort of an aphorism that needs to be repeated.
So we have to pay attention to that as we're looking at improving it.
And I would also note that while this session is mainly focused on
images and media,
virtual and augmented reality has a lot of the same problems
and solutions that we should be looking at.
Okay, thank you.
to the three for starting this discussion.
One of the things that I guess it was mentioned by all of you
and different ways, it's
the role of the end user.
And in fact,
I guess both
users were mentioned, the one that's viewing
or acquiring the the image or the description of the image,
but also the one that's creating or sharing the image.
And and for that
one, there is, the responsibility of generating a description.
And of course, we know that most people don't do that.
So that's why we also need this AI based
assistance to, to, to take on that role.
But this leads me to, to another aspect that if we have
an AI based system that
is capable of assisting both the content creator and the content consumer,
and how can this impact the agency of end users?
So will end users feel that this is no longer their responsibility
because there’s a tool that can do this for them?
Or if we explore this as something that
and now looking at this from the the content producer perspective,
if we if we see these tools as something that helps someone generating
a description, would this
producer just start relying on the on the
the output from the AI and here thinking
about what Jutta was
introducing earlier today wouldn't
and she mentioned this as an organizational monoculture
but can we also think about the description monoculture
in which all descriptions would start
conveying the same kind of information. So
what are your perspectives on
this, on the impact that this has on the agency of end users?
And I'll start with you Shivam now.
Awesome, awesome.
So it is a quite good of a question.
So let's say we are basically talking about the quality of our output
based on a user, the end user, the agency of end user.
Now the quality of these descriptions caption depend on how end users consume it.
For example, most of the models currently provide high level
and grammatically correct caption in English,
but that would not be true for captions generated in a native language of end user
because there might not be enough dataset to train
our model.
Now on the premise of training restricts this diversity of generated captions
and the use cases of what all things an AI model can comprehend.
And then the caption
which includes like a diverse text, like an email, a date or
correctly explaining graphs, which has been really a very big problem
until now and once any translational AI
is employed, how well it becomes an input, it often takes more.
So for example, you can have two different models, one specialized in one of them
and one general.
Now your general output of a model can become an input
for any specialized model and then you can refine it.
This is how now we are achieving it.
That the thing is
the caption denoting AI consumes very large amount of data to curate content.
And then in many cases of live caption generation.
AI should put in context the earlier events or early input as well.
Now this is true for a context of a conversational part,
but this can also be thought to where you have a live caption generation.
So you have to put some context there and then you have to generate the captions.
Now we have mature Indian's legibility, right?
But this is more complex than a simple image to text caption
generation, the speed, the attention, the handling of peripheral data
is very much necessary
and we have these great partnership interpreting
and we are looking forward that we will have a better solution where
end users are really satisfied with what they're getting.
Thanks,
Michael.
What about the perspective from
the end user or the agency of end users
from your point of view from
I guess the more the
the Web accessibility initiative and that role in
how how can we
guide
technical creators to
to ensure that end users remain with
autonomy to to
when creating this kind of content.
So you know first I would
you know, look at you know, what are the ways in which,
you know, machine learning generated descriptions and captions
increase user agency and then there's ways that they decrease it as well.
So you know, for instance,
although
we would prefer that authors provide
these these features, if they don't, providing them
via machine learning will help the user access the page
and, you know, give them the agency they were looking for in their task.
You know, the
you know, the descriptions don't have to be perfect to provide that agency.
That said, it's frustrating when they're not good enough.
They can often mislead users
and cause them
to not get what they were looking for, spend time, etc.
So, you know, that's
a way that this can be a risk for users.
And, you know, as you mentioned,
there's likely to be a tendency for content developers to say,
well, machine descriptions are there, so
we don't need to worry about it
now. So, you know,
I think those are simply considerations that we
you'll have to pay attention
to in our advocacy
in education work in the field
also in documenting
best practices for machine learning.
For instance, the W3C has a publication called Ethical Principles
for Web Machine Learning that, you know, you know, talk about
they address accessibility considerations, among others.
And, you know, it's
possible that, you know, the industry might want
a documented set of ethical principles or code of contact
conduct that industry organizations sign on to saying here's
here's accessibility ethics in machine learning that the
you know, in addition to other ethics we are paying attention to.
So those could be ways that we can support the growth of user agency in the end,
the end of this, yeah.
Thanks.
Thank you for that perspective and for raising awareness
to that kind of information.
That's the WAI group is
is making available.
I think that's that's really important for everyone else to know.
So, Amy, what's your take on this,
the impact that these tools can have on the agency of end users?
Yeah.
So I might answer this briefly from the sort of content creator side.
So say you are out to make a description.
How could we use A.I.
to improve the description?
Improve your quality of descriptions and the efficiency
rather than sacrificing one for the other?
So one. I'll start with what?
Like I worked on tools
a lot in this space, and so I'll kind start with what hasn't worked in the past
and then share some possibilities on things that work a little bit better.
So one thing that I've worked on for quite a while
has been creating user generated descriptions of videos.
Video descriptions currently appear mostly in highly produced TV and film,
and they're quite difficult
to produce yourself because they're sort of an art form.
You have to fit these descriptions within the dialog.
They're they're really hard to make.
So one thing we worked on was some tools to make it easier
for people to create video descriptions by using A.I..
So what didn't work was automatically generating these descriptions.
The descriptions were often uninteresting and they didn't
provide quite the depth of... that
the original content creator had included in their visual,
in the visual information in the scene.
So if the scene was really simple,
like just a house and a tree, sure it might get it.
But if it was something that was domain specific or had something
extra to it that you might want to share, it was completely missing.
And so one thing we looked at is how we could
identify areas where people could add descriptions or silences or
how we could identify things that weren't already described in the narration.
So at this point, the narration of the video talks about
is talking about something completely unrelated to the visual content.
So people might be missing out on that visual content.
So rather than trying to like, automatically generate descriptions,
I think promising approach can be to identify places
where people could put in descriptions or if they write a description, identify
parts of the image that that description doesn't cover yet.
So I think there's kind of some cool opportunities to use
AI in kind of unexpected ways to help people create better descriptions.
And then I'll briefly address the end user part.
You know, if if the user's lacking.
And so the person using the captions or the descriptions,
if they're lacking information that can decrease their ability
to have agency and responding to that information.
Right.
But if you give them all of the information
you in one big piece of alt text, then you might not be giving people
much agency over what they're what they're hearing.
You're probably not matching with the cognitive
accessibility guidelines that Michael... Michael mentioned.
And so I've experimented with some ways to try to like
maybe help people use get agency over automated descriptions.
The one thing we've played with a little bit is, you know, asking
basically alerting people to the fact that there's a mismatch
between the audio and visual.
For instance, in listening to a lecture,
hey, the lecturer hasn't talked about this piece of text that's on their slide.
Would you like to hear more about it?
And then people can optionally hear a little bit more about it.
And that's, you know, something like OCR, which automatically detects
text, works quite well.
So I think there's these opportunities that you don't want to overwhelm people
with information when they're doing a task that's not related.
But there are some cool opportunities,
I think, to like give people control over when they get more information.
Yeah, Okay.
Just and thanks for that, Amy. Also,
just before moving to the next question
that I had here, Matt Campbell
asked a follow up question on this.
So and it's about what you just mentioned, Michael So you mentioned
that descriptions not being good enough are a risk for user agency.
And what Matt's inquiring is how much can this be
mitigated by just tagging the descriptions as automatically generated?
So to
give a perspective on this and also Amy
if you if you want to make.
A try to give a quick answer.
So is the the ARIA technology, accessible rich Internet applications
technology, enhances HTML with the ability to point to a description
elsewhere in the HTML document rather than providing a simple alt text
And that gives you the rich HTML capability.
So so we have that now in terms of identifying it as a machine
generated description, we don't have a semantic for that,
but you know, that's the sort of thing that would get added to ARIA
if the use cases were emerging.
Yeah.
So I will also I'm happy to also answer this question.
Well maybe I was looking at maps other question which is kind of related.
I think so.
Are there other alternatives that are richer than alt text alone?
One thing we've looked at a little bit for,
I've worked a little bit on the accessibility of complex scientific
images, and what you end up with are these like complex multipart diagrams
that if you try to describe in like one single,
you know, alt text field, it performs quite badly.
So we're kind of starting to see like, oh, could we automatically
break that big piece of alt text down into a hierarchy
to match the image so that maybe people can more flexibly
explore like they would basically an HTML version
that sort of captures the structure of the image that people could explore.
So kind of trying
to think about some other ways to present all the information that currently gets
relegated sometimes to a single alt text into something that's a little more rich.
Yeah.
Carlos, you're on mute. Sorry. Thanks.
Uh, and
what I was saying is that since we have been coming
always around to the topic of or to the concept of quality,
also, when questioned by Mark,
Mark Urban I think it's
it would be rather interesting to know
what's your take on this. So
is there a documented metric that measures the quality of an image description?
And if if there is so, what?
What would be the most important priorities for defining quality?
Amy, you want to go first?
This is a hard question for me because I think the answer is no.
But it's really it's a really good it's a really good question
and something that we constantly sort of battle with.
So we kind of abuse in our work, you know, a four point description
that's like no description, like literally nothing, you know, one that's like
there's something in the description field, but it's in no way related.
There is something related to the image, but it's missing some key points.
And this covers most of the key points in the image and we've kind of been
using this, but the what those values mean depends a lot on the domain and what
what task the person is using the image for.
But it's been like... you know we've we've used this in a couple of papers
and it's just been like a way for us to, you know, make progress on this problem.
And we've also tried to for each domain we're working in, kind of tried to inform
it based on existing guidelines as well as like, you know,
the literally the existing W3C guidelines as well as like
what users have told us about specific to that domain.
But I don't know of a good one.
And that's something that like
we just sort of worked around, but I think it would be great to have more
efforts on that in the future.
Yeah, definitely something that's been more qualitative than quantitative.
Definitely.
That's that's what you just described.
It's a good way to start.
So Shivam, in your take on the quality of image description.
I shure, so I guess in when we come to industry
set up, right, we have certain evaluation tools.
We evaluate our models as well as some of the outputs, there’s
a rigorous testing that goes on.
But there's no set of metrics that we have.
But certainly we have some rules, we have the W3C guidelines, we have
we have some other guidelines as well that that are in place.
There are not set rules.
But yeah, we have those as a yardstick and we can really test based on that only.
So there can be some work done with there.
But yeah, certainly this is what we have currently.
Okay. Okay. Yeah. And Michael
hey, Amy just mentioned in her answer.
Looking also at the definitions that the W3C provided... they’re provided.
So do you want to add something on?
How can we measure quality of image descriptions?
The only thing I would really add to what she said is so.
So we produce resources like understanding WCAG,
understanding the web content accessibility guidelines, which
go into when you're writing image descriptions, what are the considerations?
How would you make a good one?
And one of the big challenges I think for machine
learning in particular is that the quality,
the appropriate description for an image will depend very much on its context.
We describe several different contexts in the guide,
in the support materials, and yeah, those are the
the right description for one is the wrong one for another.
So sorting that out I think is going to be one of the big challenges
beyond what others have said.
Yeah, definitely.
I have to agree with you that apparently we're losing
Shivam intermittently, but okay, he is back on that.
Okay.
And I'm going to combine two questions that
that we have here in the Q&A.
The one from Jan Benjamin
and the other one from Wilco Fiers.
So this is most it's more about
qualifying images
than really generating descriptions for, for the image.
So Jan asks if
can AI differentiate between,
for example, functional and decorative images
instead of generating a description, just differentiating between
an image that needs a description and one that doesn't?
And Wilco asks if
if it's viable to spot images where automated captions
will likely be sufficient so that content authors can focus on those
and leave the AI to
to to caption to describe others that might be easier for them.
So, Amy, would you go first?
Sure. Yeah.
So I love both of these questions.
So I would say to Jen's question, I don't think,
you know, I guess when when the question is, can AI do this?
You know, I have we've tried this a little bit, four slide presentations.
And the answer is yes.
To some extent it's going to fail some places.
But just to give you kind of an idea of how,
you know, AI could maybe help detect decorative
from non decorative from more informative images
like in the context of a slide presentation is you know,
informative images might be more complex, they might be more related
to the content on the rest of the slide and in the narration. Informative images,
they might be larger on the screen, whereas decorative images and slides
might be like, you know, like little decorations on the side.
They might be logos or, or like emojis or less
related to less related to the content on the on the screen.
So what we found out is
we can do a decent job at this, but it will fail in some cases.
Always like,
you know, maybe an image is included, but there's no other information about it.
And and so it's it's tricky.
I think in doing this, you would want to be overly inclusive
of the images that you identified as informative.
So so that maybe you could help content authors make sure that they at least
review most of the images.
And then I would say
to Wilco, I yeah, I think that's a great idea.
We've tried it a little bit on Twitter.
So one time we ran basically a bunch of different
AI methods to try to describe images on Twitter.
And so for each image we would try to run captioning OCR
we did like this URL tracing to see if we could find a caption elsewhere
on the web and basically if all of those had like low confidence or
or they didn't return anything, then we kind of automatically sent
the image for it to get more human written descriptions and another thing
we explored with is like users optionally like retrieving that description.
So I think it's possible.
I think that the like the subtleties, there's subtleties there
that would be really difficult to do automatically.
But, but at least that was the way, given how many images were on Twitter without
description, that was sort of a way for us to filter out the ones
where we definitely needed to get more information from a human.
Yeah, great.
Thanks for sharing those experiences. Shivam...
Yeah, I guess
I have had an encounter with this scenario where
I had to get descriptions of images that
most likely not get very sufficient on a machine description.
So there are always there are tools that can do that for you
there on websites.
I think there are multiple plugins that you can use.
You can get certain descriptions and people can put certain human descriptions
out there
to mark them, to spot them in a scalable manner.
It sometimes doesn't become scalable and that's the whole issue.
You can have a tool.
It might not be scalable for every user out there, every website out there.
So this can be done.
But yeah, again, there are some things that where it can used it can't.
So there's certainly this technology is the answer. How to scale
it is the question.
Great, thanks Shivam.
Michael do you have any input on this.
No not on this one. Yeah.
Okay.
Um, that, that takes me back to one question that I had here.
Uh, and, uh, I think these opportunities
to go back there and I will start with you, Michael.
Uh, it's, uh,
going in a different direction from what we have been going so far,
but How do you think that
we need to deal with
legal copyright and responsibility issues
when generating descriptions
with AI-based models?
How do we tackle that?
Yeah.
Okay.
So, you know, also, we're not speaking
as a legal professional, but the issues that I know
in general, at least for accessibility, there is often a fair use.
The right to transform content.
But to circle back to that,
you know, so you know our priority but that's my first answer.
But then there are issues around accuracy.
So, you know, if
a machine has generated a caption
or description, you know, how accurate is that description?
Who knows how accurate it is?
You know, and also publishing it, especially with potential inaccuracies,
can bring on, you know, liability consequences,
even if very useful as otherwise.
Allowing that publication is.
So another challenge is
meeting requirements.
You know, if the accuracy is high, pretty high, but still not quite right.
If it's a legal document, it might not be sufficient.
So either, depending on the accuracy of these kinds of descriptions,
is going to be a big,
you know, legal challenge, I think, from a bunch of different directions.
You know, of course there is the benefit, the reason to do it,
and this can still be better than nothing for many users,
you know, who get used to some of the inaccuracies.
And it does provide scalability,
you know, you know, given how image and video focused our web has become.
So I would
highlight one of
the ethical principles from the other goal machine learning document, which is that
it should be clear that machine, the content is machine generated
that allows many actors to evaluate,
evaluate it and then, you know, circling back to fair use,
I think who is doing the generating or publishing
of of machine learning content will probably impact that. If it's a user
agent and assistive technology probably is covered by fair use.
And if the content producer is doing it,
you know, they probably are declaring fair use for themselves.
But the responsibility for accuracy will be higher for them
because they are now the publisher.
And then there are, you know, third party agents of various
sorts accessibility remediation tools,
other other sorts
where I assume it's a legal Wild West.
Yeah, definitely.
And to make it worse, I guess there are many Wild West
because every every country, every region might have different
legal constraints there.
Shivam,
any take on this?
Yeah.
So I have a holistic view of how technical this has been.
This was when this is an ongoing issue with a lot of countries now.
So you see almost all publicly available data sets, right...
These are the data that are associated in some or other form of copyright one.
Right.
And although there is no frame, most of the part of what
deals with the legality of AI generated captions, I mean,
there is no written law with any place of what currently it might come later.
Maybe in US first, is just so this is a complexity of some other complexity.
Also like owning of AI generated... who would own that data, right?
I mean, if it's a machine generated data, who would be owning the
the industry that has built that model or the dataset that has been
gathered from different data sources.
Now, this is a very complex challenge.
The other part of it is how would you fix the responsibility?
But with that in mind, if it depends on the end user of the
ML model, when you use that,
in what context are you using?
I mean, well, for example, some some of the models are used in
Academy, right.
I know these are just for research and development purposes.
There is no way where you can
fix the responsibility on an academy of an ML output.
Right.
So these are the this this helps in two ways
like there is how you're sourcing the data.
Either you have to get the figures on the data, where it is coming from.
You, you, you gather your data based on written sources.
You have a mutual understanding between the data generator
creator and you, and then you train on the data.
But that gives you a complexity where you have very small data
and there is a large input going and training your data.
So yeah, these are the complexity currently, but yeah, it's all depends on
where the ML model or the output is being used
and that's where the fair use policy comes.
Context all the way in all scenarios, right?
Amy? Yeah,
So I am not as familiar with, kind of like the legal and copyright side of this,
but I do think, you know, oftentimes I do think about like the responsibility
aspects of the captions that we're generating,
especially when we're doing these kind of like new forms of it
where we're generating things for like user generated media.
And I think this more goes back to the
to potential harms brought up in the keynote.
So so for instance, like I guess one thing I often am thinking about is like
when are errors not that big of a deal and when are they a bigger deal?
And then, you know, kind of trade looking at their risks and trade offs
in terms of like who like who's receiving the image and who's or who's
getting identified by the the tool and who is receiving the image.
So, for instance, if I misidentified my shirt as dark blue
instead of black, this error is unlikely to be as harmful to me,
but for some people might experience
misgendering them with image classification to be harmful.
And so I guess two ways I've seen with dealing with this.
You know, not to say that either of them is good right now.
So one is like I think a lot of tools actually back off
to saying person instead of woman or man.
And another way that you could imagine doing it is also like describing
physical characteristics of the person that are less subjective.
And a final way you might imagine doing it is like take... is considering people's
own identifications of how they would like to be described,
and sometimes that varies in different contexts.
So I think that's itself a hard problem.
But yeah, I don't have much to say on the legal or copyright side.
I just wanted to bring up that.
That's something that's come up in my work before. Yeah.
Okay.
Thank you so much.
I think we're almost at the end.
We have less than 10 minutes, but and questions keep coming, which is great.
So you will have the opportunity, I guess, to to try to answer somewhat,
some of them offline if you if you wish to, But I'll still take another one.
The last one that we have here from Antonio Gambabari,
and I think it's
that the question is how do you envision the challenges of explainable A.I.
initiatives in the context of image recognition?
Right.
And I think this relates to several of the aspects
that we've dealt with, with the uncertainty of images
and how do we convey that to users even just by labeling
something as automatically generated would be a way to convey that.
But do you think that explainable A.I.
initiatives have the potential to improve this kind of
augmented context for the user?
And where did the description came from?
And this time, I'll start with you Shivam.
I think yes, and it is a good point.
Explainable AI initiative deals with how
metadata can help the end user to know the context of
what is being generated, any quantitative score on any of the models.
It is supported by a lot of data that goes beyond your training data.
There is a distinction, though, that whatever things you are getting
an output, right, the metadata can there are multiple layers of training.
If you look into training, there are multiple layers of training.
So how that decision has been made by an AI, it can give you
a certain level of metadata, but not all.
So yeah, it can augment the user, but that won't be the complete solution.
But that's how I see.
Amy,
any thoughts on this?
Yeah, so that that's a good question.
I don't, I don't know.
So I think some things that I've, I've seen
so, so one thing I would think about a little bit in this is in
and I've had to think about before is is sort of like
the tradeoff between receiving information efficiently
and explaining where you got all of that information from.
And I think both are important and I think maybe
like I think what my experience has been is that users
are used to certain types of errors and can recover from them quickly.
So for instance,
like when when a user's reviewing their own content, for example,
they took pictures or video and they hear something described is a leash.
I have had the experience of users being like, Oh no, that's my cane.
Like it always calls my cane a leash. So.
So I think in some cases, like people can get like can get used
to identifying the errors for the, for the like known unknowns.
So this is just like a wrong identification, I'm used to it.
And I do think it's harder to recover from areas that are like unknown unknowns.
You don't have any other context about it, so you're not sure what else it would be.
And I think in maybe those those cases where users haven't identified it before,
that that confidence information would be like extra important and so yeah,
I'm not really sure what the answer is, but I think that like
considering the balance between what is the what's important and to know
more information about will will be like a tricky design question as well as
a question for how to develop technology.
Okay, great.
Thanks.
And Michael, any any input on this one?
So I would just add to all that that, you know,
this again, falls into the question of of ethics, transparency
and Explainability is one of the sections of the machine learning Ethics
is intended for several aspects of it.
You should know how the machine learning was built.
It should be auditable for various issues.
These ethics are probably less specific to some of the use cases
we're discussing in this symposium, so there might be room for adding
to this section of the document.
Yeah. Yeah.
I think that might be a good idea.
And I'll I'll take just the final one
and I'll go back to the topic and one from Matt.
And because it's something that we have touched upon before
and I'll start with you Michael here, because we,
you all were mentioning this in the scope of ARIA.
And so it's the question about having richer alternatives to to the image
description, to the standard alt text, which is usually short.
And what are your thoughts on the usefulness
of having richer descriptions
for image alternatives?
Oh. Let’s see
as far as the
general idea in terms of the usefulness of of
of making use of richer descriptions. So
so for very simple
images, you know, sort of the way the web
started, where images were largely providing small
functional roles, you know, the alt attribute
was probably sufficient for many of their cases.
Images are being used in nowadays for a variety of purposes.
You know some of them are reducible to an old like photo of my dog.
But you know, that's not really providing the experience.
So, you know, there's definitely
a need for a richer alternative
and longer alternatives, you know, ones that can have structures,
you can skim them, you know, depending on the context, ones
that you can provide links to the necessary bits of alternative data,
which is...
a question about images and charts.
Often the description for a chart is much more structured semantically
than one for other kinds of images, and that's
you really want to be able to take on it, take advantage of rich text markup. So
I believe that,
you know, assistive technologies are supporting,
you know, rich text descriptions whenever they're available.
So it's a question of getting people to use them more.
And of course, for machine learning, generally, they would rather them
do richer rather than less rich output.
Okay. Yeah.
And following up on that for Shivam and for Amy, by having richer...
richer and longer descriptions,
are we increasing the, the the the chances that
AI generated descriptions will mess up
or isn't that the risk
Who wants to start?
Amy? Sure I think we're definitely yeah I agree
that like oftentimes the more details
that you get, the more
the more opportunities there are for errors.
I think one way that we've kind of explored this
a little bit is seeing if we can bring for for
like very informative images that maybe a lot of people will see.
We've thought about how we could maybe combine
automated tools
with with like human written descriptions
to hopefully make some of the descriptions better.
So maybe automated tools could help you, like help automatically extract
the structure of the image, and then humans could go in to write
more detail about the parts of the images that are really unlikely
to be fully like fully described by the computer.
So so I think for now, the way
I've been thinking about those more complex images is often in like,
how are we going to help humans create descriptions
more efficiently by while still maintaining really
high quality rather than thinking about how to do it fully automatically?
Just based on the images I've looked at in the past year.
OK, thanks and Shivam any input?
Yeah I think the inspiration behind the question would be to give a structure
to the output of any of the old images like so it can be a structure output
make more sense than to we have a fallback estimate right so you're
you can
provide more information to an output but the output would rest
should remain actually shorter and more explainable.
It may be grammatically more correct that would make more sense to the end user.
And he might have one other option to explain that.
It's not like you have a string generated out of an image, right?
When you read out to a screen, right
your screen reader, it should concisely read it shot briefly.
And for more description, there should be some other excellent
data can be supplied to it.
And then there are multiple ways we can do this.
But the description of an ultimate should remain concise and grammatically correct.
So that screen readers can try to read it,
but that's how I see it.
Okay. Thank you so much.
And I want to thank the three of you once more for agreeing to take part
in this panel, also for agreeing to take part in the next panel.
So as we can see, media accessibility, it's really a rich topic and
definitely computer generated descriptions
are also linked with natural language processing.
So what that will be the topic for the next panel
in just under 10 minutes.
So we'll have a coffee break now and I hope everyone's enjoying
and we'll be back at ten past the hour.