And now let's move to the opening keynote,
for which we are delighted to welcome Jutta Treviranus
Jutta is the director of the Inclusive Design Research Center
and a professor in the Faculty of Design at the OCAD University in Toronto.
So, Jutta, the floor is yours.
Thank you.
And if you stop sharing, then I can share my slides.
Thank you, Carlos.
And it's a great pleasure to be able to talk to you about this important topic.
I am going to
just start my slides
and I'm hoping that what you see is just the
the primary slide, correct?
Correct.
Oh, wonderful. Okay.
And thank you, everyone.
I will voice my slides and the information in the images.
And I've titled my talk first, Do no harm.
I'm usually a really optimistic person,
and I'm hoping to provide an optimistic message.
But to realize the benefits of AI I believe we need to first recognize
and take into account the harms.
I'm going to limit my discussion
to the harms that are specific to people with disabilities.
There's a great deal of work detailing the ethical concerns of currently deployed
AI from lack of representation to human bigotry,
finding its way into algorithms to manipulative practices,
unfair value extraction and exploitation and disinformation.
I'll focus on accessibility and disability,
including the recognition that disability
is at the margins of all other justice deserving groups
and therefore most vulnerable to the general and emerging harms,
but also the potential opportunities of AI.
Carlos shared a number of questions that were submitted
by those of you attending today, and they are great questions.
Shari and I have agreed that these will be better
covered through a conversation than a presentation.
So at the end of my talk,
I'm going to invite Shari to talk
about those particular questions, and we'll do the same at the book
ending talk that Shari is giving tomorrow.
So our society is plagued
by more and more difficulties
as the world becomes more and more complex
and entangled, the choices increase in ambiguity,
the risks associated with each decision become more consequential.
The factors to consider in each decision
more numerous, convoluted and confusing, and especially in times of crisis
like we've been experiencing these last few years
and in highly competitive situations where there is scarcity,
AI decision tools become more and more attractive and useful.
As a illustrative example,
it is no wonder that over 90% of organized nations
use some form of AI hiring tool, according to the U.S.
Equal Employment Opportunity Commission.
As work becomes less formulaic
and finding the right fit becomes more difficult.
They are a highly seductive tool. As an employer
when choosing who to hire from a huge pool of applicants,
what better way to sift through
and find the gems and eliminate the potential failed choices
than to use AI system with an AI tool making the decisions,
we remove the risks of conflicts of interest and nepotism.
What better way to determine who will be a successful candidate than to use
all the evidence we've gathered from our current successful employees,
especially when the jobs we're trying to fill are not formulaic.
When there isn't a valid test we can devise for candidates
to determine their suitability.
AI can use predictive analytics to find the optimal candidates.
In this way, we're applying solid, rigorous science and what would be
an unscientific decision otherwise,
we're not relying on fallible human intuition
tools are
even adding information beyond the application
to rule out falsehoods or exaggerations in the applications.
After all, you never know.
There are so many ways to fake
a work history, a cover letter, or to cheat in academia.
The AI hiring tools can verify through gleaned social media data
and information available on the web or through networked employment data.
After all, employees
have agreed to share this as part of the conditions of employment
and other employers have agreed as the conditions of using the tool.
If that is not enough, AI administered and processed
assessments can be integrated
and the tools are going beyond the practical
and qualitatively determinable capacity of candidates
to finding the best fit culturally to make sure that the chosen
candidates don't cause friction but integrate comfortably.
The tools will even analyze data from interviews to rate the socio
emotional fit of candidates.
If that's not satisfactory, an employer can tweak the system
to add additional factors such as their favored university
or to create an ideal persona.
Pick an ideal employee as the model and the systems
are becoming better and more sophisticated in finding a match.
The same system can then guide promotion and termination,
ensuring consistency of employment policies.
So what's wrong with this? Science.
Math. Statistical reasoning.
Efficiency. Accuracy. Consistency.
Better and more accurate screening for the best
fit of the scientifically determined optimal employ
accurate replication and scaling of a winning formula.
It's a very seductive opportunity.
What could be wrong
for the employing organization?
We arrive at a comfortable monoculture that recreates and intensifies
the successful patterns of the past.
With more data and more powerful analysis.
The intended target becomes more and more precise.
The employer finds more and more perfect fits.
What is wrong with that?
For the organization, what's wrong is what happens
when the context changes, when the unexpected happens.
A monoculture doesn't offer much adaptation,
flexibility, or alternative choices.
That's a visual description.
I have an image showing what happened to clone potatoes
during a blight that was survived by a diverse crop.
Of course, we have diversity, equity and inclusion
measures to compensate for discriminatory hiring and increase
the number of employees from protected underrepresented groups.
But even there,
there will be an
even greater rift between the monoculture and the candidates
hired through diversity and equity programs.
What happens to the candidate with a disability who would
otherwise be a great fit for doing the job,
when judged by these hiring systems?
When AI is analyzing, sorting, filtering data
about a large group of people, what does disability look like?
Where is disability in a complex and tangled
adaptive multivariate data set?
Self-identification is often
disallowed and many people don't self-identify.
Even if we had a way to identify the definition
and boundaries of disability are highly contested.
Disability statisticians are acutely aware of some of the challenges.
In any normal distribution, someone with a disability is an outlier.
The only common data characteristic of disability is difference
from the average or norm.
People with disabilities are also more diverse
from each other than people without disabilities.
Data points in the middle are close together, meaning they are more alike.
Data points at the periphery
are further apart, meaning they are more different from each other.
Data regarding people living with disabilities are spread
the furthest in what I call the starburst of human needs.
And as a result of this pattern,
any statistically determined prediction is highly accurate
for people that cluster in the middle, inaccurate, as you move from the middle
and wrong, as you get to the edge of a data plot
here,
I'm not talking about AI's ability to recognize and translate things
that are average or typical, like typical speech or text or from one
typical language to another or to label typical objects in the environment
or to find the path that most people are taking from one place to another.
But even there, in these miraculous tools that we're using,
if we have a disability,
if your speech is not average or the environment you're in is not typical,
AI also fails.
Disability is the Achilles heel of AI
applying statistical reasoning and disability.
You have the culmination of
diversity variability, the unexpected complexity and entanglement
and the exception to every rule or determination.
AI systems are used to find
applicants that match predetermined optima
using large data sets of successful employees and hires.
The system is optimizing the successful patterns of the past.
All data is from the past.
The analytical power tool is honing in on
and polishing the factors that worked before,
and we know how much hiring there is
of people with disabilities in the past.
The tool is built to be biased against difference.
Disability is difference.
Different ways of doing the job.
Different digital traces, different work and education history,
different social media topics, and a tangled profile of many differences
as AI gets better or more accurate
in its identification of the optima, AI gets more discriminatory
and better at eliminating applicants that don't match the optima in some way.
The assumptions these air power tools are built
upon are that scaling and replicating
past success will bring about future success.
Optimizing data characteristics associated with past successes
increases future successes and the data characteristics
that determine success need not be specified or known
to the operators of the AI or the people who are subject to the decisions.
And the AI cannot articulate at the moment
the highly diffuse and possibly adaptive reasons behind the choices.
Current AI systems cannot really explain themselves or their choices.
Despite the emergence of explainable AI, how many of you have experienced tools
like Microsoft Viva or many other similar tools that purport to help you
with be more efficient and productive by analyzing your work habits?
These surveillance systems provide more and more granular data about employment,
providing intelligence about the details of the average optimal employee.
The results of this AI design
is that the optima will not be a person with a disability.
There are not enough successfully employed persons with disability,
but it is more than data gaps.
Even if we have full representation of data from persons with disabilities,
there will not be enough consistent data regarding success
to reach probability thresholds.
Even if all data gaps are filled, each pattern will still be an outlier
or a minority and will lack probabilistic power in the algorithm.
The same pattern is happening
in all life altering difficult decisions.
AI is being applied and offered to competitive academic
admissions departments so you won't get admitted
to beleaguered health providers in the form of medical calculators
and emergency triage tools resulting
in more iatrogenic death and illness.
If you're different from your classification to policing
to parole boards to immigration and refugee adjudications
to tax
auditors, meaning more taxpayers with disabilities are flagged to loans
and mortgage officers, meaning people with unusual asset
patterns won't get credit to security departments, meaning outliers
become collateral damage.
At a community level, we have evidence based investment by governments,
AI guiding political platforms, public health decisions, urban planning,
emergency preparedness and security programs.
None will decide.
With the marginalized outlier, the outliers will be marked
as security risks.
These are monumental life changing decisions.
But even the smaller, seemingly
inconsequential decisions can harm by a million cuts.
What gets covered by the news?
What products make it to the market?
The recommended route be provided by the GPS.
The priority given to supply chain processes.
What design features make it to the market?
Statistical reasoning that is inherently biased against difference from the average
is not only used to apply the metrics, but to determine the optimal metrics.
And this harm predates AI, statistical
reasoning as the means of making decisions does harm.
It does harm to anyone not like the statistical average
or the statistically determined optima.
Assuming that that we know about the what we know about
the majority applies to the minority, it does harm
equating truth and valid evidence with singular statistically determined
findings or majority truth does harm
and AI amplifies,
accelerates and automates this harm
and it is used to exonerate us
of responsibility for this harm.
We've even heard a great deal about the concern for privacy.
Well, people with disability are most vulnerable to data abuse and misuse.
De-identification doesn't work.
If you are highly unique, you will be re-identified.
Differential privacy will remove the helpful data specifics
that you need to take to make the AI work for you and your unique needs.
Most people with disabilities are actually forced
to barter their privacy for essential services.
We need to go beyond privacy.
Assume there will be breaches and create systems to prevent data abuse and misuse.
We need to ensure transparency regarding how data is used, by whom
and for what purpose.
And it's wonderful that the EU is
organizing this talk because the EU is doing
some wonderful measures in this regard.
But wait,
we're talking about a great number
of harms, haven't we
developed some approaches, some solutions to this?
Don't we have auditing tools that detect and eliminate bias and discrimination of
AI and don't we have some systems that certify
whether an AI is ethical or not?
Can't we test tools for unwanted bias?
Unfortunately, AI auditing tools are misleading
in that they don't detect bias against outliers and small minorities
or anyone who doesn't fit the bounded groupings.
Most AI ethics auditing systems use cluster analysis,
comparing the performance regarding a bounded justice
deserving group with the performance for the general population.
There is no bounded cluster for disability.
Disability means a diffuse and highly diverse
set of differences.
Those AI ethics certification systems and the industry
that is growing around them raise the expectation of ethical conduct
that the problem has been fixed, making it even more difficult
for the individual to assert and address harm.
Many of them fall prey to Cobra effects or the unintended consequences
of oversimplistic solutions to complex problems or linear thinking.
Falling into the rut of mono-causality when the causes are
very complex and entangled,
there's
some helpful progress in regulatory guidance.
One example is the U.S.
Equal Employment Opportunity Commission, which has developed the Americans
with disabilities Act and the use of software, algorithms
and artificial intelligence to assess job applicants and employees.
A very long title.
But much of the guidance focuses on fair assessments or tests and accommodation,
not on the filtering out of applicants before they are invited
to take an assessment or by employers who don't use assessments.
The data related suggestion is to remove the disability related data
that is the basis of disability discrimination.
But what we found is that the data cannot be isolated.
For example, an interrupted work history will have other data effects
and markers, making it hard to match the optimal pattern even when that is removed
For the ethical harms there are
that are common to a whole group of marginalized individuals.
There are numerous AI ethics efforts emerging globally.
We've tried to capture the disability relevant ones in the We Count project.
These include standards, bodies, which are creating
a number of standards that act as guidance.
Government initiatives that are looking at the impact
of their decisions using automated decision tools.
Academic research units that are looking at the effects
and possible approaches and think tanks and not for profits.
One of the things that we found, though, is that disability is often
left out of the considerations or the ethics approaches.
And we as the questions that were submitted, indicate
we're at an inflection point.
And this current inflection point we’re at reminds me
of Burke and Ornstein book, The Axemaker’s Gift,
they wanted us to be
aware of the Axemaker's gifts.
Each time the Axemaker offered a new way
to cut and control the world to make us rich
or safe or invincible or more knowledgeable,
we accepted the gift and used it, and we changed the world.
We changed our minds for each gift, redefined the way we thought, the values
by which we lived, and the truths
for which we died.
But to regain
my optimism, even AI's potential
harm may be a double edged sword.
The most significant
gift of a AI is that it makes manifest the harms that have been
dismissed as unscientific concerns.
It gives us an opportunity to step back
and reconsider what we want to automate or what we want to accelerate.
It makes us consider what we mean by Best
Buy optimal truth, democracy, planning, efficiency,
fairness, progress, and the common good.
Some of the things we've done within my unit
to provoke this rethinking include our inverted word cloud,
which is a tiny little mechanism
that conventional word cloud increases the size and centrality
of the most popular or statistically frequent words.
The less popular or outlying words decrease in size and disappear.
We've simply inverted that behavior.
The novel and the unique words go to the center
and grow in size.
We've been trying to provocate with models like the Lawnmower of Justice,
where we take the top off the Gaussian curve or the bell curve
as it might be called, to remove the privilege
of being the same as the majority.
So the model needs to pay greater attention to the breadth of data,
and we're exploring bottom up community led data ecosystems
where the members govern and share in the value of the data.
This fills the gap left by things like impact investing.
For example, when social entrepreneurship efforts
that are supposedly addressing these problems can't scale
a single impactful formula sufficiently to garner support.
It also works well to grow knowledge of things like rare illnesses that
won't garner a market for the treatments and therefore are not invested in
And create,
we're also creating tools to reduce harm by signaling when a model will be wrong
or unreliable, because the evidence based guidance is wrong
for the person being decided about.
Here we're using a tool called the Data Set Nutrition label
that gives information about what data is used to train the model.
But back to the axemaker’s gifts and the opportunity to reconsider
where we're going.
From a complexity theory perspective, where I think we're collectively stuck
on a local optima and unable to unlearn our fundamental assumptions
and approach approaches to find the global optimal.
And I believe there is a global optima
At the moment,
as a society, we believe, or we act like we believe,
to succeed, we need to do what we've been doing
more effectively, efficiently, accurately and consistently.
We're hill climbing, optimizing the patterns of the past, eroding the slope.
For anyone following us, we need to stop doing
the same things more efficiently and potentially reverse course.
I've been considering the many local optima
we keep hill climbing, not just statistical reasoning
that finds a single winning answer, not just winner takes all zero sum game
capitalism and economic growth at all costs, but also majority rules.
All or nothing decisions.
And even in our community, this accessibility community,
the notion of a single checklist of full accessibility
for a group of hugely diverse people, many of whom are not represented
when coming up with the list,
the people closest to the bottom are more diverse,
closest to the path we need to follow to find the global optima.
Less invested in current conventions, we need to diversify
and learn to use our complementary skills
and learn from people who are currently marginalized.
Even in this community, focused on accessibility.
Because if anyone knows, we know that it is at the margins or outer
edge of our human starburst that we find the greatest innovation
and the weakest and the weak signals of crisis to come.
This is where you feel the extremes of both the opportunities and the risks.
One of the emerging uncertainties that holds
both greater opportunities and risks is generative AI.
What are the implications if you have a disability?
What will it do for accessibility?
I'm sure you've heard about tools like GPT, ChatGPT,
stable infusion and various word versions of Dall-E
Midjourney and other emerging tools.
Even today there's new announcements regarding new tools.
These tools do not rely on purely on statistical reasoning.
They can transfer learning from context to context.
They use new processes called transformers
that can pivot to new applications,
but they can also create convincing and toxic lies.
People with disabilities tend to be most vulnerable
to the misuse and abuse of toxic tools.
I'm going to invite Shari to help me discuss
these emerging possibilities.
Hello.
Hello, everybody.
I'm Shari Trewin.
I'm from Google.
And a middle aged white woman
with lots of smile lines on my face.
So, Jutta, you’ve given us a lot to think about there.
I wonder if we might start off where you ended there.
Talking a little bit about generative A.I.
models and language models
and the
they're trained on large corpora of data that may not reflect
the the moral values that we would like
our models to incorporate.
So one question I think would be interesting for us to talk about
is can we teach these large language models or generative A.I.
to apply these moral values,
even though the the very large datasets may not
represent that?
That's a great question.
And in thinking about how that might be done.
One of
the dilemmas is that we may need to find a way
to quantify complex, abstract, qualitative values.
And in that process, will that reduce these values?
I mean, deep learning lacks judgment.
Humans sort of value human judgment that isn't quantitative.
Perhaps one way to start is
by recognizing human diversity
and the diversity of contexts.
There is a lot of talk about
individualizing applications
without making the costs exorbitant
and to the people that need them.
The irony, of course, in that is that the people that need
that type of individualization the most are also most likely
to be the people that can't afford it.
And I think it's not yet known.
Can we do that?
And of course, there's been surprising advances
in all sorts of different areas with respect to AI and generative AI.
But I think this is the issue of values
and shared values and the the articulation
and making mechanizable, because, of course, we're talking about a machine
and mechanization values that that we have difficulty
even fully expressing is it's quite a challenge.
What do you think, Shari?
Now, I
think that you've hit a really good point there with
can we express or can we measure
whether a model meets our values or whether
we think it is free from bias
or as free from bias as we can make it?
Do we know?
Do we know how to evaluate?
That I think is an important question.
And some of the steps that that often get missed
when creating a system that uses AI might
that might help with that would be starting off
from the beginning by thinking about who are the people
who might be at risk, what are the issues that might be in the data?
What historical biases might the data represent and include?
And then actively working with members of those communities
to understand how are we going to measure fairness here?
How are we going to measure bias?
What's our goal?
And how are we going to test?
How are we going to know when we've achieved our goal?
So I think there's some
progress that can be made in the design process
and thinking about the larger system that we're embedding AI in,
everything doesn't have to be built into one AI model
and we can augment models, We can build systems around models
that take into account their limitations and
create a better overall whole system.
So thinking about what the models are currently trained on
and just the masses of data that are used to build the models
and the training data is rife
with discrimination against difference.
So how do we how do they unlearn?
I mean, this is it's sort of it matches some of the training
that I do within my program and that
students have been socialized with very similar things.
And then often the issue is not learning.
The issue is unlearning, like how do you remove those those unconscious
habituated values that that
are so embedded in our learning systems?
So it is I agree that there is is huge
opportunity, especially with more context aware systems
and maybe what we need to pursue
is even to address things like
privacy and the need to swim against this massive
amount of data that is not applicable to you
is on device
personalized or not personalized, because personalized is a term
that's also sort of been hijacked to mean
cushioning but individualized.
Let's use that term system that takes your data
and creates a bottom up picture
of what is needed.
Yeah, I think, you know, there
there is definitely interesting avenues to explore with transfer learning.
And can we take a model that's been trained on data
and has learned some of the concepts of the task that we want, but
maybe we'd like it to unlearn some of the things that it's learned.
Can we use techniques like transfer learning to layer on top
and and teach the model and direct the model
more in the direction that we want?
And the I think the the hopeful thing about that
is that it needs orders of magnitude less data
to to train such a model.
And so that makes it
a little more achievable, a little less daunting for
for the community to take on.
Yeah.
Do you think that current
regulation systems are really up to the task of
regulating current and emerging A.I.
and preventing the kinds of harms that you've been talking about?
No. A simple answer.
I don't think so.
I mean, there's so many issues.
Laws and policies are developed at a much slower pace
They’re.
We're dealing with an uncertain
very, very quickly moving, quickly adapting area.
And when laws are
well, they need to be testable.
And so in order to be testable, we have to create
these static rules that can be tested, which means we have to be
fairly specific as
opposed to sort of general and abstract.
And that tends to lead us towards one size fits,
one criteria which we know are are not great.
If we're trying to design for diversity or encourage diversity.
I think we, one of the things we need to innovate in
is the regulatory instruments that we can use here.
What's your thinking about this?
Yeah, I think some of those
regulatory instruments that we have do apply.
So if you're a company that's using a
a AI system in screening job
applicants,
the disability discrimination laws still apply to you.
Somebody can still bring a lawsuit against you
saying that your system discriminated against them and you are still liable
to defend against that and to watch out
for those kinds of issues.
So in in some ways, there are important pieces
that we need in place that can be used to tackle problems
introduced when AI systems are introduced.
But then in other ways,
there was a lot more of a gray area.
When the technology is not making
discriminatory decisions, but
it still might make harmful mistakes
or that mislead people
or that people are relying on it for.
And so, you know, if anybody here has a legal background, I would love to hear
their take as well on
how well do current consumer protections apply.
For example, if you're using an AI.
I've become aware of and worried
about the people for whom the law isn't adequate.
So the fact that we have a law, the fact that we supposedly have measures
that prevent abuse or unethical practice,
if that if you are still
being treated unethically, then it makes it even harder for you.
So I think the the measures that we do have, the regulations that we do
have have to have some way of continuously
being iterated upon so that we can catch
the individuals that are not included.
And we have to recognize that
our supposed solutions are actually not solutions,
that this is never fixed, that it's
it requires this ongoing vigilance.
And so the yeah, there's that there's much more to say about that.
But yes, you're right.
It would be great to hear from any
anyone with a legal background.
Yeah.
Let's maybe talk a little bit about it's
a bit more about generative AI
that you mentioned at the end there.
Know it. It
produces very plausible convincing
statements when when asked the question,
but it also very plausibly and convincingly,
completely makes things up and isn't always reliable.
And in fact right now is not connected
to any form of ground truth or
able to assess the accuracy of what makes
the one question I think that's interesting is
will this technology reach a stage where
it can support
the kinds of decisions that we're using, statistical reasoning for right now?
Eventually, obviously, right now it's it's not there yet.
Yeah.
And it's interesting because just recently there have been the announcements
of these systems being used for medical guidance
using large
language models to come up with answers to your medical questions,
which of course is
quite.
Yeah, it'll be interesting to see what happens.
It's scary, I think is exactly scary.
And what about the the the medical advice given to someone where
within the data set that is provided, there isn't a lot of advice.
So that given that the system does
I mean ask any of the LLMs or the chat bots
how confident they are in their their answers.
They'll answer that
they are confident because there isn't a sense of
what is the the risk level,
what is the the confidence level of this particular response.
There there's no self-awareness of what is wrong
and what is right and what is the context that is in front of me.
Yeah, I think that's actually a great opportunity.
There to explore whether we can enable
models a little better to know what they don't know,
to know when the case with right now
isn't well-represented in their models or maybe an outlier case.
That they
should perhaps pass on
to some other form of decision making or at least convey
the less confidence
in the voice of their, you know,
I think generative AI today
gives us a glimpse of the future and the kinds of interactions
that are possible, the kinds of ways we might interact with technology
in the future.
You know, clearly there's a research priority to ground it better.
In truth, and
it needs to
be much more reliable, much more trustworthy, much more accurate.
But then you go to the it can support serious applications.
And the idea of using it to get medical advice just is
that that's a very, very scary
because it is so eloquent
that it's immediately trustworthy and it gets enough things
right that we begin to trust it very quickly.
And so in some ways, the the
the advances that have been made are
it's it's so good
that it really highlights the dangers,
I think, more effectively.
So, yeah, I think that's interesting to think about
what would a human A.I.
interaction look like in the future?
Would would we need to change train
or was one to divide by bias
and kind of work with a larger language
model to adapt responses?
Would we you know how
automatic image description has sort of evolved?
At first it would
you know, we would throw out words and that might be in the picture.
And sometimes it was right and sometimes it would be wrong.
And now you see these generated alternative text
being phrased in a way that conveys the uncertainty.
So could be a tree or something like that.
And I think the large language models could do something
similar to
reduce the chances of misleading people.
So they might say things like many people seem to think
blah, blah, blah, or get better at citing sources.
I think there's a lot of ways that we can use
these in direct research to,
you know, to overcome some of the really obvious failings that are there right now.
But the limitations that we currently have,
Mark Urban has shared in in the chat that I can see
from the US government regulatory side,
much of the current laws
or regulations for access to government services are about the technical
accessibility of the interfaces rather than the more
AI focused questions around system exclusion or mismatch.
So that's
coming back to our point about the regulatory systems.
And I just noticed that Mike Calvo says what a Debbie Downer
my talk is, which
I think by design,
we decided between Shari and I that
I would provide the warnings
and then Shari would provide optimism.
I say I get the best of their.
I think there are
quite a few questions in the question and answer panel. Yes.
And so maybe what we should do is...
Over there's so many things
to explore with the emerging models and so many uncertainties,
but there's some great questions there as well.
Yeah, How about
jumping around on me...
So many new questions.
So I know this is not in the right order, but
as people are adding questions, they're kind of
they're kind of jumping.
Okay, so Bruce Bailey is asking.
He says, Fantastic keynote.
Please expound on personalization having been hijacked
to mean cushioning
time and perspective as new.
Sure.
Yeah, I can definitely talk about that.
I mean, one of the ways in which
we recognize that we're all diverse, and especially
if you have a disability, you're diverse from other people with disabilities
and that our needs are there for
a diverse has been to look at how do we personalize.
But personalization has been used
as a term to look at
using recommender engines, using
various ways in which we’re offered
only information and recommendations from people like us,
which of course removes any dissonance and any diverse
thinking and our exposure to alternative views and perspectives.
And to some extent it causes us to
it causes greater polarization because we're also
offered a personalized view
of the current stance that we're taking
so that it gets confirmed again and again and again.
So I'm not talking about that type of personalization.
I'm talking about the type of personalization where the interface
makes it easier for us to participate and addresses
our specific, very diverse requirements with respect
to that participation so that
I've moved away from the term personalization simply
because I don't want it to be mistaken for the type of personalization
that cushions us away from diverse perspectives.
Because certainly we need to be exposed to those that diversity of perspectives,
and we need to consider
the diverse stories that people have.
You know, I think personalization, personalization is
really part of accessibility in general,
but there's
you know, you're talking about a particular kind of personalization,
but AI personalization.
I'm going to talk a little bit more in the keynote at the end about an example
of AI personalization of personalized models
that are
providing personalized access
to digital content, which I think is a good use of personalization.
Yeah, Yeah.
So Carve or Convey
from EDF says thank you Jutta for this important keynote.
I've seen different toolkits to test and mitigate bias in AI.
What is your view on them and their usefulness?
Right. So we've been doing a
well actually
as part of a number of our projects, including ODD, which is optimizing
diversity with disability and our We Count project,
we've been looking at a variety of A.I.
ethics, auditing tools, but also we've done sort of the secret shopper
test of employment tools
and then seen whether we can detect the the particular biases
that unwanted biases as is is made clear by AI.
So I mean bias of course, the tools are intended to be biased.
And so it's the unwanted bias as a proviso.
And what we find is that
they're great at cluster analysis
and then they they supplement the cluster
analysis with a number of questions that is
asked of the implementer of the system.
So the primary technical key to to
the tools is determining whether there is unfair
treatment of one bounded group with another.
And that works well if you have something like
determining whether there's discrimination regarding gender or discrimination
regarding declared race or language or those sorts of things
which do cluster well, but it doesn't
none of the tools really detect whether there is discrimination
based upon disability.
And and the
because the the particular
discriminating characteristics are so diffuse and different
from person to person, we can't see how it would be possible
in a litigation perspective
or in a regulatory perspective to prove
that you have been discriminated against it.
It's going to be very, very difficult to come up with that proof
because the the particular characteristics are
themselves so entangled and diffuse.
And so it may not be one particular characteristic associated
with your disability that you would use to say,
well, look at here, I'm being discriminated against
because of this characteristic that relates to my disability.
Yeah.
So I think there are a lot of the toolkits.
Many of the methods in the toolkit are group fairness
metrics like, like you say, where.
And that's an important thing to measure.
It is when we when we do have the ability
to identify groups and to know for sure who's in which group and which one,
the boundaries of these groups are always not fuzzy.
You know, there's the there's
a deeply embedded assumption that there's
only two genders, for example, in a lot of the data.
And many of these tools.
So they have their problems and this ability exemplifies this.
The same problem. Yeah, same problems.
But there are also individual fairness metrics and measures,
and some of the toolkits include some of these kinds of measures.
And so instead of asking, is this group as a whole,
treated equivalently to this other group, they ask
is a similar are similar individuals treated similarly?
And so you could imagine with an approach like that,
if I as an individual with my unique data,
I could make a
case that I was discriminated against by
creating another person
who was similar to me in the respects
that are important for this job.
Mm hmm. Yeah.
And. And see what kind of result they got compared to my result.
And that would be, you know, a way to measure
individual fairness and and build up a case.
Yes. Yeah.
Unfortunately, there's not that many schools that that currently do that.
And the certification systems that currently exist
are not implementing those so that there is much to work on there.
Yeah.
Think it's sort of more of a case, a
case by case basis,
but this particular job to me
so it's not so easy to make a blanket statement about it,
but I think it's not impossible to
assess.
Okay, so
do we have time for one more?
How long?
How much longer do we have another three minute?
Well, you have almost 10 minutes more,
so you can definitely take one more.
Awesome, great. Um,
let's see.
So Fabian Berger
says, I feel that AI
but before it was KPIs or else are
are searched by managers to justify their decisions or run away
from the responsibility of their decisions.
It follows a need for them
but with a wrong incomplete answer.
Do you agree? Yes.
I mean, I think that the issue
and I was trying to make that point, but possibly not well enough
that AI is doing much of what
we've done before, but
it's amplifying, accelerating
and automating those things.
And certainly AI can be used to
for confirmation bias to find the specific
justification for what it is
that we need to justify whether it's something good or something bad.
So the a lot of the the harms of AI
already existed
because of course AI is learning
our past practices and our data,
but because it
I guess I've often used the analogy of a power tool
before it was this
practice that was not that we did manually.
And so there was an opportunity to make exceptions,
to reconsider, you know, is this actually what we're we're doing?
And to, to do something different?
But with the with a power tool, it's
it becomes this much more impactful thing.
And there's less opportunity to
to craft the approach that we take.
Yeah, I think that's why it's really important to
to try to design for outliers
and consider outliers in in and again
I come back to this point of a system
that the system as a whole that includes AI if we can't
guarantee that the AI
itself is going to give us the
characteristics that we want, then we need to design around that
and and be mindful of that
while we're designing.
There's also, of course, the opportunity to try to clean up our data in general
is there are in situations where we can identify
problems with the data, we should certainly tackle
that or imbalances in the data.
We should certainly tackle that.
That's one you know, one other step that
I think there's many steps to fairness and
and to ethical application of AI.
And no one step
is a magic solution to all of that.
But if we
stay aware of the risks, make sure we're talking
to the right people and involving them, then I think we can we can at least
mitigate problems and know
the limits of the technologies that we're using
better.
Yeah, I've been looking at some of the other questions that have come in
and one of the discussions was about the Gaussian curve and or the Gaussian center.
And one thing I think that
one point that I may not have made as clearly
is that in fact the the myth
that we need to have a single answer
at the very middle of the Gaussian curve, which of course matches our notion
of majority rules or as the way to decide amongst
difficult decisions, it
an alternative to that is to address
the very, very diverse edges
initially and to prioritize those.
Because what what then happens is it gives us room
for to change.
It helps us address the uncertainty.
It makes the whole design or
decision or options that are available
much more
generous and therefore prepares us better
for the vulnerabilities that we're going to experience in the future.
So that,
of course I'm an academic, and to say that
that statistical reasoning and evidence
through scientific methods is at
fault is is a fairly dangerous thing to say, especially during a time
when truth is so much under attack and
but I think what we need to do is not reduce truth to statistical reasoning,
but to acknowledge that there are
a variety of perspectives, untruth, and that we need to come up with one
that that addresses the people that we're currently excluding
in our notions of truth.
Yeah, So there's 2 minutes left.
I, I think now
maybe we could squeeze in one more
question here.
Jen Benjamin
asks, Do we think
that AI and big companies driving research on it can be problematic towards
societal issues which don't necessarily give the highest revenue?
And if so, how can it be faced?
Yeah, that's a huge question.
I think even government
efforts are facing in their decision making on profit
and economic progress and
and impact measures that are
so that I think one of the things we need to abandon
is this idea that a solution needs to be formulaic
and that we need to scale it by a formulaic replication.
And we need to recognize that
that there's a different form of scaling
that is by diversification,
and that we need to contextually apply things.
And I mean, that's one of the lessons of Indigenous and indigenous cultures
that the
what, what has been labeled as colonialist is, is actually what many governments
are still implementing, even in things like social entrepreneurship.
But yes, big companies, of course, are driven by profit and
is that the best approach
to achieve the common good?
That's a huge question.
I think.
And I think this would be a great one to come back to tomorrow, actually.
Yeah, exactly.
I mean, the symposium let's let's come back to that one
because I see that we're out of time right now.
But thank you very much.
Thank you.