Okay, so, hi Jen, would you first like to start off by introducing yourself? You can tell us your name, the degree you have, where you currently work, and where your research interests lie.

Yeah. And hi Jessica. Thanks for chatting with me. So my name is Jennifer Aguiar. I have my PhD in biology from the University of Waterloo where I was focusing on using bioinformatic tools and techniques to study various respiratory diseases. And I am now working as a bioinformatician at the Hospital for Sick Children in their department of Pediatric Laboratory medicine. And there I am focused on pediatric cancers as well as Mendelian rare diseases.

Thank you for the introduction. So today's topic overall will be about bias, clinical research data, and how it affects healthcare practices. Our interview will consist of six specific cases where clinical data led to AI systems making incorrect decisions. And they specifically stem from either gender or racial biases. The first three cases will focus on gender based biases, whereas the last two cases will focus on racial biases. And overall, the interview is kind of formatted in such a way that at the beginning of each of the six cases I'll give you very little information about the case and to try to bias you in some sort of way. And this will kind of give you the understanding or the point of view that the AI had the limited information it has. And then you could understand the bias that it came to and the bias like how the biased AI decision came about. And then we can analyze after, once we identify the bias, I'll summarize the entire case with all the details with you, and then we can talk about the impact that the biased AI decision had and how we can move forward to try to prevent this bias or this decision from happening again for future patients. And then at the very end of the six cases, I'll ask you just some overarching questions based off just all the cases and just talking about the whole idea of these biases in general and how it affects marginalized communities and what you would suggest or include in similar systems if you were test to test to design one.

Okay, sounds good.

So the first case I'll talk about, ask my first question. So imagine you're a physician and you're treating a patient with severe abdominal pain, but you see no visible issues. What diagnosis or what kind of things would come to your mind first or additional information that you would seek. Just having this limited information.

So I guess just to clarify this limited information. So I'm. Yeah, I think if I'm. If I don't know what the sex of this patient Is I not? I guess I would probably err on the side of quote, unquote caution and assume that it is something to do with the gastrointestinal tract, something along that line. So I'd probably be going down that route. But in my mind I can already there's like a red flag there of like.

And then now I'll give you a bit more information. That's. What if the patient was a woman? Is there any sort of symptom that you kind of go towards them?

Yeah. So I would say that if I knew that the patient was a woman or diagnosed female, diagnosed assigned female at birth, I would, I wouldn't necessarily all of a sudden rule out any sort of like gastrointestinal ailment. Like, I would certainly want to still investigate that. But I would all of a sudden now be bringing into it might have something to do with her ovaries. Like, I'm thinking maybe if she's having abdominal pain, she might be having some sort of like ovarian cyst or perhaps it's not even that serious. And it' um, but depending on the age of the patient, like whether she's pre her menstrual cycle, whether she's of childbearing age, whether she's like post menopausal, there could be various things going on there. But yeah, I would definitely start bringing the, the more like female centered anatomy into the equation.

Yeah, that makes sense. And then how would your thinking change if you were told the patient was a man?

If the patient was a man and they were being. Or they were coming to me with abdominal pains, I suppose I would probably stick with my original first line of questioning or first line of interrogation, which would be the kind of like gastrointestinal something to do with the GI tracked. That would certainly, I think, be my, my first go to still.

Okay, yeah, that makes sense. And then now, last question before I actually tell you about the case. What if you, what if you knew the patient was a transgender person? How would that change your clinical reasoning? Would there be some challenges that arise? Would you approach things differently.

When you say a trans person? Do we know if they're female to male or male to female or.

Because they're transgender man in this case?

Okay, so if they're a transgender man, I'm going to just assume that they were assigned female at birth. So I would then again start bringing in. It is possible that there are still like the, the female sex organs or things like ovaries that may be coming into play. Might depend on whether this person is, if they've had any sort of like bottom surgery, like if they're pre op, post op, what sort of potential, like hormone replacement therapy they might be on, like, all of that could be affecting what is going on there. But if I'm just going, if I don't know any of that, and I'm just assuming, like assigned female at birth, trans man, I'm going to assume that they still have intact ovaries and uterus and things like that. And so I would still probably investigate that hopefully a bit more sensitively given that that might cause some sort of gender dysphoria for the person. But, yeah, I would, I would start bringing that back into the equation now.

Okay, yeah, that makes sense. Yeah. You gave like, out of all the people I interviewed, I think you're the most clinical, so you give the most, like, in depth, accurate answer. I think out of everyone. Now actually summarize the case and what happened and we'll talk about the implications of it and maybe how we can prevent it from happening in the future. But the case is there was a 32 year old transgender man that arrived at an emergency room with severe abdominal pain. Although he informed staff that he was transgender, the hospital's electronic health record system had him listed as male. As a result, clinicians failed to consider pregnancy and misattributed his symptoms to factors like obesity. The patient was in fact pregnant and experiencing labor complications. But due to the delay in the clinicians recognizing this, urgent care was postponed and the baby was tragically stillborn. This case highlights how rigid binary classification health records like male or female, combined with assumptions from the clinicians themselves could lead to critical misdiagnosis. And standard pregnancy related alerts would never be triggered from, for example, AI systems, because all the AI system knows is that the gender is male and they don't know if there's a possibility that they're transgender or any other information. And so then standard pregnancy related would have never been triggered. And this kind of highlights the dangers of algorithmic biases and the need for more inclusive healthcare systems that reflect the reality of transgender and non binary individuals. It's in today's day and age. So then my question to you is that in your opinion, now, knowing that the person was transgender and pregnant, what role do you think these like, biases, system designs and other factors you could think of played in the failure to identify pregnancy risk in this case?

Yeah. So I think it's funny because when the, when you immediately started talking about, okay, now I'm going To tell you the case, my brain kind of went, huh, I guess, like, if we're talking about like women or somebody who had like a uterus, like, I wonder, I guess they could be pregnancy. But without any other information, I was just like abdominal pain. Like, it's not my first thought. But yeah, I think that's very interesting because on the one hand I'm thinking that's quite good of the hospital or this healthcare provider that they are using the gender marker that this person has requested, like that is going with their gender identity now. So I'm glad that they're saying like this person is a male because, because they are. But the fact that they're, you know, in a medical record, I think you do want a bit more detail. You do want some way to say, like, yes, they are male, but there is this other information of assigned female birth. They have transitioned. That is, that is massively relevant because, yeah, like you said, I think an AI is going to not like it's only going to be able to based on the information that you've given it and based on, you know, all the past literature and everybody kind of only now starting to come to terms with certain things. Like, I imagine that the AI is basing this on a. There's a gender binary. There's male, there's female. Males don't get pregnant. So not even considering it as a possibility. So that is, yeah, I think a very interesting case. And I think it's. It kind of makes me think of the. My, my sort of overarching thought about a lot of this is just like the, it's the garbage in, garbage out. Like if you, if you don't tell the AI certain information, like, it's not going to know.

That's kind of how it is for a lot of these cases where I think the AI wasn't given good enough data and thus it could make good enough of a decision. Another next question I have for you is what changes would you suggest, whether it be to electronic health records, triage systems, or even clinician training to prevent similar outcomes for transgender patients happening in the future? For example, an idea could be, you have two genders. One would be a sex stand of birth versus the gender you currently identify with. Maybe that way you have more information that an AI system or even just a clinician right away knows to consider from both perspectives. But do you have any other maybe suggestions or maybe clinician training? Because I guess in this case they did the person to identify that they are transgender, but still the clinician maybe didn't take the problem as seriously. Is there any suggestions you have?

Yeah, so I think that one, I think the suggestion of either having a sort of category or an area in a patient's chart where there's like sex assigned at birth and then a category in their chart that is their, their gender, or even having like, I know it might be one of the doctor's offices I go to, where they ask if you're comfortable, would you like to fill out? And you say that you, whether you are woman, trans woman, like, like CIS woman, trans woman, CIS man, trans man, non binary. Like they have several options for you and, and then there's just a lot more for the individual to choose from, which is great in terms of like an inclusivity standpoint. But then if that's the information you're feeding to the AI, it's a, there's a lot more for it to, to, you know, digest and learn from. So I think that's definitely one of them, but then I think the other is, yeah, some of it falls, I would say falls to the clinician because whether or not the AI has the correct data, ultimately the AI is not the doctor here. And so that clinician should have been taking that information into account and that should have broadened their horizons of what type of issues could have been possible. It, like, I'm not saying that this clinician needed to immediately go, ah, okay, so you're probably pregnant, like, but I would say they should have at least inquired as to whether that could be a possibility. Is this person sexually active? Have they recently had intercourse? Like, is there any reason that they would potentially think that that is a possibility themselves? Like, I would at least just be a bit more curious, I guess, as a clinician, ask a few more questions rather than just going down the, oh, no, that's not it.

Right, Yeah. I think it's also a good point that you mentioned of like all those different gender identities that could be good on one hand, but on the other hand you have to have enough people in your data set that are representative of each of the different genders. So that, that way the AI can actually make connections between different genders and different conditions. So it's kind of like a balance between having a few groups that you can have a lot of data for or a lot of different groups where you have a lot less data for each of them.

And then, yeah, that's definitely the sort of paucity of data, I'd say even, you know, just between CIS men and CIS women or different, like Racialized groups. Like we, we definitely already have a lack of data on a lot of individuals and groups. I mean so. No, that is a good, a good point as well.

And that's it for the first case. So move on to the second one now. So let's imagine a 59 year old patient presents with sudden chest pain and some nausea without knowing their gender. What would be your next, what would be your top diagnosis and what would your next steps be?

I. Without knowing their gender, again, it's not like I can already feel the biases in myself because without knowing their gender I'm immediately going to go to a more, a more male diagnosis which would be, I would, I would probably think maybe they might be having a heart attack.

That's a good guess. And then now for the next question. Let's say like regard, you know, they're 59 years old. Would you take the problem seriously right away knowing that it's chest pain and 59 year old or would you think it's not that serious or would you be thoughts in terms of severity?

That's, that's tough. I think not being a clinician, I, I can't say how I would respond for sure. I want to believe that I would be taking it seriously regardless of age. It's not as though, you know, cardiovascular disease is only an affliction of the elderly or it's only an affliction of young folks. So I want to believe that I'd be taking it seriously regardless, but very hard to say.

Yeah. And now let's imagine that, you know, the patient is woman is a woman and you're told statistically women are less likely to have heart attack than men. Would this influence your decision in any way? Do you think it should influence your decision?

I, I don't necessarily believe it should influence my decision because while, you know, trends and statistics are very helpful, we obviously like human beings, want to find patterns in things. I would say this individual, I need to kind of, to the best of my ability, treat them as an individual, as a person and they are not a statistic. And so just because perhaps broadly speaking women might be more like less likely to suffer from cardiovascular disease or heart attacks than men, then it's not impossible. So I wouldn't want to just rule it out myself. Knowing some of what I do know, I, I would probably, if I knew it was a woman, start inquiring about other symptoms because often heart attacks can present differently in women. They'll like. I think the stereotypical for man is yeah, the chest pain the, the left arm pain, all of that. A lot of times women get back pain for, like, no reason. So I'd maybe start inquiring about other symptoms. They might have to help me decide whether or not I should be ruling out a heart attack or not. But I, I don't think I should just be automatically ruling it out because, oh, it's less likely. It's still possible.

Yeah. I think even when I was talking to Pedro about this question, he said, like, the statistics, but, like, statistics, if you apply them directly to, like one person, he seems like 99 of the times wrong because. Yeah, just like a general statistic. So based off the actual symptoms that you see, you can actually make a much better diagnosis than just statistics, which is kind of what AI always does.

Yeah, exactly. Which again, it's like, it's what it, it's what we're sort of training it to do and asking it to do. We're not asking it to sit in the room and like, learn about this person.

Yeah. And now kind of the opposite side. So let's say the patient was a man, and you know that men are statistically more likely to have heart attacks than women. Would this change your sense of urgency in any way? Would you, like, treat, like, if it was a man versus a woman with the same symptoms, would you treat the man, like, man's case more seriously or, like, would you have a more sense of urgency? Would it be kind of the same?

I think it would. I think it would be the same. Or at least I would very much hope it would be if I was in that position. Because I think oftentimes a lot of where our biases come from is that we, we do treat certain people's ailments with more seriousness and more urgency than others. I would say, again, like, because cardiovascular disease and heart attacks can be more prevalent in men. Like, again, I would, I would want to be exploring that route, but. But on the other hand, I'm. I'm thinking, you know, mental health is often, like, not talked about as much with men. It's under diagnosed. This person may very well be having a panic attack and they just don't know because they don't talk about mental health and anxiety. So they're. It's like the reverse coin of. For the woman, I wouldn't want to rule out a heart attack, but I don't. I'd obviously explore other things for this man. Again, like, yeah, it might be a heart attack, but I also maybe want to explore some other things, options for them as well. It's a, I think again just the, you kind of have to go in with the, for the last, for the last one I was sort of like oh, I feel like you have to treat them as an individual and as a person. Which of course, but also part of me is like with more of the stats brain. I feel like you, you have to treat them with like what you're seeing. So like just like raw facts, like take the, try to take the biases out of the equation. Like what have they told you that they are feeling? Go with that.

And now I'll summarize the case and then we can talk about what happened. So the case was that there was an AI powered symptom checker app and it provided HSC different recommendations for a man versus a woman with identical symptoms. Both of them were 59 year old smokers reporting sudden chest pain and nausea. The only variable that was different between them was gender. The male patient was warned of potential heart issues including unstable angina or heart attack and advised to seek emergency care, whereas a female patient was told that her symptoms might be due to depression or anxiety. With no urgent care recommended. The discrepancy in the AI highlights how AI tools can replicate and amplify gender biases, potentially leading to delayed treatment or misdiagnosis. In this case, the AI appeared to rely solely or mostly on statistical trends that underrepresent heart attacks in women, overlooking the real and serious risk of cardiac events in female patients in this case. The public concern grew after this example came to light as it revealed how gender stereotypes that women are anxious or men have heart attacks more can shape algorithmic decisions ultimately. Case Topics the critical importance of designing healthcare AI systems that recognize and adjust for biases and life threatening conditions should always be considered regardless of gender, especially when the symptoms are shared between different demographics. And so my question to you is how should AI triage systems handle gender based statistics, statistical differences and should it present like for example, should it present all possibilities whatever you ask it, like oh I have chest pain and nausea. Should it list all possibilities of different things that you could get? Should it just give you the most statistically highest chance, like the most statistically probable thing that you probably get? Or should it give like an entire possibility and especially like highlight that there is extreme like even though something is very unlikely, it is still possible if it's something that's like extreme, like for example like a life or death situation, should it always be like telling you this should give you all possibilities or should it just give you the Most probable possibility. What are your thoughts on that?

I guess I do have one question. Is it when you're like a healthcare triaging, the AI is giving out these possibilities, the person who's reading them, is that the clinician or is that the patient? I'm going to assume clinician.

Yeah, goes in there.

So I was going to say in that case I'd probably go with all the possibilities even if they are high risk. If it was the patient, I was going to say, ah, we're getting into WebMD territory where you're going to tell somebody they have a headache and that means they've got brain cancer. Like you don't want to freak the, freak the patients out. But I think assuming and trusting that the clinicians are competent and are going to you follow up with whatever the AI is spitting out at them and they're not just going to like take it as fact. As long as we're going to apply some critical thinking, I would probably suggest that the AI offer up all of the possibilities and be like, okay, like you know, it's a very slim chance that it's this, but if it is this, that's a really high risk. So we're telling you. And then here's what it maybe most likely is and, and maybe there is, you know, based on the current research at least like maybe there is a, a slight difference of like oh, you know, men potentially like have a higher incidence of cardiovascular disease. So maybe it makes mention of that. But I think the AI should be sort of in this instance pulling the, the taking in the information, pulling all of the sort of like available resources. But then it, I think it should be giving as much as possible and it should be up to the clinician to then you know, make the judgment call with their own rationale. That would be my, my lean.

Okay, yeah, that was a good point too Also that if it's like a patient facing app, then you shouldn't just give like a bunch of things and like they also get scared from all the things and then.

Yeah, yeah. And I think as well like there, there's so many things like, like even like when it comes to like cancer diagnoses, like a lot of times if people get, are diagnosed with cancer, the first thing they'll look up is the survival like rate because they want to know like what is like am I going to survive x many years? But like when you look up a survival like a survival curve, like that's not like quite what it is telling you, but that's what like the patient's gonna hear and be like, okay, well in five years or in 10 years, like this is what's going to happen. So it's, it's also understandably, a lot of patients don't necessarily have like the medical literacy to be understanding this information. So I think if it's patient facing, it might need to be a little more cautious. But for a doctor who knows this stuff, I think it's okay to be like, here is the kind of worst case, slimmest chance scenario, but there's a possibility it could be this. And you can decide for yourself if that's worth investigating or not.

And my last question for this case is, do you think the AI's response in this case was defensible given it followed statistical trends, or should it have prioritized patient safety over data driven averages? Where can we draw like a line between data, statistical averages and ethical care? Do you believe it failed in its duty to provide safe recommendations in this case?

It's hard to say in the sense of does an AI have a duty to do anything? Like I, I am especially, I know AI is certainly a very powerful tool, but I always lean a little more heavily on. It's up to the, like, the, the, the, the responsibility of care, I think lies with the clinician more than it does the AI. I think, I think ideally in say a more kind of research setting, perhaps statistics is more useful. I would say in a clinical setting, you always want to err on like the side of caution, patient safety, that type of thing. And so I, I guess in that sense, like, yes, the, the outcome or like what the AI did was not ideal. I, I wouldn't, it's one of those, I wouldn't say it failed more that the, we have failed to train it.

And we'll move on to the third case now. So let's say a young patient comes to you and let's say they're around 25 years old and they present with the lump in their chest. What key factors would guide your diagnosis and decision to recommend further testing?

I mean, I'd immediately be wondering again the sex of this patient. You said they were young, but I don't, I don't think I heard you say sex though.

Yeah, just young and lump in their chest.

Yeah. I guess what I would be most curious about what the sex of this patient is. Not that, not that being male, fully 100, like precludes one from developing breast cancer, but it's significantly less likely. I would also be curious about the, when you say age like, and they're young. I. Like, how young are we talking? Like, are they kind of like 25 years? Oh, you said 25 years. Okay, so they're post. Post p. Okay. Yeah. I think I would be asking about sex to start before kind of moving into. No, I don't. I think, again, regardless of. Because regardless of whether it's a, you know, breast tissue or whether it's, like, muscle, there's a. There's an obvious physical lump that you would probably just, regardless, want to get, like, an ultrasound or a CT scan. I suppose if they don't have breasts, I wouldn't necessarily send them for, like, a mammogram, but like, an ultrasound or a ct, I feel like would be useful regardless.

Okay, that makes sense. And then now let's say I told you that the patient was a female, and how would your decision change or what you. What would you lead more towards in terms of diagnosis?

I would definitely be. I don't know if I would be able to diagnose right away, but I would be immediately, like, alarm bells ringing of potential breast cancer or some sort of tumor and would be sending them for a mammogram to. To investigate further.

And then now let's say they were male. Would you consider the same things as you would when you were considering a woman's chest pain, or would your approach be different for male chest pain and the lump in their chest?

I think it would be different both for. Like I said, like, there is the statistical. Like, I believe the incidence of breast cancer in men is only like, 2.2percent maybe, but also, I think just like, the practical of the way that mammograms work. I. I don't. I'm not sure if it would be physically possible to have somebody who doesn't have breasts have a mammogram, but I would still send them for some type of imaging, probably just not that one.

Okay, so it sounds like either way, you kind of have a similar approach and you would take the both seriously. And for the last question, I'll give you the statistic that you're talking about. Breast cancer in men is rare. It's less than 1% of all breast cancers occur in men. And my question is, should that really affect your decision to test for it? How would you balance the risk of over testing versus missing a critical diagnosis?

Yeah, so I think, for me, I think that's where it comes to. It reminds me of, like, you know, subreddits where people say, like, need more information. I wouldn't be like, yeah, I Probably wouldn't immediately go, ah, yes, it's probably breast cancer for this young gentleman. I wouldn't be ruling it out, but I, it would not be my first thought. But I don't think it needs to be my first thought for me to want to send them for imaging because I think because there's something so visibly physical with this one, I'm like, regardless of whether I think that's a, like breast tumor or a different type of tumor or just some cyst or something, there's an obvious lump. And so I'm like, you still probably want to get that checked out regardless of what you think it is. And then if you do get it imaged, then okay, if I'm, you know, if this is one of the 1%, I think the imaging would at least like start to send us down that path. So I think probably because there's something so visible about it that I would, I would be sending them for some sort of like imaging regardless. But no, yeah, like I, I, I will admit that it would not be my first thought. Yeah, of oh yes, this person probably has breast cancer. Yeah, I think I would just be like, that seems like a weird lump. We should probably get that looked at regardless.

Yep, I think those are good points. And now I'll summarize the case. So this case was in regards to Raymond Johnson, which was a 26 year old man from South Carolina. He was denied Medicaid coverage, which is health insurance coverage for breast cancer treatment, solely because of his gender. Although diagnosed with breast cancer, he was ruled ineligible for a federal Medihat program that covered patients screened through certain government programs. In this case it was programs that only screen women. As a result, the insurance algorithm automatically excluded men, leaving Raymond without coverage for chemotherapy and surgery that he needed. Originally, this case exemplifies how gender biases embedded in policy and algorithmic systems can have life threatening consequences. The program failed to account for the fact that men can also develop breast cancer, effectively barring males from life saving care that they need. Advocacy groups, including the ACLU condemned the exclusion as discriminatory and have pushed for reform. The case prompted broader conversations about the need to design healthcare systems and policies that are inclusive of all genders and reflective of real patient populations problems. Now my question to you is, in this case, what kind of role did societal assumptions like breast cancer being like a woman's disease played in the algorithm's decision to deny Raymond for coverage?

Yeah, that one's like very shocking to me because I thought this was going to be an oh like we, we missed the diagnosis because they were a man and we just assumed that men can't get breast cancer, which would already be bad. But the idea that they would be diagnosed with breast cancer by a clinician and then it would just be oh, but we won't treat you for it because you're not a woman is shocking to me. Yeah, no, that's, that's. I, I think that's one of those things where I'm not entirely in terms of, I'm not entirely sure why a, maybe not why AI is being used, but like why that is information that the AI is being given. Surely the, the gender of this patient doesn't matter. If I, if I, the clinician saying this person has breast cancer, you should just be treating them for breast cancer. Their sex or gender is irrelevant at this point. I have told you that they have breast cancer and therefore need treatment for that. So I feel like that information. I understand maybe there'd be, or maybe there'd be some instances where you would need the information about sex or gender when you're coming, when it comes to policy or insurance and the American healthcare system. But that one, I'm just like, huh. It just seems like it should be this is what they have treat them for it. Not. Yeah, but they couldn't possibly have it because so we won't treat them for it. That seems backwards.

Kind of like a tactic insurance might use to not have to like pay for the medical treatment then because.

Yeah, I'd be curious if like there's any. Because obviously I think the American healthcare system is certainly something. So I would be curious if there would be any sort of similar kind of instances like in Canada, like if like private health insurance companies would have the same issue or you know, whether you would still get the treatment because like publicly funded type of thing. I'd be interested about that. But yeah, no, that's, that's quite bad.

Uh huh. And then my next question to you is what changes would you suggest in healthcare algorithms or policies to try to prevent these gender based denial of coverage in similar cases? How could we make sure that there's equal treatment for everyone?

I don't in terms of like actually implementing this like for the AI? I will not pretend to know how this would work, but I would like my thought would be one, it needs more andor better information. Like it, it should be aware that there is 1% of breast cancer cases that are in men. So like they should like there, there shouldn't be this presumption that it would be impossible for a man to have breast cancer and therefore be denied coverage. And then I, yeah, so I guess it'd be that. And then I, I wanted to put a stronger emphasis on the diagnosis from the clinician. Like, I think to me that should be, I don't want to say like clinicians can't make mistakes or be wrong in their diagnoses, but I, I would, I would be kind of weighting that more heavily of, well, if like X, Y, Z, you know, imaging or test or whatever and like the clinician has taken all that information and said this person has this disease, I would weight that more strongly. And, and maybe they're, you know, other characteristics would come into play a little less. But I guess, you know, this also, this also involves massive sort of systemic policy changes because government bureaucracy and also just like insurance companies, they're, they're there to make money. I'm sure they're not, they're not looking for more opportunities to pay out for people. So I think it would involve like, even if we've got very good AI tools, again, it's like the. But are we gonna like, use them properly? We'll see.

And then the last question, which is kind of related to what we talked about, but should healthcare algorithms prioritize statistical likelihoods and trends or should they be designed to account for like these rare but serious conditions? Especially when there's lots of stakes. How can we try to like strike this balance in these system, these AI systems?

Yeah, I think this reminds me of another sort of US healthcare system based snafu where yeah, like, trends are useful. I understand why they are there and why we would want AI or like any algorithms that we are using to help with that. Because in terms of figuring out, you know, populations that might be most at risk and potentially need more screening or things like that, like, I think that's very useful. I understand like why at, you know, certain ages they tell women like every two years or three years or whatever to be getting like screenings and things for breast cancer at certain ages. Like I, and they're not necessarily telling men that. Like, I get why that is happening, but to assume that people, you know, that these rare cases don't exist seems crazy. And it like, it makes me think of the US health care system. They. I don't know which commercial algorithm it is, but there is a commercial algorithm that they were using to again, like notice trends and guide health decisions and determine which, you know, groups were most at risk for certain things. And basically it determined that. I suspect it's probably all people of color. But they, they honed in on black people, were less sick than white people for various different ailments. And it's because they were using amount of like, money spent on, you know, given populations in like, healthcare treatment. They were using that as like a proxy for sickness. Because I guess the idea is just, you know, the more money you're having to spend on treating somebody, the more sick they probably are. When really it turns out that it's just that the American health care system doesn't spend that much money treating people of color. And so, like, people of color aren't being treated. And now AI is thinking, oh, like. And so like, they're just like, healthier, they're just not as sick, which is. So then they're not.

They're.

No, they were no longer being included in like, high risk categories. They're like, no, those people are so healthy. So it makes me think of that where I'm like, yeah, like, sometimes trends can be helpful. And sometimes I'm like, depends what you're basing those terms off of. And also, even if they're like, super accurate, like, yes, women are more likely to get breast cancer. That is correct. It shouldn't be. It should be like, perhaps it should be like a more instead of less situation. Like, it should be okay that maybe we're screening women more, but that doesn't mean that we should say, but so men couldn't possibly, like, the rare cases still exist.

Yeah, that was a good example you brought up too, where like, in those cases it would actually like, ruin like systemic inequities even more because people that already can't pay enough for healthcare are being less prioritized. Like, it's.

That'd be like, yeah, exactly.

Yeah.

So, yeah, before it was like, we know that you're sick and you just, we won't treat you or like, you can't afford to be treated. Now it's, you couldn't afford to be treated, so we're not even gonna actually recognize that you're sick. And it's like, okay, great, excellent.

Yeah. And that's it for the gender cases, the gender bias cases. So now we're going to talk about the last three racial bias cases now. And the first question I have for you for case number four now is imagine your healthcare provider evaluating a patient with kidney disease for transplant eligibility. What clinical factors would you consider when assessing disease severity and readiness for transplant? You can kind of think of it maybe as you have like, several different patients that need a kidney transplant. How would you kind of decide which one needs it the most? What kind of factors would you consider?

Oh, that's a good question. I feel like I don't know as much about the kidneys in terms of like what would preclude somebody.

Any general ideas? Okay, sorry. Any ideas are okay. Like you'd have to be super knowledgeable.

Yeah, I think, I suppose I would be looking into, you know, things like age. Like it is horrible, but I'm like if it's, you know, a 25 year old person who needs a new kidney versus, you know, a 90 year old person, the, the sort of cost per use, for lack of a better term is going to be better with the 25 year old patient. Not to mention they would probably handle the transplant a lot better. I guess I would need to know about like blood type. My hope would be that there are enough kidneys from enough donors that that wouldn't matter, but I know that's not usually the case. So it might just be that like somebody gets bumped on the list because there's just not a suitable kidney for them. I don't think I know what sort of lifestyle factors I would or should be looking for in terms of kidney function. I suppose I would probably want to know like if they or like their family perhaps has a history of kidney disease because again, when it comes to transplants it's horrible. But you're kind of thinking like, am I going to give you this kidney and are you going to immediately ruin it? Like is kind of the same thought process as to why alcoholics don't get liver transplants type of thing. So if there was some sort of, if there was some sort of disease that like some sort of like renal disease, that means that like this kidney is either like not going to take very well or that it's, you know, probably going to be damaged quite quickly after transplant, then maybe that would be a prohibiting factor. But yeah, that's tough.

Yeah, those are good points. A couple other points I think some other, other interviewees came up with was like the severity of how bad like the person needs it. Like for example, someone's kidney is like close to failure, then maybe we want to give them a transplant first. Then another thing was also how eligible you are. Does your kidney match the donor's kidney? Will your body be able to accept it? And just other things like that. Now my next question to you is, have you heard of race based modifiers in kidney scoring systems such as egfr? The idea behind it I think is primarily for black People, for example, these kidney scoring systems will make them, their scores look healthier than they actually are because I guess generally their scores are lower with these systems. So then they have a modification for certain races that makes the kidney scores look better than they actually are. Like do you think this is a good idea or no?

And I, I was going to say that is, I didn't, I didn't know that they did that for kidneys. I, if, if you'll allow a tangent, I do know that they do that for lungs. As I mentioned, my PhD was all like kind of like respiratory research and things like that. And one of the primary ways that diseases like respiratory diseases like COPD and things like that are diagnosed is with a spirometer which you blow into and it kind of measures your lung capacity. And if your lung capacity is below a certain level, you probably are sick or there's something wrong with your lungs that is meaning that they don't have as high of a capacity as they should. But back in, you know, the 1700, late late 1700s, American Founding Father and like just prolific slaveholder Thomas Jefferson, he decided that he looked into it and claimed that slaves had a reduced lung capacity compared to white people. Then in the late 1800s another like slaveholder turned doctor quantified this at 20% that he said that slaves or just black people in general had 20% less like decreased lung capacity than white people. And then like the 1920s came along and there's all the eugenics and this started making it into clinical handbooks to the point where now like modern day if you're using a spirometer it has a race based like yeah, like correction on it. And so basically it just decides that specifically black people, I think some other racialized groups have like a different sort of like correction put on them. But specifically black people like their basically said that like their lung capacities should just normally be 20 less than white people. And so it just corrects that. And even like the most well meaning like researchers or clinicians may not even know this is happening because it's like baked into the software. And so if your lung capacity is like 20% less, which probably means you're sick, but if you're a black person then it's like oh no, that's just normal for you and so you're not sick and so you're not going to end up getting treatment for copd, which is I think especially tufts given like the primary risk factors for COPD are like smoking, air pollution, things like that. And a lot a Lot of, like, there's a lot of, you know, climate, racism and socioeconomic reasons as to why people of color are probably exposed to these pollutants, like at a higher level. And so they're probably more likely to get copd, but then also less likely to be diagnosed and treated for it because they've got, they're just told like, oh, that 20% decrease in lung capacity is just like, that's par for the course for you. So if they're doing the same things with kidneys, I will, I can say, no, that's a horrible idea.

Yeah, yeah. Now I'll actually summarize the case and talk about it. But yeah, it's kind of like a similar thing to what you just said.

Oh, that poor person.

So in this case, the person's name was Anthony Randell. He was a black man from Los Angeles who was on dialysis waiting for kidney transplant for over five years. What he didn't know was that the algorithm that was used for the kidney transplant system incorporated a race based modifier that made black patients kidney scores seem a lot better than they actually were. This modifier caused Randall's kidney disease to be perceived as less severe than it truly was, leading to his placement in the national transplant waiting list to be significantly delayed. In mid-2023, he filed a case against the hospital, which was named the Cedars Sinai Medical Center. And he also filed another one against the United Network for Organ Sharing. And he alleged that they were unfairly depriving him of a fair chance to get a transplant because of racially based formula. And apparently it wasn't a secret that the algorithm did have a bias, because I think a lot of these cases did come up. People have done research into the, into the actual scoring system behind it, and they were able to tell that there was in fact this adjustment factor for people of different races, specifically for black people. And so, yeah, these cases were coming up and the board of the transplant system understood that the modifier was resulting in black patients illnesses being severely underestimated. And by early 2023, all hospitals were directed to stop the use of this race adjustment for black patients waiting times. And their waiting times were changed then to be reflective of the postponed changes. Randall claims that these changes should have came sooner and he could have already had his kidney that he desperately needed. And this case highlights how the Google clinical algorithms are good. They try to do the scoring system to see who needs it, but the execution was not done well because of this. The insertion of race, which caused black patients to receive to not receive their quality care in a timely manner. So this is like a crazy case. Like it took five plus years for him to even get it just because the score was making him seem a lot healthier than he really was. Knowing that this modifier made black patients appear healthier than they were and delaying his transplant eligibility. How do you think this affected, you know, patients like Athena?

Yeah. Oh gosh, that's horrible. Yeah, so that's really like you said, I think it's tough because in that case, like the, that the AI was sort of doing what it was designed to do of trying to prioritize patients. And again, like, I get why that happens. You don't want it to be like a first come, first serve if there's some like this is why like emergency rooms triage, like if somebody comes in with a gunshot wound, like if you've got like a cold, like they should probably be seen before you. But it's the garbage in, garbage out of like, well, how did it decide that somebody was worse? Based on very, very poor, like biased data that we gave it. And so I'm happy to hear that it, it sounds as though you said the this like, correct. This race based correction has been like people been told to stop using it for kidneys. I'll, I, I, I mean, I hope that it's the same for the like spirometers for lung disease. I, as of 2022 or 3 it was not so like, you know, maybe in the last two years it's changed but.

And like this one was in 2023 as well.

Yeah, because I think it was like, it's something that maybe some people knew about and there was like research into it. But then it was more around Covid where like again, people were using spirometers a lot and people were like, hang on, wait a minute. And a lot of like articles are coming out eyes on it. So hopefully a similar thing happens if it hasn't already. But yeah, that's a very, just that kind of gets to the crux of like it. Part of me is like, it doesn't, it does matter. But part of me is it matters less right now at least of how good can we make AI and like the models and stuff. I'm just like, I think we got to look back at ourselves and be like, what, what, what are our own biases? What are we feeding it? And like changing those up before we kind of rely too heavily on AI which will do exactly what we want it to do. Which is say that for some Reason black people's kidneys are different or whatever it was trying to suggest.

Yeah. My next question to you. Should hospitals and national organizations be held accountable when algorithms are known to be biased, but they continue to use them? What obligations do you think they have once they become aware of them? And I could give you an example that I gave to the other interviews as well. Another. This was an example, like the one that we just talked about was they know that there was this bias towards black people, but they kept using the technology. Another example was facial recognition technology used in policing systems. It was known after some time that the facial recognition technology doesn't work as accurately for people of darker skin tones because shadows and things mess it up easily. So then whenever you have, like, a photo, it'll match it to, like, another black guy. That's not really the same. You can tell yourself that they're not the same, but it will match them. And the police will then, you know, incarcerate that person. And I guess at the end of the day, even if they were proven innocent, they could just say, oh, well, the AI system did it. It's not our fault. And then they have an excuse there. And so, like, it's beneficial for them to have it because then they have an excuse to blame it on someone. And also just makes things more efficient. As soon as they identify this person, put them in jail, we're done. The case is closed. Like, so. In that case, even though it was known that there was these biases, I think they preferred to use it because it was just efficient and easy for them, and they didn't really stop it until there was enough backlash. And that's kind of similar to the analogy here where this scoring system probably made it easy. They just use the scoring system. They know who goes first, who goes second, who goes third in the kidney transplant chain. But then once enough backlash came, then they realized that we should really make a change rather than just take the easy way out. So I was just asking about, do you think they should be held accountable for these kind of things where they can't take the easy way out? Or maybe there might be even, like, a malicious intent behind it. Maybe they don't want black people to get treatment as fast as other people. And so then my question to you is, like, what obligations do they have to take care of the problem once they become aware of its harm?

Yeah. So I think that's a really good example that you brought up, and I think it illustrates my feelings on it, which are. I think they. Yeah, the. Again, the, the burden of the responsibility of care and the, and the, the burden of responsibility falls on. It would be like the clinicians or the, the hospital or the, in the case of like the policing would be like the police services. Because I, I do not think using AI as a, as a scapegoat or an excuse is acceptable. And yeah, I think it's unfortunate because I think yes, there are ways in which AI can probably make a lot of people's lives and like the work that they're doing better, easier, like you said, like more efficient. But I guess I'm always like, but easy like more efficient or like easier for who? Because like yeah, sure, this cop didn't need to you know, spend too much extra time trying to like, like tell apart two different people to make sure that he was incarcerating the correct person. Great, he gets to like knock off early and go home. But like that's not better for the person who has now been wrongly incarcerated. So it's like, yeah, it's more efficient but in like a, it's like that the half assed done way. And I think it's tough as well because I think this is why I'm like, I think we need to be like looking at ourselves. I don't think when something goes wrong with AI, I don't think we can go like, oh well, I was just. The eyes fall. I'm like, no, because in that best case situation you're still responsible for like double checking its work basically. Like you're still like the autonomous person in the room. But then I think like the unfortunate thing is I think sometimes people will also just go with it because the AI, we have sort of biased the AI and so then the AI is giving us biases back. Like you said, it may not necessarily be like, like overtly malicious, but I think unfortunately a lot of people have this sort of like implicit, like deep within them bias of what type of people deserve treatment, deserve to be healthy. And if the AI is saying, okay, this black person, they don't need a kidney as quickly, but this like white person does, great, that works well for me because that's kind of what my preference was anyway. Like it's, that's kind of what it, we've biased it and so when it feeds us back our biases, we're obviously going to go, yeah, that makes sense. So, and, and that's not on the AI, that's on us. So I think we have to take responsibility for that. Yeah. So in like the best case of take responsibility of like you should be checking its work. In the worst case, take responsibility for if it messed up, it's because you, you kind of did that to it. So yeah, I definitely think they like for the. That's why I'm sort of both happy that this race based correction seems to be, have been removed for the kidney function scoring. But I don't know if it's been fully removed for like spirometers. And in my mind that is like shocking because I'm like, okay, if you didn't. Because it's like baked into the algorithm and space and baked into the software, maybe you didn't know that it was doing it. Now that you do, I'm like, yeah, but like you said, I think sometimes as well, even like, well meaning people might be like, but we don't have another method. And so unless. But I'm like there has to at least be a way of like okay, fine, blow into the spirometer and then manually do a calculation to like add another 20% or whatever. Like there has to be, there has to be a way that you can like at least start to remedy this. You can't just throw up your hands and be like, oh well.

Yeah, but I guess it also kind of sucks for the hospital people because if they're using like technology from another company, at the end of the day they don't know the underlying like background things like they can't fix it.

Yeah, yeah, it is very tough. Yeah, I guess that's maybe a good point of when I, it's like the human responsibility but then it's okay logistically. Is it the hospital or is it like these third party like. And then obviously it's much more difficult to hold a third party that's got their black box technology. Like, it's, it's much harder to hold them accountable. So yeah, in terms of how people would be held to account, I think that's very difficult. But yeah, I, I would at least, I would at least as the clinician, even if I can't stop using these tools, even if I'm not like, I would at least be trying to do my like level best to be finding out as much information as I could or correcting where I can or like advocating for corrections where I can. I think it's just probably like just turning a blind eye and being like, oh well, this isn't my problem and I don't think it's the right way to handle it.

Yeah, that's a good point. And in Randall's case, the board eventually Mandated the removal of race adjustments, but it came years after the issue was known. Do you think the delay was acceptable? Could we have done something differently to hold them accountable earlier?

I mean, I don't think the delay is acceptable. Whether or not we could have done something different, I'm not sure in the sense of these, like I said, it's very difficult to hold companies or like these long held systems, like bureaucracies accountable. And even when you do, it's a very slow process. There's a lot of litigation. So I'm like, like, I don't think that's acceptable. I do think that that's too like too long of a turnaround time. But I don't know what we would have done differently apart from like, we gotta be like overhauling the system here.

Yeah. And how can a health system design clinical algorithms to avoid reinforcing, you know, historical systemic inequities, especially those that are tied to race. Is there any sort of general ideas you have, like we make these AI systems. Is there something we can do to try to prevent these systems. Sorry. To prevent these biases?

Yeah, unfortunately I think that's if I've been saying like the garbage in, garbage out. Like the obvious solution is, okay, we have to give it like good data or like unbiased data. But that is easier said than done. I think the, given our history as, as a species, we don't necessarily have the best track record of being unbiased and you know, things, things like that. So I, I, I think it still comes back to, we need to, we need to go back and be like, okay, where are our current biases? Like what data do we already have and how is it maybe wrong? And then be committed to like doing better moving forward for like the like new data that we are creating. I suppose in the meantime, if we're, because obviously like one of the other issues is like there being a paucity of data and we don't want to just like lose all the like health data that we have to like be using. But at least going back, knowing where those biases are and then you know, like say for the, I think you said it was egk, like the egfk, the kidney function. Yeah. You know, maybe figuring out, okay, what is that? Like like correction. Can we like train AI to you know, take that correction off or like ignore it or something if we want to keep using that data because it's maybe otherwise good and helpful and there's just like a part of it that's bad, like maybe we can, as long as we're aware of it, train AI and like model it. Better to not take those biases into account. But ultimately it's one of a. It feels like that we need to clean up our side of the street before we, you know. Yeah, hand it over to AI.

Yeah, it's a very good point. And now the last question for this case I have is what steps, if any, should be taken to make things right for these patients that were harmed by biased algorithms, such as like adjusting wait times, issuing apologies, offering some sort of compensation. Do you think there's something that we could do to, you know, regain the trust of these people in the healthcare system?

Yeah, I think, I think that's a very good point. Like the regaining of trust. I think that is one of the, that is a very big issue in like the medical landscape is that for a variety of reasons, all of them very valid marginalized communities do not necessarily have a lot of trust in healthcare and like the healthcare systems. So I think rebuilding that trust is very important. I do think like, sort of personal and, or public apologies can go a long way, assuming that they are backed up with, you know, like, like the ensuring now that like this, this correction is not being used like some sort of tangible. Okay. And we are changing, like we are going to like, put like our words into action. I think that's very important. I think compensation is honestly probably fair. I don't know how you would quite calculate that out. And in certain, like for this one, it took obviously a long time, longer than it should have, but this individual at least did end up getting their kidney. There are, I'm confident, cases where the, the outcome is that somebody died and I don't know how you compensate somebody for that. So it, it's tough. But I do think, you know, depending on like the, the situation and severity, like perhaps conversation is important there. But yeah, I do think that there's, there's work to be done to sort of regain people's trust in that sense, which would, you know, even, even if it weren't absolutely the right thing to do, it also behooves us as like the, you know, the healthcare individuals, like the, the medical researchers, because, like, if we're concerned about bias in our data and we want to make sure, like we're doing, you know, the equity, inclusion, all of that nice stuff. I, I suspect it's difficult when people are like, well, I don't want to, I'm scared to be included because I don't trust you. So having that trust and like that two way communication, I think would then allow us to further improve our medical research because we would have the trust of people that we want to include in our medical research to then make our medical research more, you know, inclusive. So it's like this long kind of, you know, domino effect.

But yeah, those are all great points. We'll move on to the fifth case now. Almost done. So this is the second last case now.

It's all very interesting. So I do see why it takes people a long time.

Yeah, all of them took like over an hour. So now let's imagine that you're treating a patient with chronic lung disease during a respiratory pandemic. If a pulse oximeter shows normal oxygen saturation, but the patient appears visibly distressed, what would your next steps be?

So the, whatever tool I'm using, it's saying their oxygen is normal, but they clearly look distressed.

Yes. And it's like a pulse exometer. It's like those things that you put on their finger.

Oh, yeah, yeah.

There's no sensors.

Yeah, yeah. I would definitely be basing it off of the, the patient's distress, I think. Yeah, I think if they're visibly distressed or like struggling to breathe, that's, you know, the pulsometer can go out the window.

So then would you try then using other tools and other things to be able to assess oxygen levels?

I'm trying to think because my, my gut reaction is like, no, I'd probably just give them oxygen. But I'm trying to think, is there any reason why, like that would be dangerous to do if they don't have deficient oxygen? And if there was, then yes, I suppose I would need to like, have, you know, some sort of orthogonal method to test them though. I mean, as I've mentioned with the spirometer, I don't know how, how much trust I have in the, all of these, but yeah, I think I would if it, if it seemed medically necessary to, before administering oxygen, I would probably use some sort of method. Like I wouldn't just base it off the pulsometer, but if not, and it's just based on patient distress and I'm allowed to, within my judgment, give them oxygen chin, because they seem to be struggling to breathe. I'd probably just do that.

Okay, yeah, that's a good response. And now the next question I have for you is how much would you like completely rely on these tools like pulse exometers and your clinical decision making? I guess you kind of answer this, but do you think you would ever double check the accuracy of the device, Would you ever doubt it? Would you think it's 100% right? Even if they're distressed, do you think it would still possibly be right or would you be on edge, like not fully trusting it, using your own intuition more?

I mean, I would definitely be questioning it. And I would want to double check because my, my inclination is to trust the patient and trust like what they are experiencing in their own bodies. So I would want to, yes, double check these methods because I'm like, even if it's like an, even if there's no weird, you know, like the kidney thing, even if there's no like weird corrections on them and it's perfectly fine, technology fails like it, you know, there are, you know, wires get crossed, frayed, like there are reasons why things may not work. And so I think I would err on the side of assuming the patient is, knows what they're talking about when they're experiencing distress and would kind of want to double, triple check the, the methods that I'm using. If they somehow all are like saying, you know, one thing, maybe that's when we have to start looking at other avenues. But I would, I would err on the side of trust the patient first and yeah, double check the technology because technology can fail.

Yeah, yeah, I like that response. Now I'll actually talk about what the case was. So this was during the pandemic. There was race related biases in medical technology that confronted Dr. Noah Albor Albert. She was a family physician and CEO of Roots Community Health center based in Oakland. In late 2020, one of her patients was an elderly African American gentleman who suffered from chronic illness, chronic lung illness. And one of the checks that she's done in a previous visit showed that through a pulse oxygenation check that his oxygen saturation levels were high then. Now on this visit, she used the pulse oximeter and it showed that his blood oxygen level was at a normal level. But the doctor's clinical instinct indicated that the patient was much more distressed than what the pulse oximeter showed. She conducted an aerial blood gas test which confirmed reverse fears that the oxygen content in the patient's blood was too low and that the he needed oxygen. And sometime later, she came across an article in the New England Journal of Medicine that confirmed her hunch that these oximeters were unable to register low oxygen levels in darker skin patients as compared to white patients. She and her colleagues were outraged by the fact that a device that they use that was supposed to help patients was grossly inaccurate for the black population. And finally she and her clinic participated in a class action lawsuit against the manufacturers and sellers of pulse oximeters for not having detailed warnings or up to date devices. And she didn't stand idle while demanding the FDA to take this pulse oximeter discrimination towards races very seriously. This is kind of like an example where she took her own intuition, similar to how you said you would probably take action on the patient's discomfort, but it's like something that's been going on for years. People of darker skin tones. I guess the pulse 6ometers sensor doesn't work as accurately on people, darker skin tones. And so it wasn't giving good blood oxygen reading. So it's probably affected like a lot of patients over the years. But only the doctors that kind of want to like challenge the, their equipment will be the ones that actually try to go further, do more tests and try to make sure that the patient is okay.

Yeah, no, I would say kudos to that doctor for kind of not only following their intuition for like that, you know, the acute issue of like this patient is in distress, but then like the, yeah, the class action lawsuit, the doing their research, the following up. Because obviously, like, yeah, we've talked a lot. I released, I've talked a lot about like how, you know, this is all like, terrible for like the patients who are involved and like, there's a lot of accountability that needs to go on, like hospitals or companies or whatever. But there are also like, yeah, instances where I'm like, oh, like, yeah, a lot of empathy for those doctors who are trying to do their level best to treat their patients, think they're doing all the right things and then find out like, oh, I've just been given like tools that are not equipped to help certain people. And like, that's kind of scary.

Yeah. And in this case, the doctor's clinical judgment overruled, the device's reading. What do you think this says about like the limitations of relying too heavily on technology without understanding, you know, the, the, the algorithms behind it or the biases behind it.

Yeah, I think that's like, I think you said it there like that we shouldn't be relying too, too heavily on technology. And like this is a more like a pulmometer is obviously, I think, a more common form of technology. Like people have encountered them. It's not like the more I think for a lot of people, like nebulous ideas of like AI and things like that. And so when you get to like these more complex ideas like using a technology, not fully understanding it, and then just like Taking its word as gospel is dangerous. I wouldn't even like with like Google how they like it gives you like the AI summary at the top of all of your searches. Now there will be times like I will be searching something for, you know, work. I'm looking for papers, but before I get to the papers, I see what it's written and there are times where I'm like, that's just wrong. I know that's just wrong. But you know, if somebody doesn't know, if they're, if they're not in that sphere, then you might read that and go, oh, okay, like, great, log that away is a truth. So yeah, I think it's one of those things where again I, there are a lot of ways where I think technology and AI can be very helpful. I do think it's, it's not worth being afraid and therefore not implementing them. Part of me is like, it's happening, it's going to be happening. But yeah, being mindful about biases, being mindful about ethics, and also remembering that like this is a tool but ultimately you're as like the say, clinician or whoever's using the technology, you're still meant to be like the brain in the room. So you know, you can't just kind of throw up your hands and be like, well, the AI said. Because the AI is not this patient's doctor.

Yeah. And you know, pulse exometers are kind of like a call in piece of equipment. Like every hospital has it. Usually when you have a visit, you'll use it every single time. Even though they're so common. We, they were found to overestimate oxygen levels in turkey skin patients. Do you think that could be still so common? And this design flock kind of stayed for so long despite the risk it posed to patients of color.

Yeah, there are certain, there are certain tools that we use like this, the pulsometer, but also like the, like, yeah, like the aspirometers I was talking about where I'm like, we, they're so common. We use them all the time and it's either we don't realize that they have these, these shortcomings, these like limitations, or we just, you know, how bad could they be? And it's like the best option we have. Like there, like there could be a lot of reasons why. But yeah, the idea that we're still used using them is, is alarming. And I think I, I would have the same feeling of that doctor where I'm like, this is outrageous that I'm using a tool that I'm expecting to do one thing and it's perhaps not, and that's. That could be seriously impacting people. And I think that goes for, like, the technologies and then also, like. But like, it's similar just like people's own, like, subconscious biases. Like, they're. They're deeply ingrained, but they just, like, linger, unfortunately. And so then when it's in technology that now it's like, well, we've built it in and, you know, how do you. How do you slowly start to remove that or replace it with something better like it. It can be. Can be tricky. Things can. Things can really easily get kind of baked in, and then that's just how it stays, like, status quo.

Do you believe the FDA and manufacturers have done enough to address this issue? What more do you think regulatory body should be doing to ensure devices like these are accurate across diverse populations? Any sort of ideas you have there?

I think, I mean, I'm gonna say since they're still in use and it seems like at least this one doctor had no idea that there was this limitation, I'm gonna say they're probably not doing enough to, I would say, at minimum, warn people of the limitation and like, be very clear that this tool can do this. But, you know, it might not be as accurate with these populations or like, it mention the limitations so that people can be making like, their best judgment or their best, like, their decision with their eyes open. But also it's. And this, this goes for, I think, all kind of medical research going forward. Like, it. The medical research, medical tools and devices, technologies, like, they need to be tested on more than just white male bodies. Like, they're like, before we're putting like, this tool on the market, this. That's now going to be in like, every hospital and every doctor's office. It should be tested on a wide variety of people. And like a wide, like, in this case, it'd be like a wide, like, variety of skin tones to see if it actually works. Because I feel like if you can catch it early enough and then, you know, you're not just like, sending it out there to do its harm. And hopefully there are ways to, you know, I, again, I don't pretend to know, like, all the ins and outs of the tech that goes behind those readings, but there's got to be ways to somehow help them work better for everyone. But yeah, like, we have to be testing on more people, more different types of people.

Yeah, that's actually a good point you brought up, because when I was doing research for this project, we found out that a lot of times for actual clinical research that goes behind these tools or like pills or like anything in general is usually predominantly just white males compared to other people of other demographics. And even I think some medicine that's geared towards women will be mostly tested on males because they won't have, they won't have like birth complications. Like, you don't have to worry about, like, usually when you choose women, you have to make sure that they're not pregnant for some of the things. So if you just choose all men, it's easier to do. But is it really that accurate for a woman's body? That's the question.

Yeah, I think, I don't remember which scientist said that it was. They chose to use the, I mean, they didn't even touch the whole race thing. I think they just decided, of course we would use white people, but they decided to use male bodies because women's bodies and all of their hormones and stuff were just. It made the environment too complicated. And I'm like, right, but if that's the environment it has to work in. Like, it's like when we were talking about Fusion Validator and we're like, we need to like stress test it. We gotta like see like how far until it breaks. Don't be like, oh, well, it works perfectly fine in like this cool, clean, like, like whatever, perfect environment. If that's not the environment it's gonna be like used in. But they're like, women's hormones are just like too complicated. And so like we're gonna test in men's bodies that I guess are void of hormones apparently. Like it doesn't even.

That's crazy. Yeah. How should future healthcare providers be trained to detect and challenge device based bias in healthcare? You think like, maybe when they're retrained for like hospital use, for example, that they should always have like their own intuition versus what a tool might tell them. They should always like, not just like completely rely on these equipment. Or maybe even if there's like biases already known, should they be aware of them? Like for example, the pulse examiner, should they be told right away that for black patients it will look healthier than it probably actually is? Like, is there some sort of way that we could train these professionals to be able to always be prepared for biases that may arise in some of these tools?

I think there's a lot that we could do to help these physicians be, you know, be able to practice their best possible medicine. Like, I think if there are biases or Limitations in the tools that they're going to be using, they should be made aware of them. But I also understand that there are certain times where you don't know what you don't know. There might be tools that have these baked in biases from long past research that we don't know about and so you can't relay that information to people. So in that case, I think in terms of training, ensuring that the as much as possible, they know that these tools are to be used to help, they're to be used to gather information. But in terms of decision making, that needs to still fall to them and using their best judgment, if they're ever concerned, make sure that they know that it's okay to be asking questions using orthogonal methods like double checking type of thing. And ultimately I think making sure that health care is patient centered and patient focused, like making sure that what that patient is telling you is always kind of like front of mind and what is driving you to do something. I think, and that one can be, I think it sounds like, oh, well, yes, of course. But again, like, like it's not just the technology. I think we ourselves obviously have a lot of baked in biases and it can take training and it can take practice to get people to kind of like get out of those. Like even things like often, you know, from long, long ago, like black people, people of color have been described as having like higher pain thresholds. So then their pain is often not believed or when they say they're in pain, people don't believe them. And now, even though I wouldn't say I've run into any doctors who have that explicit like bias, like they're not saying out loud, I think sometimes it's still in there. And so there'll be lots of stories of, you know, black women, like I think even Serena Williams, like, who's famous and wealthy and all this, but like was just post, post having given birth, was saying she was in pain and everyone's like, meh, like no, you're fine. And then it turned out she was hemorrhaging and thankfully somebody listened to her and she's fine.

But okay.

Yeah, I think like kind of putting into practice and being like, okay, when patients tell you how they're feeling, listen to them and, and having that be kind of your, your guiding principle.

Yeah, I think that's a good point about like how like a lot of times we'll have like underlying biases, like deep rooted in us that we won't even realize that the bias is just like Your way of thinking. And that could really affect, you know, but in the case of medical, in the medical field, how they administer health care to patients.

Yeah, so I think, like, making sure people know about the limitations of the technology, but also just making sure people are as much as they can. It's hard to like, root out the biases you are like, subconscious, but, like, as much as you can, just like listening to patients and making sure that they are centered in whatever you are doing.

All right, and then now we're on the last case for this one. I want you to imagine that you're purchasing a health monitoring device, like a smartwatch. It could be like an Apple watch, for example. What level of accuracy and reliability do you expect from like these kind of devices, especially when it comes to tracking stuff like critical health metrics, like your blood oxygen levels, your pulse, your heart rate, like just basic metrics that you know, you want to monitor on a day to day basis. Maybe especially if you have a condition, it's important to check your vitals every day. How, how reliably reliable would you expect these metrics to be?

Maybe I'm just like, cynical. If I had a condition that I needed to be checking my, like, vitals, I would not be trusting Apple to be the one to give me that information. Like, I have a smartwatch and I'm using it to count my steps. And I know that it's probably not accurate entirely, but it's giving me a ballpark. I can just tell if I've been like, sedentary or if I actually like, walked around in the day. I think if I needed to know like, accurately what my heart rate was, I'm, I don't know if I would trust Apple to be doing that for me. I would hope that because they are advertising that they can, that there are rules and regulations in place that mean that they need to be accurate. But I have a healthy level of skepticism with them where I'm like, I don't know. Probably not.

Okay, so like, in general for smart watches, you wouldn't 100% trust, like these critical metrics that they would give you. You kind of think of them as a ballpark, but you wouldn't like, rely on them?

Yeah, I, yeah, I would think of them as a ballpark. Like, I'm sure I'm confident it can tell whether my heart rate is like 100, like I just exercised or whether it's 60, like I'm resting. Like, I'm sure it can tell that difference. But like, I don't, I Don't know if I'm confident in, like, it telling me what my, like, blood oxygen level is or what my, like, you know. Yeah, I'm not. I would want. I would definitely be. If I saw something and I was like, oh, my heart rate's been, like, elevated for a very long time. Like, again, I think I would be like, okay, maybe I should, like, go see a doctor. Or maybe I should. There might be another tool that I can use. But I wouldn't just necessarily be like, oh, Apple says that I'm sick.

Okay. And how would you feel if you discovered that your smartwatch didn't work accurately for people of your skin tone or demographic?

That would definitely still bother me. I think even if I'm coming from the, Like, I don't trust it to work and I'm skeptical and whatever. I guess I'm coming from the. I don't trust it to work for anybody. So to find out that, oh, it does work for some people, just not me would be. Not. No, that would be bad. I would be very frustrated by that, especially because then I'd be thinking, okay, but what if I was one of the people who really trusted it and was using it for, you know, xyz important thing. Well, okay, now great. I. Now I know it hasn't been working amazing, like, so I think that there would be a lot of anger and frustration there.

Yeah. And I'll summarize the case that's related to the smartwatch. So the person in question was Alex Morales, a New York resident. And he brought to attention to a case of possible racial bias in consumable health device Apple Watch. Morales, who had a darker skin complexion on an Apple Watch with the expectation that its blood oxygen sensor would accurately lock in oxygen levels for fitness and health purposes. To his surprise, he later found out that the device's oximeter may not work well with people from his demographic. In late 2022, Morales initiated a class action lawsuit against Apple for allegedly containing a blood oxygen app that was racially discriminatory and did not function as promised for non white customers. Supported in part by complaints of other studies claiming that more advanced false oximetry devices are massively less useful on people with darker skin. Morales asserted that Apple owed the public an explanation. After all, paying smartphone users assumed that the device would be equal for all users, which was not the case. While the judge dismissed the case in 2023, it did start an important discussion r[] products. And this case showed that. This case showed that the world that there are in fact indirect Biases in medical grade equipment like Pulse 6 ometers similar to the previous case. This shows us that Alex Morales and the rest of the community are still subjected to discrimination, discrimination based on race, even in technology that they choose to use themselves already. It demonstrates that a keen eye for responsibility in tech industry is needed in these kind of situations. And so my question to you is, do you think companies like Apple have an obligation to kind of disclose the limitations to their health sensors, especially if those limitations affect certain ratio or demographic skin tones or in general, do you think they should be disclosing these limitations so that people are actually aware prior to buying these in the first place?

Yeah, I think definitely. And it's funny because like this whole time we've been talking about like just the, how it's important like medically and how like there's a safety aspect to it of having a technology and finding out that it doesn't work for your skin tone or like whatever demographic that you're in, like that's, that's probably like the, the worst thing, the top thing. But it's funny that I'm just like, oh yeah. And I bet people would also just be incredibly frustrated to know they paid money for a product that doesn't work or at least not for them. So I think both from a kind of safety health perspective and then also from a. Just as a consumer, yes, I do think you need to tell me if I'm about to buy a product that won't work for me and if by not working for me that could have like serious or like, like medically related consequences.

And why do you think skin tone bias and exometers and wearable health devices haven't been, haven't received widespread attention until recently. What do you think some of the barriers might be to addressing it?

Yeah, it's funny because like it seems like all no 2 of the 3 cases here that have to do with race have to do with something that can't like read past or see past certain skin tones. So it does seem very prevalent in like a lot of different technologies, which is interesting. But yeah, I, why it hasn't been addressed until now. I think there part of it could be a people don't know or people kind of don't want to know, like it doesn't affect you. So meh, like moving on. I think it is also that it's not quite a stigma but that feeling of like, like people, if you're from a marginalized community, I can imagine that for again a lot of very wide Ranging and valid reasons. You have been made to feel like you don't matter or that like, you're like, people aren't going to listen to you anyway, so. Or like if, if you do bring it up, you're going to be like the insert stereotype here. Like, isn't. You're going to be seen as just causing problems. And like, and so I can imagine that there are a lot of people who are sort of just like, well, like, that's going to be a lot of like, spotlight on me or like eyes on me if I raise this almost like the kind of bystander effect, but like within your own kind of body or your own like, like demographic where you're like, it's going to be like such a hassle and like, no one's going to believe me. It's going to be like this uphill struggle to get justice. And so like, I don't know, I guess I won't mention it kind of thing. But it could also be, I think as well, like, that these people don't know, like, for all. For what I've said previously of like there being some mistrust for like the health care community. I think there's also a lot of people, they're like, doctors know best, they're well trained, they got like MDs, they are smart and they know what they're doing and I trust them. And so if they're using a device, you're kind of just assuming, assuming like that it is working. And so you like, if you're part of that community where it's not working for you, you might not even know. Especially if it's not one of these, like, kind of case studies where like, something's gone wrong and it's blown up in your face. Like, if it's just a routine thing and nothing's really come of it, like, it's. There's no reason for you to like, be suspicious that it's not working. You might just be like, yep, nope, seems fine kind of thing. Like, if you told me right now that blood pressure cuffs don't work very well on women, like, I wouldn't, I would be surprised because I'm like, I use them all the time and it like, but it's never come up because I don't have high blood pressure. So I'm like, I use them all the time and it's always said it's fine and I feel fine. Moving on. So it's possible they don't work, but I have no idea. So, yeah, I think it's like that kind of mixture of, like, people might not know, either on the patient side or the clinician side, people might not know. And then the, when you do, it's like, okay, how much do I want to rock the boat to bring this up and, or do I just wait for hopefully somebody else to bring it up? Or how important is it to me? Like, it can be tough. We, we do unfortunately put a lot of the burden of advocating, like, on the people who are like, the most marginalized and vulnerable. We're just like, well, if you want better, you're gonna really have to fight for it. And that's tough.

Yeah, that's a good point in this case. The court dismissed Morales's case. But do you think tech companies should be held accountable in other ways? What kind of responsibility should they carry when their products are shown to underperform for marginalized users?

I do think that that company should be held more accountable. And again, this is my cynical realm. Like, are they going to be. But no, I do think that they should be held more accountable. And again, even if we're not going from the moral, like, medical elements, I'm just like, you're a company who is designing a product and you are selling it in this capitalist society we live in. And I think it's fair that consumers expect that it work. So even from just a, like, you know, purely capitalistic sense, I'm like, I do think they should be held accountable if their product doesn't work work. But yeah, I do think they should also be held accountable if it's, if they're releasing a product that is like, possibly dangerous kind of thing. I feel like there are some areas like, you know, like the kind of wellness community, not super well regulated, you can be getting, you know, supplements for anything. I feel like tech is kind of like that where it's like, it's sometimes regulated, but there's a lot of, like, areas where it gets really blurry and not very well regulated. And it probably should be.

Do you think consumer tech devices with health features like smartwatches should be subject to the same regulatory scrutiny as medical devices? Why? And why not?

That's really tricky because on the one hand, if they are, if they are purporting to be a medical device, in a sense, if they are purporting to be giving the same type of information that you'd be getting from a medical device, I think it is fair that a consumer would think that that be accurate because they're going to be using it probably for the same reason that they like that they would if they were getting it from a medical device. Like, what they're doing with that information is probably the same. And so I do think that if you give. If you're saying that you are able to measure certain medical metrics and you are offering those up to patients or not patients, customers, who. People, and they're using them for, you know, probably medical reasons or at least like, you know, they're informing their medical decisions with them, I think it's fair to expect that they be tested. But then it's tough because on the other hand, like, you know, Apple's gonna be like, we're just a tech company. We're not doctors. Like, if you've got a medical thing, you should be going to your doctor. Don't come to Apple like that. It's your choice to buy an Apple watch. And, like, while that's quite, like, callous, I. A little bit, I do see it where they're like, we never pretended to be doctors. If you. If you want to be, like, properly monitoring, like, your blood O2 levels, you should be, like, getting actual medical equipment. So it's tough. It's. It's that balance where I'm like, how much of that is, like, on. Like, if you're ever. If you're offering up a product that is in the medical realm and. And we're like, people are gonna expect that it works a certain way. You should make sure it does, versus, like, how much of the due diligence has to go on, like, the person to, you know, use their best judgment. When buying and using products that aren't, like, from licensed medical places. That one's, like, a real gray area for me where I'm like, I want to say that they should be regulated like medical equipment if they are offering a medical service, but I feel like there'll be implications of that that might be difficult to get around. And in. In a small sense, I do see their point of, like, but we're not a medical company. So, like, if you're looking for actual medical things, like, don't come to us. That's not really on us that you did that, so. Oh, I don't know.

Yeah. How do you think the tech industry can ensure that innovation and health and wellness tools are tested freely across diverse populations? Who do you think should be involved in the process?

Oh, yeah. Again, it, like, it comes down to just, like, the. The bare bones of, like, your experimental design. Like, when you're selecting your cohort, make sure there are people of different skin colors or make sure there are different genders and sexes and stuff in there. Like just being mindful up front of your, your study design, I think goes a long way because like, if, if like the study itself is like a house, I'm like, that's your foundation. So like make sure that that's solid before you start like building rickety scaffolding on top of it. Who should be involved? I, again, I think this is where a lot of like the DEI and like affirmative action and things like that come into play. Because again, for, you know, maybe not malicious reasons but in like, you know, just privilege and biases there. If like everybody, if all the top researchers and all the people who are involved in like making this technology are all white men, if it's not malicious, it might just be like they don't have this experience. Like they've got blinders on, they've got their own like, privilege. That means that they're not necessarily thinking about, oh, we should double check that this works on all skin colors. So I think having people of like marginalized communities at the table to bring their expertise, not just their like technical expertise, which they have obviously, but then they're like their lived expertise of, hey, what about this? That maybe people haven't thought of, I think is really important because I think when it comes to, you know, getting your diverse cohorts and like coming up with a really well rounded design, it might need some people with that, you know, lived experience to nudge us in that direction. So I think the people who are in these like, leadership positions or these, like, heading up these projects, there should be some diversity there.

Yeah, those are good points. And the last question I have for this case is what does this case reveal about the intersection between technology, race, access to accurate health information in the digital age? How should companies rebuild or maintain trust with users who feel excluded or misled by biased product performance?

Yeah, no, that's actually, that's like, made me think like, that's also a very good point of like the access piece part because whilst companies might be going, well, if you've got medical concerns, go to a proper doctor. Awful lot of people who don't have doctors or who cannot afford doctors. And even though, you know, an Apple smartwatch might be expensive, it's probably less expensive than like certain medical, you know, apparatus that you'd be getting, especially if you're, you know, somewhere in the States where it's, you know, privatized healthcare. So, yeah, that's tough. Where we might have this situation where You've got populations who, because they've been marginalized, are probably likely to have less access to, you know, a doctor or like, medical type, like the sanctioned medical technology. So they're going the route of tech and then finding out, okay, well, the tech doesn't work for people like me. Yeah, it so just that intersectionality thing where it's just like biases and like racism, it just begets more of that where now these people are sort of, I think, at a loss of like, well, things don't work for me. And I, like the medical system doesn't work for me. But now, like, the tech that I'm buying also doesn't work for me. Like, I can imagine that you're. You eventually get to this place where you're like, okay, well, like, nothing seems to be working for me. And then you asked your last question there.

It was about how companies can rebuild or maintain trust with users who feel excluded or misled by biased product performance.

I think it's probably similar in the sense of, like the earlier when you had asked about, like, medical, like healthcare professionals doing that. I think it's similar in the sense of acknowledging that there's been a problem, acknowledging that like you made a mistake or the product that you're putting out was working. Like, acknowledging that and kind of being upfront and apologizing for that I think is important. But then following that up with action, like showing that you are, like, working to address these issues, making sure, like, that the technology you're coming out with in the future has the ability to actually, like, work properly for all the, like, different skin types. I like, I would even maybe go so far as, like, especially if this is like, kind of the only reason this person bought this watch and it doesn't work for them. Like, like reimbursing them for that product, I think would be the polite, the morally correct thing to do. And then, yeah, making sure that you are, like, putting your words into action and making sure that, like, the things that you are putting out now in the future are more mindful and will actually work. Because it's like you just don't want people to go, oh, sorry, like, here's a 200 credit to Apple. And then, you know, the next product you buy also doesn't work. Like there needs to actually be, like, real change.

Yeah, that was a good point. And then now we're done all the cases. Now we'll just ask you a couple of final overarching questions about just reflecting over everything. The first question I have is, have you ever encountered or learned about any personal bias in your work or maybe that you've seen through the Internet anywhere? And how do you think this affects healthcare decisions?

Yeah, so, yeah, like I said, my, my graduate studies, like my thesis was all on respiratory medicine. And while my, the research that I was actually like doing was all, you know, lung cells in petri dishes, like, we weren't at kind of like clinical trials or anything, I don't think there was that as much of that chance for bias there. But in terms of the research that I was like taking in and that was then informing what I was doing. Yeah, a lot of it was definitely. And like, I, I think I had a whole chapter of this on my thesis of like, we can't just have like white male bodies because it's. A lot of what I was looking at was tobacco smoke, cannabis smoke, and then, you know, we were going to do cannabis vaping, but then the pandemic happened. So we're like, covet also affects the lungs, we'll just wedge that in. But a lot of these things, like especially tobacco smoke and cannabis smoke, a lot of the studies are on men because, like, I think some people will be like, oh, it's because, like, men like smoke more. And I'm like, do they? Or is it just that, like, you know, it's the kind of default body to be working with and, and like the, it's the kind of the go to. Or like, even if there's like the kind of social students of like, oh, like people who smoke, like, that's like a guy thing. And like, you know, smoking weed's a guy thing and not. So then like, there's this whole other half of the population that's not really being included. And then when we were looking into things like the effects of like cannabis on the lungs, like hormones definitely play a role in it and like the various different cannabinoids, like, they're also kind of like getting involved. So like, the results were certainly different between men and women when we were looking into any studies that actually like included women. So yeah, I think there's definitely biases like that that I've kind of come across especially as well. Like, it's better now. Like, it's getting better. A lot of granting agencies, like the Canadian Institute for Health Research, cihr, big granting agency, they explicitly make sure that you in your grant application include how you're going to take into account sex, gender, if applicable, like, like race, things like that. They explicitly ask you about that and you need to like, Outline in your project. Like, and, like, make sure you're saying, like, yeah, if it's a. If you're actually going to be doing human trials, like, we're going to have, like, X many men and X many women. And like, so, like, that's good. But, like, the further back you go and research, like, it's all on just white men. Like, there's not. There's not a woman or a person of color to be found. So that can be certainly very difficult. I think for the research I'm doing now, it's like, say, like, the rare disease stuff, it's tough because they're rare. So I was like, we're kind of. I don't want to say there's no bias, but I'm like, we're taking whoever we can get. Because I'm like, they're so rare in terms of, like, the. There's so, like, the pool is already so few. Like, if we were only selecting men or we were only selecting white people, like, we'd have, like, no patience, basically. Like, so I. I guess it being a. Working with rare diseases kind of forces you to, like, just you. You gotta make sure you're including everybody, which is a good thing. But, yeah, no, I think it's. It's tough. I like. And then even, like, in a personal realm, like, it kind of reminds me of the first story where somebody I know went in to their doctor's office because they were complaining of, like, an abdominal situation. And it was almost the. It was almost the reverse where, like, it's a good thing. The doctor at least considered ovaries, but they became very dismissive and they're. It's just your ovaries. It's probably like, I don't know, like, PMS or something. Go home. No, no. It turned out that, like, their appendix was rupturing and they had to get it taken out. And it was just like, oh, okay. Like, would a female doctor have, like, you know, maybe thought about ovaries but then seen. Yeah, that's probably not it. And like, you know, looked further and gotten me into a hospital before my appendix burst, maybe? Hard to say. So I think both in, like, the research I'm in, but also just, like, lived experience. Like, yeah, there's medical biases everywhere. Which is always why I'm like, I know people are like, oh, I'm so excited about AI And I'm like, I am. But, like, I'm so nervous about what we're giving it because, like, science has done a lot of amazing things and there's also some bad science out there.

Yeah, it's kind of trying to find a balance between good AI, bad AI, convincing AI.

I know, but then I think part of it's also, I'm just like, yes, we want to make sure that the AI that we're programming is good, it's robust, it's ethical, all these different things. But part of me is gotta turn the lens on ourselves a bit and be like, we should be mindful of what we're giving it and not pretend that we're perfect. And if it messes up, it's obviously the AI's fault. It might be that we're giving it some real questionable data based on some real questionable thoughts that probably many of us are still holding.

The next question I have for you is how can we ensure that AI systems in healthcare reflect the diversity and complexity of human identity, including transgender and non binary people?

Yeah, I think that all it still just comes back to like the making sure that we as people and like researchers and clinicians are doing the work to include those people, like all of those marginalized folks in our studies. Making sure that we have good, robust data to then feed AI, I think is step number one. And then as well, like, you know, research takes a long time. It's probable that AI in medicine is going to be implemented before we can go back and like undo all of our wrongs in research. So I think it's also just being really mindful that we do have these biases, that the research that we have given AI is going to have biases and just not kind of leaning back in our chairs and letting AI take the wheel, so to speak, like making sure that we are still using our judgment and our discernment and not just taking AI as like gospel.

Yeah. If you were designing an AI tool, how would you balance giving statistically accurate information but also not downplaying rare but deadly possibilities?

Hmm. Yeah, it's really tough because I'm like part of the, part of the obvious, like a lot of the reason that people want to be bringing in AI is that it, it makes things like quicker, easier, like faster. Like you'll be able to like diagnose faster, you'll be able to treat faster like this, like efficiency is what it's meant to be good at or why we want it, especially with like healthcare being so like imitated and like, you know, we're trying to free up clinicians to do as much as they can, like with a little time that they have in a day. So you want those trends and you want those stats to make it so that I can be as fast and efficient as possible. But like you said, we don't want to be losing out on those rare niche possibilities that are still real and valid and should still be taken seriously. So I don't. All I can sort of think right now is almost like, almost like we were talking about earlier, like a, like a, I'm thinking of like a flagging system where like the AI's kind of gone through, it's like distilled all this information and then it's kind of flagging in terms of like most statistically likely. But also maybe like here, here's like the most like serious concerning and kind of just like have those like quick flags that then like a clinician can look at and be like, okay, it's most likely this, but maybe I should check this because if it is this, that it would be really serious, like kind of thing. And then again, put that in the clinician's hands to then use their training and knowledge and experience to do with that information what they will in term. Yeah, it's so tough. Yeah, it's like the, similar to like pipelines that we use. Like, it's like that trade off between like specificity and sensitivity. Like, you can have 100% sensitivity if you, you don't care that you've got like, you're never going to miss anything if you don't filter out anything. So you'll catch all of the false, like, you'll catch all the true positives if you're willing to accept every false positive. And obviously you want to like, minimize. Like, you want to make sure you're like minimizing false positives and increasing true positives and like all that. And, but there's, there's usually like a, a point where you're like, okay, now it's like, when would sensitivity push like specificity the wrong way kind of thing? Like, when do we lose specificity because we're trying to be too sensitive to too many things. And I think that's the same here where you're like, okay, we want to be quick and efficient, but we also need to be accurate and we also need to be cautious. And now where's the balance between those things of like, if you're super, super cautious, you're probably not going to be that quick. But if you're like super quick, you might not be that accurate or cautious. And then you're like, okay, now where's like the sweet spot where it's still pretty quick. But we're not like just fully negating true positives that we probably should have been paying attention to. I think that's in terms of the actual kind of coding and training of it. Maybe I'm naive because I haven't had to do any kind of training of any sort of machine learning models, but I suspect that would not be as difficult as figuring out what you yourself or you as the researchers figuring out what the sweet spot is. I think once you know it, it should hopefully be okay to provide to the algorithm. I think it's figuring out what it is, like how much risk you're willing to take basically.

Yeah, I like your point about like you mentioned how you should have kind of two possibilities, maybe one the most likely and one the most fatal or like most serious. And that way a clinician knows they should probably treat for this, but maybe also check for this just in case. That was like a, a nice point that I think.

Yeah. And maybe it's maybe the most serious one is like a. You might just want to like ask a couple follow up questions just to really rule this one out. It's probably this, but just FYI, this is a possibility, a slim possibility, but maybe you want to ask a couple follow up questions.

The next question I have for you is what kind of training or human oversight do you think should go into building and testing these AIs, like whether they're simple injectors or any sort of AI system to avoid any sort of gender or racially spice.

I'm sort of imagining like, and maybe it's top of mind because. Do you still get emails from SickKids? Yeah. Did you see the like one about sky yesterday, like to kids AI?

Yeah.

So, because that's, I think that's top of mind. It sounded like they almost had like a three pronged approach and one of those prongs was like the sky service where they're like, we have experts here. If you want to implement AI, come talk to us. And we will, we have people who will be able to help you do that. Like, well, but also ethically and making sure that we're like HIPAA compliant and private and like all these different things. And so I'm imagining like there being some sort of on like every research team that or like any sort of like team that's trying to make AI. I feel like there should be like, I say this because I am one, but the science nerds, like the like computer people who are like doing the, the actual like coding but there needs to Be like that. Almost like a review board to be like, okay, with our expertise? Is the tool you made actually giving results that makes, like, biological sense that are, to the best of their ability, void of these, you know, biases? Or at least, like, makes them very clear. Because, like, like I said, like, with the sensitivity, specificity, it's not as though, you know, every pipeline that we operate has, like, 100% specificity and 100% sensitivity. That's often very difficult to achieve, if not impossible. But the limitations or, like, the what the, you know, false positive rate is, the what the false negative rate is. If there are certain gaps that, like, certain things that we're not able to detect, we make those really clear. They're huge disclaimers to anybody who is, like, using our tests. And I think that would be the thing where it's like, okay, we need to make sure that we've got some sort of advisory board that is, like, testing and making sure that, like, to the best of the ability of, like, them as experts, we're not including all these biases. And if there is a limitation, like, because maybe there's biases, because, again, like, the research that we've done in the past, like, is full of biases. Maybe there's not better research to be giving. But we've decided that the benefit of the tool outweighs the risk. But you then need to have a huge disclaimer about the risk or the limitations there. Like, if for whatever reason, which I doubt, there's no possible way to make one of those Pulse readers work for different skin colors, I strongly do not believe that's the case. I believe that we could do it. But if for some reason it's impossible, then that needs to be very, very explicitly stated of. This will not work, or at least it will not work as well for these. We can't just, like, pretend that the biases aren't there.

That makes sense. At least being transparent and acknowledging those biases so that the healthcare professionals or even the normal people that are using these things will have an idea that this isn't 100% accurate. So just take it as a ballpark.

Exactly. I think that's. I feel like it's like the PAN Canada, like, AI for health kind of thing. Like, basically the government of Canada have put out their kind of guidelines for AI in health. And I think one of them was transparency. And like, obviously other ones like that we've talked about, like, diversity and making sure there's minimal bias and ethics and stuff like that. But one of Them is like transparency, I think, because AI is like the hot thing and therefore the hot thing, like, makes money because people are interested in it. I think that that initial desire to make something that is proprietary or that's in a black box and like, that's yours, that you can make money off of it, I think, in my personal opinion, is a bit at odds with like, health care. I think when it comes to health care, there needs to be transparency and that's not the place to be like, in a big, like, I understand that, like, yes, in order to keep the lights on and keep making these discoveries, like, you need, like, companies need to make money and blah, blah, blah. But I think having these sort of like, proprietary black box, like, so that we can maximize profits at the risk of. Okay, well, there's not transparency. So the people using this product might not know if they're using it right. Or like, might not be aware if there are limitations, I think is an issue. So I think, yeah, transparency regulations.

And your idea of like having an auditing kind of board is also sort like having someone after like all the developments done, maybe checking ethically doesn't make sense. Does it also make sense, like, biologically or whatever other cases and just having like a fresh pair of eyes? They're kind of like also working towards the same goal to help, like, verify that, you know, you didn't introduce any biases that you or your team thought of because then you have like another team also checking it out and making sure that it's good to go.

Exactly. I feel like I'm a big proponent of like, multidisciplinary teams. Like, like, when. Even when it comes to like, you know, pipeline design, I may not, like, I think I could, I can tell you some things, but I may not be the best person to tell you, like, the best optimizations, like, for XYZ pipeline and like the, like the, the best coding practices for this particular pipeline. But I can certainly tell you that, like, if the results coming out of it are biologically, like, if they make sense kind of thing. And so I think you need those different people. Like, you need the coding experts, you also need the biology experts and like the clinicians and things, but you also need people who understand, like, legal and ethics. And then you also need people who, you know, whether it be through research experience or lived experience, like, have things like biases like diversity, having that front of mind so that this is something that they can kind of like check and yeah, audit before we just send it out in the world is I Think a good idea.

My next question for you is, what do you think it feels like when a patient's denied care not because of medical necessity, but because of an algorithm classifies you as that way? And how can future healthcare professionals be trained to recognize and challenge biases like these in medical systems or software and just like, not just keep doing what, what's been done to actually, you know, think and challenge that Maybe some of the things that some of the tools or the processes that they're doing might inherently be bad.

Yeah, I think that's, I think it's like it'd have to come from both, like the individual, like clinician type things, but it would also need to come from like the, the environment that they're working in, basically, or like the, the company or the organization that they're working with. I think organizations can do a lot by, you know, promoting dei, making sure that there's equity and inclusion within the workplace, ensuring that, like, employees have to like, do like, DEI training, like, making sure that that's in place and then, you know, because hopefully clinicians or like, whoever the employee is is going to be coming like, in good faith with the hope that they're trying to learn and get better. But like, even if they're coming in with like, I don't have any biases, like, at least make them take the training and they can, you know, just like, I think sometimes it's like knowing what the limitations are, Knowing what your limitations are is, does a lot. But unless you're kind of like prodded to like, you're like, a lot of people aren't going to look at that, like, they're not going to kind of interrogate their own biases. So you kind of need to give them the opportunity to do that through like, mandatory practices and then. Yeah, just sort of. Yeah, I think a lot of it comes like from like the system or like the organization down and like fostering and that, that environment where like, that's just sort of like, just like how you'd have like best coding practices. Like, you know, you're going to comment your code, you're going to like, do proper, like, like commits like to your repos and stuff. I think making sure that the employer is fostering these best practices when it comes to working with patients, especially patients of marginalized communities, and implementing these good listening skills and interrogating biases. If they're offering that training and setting up those best practices and modeling that, then I think the employees are going to be following suit and that'll Just be better for everybody and then it's just kind of like a lived practice thing.

And now the final question I have for you was if you were on a team creating a new clinical algorithm, what are some like safeguards or must haves that you'd recommend to ensure that the AI doesn't cause harm to any demographic group, even unintentionally.

So these be safeguards like as I'm making the tool or if. I'm sorry, if I'm using it already.

If you're making one.

If I'm making one, yeah. I think I would probably take almost like the CIHR model where I'm like from the beginning, like I'm imagining like when you sort of map out a pipeline and you're kind of like here's like the architecture, like the workflow flow doing that as well for like the design. When it comes to like the kind of what data is coming in and what you're, what you're expecting to kind of get out type of thing, I think starting with that fundamental and like making sure that you're bringing in like a wide variety of data would be like my, that's number one. And if you're, if we're not doing that like then we can't move forward with building this tool. But then yeah, I think it's also, I don't know exactly at what points, but there would. I think there's probably just like some stop gaps. Like I suppose often with AI like the, the idea is like it's kind of a, you don't see the middle. It's just like you put something in, you get something out. You're not seeing kind of like the steps, but like as you're building it, if you're able to kind of like put these stop gaps in and be like, okay, from like step one to step two, like did anything weird happen here? Yes. No. And like make sure that you're kind of, you know what is happening and like what the, like what the algorithm is doing. Because I think, yeah, sim, like it would be the exact same for like the, the clinicians using it. Where I'm like, you want to know what the tool is doing. You shouldn't just accept that like, oh, it's like a black box. Like you shouldn't just accept like that. I guess that's just like what this tool does. And so I think for you as like the developer making sure that you know what's happening at like each step and why and if you think that's fine or not would be important. And then, yeah, just lots of rigorous testing. I like, at the end, I think on like a wide variety of people, I think sometimes again, because it's probably not like super well regulated, we're able to just make these tools and then just put them out there. And I don't know how often like how well they've been tested or like on what demographics. So I think having a sort of, I'm imagining almost a bit like, like a, A bit like a drug trial where you're like, okay, we've made the drug, but now we've got like phase one testing, phase two testing. I don't know if it would need to be like as rigorous, more or less. But having that sort of planned approach, lots of diverse data, checking it as you go, and then having like maybe some phases of testing and auditing to be like, okay, is everything looking like we'd expect? And then kind of then when you're feeling confident being like, okay, we can release it to the world.

Yeah, that makes sense. It's like a really good, like holistic, detailed view taking things step by step, it really improving and just making sure you know that what's going on at all times and you're not introducing biases because you're taking things step by step and making sure that you understand the way the AI is kind of thinking.

Yeah, I do think that'll also take like the sort of the industry and like society incentivizing that. Because again, if we're going to go with like the kind of capitalistic, like, like the move fast, break things kind of vibe because that like just get it out there and that's not even like a tech, like a tech exclusive thing. Like even in the research community, like there's the, like the getting scooped where you're like, you want to get your research out there and publish before somebody else who's in like a similar field to you hones in on the same thing maybe and publishes first. But sometimes you don't want to be rushing your research. And so it's like, yeah, I think we also just need to be incentivizing people to take it slow and go the iterative route, especially when it comes to health and not incentivize. We'll just like get it out quick and be the first one. Because then you get to be the first one and you get to be like the, the innovator and you get to be. Because like, I think innovation is important, but I also think especially when it comes to health, maybe taking a measured approach is helpful. It's just we don't really incentivize people to do that.

Yeah, that was a good point. Like we should be invent incentivizing more of good practice versus capitalization, especially in healthcare.

Yeah, exactly. There are certain things like that, I guess, COD speed, like just make. Try to make your stuff quick and make money. But I was like, I don't think healthcare should be in the place for that.

Yeah.

Yeah.

And that's all my questions for you for today. I really appreciate you taking the time to come and share your opinion on these topics. Do you have any questions for me or any closing remarks you want to make before the end?

No, I think. Thank you very much for including me. This has been a very interesting conversation. I've really enjoyed it and I hope some of what I've been able to provide has been helpful. But yeah, no, I think this has been great. And you're clearly, I think, very well versed in the topic. And so I'm confident you'll be able to put together. I assume this is for like a project of sorts. I'm sure they'll be able to distill all of these interviews into something very insightful. So, yeah, thank you for including me.

Thank you.

Awesome. I will chat to you later.