This video is going to be about helping the
doc make the right diagnosis. It’s clear why so many of our viewers relay
frustration with their docs. And this video sounds like – in the beginning it’s
going to be criticizing doc’s it’s not a criticism. It’s the truth. The evidence
would indicate that docs make the right diagnosis between half the time and five
percent of the time. That sounds bad, right? and it is bad. but
I’ll cover an article which which shows that, but the question is not so much how
to criticize docs the question is how to how to make it better
well that brings up the question of why can’t we use artificial intelligence now
think about chess this is a picture that was done in 1997 the fellow on the
screen with his head in his hands is Gary Kasparov. At that time the reigning
world champion in chess he that was when he realized he had lost to a computer
deep-blue actually at that point it was called Deeper Blue or Very Deep Blue but
not just deep blue and that’s that’s an important concept which we’ll talk about
a little bit later the year before Kasparov had also agreed to a challenge
from Deep Blue. Deep Blue came out and won the first game but then lost the
match the programmers the consultants the chess masters all got together and
learned what they where they fit – where Deep Blue failed in 96. and they’ve
learned a lot of other things as well and put all of that new stuff into the
program. Kasparov in 97 was frustrated and he said, “I think they cheated! I still
think they cheated!”Tthere was a whole book written about this potential
cheating thing. but if you google Kasparov and GoogleTalks about Deep
Blue it’s really clear he says, ” yeah in 1996 it was a good match in
97 I lost. ever since it’s just – there is no human that could in any way keep up
with artificial intelligence in terms of playing chess.” Now wait a minute I’m
going down a long story about chess. right so how is that analogous – how was
playing chess analogous to getting a medical history and making a diagnosis?
If you think about it playing chess is all is a whole bunch of if _____, then
_____. In other words, making a parry reacting to the other side. Guess what!
Making a diagnosis is – if A, then B. In other words – if a patient’s over 60 years
old, then I have a lot of things to worry – to think about. If male ,if they have a
his family history , if they have so again both of these activities are very much
an if-then type of activity. The difference between deep blue in 96 and
97 – if you look just look it up on Wikipedia, they’ll say that Deep Blue got
up – went from planning like five to seven moves ahead – to up to ten to twenty if
they needed. I believe that’s what they said. Again, it
just outgunned the individual human brain in terms of the scenarios it could
consider when it thought about the next item. Again that’s the problem that we
have with with docs in making a diagnoses There’s just too many scenarios that we
have to consider. We’re human. We’re thinking about “I got to get through this
patient. I’ve got to meet my family for dinner, I’ve got six patients
between now and then.” He’s got a bunch of other stuff on his mind and he’s not
going to he or she’s not not as likely to spend the time that they need or go
through all of the scenarios that they need so where does that leave you as a
patient? Symptom checkers . You wake up in the middle of the night with stomach
pain fever sweats. you know the story. It’s 3:00 a.m. You’re not gonna call a
doc. You don’t want to go to the ER. So you start looking it up on dr. Google
the Internet. Good luck on your chances with the
internet because I’m gonna cover a study which shows that these symptom checkers
are really bad. They make the diagnosis maybe a third of the time – and we’ll talk
about that in just a minute. So what’s the difference between those symptom
checkers and Deep Blue? Again it was that point that deep blue was constantly
learning, constantly improving. Symptom checkers are built on what we call an
expert logic. What that means is whoever’s running the program decides
who they think are experts, who they can get out of that group that will actually
come do this. Then those groups, those ” experts” sit around a table and
decide what rules should be used. The programmers program i. There you
have it. There is no relearning – not a significant relearning process or at
least there hasn’t been until recently. Now there’s a thing called K Health.
Yes it does have those history symptom checker components =but like Deep Blue –
it’s starting to learn. It’s been learning in the years that it’s being
been developed it’s been learning with tens of thousands of docs and millions
of patients in terms of their experience and getting better every day. That’s the
way Toyota =if you talk – I worked for Toyota for 10 years and the Toyota folks
will tell yo, “look we didn’t get better by any leap. There was no major
breakthrough that we created. In fact, we grew the way we grew and we attained
the status that we have in the automobile industry not with leap frogging
but learning a little bit – Kaizen. Again that’s what that’s what
we need in this symptom checker. This artificial intelligence quote artificial
intelligence process for helping doctors now the history is a huge issue. In fact
if you go back through history there are a couple of docs that were very well
known and very well respected – Hippocrates and William Osler
I’m going to go back and share my own experience (and pardon me for putting
myself in the same the same sentence or the same breath with those two docs) but
when I am a good doc (and whether I am truly a good doctor or not we’ll talk
about a little bit later) because I think I have some of the same I think we all
have some of the same problems starting off in the ER one of the things that
struck me was that other dogs tend to talk to the patient less listen to the
patient list and order more labs I talked to the patient and I felt like
you know you could get a lot more information just by listening and you
probably don’t need as many labs Then I found out later I thought – it was
kind of weird – until I found out later William Osler had the same issue he said,
there was a quote one of his most famous quotes was – he was in the amphitheater at
Hopkins and a student doctor was getting a history from a patient he interrupted
very frustrated and said, “listen to the patient! He’s telling you the diagnosis!”
So again it’s there’s a huge effort and like we said a minute ago – major failure
rate around getting a diagnosis . I said I was going to cover two studies
now the the problem is I’ve got a lot of details in both of these studies
that I’d like to cover but I’m just not going to have time in this video so I’m
just going to go through these very very quickly to help make the points that we
need to make again for this video this is Ateeva Mehrota
This was in the British Medical Journal June 2 2015
I actually have some history with Mehrota he doesn’t know that but I do. He
did a study on telemedicine back when I was the CMO chief medical officer
for MD LiVE – one of the top two telemedicine companies at that time – and
the end of his study he basically said, “look telemedicine doesn’t replace doctor
visits,” and so I ended up having to go around and say well that was a critical
comment on that study what do you respond – I mean I first heard it and
– my response was, ” what were his assumptions? – in terms of his study – his
assumptions were that telemedicine was going to replace doctor visits. That’s
sort of like saying home videos and streaming is going to replace the movie
theaters you got your assumptions wrong and actually I think he did that with
this study as well (Tthat’s why I went into that into that – in
little diatribe.) He went to the end of the study he basically took the 23 major
symptom checkers out there and he said, “they don’t work – they they get it right
30% of the time.” So here’s the conclusion symptom checkers had deficits in both
triage and diagnosis actually for the emergent cases in terms of triage. triage
use I’ve got chest pain where do I go I can’t diagnose exactly what the chest
pain is yet but I know what I need to do next in those urgent triage cases the
symptom checkers were 80 percent right so that’s – whether that’s better
or worse than then the docs like you can’t really say – because the
doctor study doesn’t compare that but it does compare diagnoses – and we’ll cover
that in a minute. The point was though he said the symptom
checkers are not that great. they only get it right 34% of the tim. again
what this study adds is that symptom checkers have deficits in both
diagnosis and triage. He missed the assumptions again like he
did on the first study back in telemedicine. Here’s how he missed it.
This is a JAMA study. I’ll cover both of these studies in more detail I
know there’s a lot of viewers that are very interested in details around
studies how did they pick which symptom checkers to look at what were the
scenarios that they put in front of symptom checkers what were the scenarios
that they put in front of doctors here how did they put the doctors there I’ll
cover those in later videos but here’s what you need to know about this study
Mehrota. This study was done two years before and published in JAMA Internal
Medicine Mehrota had obviously not done his lit review or he would have noticed
this. This line these open circles is the how often the docs got it correct on the
scenario when they on the easy cases and that was about half the time this – this
side of the graph shows how often they got it right when it was a little bit
more difficult case in other words what 5% of the time so in difficult cases
docs got the diagnosis correct five – six – eight percent of the time. What’s even
more disturbing is this. On the easy cases the docs said, “I’ve got like a 70
percent chance of getting that right. On the times when they only got one in 20
correct -they had almost the same level of confidence. In other words docs think
they’re right even when they aren’t. Why is that? (Because they’re human.) & because there’s
no gold standard. That’s what we need in medicine in terms of making a
better diagnosis. We need to have some sort of mechanism for collecting the
information and finding out was that correct or not – and how can we improve
that. That’s going to take a team approach – there’s no individual doc
that’s going to be able to accomplish this. Now you may go back speaking of
those I will cover just a couple of details about the Mehrota study of the
of the symptom checkers. You may say “you got a bunch of bad stuff off the
internet.” maybe ASKMD maybe you think that’s
bad well – ASKMD was in there but Harvard was in there and I don’t really
think that ASKMD was any worse than Harvard and some of the others that were
there ASKMD as I mentioned before – a lot of people use both of
those. WebMD was in there as well Again a lot of people use these so I’m
gonna skip over a lot of these details and again please just comment down below
I am planning to go back and cover these studies in much more detail. But as you
know from many of my comments I’ve got three or four months worth of topics
that I need to cover and things happen so I often don’t get to to go into the
details and plans that that I’ve developed so let’s go back and think
about this one more time. What happened in 96 and 97? Well there’s a
comment this came from an IT magazine – “20 years on from Deep Blue and Kasparov -How
a chess match started the Big Data revolution” Again that is an interesting
thing to think about. At that point in time the standard –
the gold standard for chess had to be handed over from an individual man to a
team driving a computer. Are we ready for that in in medicine ? I think
we’re more than ready – I think we needed this more than 20 years ago. What’s
the difference? Is K Health really going to accomplish what they plan –
what they think? Like they say -don’t take my word for it – look up
K Health on the web. that’s what they say KHealth has learned
from the experience of thousands of doctors- 10,000 doctors and millions of
real cases and they’re constantly learning. I’ve talked with – well I will
I’ll talk about that in just a minute – about my own activities with Kay health
now I have a disclaimer I’m starting to I’m probably going to do some work with
them. I wanted to go back and cover William Osler and the Hopkins amphitheater -and
make one comment and comment about me thinking I was always right in the ER
and other docs thinking they were always right and all these docs thinking
and William Osler – all thinking that William Osler was right how often do you
think he really was right ? That’s my point.
We all think we’re right because there’s no continuing learning process. I used to
work for Toyota worked for them for about a decade and did a lot of stuff to
set up and and improve the healthcare delivery for their employees/families. It was at that point
going from 80 to 120 thousand lives. Now it’s much more Toyota . It was
very clear. They said, “you know – we went from a small shop basically doing –
(they started off as weavers in Japan and they developed a weaving mechanism)
and then they got into cars. They were considered very poor quality when they
started out but look where they are today. They’re in the lead in terms of
the world’s automobile production and they’ll say, “We didn’t get there by some
giant leap. We had some giant leaps but not that many. We got there by watching
what we did and improving a little bit every day.” That’s their term. They call it Kaizen.
It’s a Japanese word . Is K Health going to be the Deep Blue – the Kaizen for
medicine – doctors’ history, making a diagnosis? They certainly want to be.
As I mentioned a minute ago – I’ve got a disclaimer. I’m gonna do some work with
these guys I went to a New York a couple of months ago &
met with Ran Schaul – the Chief Operating Officer on the left and Allon Bloch on
the right – the founder and CEO. Can that team accomplish this? Can they accomplish
all that they’ve got on their plate? I don’t know. Why did I join them? I can
tell you a couple of reasons I like Allon and Ran and the rest of
the members of the team. I like them a lot. So is it that?No it’s
not the social issue. Is it money? I’m making a little bit – but I’m not not
making very much with them. Here’s the reason – this activity needs to happen.
Somebody needs to develop the gold standard for medicine because we need to
improve the way we practice medicine. I’m going to be doing a couple of things. I
may actually even open up my panel of patients – those of you who want to be
patients may have gone in and seen that I’m not seeing and taking new patients
haven’t done it for a few months – one of the things I’m considering doing is
opening up and using K Health. We’re not quite there. I’d love for you to go in and
check out K Health tell me what you think. Put some comments down below. Again, as
usual – thank you if you’ve made it all the way to this far in the video. Thank
you again for your interest.


  1. Thanks for this video. I think that narrow AI for primary health care will happen and in a decade make a huge difference, especially for preventative care. I am sure that you see a lot of complaints by members about the health care system, and I think that many doctors will embrace new technology over time to improve overall service. Narrow AI will complement rather than replace a doctor.

  2. Today it's not as issue, the Doc is still the final arbiter and critically they have an experiential knowledge.
    Will it become an issue when there aren't so many Docs, AI's means job losses ?, they perhaps become reliant on the AI to take the load off the "front end" when the pressure's on !
    In the ER person to person communication is critical, is speaking to an AI going to deliver that ?, perhaps with time and familiarity ?
    I'm sure as they become more sophisticated they can perhaps take over a huge amount of the diagnosis and perhaps even the treatment of "their" patients
    I quite like however Flying with the Knowledge that the Pilot can Fly the plane WITHOUT the auto pilot just in case !
    As a sophisticated & knowledgeable Assistant, I think they'll be a godsend.

  3. This is so beyond necessary. Patients shouldn't know more than their docs. I'm tired of having to explain things to the docs I see. I've literally stopped going. Not worth it.

  4. Thanks for the app info! I downloaded to my iPhone, entered symptoms for ongoing sciatica issues and the diagnoses was spot on.
    Something similar was developed by NASA for diagnosing space shuttle faults. It was adapted by the appliance service industry for their use.
    It learns by receiving real time input from service techs, about the guidance provided.
    With your input K could be a success.

  5. Another thought provoking video. That is the first I have heard of K-Health. Sounds like a potential breakthrough. I worked with InterQual 15 years ago to automate their algorithmic system of books into handheld devices. That was not too difficult but it had no feedback loop to judge the correctness of the results. Later I worked with CMS on nursing home casemix and quality determination. It still frustrates me that with the huge databases we have available today that we can't track outcomes to process. The HIPPA restraints complicate the issues, but I believe the biggest issue is the human interface. We don't know how to get actionable data from patients. And many of the current interfaces require far too much of skilled doctors. I hope your work with K-Health is successful.

  6. Very good VDO! Thanks for that. I guess the difficulty with using AI for (complex) medical diagnoses is that there are too many variables which are unknown to make conclusions on treatments. At the end in order to be able to make the IF…THEN statements you will need to have the feedback of Actions taken and know all the other variables that may have influenced the end result in order to have a useful feedback-loop. When using the analogy with chess, the chess-game is much much simpler than the human body, with much less variables. In mathematic terms they are the same with respect to that both "problems" to be solved are exponential (which means there will never be an optimal solution). But again the human body is much more complex then the chess game. As some would say "It is not that simple" :-). However I think we should keep striving to improve tools like KHealth, but keep your expectations modest. Thanks!

  7. Thank you for yet another super video. I have been having some calf soreness, so after watching the video, I downloaded and used the APPLE K-Health app for a possible 'diagnosis' starting point. Result, going this Thursday to LifeLine Screening for their ultradsound scan package on the leg calves, carotid artery, etc. I have been doing low carb/semi-keto for over a year, and have had to manage my electrolytes more carefully as I am now 55 pounds lighter. I have learned so much watching your videos, watched many of them more than once! Thank you again, you have made a very big difference in my life!!

  8. My respect for you, as a doctor, grows with each video. … I have been doing estimates since 1988, along side old guys with 50 years of experience. I brought in my own detailed computer formula(s), in 1992/1993, which I constantly adjust and compare to the real world. I also, follow many different computer weather models down to the hour. Usually, computer beats man (including the old guy), but not always. I trust neither. But together, computer plus human, I might trust the combined wisdom to have s 95% rate. The bigger the view, the better the choice. Certainty can be a sign of wisdom and a sign of folly. I feel more often, it is a sign of folly.

  9. Hey Dr. Brewer. I’m so glad you are talking about this. AI maybe smart but it’s still a tool right now. Docs may get dependent on it and forget how to look at symptoms the “Old Country Doctor” did just fine for millennia. We in the “functional” community may not see as many patients I guess but I would much rather do it that way.

  10. And furthermore Dr. Brewer you can read a new Medium post from The Atlantic where they state that sometimes strings of code are attached to other strings of code poorly. Often hooked together in many many systems, which can cause outages like what happened with 911 calls. If a hospital for example uses this AI, some bad code in another system could might cause the AI to fail. We have not thought out well enough how algorithms can work together much better than they do. They may need to re-write code all over again.

  11. There’s new AI in development called Suki…it sounds very promising! Gosh, thanks for your honesty. My son is in medical administration and it’s a HUGE issue!

  12. I'm very happy that AI is being implemented as a diagnosis tool and and very glad that you're involved with it. I do have a concern that the datasets it'll be learning from will have a dearth of data about how symptoms present in women though, as most medical research has been done on men. I also wonder if it'll take into consideration racial differences, as again most research seems to have been done on white men.

    Can you please let me know how this bias will be addressed, as when machine learning systems have been allowed to educate themselves using just the internet for example, they've been shown to become severely mysogonistic very quickly, due to the environment they were set free to learn from. I see it as an opportunity to address these knowledge gaps if they're addressed directly as part of the development of the system and to continually improve the knowledge from new data as it's found.

Leave a Reply

Your email address will not be published. Required fields are marked *