CCSVI treatment effects: why patient self-reports not enough
Posted: Mon May 02, 2011 1:28 pm
Those of you who know me (from the past few days) know that are am a strong advocate of conducting objective, bias-free/limited research on the effects of ANYTHING.
As you know, because studies take so darn long to conduct, many patients rely on stories and experiences of others decide what to do. (this may be related to a procedure or as simple as a vitamin or supplement - the concepts below apply to all of this).
I thought I'd use an analogy with something unrelated to see if maybe some who are not familiar with scientific research can get a different perspective on things. The analogy is about infection control in hospitals (hospital workers need to take mini-courses on infection control to help reduce rates of nosocomial aka health care related, infections).
Health care related infections are common and sometimes preventable if we adhere to findings of the many studies done on this topic. (there has been many studies on something as seemingly simple as hand-washing with different types of soap!)
These infections are most often acquired from devices that are necessary to saving patient lives/organs (e.g., surgeries, ventilators for those who cannot breathe on their own due to an acute disease...etc.)
Now imagine if public health folks are trying to determine the prevalence (frequency) of health care-associated infections in order to come up with ways to reduce the rates. They would need accurate, objective numbers for this, right?
Let's say they ASK (not require) hospitals and other health care settings across the country, with the intent of tallying up the numbers at the end. The problem with this approach? because this is a voluntary reporting system, some hospitals will report some incidences, but not others. They are not trying to be "unethical" (because noone gets in trouble for saving a life via a ventilator, which caused a treatable infection), but sometimes other aspects of the job get in the way, one person thinks another one reported it, it's not a priority to report bc its voluntary ...etc. (btw, this is an example, health care settings are actually now required to report all infections, at least in Canada).
The numbers that public health folks get are likely to be (a) underestimating true prevalence of ALL infections, and (b) over-represented by certain "rare" infections that the hospital might be more inclined to report because they are "strange" and unexpected.
The final numbers are NOT representing the true rates of all infections and may be over-represented by the "rare" ones!
What does this have to do with MS-CCSVI?
Patients report their experiences on forums and other means on the Internet voluntarily (if they feel like doing it). Individual doctors performing these procedures may also discuss the "results" of their patients to journalists writing newspaper articles on the topic (not peer-reviewed or based on an official study). This is important to provide emotional support to fellow patients experiencing similar things. But patients often report their "readings" as evidence of the results of CCSVI. What is wrong with this you ask?
Well, as humans, we are all prone to bias. We tend to report and remember very salient (strange, over-the-top, unexpected) information more than the "average", common or expected information. (Social psychology research has shown this for years).
Of all patients getting a procedure done, most have not had objective measurements of their symptoms before and after the procedure (levels of spasticity, urinary problems and even depression can all be measured objectively). Therefore, we cannot compare their befores and afters objectively; only based on subjective experience and memory.
Of all patients who have had the procedure done, by virtue of human tendency to focus on the "extreme", it may be more likely that those with "extreme" results will feel the urge to come online and share with their experience with the rest of us. (NOT everyone of course; some are already active online and report whatever they go through - but we're talking about 1000s of people who may/may not report their experiences).
What ends up happening is we are more likely to read about experiences that were so incredibly beneficial that they are in fact not the norm in terms of what to expect OR experiences that are so terrible that they too do not represent what most patients should expect (e.g., incredibly good = the rare cases of complete "cure" of all symptoms; incredibly bad = the rare cases of death, thrombosis leading to other probs...etc.)
Of course, you know that most patients will not fall under either of those categories in any form of treatment (google "normal distribution" to see why). And again, there are those patients who are active of forums and online, who report whatever their experience is - even if it is in the middle of the extremes.
But by and large, we as humans have a tendency to want to share our extreme circumstances and experiences. (Same reason why websites rating doctors tend to have mostly two types of reviews... some saying the doc was "the best ever", some saying doc was "the worse ever"... the majority of patients who didn't feel strongly either way are less likely to take the time to complain or praise the doc online). <-- see social psych research on all this.
Same goes for physicians voluntarily reporting their patient experiences... they may focus on the very good or the very bad (unless they make a concerted effort to record and report on EVERY patient they treat, in which case, they may as well conduct an official study and at least get a publication and official recognition among colleagues for their work!)
Even on the prevalence of CCSVI in MS: what if those who tested negative are less likely to join a forum discussing CCSVI because it doesn't apply to them? Fewer folk without CCSVI will be represented on a CCSVI forum, causing us to overestimate the true prevalence of it in MS if we rely on the voluntary patient-self reports alone.
The result? If we rely self-reports to see what the results of any treatment is, we get a skewed view of what is actually happening - whether it's infections or CCSVI results. This is a basic premise of why scientific research is used to answer important questions.
As you know, because studies take so darn long to conduct, many patients rely on stories and experiences of others decide what to do. (this may be related to a procedure or as simple as a vitamin or supplement - the concepts below apply to all of this).
I thought I'd use an analogy with something unrelated to see if maybe some who are not familiar with scientific research can get a different perspective on things. The analogy is about infection control in hospitals (hospital workers need to take mini-courses on infection control to help reduce rates of nosocomial aka health care related, infections).
Health care related infections are common and sometimes preventable if we adhere to findings of the many studies done on this topic. (there has been many studies on something as seemingly simple as hand-washing with different types of soap!)
These infections are most often acquired from devices that are necessary to saving patient lives/organs (e.g., surgeries, ventilators for those who cannot breathe on their own due to an acute disease...etc.)
Now imagine if public health folks are trying to determine the prevalence (frequency) of health care-associated infections in order to come up with ways to reduce the rates. They would need accurate, objective numbers for this, right?
Let's say they ASK (not require) hospitals and other health care settings across the country, with the intent of tallying up the numbers at the end. The problem with this approach? because this is a voluntary reporting system, some hospitals will report some incidences, but not others. They are not trying to be "unethical" (because noone gets in trouble for saving a life via a ventilator, which caused a treatable infection), but sometimes other aspects of the job get in the way, one person thinks another one reported it, it's not a priority to report bc its voluntary ...etc. (btw, this is an example, health care settings are actually now required to report all infections, at least in Canada).
The numbers that public health folks get are likely to be (a) underestimating true prevalence of ALL infections, and (b) over-represented by certain "rare" infections that the hospital might be more inclined to report because they are "strange" and unexpected.
The final numbers are NOT representing the true rates of all infections and may be over-represented by the "rare" ones!
What does this have to do with MS-CCSVI?
Patients report their experiences on forums and other means on the Internet voluntarily (if they feel like doing it). Individual doctors performing these procedures may also discuss the "results" of their patients to journalists writing newspaper articles on the topic (not peer-reviewed or based on an official study). This is important to provide emotional support to fellow patients experiencing similar things. But patients often report their "readings" as evidence of the results of CCSVI. What is wrong with this you ask?
Well, as humans, we are all prone to bias. We tend to report and remember very salient (strange, over-the-top, unexpected) information more than the "average", common or expected information. (Social psychology research has shown this for years).
Of all patients getting a procedure done, most have not had objective measurements of their symptoms before and after the procedure (levels of spasticity, urinary problems and even depression can all be measured objectively). Therefore, we cannot compare their befores and afters objectively; only based on subjective experience and memory.
Of all patients who have had the procedure done, by virtue of human tendency to focus on the "extreme", it may be more likely that those with "extreme" results will feel the urge to come online and share with their experience with the rest of us. (NOT everyone of course; some are already active online and report whatever they go through - but we're talking about 1000s of people who may/may not report their experiences).
What ends up happening is we are more likely to read about experiences that were so incredibly beneficial that they are in fact not the norm in terms of what to expect OR experiences that are so terrible that they too do not represent what most patients should expect (e.g., incredibly good = the rare cases of complete "cure" of all symptoms; incredibly bad = the rare cases of death, thrombosis leading to other probs...etc.)
Of course, you know that most patients will not fall under either of those categories in any form of treatment (google "normal distribution" to see why). And again, there are those patients who are active of forums and online, who report whatever their experience is - even if it is in the middle of the extremes.
But by and large, we as humans have a tendency to want to share our extreme circumstances and experiences. (Same reason why websites rating doctors tend to have mostly two types of reviews... some saying the doc was "the best ever", some saying doc was "the worse ever"... the majority of patients who didn't feel strongly either way are less likely to take the time to complain or praise the doc online). <-- see social psych research on all this.
Same goes for physicians voluntarily reporting their patient experiences... they may focus on the very good or the very bad (unless they make a concerted effort to record and report on EVERY patient they treat, in which case, they may as well conduct an official study and at least get a publication and official recognition among colleagues for their work!)
Even on the prevalence of CCSVI in MS: what if those who tested negative are less likely to join a forum discussing CCSVI because it doesn't apply to them? Fewer folk without CCSVI will be represented on a CCSVI forum, causing us to overestimate the true prevalence of it in MS if we rely on the voluntary patient-self reports alone.
The result? If we rely self-reports to see what the results of any treatment is, we get a skewed view of what is actually happening - whether it's infections or CCSVI results. This is a basic premise of why scientific research is used to answer important questions.