Online Physician Rating and Ranking Services are Biased and Unreliable

Please tweet and retweet Tweet

A study of online physician review, rating and ranking services such as Angie’s List, healthgrades.com, RateMds.com, Vitals.com, and Yelp.com by professors of the University of Maryland’s Robert H. Smith School of Business and the School of Public Health at the University of Minnesota found that they are biased and unreliable. This is particularly worrisome as physician ratings are gaining popularity among patients and an increasing number of patients consult these rankings before making an appointment to see a prospective surgeon. According to a 2010 Pew Internet and Life Project survey,  59% of U.S. adults have looked online for health information.

The study looked at the way patients determine which physicians to rate and the intensity of opinions they express using 4 data sets:

1. online data from RateMDs.com, one of the largest American physician rating services;
2. an offline patient survey conducted by the consumer advocacy group Consumers’ Checkbook for the cities of Denver, Memphis, and Kansas City;
3. the U.S. Census 2007 Economic Census, which was a source of information about population and median income in the three cities examined;
4. and state medical board websites.

They asked if patients using these online sites reviewed the full range of doctors (including those viewed positively, neutrally, and negatively); if they were biased toward giving negative reviews to doctors (which the authors characterized as “bad mouthing”); or if they were biased toward giving mostly positive reviews (providing what the authors call “sounds of silence” about poor caregivers).  It turns out patients posting their opinions about doctors on online are much less likely to discuss physicians with low perceived quality and are more prone than offline populations to exaggerate their opinions,.

The authors came to two major conclusions:

First, they found that physicians with low ratings in offline surveys are less likely to be rated online, therefore supporting the “Sound of Silence” effect in selecting what physicians to rate.

Second, although authors found a strong correlation between the online ratings and offline population opinion, the association is strongest in the lowest quartile of opinions. These results suggest that online ratings are more informative when identifying low-quality physicians, but not as effective in discerning high quality physicians from median ones. The authors also confirm that patients are most likely to provide ratings for their most flagrant or negative experiences with physicians.

By 2014, an estimated 15% of social media reviews are expected to be fake, according to the technology research firm Gartner Inc.

In 2009 Lifestyle Lift reached a settlement with New York state over claims it had employees post false customer endorsements/reviews on third-party websites, including RealSelf.com, and on some 10 websites the company had created to appear as consumer generated praising of the procedure. Lifestyle Lift was ordered to pay $300,000 dollars to the state, and it agreed to cease the practice

Doctors and professional medical societies were initially caught like deer in headlights when these online ranking and rating (but mostly raking) sites appeared. It is a premise of marketing that the few dissatisfied customers are the most vocal. That does not always mean that the product or service is inherently bad. Some irate and sometimes irrational patients without valid complaints posted highly negative reviews. These hurt the reputations of some surgeons and they lost patients, not always through any fault of their own. Complaints to these review and ranking sites went unanswered and they refused to remove negative reviews. Lawsuits against the defamers were hard to initiate because the reviewers were largely anonymous. One surgeon was somehow able to track the negative reviewer at which time it was found out that that person had never even had the surgery that was complained about. On the other side of the coin a doctor in northern California and a cosmetic group of doctors in New York were found to be posting false positive reviews. The New York group was using employees to post multiple fake positive reviews. They were cited and fined hundreds of thousands of dollars related to false advertising. The California doctor hired a public relations firm to boost the practice and the firm posted the fake positive reviews.

Next came companies that promised to remove negative reviews and track them as they appear for removal, for a price of course. Interestingly after some investigation it was noted that some of these companies were actually the source of the negative reviews and some were even the company that owned the ranking/review website. I was contacted by someone who was writing a book about these companies to expose them.

A negative review was posted about me that contained general information that could be obtained from my website but did not mention having any specific surgical procedure. This coincided in time with the first of the companies calling me to offer their paid services to remove negative reviews. A colleague of mine had a certifiably unstable patient who underwent a minor procedure, so minor that it would be hard to have a complication much less a bad result. This patient went literally into orbit posting negative reviews and even creating websites that would appear on searches for that doctor's name.

A well-respected facial plastic surgeon was getting obliterated online by multiple patients who were accusing him of molesting them under anesthesia. What really happened is that when the economy started heading south, one of his neighbor’s businesses went south. Instead of trying to resolve things the old-fashioned way the neighbor decided to blackmail the successful neighborhood facial plastic surgeon so he had his wife go online and pose as patients who had all sorts of terrible experiences, including the molestations! The physician called on the FBI and the local police but some of the sites that had posted the face accusations refused to remove the posts.

In another case a doctor's reputation went from good to bad virtually overnight because of a sudden posting of multiple negative reviews. All those negative patient reviews showed up on one particular discussion forum. While an outside firm monitored the forum a change made on the forum allowed them to see the IP addresses behind the posts. It then became clear that all of the negative posts came from the forum moderator who placed those posts because the doctor was about to begin a competing forum.

The main problem is the Health Insurance Portability and Accountability Act (HIPAA) of 1996 prevents doctors from discussing patients so disgruntled and anonymous individuals can pick fights over their quality of medical care whether it is true or false and physicians are almost powerless to defend themselves. In some instances, attacks are made with the sole intention of inflicting irreparable damage to careers and reputations. In the life-and-death world of health care, a disgruntled few can impact the medical decisions of thousands who are struggling to make sense of new marketplace mandates and regulations.
Doctors have collectively come to the conclusion that the only way to deal with this situation is to drown it out with online content like blogs and multiple websites and have happy patients post reviews. Unfortunately, happy patients in general are less motivated to post reviews no matter how happy they are. They got the result they wanted and now they want to go on with their lives. They have nothing more to gain by posting a positive review.

One doctor accused Yelp of  filtering out positive reviews and highlighingt negative reviews. "They then contact doctors to purchase ads." When asked why the filter behaves this way, the company responded that those "reviewers do not review enough to be considered salient." Their logic is one or two postings on their site are far less credible than those from a person who reviews regularly. In addition, Yelp's semi-anonymous atmosphere seems to naturally attract people with what psychologists call a "negative bias," offering a platform for users to vent their generalized rage with impunity - regardless whether the issue concerns food, or medical care.  Disgruntled patients have the upper hand, because Yelp doesn't require its users to make sound, rational arguments for their opinions. 

Online ratings are inherently unreliable because:

1. Review sites rarely verify that the reviewer is a real patient of the doctor
2. They do not prevent reviews by non-patients, most commonly by competitors or angry former staff
3. The review sites do not consistently generate enough ratings to provide an accurate picture of practice quality
4. Low numbers of reviews affect validity of conclusions

April 12, 2012 Addendum:
Fake reviews (review spam) has become such a problem that Google commissioned University of Illinois at Chicago researchers to investigate it. They concluded the key to identifying groups working organized review fraud is their behavior. Those include multiple reviews posted within days of each other, deviation from the norm where there are already a large number of reviews, posting of similar reviews in terms of content, and the same number of reviews over multiple products. These findings do not apply to most doctors because individual doctors just do not have that many total number of reviews on the web.

June 7, 2012 Addendum:
An Orlando Plastic Surgeon just prevailed in a lawsuit against a patient who posted negative reviews on RateMds.com and had relatives post additional negative reviews despite the fact that they had never been patients of the surgeon. That patient had apparently never voiced a complaint directly to the surgeon and their name was only released to the surgeon after a Jane Doe lawsuit against the review company. As part of a settlement that patient submitted another review rescinding the negative review and apologizing for the situation.

September 23, 2012 Addendum:
The publishing industry was rocked with news about authors creating fake accounts on Amazon and other sites to pad their product pages with phony positive reviews. In another case the owner of a pizza restaurant who was photographed bear hugging President Obama, had his Yelp profile take the brunt of users upset by his encounter, who took to the page to post fake negative reviews to ruin his reputation on the site. Gartner, the world's leading information technology research and advisory company, predicts that by 2014, somewhere between 10 and 15 percent of all online reviews will be paid for by companies seeking to build bigger follower bases, generate video views, get Facebook Likes and, of course, display more positive online reviews. And they’re willing to solicit these things by offering coupons or special promotions or, in some cases, just straight cash.

December 11,2012 Addendum:

Note in the beginning she states she went on line and picked a doctor with great reviews. She then let her guard down and let a Gynecologist perform a hairgrafting procedure.

May 25, 2013 Addendum:
Negative posts placed in response to positive posts in an Australian online doctor review website turned out to be posts made by the same man under 7 different forum user names, most of which implied he was female.  Some of the negative posts included statements that the previous positive posts were fabricated by the surgeon's office staff. Over a 5 month period following the posts the number of patients booking surgery fell to 12% of those seen in consultation. When the surgeon proceeded to prosecute the poster for defamation of character it turns out the surgeon had never met the man who made the negative posts. It was alleged that the man was hired by a competing surgeon to defame his character and was never a patient of this surgeon. It got so far as the court ordering a search warrant of the poster's home to look for emails related to the surgeon and the forum, including private, sent and deleted messages.

July 7, 2013 Addendum:
A study by marketing professors Duncan Simester of the Massachusetts Institute of Technology and Eric Anderson of Northwestern University of online reviews posted on the website of a major private-label apparel company revealed that about 5 percent of the product reviews were written by customers with no record of actually buying the item. Furthermore, those reviews were "significantly more negative" than the other 95% of reviews. These 5% bad reviews also resulted in significantly less demand for an item for at least 12 months i.e. these unreliable negative reviews drove the purchasing decisions of other prospective customers and reduced sales. These researchers were able to replicate the effect using book reviews on Amazon.com. It is unclear why someone who never purchased an item would give it a negative review. It is unclear if this behavior is to presumably protect others, give feedback to a company about its products or raise the fake reviewers online social status.

September 26, 2013 Addendum:
 3 days ago The office of New York’s Attorney General announced that it fined businesses over $350,000 for generating phony reviews online charging them with false advertising. This highlights the rampant fraud present in the online review systems. Even if it is not the majority of reviews it is sufficient in quantity to tip the scale in terms of consumers making purchasing decisions. A 2011 Cornell university study concluded that 85% of the most prolific reviewers on Amazon— the site’s “most trusted” reviewers — were given free products from publishers, agents and manufacturers so their reviews were biased. A Harvard business school study in 2011 found companies were gaming their online reviews by paying freelance writers in the Phillippines, Bangladesh and Eastern Europe $1 to $10 for each fake review so that 5% of reviews on Yelp in 2006 were fake and that a one star rating hike on Yelp increased restaurant revenue 5 to 9%. A just released study from Boston University found that 16% of 316,415 restaurant reviews on Yelp were fake this year and this year is not over yet. It is clear that the number of fake online reviews is increasing in proportion to the number of people that use them. Despite these facts 79% of people surveyed by e-commerce company BrightLocal in 2013 say they trust online reviews as much as personal recommendations from family and friends, up from 72% last year.

January 9, 2014 Addendum:
A Virginia Court of Appeals today upheld a lower court's decision that Yelp must provide the names of seven reviewers that negatively reviewed Hadeed Carpet. The decision was made despite friend of the court briefs against doing so filed by the Washington Post and Gannett Col Inc. and others. The complainant attached sufficient evidence indicating that a thorough review of its customer database revealed that the 7 anonymous defendants' reviews did not match its customer database. The court ruled that because of the doubt of the reviewers being actual customers the reviews were defamatory and not entitled to first amendment protections.

January 28, 2014 Addendum:

1. Time-related concerns are the #1 complaint in plastic surgery patient reviews. Most bad reviews are complaints about waiting times, communication between patient and doctor and money instead of the results of surgery. Most high 5 star reviews on the other hand focus on comfort with the doctor and results of surgery.
2. After restaurants, consumers read reviews of doctors and dentists more than any other category. Nielsen just reported that 68% of consumers trust online reviews and 70% take action after reading these reviews despite everything I have written above. Should such reviews really be more important than TV, websites, newspaper ads, email marketing, print ads, and billboards? 
3. Consumers require negative reviews to believe positive reviews. A number of studies have determined that consumers require a small amount of negative feedback in order to confidently believe positive feedback.

January 31, 2014 Addendum:
I just found a negative review of myself on portals that gather reviews from different websites. It is dated July 8, 2012 but I have never seen it before and the sites that it is supposedly listed on are ezlocal, which has no listing for me when my name is searched, and menuism, which is a site devoted only to restaurant reviews and restaurant menus. A google search of the sentence was not able to pick up a reliable webpage source. The negative review was brought to my attention by a company that emailed me trying to sell me there services to monitor negative reviews and asked for a written response. How convenient for this company that the only way for me to respond would be through them since I cannot respond to ezlocal or menuism.

February 22, 2014 Addendum:
 A University of Michigan survey conducted in September 2012, and just released, asked more than 2,000 people of various ages and ethnicities what they thought of doctor-rating sites including Healthgrades.com, Vitals.com and RateMDs.com, etc. 65% of those surveyed were aware these existed vs. 87% who were aware of car rating sites and 81% who knew about restaurant rating sites. Of those 65% only 19 % said they had gone to one of those websites during the past year and only 17% had done so more than once. Less than half of those who visited those websites said the reviews and ratings on the websites were "somewhat important" when choosing a doctor. A smaller portion said they were "very important". It turns out that the most important factor when picking a doctor was whether the doctor accepted a participant's insurance.

April 1, 2014 Addendum:
A recent Pew Research Center study indicates that 72% of all U.S. internet users looked online for health information in 2012; 30% of them have looked specifically at provider reviews, where anyone can write anything under a near guarantee of anonymity. More than 700,000 physicians are listed on Vitals.com, the largest of the patient review sites, which attracts more than 13 million visitors a month.

A Texas general surgeon was approached by one of his patients and asked why his website was so negative. The surgeon investigated this and found out that a disgruntled patient had created a parallel website designed to look like the doctor's website but filled with negative testimonials.  The doctor and his staff "strongly believe that the best physician-patient relationships are based on mutual respect and trust. That is, until we perform your surgery, after which we just dump you in the streets like a black baby in a dumpster.” “My husband went into surgery to have his appendix removed. … he died two days later.” The disgruntled patient obtained the URL a week after his 2012 surgery and began to populate it a year later because the doctor would not give him the pain medication he requested.  That disgruntled patient was arrested as a result of this. The problem with online reviews that are negative is you cannot tell the difference between an allegation that has some merit and an allegation that’s simply someone venting, who has an axe to grind.

Another doctor in New Jersey had given up seeing patients independently and began working for a company assisting surgeons in the operating room and managing a surgical facility. The doctor had not directly treated a patient in 3 years and decided to check their name on vitals.com only to find, “She is mentally unstable and has poor skills. Stay away!” Further investigation found the post came from a computer at New York University Langone Medical Center, a huge facility with thousands of people who have access to the computer but none of whom have been treated by this doctor. The only logical conclusion was that the review was placed by a disgruntled former medical school classmate.

September 10, 2014 Addendum:
Sadly Joan Rivers passed away last week as a result of a heart attack she sustained while undergoing a procedure in an outpatient endoscopy center. That center had few if any reviews on Yelp the day before her death but the number of reviews skyrocketed after her death. Those were of course negative reviews that are not reliable. The latest news is that another doctor brought in by her people did an un-consented biopsy in the vocal area that caused the heart attack. Saving an 81 year old in that situation can be a futile. Before all the facts are in the facility will likely lose business, whether deserved or not. Yelp just adds fuel to a flame where there originally was no fire.

April 29, 2015 Addendum:
Economics and Finance professors at Yale University hypothesized that smaller businesses such as owner operated hotels are more likely to post fake reviews online than large hotel companies. They compared hotel reviews on TripAdvisor that allows anyone to post a review with Expedia which only allows reviews from those who have booked a room through the site. They found that owner operated hotels have more positive reviews on TripAdvisor relative to Expedia than large company owned hotels and the competitors of owner operated hotels have more negative reviews i.e. the smaller owner operated hotels are more likely to post fake positive reviews of themselves and fake negative reviews of their nearby competitors.

August 19, 2015 Addendum:
In 2014 Iowa prosecutor Ben Smith began investigating RipoffReport.com poster Darren Meade who was paid more than $100,000 between 2011 and 2013 by Ripoff's parent company Xcentric and who was directed by Ripoff founder Ed Magedson to post specific reports some of which were written by Magedson. In September 2014 Meade was charged with witness tampering and obstruction of justice in the murder trial of the wife of a surgeon on whom Meade posted negative reports. Xcentric then sued Smith and after an ensuing legal circus the judge declared that Magdeson and Ripoff were more than a neutral conduit for the harassing content about witnessess and materially contributed to the alleged illegality of the negative reports.

Once a complaint goes up, true or false, it’s up for good: Ripoff has a policy of never removing reports entirely despite its parent company paying for and directing some of those reports. And that policy drives Ripoff’s fee-based services for businesses desperate to fix the damage done by having “Ripoff Report” associated with their name. Ripoff then contacts companies of via its webiste elicits submissions to on-site inspection and filling out of a questionnaire followed by a pledge to “make things right” with customers. Ripoff uses the information to create a positive review; crucially, reassuring language replaces negative headlines in search results, and any new complaint gets put on hold, giving the business a chance to resolve it first. (While Ripoff doesn’t post its rates, founder and editor Ed Magedson sent one potential customer the following in 2013: a base fee of $7,500 plus $600 per report up to 20 reports, plus a minimum of three years of monitoring at a minimum of $100 a month, or $3,600.)

For those with just one or two reports, Ripoff suggests its arbitration option, starting at $2,000. The website says it will hire an independent arbitrator to weigh evidence of factual inaccuracies, and will redact details found to be untrue. There’s even an option for companies that have never experienced a Ripoff complaint to get “verified,” ensuring that they will have a chance to resolve any gripes before anyone can post about them. Ripoff is using the right of free speech to exort money from these companies.

June 27, 2016 Addendum:
South Korea's Busan District Court handed down 10-month and 6-month jail terms suspended for two years to operators of online communities of prospective or past plastic surgery patients for receiving 600 million won from the heads of plastic surgery clinics in exchange for uploading fake testimonials of patients claiming they were happy and satisfied with procedures performed at the clinics. The heads of six plastic surgery clinics in Busan were also fined 3 million to 5 million won each for their participation in the scheme. The fabrication and posting of advertisements that would mislead consumers is banned under South Korean Medical Laws.

September 1, 2016 Addendum:
36 year old Carlos Hernandez Fernandez a Denver surgical assistant opened a cosmetic surgery clinic,  H-F Care Complete Beauty, in January 2015.

For a year and half he identified himself as doctor or MD and performed facelifts, rhinoplasties, breast augmentations, tummy tucks etc. strictly under local anesthesia. During patient visits, he made recommendations and prescribed treatments, administered medications and gave prescription medications. He targeted illegal aliens offering discount plastic surgery and they described procedures where they could feel the scalpel cutting because of inadequate pain control. After complaints of botched surgery and unlawful sexual conduct his clinic was shut done in August 2016. He is facing 10 felony charges including criminal impersonation, illegally practicing as a medical doctor, child abuse, second degree assault and unlawful sexual contact with a patient.

Fernandez violated Healthgrades policy and fraudulently updated his profile to list himself as a medical doctor. At the time of his arrest, the fake doctor had a five-star rating based on two user reviews. His Healthgrades.com profile was immediately disabled after his arrest. So much for online doctor ratings.

November 18, 32017 Addendum:
On an online board for physicians one physician described his on line reviews. One said he was the best doctor he every had because the doctor listened. The next said he was the worst because he wouldn't listen and only spent a few minutes with the patient. Another rated his physician assistant at 75% while the doctor did not employ a physician assistant. That doctor shares front office staff with other doctors yet these same employees were rated 10% lower by his patients than the patients of the other doctors. Another patient praised his care but only rated him 1 out of 5 on a scale of 5.  People have no idea what they are rating or how to submit a review.

Aaron Stone MD - twitter
Aaron Stone MD - Plastic Surgeon Los Angeles