The Importance of Evidence

A place to get your questions answered from McDougall staff dietitian, Jeff Novick, MS, RDN.

Moderators: JeffN, carolve, Heather McDougall

Re: The Importance of Evidence

Postby JeffN » Sun Dec 20, 2015 6:26 pm

An editorial in the New England Journal of Medicine has called attention to the problem of peer-review fraud, wherein people chosen to review manuscripts are associated with the author or are the author using a fake identify.

Haug CJ. Peer-review fraud: Hacking the scientific publication process. New England Journal of Medicine 373:2393-2395, 2015

http://www.nejm.org/doi/pdf/10.1056/NEJMp1512330

The editorial writer notes:

- Within the past few years, more than 250 articles that underwent fake peer-review have been retracted.

- Most of the articles originated from China and Southeast Asia.

- Pressure to publish, quickly and preferably in the very best journals, can influence both authors and editors to game the system.

- Verifying the identity of peer-reviewers should reduce the problem, but new ways of gaming the traditional publication models will be invented more quickly than new control measures can be put in place.
User avatar
JeffN
 
Posts: 9412
Joined: Tue Jan 08, 2008 5:56 am

Re: The Importance of Evidence

Postby JeffN » Wed Jan 13, 2016 6:32 am

Differences in Meta-Analyses’ Conclusions Depend on Methods:


http://www.bu.edu/sph/2016/01/06/differ ... n-methods/

Meta-analysis—a research method that aggregates multiple studies on a given topic—is a powerful tool in public health. But the way such analyses are conducted can influence their conclusions, a team of School of Public Health researchers has found.
In an article published in PLOS ONE, researchers led by Christopher Gill, associate professor of global health, found that meta-analyses conducted via the Cochrane Collaboration, which follow a standardized set of methods, produced different results than non-Cochrane meta-analyses on the same topics. In addition, the overlap among studies used in the two kinds of reviews was “surprisingly low,” raising questions about whether the two methods had different criteria for inclusion.

“Our results indicate a substantial divide between the Cochrane and non-Cochrane literature,” Gill said. “While our analysis covers a very small fraction of the vast body of work that comprises the meta-analytic literature, readers should be aware that the two types of meta-analyses are not synonymous—and that in some cases, the discrepancies could lead to fundamentally different conclusions about whether a given intervention is effective or not.”

The study found that non-Cochrane reviews reported significantly higher effect sizes of interventions, with lower precision—a systemic difference that indicates that such reviews “may be overstating the evidence,” Gill said. Meta-analyses conducted via the Cochrane Collaboration adhere to strict methodological and reporting standards that aim to minimize bias, maximize transparency, and improve the accuracy of summarized data, the authors said.

Gill and colleagues conducted a “matched-pair” analysis of 40 meta-analyses related to cardiovascular health, identifying pairs that addressed the same interventions and outcomes. The two sets were similar in terms of publication dates, how many studies were included, and average sample size. But of the 344 studies included in the paired reviews, only 129 (37.5 percent) were included in both kinds of analyses—and only 2 of the 40 matched pairs included exactly the same studies.

Overall, 37.5 percent of the pairs had discrepant results—a finding that the authors called troubling.

“A number of reviews reported markedly different effect sizes, or reported effect sizes that contradicted each other (as from a protective effect to a null or harmful one, or vice versa). These are all instances where the bottom line interpretation by a reader of a review could differ qualitatively,” they wrote.

Gill said that, because meta-analyses are important tools in clinical research, there is a need for such analyses conducted outside of the rigorous Cochrane Collaboration. Still, “it is concerning when two meta-analyses addressing the same question, within a similar time frame, reach different conclusions,” he said. “How should the average doctor or health policy maker react when two ‘gold standards’ disagree with each other? That is the question we have to think about.”

Co-authors on the study included Michael LaValley, professor of biostatistics; doctoral students Alana Brennan and Omid Ameli; and recent MPH graduates Johanna Useem, Michelle Vickery, and Nichole Reinen.
User avatar
JeffN
 
Posts: 9412
Joined: Tue Jan 08, 2008 5:56 am

Re: The Importance of Evidence

Postby JeffN » Mon May 09, 2016 8:25 am

John Oliver discusses how and why media outlets so often report untrue or incomplete information as science.

https://youtu.be/0Rnq1NpHdmw

Enjoy!!

In Health
Jeff
User avatar
JeffN
 
Posts: 9412
Joined: Tue Jan 08, 2008 5:56 am

Re: The Importance of Evidence

Postby JeffN » Sat Jun 11, 2016 10:05 am

A great read

In Health
Jeff Novick

Ten Simple Rules for Effective Statistical Practice

http://journals.plos.org/ploscompbiol/a ... bi.1004961

Introduction

Rule 1: Statistical Methods Should Enable Data to Answer Scientific Questions
Rule 2: Signals Always Come with Noise
Rule 3: Plan Ahead, Really Ahead
Rule 4: Worry about Data Quality
Rule 5: Statistical Analysis Is More Than a Set of Computations
Rule 6: Keep it Simple
Rule 7: Provide Assessments of Variability
Rule 8: Check Your Assumptions
Rule 9: When Possible, Replicate!
Rule 10: Make Your Analysis Reproducible
Conclusion
Acknowledgments


Kass RE, Caffo BS, Davidian M, Meng XL, Yu B, et al. (2016) Ten Simple Rules for Effective Statistical Practice. PLoS Comput Biol 12(6): e1004961. doi: 10.1371/journal.pcbi.1004961
http://journals.plos.org/ploscompbiol/a ... bi.1004961
User avatar
JeffN
 
Posts: 9412
Joined: Tue Jan 08, 2008 5:56 am

Re: The Importance of Evidence

Postby JeffN » Mon Jun 20, 2016 8:06 pm

Great article. Only problem is, everyone thinks they have (or know) the real science & everyone else is the pseudoscience

In Health
Jeff



THE MISTRUST OF SCIENCE
By Atul Gawande , JUNE 10, 2016

"The mistake, then, is to believe that the educational credentials you get today give you any special authority on truth. What you have gained is far more important: an understanding of what real truth-seeking looks like. It is the effort not of a single person but of a group of people—the bigger the better—pursuing ideas with curiosity, inquisitiveness, openness, and discipline. As scientists, in other words.

Even more than what you think, how you think matters. The stakes for understanding this could not be higher than they are today, because we are not just battling for what it means to be scientists. We are battling for what it means to be citizens."


http://www.newyorker.com/news/news-desk ... of-science
User avatar
JeffN
 
Posts: 9412
Joined: Tue Jan 08, 2008 5:56 am

Re: The Importance of Evidence

Postby JeffN » Sat Jul 23, 2016 7:12 am

Not only is understanding science and all the breaking health news, and how to put it in proper perspective, important, so is the skill in how we communicate it.

Whenever I am invited to give a talk, there are a few basics pieces of information I need in order to make sure my talk will be as effective as possible. One of these pieces of info is, who is my audience? And, the more details I know (age, gender, health, education, perspective, etc), the better. Not only it is very important to know my topic inside and out, it is also very important to know as much as I can about the audience.

As you all know, I am not one to argue the topics covered in my forum, nor do I encourage it. I share Dr Lisle's "seems" strategy. However, I am all for sharing this information when the situation is right and appropriate.

I spent many years in Sales, with lots of success. One of the things I enjoyed during those years, was all the education and training I received in personal communication, listening, persuasion, selling, etc. It helped me not only professionally but also personally. I also realized that many of the skills I learned reminded me of the same skills I learned when I studied debate and also reason and logic. All of them have helped me to do what I do, better.

One of my favorite shows is Science Friday. Yesterday, they had a discussion on the political discourse that is going on in this country right now. Several of the guests, discussed strategies on how to be more effective in conveying your point to "the other side" with some examples of how they have recently done that. It reminded me so much on why knowing your audience is so important.

The current political debate has so much in common with the vegan/diet debates and while listening, I thought how helpful this would be to those who are out there trying to discuss this with friends, families and loved ones (of course at the appropriate time).

Science Friday - Life In The Wrong Political Bubble.

"Stanford sociology professor Robb Willer shares his findings about successfully persuading “the other side” on even divisive partisan issues. And Liz Joyner, director of The Village Square in Tallahassee, Florida, talks about the evidence-based work her group has done to create community across political divides."

http://www.sciencefriday.com/segments/l ... al-bubble/

In Health
Jeff
User avatar
JeffN
 
Posts: 9412
Joined: Tue Jan 08, 2008 5:56 am

Re: The Importance of Evidence

Postby JeffN » Mon Jul 25, 2016 2:43 pm

The 7 biggest problems facing science, according to 270 scientists
Vox

by Julia Belluz, Brad Plumer, and Brian Resnick on July 14, 2016

http://www.vox.com/2016/7/14/12016710/s ... ew-process

"Science, I had come to learn, is as political, competitive, and fierce a career as you can find, full of the temptation to find easy paths." — Paul Kalanithi, neurosurgeon and writer (1977–2015)

Science is in big trouble. Or so we’re told.

Explore the biggest challenges facing science, and how we can fix them:

- Academia has a huge money problem
- Too many studies are poorly designed
- Replicating results is crucial — and rare
- Peer review is broken
- Too much science is locked behind paywalls
- Science is poorly communicated
- Life as a young academic is incredibly stressful

Conclusion:

- Science is not doomed


In Health
Jeff
User avatar
JeffN
 
Posts: 9412
Joined: Tue Jan 08, 2008 5:56 am

Re: The Importance of Evidence

Postby JeffN » Fri Aug 19, 2016 8:59 am

How social media can distort and misinform when communicating science
The Conversation
June 30, 2016

https://theconversation.com/how-social- ... ence-59044

"Even many traditional news organizations and media outlets report incomplete aspects of scientific studies, or misinterpret the findings and highlight unusual claims. Once these items enter into the social media echo chamber, they’re amplified. The facts become lost in the shuffle of competing information, limited attention or both."

"In a nationwide survey, we found that the more frequently respondents reported posting and sharing any information online to social media, they were increasingly likely to be highly misinformed about AMR. This suggests that those individuals most active in contributing to social media were actually propagating inaccurate information."
User avatar
JeffN
 
Posts: 9412
Joined: Tue Jan 08, 2008 5:56 am

Re: The Importance of Evidence

Postby JeffN » Tue Aug 23, 2016 7:09 am

NPR & Quad-City Times join Popular Science, CNN, Reuters and the Chicago Sun-Times.in shutting down comments

Pulling the plug on online comments
Rem Rieder
USA TODAY

http://usat.ly/2bgzrlS

"Since we made the announcement, I’ve received an outpouring of gratitude from our readers," she says. "I’ve heard from parents whose children were bullied in our online comments. I’ve heard from people who said they wouldn’t send in letters to the editor because they were attacked so harshly by commenters, and it wasn’t worth it."

"Comments sections too often were taken over by a small, nasty slice of the audience. Rather than a place for exchanging ideas, they became havens for ugly name-calling, for intimidation, for racism and misogyny."

"The digital world has brought with it many wonderful innovations. But comments sections became vivid sections of the extremely dark side of the Web."
User avatar
JeffN
 
Posts: 9412
Joined: Tue Jan 08, 2008 5:56 am

Re: The Importance of Evidence

Postby JeffN » Tue Aug 30, 2016 4:57 am

Nutrition, Politics, and the Destruction of Scientific Integrity
By T. Colin Campbell, PhD
August 16, 2016

http://nutritionstudies.org/nutrition-p ... integrity/

"This brings me to my experiences during the past two decades, mostly outside of academia where I find there to be little to no rules of scientific discourse, where far too many public figures (both those for and against the whole food, plant-based diet hypothesis) seem not to know what science really means. Too many of these would-be scientists have almost no regard for the concept of being held accountable for what they say to the public.

This therefore presents a huge problem. Instead of legitimate scientific findings being told to the public by qualified people who are held accountable for their views, we are left with a massive din of far too much noise that denies the public an honest airing of views. We must find ways to change this scenery. The exceptionally beneficial evidence on health that we now have must be told with defensible facts, not be distorted for personal gain. We have huge problems and we have evidence that can go a long way to help resolve these problems."
User avatar
JeffN
 
Posts: 9412
Joined: Tue Jan 08, 2008 5:56 am

Re: The Importance of Evidence

Postby JeffN » Tue Sep 20, 2016 8:19 am

What exactly is the scientific method and why do so many people get it wrong?
September 14, 2016
The Conversation

Peter Ellerton
Lecturer in Critical Thinking,
The University of Queensland


http://theconversation.com/what-exactly ... rong-65117
User avatar
JeffN
 
Posts: 9412
Joined: Tue Jan 08, 2008 5:56 am

Re: The Importance of Evidence

Postby JeffN » Thu Nov 03, 2016 8:37 pm

How the Internet Is Loosening Our Grip on the Truth
NYTimes
11/3/2016

A wider variety of news sources was supposed to be the bulwark of a rational age. Instead, we are roiled by biases, gorging on what confirms our ideas and shunning what does not.

http://www.nytimes.com/2016/11/03/techn ... truth.html
User avatar
JeffN
 
Posts: 9412
Joined: Tue Jan 08, 2008 5:56 am

Re: The Importance of Evidence

Postby JeffN » Mon Nov 14, 2016 3:09 pm

This article does a decent job in describing the influence of Social Media on our current state on "misinformation." While it focuses on the current politics we could easily substitute nutrition, diet and/or health and it would be just as accurate.

In my lecture, Lighten Up, I use the same analogy of how the way we receive news and information and how social media influences it, has contributed to our current state of nutrition, diet and health misinformation.


How We Broke Democracy
Our technology has changed this election, and is now undermining our ability to empathize with the world

https://medium.com/@tobiasrose/empathy- ... .9v7ctnxae

In health
Jeff
User avatar
JeffN
 
Posts: 9412
Joined: Tue Jan 08, 2008 5:56 am

Re: The Importance of Evidence

Postby JeffN » Sat Mar 04, 2017 10:51 am

JeffN wrote:I started making a similar point about 2.5 years ago as part of the intro to some of my talks and have continued to do so on many occasions.

I showed a few slides on "Who Do You Believe" and discuss the concepts of "Burden of Proof" and the accepted "Standards (Hierarchy) of Evidence."

I also discuss the 3 different categories that most health information falls under, which are similar to the 4 he describes above (evidence, tradition, revelation and authority).

My three were

1) Science & Evidence Based

2) Advertising/Marketing

3) Guru/philosophical/religious.

The main point was the same.

Here is the slide on the Standards of Evidence

Image

In Health
Jeff



Oh well,,,,

Fake news wins

"I just know" replaces systematic reviews at top of evidence pyramid
The Science Post
3/4/17

"The Society for Healthcare Epidemiology of America (SHEA) announced today that “I just know” will replace systematic reviews as the top level of evidence available in medical research. For years scientists and doctors have dismissed anecdotes from the likes of anti-vaxxers and pseudoscience pushers, but it appears they are finally ready to listen.

http://thesciencepost.com/i-just-know-r ... e-pyramid/

Image
User avatar
JeffN
 
Posts: 9412
Joined: Tue Jan 08, 2008 5:56 am

Re: The Importance of Evidence

Postby JeffN » Mon May 29, 2017 7:15 am

We can even teach kids how to detect BS in health research.

The study

Effects of the Informed Health Choices primary school intervention on the ability of children in Uganda to assess the reliability of claims about treatment effects: a cluster-randomised controlled trial
Published: 21 May 2017
http://dx.doi.org/10.1016/S0140-6736(17)31226-6

Background
Claims about what improves or harms our health are ubiquitous. People need to be able to assess the reliability of these claims. We aimed to evaluate an intervention designed to teach primary school children to assess claims about the effects of treatments (ie, any action intended to maintain or improve health).

Methods
In this cluster-randomised controlled trial, we included primary schools in the central region of Uganda that taught year-5 children (aged 10–12 years). We excluded international schools, special needs schools for children with auditory and visual impairments, schools that had participated in user-testing and piloting of the resources, infant and nursery schools, adult education schools, and schools that were difficult for us to access in terms of travel time. We randomly allocated a representative sample of eligible schools to either an intervention or control group. Intervention schools received the Informed Health Choices primary school resources (textbooks, exercise books, and a teachers' guide). Teachers attended a 2 day introductory workshop and gave nine 80 min lessons during one school term. The lessons addressed 12 concepts essential to assessing claims about treatment effects and making informed health choices. We did not intervene in the control schools. The primary outcome, measured at the end of the school term, was the mean score on a test with two multiple-choice questions for each of the 12 concepts and the proportion of children with passing scores on the same test. This trial is registered with the Pan African Clinical Trial Registry, number PACTR201606001679337.

Findings
Between April 11, 2016, and June 8, 2016, 2960 schools were assessed for eligibility; 2029 were eligible, and a random sample of 170 were invited to recruitment meetings. After recruitment meetings, 120 eligible schools consented and were randomly assigned to either the intervention group (n=60, 76 teachers and 6383 children) or control group (n=60, 67 teachers and 4430 children). The mean score in the multiple-choice test for the intervention schools was 62·4% (SD 18·8) compared with 43·1% (15·2) for the control schools (adjusted mean difference 20·0%, 95% CI 17·3–22·7; p<0·00001). In the intervention schools, 3967 (69%) of 5753 children achieved a predetermined passing score (≥13 of 24 correct answers) compared with 1186 (27%) of 4430 children in the control schools (adjusted difference 50%, 95% CI 44–55). The intervention was effective for children with different levels of reading skills, but was more effective for children with better reading skills.

Interpretation
The use of the Informed Health Choices primary school learning resources, after an introductory workshop for the teachers, led to a large improvement in the ability of children to assess claims about the effects of treatments. The results show that it is possible to teach primary school children to think critically in schools with large student to teacher ratios and few resources. Future studies should address how to scale up use of the resources, long-term effects, including effects on actual health choices, transferability to other countries, and how to build on this programme with additional primary and secondary school learning resources.


The Resources

Website

Testing Treatments interactive
Promoting critical thinking about treatment claims.

http://www.testingtreatments.org/?nabm=0


PDF book

Testing Treatments: Better Research for Better Healthcare
http://www.testingtreatments.org/wp-con ... ct2011.pdf
User avatar
JeffN
 
Posts: 9412
Joined: Tue Jan 08, 2008 5:56 am

PreviousNext

Return to Jeff Novick, RD

Who is online

Users browsing this forum: No registered users and 10 guests



Welcome!

Sign up to receive our regular articles, recipes, and news about upcoming events.