Reading Studies | Funwithfacts

Science

06 - Reading Studies

Scientific studies can be extremely daunting to start reading when you have no idea how. Let's go over the common structure of a scientific study first.

Abstract

This section is a general summary of the paper. It might include a brief summary of the hypothesis, purpose, method, results and implications of the study. It might include some elements but not others. There doesn't seem to be a hard rule on what needs to be in it. It's just a taste of the study.

Some people only use the abstracts to prove their points which is problematic, as the intricacies of the study are ignored. Having said that, some studies are locked behind a pay wall so sometimes the abstract is all you have.

Introduction

In this section the author of the study will outline the purpose of the study and hypothesis that will be investigated. It's important to read the introduction as any presuppositions will be stated and supported with references. It will also be visible from the introduction whether the scientist has any bias or assumptions about the outcome of the paper.

Method

In this section the testing or data gathering method will be outlined. The author wil explain how they gathered the information they used in their paper. They will also note down any quality control measures they implemented, as well as any precautions they might have taken. There is a lot the method could include and they can be pretty long, but take it in your own time.

Results

In this section the data gathered will be collated and given. The data will be arranged into a form that it can be read properly such as graphs and tables. Certain terminology will be thrown around which might seem alien, but don't worry, I will explain that, too.

Conclusion

Here the results are then explained by the author. The results will be interpreted to show the implications of the data. It can be tempting to go right for the conclusion and leave the rest, but you miss the nuance in science that way. Important information can be dotted around so it's important to give every section a read if you can. Sometimes the conclusion is combined with the results.

Discussion

At the end of a study is a section called the discussion where the author will talk about the implications of their study. It will usually contain some information on the current state of the field or current understanding of the subject and the need for more research. There can be some really important information in the discussion. Authors can share information in the discussion which can change your perception as opposed to just looking at the abstract or the conclusion.

Funding

Any conflict of interest and funding is likely to be noted, too. While this is important and as I've discussed the potential for funder bias is a real issue, it's worth remembering that because something is funded by industry it doesn't make that study automatically false. I see this line of thinking employed by people trying to discredit "mainstream" scientific papers to push their interpretation of the science.

Let's take a look at an actual scientific study. Before we start it's worth noting that some studies require payment to be read in full so sometimes, all you have is the abstract.

We're going to go through and break down together this study from the British Medical Journal titled:
Intake of whole grain foods and risk of type 2 diabetes: results from three prospective cohort studies.

Before we click the link, this study looks at 3 prospective cohort studies. Referencing the terminology page:

Cohort Study - "A type of epidemiological study in which a particular outcome, such as a medical condition, is compared according to a putative factor (a factor suspected to influence the chances of acquiring the medical condition) in a group of individuals who are linked in some way (the cohort). In a prospective cohort study the group of individuals is followed over time in order to determine how the putative factor affects rates of the outcome of interest. In a retrospective cohort study, the data is collected from past records of the cohort."

Or to rephrase for this particular study:

This Cohort Study - A type of epidemiological study in type 2 diabetes, is compared according to intake of wholegrain foods in a group of individuals who are linked in some way (the cohort). In a prospective cohort study the group of individuals is followed over time in order to determine how the intake of wholegrain foods affects rates of type 2 diabetes

OK, we can click on the link now.

At the top we have a list of the scientists involved:

Yang Hu, postdoctoral fellow1,
Ming Ding, research fellow1,
Laura Sampson, senior research dietitian1,
Walter C Willett, professor1 2 3,
JoAnn E Manson, professor2 3 4,
Molin Wang, associate professor2 5,
Bernard Rosner, professor3 5,
Frank B Hu, associate professor1 2 3,
Qi Sun, professor1 3

The numbers by their name indicate the institutions they belong to:

1. Department of Nutrition, Harvard TH Chan School of Public Health, 665 Huntington Avenue, Boston, MA 02115, USA

2. Department of Epidemiology, Harvard TH Chan School of Public Health, Boston, MA, USA

3. Channing Division of Network Medicine, Department of Medicine, Brigham and Women’s Hospital and Harvard Medical School, Boston, MA, USA

4. Division of Preventive Medicine, Department of Medicine, Brigham and Women’s Hospital and Harvard Medical School, Boston, MA, USA

5. Department of Biostatistics, Harvard TH Chan School of Public Health, Boston, MA, USA

If you become acquainted with certain circles of truthers such as with the radiowave alarmist movement, you will come to recognise names that pop up over and over across various very biased studies. It's important to at least read who's involved.

Starting off the researcher outlined the purpose of the study and provides some background information on why this study is being performed. Also be mindful of the language. Language in a study should be neutral. If you detect anger, or bias, or mistrust for government, then something's wrong. Science isn't a product of emotional positions, it's off of evidence. As we should know by now, emotional arguments and bias cloud judgement.

"Whole grains have been widely recognized as healthy foods because of their high content of fiber, antioxidants, and phytochemicals.1"

That "1" at the end is a link to the reference paper. Good scientists don't assume, they reference other science. When looking at the introduction, if the statements made seem off, check the source they use and see the quality of that paper. Science isn't proppogated on cherry picking supporting papers, it's proppogated off of good studies. Occasionally you will find a study which doesn't add up and checking their sources (if they even had any) is a good way to check what's going on.

"Epidemiological studies have shown inverse associations between consumption of whole grains and the risk of developing several major chronic diseases, including type 2 diabetes"

Other studies have shown an inverse association, meaning the more wholegrains people ate the less type 2 diabetes they had among others. The language is also very neutral and the sources are numerous.

"To provide evidence to bridge this knowledge gap, in this study, we prospectively examined the associations between the consumption of several commonly eaten whole grain foods, such as whole grain cold breakfast cereal, oatmeal, dark bread, brown rice, popcorn, wheat germ, and added bran, and the risk of type 2 diabetes, in the Nurses’ Health Study, the Nurses’ Health Study II, and the Health Professionals Follow-up Study, three large, well characterized cohort studies with diet and other characteristics repeatedly assessed over three decades of follow-up."

So that's the introduction. Read it all, but those are the key points. This study is to further the knowledge around how wholegrain foods may help protect us against type 2 diabetes.

So let's look at the method.

"The Nurses’ Health Study cohort was established in 1976 when 121 700 registered nurses, all women, aged 30-55, completed a questionnaire on their medical history and lifestyle characteristics. The Nurses’ Health Study II was initiated in 1989 and included 116 340 eligible women aged 25-42. A questionnaire similar to that used in the Nurses’ Health Study was administered at baseline to assess medical history, lifestyle factors, and diet. The Health Professionals Follow-up Study began in 1986 when 51 529 American health professionals, all men, aged 40-75, answered a similar baseline questionnaire. In all three cohorts, similar follow-up questionnaires were sent to the participants to update the information and to identify newly diagnosed type 2 diabetes and other diseases every two years. The cumulative response rates in the three cohorts exceeded 90%"

So here we can see that:
Study 1 - 121 700 nurses
Study 2 - 116 340 nurses

Study 3 - 51 529 health professionals

Initiated with a health questionaire, following up every 2 years with a 90% response rate, meaning that 90% of participants remained throughout.

"We excluded participants diagnosed with type 2 diabetes, cardiovascular disease (including non-fatal myocardial infarction, fatal coronary heart disease, and fatal and non-fatal stroke), or cancer at baseline, those who did not return a semiquantitative food frequency questionnaire or had an unusual total energy intake at baseline (<500 or >3500 kcal/day for the Nurses’ Health Study and Nurses’ Health Study II, and <800 or >4200 kcal/day for the Health Professionals Follow-up Study; 1 kcal=4.18 kJ=0.00418 MJ), those with unconfirmed type 2 diabetes, and those who completed only the baseline questionnaire."

Those who had conditions at the start that were to be measured were excluded. Those who didn't return decent food questionaires were excluded, and those who under or over ate. Also those with unconfirmed type 2 diabetes were excluded, and those who didn't fill out the follow up questionaires.

"After these exclusions, 69 139 participants from the Nurses’ Health Study, 89 120 participants from the Nurses’ Health Study II, and 36 525 participants from the Health Professionals Follow-up Study were included in the final analysis."

So there we have the final participant numbers after exclusion. From the method and introduction we have seen:

-Why this study is being done
-Background on the current understanding of wholegrains on diabetes

-Which studies they are going to evaluate

-The quality control measures being excluding participants based on certain criteria

Just under we get clarrification on what they're determining to be whole grains:

"By definition, foods and ingredients considered whole grains were: whole wheat and whole wheat flour, whole oats and whole oat flour, whole cornmeal and whole corn flour, whole rye and whole rye flour, whole barley, bulgur, buckwheat, brown rice and brown rice flour, popcorn, amaranth, and psyllium."

Below is a further explanation of the quality control in one of the studies they're evaluating:

"In the Nurses’ Health Study, Nurses’ Health Study II, and Health Professionals Follow-up Study, we sent follow-up questionnaires every two years to collect and update the occurrence of diseases and many lifestyle and personal risk factors, including smoking status, use of vitamin supplements, alcohol consumption, menopausal status, years of use of postmenopausal hormones (Nurses’ Health Study and Nurses’ Health Study II only), body weight, physician diagnosed hypertension and hypercholesterolemia, and other variables."

"Baseline characteristics are expressed as mean (standard deviation) or median (interquartile range)"

We now know what standard deviation and interquartile range means from the previous page.

"To minimize potential bias resulting from a change in usual diet because of a diagnosis of a chronic disease or condition, we stopped updating dietary information when participants first reported having myocardial infarction, stroke, cancer, hypertension, or hypercholesterolemia."

This is important as it's a quality control measure to be aware of.

"For the primary analysis, a multivariable Cox proportional hazards model was used to calculate hazard ratios and 95% confidence intervals for the associations between individual whole grain foods and total whole grain intake, and the risk of type 2 diabetes."

For what the Cox hazard model is, here's a link to read.

"The proportional hazards assumption was evaluated by including an interaction term between whole grain intake and the duration of follow-up. No evidence of violations of the assumption was detected in each cohort (P>0.05 for all tests)."

From the previous page, we know roughly what the P values mean. Roughly it means the assumption is that the values have a 5% chance or less of being wrong.

"The statistical models were adjusted for age (month), ethnicity (white, African American, Asian, other), smoking status (never smoked, past smoker, or currently smoke 1-14 cigarettes/day, 15-24 cigarettes/day, or ≥25 cigarettes/day), alcohol intake (0, 0.1-4.9, 5.0-9.9, 10.0-14.9, 15.0-29.9, and ≥30.0 g/day), multivitamin use (yes, no), physical activity (divided into five equal groups), modified alternative healthy eating index (divided into five equal groups), total energy (divided into five equal groups), family history of diabetes (yes, no), postmenopausal hormone use (women only; never, former, or current hormone use, or missing), and oral contraceptive use (yes, no; women only). Because the time varying body mass index (<21.0, 21.0-22.9, 23.0-24.9, 25.0-26.9, 27.0-29.9, 30.0-32.9, 33.0-34.9, or ≥35.0) could be both confounder and mediator, we also adjusted for it in a separate model."

From this section we can see that the researchers adjusted for a wide range of variables to account for variation based on different factors, and very importantly, they adjusted for BMI in a separate model as it could be a cofounding variable.

"we used a Markov Chain Monte Carlo based method to impute missing data on total energy (3.1% missing), physical activity (5.6% missing), body mass index (2.7% missing), and smoking status (3.0% missing) before categorizing these variables."

For what the Markov Chain Monte Carlo method is, you can view this wikipedia article which says:

"MCMC methods are primarily used for calculating numerical approximations of multi-dimensional integrals, for example in Bayesian statistics, computational physics,[1] computational biology [2] and computational linguistics."

It's important to understand that they used a method of calculating missing variables before completing the analysis.

"we restricted the analyses to participants with symptomatic type 2 diabetes to look at potential detection bias."

Another bit of information about potential bias detection. This is what you find in good studies.

"A total of 18 629 participants with type 2 diabetes were identified and confirmed during 4 618 796 person years of follow-up. The average follow-up time was 24 years. In all three cohorts at baseline, participants with higher total whole grain consumption, on average, were more likely to be white participants, were slightly older, were less likely to be current smokers, were more likely to be leaner and multivitamin users, and were more physically active, compared with participants with a lower intake. They also tended to have a lower prevalence of hypertension and family history of diabetes, higher diet quality, and more frequent screening for fasting glucose"

The start just gives you a general idea of what was tested and what the potential factors could be, which is a sign of an unbiased study. You'll generally find the biased studies don't account for cofounding variables, don't employ robust quality control measures, or even discuss potential bias.

"After adjusting for body mass index and other lifestyle and dietary risk factors for diabetes, higher total whole grain consumption was consistently associated with a lower risk of type 2 diabetes in all three cohorts (table 2). In pooled results, comparing the extremes of the five equal categories for total whole grain intake, a 29% (hazard ratio 0.71, 95% confidence interval 0.67 to 0.74, P<0.001 for trend) lower rate of type 2 diabetes was found."

So from this, we can see they found that all three studies consistently showed a decrease in the risk of type II diabetes from whole grain food intake. We also understand that with less than a 1% chance of being wrong, they found that with 95% confidence that the rate of diabetes was 29% lower.

"The goodness of fit of the fully adjusted model was significantly improved by also adjusting for individual whole grain foods (Nurses’ Health Study, P<0.001; Nurses’ Health Study II, P<0.001; Health Professionals Follow-up Study, P<0.001) suggesting potentially heterogeneous associations in different individual whole grain foods with diabetes risk. To test if the significant heterogeneity was driven by the positive association for popcorn, we repeated the likelihood ratio test by removing popcorn in the model and found similar results."

Interestingly, they found that certain whole grain foods were more prone to reducing risk than others. They tested this idea by removing popcorn from the model and found a similar result.

"The rate reduction slightly plateaued at consumption of more than two servings a day of total whole grains."

"For individual whole grain foods, non-linear associations were seen for consumption of whole grain breakfast cereal and dark bread, and the risk of diabetes, where the rate reduction plateaued at about 0.5 servings a day"

"In the stratified analysis that adjusted for the same covariates in table 2, the inverse associations between total whole grain intake and the risk of type 2 diabetes seemed to be stronger in participants who were lean or overweight compared with participants who were obese (P=0.003 for interaction) whereas we found no significant effect modification for smoking status, physical activity, or family history of diabetes"

Inverse meaning inverted. A normal association would mean as variable 1 increases, so does variable 2. Inverse association means as variable 1 increases, variable 2 decreases. Obesity saw a weaker inverse association between whole grain intake and decreased risk of type II diabetes. No significant effect is an important distinction, as well. It doesn't mean no effect, it means no significant effect.

Here are some graphs of their results.

"Findings from three prospective cohorts showed that higher total whole grain intake was significantly associated with a lower risk of type 2 diabetes. Although these inverse associations were seen for most individual whole grain foods, we also found an elevated increased rate of type 2 diabetes associated with consumption of one of more servings of popcorn a day. The association for individual whole grain foods seemed to be independent of other whole grain foods, except for wheat germ. Dose-response analyses showed that the rate reduction plateaued at high intake levels for total whole grains, and whole grain cold breakfast cereal and dark bread, whereas a J shaped association was found for popcorn intake where the rate of diabetes did not increase until intake exceeded about one serving of popcorn a day."

Firstly they outline what they found. Next, they go on to discuss this study in context to the available science.

"A meta-analysis of prospective cohort studies showed that total whole grain consumption was associated with a lower risk of type 2 diabetes.43"

"Moreover, this meta-analysis showed that a higher intake of several whole grain foods, including whole grain bread, whole grain breakfast cereals, wheat bran, and brown rice, were associated with a similar risk reduction in type 2 diabetes."

Reference 43 is a study called Whole grain and refined grain consumption and the risk of type 2 diabetes: a systematic review and dose–response meta-analysis of cohort studies from the European Journal of Epidemiology.

"Because individual whole grain foods contain various amounts of dietary fiber, magnesium, antioxidants, and phytochemicals, they might have distinct associations with the risk of type 2 diabetes.44"

Reference 44 is Fiber and Magnesium Intake and Incidence of Type 2 Diabetes. A Prospective Study and Meta-analysis from the JAMA journal.

"The statistically significant P values from the goodness of fit tests across the three cohorts, even after removing popcorn, suggested potential heterogeneous effects in individual whole grain foods with the risk of type 2 diabetes. Nevertheless, this finding should be interpreted with caution because of similar effect estimates for most individual whole grain foods. The highly statistically significant P values could be because of the abundant statistical power in the three cohorts."

You often find this in unbiased and decently conducted studies. They demonstrate their results, but also explain elements which should be taken with caution because of possible cofounding variables.

"In our analysis, we saw a non-linear dose-response association between total whole grain intake and diabetes. This observation is consistent with previous studies collectively suggesting that the rate reduction of diabetes might plateau at two to three servings a day."

It's important to recognise when good studies find the same thing because it increases the likeliness that what they found is correct.

"An interesting finding in the subgroup analysis was a relatively weaker inverse association between total whole grain intake and the risk of type 2 diabetes in participants who were obese compared with participants who were lean or overweight. Participants who were obese might have had a high risk profile characterized by chronic inflammation, dyslipidemia, hypertension, and insulin resistance, which could partially offset the beneficial effects of whole grain intake on glucose metabolism."

"This disproportional higher consumption of popcorns in participants who were obese might also contribute to the weaker association of whole grain intake with the risk of type 2 diabetes. Lastly, given a borderline significant P value and potential multiple testing issues in the stratified analysis, we cannot rule out the possibility of a chance finding for this effect modification by body mass index."

The researchers also discuss the findings that obese individuals showed less reduction in the risk of type II diabetes based on whole grain intake.

Strengths and weaknesses

They then discuss the strengths and weaknesses of the study, which again, is a sign of an unbiased and well-rounded study.

"The strengths of our study include the use of data from three large cohort studies with long term follow-ups, comprehensive, repeated assessments of diet and potential confounders, and high follow-up rates. Also, along with total whole grains, our study included seven commonly consumed individual whole grain foods and assessed their associations with the risk of type 2 diabetes. Moreover, we conducted a series of sensitivity analyses to show the robustness of the findings."

"Potential limitations warrant consideration. First, although we adjusted for many lifestyle practices and diet quality, residual or unmeasured confounding cannot be excluded in observational studies. Second, multiple comparisons could result in false positive results because we examined the associations for seven whole grain foods simultaneously. Our main results remained statistically significant for most whole grain foods, however, even after adjusting for the multiplicity with the conservative Bonferroni correction, and the consistent associations across three cohorts made the chance findings less likely. Lastly, our observations might largely relate to white health professionals and lack generalizability to other populations with different characteristics."

"In conclusion, higher consumption of total whole grains and the most commonly consumed whole grain foods was significantly associated with a lower risk of type 2 diabetes. These findings provide further support for the current recommendations that promote increased consumption of whole grain as part of a healthy diet for the prevention of type 2 diabetes."

And finally, the funding

"Funding: This study was funded by research grants CA186107, CA176726, CA167552, HL034594, HL035464, DK112940, and DK120870 from the National Institutes of Health. The sponsors had no role in the study design; the collection, analysis, or interpretation of data; the writing of the report; or in the decision to submit the article for publication. QS had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis."

Or is it? This is one study of many. On any one subject, there exists a multitude of studies. Some are fantastic as this one, some are biased works that somehow got through peer-review. The science will be cherry-picked by people wanting to prove a point, misrepresented in the media, ignored by the public, and denied by those with a conflict of interest.

There might be overwhelming scientific consensus among experts such as with climate change, yet there are people who deny the consensus and grasp at anything anyone says as a way to discredit it.

It might be a mixed bag, with many biased papers utilising close to no quality control with the harm from radiowave scientific literature, that gets latched on to by those trying to sell their emf shielding and meters.

When I hear that an institution like the WHO who's job is to understand the ever evolving science and recommend the world's population what to do have read thousands of studies, I have an easier time believing that than when a small group of 1-5 people say they've read the same amount.

As we've seen with papers, they have a lot of information and to understand which study is good and which is biased, you have to read all of it. I find it highly unlikely that a few people can thoroughly read thousands of studies. It might be possible, but to me it seems like they're grabbing abstracts and conclusions without acknowledging the introduction, method, results, or discussion.

Off on a tangent, I know. The point is that science is not simply just the sum of all papers. It's a constantly evolving understanding which involves filtering out the biased works and the badly composed papers which cloud the waters. It involves listening to the right institutions and being sceptical of anyone claiming the consensus is wrong.

If an institution like the WHO, the CDC, various safety agencies, regulatory bodies, and the consensus of experts in their respective fields say something collectively, you listen. They understand the nuance, complexity, and know their context to understand the science. They know how to spot and filter out the bad papers, and can fully comprehend the implications of the sum of the papers in their field.

With this in mind, when someone outside of that field or those institutions claims to know the "truth" I'm extremely sceptical. I have a hard time believing that the people who have dedicated their lives to any subject are all wrong, despite having the highest possible knowledge of that subject. Anti-expert rhetoric is ruining this world and is voiced proudly by those basking in their own ignorance.

Don't be one of those people. Either they don't know the magnitude of the knowledge they're missing, or they have a conflict of interest. Either way, don't deny expert advice. Additionally, get your expert advice from the experts, not from the people claiming to know what the experts are saying. Don't get your news on the WHO from a news site, go to the WHO website. Don't get your space or infectious disease news from a mainstream media outlet, go to a scientific journal or follow science advocate social pages.

That's it for now. Next we move onto the critical thinking section. These following pages might have slight occurences in previous pages, but this next section is the most comprehensive explanation and analysis I have of the subject.

Next page: Critical Thinking - What is Scepticism?