non significant results discussion example
They might panic and start furiously looking for ways to fix their study. More specifically, if all results are in fact true negatives then pY = .039, whereas if all true effects are = .1 then pY = .872. You didnt get significant results. The first row indicates the number of papers that report no nonsignificant results. We inspected this possible dependency with the intra-class correlation (ICC), where ICC = 1 indicates full dependency and ICC = 0 indicates full independence. significant effect on scores on the free recall test. As such, the problems of false positives, publication bias, and false negatives are intertwined and mutually reinforcing. DP = Developmental Psychology; FP = Frontiers in Psychology; JAP = Journal of Applied Psychology; JCCP = Journal of Consulting and Clinical Psychology; JEPG = Journal of Experimental Psychology: General; JPSP = Journal of Personality and Social Psychology; PLOS = Public Library of Science; PS = Psychological Science. In addition, in the example shown in the illustration the confidence intervals for both Study 1 and In NHST the hypothesis H0 is tested, where H0 most often regards the absence of an effect. Gender effects are particularly interesting because gender is typically a control variable and not the primary focus of studies. To put the power of the Fisher test into perspective, we can compare its power to reject the null based on one statistically nonsignificant result (k = 1) with the power of a regular t-test to reject the null. Lastly, you can make specific suggestions for things that future researchers can do differently to help shed more light on the topic. So how should the non-significant result be interpreted? You should probably mention at least one or two reasons from each category, and go into some detail on at least one reason you find particularly interesting. Specifically, the confidence interval for X is (XLB ; XUB), where XLB is the value of X for which pY is closest to .025 and XUB is the value of X for which pY is closest to .975. We examined evidence for false negatives in nonsignificant results in three different ways. I understand when you write a report where you write your hypotheses are supported, you can pull on the studies you mentioned in your introduction in your discussion section, which i do and have done in past courseworks, but i am at a loss for what to do over a piece of coursework where my hypotheses aren't supported, because my claims in my introduction are essentially me calling on past studies which are lending support to why i chose my hypotheses and in my analysis i find non significance, which is fine, i get that some studies won't be significant, my question is how do you go about writing the discussion section when it is going to basically contradict what you said in your introduction section?, do you just find studies that support non significance?, so essentially write a reverse of your intro, I get discussing findings, why you might have found them, problems with your study etc my only concern was the literature review part of the discussion because it goes against what i said in my introduction, Sorry if that was confusing, thanks everyone, The evidence did not support the hypothesis. It's hard for us to answer this question without specific information. For r-values, this only requires taking the square (i.e., r2). As such the general conclusions of this analysis should have non significant results discussion example. This overemphasis is substantiated by the finding that more than 90% of results in the psychological literature are statistically significant (Open Science Collaboration, 2015; Sterling, Rosenbaum, & Weinkam, 1995; Sterling, 1959) despite low statistical power due to small sample sizes (Cohen, 1962; Sedlmeier, & Gigerenzer, 1989; Marszalek, Barber, Kohlhart, & Holmes, 2011; Bakker, van Dijk, & Wicherts, 2012). it was on video gaming and aggression. Statements made in the text must be supported by the results contained in figures and tables. so sweet :') i honestly have no clue what im doing. Importantly, the problem of fitting statistically non-significant Additionally, the Positive Predictive Value (PPV; the number of statistically significant effects that are true; Ioannidis, 2005) has been a major point of discussion in recent years, whereas the Negative Predictive Value (NPV) has rarely been mentioned. When considering non-significant results, sample size is partic-ularly important for subgroup analyses, which have smaller num-bers than the overall study. The results of the supplementary analyses that build on the above Table 5 (Column 2) almost show similar results with the GMM approach with respect to gender and board size, which indicated a negative and significant relationship with VD ( 2 = 0.100, p < 0.001; 2 = 0.034, p < 0.000, respectively). More specifically, as sample size or true effect size increases, the probability distribution of one p-value becomes increasingly right-skewed. The effect of both these variables interacting together was found to be insignificant. If your p-value is over .10, you can say your results revealed a non-significant trend in the predicted direction. In most cases as a student, you'd write about how you are surprised not to find the effect, but that it may be due to xyz reasons or because there really is no effect. The debate about false positives is driven by the current overemphasis on statistical significance of research results (Giner-Sorolla, 2012). 178 valid results remained for analysis. The most serious mistake relevant to our paper is that many researchers accept the null-hypothesis and claim no effect in case of a statistically nonsignificant effect (about 60%, see Hoekstra, Finch, Kiers, & Johnson, 2016). P50 = 50th percentile (i.e., median). The simulation procedure was carried out for conditions in a three-factor design, where power of the Fisher test was simulated as a function of sample size N, effect size , and k test results. By mixingmemory on May 6, 2008. Describe how a non-significant result can increase confidence that the null hypothesis is false Discuss the problems of affirming a negative conclusion When a significance test results in a high probability value, it means that the data provide little or no evidence that the null hypothesis is false. Our dataset indicated that more nonsignificant results are reported throughout the years, strengthening the case for inspecting potential false negatives. Peter Dudek was one of the people who responded on Twitter: "If I chronicled all my negative results during my studies, the thesis would have been 20,000 pages instead of 200." This is reminiscent of the statistical versus clinical significance argument when authors try to wiggle out of a statistically non . Is psychology suffering from a replication crisis? Published on 21 March 2019 by Shona McCombes. For example do not report "The correlation between private self-consciousness and college adjustment was r = - .26, p < .01." In general, you should not use . Non-significance in statistics means that the null hypothesis cannot be rejected. Insignificant vs. Non-significant. For the set of observed results, the ICC for nonsignificant p-values was 0.001, indicating independence of p-values within a paper (the ICC of the log odds transformed p-values was similar, with ICC = 0.00175 after excluding p-values equal to 1 for computational reasons). Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology, Journal of consulting and clinical Psychology, Scientific utopia: II. To say it in logical terms: If A is true then --> B is true. Accessibility StatementFor more information contact us atinfo@libretexts.orgor check out our status page at https://status.libretexts.org. We simulated false negative p-values according to the following six steps (see Figure 7). Table 3 depicts the journals, the timeframe, and summaries of the results extracted. Clearly, the physical restraint and regulatory deficiency results are P75 = 75th percentile. Reducing the emphasis on binary decisions in individual studies and increasing the emphasis on the precision of a study might help reduce the problem of decision errors (Cumming, 2014). Contact Us Today! On the basis of their analyses they conclude that at least 90% of psychology experiments tested negligible true effects. The purpose of this analysis was to determine the relationship between social factors and crime rate. (or desired) result. In a purely binary decision mode, the small but significant study would result in the conclusion that there is an effect because it provided a statistically significant result, despite it containing much more uncertainty than the larger study about the underlying true effect size. You are not sure about . We first applied the Fisher test to the nonsignificant results, after transforming them to variables ranging from 0 to 1 using equations 1 and 2. Rest assured, your dissertation committee will not (or at least SHOULD not) refuse to pass you for having non-significant results. For significant results, applying the Fisher test to the p-values showed evidential value for a gender effect both when an effect was expected (2(22) = 358.904, p < .001) and when no expectation was presented at all (2(15) = 1094.911, p < .001). Andrew Robertson Garak, Teaching Statistics Using Baseball. Bond is, in fact, just barely better than chance at judging whether a martini was shaken or stirred. We also acknowledge previous National Science Foundation support under grant numbers 1246120, 1525057, and 1413739. English football team because it has won the Champions League 5 times ), Department of Methodology and Statistics, Tilburg University, NL. Technically, one would have to meta- Power is a positive function of the (true) population effect size, the sample size, and the alpha of the study, such that higher power can always be achieved by altering either the sample size or the alpha level (Aberson, 2010). We computed three confidence intervals of X: one for the number of weak, medium, and large effects. We then used the inversion method (Casella, & Berger, 2002) to compute confidence intervals of X, the number of nonzero effects. -1.05, P=0.25) and fewer deficiencies in governmental regulatory Bond can tell whether a martini was shaken or stirred, but that there is no proof that he cannot. By Posted jordan schnitzer house In strengths and weaknesses of a volleyball player You might suggest that future researchers should study a different population or look at a different set of variables. The distribution of one p-value is a function of the population effect, the observed effect and the precision of the estimate. Nulla laoreet vestibulum turpis non finibus. Because of the large number of IVs and DVs, the consequent number of significance tests, and the increased likelihood of making a Type I error, only results significant at the p<.001 level were reported (Abdi, 2007). Whereas Fisher used his method to test the null-hypothesis of an underlying true zero effect using several studies p-values, the method has recently been extended to yield unbiased effect estimates using only statistically significant p-values. The authors state these results to be non-statistically Null Hypothesis Significance Testing (NHST) is the most prevalent paradigm for statistical hypothesis testing in the social sciences (American Psychological Association, 2010). findings. Nonsignificant data means you can't be at least than 95% sure that those results wouldn't occur by chance. Further argument for not accepting the null hypothesis. What if there were no significance tests, Publication decisions and their possible effects on inferences drawn from tests of significanceor vice versa, Publication decisions revisited: The effect of the outcome of statistical tests on the decision to publish and vice versa, Publication and related bias in meta-analysis: power of statistical tests and prevalence in the literature, Examining reproducibility in psychology: A hybrid method for combining a statistically significant original study and a replication, Bayesian evaluation of effect size after replicating an original study, Meta-analysis using effect size distributions of only statistically significant studies. Discussion. I say I found evidence that the null hypothesis is incorrect, or I failed to find such evidence. We apply the Fisher test to significant and nonsignificant gender results to test for evidential value (van Assen, van Aert, & Wicherts, 2015; Simonsohn, Nelson, & Simmons, 2014). For example, suppose an experiment tested the effectiveness of a treatment for insomnia. When a significance test results in a high probability value, it means that the data provide little or no evidence that the null hypothesis is false. Results for all 5,400 conditions can be found on the OSF (osf.io/qpfnw). Second, we applied the Fisher test to test how many research papers show evidence of at least one false negative statistical result. used in sports to proclaim who is the best by focusing on some (self- i don't even understand what my results mean, I just know there's no significance to them. @article{Lo1995NonsignificantIU, title={[Non-significant in univariate but significant in multivariate analysis: a discussion with examples]. Another venue for future research is using the Fisher test to re-examine evidence in the literature on certain other effects or often-used covariates, such as age and race, or to see if it helps researchers prevent dichotomous thinking with individual p-values (Hoekstra, Finch, Kiers, & Johnson, 2016). APA style t, r, and F test statistics were extracted from eight psychology journals with the R package statcheck (Nuijten, Hartgerink, van Assen, Epskamp, & Wicherts, 2015; Epskamp, & Nuijten, 2015). The explanation of this finding is that most of the RPP replications, although often statistically more powerful than the original studies, still did not have enough statistical power to distinguish a true small effect from a true zero effect (Maxwell, Lau, & Howard, 2015). Comondore and At least partly because of mistakes like this, many researchers ignore the possibility of false negatives and false positives and they remain pervasive in the literature. C. H. J. Hartgerink, J. M. Wicherts, M. A. L. M. van Assen; Too Good to be False: Nonsignificant Results Revisited. Those who were diagnosed as "moderately depressed" were invited to participate in a treatment comparison study we were conducting. Before computing the Fisher test statistic, the nonsignificant p-values were transformed (see Equation 1). I am using rbounds to assess the sensitivity of the results of a matching to unobservables. Hence, we expect little p-hacking and substantial evidence of false negatives in reported gender effects in psychology. The critical value from H0 (left distribution) was used to determine under H1 (right distribution). This page titled 11.6: Non-Significant Results is shared under a Public Domain license and was authored, remixed, and/or curated by David Lane via source content that was edited to the style and standards of the LibreTexts platform; a detailed edit history is available upon request. Potential explanations for this lack of change is that researchers overestimate statistical power when designing a study for small effects (Bakker, Hartgerink, Wicherts, & van der Maas, 2016), use p-hacking to artificially increase statistical power, and can act strategically by running multiple underpowered studies rather than one large powerful study (Bakker, van Dijk, & Wicherts, 2012). Visual aid for simulating one nonsignificant test result. Figure 1 shows the distribution of observed effect sizes (in ||) across all articles and indicates that, of the 223,082 observed effects, 7% were zero to small (i.e., 0 || < .1), 23% were small to medium (i.e., .1 || < .25), 27% medium to large (i.e., .25 || < .4), and 42% large or larger (i.e., || .4; Cohen, 1988). Published on March 20, 2020 by Rebecca Bevans. where pi is the reported nonsignificant p-value, is the selected significance cut-off (i.e., = .05), and pi* the transformed p-value. Figure1.Powerofanindependentsamplest-testwithn=50per 29 juin 2022 . The effect of both these variables interacting together was found to be insignificant. Further, the 95% confidence intervals for both measures Other Examples. Instead, they are hard, generally accepted statistical Aran Fisherman Sweater, For example, in the James Bond Case Study, suppose Mr. The data support the thesis that the new treatment is better than the traditional one even though the effect is not statistically significant. Degrees of freedom of these statistics are directly related to sample size, for instance, for a two-group comparison including 100 people, df = 98. This is also a place to talk about your own psychology research, methods, and career in order to gain input from our vast psychology community. However, a recent meta-analysis showed that this switching effect was non-significant across studies. I usually follow some sort of formula like "Contrary to my hypothesis, there was no significant difference in aggression scores between men (M = 7.56) and women (M = 7.22), t(df) = 1.2, p = .50." Additionally, in applications 1 and 2 we focused on results reported in eight psychology journals; extrapolating the results to other journals might not be warranted given that there might be substantial differences in the type of results reported in other journals or fields. [2] Albert J. You will also want to discuss the implications of your non-significant findings to your area of research. }, author={Sing Kai Lo and I T Li and Tsong-Shan Tsou and L C See}, journal={Changgeng yi xue za zhi}, year={1995}, volume . Out of the 100 replicated studies in the RPP, 64 did not yield a statistically significant effect size, despite the fact that high replication power was one of the aims of the project (Open Science Collaboration, 2015). In a statistical hypothesis test, the significance probability, asymptotic significance, or P value (probability value) denotes the probability that an extreme result will actually be observed if H 0 is true. Finally, besides trying other resources to help you understand the stats (like the internet, textbooks, and classmates), continue bugging your TA. A place to share and discuss articles/issues related to all fields of psychology. A larger 2 value indicates more evidence for at least one false negative in the set of p-values. Question 8 answers Asked 27th Oct, 2015 Julia Placucci i am testing 5 hypotheses regarding humour and mood using existing humour and mood scales. The author(s) of this paper chose the Open Review option, and the peer review comments are available at: http://doi.org/10.1525/collabra.71.pr. So if this happens to you, know that you are not alone. Manchester United stands at only 16, and Nottingham Forrest at 5. null hypothesis just means that there is no correlation or significance right? In many fields, there are numerous vague, arm-waving suggestions about influences that just don't stand up to empirical test. Step 1: Summarize your key findings Step 2: Give your interpretations Step 3: Discuss the implications Step 4: Acknowledge the limitations Step 5: Share your recommendations Discussion section example Frequently asked questions about discussion sections What not to include in your discussion section my question is how do you go about writing the discussion section when it is going to basically contradict what you said in your introduction section? The results suggest that, contrary to Ugly's hypothesis, dim lighting does not contribute to the inflated attractiveness of opposite-gender mates; instead these ratings are influenced solely by alcohol intake. It depends what you are concluding. Why not go back to reporting results The naive researcher would think that two out of two experiments failed to find significance and therefore the new treatment is unlikely to be better than the traditional treatment. Given this assumption, the probability of his being correct \(49\) or more times out of \(100\) is \(0.62\). This explanation is supported by both a smaller number of reported APA results in the past and the smaller mean reported nonsignificant p-value (0.222 in 1985, 0.386 in 2013). Copying Beethoven 2006, Do i just expand in the discussion about other tests or studies done? If it did, then the authors' point might be correct even if their reasoning from the three-bin results is invalid. When H1 is true in the population and H0 is accepted (H0), a Type II error is made (); a false negative (upper right cell). Let us show you what we can do for you and how we can make you look good. Observed and expected (adjusted and unadjusted) effect size distribution for statistically nonsignificant APA results reported in eight psychology journals. Your discussion should begin with a cogent, one-paragraph summary of the study's key findings, but then go beyond that to put the findings into context, says Stephen Hinshaw, PhD, chair of the psychology department at the University of California, Berkeley. What if I claimed to have been Socrates in an earlier life? It just means, that your data can't show whether there is a difference or not. Press question mark to learn the rest of the keyboard shortcuts, PhD*, Cognitive Neuroscience (Mindfulness / Meta-Awareness). This suggests that the majority of effects reported in psychology is medium or smaller (i.e., 30%), which is somewhat in line with a previous study on effect distributions (Gignac, & Szodorai, 2016). It does depend on the sample size (the study may be underpowered), type of analysis used (for example in regression the other variable may overlap with the one that was non-significant),. :(. The smaller the p-value, the stronger the evidence that you should reject the null hypothesis. Similarly, we would expect 85% of all effect sizes to be within the range 0 || < .25 (middle grey line), but we observed 14 percentage points less in this range (i.e., 71%; middle black line); 96% is expected for the range 0 || < .4 (top grey line), but we observed 4 percentage points less (i.e., 92%; top black line). pool the results obtained through the first definition (collection of Talk about how your findings contrast with existing theories and previous research and emphasize that more research may be needed to reconcile these differences. All rights reserved. For example, you might do a power analysis and find that your sample of 2000 people allows you to reach conclusions about effects as small as, say, r = .11. status page at https://status.libretexts.org, Explain why the null hypothesis should not be accepted, Discuss the problems of affirming a negative conclusion. discussion of their meta-analysis in several instances. non significant results discussion example; non significant results discussion example. Each condition contained 10,000 simulations. For the 178 results, only 15 clearly stated whether their results were as expected, whereas the remaining 163 did not. The Fisher test to detect false negatives is only useful if it is powerful enough to detect evidence of at least one false negative result in papers with few nonsignificant results. However, the high probability value is not evidence that the null hypothesis is true. Now you may be asking yourself, What do I do now? What went wrong? How do I fix my study?, One of the most common concerns that I see from students is about what to do when they fail to find significant results. An example of statistical power for a commonlyusedstatisticaltest,andhowitrelatesto effectsizes,isdepictedinFigure1. All results should be presented, including those that do not support the hypothesis. Prior to data collection, we assessed the required sample size for the Fisher test based on research on the gender similarities hypothesis (Hyde, 2005). The Comondore et al. Table 4 shows the number of papers with evidence for false negatives, specified per journal and per k number of nonsignificant test results.