SPSS Statistics

 View Only
Expand all | Collapse all

Benjamini & Hochberg method to control the false discovery rate

  • 1.  Benjamini & Hochberg method to control the false discovery rate

    Posted Tue May 10, 2022 03:35 AM
    Hello fellow SPSS users.

    I am using the following tech note for FDR correction implementation:

    Does SPSS Statistics offer multiple comparisons using the Benjamini & Hochberg method to control the false discovery rate?

    although it provides the technical basis for the correction, it does not offer any theoretical background.

    Does anybody have some info regarding the rationale behind the BH FDR? any theoretical resources?

    Thanks!!



    ------------------------------
    Meni Berger
    ------------------------------

    #SPSSStatistics


  • 2.  RE: Benjamini & Hochberg method to control the false discovery rate

    IBM Champion
    Posted Tue May 10, 2022 08:31 AM
    The STATS PADJUST extension command available via the Extensions > Extension Hub menu provides six multiple-testing corrections and works with the output from any command that includes the sig levels.

    For background on BH, check out this link

    There are many others as well.

    --



    Original Message


  • 3.  RE: Benjamini & Hochberg method to control the false discovery rate

    Posted Wed May 11, 2022 03:56 AM
    Thanks, Jon

    This package looks great. better than writing code for those who aren't proficient in Syntax.

    although Wikipedia offers some excellent theoretical background, I need to keep my students awake ;) 

    I Am looking for the summary of the subjet.




    ------------------------------
    Meni Berger
    ------------------------------

    Original Message


  • 4.  RE: Benjamini & Hochberg method to control the false discovery rate

    Posted Wed May 11, 2022 07:24 AM
    Try these. Maybe the students will stay awake:

    A Guide to the Benjamini-Hochberg Procedure - Statology

    https://cpb-us-w2.wpmucdn.com/blog.nus.edu.sg/dist/0/3425/files/2018/10/Understanding-Benjamini-Hochberg-method-2ijolq0.pdf


    ------------------------------
    Rick Marcantonio
    Quality Assurance
    IBM
    ------------------------------

    Original Message


  • 5.  RE: Benjamini & Hochberg method to control the false discovery rate

    IBM Champion
    Posted Wed May 11, 2022 08:45 AM
    One thing that is glossed over in these notes is the assumption underlying the B-H procedure.

    This link has a useful discussion about this.

    An extract...

    Assumption

    The Bonferroni correction and Benjamini-Hochberg procedure assume that the individual tests are independent of each other, as when you are comparing sample A vs. sample B, C vs. D, E vs. F, etc. If you are comparing sample A vs. sample B, A vs. C, A vs. D, etc., the comparisons are not independent; if A is higher than B, there's a good chance that A will be higher than C as well. One place this occurs is when you're doing unplanned comparisons of means in anova, for which a variety of other techniques have been developed, such as the Tukey-Kramer test. Another experimental design with multiple, non-independent comparisons is when you compare multiple variables between groups, and the variables are correlated with each other within groups. An example would be knocking out your favorite gene in mice and comparing everything you can think of on knockout vs. control mice: length, weight, strength, running speed, food consumption, feces production, etc. All of these variables are likely to be correlated within groups; mice that are longer will probably also weigh more, would be stronger, run faster, eat more food, and poop more. To analyze this kind of experiment, you can use multivariate analysis of variance, or manova, which I'm not covering in this textbook.


    --



    Original Message


  • 6.  RE: Benjamini & Hochberg method to control the false discovery rate

    Posted Tue May 17, 2022 04:51 AM
    Hello, Rick!

    I've used https://www.statology.org/ summary in my presentation and handed out Linda's paper for the nerdy students who need to know the nitty-gritty details. 

    much success very thank you.

    ------------------------------
    Meni Berger
    ------------------------------

    Original Message


  • 7.  RE: Benjamini & Hochberg method to control the false discovery rate

    Posted Tue May 17, 2022 06:32 AM
    @Jon Peck, thank you for the info. naturally, I presented my students with the issue of correlated IV variables and multiple comparisons.

    I have a question regarding ​PADJUST. 

    the syntax alternative has a Q value for controlling FDR. the PADJUST does not offer such an option for B-H. I've searched the R documentation for such a parameter for B-H and found none. 
    what Q does it use then?

    P.S.
    Respect for implementing R p.adjust into SPSS Statistics.

     




    ------------------------------
    Meni Berger
    ------------------------------

    Original Message


  • 8.  RE: Benjamini & Hochberg method to control the false discovery rate

    IBM Champion
    Posted Tue May 17, 2022 08:50 AM
    I don't know what Q is.  The command returns a set of adjusted p values according to the selected method(s).  Here is some additional information from the underlying package.

    The adjustment methods include the Bonferroni correction ("bonferroni") in which the p-values are multiplied by the number of comparisons. Less conservative corrections are also included by Holm (1979) ("holm"), Hochberg (1988) ("hochberg"), Hommel (1988) ("hommel"), Benjamini & Hochberg (1995) ("BH" or its alias "fdr"), and Benjamini & Yekutieli (2001) ("BY"), respectively. 

    The first four methods are designed to give strong control of the family-wise error rate. There seems no reason to use the unmodified Bonferroni correction because it is dominated by Holm's method, which is also valid under arbitrary assumptions.

    Hochberg's and Hommel's methods are valid when the hypothesis tests are independent or when they are non-negatively associated (Sarkar, 1998; Sarkar and Chang, 1997). Hommel's method is more powerful than Hochberg's, but the difference is usually small and the Hochberg p-values are faster to compute.

    The "BH" (aka "fdr") and "BY" methods of Benjamini, Hochberg, and Yekutieli control the false discovery rate, the expected proportion of false discoveries amongst the rejected hypotheses. The false discovery rate is a less stringent condition than the family-wise error rate, so these methods are more powerful than the others.

    Note that you can set n larger than the number of the values listed which means the unobserved p-values are assumed to be greater than all the observed p for "bonferroni" and "holm" methods and equal to 1 for the other methods.

    --



    Original Message


  • 9.  RE: Benjamini & Hochberg method to control the false discovery rate

    Posted Mon May 23, 2022 03:23 AM

    Hello Jon,

     

    The Q is the proportion of the FDR you can control manually.

     

    In the syntax you can edit this value (which is set to .05) to any value you desire.

     

    If I understand correctly the R package will calculate the BH adjustment and it is up to the user to apply the desired Q?

     

     

     

     

     

     

    Meni Berger | Data scientist and Head of Tech Support E-mail    Meni@genius.co.il

    11 Menachem Begin st.,  Ramat Gan

    www.genius.co.il

    Title: LinkedIn - Description: image of LinkedIn icon

    Click here to open a support ticket  

     

     




    Original Message


  • 10.  RE: Benjamini & Hochberg method to control the false discovery rate

    IBM Champion
    Posted Mon May 23, 2022 08:40 AM
    The command provides a parameter for the number of tests.
    By default, the number of tests is assumed to be the
    same as the number of p values supplied.  However, you
    can use NTESTS to set a larger number.  This assumes that
    the omitted p values are either larger than any of the
    included ones (Bonferroni and Holm) or are equal to 1.

    That corresponds to the "n" parameter in the R p.adjust function.

    It is up to the user to decide on the cutoff point.  Here is a plot from the command.

    image.png



    --



    Original Message


  • 11.  RE: Benjamini & Hochberg method to control the false discovery rate

    Posted 30 days ago

    Hello, I can't seem to figure out where or how to actually install the FDR extension for SPSS. Can someone post a direct link to the extension hub with the multiple comparison options? Thank you! 



    ------------------------------
    Desiree Delgadillo
    ------------------------------

    Original Message


  • 12.  RE: Benjamini & Hochberg method to control the false discovery rate

    IBM Champion
    Posted 30 days ago
    Go to the Extension Hub and search for PADJUST.

    If you don't see it, the extension may already be installed, and it would appear under Analyze > Descriptive Statistics

    --



    Original Message


  • 13.  RE: Benjamini & Hochberg method to control the false discovery rate

    Posted 30 days ago

    Thank you for the replies! I was able to locate the 'calculate adjusted p-values' option in descriptive statistics. But I cannot figure out how to use this for my GLM. 

    For example, when conducting the general linear model (GLM), SPSS gives you a post-hoc option to apply multiple comparison corrections such as Bonferroni and Tukey etc. However, after installing the "Calculate Adjusted P-Values" extension in SPSS, the Benjamini-Hochberg (BH) correction only appears in the descriptive statistics option but NOT when I conduct the GLM. Do I need to create a variable of only my p-values from the GLM and then conduct the BH correction? If so, how do I do that?

    I tried conducting the general linear model but the extension was not available to me there. How do I apply the benjamini-hochberg correction for my GLM?



    ------------------------------
    Desiree Delgadillo
    ------------------------------

    Original Message


  • 14.  RE: Benjamini & Hochberg method to control the false discovery rate

    IBM Champion
    Posted 30 days ago
    To apply a multiple-testing correction, you need all the reported sig levels you want to correct and, in some cases the number of tests.  You can enter those in the STATS PADJUST command syntax or dialog box.--



    Original Message


  • 15.  RE: Benjamini & Hochberg method to control the false discovery rate

    Posted 30 days ago

    I made a YouTube tutorial on FDR using this extension. although it's in Hebrew, try to follow my doing on the screen.

    If it still doesn't make sense let me know and I'll try to help you:

    Conducting Benjamini & Hochberg method to control the false discovery rate in SPSS Statistics 28.0.1

    YouTube remove preview
    Conducting Benjamini & Hochberg method to control the false discovery rate in SPSS Statistics 28.0.1
    בסרטון קצת הקדמה על הרציונל של השיטה וכיצד לערוך את תיקון FDR של בנימיני הוכברג להשוואות מרובות, באמצעות סינטקס של SPSS. קישור למאמר הטכני של IBM המכיל הסבר נוסף והסינטקס המקורי: https://www.ibm.com/support/pages/does-spss-statistics-offer-multiple-comparisons-using-benjamini-hochberg-method-control-false-discovery-rate
    View this on YouTube >



    ------------------------------
    Meni Berger
    ------------------------------

    Original Message