I used SPSS Complex Sampling package with the Mersenne Twister algorithm to randomly select 30 sample units from 6 strata in my population with a given seed of 1328256. I want to keep my results the same as above (keep the same sample units) while adding an additional 5 sample units to each stratum. Is this possible, and if so, how? Is it possible to add just 5 to just 1 stratum while leaving the results from the other strata unchanged?

------------------------------
Orion Lawler
------------------------------

#SPSSStatistics

Hi,
Since sampling is independent in each of the strata you can do whatever you want in one without affecting the others. In practice, just select out the one stratum that you need to add cases to. A quick and dirty way to select five extra is to remove the selected ones from your sample, and then select again (and for estimation purposes happily ignoring that you did). Another, slightly more cumbersome approach is to select five extra from the whole (including your original 30) and then if you have selected an original one substitute with year another selection. The simplest way to deal with inclusion probabilities is to assume they are 35/N (provided you don't do PPS). You do not say what kind of sampling you do, or anything about population sizes, so difficult to be more specific. Leslie Kish has a paper on adding units to strata and changing strata, but most of the stuff there is for more complex situations than the one you describe.
hope this helps,
Jon

------------------------------
Jon Pedersen
------------------------------

Thank you Jon. Your response was very insightful. The sample needs to be random and so unfortunately I cannot simply substitute sample units that are reselected. The population is large, at ~2,500 records, with each strata containing various amounts: s1 ~ 1,000, s2 ~ 450, s3 ~ 300, s4 ~ 250 s5 ~ 200, s6 ~ 150. Is it statistically valid to just remove the sampling units from the population and sample for an additional 5? The results need to be able to be replicated, so will using the same seed as the original sampling yield replicable results? I'm extrapolating the results from the sample to the population so I'm a little worried that simply picking another 5 from the population and mixing them in will invalidate the extrapolation process. Do you know the name of Leslie Kush's paper on adding units to strata?

Thanks!

------------------------------
Orion Lawler
------------------------------

Original Message:
Sent: Thu April 30, 2020 03:10 AM
From: Jon Pedersen
Subject: Is it possible to maintain the same sample units while adding sample units to each stratum?

Hi,
Since sampling is independent in each of the strata you can do whatever you want in one without affecting the others. In practice, just select out the one stratum that you need to add cases to. A quick and dirty way to select five extra is to remove the selected ones from your sample, and then select again (and for estimation purposes happily ignoring that you did). Another, slightly more cumbersome approach is to select five extra from the whole (including your original 30) and then if you have selected an original one substitute with year another selection. The simplest way to deal with inclusion probabilities is to assume they are 35/N (provided you don't do PPS). You do not say what kind of sampling you do, or anything about population sizes, so difficult to be more specific. Leslie Kish has a paper on adding units to strata and changing strata, but most of the stuff there is for more complex situations than the one you describe.
hope this helps,
Jon

------------------------------
Jon Pedersen
------------------------------

Hi,
If your selection method in SPSS complex samples is SIMPLE_WOR I think it will draw extra ones and keep your originals selected if you use the same seed. I am not altogether sure if it makes a big difference to the other methods.
Kish's paper is called "Changing strata and selection probabilities" It is in an American statistical association conference proceeding from 1963. http://www.asasrms.org/Proceedings/y1963/Changing%20Strata%20And%20Selection%20Probabilities.pdfThere is also a rather similar piece by Kish and Scott in the Leslie Kish Selected Papers book (Edited by Heeringa and Kalton). I think both papers refer to your situation briefly, although they are more focused on more complex surveys.
best
Jon

------------------------------
Jon Pedersen
------------------------------

Original Message:
Sent: Thu April 30, 2020 12:39 PM
From: Orion Lawler
Subject: Is it possible to maintain the same sample units while adding sample units to each stratum?

Thank you Jon. Your response was very insightful. The sample needs to be random and so unfortunately I cannot simply substitute sample units that are reselected. The population is large, at ~2,500 records, with each strata containing various amounts: s1 ~ 1,000, s2 ~ 450, s3 ~ 300, s4 ~ 250 s5 ~ 200, s6 ~ 150. Is it statistically valid to just remove the sampling units from the population and sample for an additional 5? The results need to be able to be replicated, so will using the same seed as the original sampling yield replicable results? I'm extrapolating the results from the sample to the population so I'm a little worried that simply picking another 5 from the population and mixing them in will invalidate the extrapolation process. Do you know the name of Leslie Kush's paper on adding units to strata?

Thanks!

------------------------------
Orion Lawler
------------------------------


Original Message:
Sent: Thu April 30, 2020 03:10 AM
From: Jon Pedersen
Subject: Is it possible to maintain the same sample units while adding sample units to each stratum?

Hi,
Since sampling is independent in each of the strata you can do whatever you want in one without affecting the others. In practice, just select out the one stratum that you need to add cases to. A quick and dirty way to select five extra is to remove the selected ones from your sample, and then select again (and for estimation purposes happily ignoring that you did). Another, slightly more cumbersome approach is to select five extra from the whole (including your original 30) and then if you have selected an original one substitute with year another selection. The simplest way to deal with inclusion probabilities is to assume they are 35/N (provided you don't do PPS). You do not say what kind of sampling you do, or anything about population sizes, so difficult to be more specific. Leslie Kish has a paper on adding units to strata and changing strata, but most of the stuff there is for more complex situations than the one you describe.
hope this helps,
Jon

------------------------------
Jon Pedersen

The selection method is SIMPLE_WOR, but when I expand the sample size per strata, with the same seed, I get much different results than the original sample.

------------------------------
Orion Lawler
------------------------------

Original Message:
Sent: Thu April 30, 2020 02:11 PM
From: Jon Pedersen
Subject: Is it possible to maintain the same sample units while adding sample units to each stratum?

Hi,
If your selection method in SPSS complex samples is SIMPLE_WOR I think it will draw extra ones and keep your originals selected if you use the same seed. I am not altogether sure if it makes a big difference to the other methods.
Kish's paper is called "Changing strata and selection probabilities" It is in an American statistical association conference proceeding from 1963. http://www.asasrms.org/Proceedings/y1963/Changing%20Strata%20And%20Selection%20Probabilities.pdfThere is also a rather similar piece by Kish and Scott in the Leslie Kish Selected Papers book (Edited by Heeringa and Kalton). I think both papers refer to your situation briefly, although they are more focused on more complex surveys.
best
Jon

------------------------------
Jon Pedersen
------------------------------


Original Message:
Sent: Thu April 30, 2020 12:39 PM
From: Orion Lawler
Subject: Is it possible to maintain the same sample units while adding sample units to each stratum?

Thank you Jon. Your response was very insightful. The sample needs to be random and so unfortunately I cannot simply substitute sample units that are reselected. The population is large, at ~2,500 records, with each strata containing various amounts: s1 ~ 1,000, s2 ~ 450, s3 ~ 300, s4 ~ 250 s5 ~ 200, s6 ~ 150. Is it statistically valid to just remove the sampling units from the population and sample for an additional 5? The results need to be able to be replicated, so will using the same seed as the original sampling yield replicable results? I'm extrapolating the results from the sample to the population so I'm a little worried that simply picking another 5 from the population and mixing them in will invalidate the extrapolation process. Do you know the name of Leslie Kush's paper on adding units to strata?

Thanks!

------------------------------
Orion Lawler


Original Message:
Sent: Thu April 30, 2020 03:10 AM
From: Jon Pedersen
Subject: Is it possible to maintain the same sample units while adding sample units to each stratum?

Hi,
Since sampling is independent in each of the strata you can do whatever you want in one without affecting the others. In practice, just select out the one stratum that you need to add cases to. A quick and dirty way to select five extra is to remove the selected ones from your sample, and then select again (and for estimation purposes happily ignoring that you did). Another, slightly more cumbersome approach is to select five extra from the whole (including your original 30) and then if you have selected an original one substitute with year another selection. The simplest way to deal with inclusion probabilities is to assume they are 35/N (provided you don't do PPS). You do not say what kind of sampling you do, or anything about population sizes, so difficult to be more specific. Leslie Kish has a paper on adding units to strata and changing strata, but most of the stuff there is for more complex situations than the one you describe.
hope this helps,
Jon

------------------------------
Jon Pedersen

------Original Message------

The selection method is SIMPLE_WOR, but when I expand the sample size per strata, with the same seed, I get much different results than the original sample.

------------------------------
Orion Lawler
------------------------------

#SPSSStatistics

Previously I was sampling on the population with all strata less the original sample units, and I was getting different results for strata 2-6 when I attempted to sample an additional 3, 5 and 10 units. I then realized if I broke the population into each stratum and sampled from this data set then I would end up with the same sample units in the 10 unit sample that I had in the 3 and 5 unit samples. My concern is with extrapolation now. Is it statistically valid to extrapolate these new sample units along with the original sample units to the population?

------------------------------
Orion Lawler
------------------------------

Original Message:
Sent: Fri May 01, 2020 01:19 PM
From: Jon Pedersen
Subject: Is it possible to maintain the same sample units while adding sample units to each stratum?

Hi

That should not happen. The method is sequential and deterministic given the seed. I tried with a small example and that gave the expected results. Do you select the whole sample? If you selected only the stratum of interest this time you would get different results unless it is the first stratum. 

Best

Jon

Original Message------

The selection method is SIMPLE_WOR, but when I expand the sample size per strata, with the same seed, I get much different results than the original sample.

------------------------------
Orion Lawler
------------------------------

#SPSSStatistics

Hi,
Since both the original and the new are sequential samples with the same random start, your new sample is what would have been obtained if you had sampled them originally. SPSS has a rather terse explanation in their statistics algorithms pdf, but it brings out the deterministic nature. Their reference: Fan, C. T., M. E. Muller, and I. Rezucha. 1962. Development of Sampling Plans by Using
Sequential (Item by Item) Selection Techniques and Digital Computers. Journal of the American
Statistical Association, 57, 387–402.  gives a quite clear exposition. 
But anyway, yes, it should be fine, provided you did not increase the sample size as a consequence of what was observed. That could easily lead you into p-hacking. (i.e. if something was nearly significant, and you decided to increase the sample size to get it to be significant) There is sequential/adaptive sampling literature that deals with inference and estimation when you have some criterion for adding to the sample. But that may be overkill in your case.

------------------------------
best
Jon
------------------------------

Original Message:
Sent: Fri May 01, 2020 05:35 PM
From: Orion Lawler
Subject: Is it possible to maintain the same sample units while adding sample units to each stratum?

Previously I was sampling on the population with all strata less the original sample units, and I was getting different results for strata 2-6 when I attempted to sample an additional 3, 5 and 10 units. I then realized if I broke the population into each stratum and sampled from this data set then I would end up with the same sample units in the 10 unit sample that I had in the 3 and 5 unit samples. My concern is with extrapolation now. Is it statistically valid to extrapolate these new sample units along with the original sample units to the population?

------------------------------
Orion Lawler
------------------------------


Original Message:
Sent: Fri May 01, 2020 01:19 PM
From: Jon Pedersen
Subject: Is it possible to maintain the same sample units while adding sample units to each stratum?

Hi

That should not happen. The method is sequential and deterministic given the seed. I tried with a small example and that gave the expected results. Do you select the whole sample? If you selected only the stratum of interest this time you would get different results unless it is the first stratum. 

Best

Jon

Original Message------

The selection method is SIMPLE_WOR, but when I expand the sample size per strata, with the same seed, I get much different results than the original sample.

------------------------------
Orion Lawler
------------------------------

#SPSSStatistics

Well originally I sampled 180 from the ~2,500, but when I received unexpected results I reduced the 2,500 by the 180 and sampled another 30 from this new population. I used the same seed for both samples, but from what you're saying above, this effects the inclusion probability and therefore is considered p-hacking, correct? Do you know of the article that references how to properly draw additional sample units from the population after the original draw?

------------------------------
Orion Lawler
------------------------------

Original Message:
Sent: Sat May 02, 2020 08:04 AM
From: Jon Pedersen
Subject: Is it possible to maintain the same sample units while adding sample units to each stratum?

Hi,
Since both the original and the new are sequential samples with the same random start, your new sample is what would have been obtained if you had sampled them originally. SPSS has a rather terse explanation in their statistics algorithms pdf, but it brings out the deterministic nature. Their reference: Fan, C. T., M. E. Muller, and I. Rezucha. 1962. Development of Sampling Plans by Using
Sequential (Item by Item) Selection Techniques and Digital Computers. Journal of the American
Statistical Association, 57, 387–402.  gives a quite clear exposition. 
But anyway, yes, it should be fine, provided you did not increase the sample size as a consequence of what was observed. That could easily lead you into p-hacking. (i.e. if something was nearly significant, and you decided to increase the sample size to get it to be significant) There is sequential/adaptive sampling literature that deals with inference and estimation when you have some criterion for adding to the sample. But that may be overkill in your case.

------------------------------
best
Jon
------------------------------


Original Message:
Sent: Fri May 01, 2020 05:35 PM
From: Orion Lawler
Subject: Is it possible to maintain the same sample units while adding sample units to each stratum?

Previously I was sampling on the population with all strata less the original sample units, and I was getting different results for strata 2-6 when I attempted to sample an additional 3, 5 and 10 units. I then realized if I broke the population into each stratum and sampled from this data set then I would end up with the same sample units in the 10 unit sample that I had in the 3 and 5 unit samples. My concern is with extrapolation now. Is it statistically valid to extrapolate these new sample units along with the original sample units to the population?

------------------------------
Orion Lawler


Original Message:
Sent: Fri May 01, 2020 01:19 PM
From: Jon Pedersen
Subject: Is it possible to maintain the same sample units while adding sample units to each stratum?

Hi

That should not happen. The method is sequential and deterministic given the seed. I tried with a small example and that gave the expected results. Do you select the whole sample? If you selected only the stratum of interest this time you would get different results unless it is the first stratum. 

Best

Jon

Original Message------

The selection method is SIMPLE_WOR, but when I expand the sample size per strata, with the same seed, I get much different results than the original sample.

------------------------------
Orion Lawler
------------------------------

#SPSSStatistics

------Original Message------

Well originally I sampled 180 from the ~2,500, but when I received unexpected results I reduced the 2,500 by the 180 and sampled another 30 from this new population. I used the same seed for both samples, but from what you're saying above, this effects the inclusion probability and therefore is considered p-hacking, correct? Do you know of the article that references how to properly draw additional sample units from the population after the original draw?

------------------------------
Orion Lawler
------------------------------

#SPSSStatistics