Looking for help solving the following problem, which I will try to describe with edited in code. Question is at the end.

I have the following setup within the SPSS macro language:

** start.

!let !pv = X

!let !rwgt = Y

!do !p = 1 !to !pv.

get file with !n cases.

sort cases by sortvar (40 groups).

oms to output regression results to file Output(!p)

!do !r = 1 to !!rwgt

split file by sortvar

run a regression

split file off

!doend.

omsend.

dataset close.

Post-process the !rwgt results for !pv using the files created with OMS.

omsend.

dataset close.

!doend.

** end.

the macro runs fine with relatively small !n and values for !p and !r, but two problems:

1. as it makes its way through the !p's, performance degrades noticeably. each subsequent !pv cycle seems to take 1.5x - 2.0x the previous cycle.
2. i need to run this on a file with !n = 200,000, !pv = 20 and !rwgt =250 and it completely stalls at around !p = 10, never completing.

i tried looking for help with OpenAI, but i received no useful hints.

i realize it's a memory problem, which is why is why i add the omsend and dataset close statements at the end of each !pv loop, but this has done nothing to speed up or complete the run.

i would welcome any suggestions, and would be happy to share the full code and data file if that helps.

thank you!

eugene

I can't tell very much from what is here, but here are a few thoughts.

Use SET MPRINT ON. at the start to see what the actual expanded code looks like. 

The macro will be fully expanded before any of the generated code is run.  There might be a lot of code generated then that might choke the command queue (technical term :-))

If you are generating a lot of output in the Viewer, you might want to close the Viewer periodically so that it doesn't consume all the memory or get tied up in handling a huge number of objects.

You might use VIEWER=NO  on the OMS command to avoid writing the output to the Viewer.  It appears that you are writing the output to a file but not suppressing it in the Viewer.

If you can use Python external mode to run the code, that could be a big speedup.  Some years ago (actually quite a few years ago), I worked with a customer who was running many tens of thousands of regressions but needed only a little bit of the output.  Converting it to use external mode  greatly sped it up - more than an order of magnitude improvement, though I don't remember the exact details.  There is no Viewer in external mode.

hi jon,

as always, thanks for the quick reply.

here are answers to your comments/questions:

1. except for a few before the regressions are run, all the regression OMS commands have viewer = NO to suppress the output to the spv file.

2. the MPRINT ON shows nothing suspect, at least that i can see

3. there is a lot of code generated, but if it gets through the first PV in an ok time, why/how could there be a "choke" in the command line?

4. i also turned off the autorecovery of files and there was no change in performance.

5. i also close the output window and open a blank new one a the start of the run.

clearly something is accumulating somewhere, but what could it be, and how can i clear it...?

Hello Eugenio.  Please show us what the REGRESSION command looks like where you have "run a regression" in your pseudo-code.  I'm thinking of the so-called XY problem, and wondering if there could be a more efficient way to achieve what you want to do.  Thanks for clarifying.  

Cheers,
Bruce

hi bruce,

here are the attachments.

e.

Attachment(s)

test_LR_20pvs_SPSS.sps.txt   770 B 1 version

JB_RegGP_new3.ieasps.txt   46 KB 1 version

hi,

none of the suggestions have provided additional information or solved the problem.

before filing in the "cold case" folder, any additional suggestions?

thank you!

e.