Hi,

I'm trying to batch process about 80 txt files (filenames begin with "U") and would like to append them into a single SPSS file. My syntax looks like this:

define !out () !quote(!concat(!unquote(!eval(!JOB_OUTPUTDATADIR)),

!unquote(!eval(!JOB_DATAFILEROOT)),

".sav")) !enddefine.

SPSSINC PROCESS FILES

INPUTDATA="/Users/sarah/Desktop/Data/output data/U*.txt"

SYNTAX="/Users/sarah/Desktop/Data/output data/STUDYID.sps"

OUTPUTDATADIR="/Users/sarah/Desktop/Data/output data" CONTINUEONERROR=YES

VIEWERDIR= "/Users/sarah/Desktop/Data/output data" CLOSEDATA=YES

LOGFILE= "/Users/sarahherzog/Desktop/Data/output data/masterfile.txt" MACRONAME="!JOB"

LOGFILEMODE=APPEND

/MACRODEFS ITEMS QPARM1="JOB".

When I run this, I get the error "“The procedure cannot access a file with the given file specification: /Users/sarah/Desktop/Data/output data/masterfile.txt for keyword LOGFILE.”

'masterfile.txt' doesn't actually exist, I had assumed the LOGFILE path required the name for the file that the output would be appended to. Did I get that wrong?

I also tried the above syntax without the LOGFILE line, and it transformed several txt files to SPSS before stopping with the error: "Output document NAME='^1"

In case the problem is my syntax file STUDYID, this is the content of the syntax:

PRESERVE.

SET DECIMAL DOT.

GET DATA /TYPE=TXT

/FILE=JOB_INPUTFILE

/ENCODING='UTF8'

/DELIMITERS="\t"

/QUALIFIER='"'

/ARRANGEMENT=DELIMITED

/FIRSTCASE=2

/DATATYPEMIN PERCENTAGE=95.0

/VARIABLES=

StartEvent AUTO

StartTime AUTO

EndEvent AUTO

EndTime AUTO

SegmentDuration AUTO

ManualOverride AUTO

Thresholdvolts AUTO

MeanHeartRate AUTO

RSA AUTO

MeanIBI AUTO

Support Member#ofRsFound AUTO

RespirationRate AUTO

RespirationAmplitude AUTO

RespirationPeakFrequency AUTO

RespirationPower AUTO

FirstECGRTime AUTO

LastECGRTime AUTO

FirstRtoLastRDuration AUTO

SDNN AUTO

AVNN AUTO

RMSSD AUTO

NN50 AUTO

pNN50 AUTO

/MAP.

RESTORE.

CACHE.

EXECUTE.

DATASET NAME DataSet3 WINDOW=FRONT.

STRING STUDYID (A10).

COMPUTE STUDYID=CHAR.SUBSTR(StartEvent,1,6).

EXECUTE.

ADD FILES file *

/KEEP STUDYID all.

EXECUTE.

Thanks so much!

First, get rid of all those EXECUTE commands and the CACHE command. Not the problem, but just wastes data passes.

The logfile will be created if it doesn't already exist. However, it might be that

/Users/sarahherzog/Desktop/Data/output data/masterfile.txt

is not a valid filespec. That is, could

/Users/sarahherzog/Desktop/Data/output data

not be a valid path? Or could the current drive not be the one where it exists, since there is no drive letter there.

Also, I think there is a problem with the ADD FILES syntax. There should be an = after file. Bu the bigger problem is that at that point, your just-read file is the active file, so this won't achieve what you want. If you have assigned a dataset name to what will be the master file (not starting with U!), then you would want something like file=master /file=*. Check the CSR for the subtle details.

Now this assumes that a master file is already open. The first time through, it might not be. You can open one file manually and assign it a dataset name or you can use the BEFORESYNTAX keyword to execute another syntax file before starting on the loop.

Also, I don't see !out being used. Harmless to define it, but if you had something else in mind, it needs looking at.

SPSSINC PROCESS FILES can be a huge time saver, but it does take some work to get it right.

Thanks, Jon, this was hugely helpful, I managed to use it on multiple jobs. Do you know if it's possible to use PROCESS to merge multiple datasets?

Thanks again!

Great. As for merging, as we discussed before, I think, you would open the first one and then iteratively run ADD FILES to add them one at a time. However, you might be better off taking advantage of ADD FILES ability to combine up to 50 files at a time as long as these are already sav files. If you need a way to avoid having to list them one by one, a little bit of Python code could create a macro from directory contents. If you want to explain further, I can help with that.