IBM Security Guardium

 View Only

Guardium Windows Must Gather V3.0 - Part 7 - What if Must Gather doesn't generate output?

By SATOSHI KAWASE posted Sun October 30, 2022 07:00 AM

  

About Windows Agent Must Gather V3.0

Guardium Windows Agent Must Gather V3.0 (a.k.a. Windows S-TAP Must Gather) is the latest must gather script, which is released in Guardium V11.5.

It's included in all Guardium Windows agents (GIM, S-TAP, GAM, CAS, FAM monitor, FAM crawler, FDEC for NAS/SP, FAM for NAS/SP) in V11.5, and will be back-ported to all supported versions.

Index

  1. What's new in V3.0?
  2. S-TAP mode and STANDALONE mode
  3. How to run Must Gather V3.0?
  4. Must Gather V3.0 command options
  5. Where is the output of Must Gather?
  6. What files are collected?
  7. What if Must Gather doesn't generate output?

NOTE: This blog article covers the section 7. Click links to read other sections.

7. What if Must Gather doesn't generate output?

What if you run Must Gather but no output zip file is generated? There are several possibilities. It could be a Must Gather issue. It could be a communication issue between the Guardium GUI and S-TAP.

7.1 When you run Must Gather on DB server

Before starting troubleshooting, let me share a couple of example outputs of diag.log for good cases.

Here is an example output of diag.log in a good case in S-TAP mode. Must Gather uses ExternalZip.exe and generate a zip file. You can see the exact zip file location with the file size and the time generated.

`I 2022-08-29T04:35:05.9327  STEP7.1: === STEP 7.1 : Zipping diagnostic files.
`I 2022-08-29T04:35:05.9409  STEP7.1: ZipSourceDir : "C:\Program Files\IBM\Windows S-TAP\Logs"
`I 2022-08-29T04:35:05.9429  STEP7.1: ZipTargetDir : "C:\Program Files\IBM\Windows S-TAP\Bin\zipTmp"
`I 2022-08-29T04:35:05.9429  STEP7.1: Creating C:\Program Files\IBM\Windows S-TAP\Logs\ZipSource.dir.txt.
`I 2022-08-29T04:35:06.0049  STEP7.1: Printing the list of files using 'Get-ChildItem' to ZipSource.dir.txt.
`I 2022-08-29T04:35:06.5776  STEP7.1: Mode: Running in S-TAP mode.
`I 2022-08-29T04:35:06.5776  STEP7.1: Calling "ExternalZip.exe -z".
`I 2022-08-29T04:35:34.2087  STEP7.2: === STEP 7.2 : Checking the zip file and the size.
`I 2022-08-29T04:35:34.2590  STEP7.2: ZIP file was generated.
`I 2022-08-29T04:35:34.2590  STEP7.2: Folder : "C:\Program Files\IBM\Windows S-TAP\Bin\zipTmp"
`I 2022-08-29T04:35:34.2590  STEP7.2: Name   : "WSTAP_DBSERVER1_2022-08-29T04-35-17-2464850-04-00.zip"
`I 2022-08-29T04:35:34.2671  STEP7.2: Size   : 9,843 [KB]
`I 2022-08-29T04:35:34.2671  STEP7.2: Time   : 2022-08-29T04:35:33


Here is an example output of diag.log in a good case in
STANDALONE mode. Must Gather uses Compress-Archive cmdlet and generates a zip file. You can see the exact zip file location with the file size and the time generated.

`I 2022-08-24T03:20:40.5564  STEP7.1: === STEP 7.1 : Zipping diagnostic files.
`I 2022-08-24T03:20:40.5564  STEP7.1: ZipSourceDir : "C:\tmp\diag"
`I 2022-08-24T03:20:40.5564  STEP7.1: ZipTargetDir : "C:\tmp\zip"
`I 2022-08-24T03:20:40.5564  STEP7.1: Creating C:\tmp\diag\ZipSource.dir.txt.
`I 2022-08-24T03:20:40.5720  STEP7.1: Printing the list of files using 'Get-ChildItem' to ZipSource.dir.txt.
`I 2022-08-24T03:20:40.9110  STEP7.1: Mode: Running in STANDALONE mode.
`I 2022-08-24T03:20:40.9110  STEP7.1: PowerShell version is System.Collections.Hashtable.PSVersion.Major. Use Compress-Archive command to create zip file.
`I 2022-08-24T03:20:40.9267  STEP7.1: Previous ZIP and BAK files are existing in "C:\tmp\zip". Deleting the BAK file.
`I 2022-08-24T03:20:40.9425  STEP7.1: Renaming existing ZIP file in "C:\tmp\zip".
`I 2022-08-24T03:20:40.9893  STEP7.1: "GRD_WIN_DIAG_compress-archive_failed.txt" file doesn't exist in "C:\tmp\zip".
`I 2022-08-24T03:20:40.9893  STEP7.1: Zip file name is defined as "GRD_WIN_DIAG_2022-08-24T03-20-40-04.zip".
`I 2022-08-24T03:20:41.0049  STEP7.1: Creating Zip target folder "C:\tmp\zip".
`I 2022-08-24T03:20:41.0049  STEP7.1: Copying files in "C:\tmp\diag" to temp folder "C:\Users\user1\AppData\Local\Temp\diag" before zipping.
`I 2022-08-24T03:20:44.9937  STEP7.1: Creating "GRD_WIN_DIAG_2022-08-24T03-20-40-04.zip" using compress-archive command.
`I 2022-08-24T03:20:59.9584  STEP7.1: ZIP file was created at  "C:\tmp\zip\GRD_WIN_DIAG_2022-08-24T03-20-40-04.zip".
`I 2022-08-24T03:20:59.9584  STEP7.1: Deleting files from the temp folder "C:\Users\user1\AppData\Local\Temp\diag".
`I 2022-08-24T03:21:00.4396  STEP7.2: === STEP 7.2 : Checking the zip file and the size.
`I 2022-08-24T03:21:00.4856  STEP7.2: ZIP file was generated.
`I 2022-08-24T03:21:00.4856  STEP7.2: Folder : "C:\tmp\zip"
`I 2022-08-24T03:21:00.4856  STEP7.2: Name   : "GRD_WIN_DIAG_2022-08-24T03-20-40-04.zip"
`I 2022-08-24T03:21:00.4856  STEP7.2: Size   : 1,059 [KB]
`I 2022-08-24T03:21:00.4856  STEP7.2: Time   : 2022-08-24T03:20:59

 

If you don't see the above messages in diag.log and no zip file is created after you run Must Gather on DB server, please take the following actions.

7.1.1 Make sure to run diag.bat with Administrator privilege

If you run Must Gather on non-elevated Windows Command (non Administrator), Must Gather will fail with the following error message.

C:\Program Files\IBM\Windows S-TAP\Bin>diag.bat
Guardium Windows Agent Must Gather V3 - PowerShell version
DIAG VERSION V3.0.11 (2022/08/27)

Running with no Administrator role. The current user is "Domain1\user1". Exiting.

C:\Program Files\IBM\Windows S-TAP\Bin>

 

Solution: Open Windows Command Prompt as Administrator and run Must Gather.

7.1.2 Check if the path to powershell.exe is defined in PATH environment variable

The diag.bat calls powershell.exe to run diag.ps1. It should work because the path to powershell.exe (i.e. %SystemRoot%\System32\WindowsPowerShell\v1.0\) is defined in PATH environment variable by default. If you remove the path, diag.bat will fail to execute powershell.exe and will see the following error. In this case, no zip file will be generated.

C:\Windows\system32>cd \tmp

C:\tmp>diag.bat
'powershell' is not recognized as an internal or external command,
operable program or batch file.

C:\tmp>powershell
'powershell' is not recognized as an internal or external command,
operable program or batch file.

C:\tmp>set PATH | findstr WindowsPowerShell

C:\tmp>

 

Solution: Add the path to powershell.exe (e.g. C:\Windows\System32\WindowsPowerShell\v1.0\) in PATH environment variable.

NOTE: It should be defined by default. You need to add the path only if you removed it by yourself. The following output is the default (and expected) behavior.

C:\Windows\system32>set | findstr WindowsPowerShell
Path=...;C:\Windows\System32\WindowsPowerShell\v1.0\;...

C:\Windows\system32>

 

7.1.3 Check if the PowerShell version is 5.1 or newer

In STANDALONE mode, the diag.ps1 uses Compress-Archive cmdlet, which is supported in PowerShell 5.1 and newer. 

Unfortunately, Windows Server 2012 and 2012 R2 don't support this cmdlet by default, because the default PowerShell version is lower than 5.1 on these servers.

C:\Windows\system32>wmic os get caption
Caption
Microsoft Windows Server 2012 R2 Standard


C:\Windows\system32>powershell -c "Get-Host | Select-Object Version"

Version
-------
4.0



C:\Windows\system32>


So, when you run diag.bat (or diag.ps1) in STANDALONE mode on Windows Server 2012 / 2012 R2, you will see the following message:

C:\tmp>diag.bat
Guardium Windows Agent Must Gather V3 - PowerShell version
DIAG VERSION V3.0.11 (2022/08/27)

Running with Administrator role.

Diag Log   : C:\tmp\diag\diag.log
Start Time : 2022-08-28T22:37:13
End Time   : 2022-08-28T22:39:56

ZIP file was not created because it's STANDALONE mode and the powershell versio
n (4) doesn't support Compress-Archive command.
All files are gathered to "C:\tmp\diag\" folder. See "C:\tmp\diag\diag.log" for
 details.

C:\tmp>



Solution: Apply one of the following actions.

  • Create a zip file manually. When this happens, Must Gather works in KEEP mode and doesn't clean up files from ZIP Source directory, which is the diag folder under the folder where diag.bat and diag.ps1 are located (e.g. C:\tmp\diag). 
  • Download Microsoft Windows Management Framework 5.1 from Microsoft web site and install it. Then, Windows PowerShell version will be upgraded to 5.1 and Compress-Archive cmdlet will work.
  • Install Windows S-TAP and use S-TAP mode instead of STANDALONE mode. In S-TAP mode, Must Gather will use ExternalZip.exe instead of Compress-Archive cmdlet. 

7.1.4 If you see any other issues

Read diag.log to know what's happening, or open a support ticket so that Guardium support will help you to resolve the issue.

7.2 When you run Must Gather from Guardium GUI

When you run Must Gather from Guardium GUI and the output zip file doesn't seem to be uploaded to the appliance, make sure that S-TAP received the request, executed diag.bat, and uploaded the output to the appliance. See below for details.

7.2.1 Check if Windows S-TAP received a request to run diagnostics, and if Must Gather started

If you're using Windows S-TAP V8 protocol, you should see the following message in Stap.ctl.

`I 08/29/2022 04:29:48.488 Snapshot: Starting debug snapshot pass for DBSERVER1 targeting COLLECTOR1
`I 08/29/2022 04:35:35.515 Snapshot: Ending debug snapshot pass


Also, the start time of diag.log should be slightly after the time of "Starting debug snapshot ...." in Stap.ctl.

Guardium Windows Agent Must Gather V3 - PowerShell version
DIAG VERSION V3.0.11 (2022/08/27)

`I 2022-08-29T04:30:15.5856  STEP1: === STEP 1 : Preparation
`I 2022-08-29T04:30:15.5856  STEP1.1: === STEP 1.1 : Starting up
`I 2022-08-29T04:30:15.5856  STEP1.1: Guardium Windows Agent Must Gather started.
...
`I 2022-08-29T04:35:34.7799  STEP8: === STEP 8 : Must Gather completed. Check "C:\Program Files\IBM\Windows S-TAP\Bin\diag\diag.log" for details.

If you're using Windows S-TAP V7 protocol, you won't see the above Stap.ctl message, and  Windows S-TAP debug log doesn't generate any message for Must Gather. Please just focus on diag.log.

If you don't see any update in diag.log after you run Must Gather from Guardium GUI, please open a support ticket. We may ask for application debug log on the collector and tcpdump on the DB server to go further investigation.

7.2.2 Check if Must Gather completed successfully and generated a zip file

Refer to the section "7.1 When you run Must Gather on DB server" for details.

Note that Must Gather always runs as S-TAP mode when you run it from Guardium GUI, so it always uses ExternalZip.exe and it should always generate a zip file on any supported Windows OS.

 

7.2.3 Check if the zip file was uploaded to the appliance

Must Gather will upload the zip file to the appliance when the file size is less than 100MB and UPLOAD_FEATURE in not 0.

7.2.3.1 Check the zip file size

Open diag.log and check the size. You can also check the zip file directly in the DB server.

`I 2022-08-29T04:35:06.5776  STEP7.1: Calling "ExternalZip.exe -z".
`I 2022-08-29T04:35:34.2087  STEP7.2: === STEP 7.2 : Checking the zip file and the size.
`I 2022-08-29T04:35:34.2590  STEP7.2: ZIP file was generated.
`I 2022-08-29T04:35:34.2590  STEP7.2: Folder : "C:\Program Files\IBM\Windows S-TAP\Bin\zipTmp"
`I 2022-08-29T04:35:34.2590  STEP7.2: Name   : "WSTAP_DBSERVER1_2022-08-29T04-35-17-2464850-04-00.zip"
`I 2022-08-29T04:35:34.2671  STEP7.2: Size   : 9,843 [KB]
`I 2022-08-29T04:35:34.2671  STEP7.2: Time   : 2022-08-29T04:35:33

 

7.2.3.2 Check the value of UPLOAD_FEATURE in Guard_Tap.ini

Open Guard_Tap.ini and check the value of UPLOAD_FEATURE.

Guard_Tap.ini Default value Description
UPLOAD_FEATURE 1

Controls uploading of all log files from Program Files\IBM\Windows S-TAP\Logs on to the collector and/or central manager. Valid values:

0: No automatic upload.
1: Upload files to the collector and the central manager.
2: Upload files to the collector even if a central manager is available. For more information, see Windows: Upload dump files from the S-TAP to the collector and central manager.

Ref) https://www.ibm.com/docs/en/guardium/11.5?topic=parameters-protocol-7-general 

If the zip file size is less than 100MB and UPLOAD_FEATURE is not 0, you should see the following message in Stap.ctl (in both V7 protocol and V8 protocol).

`I 08/29/2022 04:46:54.607   Upload: Transferred file C:\Program Files\IBM\Windows S-TAP\Bin\..\LOGS\WSTAP_DBSERVER1_2022-08-29T04-45-40-2039813-04-00.zip to appliance COLLECTOR1

 

7.2.3.3 Logon to the target appliance and check if the zip is uploaded.

The target appliance hostname is written in the above Stap.ctl message. For example, if the zip file name is WSTAP_DBSERVER1_2022-08-29T04-45-40-2039813-04-00.zip, DBSERVER1 is the target appliance host name.

Logon to Guardium GUI, and navigate to Manage > Maintenance > Support Information Results.

 

If you see the zip file in the GUI, everything is fine.

If you have any problems, please open a support ticket so that Guardium support will help you to resolve the issue.

What's next?

Now you know everything about Must Gather V3.0. It's time to use the tool!

Questions?

If you have any questions, please feel free to comment on this article. You can also ask questions to IBM Security Guardium discussion in IBM Security Community or open a technical support ticket.

Reference

0 comments
6 views

Permalink