IBM Security QRadar

 View Only

Napatech3 - "The New Frontier"

By DOGA TAV posted Fri January 20, 2023 01:28 PM

  

Napatech Software Suite, which is the 3rd generation driver for Napatech adapters (3GD for short) is the Napatech's new stack that was released along with migration to RHEL7 in QRadar ("napatech" service is no longer supported in RHEL7). It is a broad topic but I will discuss things in the following headers and try to provide bite-size information.

Basic Info:

Following our older stack convention, pretty much everything there is to do lives under /opt/napatech3/. Below is the directory structure: (the non-starred ones are optional rpms and they are not included in default install). 

napatech3
      |____bin (*)(all functional stuff)
      |____config (*) (configuration)
      |____doc
      |____driver (*) (driver)
      |____examples
      |____images (*) (card images)
      |____include
      |____lib (*) (NT libraries)
 
In addition, the images folder contents are as follows:
 
napatech3
  |____images
    |____nt20e2_fpga_200-9220-50-05.tar.gz
    |____nt20e_fpga_200-9014-42-12.tar.gz
    |____nt40e3-4-ptp_fpga_200-9232-55-05.tar.gz
    |____nt40e3-4-ptp_fpga_200-9234-55-01.tar.gz
    |____nt4estd_fpga_200-9017-42-10.tar.gz

In case you are wondering, the bold numbers denotes the type of the card. At the time of writing, we have 5 types of cards offered in QRadar Portfolio.

  • nt20e2_fpga_200-9220-50-05.tar.gz
  • nt20e_fpga_200-9014-42-12.tar.gz
  • nt40e3-4-ptp_fpga_200-9232-55-05.tar.gz
  • nt40e3-4-ptp_fpga_200-9234-55-01.tar.gz
  • nt4estd_fpga_200-9017-42-10.tar.gz

Start / Stop / Status napatech3 service:

~]# systemctl start napatech3
 
~]# systemctl stop napatech3
 
~]# systemctl status napatech3
● napatech3.service - Napatech3 3GD Service
   Loaded: loaded (/usr/lib/systemd/system/napatech3.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2017-03-22 22:03:13 ADT; 4 days ago
  Process: 901 ExecStop=/opt/napatech3/bin/napatech3 stop (code=exited, status=0/SUCCESS)
  Process: 1319 ExecStart=/opt/napatech3/bin/napatech3 start (code=exited, status=0/SUCCESS)
  Process: 1218 ExecStartPre=/opt/napatech3/bin/napatech3 initialize (code=exited, status=0/SUCCESS)
  Process: 1162 ExecStartPre=/opt/napatech3/bin/test_fpgaupdate (code=exited, status=0/SUCCESS)
 Main PID: 1506 (ntservice)
   Memory: 26.8M
   CGroup: /system.slice/napatech3.service
           └─1506 /opt/napatech3/bin/ntservice -d -n /opt/napatech3/config/qni_ntconfig_6300.ntpl -f /opt/napatech3/config/ntservice.ini
 
Mar 26 23:00:26 m5pl039.q1labs.lab ntservice[1506]: Adapter 0 timestamp clock is now In-Sync with reference
Mar 26 23:10:02 m5pl039.q1labs.lab ntservice[1506]: Adapter 0 timestamp clock got Out-of-Sync with reference
Mar 26 23:10:02 m5pl039.q1labs.lab ntservice[1506]: Adapter 0 timestamp clock is hard reset to reference
Mar 26 23:10:25 m5pl039.q1labs.lab ntservice[1506]: Adapter 0 timestamp clock is now In-Sync with reference
Mar 26 23:20:03 m5pl039.q1labs.lab ntservice[1506]: Adapter 0 timestamp clock got Out-of-Sync with reference
Mar 26 23:20:04 m5pl039.q1labs.lab ntservice[1506]: Adapter 0 timestamp clock is hard reset to reference
Mar 26 23:20:28 m5pl039.q1labs.lab ntservice[1506]: Adapter 0 timestamp clock is now In-Sync with reference
Mar 26 23:30:03 m5pl039.q1labs.lab ntservice[1506]: Adapter 0 timestamp clock got Out-of-Sync with reference
Mar 26 23:30:03 m5pl039.q1labs.lab ntservice[1506]: Adapter 0 timestamp clock is hard reset to reference
Mar 26 23:30:26 m5pl039.q1labs.lab ntservice[1506]: Adapter 0 timestamp clock is now In-Sync with reference


Get card details:

~]# /opt/napatech3/bin/adapterinfo
adapterinfo (v. 2.12.1.1-0a8051)                                                                                                                                                                                    
==============================================================================                                                                                                                                      
System: 2.12.1.1                                                                                                                                                                                                    
------------------------------------------------------------------------------                                                                                                                                      
Adapters:  1                                                                                                                                                                                                        
Ports:     4                                                                                                                                                                                                        
==============================================================================                                                                                                                                      
Adapter: 0 - NT40E3-4-PTP-ANL Analysis Network Adapter PCIe Gen3 4x1Gb SFP                                                                                                                                          
------------------------------------------------------------------------------                                                                                                                                      
Bus ID:             0000:0b:00.0                                                                                                                                                                                    
PCI ID:             18F4:0145                                                                                                                                                                                       
FPGA ID:            200-9234-55-01-00                                                                                                                                                                               
Feature level:      N-ANL3A                                                                                                                                                                                         
Profile:            CaptureReplay                                                                                                                                                                                   
Supported profiles: Capture Inline CaptureReplay TrafficGen                                                                                                                                                         
                                                                                                                                                                                                                      
AVR version:  3.4b                                                                                                                                                                                                  
                                                                                                                                                                                                                      
Ports:            4                                                                                                                                                                                                 
Port Range:       0 - 3                                                                                                                                                                                             
Descriptor:       Standard                                                                                                                                                                                          
Timestamp:        Native Unix                                                                                                                                                                                       
TimestampMethod:  EOF (End of Frame)                                                                                                                                                                                
 
RX host buffers:   4
TX host buffers:   2
 
Port #0 MAC : 00.0D.E9.05.8B.2F
Port #1 MAC : 00.0D.E9.05.8B.30
Port #2 MAC : 00.0D.E9.05.8B.31
Port #3 MAC : 00.0D.E9.05.8B.32
 
SDRAM size  : 4GB
 
Bus type                : PCI Express 3
Link width supported    : x8
Link width negotiated   : x8
Link speed              : 8.0 GT/s
Max payload supported   : 512
Max payload negotiated  : 256
Max expected throughput : 56691 Mbps
==============================================================================

Firmware check:

~]# /opt/napatech3/bin/productinfo
 
productinfo (v. 2.12.1.1-0a8051)
==============================================================================
System: 2.12.1.1
------------------------------------------------------------------------------
Adapters:  1
==============================================================================
Adapter 0  NT40E3-4-PTP-ANL Analysis Network Adapter PCIe Gen3 4x1Gb SFP
------------------------------------------------------------------------------
P/N                      : 801-0101-01-10
Serial No                : 0000-231948
PBA                      : 073-011901-07
FPGA flash image #0      : 200-9234-55-01
FPGA flash image #1      : 200-9234-55-01
AVR                      : 3.4
------------------------------------------------------------------------------

Firmware update (Automatic):

In this case, we do not need much of a user intervention, upon the start of the napatech3 service (also configured to start on boot), we run the fpgaupdate script which checks the FW and compares the image file with the corresponding one under the "images" folder and starts the process so before anything happens we compare the FW version. Note: In older cards, the box may require an OS reboot to update both banks and switch the running bank to the new version.

Firmware update (Manual):

This case requires some validation and manual steps and ideally, one should only use this for downgrading the card to a specific FW. Napatech3 stack should perform its FW update automatically as discussed earlier.

The FW upgrade/downgrade process is simple:

  • First, pick the correct image which corresponds to the card. Stop any service which would use napatech3 (or napatech for earlier QRadar releases): hostcontext, napatech3, qflow, napatech etc.
  • Program (-p) one of the banks with the img file. Then switch(-s) to it, after verifying with  "/opt/napatech3/bin/imgctrl  -q"  output, if all looks good, confirm it. Finally repeat the process for the other bank as well. In older card models such as NT20E one may  need to reboot the box after updating both of the banks. As a final step, do not forget to take a look at the "/opt/napatech3/bin/imgctrl  -q" output to make sure that all the FPGA banks (banks 0,1,running) are updated with the correct version of the FW.

Below is an example for updating NT20E card.

[root@pl22 1310 - NT20E]# systemctl stop hostcontext
[root@pl22 1310 - NT20E]# systemctl status qflow
[root@pl22 1310 - NT20E]# systemctl stop napatech3
[root@pl22 1310 - NT20E]# /opt/napatech3/bin/imgctrl -p -a0 --ntimg 200-9014-42-08-00-120604-1337_0.img --ini tag
imgctrl v3.4.4.29558 (Jul  8 2016 12:34:13) (Logfile: /tmp/imgctrl.log)
Scanning for Napatech adapters..
Writing FPGA bank 0
  Erasing................... 100% done
  Writing................... 100% done
  Verifying................. 100% done
Done writing FPGA bank 0
Note: You can inspect the upload result through the Bank0 Contents
      section using the --query mode. To try the new image, remember
      to switch bank using the --switch mode. See --help for more information.
 
[root@pl22 1310 - NT20E]# /opt/napatech3/bin/imgctrl -s -a0
imgctrl v3.4.4.29558 (Jul  8 2016 12:34:13) (Logfile: /tmp/imgctrl.log)
Scanning for Napatech adapters..
Switching image on adapter 0:
  OK. Image switched. Remember to issue --confirm or --discard to accept/discard the change. Inquire the adapter via --query
 
[root@pl22 1310 - NT20E]# /opt/napatech3/bin/imgctrl -s -a0 --confirm
imgctrl v3.4.4.29558 (Jul  8 2016 12:34:13) (Logfile: /tmp/imgctrl.log)
Scanning for Napatech adapters..
Applying current running image as safeboot image..
  OK. Adapter PrimaryImage has been updated to reflect current running image.
 
[root@pl22 1310 - NT20E]# /opt/napatech3/bin/imgctrl -p -a0 --ntimg 200-9014-42-08-00-120604-1337_0.img --ini tag
imgctrl v3.4.4.29558 (Jul  8 2016 12:34:13) (Logfile: /tmp/imgctrl.log)
Scanning for Napatech adapters..
Writing FPGA bank 1
  Erasing................... 100% done
  Writing................... 100% done
  Verifying................. 100% done
Done writing FPGA bank 1
Note: You can inspect the upload result through the Bank1 Contents
      section using the --query mode. To try the new image, remember
      to switch bank using the --switch mode. See --help for more information.

Tools

monitoring

The Monitoring tool is an interactive tool used to monitor port statistics, sensors, time synchronization, stream statistics etc.

In the Port Stat screen (shortcut "P"), P column lists the ports on the card, Type column lists the type of the connection, Link column lists the ports which have SFP+ connectors, Rx and Tx lists the receive and transmit rate, Max lists max packet size and Temp lists the temperature for the card.

Note: In production environment, you are not likely to see connection between port 3 and 4 but in this example there is. Port 0 is receiving traffic, the last port of the first card and the first port of the second card has a 10G external connection between them. You can see that data is received from port 0, gets replicated to port 3 via port 4 and it is received by the second card. The idea is, this will act as a backup configuration and there is a separate configuration file delivered with napatech3-base rpm if the customers experience some issues with the internal cabling (broken, faulty etc). While waiting for a replacement, they can keep using their QNI appliance in the meantime.

Hit "X" for time sync, it should show "OsTime"



==Troubleshooting== (Disclaimer: When you hit an issue please go through QRadar Support and proper channels, these cases showcase how we mitigate in the Lab and Development side).

In case of full QNI appliance 6200, if internal connection fails what can we do for an interim fix?

Suppose. the internal connection failed due to the cable itself, we ordered a replacement. In the meantime, we put a config file as a workaround so that the customer system would not need to be down in the meantime. We need to ask customer to add an external connection between the last port of the first card and the first port of the second card (10G connection with SFPs and fiber cable). After that step: 

Edit ntservice.ini to change the master, slave BondingType structure. Note that we need to apply this change for both of the cards defined in the ini file.

[Adapter0]
BondingType = Separate
 
[Adapter1]
BondingType = Separate

If you look at /opt/napatech3/config/ there is a qni_config.ntpl (default) and qni_ntconfig_EXTCON.ntpl. You just need to edit /opt/napatech3/bin/napatech3 at the very top change the QNI_NTPL global variable to point the extcon ntpl.

QNI_NTPL=$NAPATECH3_ROOT/config/qni_ntconfig_EXTCON.ntpl

Then issue a "systemctl restart napatech3". If everything is correct, the system will come up and /opt/napatech3/bin/monitoring will show the TX activity on the last port of the first card ( port3) to the first port of the second card (port4). Be sure to check the data flow to decapper and out of QNI box to verify the expected rate is send to the rest of the QRadar Pipeline. Note: If you delete ntservice.ini the napatech3 service will create it with default (internal connection) settings.


The issue might not be a problem anymore but sharing just in case:

If napatech3 fails to load drivers and can not start

In /opt/napatech3/bin/ntload.sh, comment out the " build_driver " function calls (line 172,177 at the time of writing) as we are going to push the precompiled driver in our rpms. The reason for this to happen is, the kernel version which the driver is built and the kernel version in the OS are not the same. This would be a work around and due to QRadar production code limitations, we can not build driver on a production system. We are in touch with Napatech for them to parameterize this function.

Comment out the build_driver calls at the very end of the script (at the time of writing)
171
172
173
174
175
176
177
178
179
180
${SCRIPTDIR}/ntunload.sh silent
 # build_driver ${DRIVER} ""
        load_driver ${DRIVER}
        create_device "nt3gd"
 
if[ $PLATFORM = "linux" ]; then
 # build_driver ${DRIVER_NETDEV} "_netdev"
        load_driver ${DRIVER_NETDEV}
        create_device "nt3gd_netdev"
fi

Capture data to verify that traffic is coming in to the card

  • Capture 100MB of traffic from port 0
/opt/napatech3/bin/capture -f src.cap -p 0 -b 100M
  • Convert to PCAP
/opt/napatech3/bin/capfileconvert -i src.cap -o dest.cap --outputformat=pcap
  •  Verify the contents (first 10 records)
tcpdump -r  dest.cap  -c10
0 comments
22 views

Permalink