IBM Security QRadar SOAR

 View Only
  • 1.  Data Feeder - records not updated due to ValueError

    Posted Thu November 18, 2021 09:13 AM
    Edited by Francois Chometton Thu November 18, 2021 09:28 AM
    Hello,

    I have a lot of records that are not inserted/updated in my PostgreSQL 12.9 DB using ODBC DataFeeder, and I get these errors in the log :

    2021-11-18 14:43:11,901 WARNING [type_info] field_id no longer exists: 511


    2021-11-18 14:43:11,901 ERROR [feed_ingest] Failure in update to ODBCFeedDestination 

    2021-11-18 14:43:11,901 ERROR [feed_ingest] Traceback Traceback (most recent call last):

      File "/home/integration/.local/lib/python3.6/site-packages/rc_data_feed/lib/type_info.py", line 220, in flatten

        values = self._get_field_values(payload, all_fields, translate_func)

      File "/home/integration/.local/lib/python3.6/site-packages/rc_data_feed/lib/type_info.py", line 238, in _get_field_values

        raw_value = self._get_raw_value_for_field(payload, field)

      File "/home/integration/.local/lib/python3.6/site-packages/rc_data_feed/lib/type_info.py", line 341, in _get_raw_value_for_field

        raise ValueError()

    ValueError

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):

      File "/home/integration/.local/lib/python3.6/site-packages/rc_data_feed/components/feed_ingest.py", line 113, in send_data

        feed_output.send_data(context, payload)

      File "/home/integration/.local/lib/python3.6/site-packages/data_feeder_plugins/sqllib/sql_feed_base.py", line 175, in send_data

        TypeInfo.translate_value))

      File "/home/integration/.local/lib/python3.6/site-packages/rc_data_feed/lib/type_info.py", line 224, in flatten

        values = self._get_field_values(payload, all_fields, translate_func)

      File "/home/integration/.local/lib/python3.6/site-packages/rc_data_feed/lib/type_info.py", line 238, in _get_field_values

        raw_value = self._get_raw_value_for_field(payload, field)

      File "/home/integration/.local/lib/python3.6/site-packages/rc_data_feed/lib/type_info.py", line 341, in _get_raw_value_for_field

        raise ValueError()

    ValueError

    I also tried FileFeed plugin, it is generating the same errors and related files are not generated.

    The field_id that does not exist are 511, 536, 537, 538, 539, 540, 541, 542, 546, 547


    There are also tons of such Warning messages in the log, but seems to have no impact :
    2021-11-18 14:57:34,558 WARNING [sql_dialect] Unable to find a mapping for field type: text

    2021-11-18 14:57:34,558 WARNING [sql_dialect] Unable to find a mapping for field type: boolean


    I had a look at the source code, the comment is interesting but I do not understand what to do to solve the issue

        def _get_raw_value_for_field(self, payload, field):

            """

            this function will raise ValueError if a schema field is not found in the payload. This is used to trap changes in the

            schema not reflected in the cached schema


    I use the latest Data Feed plugins (resilient-circuits list):

    The following packages and components are installed:

    rc-data-feed-plugin-odbcfeed==1.0.5:

    (Package does not define any components)

    rc-data-feed==2.1.0:

    FeedComponent

    SyncIncidentsFunctionComponent


    I also get this Warning when running resilient-circuits customize :

    WARN: The configuration version detected is '30.0.3438' and the product platform version is '41.0.6783'. It is not recommended for these to be more than 1 major version number apart.

    Is there a configuration file to update ? 

    Thanks in advance for your help.



    ------------------------------
    Francois Chometton
    ------------------------------


  • 2.  RE: Data Feeder - records not updated due to ValueError

    Posted Fri November 19, 2021 08:39 AM
    Hi Francois,

    In no particular order,  the second issue regarding the configuration version detected, this warning is informational and has no effect on the operation of the integration.

    Regarding the message: "Unable to find a mapping for field type: text/bool", we will publish a fix so this warning does not appear.

    Finally, for field Ids which no longer exist, a fix will be published to allow synchronization of the record. Look for v2.1.1 in December, 2021.

    Regards,
    Mark

    ------------------------------
    Mark Scherfling
    ------------------------------



  • 3.  RE: Data Feeder - records not updated due to ValueError

    Posted Wed January 12, 2022 05:47 AM
    Hi @Mark Scherfling, any news regarding publication of Data Feeder v2.1.1 on App Exchange ?


    ------------------------------
    Francois Chometton
    ------------------------------



  • 4.  RE: Data Feeder - records not updated due to ValueError

    Posted Fri February 11, 2022 08:48 AM
    We release 2.2.1 on January 28, 2022: https://exchange.xforce.ibmcloud.com/hub/extension/86511b76452a58f4cb3bccfd6b01b842

    ------------------------------
    Mark Scherfling
    ------------------------------