Content Management and Capture

 View Only
Expand all | Collapse all

Datacap RuleRunner aborts after successful Task

  • 1.  Datacap RuleRunner aborts after successful Task

    Posted Tue September 26, 2023 05:21 AM
    Edited by Julian Fiegenbaum Tue September 26, 2023 09:57 AM

    Hello community,

    we currently have a problem with aborting Batches after migrating from Datacap 9.1.6 to 9.1.9 iFix 03. After successfully finishing a task, the batch status gets set to 'aborted' instead of 'finished'. This reliably happens in two out of our four applications at specific tasks. The only commonality shared by these tasks i can see is the usage of Abby OCR, but that is used in the other applications as well, which work just fine.

    The last action of the last rule of the last ruleset gets executed.Then the batch is aborted. Task status and pageFile.xml are not updated and the last buffered content for the rrs-log is not written to disk. So if there was an error message, it does get lost.

    EDIT: Thanks to Varun the full log is available. Just does not show any errors/exceptions.

    There are no error/exception to be found in rrs batch-logs, RuleRunner thread logs or RuleRunner service logs. It is not IO28998 - the 64bit dlls are not used, which can be affirmed in the logs.

    Have you ever had something like this? Or any ideas how to analyse further?

    Many thanks and best regards,

    Julian

    End of Log after last executed action. Somwhere here the process fails.

    15:39:36.772 (0)	t:1A18 p:EDA8C0                /call
    15:39:36.772 (0)	t:1A18 p:EDA8C0                action returned true
    15:39:36.772 (0)	t:1A18 p:EDA8C0                execute statement On Action True
    15:39:36.772 (0)	t:1A18 p:EDA8C0                   executing code:
    15:39:36.773 (0)	t:1A18 p:EDA8C0                   Call OnActionSuccess()
    15:39:36.773 (0)	t:1A18 p:EDA8C0  Action esec='0.03125'
    15:39:36.773 (0)	t:1A18 p:EDA8C0                /execute statement On Action True
    15:39:36.773 (0)	t:1A18 p:EDA8C0             /action
    15:39:36.774 (0)	t:1A18 p:EDA8C0             func result: "true"
    15:39:36.774 (0)	t:1A18 p:EDA8C0             execute statement On Function True
    15:39:36.774 (0)	t:1A18 p:EDA8C0                executing code:
    15:39:36.774 (0)	t:1A18 p:EDA8C0                Call OnFunctionEnd()
    15:39:36.775 (0)	t:1A18 p:EDA8C0  Function esec='0.12500'
    15:39:36.775 (0)	t:1A18 p:EDA8C0             /execute statement On Function True
    15:39:36.775 (0)	t:1A18 p:EDA8C0          /func
    15:39:36.775 (0)	t:1A18 p:EDA8C0          execute statement On Rule True
    15:39:36.776 (0)	t:1A18 p:EDA8C0             executing code:
    15:39:36.776 (0)	t:1A18 p:EDA8C0             Call StatCounter():Call OnRuleEnd():Call RRLogicOnRuleEnd()
    15:39:36.777 (0)	t:1A18 p:EDA8C0  DocCount 0,  PageCount 6
    15:39:36.778 (0)	t:1A18 p:EDA8C0  rrRuleEnd OVERRIDEABLE:True
    15:39:36.778 (0)	t:1A18 p:EDA8C0  Rule esec='0.14063'
    15:39:36.791 (15)	t:1A18 p:EDA8C0 nDefaultRtn default: 1
    15:39:36.792 (0)	t:1A18 p:EDA8C0 cOnRuleEnd: OVERRIDEABLE:True
    15:39:36.792 (0)	t:1A18 p:EDA8C0          /execute statement On Rule True
    15:39:36.792 (0)	t:1A18 p:EDA8C0          execute statement Set CurrentObj to Nothing
    15:39:36.793 (0)	t:1A18 p:EDA8C0             executing code:
    15:39:36.793 (0)	t:1A18 p:EDA8C0             Call set_CDCOnothing()
    15:39:36.793 (0)	t:1A18 p:EDA8C0          /execute statement Set CurrentObj to Nothing
    15:39:36.794 (0)	t:1A18 p:EDA8C0          rule result: "true"
    15:39:36.794 (0)	t:1A18 p:EDA8C0       /rule
    15:39:36.794 (0)	t:1A18 p:EDA8C0    /dco
    15:39:36.794 (0)	t:1A18 p:EDA8C0    dco close tag="P" id="TM000006" type="Other" progress="2"
    15:39:36.795 (0)	t:1A18 p:EDA8C0       Looked up map for current dco: dco tag:[P]; dco type:[Other]; base:[];
    did not match [P.Other]
    did not match [.Other]
    looked up DCO object type map
    map for DCO object type [2] is blank
    *(star) was not specified
    did not find matches
    15:39:36.795 (0)	t:1A18 p:EDA8C0       skip: current DCO has no associated rules from this ruleset
    15:39:36.795 (0)	t:1A18 p:EDA8C0    /dco
    15:39:36.795 (0)	t:1A18 p:EDA8C0    dco close tag="B" id="20230926.000004" type="AutoCapture" progress="2"
    15:39:36.796 (0)	t:1A18 p:EDA8C0       Looked up map for current dco: dco tag:[B]; dco type:[AutoCapture]; base:[];
    did not match [B.AutoCapture]
    did not match [.AutoCapture]
    looked up DCO object type map
    map for DCO object type [0] is blank
    *(star) was not specified
    did not find matches
    15:39:36.796 (0)	t:1A18 p:EDA8C0       skip: current DCO has no associated rules from this ruleset
    15:39:36.796 (0)	t:1A18 p:EDA8C0    /dco
    15:39:36.796 (0)	t:1A18 p:EDA8C0    stop: back to the top level and finished it
    15:39:36.797 (0)	t:1A18 p:EDA8C0    execute statement On Ruleset End
    15:39:36.797 (0)	t:1A18 p:EDA8C0       executing code:
    15:39:36.797 (0)	t:1A18 p:EDA8C0       Call OnRuleSetEnd()
    15:39:36.798 (0)	t:1A18 p:EDA8C0  RuleSet finished with TaskStatus '2'
    15:39:36.798 (16)	t:1A18 p:EDA8C0  Total Ruleset Action   esec:'0.95313'
    15:39:36.798 (0)	t:1A18 p:EDA8C0  Total Ruleset Function esec:'1.01563'
    15:39:36.799 (0)	t:1A18 p:EDA8C0  Total Ruleset Rule     esec:'1.04688'
    15:39:36.799 (0)	t:1A18 p:EDA8C0  Total Profile Action   esec:'17.77344'
    15:39:36.799 (0)	t:1A18 p:EDA8C0  Total Profile Function esec:'17.97656'
    15:39:36.800 (0)	t:1A18 p:EDA8C0  Total Profile Rule     esec:'18.10156'
    15:39:36.800 (0)	t:1A18 p:EDA8C0  OnRuleSetEnd Pilot XML:
    <B id="20230926.000004"><P n="BatchDir">\\<REDACTED_PATH_TO_BATCH_DIR>\20230926.000004</P><P n="AppName"></P><P n="Workflow"></P><P n="DCOFile">\\<REDACTED_PATH_TO_BATCH_DIR>\20230926.000004\PDFConversion.xml</P><P n="Operator">admin</P><P n="Station">1.0</P><P n="JobName">CNPartList</P><P n="TaskName">PDFConversion</P><P n="Priority">5</P><P n="PagesInBatch">1</P><P n="DocsInBatch">0</P><P n="ExpectedPages">0</P><P n="ExpectedDocs">0</P><P n="AdjustedPages">0</P><P n="AdjustedDocs">0</P><P n="ProjectPath">\\<REDACTED_PATH_TO_DCO_DIR>\PDFConversion.set.xml</P><X n="SAPID"></X></B>
    
    
    15:39:36.802 (0)	t:1A18 p:EDA8C0    /execute statement On Ruleset End
    15:39:36.802 (0)	t:1A18 p:EDA8C0 /ruleset
    15:39:36.803 (0)	t:1A18 p:EDA8C0 CIMainAlgorithm::Execute::finalize_after_process:
    15:39:36.803 (0)	t:1A18 p:EDA8C0 execute statement On Process End
    15:39:36.803 (0)	t:1A18 p:EDA8C0    executing code:
    15:39:36.803 (0)	t:1A18 p:EDA8C0    Quit()
    15:39:36.813 (0)	t:1A18 p:EDA8C0 /execute statement On Process End
    15:39:36.813 (0)	t:1A18 p:EDA8C0 CIMainAlgorithm::Execute::finalize_after_process:SaveDCO:
    15:39:36.818 (16)	t:1A18 p:EDA8C0 unload script [m0x="2028a44"]
    15:39:36.819 (0)	t:1A18 p:EDA8C0    execute statement (null)
    15:39:36.819 (0)	t:1A18 p:EDA8C0       executing code:
    15:39:36.819 (0)	t:1A18 p:EDA8C0       Call set_CDCOnothing()
    15:39:36.819 (0)	t:1A18 p:EDA8C0    /execute statement (null)
    15:39:36.820 (0)	t:1A18 p:EDA8C0    script released
    15:39:36.821 (0)	t:1A18 p:EDA8C0 /unload
    15:39:36.821 (0)	t:1A18 p:EDA8C0 unload COM m0x="2028a14">
    



    ------------------------------
    Julian Fiegenbaum | ISR Information Products AG | Consultant | Germany
    ------------------------------



  • 2.  RE: Datacap RuleRunner aborts after successful Task

    Posted Tue September 26, 2023 09:04 AM

    Hi Julian,

    Did you tried to use the Flush buffer option for RRS logs to flush out the logs?



    ------------------------------
    Varun Gupta
    ------------------------------



  • 3.  RE: Datacap RuleRunner aborts after successful Task

    Posted Tue September 26, 2023 09:53 AM

    Hi Varun,

    for some reason i was not aware of this functionality in the RRS log. I activated it and now the log gets fully written, which is great - thank you :) I'll edit the original post to reflect that change in information.

    Sadly still no error/exception to be found :(



    ------------------------------
    Julian Fiegenbaum | ISR Information Products AG | Consultant | Germany
    ------------------------------



  • 4.  RE: Datacap RuleRunner aborts after successful Task

    Posted Wed September 27, 2023 06:10 AM

    Hi Julian,

    Without seeing the application and corresponding runtime XML it's difficult to visualise the possible cause of the abort.

    The log seems to suggest that no actions are being executed but perhaps that is due to the log level that has been set?

    I did notice this:

    <P n="DocsInBatch">0</P>

    Which suggests that there are no documents in the batch and I wonder if that is the cause of the abort?

    Are you able to share the PDFConversion.xml?

    Is this reproducible (i.e. does it occur every time you execute this batch via Rulerunner)? If it's intermittent, it will be more difficult to troubleshoot.

    If it is reproducible, does the same batch abort when you run it in Datacap Studio?

    Many thanks,

    Shaun



    ------------------------------
    Shaun McDowall
    ------------------------------



  • 5.  RE: Datacap RuleRunner aborts after successful Task

    Posted Thu September 28, 2023 04:34 AM

    Hi Shaun,

    thanks for your comments! I uploaded the runtime-xml, the rulesets and the complete log. The step is really mostly the start of the process, where a pdf is split into tifs and recognized.

    It is reproducible every time. I'm not quite sure what to make of the DocsInBatch - the PDFConversion.xml correctly shows a properly built structure. Doesn't the runtime-xml get updated after the task is fully processed (at least in this case the task is fully processed, but still the old page file is referenced)? In that case the 0 documents (not yet created) and 1 page (other) makes sense as we are at the start of the job.

    I can not process the task in DatacapStudio as PDFFREDocumentToImage (and the corresponding release) is used and as far as i know (seen multiple client systems) this leads to an error in DatacapStudio.

    Many thanks and regards,

    Julian



    ------------------------------
    Julian Fiegenbaum | ISR Information Products AG | Consultant | Germany
    ------------------------------



  • 6.  RE: Datacap RuleRunner aborts after successful Task

    Posted Thu September 28, 2023 09:37 AM

    Hi Julian

    Thanks for sharing the XML and log files. The PDFConversion.xml file doesn't display, but the other files provide a good overview of the process. You should be able to run this in Datacap Studio, unless there's a Branch operation (see my screenshot from TravelDocs). Branching and splitting of batches was never implemented in Studio. 

    I've worked on a couple of projects recently using DC9.1.9 iFix03 which leverage PDFFREDocumentToImage() and have been able to execute this in Studio with no issues.

    My thinking is that if you're able to run the PDFConversion task profile in the Studio Test tab and it aborts, you should be able to see exactly which action causes the failure.

    Kind regards,

    Shaun



    ------------------------------
    Shaun McDowall
    ------------------------------



  • 7.  RE: Datacap RuleRunner aborts after successful Task

    Posted Thu September 28, 2023 09:40 AM

    Hi Julian,

    Did you checked the rulerunner logs after enabling flush buffer and detailed logging?



    ------------------------------
    Varun Gupta
    ------------------------------



  • 8.  RE: Datacap RuleRunner aborts after successful Task

    Posted Mon October 02, 2023 03:00 AM

    Hi Varun,

    yes i did - See post #5. Thanks for the tip!



    ------------------------------
    Julian Fiegenbaum | ISR Information Products AG | Consultant | Germany
    ------------------------------



  • 9.  RE: Datacap RuleRunner aborts after successful Task

    Posted Mon October 02, 2023 04:52 AM

    Hi Shaun,

    thanks for encouraging me to further investigate the PDFFREDocumentToImage() issue. I figured out, that for some reason Interop.FREngine.dll could not be found by the Convert library. As a temporary workaround i copied it in the rules folder and that fixed that issue.

    Manual stepping now shows me that the application crashes at several, not particularly reasonable points including SetNamePattern("2"). When the studio crashes i at least get a meaningfull entry in the windows event logs:

    Application: DStudio.exe
    Framework Version: v4.0.30319
    Description: The process was terminated due to an internal error in the .NET Runtime at IP 739ED9C2 (738F0000) with exit code 80131506.

    The exit code indicates an ExecutionEngineException. I have been guaranteed (and i asked multiple times) that all kinds of anti virus measures are disabled for the time being, which would be the main suspect for this.

    Given that the issue is 100% reproducible and only affects 2 out of 4 applications all using basically the same patterns (including naming patterns), i am reluctant to suspect something like physical or virtualized RAM or something like that either. Anyone ever experienced something like this or any ideas how to analyse this further?

    As always many thanks and regards,
    Julian



    ------------------------------
    Julian Fiegenbaum | ISR Information Products AG | Consultant | Germany
    ------------------------------



  • 10.  RE: Datacap RuleRunner aborts after successful Task

    Posted Mon October 02, 2023 06:43 AM

    Hi Julian,

    Are all 4 applications running in the same environment?

    If 2 of them are running in a different environment, I suspect that either the installation in the environment that has the issue has become corrupt or that there are some components missing. If all 4 of them are running in the same environment, there could be subtle differences between them which surface the issue for 2 but not for all 4.

    I've seen this type of issue previously, particularly where the Email and Electronic Documents Connector hadn't been installed:

    https://www.ibm.com/support/pages/convertrrx-missing-custom-install-ibm-datacap-taskmaster-capture

    (brought to mind by the fact that you copied the Interop.FREngine.dll to the rules folder and that resolved the PDFFREDocumentToImage issue).

    I've also had issues trying to repair Datacap 9.1.9 or add missing features when fix packs have been installed :(

    In a couple of cases, I've ended up having to uninstall Datacap completely and reinstall it, selecting and installing the missing components prior to running the Fix Pack installation.

    Does that help?

    Kind regards,

    Shaun



    ------------------------------
    Shaun McDowall
    ------------------------------



  • 11.  RE: Datacap RuleRunner aborts after successful Task

    Posted Thu October 05, 2023 04:46 AM

    Hi Shaun,

    thanks again for your replies and solidarity :)

    Same environment and we actually figured out the subtle difference between the applications. It is the usage of DocumentAnalytics-Actions from the Insight Edition that causes the batch to abort. I have further been able to get the sporadic problems i had with DStudio and now get the same crash after full processing of the task in both DStudio and RulerRunner:

    Application: DStudio.exe (or RRProcessor.exe)
    Framework Version: v4.0.30319
    Description: The process was terminated due to an unhandled exception.
    Exception Info: System.IO.FileNotFoundException
    at DocumentAnalytics.TextAnalysis.Cleanup()
    at DocumentAnalytics.Actions.Finalize()

    Given that DocumentAnalytics shows up in current iFix Patch notes, i would assume there are systems out there, that successfully run DocumentAnalytics under 9.1.9. Maybe someone here knows of some settings i missed?

    Best Regards,
    Julian



    ------------------------------
    Julian Fiegenbaum | ISR Information Products AG | Consultant | Germany
    ------------------------------



  • 12.  RE: Datacap RuleRunner aborts after successful Task

    Posted Thu October 05, 2023 05:23 AM

    Hi Julian,

    I haven't used the DocumentAnalytics actions in DC 9.1.9 yet.

    Are you able to share a screenshot of the ruleset that causes the crash? Or say which action(s) cause it?

    I've noticed this entry in the Release Notes for iFix004 and I wonder whether this is the issue you're facing:

    IO28743 Fix DC919 IFIX004
    Many thanks,
    Shaun


    ------------------------------
    Shaun McDowall
    ------------------------------



  • 13.  RE: Datacap RuleRunner aborts after successful Task

    Posted Thu October 05, 2023 09:26 AM

    Hi Shaun,

    the two Actions in question are CopyAllBlocks() and AnalyzeLayout(). I am still a little unsure if above error message is symptomatic or causal to our problems (dotnet not being able to clean up after crash or not being able to clean up causes the crash).

    I have not found strong candidates in the iFix04 patch notes, but the module has been touched in some way. So yeah - iFix04 would be next in line to try out.

    Maybe I should experiment, if other Actions of that module have similar impact and if other tasks can be affected in the same way first. Either way, i will have a look :)

    Many thanks an best regards,

    Julian



    ------------------------------
    Julian Fiegenbaum | ISR Information Products AG | Consultant | Germany
    ------------------------------



  • 14.  RE: Datacap RuleRunner aborts after successful Task

    Posted Wed October 25, 2023 07:11 AM

    For future readers,

    this particular symptom was caused by the DCSTConnector.dll missing. (Re)installation of the insight package fixed it.



    ------------------------------
    Julian Fiegenbaum | ISR Information Products AG | Consultant | Germany
    ------------------------------



  • 15.  RE: Datacap RuleRunner aborts after successful Task

    Posted Thu September 28, 2023 11:55 AM

    Your pdf coversion rrs log was cut short and didn't see the error.  I can see profile is the longest.

    You are calling the release engine which is good. Try having just one thread and reprocess just this  task.  See if it complete.

    If Rule runner is saturated try configuring the restart after batch execution to help rulerunner memory. 

    https://www.ibm.com/docs/en/datacap/9.1.9?topic=service-configuring-rulerunner-restart-after-batch-execution

    Total Profile Rule     esec:'18.10156'


    ------------------------------
    Blue Devil
    ------------------------------



  • 16.  RE: Datacap RuleRunner aborts after successful Task

    Posted Mon October 02, 2023 03:25 AM
    Edited by Julian Fiegenbaum Mon October 02, 2023 03:30 AM

    Hi Blue Devil,

    Thanks for your reply! What do you mean with "Your pdf coversion rrs log was cut short"? The log ends with OnRulesetEnd of the last ruleset and subsequent task unloading. As far as i can tell, this is how a task log shoud end or am i missing something?

    The sad thing is that we currently run only one thread and it fails at the very first batch :/ Windows performance monitor does not suggest problematic amount of memory usage, but i have set "restart after batch execution" just in case as per these instructions: Configuring Rulerunner to restart after batch execution - IBM Documentation. Thanks for the tip :)



    ------------------------------
    Julian Fiegenbaum | ISR Information Products AG | Consultant | Germany
    ------------------------------