Rather stay with the trace as it is easily disabled and not filling our log infrastructure unnecessary. We tend to dump the whole message context [body, exception, local env] in it, to analyze an issue easier.
So current workaround is restarts and re-deployments, or looking at logs of integrated services to get a better picture.
As I said, this thread here is the closest to a support case with IBM we are making, as there is no pressure to fix this.
Original Message:
Sent: Wed June 18, 2025 04:08 AM
From: Matthias Blomme
Subject: ACE v12 - empty user trace package
Hi Vid
Not a solution to your problem, but you could (temporarily) use the log node? You could only wire it to the error branch and then check the activity log to see the output.
As a workaround until this issue is fixed.
I would start by opening a support case with IBM, not a PMR directly.
------------------------------
Regards
Matthias Blomme
Integration Specialist
Integration Designers
Turnhout
Original Message:
Sent: Tue June 10, 2025 04:00 AM
From: Vid Romac
Subject: ACE v12 - empty user trace package
We don't stop the trace until we are finished with the debug. The file is completely empty, so there is 0 data in it, and we'd expect at least some 'passive' data to be as it usually is [as is the case when the user trace works].
No pod restarts happen, a the problematic scenario goes thru an error branch that logs the details and then finishes cleanly. Other operations and invocations continue to work correctly.
Yeah, a PMR is an option, but that is a very bureaucratized and slow process and this is not something that is critical enough. Hopes are someone had a quick-fix or a thing to note that could help going forward.
But if IBM sees this thread and decides it is worth looking into (esp. if other people mention seeing this issue), that is a win for me.
------------------------------
Vid Romac
Original Message:
Sent: Tue June 10, 2025 03:49 AM
From: Francis Cocx
Subject: ACE v12 - empty user trace package
Hi,
I understand that you only started noticing this since the version upgrade. And if this is really no user/configuration error, than this might be worth starting a case with IBM to further investigate the issue.
Some other things you can check:
The trace reconcile loop is scheduled to run every minute. This means there is a possibility that some trace data might be lost during this interval. Could it be that you are just stopping the trace to quickly sometimes? If so, definitely try and wait one minute before stopping the trace after triggering the problematic scenario.
Another possible cause could be that your pod restarts because of the problematic scenario. You could prevent this by setting the following settings as testing purposes to see if this impact the behavior that you are noticing.
spec: env: - name: MQSI_PREVENT_CONTAINER_SHUTDOWN value: 'true'
Hope this helps!
------------------------------
Francis Cocx
Original Message:
Sent: Tue June 10, 2025 03:23 AM
From: Vid Romac
Subject: ACE v12 - empty user trace package
Mostly its for first time activation, as we don't activate it unless an odd issue is raised; but as I said the only way to fix it is to restart/redeploy the same thing and then re-activate the user trace so in the end it works on 'first time' activation.
No changes are in the trace nodes themselves, same flow/BAR/config is deployed when it works or doesn't work.
We are sure the flow + trace is triggered as the normal logs would show that the problematic flow has occurred.
I repeat, the main things to note is:
- service A with config A is deployed on integration server A [each container/integration server hosts only a single app/service]
- it is active for a extended period, uninterrupted
- an issue is raised, regarding data that entered the flow
- we activate the user trace
- we re-trigger the problematic scenario
- we download the user trace -> it is empty
- we restart the container [kill the pod e.g.]
- after a new one is started we re-activate the user trace
- we re-trigger the problematic flow
- we download the user trace -> 30% chance that it is created now, if not repeat above restart steps
And the other thing of note is the version upgrade; this behavior was not observed on the prev. integration server version.
------------------------------
Vid Romac
Original Message:
Sent: Tue June 10, 2025 02:42 AM
From: Francis Cocx
Subject: ACE v12 - empty user trace package
Hi Vid,
You mentioned that the issue only occurs sometimes, do you notice if it always happens the first time you enable trace after deploying the server? Or does it also occur after trace has been enabled and disabled previously?
Have you also checked the state of the Trace nodes within the message flows? By default, these are disabled, which could explain why the user trace file ends up empty. If the behavior is different for the same configuration after you have deployed the integration server, are you sure there was information to be logged during the time the trace was enable? If you have configured tracing for a specific scope (e.g., only a single node), and the event you are looking for is happening outside of that scope, no trace data will be generated.
Just to clarify: if the Trace node setting is disabled at the integration server level, all Trace nodes within its flows will be inactive. You can configure Trace node settings individually for each message flow, but they only take effect once Trace nodes are enabled at the server level. When Trace nodes are enabled on the integration server, the per-flow settings then determine whether trace data is actually collected.
Regards, Francis
------------------------------
Francis Cocx
Original Message:
Sent: Mon June 09, 2025 10:30 AM
From: Vid Romac
Subject: ACE v12 - empty user trace package
We are stumbling upon an issue with ACE user traces not always being generated.
We have a flow that has a trace node to log some info.
The steps we use are the following to enable and fetch the trace [start after deployment] - Enabling and managing trace for a deployed integration server
The thing is that this happens only sometimes. We deploy a server, we enable a trace, trigger an event and then download the trace. In most cases it works, and a zip file is created that contains the file with all the details we expect. However sometimes, we get a minuscule zip [22b large], that is empty, no file in it. Attempts to re-download or restart the user trace yields no changes.
Only if we restart the whole container [i.e. kill the pod or scale it down and up again] does it 'fix' it self, and the same server/service/flow starts writing user traces [and even this is not 100% reliable, I'd say 70% of the time the same behavior is observed even after restarts].
This is a headache as we'd prefer not to have to restart/redeploy our services in case of user-trace debugging, and enabling it on the admin page on a per-need basis usually works exactly as we need it. The user trace is enabled only when needed, for a short period and disabled [and reset] afterwards.
I am asking here if someone else has seen this behavior already and if there is a fix for it [either on our side or a planned one from IBM]?
We are using version ACE on CP4I, integration server version: 12.0.12-r2
NOTE - This behavior has been noticed only after upgrading from a previous v12 [I think it was 12.0.7.0-r5 but not 100% sure]
------------------------------
Vid Romac
------------------------------