Hi David,
Wow, 6 years already, and yes my comment wrt queries against transaction log being contentious is most definitely still true, nothing changed in that regard, nothing will. What did change however is that I introduced an query execution optimization when filtering, most notably when filtering based on time ranges, but sounds like you are not doing that.
Long story short, you shouldn't query transaction logs over and over again if you can prevent it. Trailing the transaction log, using delta pulls, and storing the transaction in either a database or something even more specific for log management like the Elastic Stack, would be the way to go IMHO (and which is what @Vlad Didenko is talking about as well whilst taking it a step further and give you a nice, integrated, UI on top of that).
This tracking/trailing of the log is also what I've shown in my tm1-log-tracker utility already back in 2017 (https://github.com/Hubert-Heijkers/tm1-log-tracker) a pattern later used for the so called, customer inspired, 'blackhawk pattern' to replay/apply those transactions immediately on a secondary TM1 server (https://github.com/Hubert-Heijkers/tm1-blackhawk). The trailing of the transaction log remains the same, it is just for you to fill in the blanks and where to send the data and in which form.
Good news ahead though, in v12 you can simply subscribe to the transaction log and we'll happily push transaction to any URL you provide us it while they happen (without impacting performance of the TM1 database in question), you might have seen me demonstrate this using Elastic.
------------------------------
Hubert Heijkers
STSM, Program Director TM1 Functional Database Technology and OData Evangelist
------------------------------
Original Message:
Sent: Thu March 12, 2026 10:36 AM
From: David Usherwood
Subject: REST - FilterableProperties on TransactionLogEntries
"Things being as they are (TM1Server v11 and before) queries against the transaction log are contentious (effectively block any other commits). The occasional brief query doesn't hurt as (hence: why I did a special implementation for deltas) but longer running queries could, especially with larger transaction logs, seriously impact the system."
Apologies for following up a 6 year old post - and please advise if this no longer applies.
I have been doing some work with the Rest API transaction log functionality using TM1Py (2.0.3), with PAoC 11.8.02800.9. The client keeps three month's worth of transactions. They want to capture user changes to four specific mapping cubes.
I started out issuing four calls specifying cube and date range. This worked well on Dev but migrating to prod with the larger volumes became slow - 6 minutes per call.
I rethought the approach and changed to a single call for all transactions in a date range, then filtering the results in Python. This works much better, taking 3 minutes. It does of course use more memory in python but not an unreasonable amount - 5 gig for a month.
Can you comment on the difference?
The other test I carried out was to write values to a cube via PAW workbench while the call was running. I found that this updated without issues.
Can you comment on this?
------------------------------
David Usherwood
Original Message:
Sent: Thu January 09, 2020 02:58 AM
From: Hubert Heijkers
Subject: REST - FilterableProperties on TransactionLogEntries
Ryan, if you hadn't told us that you were putting them already in a database I'd have suggested you should;-)
Things being as they are (TM1Server v11 and before) queries against the transaction log are contentious (effectively block any other commits). The occasional brief query doesn't hurt as (hence: why I did a special implementation for deltas) but longer running queries could, especially with larger transaction logs, seriously impact the system.
We contemplated at some point to allow users to see their own transactions (like we do in the REST API for the threads/sessions) but shed away from doing so for the exact same reason, not wanting to invite clever people to add contention to the most contentious resource we have.
So, for now, I'd indeed put it in a separate database and, if I was writing my own client, I wouldn't even use TM1's drill through as the middle man but directly go to that database to retrieve the transactions I'd be looking for. Any reason why you wouldn't?
Now the 'good' news perhaps is that in the next major version of TM1 server we already have these transactions in a database (part of the high availability work we've been working on for a while). We have not finalized any new APIs for that version, might not need any, but at least we'll have tackled the contention issues plus the performance presumably would be way better than ploughing through old fashion text files;-).
------------------------------
Hubert Heijkers
Original Message:
Sent: Wed January 08, 2020 03:02 PM
From: Ryan Clapp
Subject: REST - FilterableProperties on TransactionLogEntries
Today we extract transactions, push them into a database, and then expose them via drill through. There is an inherent time lag due to the length of the processing. In our UI we want users to be able to right click on a data point, cell in a view, and select an option to "Show all Changes".
Original Message------
Hi Ryan,
Interesting question...Not currently possible and it's not on our roadmap. Why do users need to see transactions? Is it transactions for all users or just the data changes made by the user running the query?
I think we do want to look into storing transactions in proper database at some point in the distant (or maybe not so distant) future. We might look at security for transactions at that point. We won't plan any changes in security in this area in the current version of TM1 Server.
------------------------------
Stuart King
IBM Planning Analytics Offering Manager
------------------------------
#PlanningAnalyticswithWatson