Hi George, Alexander,
The biggest issues with SmartCache were that 1) the supporting structures quickly ended up taking more memory then the source cube and 2) that figuring out what to invalidate took longer then it took to recalculate. For very simple rules it worked marvelous but anything a tad more involved (still simple for most of you TM1 experts), it pretty rapidly went nowhere.
But the solution IMHO is indeed that the engine figures this out as opposed to adding more sections to the rules to 'help' the engine be smarter. More importantly, we need the answers to be correct 100% of the time, especially in the v12 world with HA where the query can be answered by any of the replicas running, we can't have the answer be influenced by previously executed queries (read: stargates that were created to answer those queries), we've have had enough issues with with underfeeding, even when people deliberately did so and knew they did it right, not.
We are, and have been for a little while now, working on a completely new consolidation engine for TM1 (a.k.a. NQE or 'the feeder less engine'). This also builds on the knowledge gained by trying to do smart-cache (specifically in the rule analysis side of things), and apart from it begin way more efficient in executing these consolidating and rule executing queries in the first place, whilst not needing any feeders, we expect that we'll be able to do invalidations, whilst less needed, more efficiently over time as well.
This new engine, apart from it no longer requiring feeders (which is what most people might like the most about it), is also going to be the base for true horizontal scaling for TM1. Whilst you might not need horizontal scaling for size of the model, and not even need it either to throw more compute power at the model either, it would allow you to effectively partition by country/countries, presuming no rules cross those country boundaries (as in your example if I understand correctly Alexander). A simplified, manually driven (perhaps like a STARGATES section like you propose), partitioning scheme might be considered for TM1 v12 (based on our learnings with the new engine which will go into a next major version) which might get you closer to what you are trying to accomplish at that point.
None of this directly targets stargates but I would think it would help get you closer to your goal (stargates are just TM1's caches, you shouldn't need to be aware of them to begin with let alone get involved in their life cycle management IMHO;-).
------------------------------
Hubert Heijkers
STSM, Program Director TM1 Functional Database Technology and OData Evangelist
------------------------------
Original Message:
Sent: Mon March 31, 2025 06:09 AM
From: George Tonkin
Subject: TM1 caching and stargates: distinguish them by specified dimension's elements?
Hi Alexander,
This is a good idea but I seem to recall that it is similar to what "SmartCache" was supposed to achieve but never did and thus did not make it to release.
@Hubert Heijkers may have more insights in terms of how this may work in the future.
------------------------------
George Tonkin
Business Partner
MCI Consultants
Johannesburg
Original Message:
Sent: Mon March 31, 2025 03:59 AM
From: Alexander Dvoynev
Subject: TM1 caching and stargates: distinguish them by specified dimension's elements?
I've wanted to post it as a new idea, but I want to discuss it first and collect some ideas.
Stargates are very fragile and delicate. A lot of actions can cause stargate removal (we've discussed them in TM1 Quiz)
- Change any cell in the cube (Or even in any cube your cube depends on)
- Change cube's rule (Or other cube's rule your cube depends on)
- add+delete element to any dimension in the cube
- Change alias of any element of any dimension in the cube
- DebugUtility( 125, 0, 0, '', '', '' );
- Server restart
My idea is to avoid first 3 causes.
As an example, we have a multi-country model. Countries here are independent: One country's data do not affect another country's data. The 1st one has approved and locked her data and now deserves fully-cached and fast calculations. But other 100 countries are changing their data every 5 minutes, this causes stargate removal, and our 1st country needs to wait for model to recalculate everything from scratch. This can be annoying, and I've even had to make a cube for each country on one of my project to handle independent countries stargates.
My idea is to add a new STARGATES; section after FEEDERS;, where we can specify dimensions, where we guarantee their independent calculations, and thus for which stargates should be independent:
STARGATES;[Country];[Brand];
Having this, when a country changes a cell (and same for Brand in this example), only it's stargate (and all consolidations' it is a part of) should be destroyed. All other countries should not be affected.
Or, if stargate cannot be divided into parts, stargates should be created for each country independently.
------------------------------
Best regards, Alexander Dvoynev
------------------------------