Planning Analytics with Watson

Always use temporary subsets and views!

By Errol Brandt posted Fri October 22, 2021 06:10 AM

  

Always use temporary subsets and views!


I came across an interesting problem yesterday when I was speaking with another Planning Analytics administrator.  

He asked me how often do we restart our model. When I told him that I hadn't restarted our model for a few weeks, he almost fell off his chair.  He was just as suprised when I told him that our model sits stable at around 180gb.  

It turns out that his model is constantly growing in size and daily restarts were required to free up garbage memory.

Suspect

I remember having this issue a few years ago, but had found that the "problem" seemed to go away by itself.  It occurred around the same time I switched every process over to using temporary subsets and views to avoid contention locks when running threads in parallel.

Today, I joined the dots and wondered whether it was the permanent views were causing the bloat.  Now you might think that's obvious, and that would be true except for the fact that the epilog called processes that subsequently deleted the permanent views and subsets. 

Temporary view/subset, or deleted permanent view/subset - same thing then, right?   Well apparently not....

Experiment

I formed a hypothesis that permanent subsets that are deleted at the end of the process will consume more memory than temporary subsets which are deleted automatically by the process.  That is, although they appear to achieve the same outcome, the impact on memory is different.

To test this hypothesis, I created a cube with four dimensions.


I populated the cube with test data, created rules, internal feeders etc.  Something with a bit of complexity to model the real world.

Then I created two processes:

      TEST SUBPROCESS 1:      Temporary object variable is set to 0 (i.e., views and subsets are permanent) 
      TEST SUBPROCESS 2:       Temporary object variable is set to 1 (i.e., view and subsets are temporary)

The first process creates permanent views, which are subsequently deleted in the epilog.


##================ Prolog ================

cTemp = 0;

# create views

IF(ViewExists(sCube, sView) = 1);
ViewDestroy(sCube, sView);
ENDIF;

ViewCreate(sCube, sView, cTemp);

# create subsets

sDim = 'TEST State';
sSub = sView;
sConsol = 'All TEST state';

IF(SubsetExists(sDim, sSub) = 1);
SubsetDeleteAllElements(sDim, sSub);
ELSE;
SubsetCreate(sDim, sSub, cTemp);
ENDIF;

##================ Epilog ================

ExecuteProcess ('sys.view.cleanup');
ExecuteProcess ('sys.subset.cleanup');




The second process used temporary views.  


##================ Prolog ================

cTemp = 1;

# create views

IF(ViewExists(sCube, sView) = 1);
ViewDestroy(sCube, sView);
ENDIF;

ViewCreate(sCube, sView, cTemp);

# create subsets

sDim = 'TEST State';
sSub = sView;
sConsol = 'All TEST state';

IF(SubsetExists(sDim, sSub) = 1);
SubsetDeleteAllElements(sDim, sSub);
ELSE;
SubsetCreate(sDim, sSub, cTemp);
ENDIF;

##================ Epilog ================





To set the experiment up properly I deactivated all chores, cleaned-up any existing temporary views and subsets, checked that performance monitor was not active, and then restarted the model. 

When the model started up I ran the process 1.  I then shut down the server, restarted the model and executed the same process again. To be double-sure, I repeated the same process two more times to account for any impact of caching. 

Results

My suspicions were confirmed:  The process that built and deleted permanent views and subsets took longer and more generated garbage memory.  By contrast, the process that built temporary views and subsets, made a negligable impact on memory.

Test 1

After setting up the test conditions and running the first process, the model's memory consumption increased from 116.01GB to 116.74GB.  Now that's not huge, but imagine if this keeps happening in the same process, or across multiple processes. This will almost certainly bloat the model.

Memory Before: 116.01GB Memory After: 116.74GB

Test 2

After resetting the test conditions and running the second process, the model's memory increased from 116.01GB to 116.02GB.  Basically there was a neglible increase in garbage memory.

Memory Before: 116.01GB Memory After: 116.02GB

So what?

Although the two methods might appear to be equivalent, deleting permanent views within a process is expensive and probably leads to unnecessary buildup of garbage memory.  This means that developers who are not using temporary views and subsets are making their clients restart the model unnecessarily.  As a consequence of this, models not using this parameter are not making the best use of stargates, thereby causing unnecessary user delays.  This could be quite significant when users attempt to access large views.

In my view, I believe developers should always use temporary views and temporary subsets used except where there is an explicit requirement to do so. Failing to do so could be significantly impacting the user experience!


​​​​
1 comment
42 views

Permalink

Comments

Mon October 25, 2021 02:06 AM

Thanks for running these tests and posting Errol. I would like to have tested on a client where we often end up with about 30Gb in garbage but their version does not support the way we create our views. We use a child process and the view is no longer available in the parent or related processes.

Worthwhile pointing out that IBM changed this in PA 2.0.5 or V11.3 per this article.