I started a promotion to Production probably 25/26 hours ago and it's still calculating (almost finished) and I was wondering if anyone had any tips to making productions go faster or whether this was just a one off and it'll be fine next month?

Usual promotions for me are about 2.5 hours and I don't think that I've changed the way it was done apart from the time that I started it.

Any tips/ help would be greatly appreciated!

Hi Oliver,

If you have not done so already, would you mind logging a support ticket for this too.

Thanks.

Michael

I have the same issue going on. I have two support tickets in progress.

Dev to stg started over 15 hours ago

Stg to prod started over 12 hours ago

The support team found the problem, and I'm hopeful that the fix they applied will resolve it shortly

Thanks for the updates, have they managed to fix whatever the problem was?

I'm curious as well - did they get to the root cause?

After a solid effort from both the Apptio support team and us we recovered the service on Saturday (around 44 hours down), and we're back on track.

In terms of root cause we're documenting all events to get to it. As is usually the case I think it was a combination of factors, some self-inflicted, that coincided at the worst possible time.

We recently had a snafu ourselves - turning on/off certain objects kicked off new reports, broke some others - all of which caused the system to hang, max out on heap memory, and then ultimately reset several times.  We wound up having to fix on the dev side, push back to stage to fix that, and then stage would act right so we could continue our monthly copies.

Our TAM had us install a "PreCalc Analyzer" in our stage environment, which is cool.  It's its own project and pulls in details from the other environments and lets you see the 'trouble-makers'.  Kind of like task manager in windows...  Was able to quickly point out certain reports that were taking several hours to calc through.  If you don't have that, definitely ck into it. 

One suggestion I have for Apptio (maybe this is an enhancement type request I need to enter?) is if there was a list of the OOTB reports and what objects they were connected to - that may exist, but we haven't seen anything.  Not having that is making us go through and catalog each report to pull those details together manually.  Then we can quickly check if we need to add/remove objects and see what potential impacts we may have on the reporting surface. 

BTW, deleting unused OOTB reports didn't work in our experience - since OOTB, they're hard-coded in somehow, so they just pop back up.  In those cases, I just deselected the pre-calc button and hit save. 

Interestingly enough, I was also having the same issue. In dev & prod, it turned out to be a heap size issue. Stage is still being investigated. It seems odd that several instances are having the same problem at the same time.

I agree Tony. This is something Apptio should investigate and explain.

Is anyone having the same issue again? Our Prod environment has been calculating all night and still hasn't finished. Seems like it's going to be a very long time again

I had a similar event last Friday. This time we were successful in containing the impacts to end users and recovering faster (rolling back). Applying lessons learned . The reason for the latest event is unresolved model configuration issues from the first event, causing ultra long calculation cycles.

The jury is still out on what if anything can be done from the Apptio side to reduce/anticipate/prevent situations like this. It is odd that we had two events in such a short time, while we have been very active in config work for two years.

There are land mines in the model (at least in mine), that go off for reasons that appear unrelated to the reason why it was planted, and that makes it much more difficult to correct. 

Did you guys get the PreCalc Analyzer installed to see if there were any unusually long calc times associated w/any of the reports?  That's been our big deal - whenever we add/remove objects, it hoses up the reports that we didn't know were linked.  We know to research now, but some reports that were fixed once came back to haunt us.  That got back to 'hard coded, OOTB' reports that we deleted, and then popped back up - so we just saved them down without 'precalc' checked.  Then we ran into issues with syntax in terms of project copy requests, so have a common doc to use across TBMAs now where they can just copy/paste into the ticket depending on the step.  More lessons learned and whatnot.

I'm using now a calc analyzer. Thank you for the suggestion. It was one of the keys to our quick recovery this time.

Excellent - we love having it available - definitely helps identify issues quickly

Let me clear up a couple of items here.

1.)The "PreCalc Analyzer" is an excel spreadsheet I created to parse out the calculation data that is generated when a project Pre-calculates.  It has to be manually generated by someone in Apptio/Support for every calculation that needs to be investigated.

2.)There is a special project that is in Beta that does this inside of Apptio, that I designed, that consumes the calculation data in a similar way, but it is not readily available at this time.  Some customers have access to this.  If you're having issues with a long pre-calculation in any environment the first place to start is by filing a support ticket, and they can start the initial investigation.

When it comes to long running pre-calculations there can be a variety of reason/causes that have to be considered.

Let me know if you have any questions.