Planning Analytics

Hi,

Before restarting a TM1 instance, via IBM Cognos Configuration or via Services, I always do a SaveDataAll.
That's how I learned it a long time ago.

I'm wondering now if that SaveDataAll is really necessary, does a restart via IBM Cognos Configuration or via Services also save data before stopping?

Best regards,

Hi Herman,

I think you learned from someone who lost data when the server crashed shutting down and they did not want to risk this in the future.

I still do this more often than not as it is just habit now.

However, stopping the service tells TM1/PA to shutdown and will go through the SaveDataAll process during the shutdown.

What you could do to confirm is to add the TM1.SaveDataAll logger to your tm1s-log.properties file so that you can see what is being done in terms of saving data when you shut down.

Hope that helps. Let me know if you need guidance on adding the logger.

Hi,

Before restarting a TM1 instance, via IBM Cognos Configuration or via Services, I always do a SaveDataAll.
That's how I learned it a long time ago.

I'm wondering now if that SaveDataAll is really necessary, does a restart via IBM Cognos Configuration or via Services also save data before stopping?

Best regards,

Hi Herman,

I think you learned from someone who lost data when the server crashed shutting down and they did not want to risk this in the future.

I still do this more often than not as it is just habit now.

However, stopping the service tells TM1/PA to shutdown and will go through the SaveDataAll process during the shutdown.

What you could do to confirm is to add the TM1.SaveDataAll logger to your tm1s-log.properties file so that you can see what is being done in terms of saving data when you shut down.

Hope that helps. Let me know if you need guidance on adding the logger.

Hi,

Before restarting a TM1 instance, via IBM Cognos Configuration or via Services, I always do a SaveDataAll.
That's how I learned it a long time ago.

I'm wondering now if that SaveDataAll is really necessary, does a restart via IBM Cognos Configuration or via Services also save data before stopping?

Best regards,

Hi George,

Has something changed on this?

My understanding was that it used to be that if you stopped a service on windows it would try and save the data but if it took longer than a certain time (I think about only 30 seconds) then it could just stop the service anyway meaning some data could be lost.

It's not something I have really looked into since as like many I have stuck with the habit of always saving data before stopping a service (and having regular scheduled saves just in case.)

Thanks,
Declan

Hi Herman,

I think you learned from someone who lost data when the server crashed shutting down and they did not want to risk this in the future.

I still do this more often than not as it is just habit now.

However, stopping the service tells TM1/PA to shutdown and will go through the SaveDataAll process during the shutdown.

What you could do to confirm is to add the TM1.SaveDataAll logger to your tm1s-log.properties file so that you can see what is being done in terms of saving data when you shut down.

Hope that helps. Let me know if you need guidance on adding the logger.

Hi,

Before restarting a TM1 instance, via IBM Cognos Configuration or via Services, I always do a SaveDataAll.
That's how I learned it a long time ago.

I'm wondering now if that SaveDataAll is really necessary, does a restart via IBM Cognos Configuration or via Services also save data before stopping?

Best regards,

Hi Declan,

That sounds like something on the OS side as I am not sure I have seen that.

If I think about the numerous sites still running with PersistenFeeders=T, it can take 10 minutes for the server to come down and the feeders to save.

I do not recall seeing that the process is terminated after 30 seconds on these.

But again, habit is to do the save via chore or manually as we have all likely been burnt in the past.

Maybe someone else has something to add in terms of this and their experiences...

Here is a piece of the SaveDataAll log for those that curious:

60560   []   DEBUG   2023-08-30 12:17:49.866   TM1.SaveDataAll   Serializing cube 'Client Assumption'60560   []   DEBUG   2023-08-30 12:17:49.866   TM1.SaveDataAll   Serializing cube 'Exchange Rate'60560   []   DEBUG   2023-08-30 12:17:49.866   TM1.SaveDataAll   Serializing cube 'Expense'...60560   []   DEBUG   2023-08-30 12:17:49.870   TM1.SaveDataAll   Serializing cube '}StatsByCubeByClient'60560   []   DEBUG   2023-08-30 12:17:49.871   TM1.SaveDataAll   Serializing cube '}StatsByProcess'60560   []   DEBUG   2023-08-30 12:17:49.872   TM1.SaveDataAll   Serializing cube '}StatsByRule'60560   []   DEBUG   2023-08-30 12:17:49.872   TM1.SaveDataAll   Serializing cube '}StatsForServer'60560   []   DEBUG   2023-08-30 12:17:49.873   TM1.SaveDataAll   Saving sets60560   []   DEBUG   2023-08-30 12:17:49.873   TM1.SaveDataAll   Saving subsets60560   []   DEBUG   2023-08-30 12:17:49.873   TM1.SaveDataAll   Saving calculated members60560   []   DEBUG   2023-08-30 12:17:49.873   TM1.SaveDataAll   Saving processes60560   []   DEBUG   2023-08-30 12:17:49.873   TM1.SaveDataAll   Saving chores

Hi George,

Has something changed on this?

My understanding was that it used to be that if you stopped a service on windows it would try and save the data but if it took longer than a certain time (I think about only 30 seconds) then it could just stop the service anyway meaning some data could be lost.

It's not something I have really looked into since as like many I have stuck with the habit of always saving data before stopping a service (and having regular scheduled saves just in case.)

Thanks,
Declan

Hi Herman,

I think you learned from someone who lost data when the server crashed shutting down and they did not want to risk this in the future.

I still do this more often than not as it is just habit now.

However, stopping the service tells TM1/PA to shutdown and will go through the SaveDataAll process during the shutdown.

What you could do to confirm is to add the TM1.SaveDataAll logger to your tm1s-log.properties file so that you can see what is being done in terms of saving data when you shut down.

Hope that helps. Let me know if you need guidance on adding the logger.

Hi,

Before restarting a TM1 instance, via IBM Cognos Configuration or via Services, I always do a SaveDataAll.
That's how I learned it a long time ago.

I'm wondering now if that SaveDataAll is really necessary, does a restart via IBM Cognos Configuration or via Services also save data before stopping?

Best regards,

Hi George,

Has something changed on this?

My understanding was that it used to be that if you stopped a service on windows it would try and save the data but if it took longer than a certain time (I think about only 30 seconds) then it could just stop the service anyway meaning some data could be lost.

It's not something I have really looked into since as like many I have stuck with the habit of always saving data before stopping a service (and having regular scheduled saves just in case.)

Thanks,
Declan

Hi Herman,

I think you learned from someone who lost data when the server crashed shutting down and they did not want to risk this in the future.

I still do this more often than not as it is just habit now.

However, stopping the service tells TM1/PA to shutdown and will go through the SaveDataAll process during the shutdown.

What you could do to confirm is to add the TM1.SaveDataAll logger to your tm1s-log.properties file so that you can see what is being done in terms of saving data when you shut down.

Hope that helps. Let me know if you need guidance on adding the logger.

Hi,

Before restarting a TM1 instance, via IBM Cognos Configuration or via Services, I always do a SaveDataAll.
That's how I learned it a long time ago.

I'm wondering now if that SaveDataAll is really necessary, does a restart via IBM Cognos Configuration or via Services also save data before stopping?

Best regards,

Hi,

Before restarting a TM1 instance, via IBM Cognos Configuration or via Services, I always do a SaveDataAll.
That's how I learned it a long time ago.

I'm wondering now if that SaveDataAll is really necessary, does a restart via IBM Cognos Configuration or via Services also save data before stopping?

Best regards,

Hi all
I have experienced data loss due to the Windows-update, and not waiting long enough for the TM1 servers to shut down gracefully.
I would love to hear some recommendations regarding this. 

Hi,

Before restarting a TM1 instance, via IBM Cognos Configuration or via Services, I always do a SaveDataAll.
That's how I learned it a long time ago.

I'm wondering now if that SaveDataAll is really necessary, does a restart via IBM Cognos Configuration or via Services also save data before stopping?

Best regards,

Hi Asgeir,

The general concept, when a service starts, is that it checks whether all data was saved successfully to disk when it was last stopped; if the answer is no - it will then start the service with the data it did have saved to disk... but also on top of that it will scan through the tm1s.log files and reload the transactions which were logged from there. (should add that this is for V11 - it seems that logging etc has changed in V12 so this probably won't be applicable.)

So with that in mind, the trick is to consider which transactions you record when the service is up. There is always the risk that the whole Server has an issue etc so you should plan for how to make sure all data comes back if the worst happens.

My general approach has been:

1/ Any data loading TI processes (from source systems) - I tend to turn off logging (to make the process faster) so when these are finished I make sure that I save data to disk. However, I may leave logging on anyway if the source is fast changing - but that is more so I can explain why a number is X if I can't go straight back to the source at a later point in time.

2/ Any cubes that users will interact with - make sure cube logging is enabled. (also anything that a "Super User/Admin" may change directly)

3/ Try and have regular Save Data processes (this normally comes down to a bit of common sense and trial and error to aim for the times that will cause the least impact to users.) - As a very minimum there is at least 1 save data that would occur "out of hours" but ideally I aim to have more than that.

Thanks,
Declan

Hi all
I have experienced data loss due to the Windows-update, and not waiting long enough for the TM1 servers to shut down gracefully.
I would love to hear some recommendations regarding this. 

Hi,

Before restarting a TM1 instance, via IBM Cognos Configuration or via Services, I always do a SaveDataAll.
That's how I learned it a long time ago.

I'm wondering now if that SaveDataAll is really necessary, does a restart via IBM Cognos Configuration or via Services also save data before stopping?

Best regards,

Hi Asgeir,

The general concept, when a service starts, is that it checks whether all data was saved successfully to disk when it was last stopped; if the answer is no - it will then start the service with the data it did have saved to disk... but also on top of that it will scan through the tm1s.log files and reload the transactions which were logged from there. (should add that this is for V11 - it seems that logging etc has changed in V12 so this probably won't be applicable.)

So with that in mind, the trick is to consider which transactions you record when the service is up. There is always the risk that the whole Server has an issue etc so you should plan for how to make sure all data comes back if the worst happens.

My general approach has been:

1/ Any data loading TI processes (from source systems) - I tend to turn off logging (to make the process faster) so when these are finished I make sure that I save data to disk. However, I may leave logging on anyway if the source is fast changing - but that is more so I can explain why a number is X if I can't go straight back to the source at a later point in time.

2/ Any cubes that users will interact with - make sure cube logging is enabled. (also anything that a "Super User/Admin" may change directly)

3/ Try and have regular Save Data processes (this normally comes down to a bit of common sense and trial and error to aim for the times that will cause the least impact to users.) - As a very minimum there is at least 1 save data that would occur "out of hours" but ideally I aim to have more than that.

Thanks,
Declan

------------------------------
Declan Rodger
Technical Director
Spitfire Analytics

Original Message:
Sent: Thu August 31, 2023 06:12 AM
From: Asgeir Thorgeirsson
Subject: Restarting TM1 intance and SaveDataAll

Hi all
I have experienced data loss due to the Windows-update, and not waiting long enough for the TM1 servers to shut down gracefully.
I would love to hear some recommendations regarding this. 

------------------------------
Asgeir Thorgeirsson

Original Message:
Sent: Wed August 30, 2023 04:08 AM
From: Herman Teeuwen
Subject: Restarting TM1 intance and SaveDataAll

Hi,

Before restarting a TM1 instance, via IBM Cognos Configuration or via Services, I always do a SaveDataAll.
That's how I learned it a long time ago.

I'm wondering now if that SaveDataAll is really necessary, does a restart via IBM Cognos Configuration or via Services also save data before stopping?

Best regards,

------------------------------
Herman Teeuwen
------------------------------

The advice I got is to run SaveDataAll sparely

According to SaveDataAll Best Practices (ibm.com)

SaveDataAll:

• Writes cube data and feeders to disk. 
• Restarts the TM1 Servers transaction file (tm1s.log).  

We did run a "Save Data All" process hourly in the past but that created other problems

1. Users lose the ability to rollback data in PAW 
2. SaveDataAll creates a lot of write locks in the system and that can create various problems for other running processes, especially for the time-consuming ones or parallel processing. 

The advice I got is to run SaveDataAll sparely but use the CubeSaveData in the epilog of any data import. Any manually inserted data is automatically written to disk so I do not have to worry about that.  

I run SaveDataAll only once a day, outside business hours, and just before I run the daily extra backup of all TM1 models. I keep ten days of all models in a separate backup drive. Additionally, the server is included in the company's general backup system.  

Despite all this, I have experienced data loss due to the Windows Backup not waiting for the TM1 servers to shut down gracefully. 

Hi Asgeir,

The general concept, when a service starts, is that it checks whether all data was saved successfully to disk when it was last stopped; if the answer is no - it will then start the service with the data it did have saved to disk... but also on top of that it will scan through the tm1s.log files and reload the transactions which were logged from there. (should add that this is for V11 - it seems that logging etc has changed in V12 so this probably won't be applicable.)

So with that in mind, the trick is to consider which transactions you record when the service is up. There is always the risk that the whole Server has an issue etc so you should plan for how to make sure all data comes back if the worst happens.

My general approach has been:

1/ Any data loading TI processes (from source systems) - I tend to turn off logging (to make the process faster) so when these are finished I make sure that I save data to disk. However, I may leave logging on anyway if the source is fast changing - but that is more so I can explain why a number is X if I can't go straight back to the source at a later point in time.

2/ Any cubes that users will interact with - make sure cube logging is enabled. (also anything that a "Super User/Admin" may change directly)

3/ Try and have regular Save Data processes (this normally comes down to a bit of common sense and trial and error to aim for the times that will cause the least impact to users.) - As a very minimum there is at least 1 save data that would occur "out of hours" but ideally I aim to have more than that.

Thanks,
Declan

Hi all
I have experienced data loss due to the Windows-update, and not waiting long enough for the TM1 servers to shut down gracefully.
I would love to hear some recommendations regarding this. 

Hi,

Before restarting a TM1 instance, via IBM Cognos Configuration or via Services, I always do a SaveDataAll.
That's how I learned it a long time ago.

I'm wondering now if that SaveDataAll is really necessary, does a restart via IBM Cognos Configuration or via Services also save data before stopping?

Best regards,

Additionally, TM1 does not save the code we write in the PAW workbench automatically to disk so there is a risk of losing a whole day's work. Please correct me if I am wrong but I have a story to support this statement. 

I was very productive one day, working in the workbench, rewriting rules and code in TM1 cubes and processes. For some reason, the Windows server restarted. When I opened a new rules tab for some cube in the workbench, I noticed that all my newly written code was gone. Then, I checked the cube rule objects directly on the server and found no trace of almost a whole day of productive work. Luckily some browser windows were open on my desktop, containing my latest work. I could just hit the save button on each page or a tab but still, I had to look through everything carefully and could not be sure what was right and tested and what was not. I found this experience a bit scary. 

The advice I got is to run SaveDataAll sparely

According to SaveDataAll Best Practices (ibm.com)

SaveDataAll:

• Writes cube data and feeders to disk. 
• Restarts the TM1 Servers transaction file (tm1s.log).  

We did run a "Save Data All" process hourly in the past but that created other problems

1. Users lose the ability to rollback data in PAW 
2. SaveDataAll creates a lot of write locks in the system and that can create various problems for other running processes, especially for the time-consuming ones or parallel processing. 

The advice I got is to run SaveDataAll sparely but use the CubeSaveData in the epilog of any data import. Any manually inserted data is automatically written to disk so I do not have to worry about that.  

I run SaveDataAll only once a day, outside business hours, and just before I run the daily extra backup of all TM1 models. I keep ten days of all models in a separate backup drive. Additionally, the server is included in the company's general backup system.  

Despite all this, I have experienced data loss due to the Windows Backup not waiting for the TM1 servers to shut down gracefully. 

Hi Asgeir,

The general concept, when a service starts, is that it checks whether all data was saved successfully to disk when it was last stopped; if the answer is no - it will then start the service with the data it did have saved to disk... but also on top of that it will scan through the tm1s.log files and reload the transactions which were logged from there. (should add that this is for V11 - it seems that logging etc has changed in V12 so this probably won't be applicable.)

So with that in mind, the trick is to consider which transactions you record when the service is up. There is always the risk that the whole Server has an issue etc so you should plan for how to make sure all data comes back if the worst happens.

My general approach has been:

1/ Any data loading TI processes (from source systems) - I tend to turn off logging (to make the process faster) so when these are finished I make sure that I save data to disk. However, I may leave logging on anyway if the source is fast changing - but that is more so I can explain why a number is X if I can't go straight back to the source at a later point in time.

2/ Any cubes that users will interact with - make sure cube logging is enabled. (also anything that a "Super User/Admin" may change directly)

3/ Try and have regular Save Data processes (this normally comes down to a bit of common sense and trial and error to aim for the times that will cause the least impact to users.) - As a very minimum there is at least 1 save data that would occur "out of hours" but ideally I aim to have more than that.

Thanks,
Declan

Hi all
I have experienced data loss due to the Windows-update, and not waiting long enough for the TM1 servers to shut down gracefully.
I would love to hear some recommendations regarding this. 

Hi,

Before restarting a TM1 instance, via IBM Cognos Configuration or via Services, I always do a SaveDataAll.
That's how I learned it a long time ago.

I'm wondering now if that SaveDataAll is really necessary, does a restart via IBM Cognos Configuration or via Services also save data before stopping?

Best regards,

HI Asgeir,

I think there must be something else at play here. Saving rules and TIs has always written to disk immediately updating the .rux or .pro file.

Updates via the REST API do the same from my testing and this applies to PAW which would patch any existing rules or processes.

Is the server a VM and possibly loaded back to a previous checkpoint/version?

Additionally, TM1 does not save the code we write in the PAW workbench automatically to disk so there is a risk of losing a whole day's work. Please correct me if I am wrong but I have a story to support this statement. 

I was very productive one day, working in the workbench, rewriting rules and code in TM1 cubes and processes. For some reason, the Windows server restarted. When I opened a new rules tab for some cube in the workbench, I noticed that all my newly written code was gone. Then, I checked the cube rule objects directly on the server and found no trace of almost a whole day of productive work. Luckily some browser windows were open on my desktop, containing my latest work. I could just hit the save button on each page or a tab but still, I had to look through everything carefully and could not be sure what was right and tested and what was not. I found this experience a bit scary. 

The advice I got is to run SaveDataAll sparely

According to SaveDataAll Best Practices (ibm.com)

SaveDataAll:

• Writes cube data and feeders to disk. 
• Restarts the TM1 Servers transaction file (tm1s.log).  

We did run a "Save Data All" process hourly in the past but that created other problems

1. Users lose the ability to rollback data in PAW 
2. SaveDataAll creates a lot of write locks in the system and that can create various problems for other running processes, especially for the time-consuming ones or parallel processing. 

The advice I got is to run SaveDataAll sparely but use the CubeSaveData in the epilog of any data import. Any manually inserted data is automatically written to disk so I do not have to worry about that.  

I run SaveDataAll only once a day, outside business hours, and just before I run the daily extra backup of all TM1 models. I keep ten days of all models in a separate backup drive. Additionally, the server is included in the company's general backup system.  

Despite all this, I have experienced data loss due to the Windows Backup not waiting for the TM1 servers to shut down gracefully. 

Hi Asgeir,

The general concept, when a service starts, is that it checks whether all data was saved successfully to disk when it was last stopped; if the answer is no - it will then start the service with the data it did have saved to disk... but also on top of that it will scan through the tm1s.log files and reload the transactions which were logged from there. (should add that this is for V11 - it seems that logging etc has changed in V12 so this probably won't be applicable.)

So with that in mind, the trick is to consider which transactions you record when the service is up. There is always the risk that the whole Server has an issue etc so you should plan for how to make sure all data comes back if the worst happens.

My general approach has been:

1/ Any data loading TI processes (from source systems) - I tend to turn off logging (to make the process faster) so when these are finished I make sure that I save data to disk. However, I may leave logging on anyway if the source is fast changing - but that is more so I can explain why a number is X if I can't go straight back to the source at a later point in time.

2/ Any cubes that users will interact with - make sure cube logging is enabled. (also anything that a "Super User/Admin" may change directly)

3/ Try and have regular Save Data processes (this normally comes down to a bit of common sense and trial and error to aim for the times that will cause the least impact to users.) - As a very minimum there is at least 1 save data that would occur "out of hours" but ideally I aim to have more than that.

Thanks,
Declan

Hi all
I have experienced data loss due to the Windows-update, and not waiting long enough for the TM1 servers to shut down gracefully.
I would love to hear some recommendations regarding this. 

Hi,

Before restarting a TM1 instance, via IBM Cognos Configuration or via Services, I always do a SaveDataAll.
That's how I learned it a long time ago.

I'm wondering now if that SaveDataAll is really necessary, does a restart via IBM Cognos Configuration or via Services also save data before stopping?

Best regards,