Original Message:
Sent: Wed September 13, 2023 05:42 AM
From: Edward Stuart
Subject: Multiple DataBaseDirectory declaration issue
# Note to self to read the whole thread before replying! Pre v12 this holds a different set of challenges as is discussed. The potential to utilise most programming languages to replicate ExecuteHttpRequest exists but is a stopgap ahead of v12 adoption
I believe the opportunity we have with the Planning Analytics Engine will make our historical processes seem arcane,
I've yet to test the theory and I have a blog post waiting to flesh this out but I can see something along the lines of:
Process: Copy.Object
This takes parameters for Object type (Process, Dimension, Chore, Rules etc..), Object Name, Source Server and Target Server
Uses ExecuteHTTPRequest to store the details of the Object to be Copied
https://www.ibm.com/docs/en/planning-analytics/2.0.0?topic=functions-executehttprequest
- This process can be run standalone to migrate Process A from Server A to Server B
- This process can also be run as a worker process to a master process
Process: Copy.Selected.Objects
This loops the Control dimensions (}Clients, }Dimensions, }Cubes, }Processes etc..) which have an attribute to flag that the object is to be moved
For each instance were a flag is set then run Process Copy.Object
Server Control Cube:
This Cube will hold the details for each Server in the environment, each server will need to be manually added as expected but again a ExecuteHTTPRequest call to "http://<ServerName>:<PortNumber>/api/v1/Servers" will pull back data for those selected servers
This needs testing and appropriate due diligence on setting the flags for objects to be copied, considerations also need to be made on backups of existing objects before copyover should anything go awry/ mistakes be made.
Of course, we also have the alternative to set each server up against a Git Repository and pull/ push objects between servers direct from Github.
I can see processes being created to check the differences between two servers and creating automated migration packages to keep models in sync.
This do not scratch the surface of what is possible with ExecuteHTTPRequest to send objects, data, messages to third party tools to fulfill your workflow requirements.
------------------------------
Edward Stuart
Original Message:
Sent: Wed September 13, 2023 04:42 AM
From: Mucahit Erdal
Subject: Multiple DataBaseDirectory declaration issue
Dear TM1 Team,
In all TM1 models I built (over the last 13 years), I used multiple data directories feature to separate, Code, Config and Data objects in different folders.
Under the main TM1 model directory, I use the following typical subfolder structure as standard;
D:\TM1\[ServiceName]\TM1Code\
D:\TM1\[ServiceName]\TM1Config\
D:\TM1\[ServiceName]\TM1Data\
D:\TM1\[ServiceName]\TM1LogFiles\
D:\TM1\[ServiceName]\TM1MasterDataFiles\
D:\TM1\[ServiceName]\TM1SourceFiles\
D:\TM1\[ServiceName]\TM1ExportFiles\
In my tm1s.cfg file, I have the following definition for directory settings:
DataBaseDirectory=D:\TM1\[ServiceName]\TM1Data;D:\TM1\[ServiceName]\TM1Config;D:\TM1\[ServiceName]\TM1Code
LoggingDirectory=D:\TM1\[ServiceName]\TM1LogFiles
TM1Data folder is the default model directory where all new objects are stored. Hence if I do not move any object to TM1Code or TM1Config folders, then they stay in this folder. Remaining objects are treated as data objects for the environment.
TM1Code folder is used for code object that I can move from NonProd to PROD environment, such as pro files…etc. When I create a new pro file, I move it from TM1Data folder to TM1Code folder manually in Dev environment, and then any updates to this process from the UI updates the file in TM1Code folder, without any need to restart the service. If I need to deploy code changes from one environment to the other, then I only need to move objects from this folder only.
TM1Config folder is used for objects that needs to be environment specific by nature. I move any such object from TM1Data folder to TM1Config folder when the object is initially created, such as tm1s.cfg, }Clients.dim, }ClientSettings.cub, }ClientsGroups.cub or }ClientProperties.cub…etc. In case I need to move data and code from one environment to the other, I do not need to move anything in this folder to keep basic environment configuration unchanged in each environment.
This method works very well in a way that we are able to separate, Code, Config and Data objects of the TM1 Model. Therefore having multiple database directories feature is very important for us. If this feature will be removed, then it will be very useful if additional parameters are introduced such as CodeBaseDirectory and ConfigDirectory…etc.
Having Code, Config, and Data objects separated is very important in deploying model changes between environments in a programmatic way. We only move TM1Code folder objects between environments for the sake of keeping security and data separate between Prod, Dev and Test environments.
Hope removal of multiple database directories feature is reconsidered based on the explanations above.
Regards,
Mücahit
------------------------------
Mucahit Erdal
Original Message:
Sent: Mon September 11, 2023 01:25 PM
From: Médard ATANNON
Subject: Multiple DataBaseDirectory declaration issue
Hello Everyone,
Hello Hubert and Stuart,
I'm taking the liberty of relaunching this subject to find out what's in the pipes about it. Are we moving towards not supporting multiple DatabaseDirectories as Stuart said, or is the notion of modules/libraries proposed by Hubert?
In the below link, the updated precised that DataBaseDirectory parameter is not applicable to Planning Analytics Engine only and we can read "You can list multiple data directories by separating them with semicolons."
https://www.ibm.com/docs/en/planning-analytics/2.0.0?topic=file-databasedirectory
Thank you for your time and regards,
------------------------------
MEDARD ATANNON
Original Message:
Sent: Fri July 30, 2021 01:41 AM
From: Christoph Hein
Subject: Multiple DataBaseDirectory declaration issue
I am all in for the submodule approach and would love to see it in v11 as well. Would help us with our 100+ internal databases a lot. It is hard to maintain standards if you have to deploy every change on them to every server. (We are not using a shared folder approach like Robert).
Robert! I would love to see your infrastructure as well. ;-) Can share mine as a bargain. :-D
------------------------------
TM1 Fanboy and outspoken advocate for functional databases
Christoph Hein
Deutsche Bahn - Frankfurt
Original Message:
Sent: Thu July 29, 2021 05:05 AM
From: Hubert Heijkers
Subject: Multiple DataBaseDirectory declaration issue
<LOL>
Pity, TM1 12 would probably be able to help you with a bunch of challenges, but I reckon that this containerized world comes with it's own set, non-TM1 related, challenges as well ;-)
GIT integration already exists in TM1 v11, and whilst that version is in LTSR and we are focussed on the future and as such v12, such an extension to our GIT support could solve quite a number of challenges for customers, like yourself, that might be stuck on v11 until it goes out of support (not before the end of 2024 I've been told). I.o.w. I wouldn't be surprised, if at all sensibly possible, if we were to add sub-module support, that we'd be adding it in a future v11 release as well <just don't tell anybody that yet;->.
Happy to discuss, talk about advantages of moving to the next generation when it becomes available, and learn about your environment as well. I'm, traditionally, taking the month of September off but happy to get in touch and plan a meeting after I return reenergized;-!
Merci!
------------------------------
Hubert Heijkers
Original Message:
Sent: Wed July 28, 2021 07:13 AM
From: Robert MILLI
Subject: Multiple DataBaseDirectory declaration issue
Hi Hubert,
Thanks for your answer. Always very instructive :-)
For sure, a way to handle module/sub-module/libraries will be a much more smart way to handle common standard code/procedures than mutliple data directories...
The key point, for me and the organization I'm working for, is the ability, for these standard material, to update it once, deploy it every where so somekind of GIT repository should do the job and ideally, these material shouldn't be updatable in any of the targeted database.
My IT team has provided a GIT server but to be transparent, I do not know, at this time, how to use it in the best way.
You're talking of "containerized" TM1 server. If it's TM1 V12 related, I'm almost convinced that I'll be retired prior the V12 deployment in the company I work for 😂
If it comes with the current TM1 version, V11, it will have a so important impact on our infra that I'm already frightened !
We've got a huge technical infrastructure to support our TM1 applications. Several tens of Windows server are dealing individually softwares such as PAL, PASS, PAW, PAX, Adminhost... We've got DRP servers. We've tried to manage load balancing everywhere it was possible. All of our infra reside in a highly secured vlan. We're using our own security certificates...
When I was a consultant, most of the time, customers I was visiting were dealing with a single Windows server hosting all data and presentation layers. Today, I'm dealing with many PAL servers...
If you're curious about what we've got and want to see it more in details, I can settle a meeting sometime in september, just let me know.
Hartelijkheid,
------------------------------
Bob
Original Message:
Sent: Wed July 28, 2021 05:54 AM
From: Hubert Heijkers
Subject: Multiple DataBaseDirectory declaration issue
Hi Ifthen,
I'm thinking about a kind of 'opt-in' model where you, as part of the project specification, 'include' objects from the sub-modules. That way these could be regular TM1 source repos from which you could 'borrow/inherit' objects.
Those repos, the one a database would be sourcing from as well as the referenced sub-modules/libraries, could all be maintained separately, but that leaves questions like if we even would allow changes to those objects sourced from sub-modules. And presuming we wouldn't (read: changes would have to come in through changes to the respective sub-modules/libraries), which I think is what Bob would like to see as well, would that apply to data as well?
Again, just ideas at this point, but an interesting topic/extension to explore further.
Cheers,
------------------------------
Hubert Heijkers
Original Message:
Sent: Tue July 27, 2021 09:01 AM
From: Ifthen CHERMAK
Subject: Multiple DataBaseDirectory declaration issue
Hi Hubert,
If the notion of modules/libraries includes all TM1 objects, this would definitely solve the mentioned use case for multiple database directories.
Regards,
Ifthen
------------------------------
Ifthen CHERMAK
Original Message:
Sent: Tue July 27, 2021 05:43 AM
From: Hubert Heijkers
Subject: Multiple DataBaseDirectory declaration issue
Hi Bob,
Parsing indeed would be the simplest part of it but the implications of just saying you have multiple database directories weren't necessarily 100% clear nor what the ramifications were of modification of objects, provided one could. That aside, and on top of what Stuart already said about the existing cloud offering, the future, containerized cloud-native, version of TM1 you don't even have the notion of 'database directories' any longer, albeit there is most definitely still disk storage that ultimately underpins the system.
The use-case that you refer to however has come up before, not only as the only real example that I've ever seen for using multiple database directories, but also in the context of our GIT integration that's been out several years now. In that context, the ask has been if we'd be able to support 'modules', libraries as you are talking about, and be able to reference/include them as part of a model definition using GIT sub-modules. Ironically the GIT sub-module addition, effectively just a reference to another repository, is the simple part of this but the notion of proper libraries/modules of processes, and perhaps chores, could be an interesting addition to TM1.
Out of curiosity, would this notion of modules/libraries solve all your cases for multiple database directories?
Cordialement,
------------------------------
Hubert Heijkers
Original Message:
Sent: Tue July 27, 2021 02:39 AM
From: Robert MILLI
Subject: Multiple DataBaseDirectory declaration issue
Hello Stuart,
I should say that I'm quite puzzled by your answer !
Parsing a muli-value parameter didn't seem a big deal for such a company as IBM.
We're using this feature on the backend part as it's, IMHO, the best way to ensure that a code (mainly TI) which is common to a bunch of TM1/PAL server could be isolated in it's own folder and easily maintained as a dedicated library. I was even searching to have a way to ensure that nobody, including admins, could update the contents of our common objects libraries.
Regards,
------------------------------
Bob
Original Message:
Sent: Mon July 26, 2021 04:29 PM
From: STUART KING
Subject: Multiple DataBaseDirectory declaration issue
Hi Ifthen and Christoph,
I can confirm we never planned to support multiple DatabaseDirectories with the PAA agent. PAA was born in Planning Analytics on Cloud and then adopted to local. In our cloud environment the DatabaseDirectory is always a single directory (cannot be reconfigured by our customers). During the adoption for Planning Analytics local we made the determination that multiple DatabaseDirectories was too uncommon to implement support for. We are going to update our docs to mention this limitation.
------------------------------
Stuart King
IBM Planning Analytics Offering Manager
Original Message:
Sent: Wed July 21, 2021 07:50 AM
From: Christoph Hein
Subject: Multiple DataBaseDirectory declaration issue
I guess that this is one of those features... It was once implemented, sparsely used and now it is only partly supported.
I know that you can do it but I never used it in a production system. I am guessing that they either forgot or decided to not support this functionality in the cloud (aka PAW).
------------------------------
TM1 Fanboy and outspoken advocate for functional databases
Christoph Hein
Deutsche Bahn - Frankfurt
Original Message:
Sent: Mon July 19, 2021 05:48 AM
From: Ifthen CHERMAK
Subject: Multiple DataBaseDirectory declaration issue
Hello,
I usually define several directories for my servers by passing a value to the server configuration like the one below.
DataBaseDirectory=...\Data ;..\Lib\Bedrock ;..\Lib\Izi
This is very useful because it allows to separate the objects linked to an application from those of a library.
Since the introduction of the possibility to upload a local file to a server (to ..\Data\model_upload), I have observed a problem with the interpretation of the DataBaseDirectory value. It seems that it is considered a unique directory and causes problems when creating the model_upload directory.
Has anyone experienced the same issue as mine?
Do you consider, as I do, that this is a regression?
PAL version 2.0.9
PAW version 2.0.63
------------------------------
Ifthen CHERMAK
------------------------------
#PlanningAnalyticswithWatson