Planning Analytics

 View Only
Expand all | Collapse all

Multiple DataBaseDirectory declaration issue

  • 1.  Multiple DataBaseDirectory declaration issue

    Posted Mon July 19, 2021 10:37 AM
    Hello,

    I usually define several directories for my servers by passing a value to the server configuration like the one below.

    DataBaseDirectory=...\Data ;..\Lib\Bedrock ;..\Lib\Izi

    This is very useful because it allows to separate the objects linked to an application from those of a library.

    Since the introduction of the possibility to upload a local file to a server (to ..\Data\model_upload), I have observed a problem with the interpretation of the DataBaseDirectory value. It seems that it is considered a unique directory and causes problems when creating the model_upload directory.

    Has anyone experienced the same issue as mine?
    Do you consider, as I do, that this is a regression?

    PAL version 2.0.9
    PAW version 2.0.63

    ------------------------------
    Ifthen CHERMAK
    ------------------------------

    #PlanningAnalyticswithWatson


  • 2.  RE: Multiple DataBaseDirectory declaration issue

    IBM Champion
    Posted Wed July 21, 2021 07:50 AM
    I guess that this is one of those features... It was once implemented, sparsely used and now it is only partly supported. 
    I know that you can do it but I never used it in a production system. I am guessing that they either forgot or decided to not support this functionality in the cloud (aka PAW).

    ------------------------------
    TM1 Fanboy and outspoken advocate for functional databases

    Christoph Hein
    Deutsche Bahn - Frankfurt
    ------------------------------



  • 3.  RE: Multiple DataBaseDirectory declaration issue

    Posted Mon July 26, 2021 04:30 PM
    Hi Ifthen and Christoph,

    I
    can confirm we never planned to support multiple DatabaseDirectories with the PAA agent.   PAA was born in Planning Analytics on Cloud and then adopted to local.   In our cloud environment the DatabaseDirectory is always a single directory (cannot be reconfigured by our customers).   During the adoption for Planning Analytics local we made the determination that multiple DatabaseDirectories was too uncommon to implement support for.   We are going to update our docs to mention this limitation.





    ------------------------------
    Stuart King
    IBM Planning Analytics Offering Manager
    ------------------------------



  • 4.  RE: Multiple DataBaseDirectory declaration issue

    Posted Tue July 27, 2021 02:39 AM
    Hello Stuart,
    I should say that I'm quite puzzled by your answer !
    Parsing a muli-value parameter didn't seem a big deal for such a company as IBM.
    We're using this feature on the backend part as it's, IMHO, the best way to ensure that a code (mainly TI) which is common to a bunch of TM1/PAL server could be isolated in it's own folder and easily maintained as a dedicated library. I was even searching to have a way to ensure that nobody, including admins, could update the contents of our common objects libraries.
    Regards,



    ------------------------------
    Bob
    ------------------------------



  • 5.  RE: Multiple DataBaseDirectory declaration issue

    Posted Tue July 27, 2021 05:44 AM
    Hi Bob,

    Parsing indeed would be the simplest part of it but the implications of just saying you have multiple database directories weren't necessarily 100% clear nor what the ramifications were of modification of objects, provided one could. That aside, and on top of what Stuart already said about the existing cloud offering, the future, containerized cloud-native, version of TM1 you don't even have the notion of 'database directories' any longer, albeit there is most definitely still disk storage that ultimately underpins the system.

    The use-case that you refer to however has come up before, not only as the only real example that I've ever seen for using multiple database directories, but also in the context of our GIT integration that's been out several years now. In that context, the ask has been if we'd be able to support 'modules', libraries as you are talking about, and be able to reference/include them as part of a model definition using GIT sub-modules. Ironically the GIT sub-module addition, effectively just a reference to another repository, is the simple part of this but the notion of proper libraries/modules of processes, and perhaps chores, could be an interesting addition to TM1.

    Out of curiosity, would this notion of modules/libraries solve all your cases for multiple database directories?

    Cordialement,

    ------------------------------
    Hubert Heijkers
    ------------------------------



  • 6.  RE: Multiple DataBaseDirectory declaration issue

    Posted Tue July 27, 2021 09:01 AM
    Hi Hubert,

    If the notion of modules/libraries includes all TM1 objects, this would definitely solve the mentioned use case for multiple database directories.

    Regards,
    Ifthen

    ------------------------------
    Ifthen CHERMAK
    ------------------------------



  • 7.  RE: Multiple DataBaseDirectory declaration issue

    Posted Wed July 28, 2021 05:55 AM
    Hi Ifthen,

    I'm thinking about a kind of 'opt-in' model where you, as part of the project specification, 'include' objects from the sub-modules. That way these could be regular TM1 source repos from which you could 'borrow/inherit' objects.
    Those repos, the one a database would be sourcing from as well as the referenced sub-modules/libraries, could all be maintained separately, but that leaves questions like if we even would allow changes to those objects sourced from sub-modules. And presuming we wouldn't (read: changes would have to come in through changes to the respective sub-modules/libraries), which I think is what Bob would like to see as well, would that apply to data as well?

    Again, just ideas at this point, but an interesting topic/extension to explore further.

    Cheers,

    ------------------------------
    Hubert Heijkers
    ------------------------------



  • 8.  RE: Multiple DataBaseDirectory declaration issue

    Posted Wed July 28, 2021 07:13 AM
    Hi Hubert,
    Thanks for your answer. Always very instructive :-)
    For sure, a way to handle module/sub-module/libraries will be a much more smart way to handle common standard code/procedures than mutliple data directories...
    The key point, for me and the organization I'm working for, is the ability, for these standard material, to update it once, deploy it every where so somekind of GIT repository should do the job and ideally, these material shouldn't be updatable in any of the targeted database.
    My IT team has provided a GIT server but to be transparent, I do not know, at this time, how to use it in the best way.

    You're talking of "containerized" TM1 server. If it's TM1 V12 related, I'm almost convinced that I'll be retired prior the V12 deployment in the company I work for 😂
    If it comes with the current TM1 version, V11, it will have a so important impact on our infra that I'm already frightened !
    We've got a huge technical infrastructure to support our TM1 applications. Several tens of Windows server are dealing individually softwares such as PAL, PASS, PAW, PAX, Adminhost... We've got DRP servers. We've tried to manage load balancing everywhere it was possible. All of our infra reside in a highly secured vlan. We're using our own security certificates...
    When I was a consultant, most of the time, customers I was visiting were dealing with a single Windows server hosting all data and presentation layers. Today, I'm dealing with many PAL servers...
    If you're curious about what we've got and want to see it more in details, I can settle a meeting sometime in september, just let me know.

    Hartelijkheid,

    ------------------------------
    Bob
    ------------------------------



  • 9.  RE: Multiple DataBaseDirectory declaration issue

    Posted Thu July 29, 2021 05:05 AM
    <LOL>

    Pity, TM1 12 would probably be able to help you with a bunch of challenges, but I reckon that this containerized world comes with it's own set, non-TM1 related, challenges as well ;-)

    GIT integration already exists in TM1 v11, and whilst that version is in LTSR and we are focussed on the future and as such v12, such an extension to our GIT support could solve quite a number of challenges for customers, like yourself, that might be stuck on v11 until it goes out of support (not before the end of 2024 I've been told). I.o.w. I wouldn't be surprised, if at all sensibly possible, if we were to add sub-module support, that we'd be adding it in a future v11 release as well <just don't tell anybody that yet;->.

    Happy to discuss, talk about advantages of moving to the next generation when it becomes available, and learn about your environment as well. I'm, traditionally, taking the month of September off but happy to get in touch and plan a meeting after I return reenergized;-!

    Merci!


    ------------------------------
    Hubert Heijkers
    ------------------------------



  • 10.  RE: Multiple DataBaseDirectory declaration issue

    IBM Champion
    Posted Fri July 30, 2021 01:41 AM
    I am all in for the submodule approach and would love to see it in v11 as well. Would help us with our 100+ internal databases ​a lot. It is hard to maintain standards if you have to deploy every change on them to every server. (We are not using a shared folder approach like Robert).
    Robert! I would love to see your infrastructure as well. ;-) Can share mine as a bargain. :-D

    ------------------------------
    TM1 Fanboy and outspoken advocate for functional databases

    Christoph Hein
    Deutsche Bahn - Frankfurt
    ------------------------------



  • 11.  RE: Multiple DataBaseDirectory declaration issue

    Posted Tue September 12, 2023 11:12 AM

    Hello Everyone,

    Hello Hubert and Stuart,

    I'm taking the liberty of relaunching this subject to find out what's in the pipes about it. Are we moving towards not supporting multiple DatabaseDirectories as Stuart said, or is the notion of modules/libraries proposed by Hubert?

    In the below link, the updated precised that DataBaseDirectory parameter is not applicable to Planning Analytics Engine only and we can read "You can list multiple data directories by separating them with semicolons."

    https://www.ibm.com/docs/en/planning-analytics/2.0.0?topic=file-databasedirectory

    Thank you for your time and regards,



    ------------------------------
    MEDARD ATANNON
    ------------------------------



  • 12.  RE: Multiple DataBaseDirectory declaration issue

    Posted Wed September 13, 2023 04:42 AM
    Edited by Mucahit Erdal Wed September 13, 2023 04:56 AM

    Dear TM1 Team,

    In all TM1 models I built (over the last 13 years), I used multiple data directories feature to separate, Code, Config and Data objects in different folders.

    Under the main TM1 model directory, I use the following typical subfolder structure as standard;

    D:\TM1\[ServiceName]\TM1Code\

    D:\TM1\[ServiceName]\TM1Config\

    D:\TM1\[ServiceName]\TM1Data\

    D:\TM1\[ServiceName]\TM1LogFiles\

    D:\TM1\[ServiceName]\TM1MasterDataFiles\

    D:\TM1\[ServiceName]\TM1SourceFiles\

    D:\TM1\[ServiceName]\TM1ExportFiles\

    In my tm1s.cfg file, I have the following definition for directory settings:

    DataBaseDirectory=D:\TM1\[ServiceName]\TM1Data;D:\TM1\[ServiceName]\TM1Config;D:\TM1\[ServiceName]\TM1Code

    LoggingDirectory=D:\TM1\[ServiceName]\TM1LogFiles

    TM1Data folder is the default model directory where all new objects are stored. Hence if I do not move any object to TM1Code or TM1Config folders, then they stay in this folder. Remaining objects are treated as data objects for the environment.

    TM1Code folder is used for code object that I can move from NonProd to PROD environment, such as pro files…etc. When I create a new pro file, I move it from TM1Data folder to TM1Code folder manually in Dev environment, and then any updates to this process from the UI updates the file in TM1Code folder, without any need to restart the service. If I need to deploy code changes from one environment to the other, then I only need to move objects from this folder only.

    TM1Config folder is used for objects that needs to be environment specific by nature. I move any such object from TM1Data folder to TM1Config folder when the object is initially created, such as tm1s.cfg, }Clients.dim, }ClientSettings.cub, }ClientsGroups.cub or }ClientProperties.cub…etc. In case I need to move data and code from one environment to the other, I do not need to move anything in this folder to keep basic environment configuration unchanged in each environment. 

    This method works very well in a way that we are able to separate, Code, Config and Data objects of the TM1 Model. Therefore having multiple database directories feature is very important for us. If this feature will be removed, then it will be very useful if additional parameters are introduced such as CodeBaseDirectory and ConfigDirectory…etc.

    Having Code, Config, and Data objects separated is very important in deploying model changes between environments in a programmatic way. We only move TM1Code folder objects between environments for the sake of keeping security and data separate between Prod, Dev and Test environments.

    Hope removal of multiple database directories feature is reconsidered based on the explanations above.

    Regards,

    Mücahit



    ------------------------------
    Mucahit Erdal
    ------------------------------



  • 13.  RE: Multiple DataBaseDirectory declaration issue

    IBM Champion
    Posted Wed September 13, 2023 05:42 AM
    Edited by Edward Stuart Wed September 13, 2023 05:49 AM

    # Note to self to read the whole thread before replying! Pre v12 this holds a different set of challenges as is discussed. The potential to utilise most programming languages to replicate ExecuteHttpRequest exists but is a stopgap ahead of v12 adoption

    I believe the opportunity we have with the Planning Analytics Engine will make our historical processes seem arcane, 

    I've yet to test the theory and I have a blog post waiting to flesh this out but I can see something along the lines of:

    Process: Copy.Object

    This takes parameters for Object type (Process, Dimension, Chore, Rules etc..), Object Name, Source Server and Target Server

    Uses ExecuteHTTPRequest to store the details of the Object to be Copied

    https://www.ibm.com/docs/en/planning-analytics/2.0.0?topic=functions-executehttprequest

    • This process can be run standalone to migrate Process A from Server A to Server B
    • This process can also be run as a worker process to a master process

    Process: Copy.Selected.Objects

    This loops the Control dimensions (}Clients, }Dimensions, }Cubes, }Processes etc..) which have an attribute to flag that the object is to be moved

    For each instance were a flag is set then run Process Copy.Object

    Server Control Cube:

    This Cube will hold the details for each Server in the environment, each server will need to be manually added as expected but again a ExecuteHTTPRequest call to "http://<ServerName>:<PortNumber>/api/v1/Servers" will pull back data for those selected servers

    This needs testing and appropriate due diligence on setting the flags for objects to be copied, considerations also need to be made on backups of existing objects before copyover should anything go awry/ mistakes be made.

    Of course, we also have the alternative to set each server up against a Git Repository and pull/ push objects between servers direct from Github.

    I can see processes being created to check the differences between two servers and creating automated migration packages to keep models in sync. 

    This do not scratch the surface of what is possible with ExecuteHTTPRequest to send objects, data, messages to third party tools to fulfill your workflow requirements.



    ------------------------------
    Edward Stuart
    ------------------------------



  • 14.  RE: Multiple DataBaseDirectory declaration issue

    IBM Champion
    Posted Wed September 13, 2023 06:56 AM

    Thanks for the useful insights Ed!



    ------------------------------
    George Tonkin
    Business Partner
    MCI Consultants
    Johannesburg
    ------------------------------



  • 15.  RE: Multiple DataBaseDirectory declaration issue

    Posted Wed September 13, 2023 09:31 AM

    Hence there will be no access to the filesystem in v12, this functionality is gone either way. No more copying individual files from one server to the other. 

    Not my opinion, just a fact. Migration to v12 will lead to a lot of change of our existing ways of doing things. For better or for worse. 



    ------------------------------
    Christoph Hein TM1 Fanboy
    ------------------------------



  • 16.  RE: Multiple DataBaseDirectory declaration issue

    Posted Wed September 13, 2023 10:09 AM

    Chris,

     

    So how will this be done in future, as developers still need access to those files for release from Dev to Prod environments or for developing TI's that work on Flatfiles which sit on the server etc.

     

    regards

     

    Phil Lidstone

    Finance Data and Systems Manager | UK Strategic Finance

    Zurich Insurance Company Ltd

     

    ******************* PLEASE NOTE ******************* This message, along with any attachments, may be confidential or legally privileged. It is intended only for the named person(s), who is/are the only authorized recipients. If this message has reached you in error, kindly destroy it without review and notify the sender immediately. Thank you for your help.
    -------------------------------------------------

    All of the companies listed below are part of a group of companies of which the ultimate parent company is Zurich Insurance Group Ltd, a company registered in Switzerland No. CH-020.3.023.083-6/a. Zurich is a trading name of this group of companies.

    Zurich Insurance Company Ltd. A public limited company incorporated in Switzerland. Registered in the Canton of Zurich, No. CHE-105.833.114, registered offices at Mythenquai 2, 8002 Zurich. UK Branch registered in England and Wales no BR000105. UK Branch Head Office: The Zurich Centre, 3000 Parkway, Whiteley, Fareham, Hampshire PO15 7JZ.

    Zurich Insurance Company Ltd is authorised and regulated in Switzerland by the Swiss Financial Market Supervisory Authority FINMA. Authorised by the Prudential Regulation Authority. Subject to regulation by the Financial Conduct Authority and limited regulation by the Prudential Regulation Authority. Details about the extent of our regulation by the Prudential Regulation Authority are available from us on request. Our firm reference number is 959113.

    Zurich Insurance plc. A public limited company incorporated in Ireland. Registration No.13460. Registered Office: Zurich House, Frascati Road, Blackrock, County Dublin, A94 X9Y3, Ireland. UK branch registered in England and Wales Registration No. BR7985. UK Branch Head Office: The Zurich Centre, 3000 Parkway, Whiteley, Fareham, Hampshire PO15 7JZ

    Zurich Insurance plc is authorised and regulated by the Central Bank of Ireland. Authorised by the Prudential Regulation Authority. Subject to regulation by the Financial Conduct Authority and limited regulation by the Prudential Regulation Authority. Details about the extent of our regulation by the Prudential Regulation Authority are available from us on request. Our firm reference number is 203093.

    Zurich Insurance plc is registered with the Portuguese Insurance and Pension Funds Authority (Autoridade de Supervisão de Seguros e Fundos de Pensões) under no. 4402 to operate in Portugal under the freedom to provide services

    Zurich Insurance plc is authorised to operate in Spain under the right of establishment through its branch Zurich Insurance plc, Sucursal en España: Tax ID (NIF) W0072130H, registered address Paseo de la Castellana, 81, planta 22, 28046 Madrid, and registered in the Directorate General of Insurance and Pension Funds Administrative Register with code no. E0189. In enforcement of Article 123 of Royal Decree 1060/2015, of 20 November, on the organisation, supervision and solvency of insurers and reinsurers, it is stated that in the event of the insurer’s liquidation, Spanish liquidation regulations do not apply, but rather those of Ireland.

    Each of the following companies has its registered office at Unity Place, 1 Carfax Close, Swindon, SN1 1AP.

    Zurich Financial Services (UKISA) Limited is registered in England and Wales No. 01860680.

    Zurich Assurance Ltd is registered in England and Wales No.02456671. Zurich Assurance Ltd is authorised by the Prudential Regulation Authority and regulated by the Financial Conduct Authority and the Prudential Regulation Authority. Zurich is used as a trading name of Zurich Assurance Ltd.

    Zurich Intermediary Group Limited is registered in England and Wales No. 01909111 and is authorised and regulated by the Financial Conduct Authority.

    Any views or opinions expressed in this email are those of the author only. Communications may be monitored to improve our service and for security and regulatory purposes.

    The information contained in this message is confidential and may be legally privileged. This message is intended for the addressee(s) only. If you are not the intended recipient, please do not read, copy or otherwise use it and do not disclose it to anyone else. Please notify the sender of the delivery error and then delete the message from your system. Thank you for your assistance.





  • 17.  RE: Multiple DataBaseDirectory declaration issue

    IBM Champion
    Posted Wed September 13, 2023 10:54 AM

    There is a file manager for loading data from flat files:

    https://www.ibm.com/docs/en/planning-analytics/2.0.0?topic=2022-manage-files-file-manager

    and you'll likely want to get familiar with FTPS for automating uploads of files to the cloud/ container to run via TI processes in the standard way.

    A potential enhancement is to see if you are able to access local network shares via the Rest Api (for example if you use Microsoft Office/ Sharepoint - https://learn.microsoft.com/en-us/graph/use-the-api) to cut FTPS out of the loop and go direct to files in file explorer. 

    As for Dev to Prod migrations, as mentioned above it is not going to be via File Explorer. Therefore it 'has' to be via Rest Api, however, ExecuteHTTPRequest will facilitate this without the need to use third party applications and/ or pushing models into Github. I have tested this and it works well but it is all about your workflow and how you want it to be implemented. 



    ------------------------------
    Edward Stuart
    ------------------------------



  • 18.  RE: Multiple DataBaseDirectory declaration issue

    Posted Wed September 13, 2023 12:39 PM

    Migrations will no longer be manual. They will need to be built into a CI/CD process. You can use the built in GIT functionality, or create your own. The days of manually copying files will be gone, finally. 



    ------------------------------
    Ryan Clapp
    ------------------------------



  • 19.  RE: Multiple DataBaseDirectory declaration issue

    Posted Wed September 13, 2023 01:51 PM

    Hi Médard,

    Let me start by saying that support for multiple database directories, as described in the tm1s.cfg configuration documentation referenced by you, is and will continue to be supported in TM1 v11. 

    Stuart's comments were wrt PAA. I'll leave it up to Stuart to comment if anything changed in that perspective.

    But, unless you are running Planning Analytics on Cloud on which the one database directory is what that 'solution' offers, you can still simply go into your tm1s.cfg file of your TM1 server and set your multiple database directories.

    TM1 v12 is a whole different/separate topic. Think of it as not having a file system but a lot of means to deal with what you need to do which we could still improve with the notion of modules/libraries in the GIT integration feature (which would also apply to v11) but we that's still on the highly desired features list (it's not forgotten yet;-).

    Cheers,



    ------------------------------
    Hubert Heijkers
    STSM, Program Director TM1 Functional Database Technology and OData Evangelist
    ------------------------------