IBM i Global

IBM i Global

Connect, learn, share, and engage with IBM Power.

 View Only
Expand all | Collapse all

Database modernization

  • 1.  Database modernization

    Posted Tue January 23, 2024 08:34 AM

    In traditional DDS coding for a database, you can use the following to create an exact duplicate of a file by another name:

         A          R EDR832BH1                 FORMAT(EDP832BH1)
          *
         A          K H1CMPY
         A          K H1CST1
         A          K H1ID#                                              

    And using DDL to define the same file duplication, you can do the following:

    CREATE OR REPLACE TABLE EDP832BPH1(
    LIKE EDP832BH1
    )
    RCDFMT EDR832BH1;

    LABEL ON TABLE EDP832BPH1 IS 'EDI B2B 832 HEADER FILE, PURGED DATA'    ;

    ALTER TABLE EDP832BPH1
    PRIMARY KEY (H1CMPY, H1CST1, H1ID#);                   

      Can anyone explain why the SQL table version does not end up with the same level Id as the parent, and also has no column headings or field text for each field?  It is difficult to understand why these two coding techniques do not produce the same result, and if trying to convert to using the tables instead of the file with native I/O you have to recompile all of the programs that depend on the level Id. 

    Thank you.



    ------------------------------
    Mike Overlander
    ------------------------------


  • 2.  RE: Database modernization

    Posted Tue January 23, 2024 09:02 AM

    Hi Mike,

    AFAIK you shouldn't specify the RCDFMT keyword on the CREATE ... LIKE ... statement.

    If you specify RCDFMT you always get a new record format level id - if you don't specify it, the record format will be the same, und the record format level id will match the original table.

    HTH

    Daniel



    ------------------------------
    Daniel Gross
    ------------------------------



  • 3.  RE: Database modernization

    Posted Tue January 23, 2024 09:15 AM

    Daniel, 

      Thank you, good thought.  But I just tested this and it ended up with a record format name that is the same as the table name instead of that of the parent, and the level Id is still not the same as the parent file.  It has the same record layout, but still has no column headings or field text for each field either, where DDS duplicates have all this information.



    ------------------------------
    Mike Overlander
    ------------------------------



  • 4.  RE: Database modernization

    Posted Wed January 24, 2024 04:26 AM

    Hi Mike,

    could you try with

    ... LIKE ...

    and not with

    ... ( LIKE ....)

    I think this could be the reason.

    With (LIKE ...) you IMHO create a new record format, that is "like" the other. 

    With LIKE ... you define the table exactly like the other table incl. all attributes. I use this to create the temporal history tables - and AFAIK these share the same record format incl. the same record format level id. 

    Please try and tell me what happens. 

    HTH

    Daniel



    ------------------------------
    Daniel Gross
    ------------------------------



  • 5.  RE: Database modernization

    Posted Tue January 23, 2024 09:13 PM
    Edited by Satid Singkorapoom Tue January 23, 2024 09:19 PM

    Dear Mike 

    >>>>>   It is difficult to understand why these two coding techniques do not produce the same result,  It is difficult to understand why these two coding techniques do not produce the same result,  <<<<<

    The worlds of IBM i DDS and SQL are really different ones.  One example of the stark difference is that data verification in DDS PF is done at read time as opposed to write time which makes it possible to write corrupted data into DDS PF and be frustrated with the error when reading it and this was what bugged me a lot in my early days working with AS/400.  An SQL table does data verification at write time and therefore prevents corrupted data to be in there. It also means reading a lot of data from SQL table is faster.   The developers of SQL world for DB2 for i try their best to accommodate features from DDS world but they can only do so much. This article is one example: SQL indexes and native I/O – no contradiction at  https://developer.ibm.com/articles/i-sql-indexs-and-native-io/.  

    We are creature of familiarity but when in Rome, we should do like the Romans do.  I believe you do realize that there is no concept of level ID in SQL world and they are doing fine without it.   However, there is a possible solution to level ID issue I learned from the expert which is to change all your programs to access only logical files so that when changes are made to the underlining physical files, there are no level ID changes at the LFs to frustrate the programs.    



    ------------------------------
    Chance favors only the prepared mind.
    -- Louis Pasteur
    ------------------------------
    Satid S.
    ------------------------------



  • 6.  RE: Database modernization

    Posted Wed January 24, 2024 02:11 AM

    DDL does not support Keys! 

    And since the key in your physical file is not unique it can not even generate a primary Key constraint.

    The result might be a table with the same data, but without key! so, it is not usable for native I/O with keyed access.

    If you want to modernize your tables you should:

    1. Check whether there are keys in your physical file. If so generate an additional logical file (or better an index) replace the keyed access in your programs with a keyed access on the new logical or index (BTW an SQL index can be used with native I/O as a keyed logical file)
    2. Generate the SQL Script for the physical file with ACS (Generate SQL) or the SQL Stored Procedure GENERATE_SQL (make sure you selected CREATE OR REPLACE or the SQL Script includes CREATE OR REPLACE TABLE). BTW everything that cannot be converted is commented, so you may double check what is missing
    3. After everything is okay just run the SQL Script

    If nothing more is changed the Format Level will be the same



    ------------------------------
    Birgitta Hauser
    Database and Software Engineer
    Selfemployed - Modernization-Education-Consulting on IBM i
    Kaufering
    +49 170 5269964
    ------------------------------



  • 7.  RE: Database modernization

    Posted Wed January 24, 2024 04:29 AM

    Yes - instead of (non unique) DDS keys, you would use CREATE INDEX - and you can use these indexes for native RPG I/O like any other DDS LF. 

    But you should really define at least one "primary key" on the table - preferably with a unique row id. 



    ------------------------------
    Daniel Gross
    ------------------------------



  • 8.  RE: Database modernization

    Posted Wed January 24, 2024 05:08 AM

    >>But you should really define at least one "primary key" on the table - preferably with a unique row id. 

    You can only define a single Primary Key (Constraint) but additionally multiple Unique Key (Constraints or Indexes).



    ------------------------------
    Birgitta Hauser
    Database and Software Engineer
    Selfemployed - Modernization-Education-Consulting on IBM i
    Kaufering
    +49 170 5269964
    ------------------------------



  • 9.  RE: Database modernization

    Posted Tue January 30, 2024 01:51 PM

    ... Also, you have the option of creating a self generated primary key (something like 

    create table mytable (

    my_self_generated_primary_key integer generated always as identity,

    other_colum varchar(128).

    ...

    )

    I think that option was not available in DDS.  And there are more: you can populate a column with a generated transaction timestamp (both at insertion or at update).  

    More details in CREATE TABLE - IBM Documentation



    ------------------------------
    Daniel Jose Lema Guanziroli
    ------------------------------



  • 10.  RE: Database modernization

    Posted Wed January 24, 2024 07:45 AM

    I've learned that if I disagree with Birgitta then I need to reread her post again.  It took me a minute to notice that your DDS does not include the UNIQUE keyword.  Therefore the primary key constraint would be different.

    But, one thing I am confused about is that I think she is wrong that you cannot use a primary key constraint as a key in HLL's with native access.  Granted, there are newer services but one old technique we used was DSPUSRPRF to an OUTFILE then simply run ADDPFCST on it to make the userid column a primary key.  After that it was quite easy for an RPG program to do a CHAIN to it on user id.

    BTW a continued use for this old method is that the users using this do not need access to the user profiles to chain to this outfile.  The outfile is regenerated every night. Oh, I'm sure there may be security concerns like a publicly available list of your user profiles.  But most people know a backdoor to getting a list of users on your system without any special authority.



    ------------------------------
    Robert Berendt IBMChampion
    ------------------------------



  • 11.  RE: Database modernization

    Posted Tue January 30, 2024 01:43 PM

    I am almost sure many people has written answers to this question.

    DDL does support keys, but they are called indexes.  You can create indexes as part of the CREATE TABLE command or using commands CREATE INDEX or CREATE UNIQUE INDEX.  You can also explore different type of indexes.  As far as I know, DDS only support btree indexes, while DDL support btree (traditional) indexes, bitmap (encoded vector indexes or EVI) indexes.  See CREATE INDEX - IBM Documentation

    What may cause some confusion is that keyed logical file cannot be created in just one DDL instruction, but two: CREATE VIEW and CREATE INDEX.  Create view to create the data structure and underlined query implementing the "logical file" and a create [unique] index to implement the "key" for the logical file.

    Fortunately, DB2 for IBM i documentation y very good at Database - IBM Documentation, just be careful of choose your version given that there are some added functionality from version to version.

    I hope this links helps.  Have a nice day 



    ------------------------------
    Daniel Jose Lema Guanziroli
    ------------------------------



  • 12.  RE: Database modernization

    Posted Tue January 30, 2024 02:24 PM

    Daniel,

    Actually, it is quite easy to create a keyed logical file in one simple statement.

    A bulk of our ERP logical files were created with DDS like the following.

    R IPI100IM                  PFILE(IIM       )        
      IID            2A         TEXT('Record ID (IM/IZ)')
                                COLHDG('Rec' +           
                                       'ID' +            
                                       'IM/IZ')          
      IPROD         35O         TEXT('Item Number')      
                                COLHDG('Item' +          
                                       'Number' )        
    ...
    K IPROD                                  
    S IID                       COMP(EQ 'IM')

    your basic keyed LF with select.  Selecting only 'active' records.  Safe to say that a significant percentage of LF's in legacy software looked like this.

    To do this in DDL is simply

    CREATE INDEX ROB.IIML01_DDL
        ON ERPLXF.IIM
        (IPROD ASC)
        WHERE IID = 'IM'
        RCDFMT IPI100IM
        ADD IID, IPROD
    ;

    Forget about using the generate_sql procedure with this.  It will hurl.  IDK if you can even use the underlying API directly

    Generate Data Definition Language (QSQGNDDL) API - IBM Documentation



    ------------------------------
    Robert Berendt IBMChampion
    ------------------------------



  • 13.  RE: Database modernization

    Posted Wed January 24, 2024 07:36 AM

    Have you tried the generate DDL to see if it looks any different than your example?

    https://www.ibm.com/docs/en/i/7.5?topic=services-generate-sql-procedure

    There's also a GUI tool in iACS.  Point it to your table and click the button to generate sql for it.



    ------------------------------
    Robert Berendt IBMChampion
    ------------------------------



  • 14.  RE: Database modernization

    Posted Wed January 24, 2024 09:52 AM

    Hello, I was trying not to describe all the complications of our system when I asked about the LIKE feature of creating a table.  All of your points are well taken and many have already been considered, but since I posed a simple version of my problem, the lack of information may have influenced them.  I will expand the information, but try for simplicity.

    This is a 2-stage problem.  The file in the DDS Format(name) function can easily be converted to source for a table.  RunSqlScripts gives us the record format keyword and name, so that is why it's used in the source code.  This 'parent' file data is replaced more than 3,000 times a day and is keyed as the example for it's duplicate.  99% of our DDS files are not keyed unique, but programming code usually has maintained the data as unique per the key on the physical file.  The data almost always copies to a DDL Table which has the unique key set like the example, without a problem.  And one of the biggest points of modernization is to move this enforcement code out of programs.

    In stage 1, the application that builds this 'parent' data also needs to compare it to yesterdays version of the data, so stage 2  is the duplicate file created with the DDS Format(name) function, and the data copied to it after every successful load of the parent.  In order to successfully convert both of these files to DDL tables, we want them to be exactly the same with the least amount of complexity, work, or opportunity for error.  That is why I'm searching for the answers to the LIKE function.  And there are 14 files involved in this one job stream that need duplicates containing the history of yesterday.

    Daniel, I'm no whiz with the command diagrams in the SQL manual, but I thought LIKE showed that it had to be in parenthesis, so I will look at it again and try it without.

    Robert, yes I ran the extract tool over the cloned file, and it doesn't interpret it as a LIKE scenario, it extracts all the fields and labels just like it does for the parent.  I'd prefer not to use this to be sure they never get out of sync, which is why the DDS versions use the Format(name) keyword.

    And to the points about an identity column, with thousands of DDS files, we are not in a position to make use of this concept and recompile the system.

    This isn't the only challenge, we are just trying to incrementally upgrade the system we have.  And may programs use the physical files directly, and in some cases we already have hundreds of LF objects over files and we need the record formats to be the same if we hope to avoid level-check issues.  We cannot change all access to SQL to avoid it, and I don't believe complicating the system with logical files that are not the complete record layout to achieve a certain level id is a good idea either.

      I appreciate all the feedback.  It's quite involved to update a 35-40 year old system, and this may just be one part that remains DDS defined.

      Thank you all.



    ------------------------------
    Mike Overlander
    ------------------------------



  • 15.  RE: Database modernization

    Posted Wed January 24, 2024 10:11 AM

    Hi Mike,

    there are 2 variants of the LIKE clause:

    The variant without (..) does "copy" more column attributes.
    HTH
    Daniel


    ------------------------------
    Daniel Gross
    ------------------------------



  • 16.  RE: Database modernization

    Posted Wed January 24, 2024 12:49 PM

    Daniel, 

      Thank you.  I knew there had to be something I was missing.  This option, and keeping the record format statement, will produce a perfect copy of the parent file.  Just what was needed.  Case closed. Mike.



    ------------------------------
    Mike Overlander
    ------------------------------



  • 17.  RE: Database modernization

    Posted Wed January 24, 2024 05:15 PM

    There are two notable differences to keep in mind.

    PFs created from DDS have the following default values :

    ·       REUSEDLT(*NO)

    ·       SIZE(10000 1000 3)

    Table created with SQL have the following default values :

    ·       REUSEDLT(*YES)

    ·       SIZE(*NOMAX)

    that may impact the operation of certain applications.



    ------------------------------
    Dominique GAYTE
    ------------------------------



  • 18.  RE: Database modernization

    Posted Thu January 25, 2024 02:47 AM

    Dear Dominique

    To avoid the undesired impact, those 2 attributes can be changed by CHGPF command right after the file is created. 



    ------------------------------
    Chance favors only the prepared mind.
    -- Louis Pasteur
    ------------------------------
    Satid S.
    ------------------------------



  • 19.  RE: Database modernization

    Posted Thu January 25, 2024 07:37 AM

    But I would ask yourself:  Do you really want to change them?

    Sure there are some cases where reusing deleted records might make be a negative but how many people are using direct access to files (by record number vs key, etc) anymore?  I've not seen that in over 35 years.  There are many positive reasons to reuse deleted records.  Security, reduces possibility of using undelete tools.  Saves space.  Reduces need to run RGZPFM to remove deleted rows.

    Had a few jobs this week blow because they hit the maximum row count.  Changing to *nomax stops that, especially since the developers always reply to the message with *NOMAX.  Now, if you 'never' exceed a certain row count and you want to safety measure to put the brakes on a runaway job filling your disk then you could change it.  There are alternative methods to spot disk spikes.



    ------------------------------
    Robert Berendt IBMChampion
    ------------------------------



  • 20.  RE: Database modernization

    Posted Thu January 25, 2024 09:23 AM

    Morning Robert

    On Reuse deleted rows, On some instances I do need to not reuse deleted records and I do not want the rgzpf to reorganize by keys because in some of my designs, when there are duplicate keys, I assume the 1st row (record) entered will be first in the file

    I.E. in keeping modified records in a new file, when the same key is modified multiple times, the 1st modification will be the 1st occurrence of the key in the file



    ------------------------------
    Carlos Romero
    ------------------------------



  • 21.  RE: Database modernization

    Posted Thu January 25, 2024 09:33 AM

    Carlos,

    I appreciate the explanation.  Thank you.

    I might use a pseudo key, like an automatically generated identity column, or even a timestamp out to the most precision.

    However, now that you mention it, your case is not the first I've heard of this.

    Hmm, "same key modified multiple times"... This sounds like a homegrown temporal table history table.  Have you looked at implementing temporal tables?



    ------------------------------
    Robert Berendt IBMChampion
    ------------------------------



  • 22.  RE: Database modernization

    Posted Thu January 25, 2024 09:50 AM

    I would save the deleted record in the same file, in a different member. Part of the design

    It worked fine in the old days

    I would use a different approach now



    ------------------------------
    Carlos Romero
    ------------------------------