Maximo

I think I have found that the OOTB Maximo ARCGISSYNC process does not support Pagination so thereby cannot load any more then about 2000 rows at a time from single cron task?

Am I misundersatnding and how do you get past this limit?  The row count seems to only be a part of it as the number of columns (attributes) has a role in limiting the number of rows I can get back as well.

Can anyone shed any light on how I can get a million record feature class to sync using OOTB sync functionality? As I am seeing we are going to need to write our own custom auto scripts to accomplish this?

Thanks,

------------------------------
David Miller
Managing Partner - Maximo Consultant
Enterprise Integration Partners LLC
------------------------------



#MaximoSpatial
#Maximo
#AssetandFacilitiesManagement

I believe there is currently a code bug in the latest version of Maximo Spatial that is limiting the number of records that can be processed. I believe that is may be tied to the network adapter that is configured with ArcGIS Server. 
This has been reported by the City of Austin and by the Southern Company Electric group.  
This same bug has been reported several times in the past, been fixed and then reappears. 
I suggest you also put in a ticket to IBM support to help everyone find a solution. 
A poperly configured Maximo Spatial system should be able to process hundreds of thousands of ArcGIS records as we were doing at Entergy several months ago on older versions of the Maximo Spatial software.

------------------------------
Jim Saunders
------------------------------

Original Message:
Sent: Thu July 14, 2022 10:26 AM
From: David Miller
Subject: ArcGISSync Support for Pagination?

I think I have found that the OOTB Maximo ARCGISSYNC process does not support Pagination so thereby cannot load any more then about 2000 rows at a time from single cron task?

Am I misundersatnding and how do you get past this limit?  The row count seems to only be a part of it as the number of columns (attributes) has a role in limiting the number of rows I can get back as well.

Can anyone shed any light on how I can get a million record feature class to sync using OOTB sync functionality? As I am seeing we are going to need to write our own custom auto scripts to accomplish this?

Thanks,

------------------------------
David Miller
Managing Partner - Maximo Consultant
Enterprise Integration Partners LLC
------------------------------



#AssetandFacilitiesManagement
#MaximoSpatial
#Maximo

Syrely a million records of any kind will bring down your MIF surely if done in one hit…
On the Esri side you can increase the max record response, but regardless, you should be
1. Setting a flag on Esri side to mark which records need to be updated/added in Maximo (ibm suggest an MXCREATIONSTATE field but can be anything. You can default it on Esri side on add/change with an ‘attribute rule’ eg to 1
2. using the parameter on the cron to send make a response to mark the records as processed, eg set above to 0
3. Setting the where clause in the url parameter in the cron to only pick up records flagged for update eg where mxcreationstate=1
then each time cron runs it’ll do 2000 at a time and eventually get through them all.


Original Message:
Sent: 7/14/2022 10:26:00 AM
From: David Miller
Subject: ArcGISSync Support for Pagination?

I think I have found that the OOTB Maximo ARCGISSYNC process does not support Pagination so thereby cannot load any more then about 2000 rows at a time from single cron task?

Am I misundersatnding and how do you get past this limit?  The row count seems to only be a part of it as the number of columns (attributes) has a role in limiting the number of rows I can get back as well.

Can anyone shed any light on how I can get a million record feature class to sync using OOTB sync functionality? As I am seeing we are going to need to write our own custom auto scripts to accomplish this?

Thanks,

------------------------------
David Miller
Managing Partner - Maximo Consultant
Enterprise Integration Partners LLC
------------------------------



#MaximoSpatial
#AssetandFacilitiesManagement
#Maximo

Kelly,

I beg to differ with your mentality that a million records should bring down MIF. I know for a fact that I have loaded 3 Million+ WO using an IFACE table and its MXINTER_TRANS to manage (it was not fast but worked). And that dates back many years and MIF should be getting better in the day and age of Big Data and the Cloud and not getting worse. I think the design of the ARCGISSYNC is the problem here and we don't get an intertrans table  to manage the rows being brought in by the sync. A 6 GB JVM should be able to handle a million rows of data with not many attributes in connection with a SIB table in the DB (which should be unlimited)

Thanks,

------------------------------
David Miller
Managing Partner - Maximo Consultant
Enterprise Integration Partners LLC
------------------------------

Original Message:
Sent: Fri July 15, 2022 02:35 AM
From: Zoltan Kelly
Subject: ArcGISSync Support for Pagination?

Syrely a million records of any kind will bring down your MIF surely if done in one hit…
On the Esri side you can increase the max record response, but regardless, you should be
1. Setting a flag on Esri side to mark which records need to be updated/added in Maximo (ibm suggest an MXCREATIONSTATE field but can be anything. You can default it on Esri side on add/change with an 'attribute rule' eg to 1
2. using the parameter on the cron to send make a response to mark the records as processed, eg set above to 0
3. Setting the where clause in the url parameter in the cron to only pick up records flagged for update eg where mxcreationstate=1
then each time cron runs it'll do 2000 at a time and eventually get through them all.


Original Message:
Sent: 7/14/2022 10:26:00 AM
From: David Miller
Subject: ArcGISSync Support for Pagination?

I think I have found that the OOTB Maximo ARCGISSYNC process does not support Pagination so thereby cannot load any more then about 2000 rows at a time from single cron task?

Am I misundersatnding and how do you get past this limit?  The row count seems to only be a part of it as the number of columns (attributes) has a role in limiting the number of rows I can get back as well.

Can anyone shed any light on how I can get a million record feature class to sync using OOTB sync functionality? As I am seeing we are going to need to write our own custom auto scripts to accomplish this?

Thanks,

------------------------------
David Miller
Managing Partner - Maximo Consultant
Enterprise Integration Partners LLC
------------------------------



#AssetandFacilitiesManagement
#MaximoSpatial
#Maximo

Original Message:
Sent: 7/15/2022 5:56:00 PM
From: David Miller
Subject: RE: ArcGISSync Support for Pagination?

Kelly,

I beg to differ with your mentality that a million records should bring down MIF. I know for a fact that I have loaded 3 Million+ WO using an IFACE table and its MXINTER_TRANS to manage (it was not fast but worked). And that dates back many years and MIF should be getting better in the day and age of Big Data and the Cloud and not getting worse. I think the design of the ARCGISSYNC is the problem here and we don't get an intertrans table  to manage the rows being brought in by the sync. A 6 GB JVM should be able to handle a million rows of data with not many attributes in connection with a SIB table in the DB (which should be unlimited)

Thanks,

------------------------------
David Miller
Managing Partner - Maximo Consultant
Enterprise Integration Partners LLC
------------------------------

Original Message:
Sent: Fri July 15, 2022 02:35 AM
From: Zoltan Kelly
Subject: ArcGISSync Support for Pagination?

Syrely a million records of any kind will bring down your MIF surely if done in one hit…
On the Esri side you can increase the max record response, but regardless, you should be
1. Setting a flag on Esri side to mark which records need to be updated/added in Maximo (ibm suggest an MXCREATIONSTATE field but can be anything. You can default it on Esri side on add/change with an 'attribute rule' eg to 1
2. using the parameter on the cron to send make a response to mark the records as processed, eg set above to 0
3. Setting the where clause in the url parameter in the cron to only pick up records flagged for update eg where mxcreationstate=1
then each time cron runs it'll do 2000 at a time and eventually get through them all.


Original Message:
Sent: 7/14/2022 10:26:00 AM
From: David Miller
Subject: ArcGISSync Support for Pagination?

I think I have found that the OOTB Maximo ARCGISSYNC process does not support Pagination so thereby cannot load any more then about 2000 rows at a time from single cron task?

Am I misundersatnding and how do you get past this limit?  The row count seems to only be a part of it as the number of columns (attributes) has a role in limiting the number of rows I can get back as well.

Can anyone shed any light on how I can get a million record feature class to sync using OOTB sync functionality? As I am seeing we are going to need to write our own custom auto scripts to accomplish this?

Thanks,

------------------------------
David Miller
Managing Partner - Maximo Consultant
Enterprise Integration Partners LLC
------------------------------



#Maximo
#MaximoSpatial
#AssetandFacilitiesManagement

Maximo Spatial is a live, bi-directional interface between Maximo and ArcGIS. It is not a one-time data load. Large organizations (hundreds of editors) may be working in either ArcGIS or Maximo and their editing efforts will be synched across both systems. The timing of the data synchs is configurable within the cron tasks. You may run the synching process once a day or every minute, depending upon need and computer resources.
The Maximo Spatial cron tasks use the MIF to make calls through the individually configured FeatureServices to make edits in either ArcGIS or Maximo. The cron task makes a call to ArcGIS and looks for values that match the configured trigger.  (i.e. MxCreationState = 1). It then makes multiple queries to find all ArcGIS objects that match the trigger. If you have 10,000 matching ArcGIS objects, the cron task creates 10 queries that return only a list of 1,000 ObjectIDs of the matching objects. Maximo Spatial then uses the MIF to process all 10,000 records in batches of 1,000 at a time. While processing these 10 queries, the cron task knows it is running 10 queries and will not kick off a new cron task instance until all 10 have been processed.

There appears to be a bug within the latest version of Maximo Spatial in that it cannot make proper queries to ArcGIS. Clients may restrict the ArcGIS Max Record count for the FeatureService, to limit how many records can be processed at one time (sometimes this needs to be as low as 100 records) and force Maximo Spatial to operate. It has been seen that this occurs when ArcGIS has a WebAdaptor configured with ArcGIS Server as a part of ArcGIS Enterprise. Sometimes, removing the WebAdaptor from ArcGIS Server, allows Maximo Spatial to operate properly. 

David, you may want to try removing the WebAdaptor temporarily and see if that resolves your issue.

Several Spatial clients currently have tickets with IBM support to try to rectify this issue.  If you are having this issue, I urge you to also enter a support ticket.  The more tickets, the more time will be spent in fixing the problem. And the problem may be partially with the WebAdaptor and partially with Maximo Spatial and may require changes in both environments.

The speed of processing the ArcGIS records is a function of the number of JVMs you assign to Maximo Spatial (and the memory allocated to each JVM) and the queues you set up to handle the processing requests. For large clients, we have created dedicated clusters with dedicated JVMs and dedicated queues for Maximo Spatial, sized to the expected number of users and records to be processed. This is so that other MIF operations and/or processes are not impacted by the Spatial operations and vice versa.

The IBM SWG has previously demonstrated that with as properly setup and properly tuned Maximo environment, they could successfully process over 1 million Pole edits in under 15 minutes, using Maximo Spatial.

------------------------------
Jim Saunders
Saunders International, Inc.
Maximo Spatial and ArcGIS
+1 (256) 808-7512
www.SaundersIntl.com
Jim@SaundersIntl.com
Huntsville, AL
------------------------------

Original Message:
Sent: Fri July 15, 2022 07:42 PM
From: Zoltan Kelly
Subject: ArcGISSync Support for Pagination?

Yeah I get you in that MIF can handle millions of transactions, but you're right, that's not the design of OOTB Maximo Spatial.

There's some logic on the ArcGISDataSync class files thatloops through the each line and sends back a query to ArcGIS to find out if it's an update or insert that slows it down.

On the Esri side you can push the max record count up, but even from their side it's recommended to keep it to <5000 for performance https://enterprise.arcgis.com/en/sites/latest/content/server-configuration-details.htm

There's the Geocortex Maximo data sync product that might be more what you want.

I'm curious about your use case, if this is a one off load, if you're using it for transactions (eg wo etc), or if your assets in ArcGIS change often and and frequently in the order of millions.

You could probably script it to paginate the requests and still use parts of the ArcGISDataSync cron or the classes, but I don't think you'd get the performance jump you want due to the processing classes. You could always set the cron to every 30 seconds or something?

Even then 2000 x 30 seconds is still 4 hours for a million records.

Normally massive bulk requests from ArcGIS > Maximo  for us are one offs, so we use FME to query esri SQL dev > load via flat file consumer.

 

Original Message:
Sent: 7/15/2022 5:56:00 PM
From: David Miller
Subject: RE: ArcGISSync Support for Pagination?

Kelly,

I beg to differ with your mentality that a million records should bring down MIF. I know for a fact that I have loaded 3 Million+ WO using an IFACE table and its MXINTER_TRANS to manage (it was not fast but worked). And that dates back many years and MIF should be getting better in the day and age of Big Data and the Cloud and not getting worse. I think the design of the ARCGISSYNC is the problem here and we don't get an intertrans table  to manage the rows being brought in by the sync. A 6 GB JVM should be able to handle a million rows of data with not many attributes in connection with a SIB table in the DB (which should be unlimited)

Thanks,

------------------------------
David Miller
Managing Partner - Maximo Consultant
Enterprise Integration Partners LLC

Original Message:
Sent: Fri July 15, 2022 02:35 AM
From: Zoltan Kelly
Subject: ArcGISSync Support for Pagination?

Syrely a million records of any kind will bring down your MIF surely if done in one hit…
On the Esri side you can increase the max record response, but regardless, you should be
1. Setting a flag on Esri side to mark which records need to be updated/added in Maximo (ibm suggest an MXCREATIONSTATE field but can be anything. You can default it on Esri side on add/change with an 'attribute rule' eg to 1
2. using the parameter on the cron to send make a response to mark the records as processed, eg set above to 0
3. Setting the where clause in the url parameter in the cron to only pick up records flagged for update eg where mxcreationstate=1
then each time cron runs it'll do 2000 at a time and eventually get through them all.


Original Message:
Sent: 7/14/2022 10:26:00 AM
From: David Miller
Subject: ArcGISSync Support for Pagination?

I think I have found that the OOTB Maximo ARCGISSYNC process does not support Pagination so thereby cannot load any more then about 2000 rows at a time from single cron task?

Am I misundersatnding and how do you get past this limit?  The row count seems to only be a part of it as the number of columns (attributes) has a role in limiting the number of rows I can get back as well.

Can anyone shed any light on how I can get a million record feature class to sync using OOTB sync functionality? As I am seeing we are going to need to write our own custom auto scripts to accomplish this?

Thanks,

------------------------------
David Miller
Managing Partner - Maximo Consultant
Enterprise Integration Partners LLC
------------------------------



#Maximo
#MaximoSpatial
#AssetandFacilitiesManagement

Jim,

I hear what you are saying and agree. We are working in a DEV instance right now and will have a CONVERSION & PROD environments that will be VERY ROBUST compared to our current DEV.

I am missing something on the issue your are speaking of as we have found that we were able to change the MaxRecordCount: 3000  (from default of 2000) in Esri for a Valve Layer (uses a WebMap and a Query to filter it). I was able to push through sucessfully 2,989 asset records with 15 Attributes on a spec template mapped in JSON Mapping. This ran in 49 seconds. I am thrilled with this respoce time in a loiwly DEV environment running on a single JVM for CRON but another for MIF. The math says we could load 1 million Valves in 4.5 hours at that rate. This should be able to be cut in half when we get to our PROD env which will also be using cutting edge Oracle Exadata that I am hoping will crank this up even more not to mention actuial physical servers (not virtual) dedicated to JVM for CRON/MIF etc.

As far as removing the Web Adaptor, I ran this idea by my ESRI person today and he does no think that is possible with our architecture and he mentioned the use of Federated Servers would not allow us to reference ArcGIS by a Port Number due to this design. If you believe differently I would really like more background on how to do that?

Just for additional background we are using ArcGIS Current Version: 10.91 with the UPDM. ANd we have
IBM Maximo Spatial Asset Management 7.6.1.0-20220502-0822 Build 20200716-1706 DB Build V7610-15 HFDB Build HF7610-13

Thanks,

------------------------------
David Miller
Managing Partner - Maximo Consultant
Enterprise Integration Partners LLC
------------------------------

Original Message:
Sent: Mon July 18, 2022 11:02 AM
From: Jim Saunders
Subject: ArcGISSync Support for Pagination?

Maximo Spatial is a live, bi-directional interface between Maximo and ArcGIS. It is not a one-time data load. Large organizations (hundreds of editors) may be working in either ArcGIS or Maximo and their editing efforts will be synched across both systems. The timing of the data synchs is configurable within the cron tasks. You may run the synching process once a day or every minute, depending upon need and computer resources.
The Maximo Spatial cron tasks use the MIF to make calls through the individually configured FeatureServices to make edits in either ArcGIS or Maximo. The cron task makes a call to ArcGIS and looks for values that match the configured trigger.  (i.e. MxCreationState = 1). It then makes multiple queries to find all ArcGIS objects that match the trigger. If you have 10,000 matching ArcGIS objects, the cron task creates 10 queries that return only a list of 1,000 ObjectIDs of the matching objects. Maximo Spatial then uses the MIF to process all 10,000 records in batches of 1,000 at a time. While processing these 10 queries, the cron task knows it is running 10 queries and will not kick off a new cron task instance until all 10 have been processed.

There appears to be a bug within the latest version of Maximo Spatial in that it cannot make proper queries to ArcGIS. Clients may restrict the ArcGIS Max Record count for the FeatureService, to limit how many records can be processed at one time (sometimes this needs to be as low as 100 records) and force Maximo Spatial to operate. It has been seen that this occurs when ArcGIS has a WebAdaptor configured with ArcGIS Server as a part of ArcGIS Enterprise. Sometimes, removing the WebAdaptor from ArcGIS Server, allows Maximo Spatial to operate properly. 

David, you may want to try removing the WebAdaptor temporarily and see if that resolves your issue.

Several Spatial clients currently have tickets with IBM support to try to rectify this issue.  If you are having this issue, I urge you to also enter a support ticket.  The more tickets, the more time will be spent in fixing the problem. And the problem may be partially with the WebAdaptor and partially with Maximo Spatial and may require changes in both environments.

The speed of processing the ArcGIS records is a function of the number of JVMs you assign to Maximo Spatial (and the memory allocated to each JVM) and the queues you set up to handle the processing requests. For large clients, we have created dedicated clusters with dedicated JVMs and dedicated queues for Maximo Spatial, sized to the expected number of users and records to be processed. This is so that other MIF operations and/or processes are not impacted by the Spatial operations and vice versa.

The IBM SWG has previously demonstrated that with as properly setup and properly tuned Maximo environment, they could successfully process over 1 million Pole edits in under 15 minutes, using Maximo Spatial.

------------------------------
Jim Saunders
Saunders International, Inc.
Maximo Spatial and ArcGIS
+1 (256) 808-7512
www.SaundersIntl.com
Jim@SaundersIntl.com
Huntsville, AL

Original Message:
Sent: Fri July 15, 2022 07:42 PM
From: Zoltan Kelly
Subject: ArcGISSync Support for Pagination?

Yeah I get you in that MIF can handle millions of transactions, but you're right, that's not the design of OOTB Maximo Spatial.

There's some logic on the ArcGISDataSync class files thatloops through the each line and sends back a query to ArcGIS to find out if it's an update or insert that slows it down.

On the Esri side you can push the max record count up, but even from their side it's recommended to keep it to <5000 for performance https://enterprise.arcgis.com/en/sites/latest/content/server-configuration-details.htm

There's the Geocortex Maximo data sync product that might be more what you want.

I'm curious about your use case, if this is a one off load, if you're using it for transactions (eg wo etc), or if your assets in ArcGIS change often and and frequently in the order of millions.

You could probably script it to paginate the requests and still use parts of the ArcGISDataSync cron or the classes, but I don't think you'd get the performance jump you want due to the processing classes. You could always set the cron to every 30 seconds or something?

Even then 2000 x 30 seconds is still 4 hours for a million records.

Normally massive bulk requests from ArcGIS > Maximo  for us are one offs, so we use FME to query esri SQL dev > load via flat file consumer.

 

Original Message:
Sent: 7/15/2022 5:56:00 PM
From: David Miller
Subject: RE: ArcGISSync Support for Pagination?

Kelly,

I beg to differ with your mentality that a million records should bring down MIF. I know for a fact that I have loaded 3 Million+ WO using an IFACE table and its MXINTER_TRANS to manage (it was not fast but worked). And that dates back many years and MIF should be getting better in the day and age of Big Data and the Cloud and not getting worse. I think the design of the ARCGISSYNC is the problem here and we don't get an intertrans table  to manage the rows being brought in by the sync. A 6 GB JVM should be able to handle a million rows of data with not many attributes in connection with a SIB table in the DB (which should be unlimited)

Thanks,

------------------------------
David Miller
Managing Partner - Maximo Consultant
Enterprise Integration Partners LLC

Original Message:
Sent: Fri July 15, 2022 02:35 AM
From: Zoltan Kelly
Subject: ArcGISSync Support for Pagination?

Syrely a million records of any kind will bring down your MIF surely if done in one hit…
On the Esri side you can increase the max record response, but regardless, you should be
1. Setting a flag on Esri side to mark which records need to be updated/added in Maximo (ibm suggest an MXCREATIONSTATE field but can be anything. You can default it on Esri side on add/change with an 'attribute rule' eg to 1
2. using the parameter on the cron to send make a response to mark the records as processed, eg set above to 0
3. Setting the where clause in the url parameter in the cron to only pick up records flagged for update eg where mxcreationstate=1
then each time cron runs it'll do 2000 at a time and eventually get through them all.


Original Message:
Sent: 7/14/2022 10:26:00 AM
From: David Miller
Subject: ArcGISSync Support for Pagination?

I think I have found that the OOTB Maximo ARCGISSYNC process does not support Pagination so thereby cannot load any more then about 2000 rows at a time from single cron task?

Am I misundersatnding and how do you get past this limit?  The row count seems to only be a part of it as the number of columns (attributes) has a role in limiting the number of rows I can get back as well.

Can anyone shed any light on how I can get a million record feature class to sync using OOTB sync functionality? As I am seeing we are going to need to write our own custom auto scripts to accomplish this?

Thanks,

------------------------------
David Miller
Managing Partner - Maximo Consultant
Enterprise Integration Partners LLC
------------------------------



#MaximoSpatial
#Maximo
#AssetandFacilitiesManagement

You can load the initial millions of records through the MIF without the cron task.  You just need to have the PLUSSISGIS, PLUSSFEATURECLASS and whatever fields you are using for your relationship to be set and then the Maximo records will be linked.  And of course whatever other fields you would like populate too.

Going forward, you can set your cron task URL to either use a creation attribute like Zoltan mentions, or you can use the last successful start time in your URL to compare the last time your cron task instance successfully ran with an edited date attribute on your features to synchronize only the changes.  I added a post about this capability in the discussions recently.

------------------------------
Lacey Lurges
------------------------------

Original Message:
Sent: Mon July 18, 2022 04:22 PM
From: David Miller
Subject: ArcGISSync Support for Pagination?

Jim,

I hear what you are saying and agree. We are working in a DEV instance right now and will have a CONVERSION & PROD environments that will be VERY ROBUST compared to our current DEV.

I am missing something on the issue your are speaking of as we have found that we were able to change the MaxRecordCount: 3000  (from default of 2000) in Esri for a Valve Layer (uses a WebMap and a Query to filter it). I was able to push through sucessfully 2,989 asset records with 15 Attributes on a spec template mapped in JSON Mapping. This ran in 49 seconds. I am thrilled with this respoce time in a loiwly DEV environment running on a single JVM for CRON but another for MIF. The math says we could load 1 million Valves in 4.5 hours at that rate. This should be able to be cut in half when we get to our PROD env which will also be using cutting edge Oracle Exadata that I am hoping will crank this up even more not to mention actuial physical servers (not virtual) dedicated to JVM for CRON/MIF etc.

As far as removing the Web Adaptor, I ran this idea by my ESRI person today and he does no think that is possible with our architecture and he mentioned the use of Federated Servers would not allow us to reference ArcGIS by a Port Number due to this design. If you believe differently I would really like more background on how to do that?

Just for additional background we are using ArcGIS Current Version: 10.91 with the UPDM. ANd we have
IBM Maximo Spatial Asset Management 7.6.1.0-20220502-0822 Build 20200716-1706 DB Build V7610-15 HFDB Build HF7610-13

Thanks,

------------------------------
David Miller
Managing Partner - Maximo Consultant
Enterprise Integration Partners LLC

Original Message:
Sent: Mon July 18, 2022 11:02 AM
From: Jim Saunders
Subject: ArcGISSync Support for Pagination?

Maximo Spatial is a live, bi-directional interface between Maximo and ArcGIS. It is not a one-time data load. Large organizations (hundreds of editors) may be working in either ArcGIS or Maximo and their editing efforts will be synched across both systems. The timing of the data synchs is configurable within the cron tasks. You may run the synching process once a day or every minute, depending upon need and computer resources.
The Maximo Spatial cron tasks use the MIF to make calls through the individually configured FeatureServices to make edits in either ArcGIS or Maximo. The cron task makes a call to ArcGIS and looks for values that match the configured trigger.  (i.e. MxCreationState = 1). It then makes multiple queries to find all ArcGIS objects that match the trigger. If you have 10,000 matching ArcGIS objects, the cron task creates 10 queries that return only a list of 1,000 ObjectIDs of the matching objects. Maximo Spatial then uses the MIF to process all 10,000 records in batches of 1,000 at a time. While processing these 10 queries, the cron task knows it is running 10 queries and will not kick off a new cron task instance until all 10 have been processed.

There appears to be a bug within the latest version of Maximo Spatial in that it cannot make proper queries to ArcGIS. Clients may restrict the ArcGIS Max Record count for the FeatureService, to limit how many records can be processed at one time (sometimes this needs to be as low as 100 records) and force Maximo Spatial to operate. It has been seen that this occurs when ArcGIS has a WebAdaptor configured with ArcGIS Server as a part of ArcGIS Enterprise. Sometimes, removing the WebAdaptor from ArcGIS Server, allows Maximo Spatial to operate properly. 

David, you may want to try removing the WebAdaptor temporarily and see if that resolves your issue.

Several Spatial clients currently have tickets with IBM support to try to rectify this issue.  If you are having this issue, I urge you to also enter a support ticket.  The more tickets, the more time will be spent in fixing the problem. And the problem may be partially with the WebAdaptor and partially with Maximo Spatial and may require changes in both environments.

The speed of processing the ArcGIS records is a function of the number of JVMs you assign to Maximo Spatial (and the memory allocated to each JVM) and the queues you set up to handle the processing requests. For large clients, we have created dedicated clusters with dedicated JVMs and dedicated queues for Maximo Spatial, sized to the expected number of users and records to be processed. This is so that other MIF operations and/or processes are not impacted by the Spatial operations and vice versa.

The IBM SWG has previously demonstrated that with as properly setup and properly tuned Maximo environment, they could successfully process over 1 million Pole edits in under 15 minutes, using Maximo Spatial.

------------------------------
Jim Saunders
Saunders International, Inc.
Maximo Spatial and ArcGIS
+1 (256) 808-7512
www.SaundersIntl.com
Jim@SaundersIntl.com
Huntsville, AL

Original Message:
Sent: Fri July 15, 2022 07:42 PM
From: Zoltan Kelly
Subject: ArcGISSync Support for Pagination?

Yeah I get you in that MIF can handle millions of transactions, but you're right, that's not the design of OOTB Maximo Spatial.

There's some logic on the ArcGISDataSync class files thatloops through the each line and sends back a query to ArcGIS to find out if it's an update or insert that slows it down.

On the Esri side you can push the max record count up, but even from their side it's recommended to keep it to <5000 for performance https://enterprise.arcgis.com/en/sites/latest/content/server-configuration-details.htm

There's the Geocortex Maximo data sync product that might be more what you want.

I'm curious about your use case, if this is a one off load, if you're using it for transactions (eg wo etc), or if your assets in ArcGIS change often and and frequently in the order of millions.

You could probably script it to paginate the requests and still use parts of the ArcGISDataSync cron or the classes, but I don't think you'd get the performance jump you want due to the processing classes. You could always set the cron to every 30 seconds or something?

Even then 2000 x 30 seconds is still 4 hours for a million records.

Normally massive bulk requests from ArcGIS > Maximo  for us are one offs, so we use FME to query esri SQL dev > load via flat file consumer.

 

Original Message:
Sent: 7/15/2022 5:56:00 PM
From: David Miller
Subject: RE: ArcGISSync Support for Pagination?

Kelly,

I beg to differ with your mentality that a million records should bring down MIF. I know for a fact that I have loaded 3 Million+ WO using an IFACE table and its MXINTER_TRANS to manage (it was not fast but worked). And that dates back many years and MIF should be getting better in the day and age of Big Data and the Cloud and not getting worse. I think the design of the ARCGISSYNC is the problem here and we don't get an intertrans table  to manage the rows being brought in by the sync. A 6 GB JVM should be able to handle a million rows of data with not many attributes in connection with a SIB table in the DB (which should be unlimited)

Thanks,

------------------------------
David Miller
Managing Partner - Maximo Consultant
Enterprise Integration Partners LLC

Original Message:
Sent: Fri July 15, 2022 02:35 AM
From: Zoltan Kelly
Subject: ArcGISSync Support for Pagination?

Syrely a million records of any kind will bring down your MIF surely if done in one hit…
On the Esri side you can increase the max record response, but regardless, you should be
1. Setting a flag on Esri side to mark which records need to be updated/added in Maximo (ibm suggest an MXCREATIONSTATE field but can be anything. You can default it on Esri side on add/change with an 'attribute rule' eg to 1
2. using the parameter on the cron to send make a response to mark the records as processed, eg set above to 0
3. Setting the where clause in the url parameter in the cron to only pick up records flagged for update eg where mxcreationstate=1
then each time cron runs it'll do 2000 at a time and eventually get through them all.


Original Message:
Sent: 7/14/2022 10:26:00 AM
From: David Miller
Subject: ArcGISSync Support for Pagination?

I think I have found that the OOTB Maximo ARCGISSYNC process does not support Pagination so thereby cannot load any more then about 2000 rows at a time from single cron task?

Am I misundersatnding and how do you get past this limit?  The row count seems to only be a part of it as the number of columns (attributes) has a role in limiting the number of rows I can get back as well.

Can anyone shed any light on how I can get a million record feature class to sync using OOTB sync functionality? As I am seeing we are going to need to write our own custom auto scripts to accomplish this?

Thanks,

------------------------------
David Miller
Managing Partner - Maximo Consultant
Enterprise Integration Partners LLC
------------------------------



#AssetandFacilitiesManagement
#MaximoSpatial
#Maximo

Lacey,

Thanks for the suggestion. WE will keep this in mind as we get more data to be able to test load and benchmark the differeent methods. I was able to use the cron and a full featuyre class with 80k assets and able to load 73k of them in 1.25 hours. I like your sugeestion to use regular MIF and I would try to spread the load out over multiple JVM to speed it up as well.

Will see how this goies when we get out first data set with millions to load.

Thanks,

------------------------------
David Miller
Managing Partner - Maximo Consultant
Enterprise Integration Partners LLC
------------------------------

Original Message:
Sent: Mon July 18, 2022 07:58 PM
From: Lacey Lurges
Subject: ArcGISSync Support for Pagination?

You can load the initial millions of records through the MIF without the cron task.  You just need to have the PLUSSISGIS, PLUSSFEATURECLASS and whatever fields you are using for your relationship to be set and then the Maximo records will be linked.  And of course whatever other fields you would like populate too.

Going forward, you can set your cron task URL to either use a creation attribute like Zoltan mentions, or you can use the last successful start time in your URL to compare the last time your cron task instance successfully ran with an edited date attribute on your features to synchronize only the changes.  I added a post about this capability in the discussions recently.

------------------------------
Lacey Lurges

Original Message:
Sent: Mon July 18, 2022 04:22 PM
From: David Miller
Subject: ArcGISSync Support for Pagination?

Jim,

I hear what you are saying and agree. We are working in a DEV instance right now and will have a CONVERSION & PROD environments that will be VERY ROBUST compared to our current DEV.

I am missing something on the issue your are speaking of as we have found that we were able to change the MaxRecordCount: 3000  (from default of 2000) in Esri for a Valve Layer (uses a WebMap and a Query to filter it). I was able to push through sucessfully 2,989 asset records with 15 Attributes on a spec template mapped in JSON Mapping. This ran in 49 seconds. I am thrilled with this respoce time in a loiwly DEV environment running on a single JVM for CRON but another for MIF. The math says we could load 1 million Valves in 4.5 hours at that rate. This should be able to be cut in half when we get to our PROD env which will also be using cutting edge Oracle Exadata that I am hoping will crank this up even more not to mention actuial physical servers (not virtual) dedicated to JVM for CRON/MIF etc.

As far as removing the Web Adaptor, I ran this idea by my ESRI person today and he does no think that is possible with our architecture and he mentioned the use of Federated Servers would not allow us to reference ArcGIS by a Port Number due to this design. If you believe differently I would really like more background on how to do that?

Just for additional background we are using ArcGIS Current Version: 10.91 with the UPDM. ANd we have
IBM Maximo Spatial Asset Management 7.6.1.0-20220502-0822 Build 20200716-1706 DB Build V7610-15 HFDB Build HF7610-13

Thanks,

------------------------------
David Miller
Managing Partner - Maximo Consultant
Enterprise Integration Partners LLC

Original Message:
Sent: Mon July 18, 2022 11:02 AM
From: Jim Saunders
Subject: ArcGISSync Support for Pagination?

Maximo Spatial is a live, bi-directional interface between Maximo and ArcGIS. It is not a one-time data load. Large organizations (hundreds of editors) may be working in either ArcGIS or Maximo and their editing efforts will be synched across both systems. The timing of the data synchs is configurable within the cron tasks. You may run the synching process once a day or every minute, depending upon need and computer resources.
The Maximo Spatial cron tasks use the MIF to make calls through the individually configured FeatureServices to make edits in either ArcGIS or Maximo. The cron task makes a call to ArcGIS and looks for values that match the configured trigger.  (i.e. MxCreationState = 1). It then makes multiple queries to find all ArcGIS objects that match the trigger. If you have 10,000 matching ArcGIS objects, the cron task creates 10 queries that return only a list of 1,000 ObjectIDs of the matching objects. Maximo Spatial then uses the MIF to process all 10,000 records in batches of 1,000 at a time. While processing these 10 queries, the cron task knows it is running 10 queries and will not kick off a new cron task instance until all 10 have been processed.

There appears to be a bug within the latest version of Maximo Spatial in that it cannot make proper queries to ArcGIS. Clients may restrict the ArcGIS Max Record count for the FeatureService, to limit how many records can be processed at one time (sometimes this needs to be as low as 100 records) and force Maximo Spatial to operate. It has been seen that this occurs when ArcGIS has a WebAdaptor configured with ArcGIS Server as a part of ArcGIS Enterprise. Sometimes, removing the WebAdaptor from ArcGIS Server, allows Maximo Spatial to operate properly. 

David, you may want to try removing the WebAdaptor temporarily and see if that resolves your issue.

Several Spatial clients currently have tickets with IBM support to try to rectify this issue.  If you are having this issue, I urge you to also enter a support ticket.  The more tickets, the more time will be spent in fixing the problem. And the problem may be partially with the WebAdaptor and partially with Maximo Spatial and may require changes in both environments.

The speed of processing the ArcGIS records is a function of the number of JVMs you assign to Maximo Spatial (and the memory allocated to each JVM) and the queues you set up to handle the processing requests. For large clients, we have created dedicated clusters with dedicated JVMs and dedicated queues for Maximo Spatial, sized to the expected number of users and records to be processed. This is so that other MIF operations and/or processes are not impacted by the Spatial operations and vice versa.

The IBM SWG has previously demonstrated that with as properly setup and properly tuned Maximo environment, they could successfully process over 1 million Pole edits in under 15 minutes, using Maximo Spatial.

------------------------------
Jim Saunders
Saunders International, Inc.
Maximo Spatial and ArcGIS
+1 (256) 808-7512
www.SaundersIntl.com
Jim@SaundersIntl.com
Huntsville, AL

Original Message:
Sent: Fri July 15, 2022 07:42 PM
From: Zoltan Kelly
Subject: ArcGISSync Support for Pagination?

Yeah I get you in that MIF can handle millions of transactions, but you're right, that's not the design of OOTB Maximo Spatial.

There's some logic on the ArcGISDataSync class files thatloops through the each line and sends back a query to ArcGIS to find out if it's an update or insert that slows it down.

On the Esri side you can push the max record count up, but even from their side it's recommended to keep it to <5000 for performance https://enterprise.arcgis.com/en/sites/latest/content/server-configuration-details.htm

There's the Geocortex Maximo data sync product that might be more what you want.

I'm curious about your use case, if this is a one off load, if you're using it for transactions (eg wo etc), or if your assets in ArcGIS change often and and frequently in the order of millions.

You could probably script it to paginate the requests and still use parts of the ArcGISDataSync cron or the classes, but I don't think you'd get the performance jump you want due to the processing classes. You could always set the cron to every 30 seconds or something?

Even then 2000 x 30 seconds is still 4 hours for a million records.

Normally massive bulk requests from ArcGIS > Maximo  for us are one offs, so we use FME to query esri SQL dev > load via flat file consumer.

 

Original Message:
Sent: 7/15/2022 5:56:00 PM
From: David Miller
Subject: RE: ArcGISSync Support for Pagination?

Kelly,

I beg to differ with your mentality that a million records should bring down MIF. I know for a fact that I have loaded 3 Million+ WO using an IFACE table and its MXINTER_TRANS to manage (it was not fast but worked). And that dates back many years and MIF should be getting better in the day and age of Big Data and the Cloud and not getting worse. I think the design of the ARCGISSYNC is the problem here and we don't get an intertrans table  to manage the rows being brought in by the sync. A 6 GB JVM should be able to handle a million rows of data with not many attributes in connection with a SIB table in the DB (which should be unlimited)

Thanks,

------------------------------
David Miller
Managing Partner - Maximo Consultant
Enterprise Integration Partners LLC

Original Message:
Sent: Fri July 15, 2022 02:35 AM
From: Zoltan Kelly
Subject: ArcGISSync Support for Pagination?

Syrely a million records of any kind will bring down your MIF surely if done in one hit…
On the Esri side you can increase the max record response, but regardless, you should be
1. Setting a flag on Esri side to mark which records need to be updated/added in Maximo (ibm suggest an MXCREATIONSTATE field but can be anything. You can default it on Esri side on add/change with an 'attribute rule' eg to 1
2. using the parameter on the cron to send make a response to mark the records as processed, eg set above to 0
3. Setting the where clause in the url parameter in the cron to only pick up records flagged for update eg where mxcreationstate=1
then each time cron runs it'll do 2000 at a time and eventually get through them all.


Original Message:
Sent: 7/14/2022 10:26:00 AM
From: David Miller
Subject: ArcGISSync Support for Pagination?

I think I have found that the OOTB Maximo ARCGISSYNC process does not support Pagination so thereby cannot load any more then about 2000 rows at a time from single cron task?

Am I misundersatnding and how do you get past this limit?  The row count seems to only be a part of it as the number of columns (attributes) has a role in limiting the number of rows I can get back as well.

Can anyone shed any light on how I can get a million record feature class to sync using OOTB sync functionality? As I am seeing we are going to need to write our own custom auto scripts to accomplish this?

Thanks,

------------------------------
David Miller
Managing Partner - Maximo Consultant
Enterprise Integration Partners LLC
------------------------------



#MaximoSpatial
#Maximo
#AssetandFacilitiesManagement

David,
After a lot of digging around at COA, we have discovered that if the ArcGIS Network Adaptor is running on MicroSoft IIS, there are two places where the size of urls can be constrained or truncated if the urls get too long.  One can be set in the IIS "Request Limits" user interface, which will add a section into the web.config file.  But you must also increase the size of the httpRuntime parameters by adding a second section to web.config.  The httpRuntime default parameters are not listed directly in the web.config file, but must be manually added into the file.  We have discovered this and tested it today and it has allowed us to work with very large numbers of records processed.

https://support.esri.com/en/technical-article/000015245
Problem: GET requests fail when passing through the ArcGIS Web Adaptor

There is NOT a bug in the Spatial software now.  That was my mistake, from past experiences.

------------------------------
Jim Saunders
Saunders International, Inc.
Maximo Spatial and ArcGIS
+1 (256) 808-7512
www.SaundersIntl.com
Jim@SaundersIntl.com
Huntsville, AL
------------------------------

Original Message:
Sent: Thu July 28, 2022 10:32 AM
From: David Miller
Subject: ArcGISSync Support for Pagination?

Lacey,

Thanks for the suggestion. WE will keep this in mind as we get more data to be able to test load and benchmark the differeent methods. I was able to use the cron and a full featuyre class with 80k assets and able to load 73k of them in 1.25 hours. I like your sugeestion to use regular MIF and I would try to spread the load out over multiple JVM to speed it up as well.

Will see how this goies when we get out first data set with millions to load.

Thanks,

------------------------------
David Miller
Managing Partner - Maximo Consultant
Enterprise Integration Partners LLC

Original Message:
Sent: Mon July 18, 2022 07:58 PM
From: Lacey Lurges
Subject: ArcGISSync Support for Pagination?

You can load the initial millions of records through the MIF without the cron task.  You just need to have the PLUSSISGIS, PLUSSFEATURECLASS and whatever fields you are using for your relationship to be set and then the Maximo records will be linked.  And of course whatever other fields you would like populate too.

Going forward, you can set your cron task URL to either use a creation attribute like Zoltan mentions, or you can use the last successful start time in your URL to compare the last time your cron task instance successfully ran with an edited date attribute on your features to synchronize only the changes.  I added a post about this capability in the discussions recently.

------------------------------
Lacey Lurges

Original Message:
Sent: Mon July 18, 2022 04:22 PM
From: David Miller
Subject: ArcGISSync Support for Pagination?

Jim,

I hear what you are saying and agree. We are working in a DEV instance right now and will have a CONVERSION & PROD environments that will be VERY ROBUST compared to our current DEV.

I am missing something on the issue your are speaking of as we have found that we were able to change the MaxRecordCount: 3000  (from default of 2000) in Esri for a Valve Layer (uses a WebMap and a Query to filter it). I was able to push through sucessfully 2,989 asset records with 15 Attributes on a spec template mapped in JSON Mapping. This ran in 49 seconds. I am thrilled with this respoce time in a loiwly DEV environment running on a single JVM for CRON but another for MIF. The math says we could load 1 million Valves in 4.5 hours at that rate. This should be able to be cut in half when we get to our PROD env which will also be using cutting edge Oracle Exadata that I am hoping will crank this up even more not to mention actuial physical servers (not virtual) dedicated to JVM for CRON/MIF etc.

As far as removing the Web Adaptor, I ran this idea by my ESRI person today and he does no think that is possible with our architecture and he mentioned the use of Federated Servers would not allow us to reference ArcGIS by a Port Number due to this design. If you believe differently I would really like more background on how to do that?

Just for additional background we are using ArcGIS Current Version: 10.91 with the UPDM. ANd we have
IBM Maximo Spatial Asset Management 7.6.1.0-20220502-0822 Build 20200716-1706 DB Build V7610-15 HFDB Build HF7610-13

Thanks,

------------------------------
David Miller
Managing Partner - Maximo Consultant
Enterprise Integration Partners LLC

Original Message:
Sent: Mon July 18, 2022 11:02 AM
From: Jim Saunders
Subject: ArcGISSync Support for Pagination?

Maximo Spatial is a live, bi-directional interface between Maximo and ArcGIS. It is not a one-time data load. Large organizations (hundreds of editors) may be working in either ArcGIS or Maximo and their editing efforts will be synched across both systems. The timing of the data synchs is configurable within the cron tasks. You may run the synching process once a day or every minute, depending upon need and computer resources.
The Maximo Spatial cron tasks use the MIF to make calls through the individually configured FeatureServices to make edits in either ArcGIS or Maximo. The cron task makes a call to ArcGIS and looks for values that match the configured trigger.  (i.e. MxCreationState = 1). It then makes multiple queries to find all ArcGIS objects that match the trigger. If you have 10,000 matching ArcGIS objects, the cron task creates 10 queries that return only a list of 1,000 ObjectIDs of the matching objects. Maximo Spatial then uses the MIF to process all 10,000 records in batches of 1,000 at a time. While processing these 10 queries, the cron task knows it is running 10 queries and will not kick off a new cron task instance until all 10 have been processed.

There appears to be a bug within the latest version of Maximo Spatial in that it cannot make proper queries to ArcGIS. Clients may restrict the ArcGIS Max Record count for the FeatureService, to limit how many records can be processed at one time (sometimes this needs to be as low as 100 records) and force Maximo Spatial to operate. It has been seen that this occurs when ArcGIS has a WebAdaptor configured with ArcGIS Server as a part of ArcGIS Enterprise. Sometimes, removing the WebAdaptor from ArcGIS Server, allows Maximo Spatial to operate properly. 

David, you may want to try removing the WebAdaptor temporarily and see if that resolves your issue.

Several Spatial clients currently have tickets with IBM support to try to rectify this issue.  If you are having this issue, I urge you to also enter a support ticket.  The more tickets, the more time will be spent in fixing the problem. And the problem may be partially with the WebAdaptor and partially with Maximo Spatial and may require changes in both environments.

The speed of processing the ArcGIS records is a function of the number of JVMs you assign to Maximo Spatial (and the memory allocated to each JVM) and the queues you set up to handle the processing requests. For large clients, we have created dedicated clusters with dedicated JVMs and dedicated queues for Maximo Spatial, sized to the expected number of users and records to be processed. This is so that other MIF operations and/or processes are not impacted by the Spatial operations and vice versa.

The IBM SWG has previously demonstrated that with as properly setup and properly tuned Maximo environment, they could successfully process over 1 million Pole edits in under 15 minutes, using Maximo Spatial.

------------------------------
Jim Saunders
Saunders International, Inc.
Maximo Spatial and ArcGIS
+1 (256) 808-7512
www.SaundersIntl.com
Jim@SaundersIntl.com
Huntsville, AL

Original Message:
Sent: Fri July 15, 2022 07:42 PM
From: Zoltan Kelly
Subject: ArcGISSync Support for Pagination?

Yeah I get you in that MIF can handle millions of transactions, but you're right, that's not the design of OOTB Maximo Spatial.

There's some logic on the ArcGISDataSync class files thatloops through the each line and sends back a query to ArcGIS to find out if it's an update or insert that slows it down.

On the Esri side you can push the max record count up, but even from their side it's recommended to keep it to <5000 for performance https://enterprise.arcgis.com/en/sites/latest/content/server-configuration-details.htm

There's the Geocortex Maximo data sync product that might be more what you want.

I'm curious about your use case, if this is a one off load, if you're using it for transactions (eg wo etc), or if your assets in ArcGIS change often and and frequently in the order of millions.

You could probably script it to paginate the requests and still use parts of the ArcGISDataSync cron or the classes, but I don't think you'd get the performance jump you want due to the processing classes. You could always set the cron to every 30 seconds or something?

Even then 2000 x 30 seconds is still 4 hours for a million records.

Normally massive bulk requests from ArcGIS > Maximo  for us are one offs, so we use FME to query esri SQL dev > load via flat file consumer.

 

Original Message:
Sent: 7/15/2022 5:56:00 PM
From: David Miller
Subject: RE: ArcGISSync Support for Pagination?

Kelly,

I beg to differ with your mentality that a million records should bring down MIF. I know for a fact that I have loaded 3 Million+ WO using an IFACE table and its MXINTER_TRANS to manage (it was not fast but worked). And that dates back many years and MIF should be getting better in the day and age of Big Data and the Cloud and not getting worse. I think the design of the ARCGISSYNC is the problem here and we don't get an intertrans table  to manage the rows being brought in by the sync. A 6 GB JVM should be able to handle a million rows of data with not many attributes in connection with a SIB table in the DB (which should be unlimited)

Thanks,

------------------------------
David Miller
Managing Partner - Maximo Consultant
Enterprise Integration Partners LLC

Original Message:
Sent: Fri July 15, 2022 02:35 AM
From: Zoltan Kelly
Subject: ArcGISSync Support for Pagination?

Syrely a million records of any kind will bring down your MIF surely if done in one hit…
On the Esri side you can increase the max record response, but regardless, you should be
1. Setting a flag on Esri side to mark which records need to be updated/added in Maximo (ibm suggest an MXCREATIONSTATE field but can be anything. You can default it on Esri side on add/change with an 'attribute rule' eg to 1
2. using the parameter on the cron to send make a response to mark the records as processed, eg set above to 0
3. Setting the where clause in the url parameter in the cron to only pick up records flagged for update eg where mxcreationstate=1
then each time cron runs it'll do 2000 at a time and eventually get through them all.


Original Message:
Sent: 7/14/2022 10:26:00 AM
From: David Miller
Subject: ArcGISSync Support for Pagination?

I think I have found that the OOTB Maximo ARCGISSYNC process does not support Pagination so thereby cannot load any more then about 2000 rows at a time from single cron task?

Am I misundersatnding and how do you get past this limit?  The row count seems to only be a part of it as the number of columns (attributes) has a role in limiting the number of rows I can get back as well.

Can anyone shed any light on how I can get a million record feature class to sync using OOTB sync functionality? As I am seeing we are going to need to write our own custom auto scripts to accomplish this?

Thanks,

------------------------------
David Miller
Managing Partner - Maximo Consultant
Enterprise Integration Partners LLC
------------------------------



#AssetandFacilitiesManagement
#Maximo
#MaximoSpatial