Planning Analytics

Planning Analytics

Get AI-infused integrated business planning

 View Only
Expand all | Collapse all

PAoC outage Amsterdam

  • 1.  PAoC outage Amsterdam

    Posted 29 days ago

    Hi,

    Could we have some clear comms about DR etc for the Amsterdam outage please?  (IBM Watson IoT Service Health Dashboard)

    The current state of the comms is putting customers into a panic, with many in the middle of month and year end. 

    To just say the data centre is on fire with no information about next steps is not great.

    If the datacentre is going to be unavailable what are the next steps, we really need to explain to the customer what the worst case scenario is.



    ------------------------------
    Steven Rowe
    Technical Director
    InfoCat Ltd
    ------------------------------


  • 2.  RE: PAoC outage Amsterdam

    Posted 29 days ago

    100% agree Steve, we have customers asking questions and we need to be able to give them some answers.



    ------------------------------
    Mark Wragg
    ------------------------------



  • 3.  RE: PAoC outage Amsterdam

    Posted 29 days ago

    So an hour and a half to close of business in the UK, the data centre is still on fire and has been since 0730 UK Time.

    What is the plan IBM?

    Please can we have some comms about what is going to happen?

    Trying to stay professional here but the lack of even an outline plan and some expectation setting is really not acceptable.



    ------------------------------
    Steven Rowe
    Technical Director
    InfoCat Ltd
    ------------------------------



  • 4.  RE: PAoC outage Amsterdam

    Posted 28 days ago

    Last update on Planning Analytics on Cloud status dashboard does not give anything new.

    PAW/TM1 still inaccessible, SFTP not working.

    Customers are asking questions like:

    • Is there a DR server to switch over too? When?
    • What version of the data would be restored i.e. when was last backup taken
    • Who is communicating this
    • What are the timelines - we need to provide reports to central banks daily...

    @Tracy Peppy - I know it is not a good time but please can someone give some assurances and timelines.



    ------------------------------
    George Tonkin
    Business Partner
    MCI Consultants
    Johannesburg
    ------------------------------



  • 5.  RE: PAoC outage Amsterdam

    Posted 28 days ago

    Hi Steve, hope you're well.  We were promised an update at 3:00pm BST and received nothing until 6:00pm.  Plans should have kicked in hours ago as it was clear then this centre was not going to be online anytime soon.   I can't believe how poor I think IBM's crisis communication has been.  It's funny that years ago I used to used them for Disaster Recovery services.



    ------------------------------
    David Anderson
    ------------------------------



  • 6.  RE: PAoC outage Amsterdam

    Posted 28 days ago

    Hi David, How's you, hope you are well?  The comms has indeed been poor.  My latest information is that they are restoring customer environments into the Frankfurt data centre, so progress is being made.



    ------------------------------
    Steven Rowe
    Technical Director
    InfoCat Ltd
    ------------------------------



  • 7.  RE: PAoC outage Amsterdam

    Posted 28 days ago

    Hi Steve,

    when you say customers, is that all customers or the ones with premium contracts?



    ------------------------------
    Mark Wragg
    ------------------------------



  • 8.  RE: PAoC outage Amsterdam

    Posted 28 days ago

    Hi Mark, I'm unclear all I know is that they are DRing people into Frankfurt and that it is actively happening.  No indication of delivery times etc



    ------------------------------
    Steven Rowe
    Technical Director
    InfoCat Ltd
    ------------------------------



  • 9.  RE: PAoC outage Amsterdam

    Posted 28 days ago

    The good news:
    No injured persons.

    Some pictures:
    https://www.hv-almere.nl/2026/05/zeer-grote-brand-bij-datacenter-rondebeltweg/

    A video:
    https://nos.nl/artikel/2613443-enorme-rookpluimen-boven-almere-door-grote-brand-in-datacenter



    ------------------------------
    Bernd Siebert
    ------------------------------



  • 10.  RE: PAoC outage Amsterdam

    Posted 28 days ago

    IBM you need to inform us, I hope you have a diaster recovery plan, and you need to roll that our NOW.



    ------------------------------
    Rikke Jacobsen
    CEO
    CogniTech A/S
    Herning
    +4520859352
    ------------------------------



  • 11.  RE: PAoC outage Amsterdam

    Posted 28 days ago

    Something to read whilst we wait:
    https://www.ibm.com/support/customer/csol/terms/?id=i126-6398&lc=en



    ------------------------------
    George Tonkin
    Business Partner
    MCI Consultants
    Johannesburg
    ------------------------------



  • 12.  RE: PAoC outage Amsterdam

    Posted 28 days ago

    Thank you for that George, useful if sobering information!



    ------------------------------
    Mark Wragg
    ------------------------------



  • 13.  RE: PAoC outage Amsterdam

    Posted 22 days ago

    I'm not sure if this is system-wide, but we are finding that once customer environments are restored to FRA, all workspace assets seem to be from a restore point of mid-march. Anything created or edited since mid-march is missing. I'd recommend checking this before giving customers the all clear.

    We have reached out to IBM on all channels, but radio silence so far. We urgently need to update customers on what is going on here. We don't want to waste any time if we have to start recreating everything, but also don't want to go down that route if it is just a simple oversight or if we're imminently going to get access to the original environment now that power has been restored to the AMS data center.



    ------------------------------
    Joseph White
    ------------------------------



  • 14.  RE: PAoC outage Amsterdam

    Posted 22 days ago

    Hi Joseph,

    Can confirm what you found. PAW related assets were restored from mid-March.

    Users and Groups seem to be very current, within a week or two.

    What is still uncertain is when the PAW assets are backed up i.e. weekly, monthly, ad-hoc during maintenance for specific upgrades where MongoDB or other component requires upgrade.

    Also do not forget that all Satellite Connector endpoints need to be recreated and if you previously supplied IBM with a 3rd party/custom ODBC driver or settings, you will need to do this again.



    ------------------------------
    George Tonkin
    Business Partner
    MCI Consultants
    Johannesburg
    ------------------------------



  • 15.  RE: PAoC outage Amsterdam

    Posted 22 days ago

    We are still waiting on restoration despite given an original ETA of end of the 12th May. They've missed two of these target restoration dates now and seemingly any goodwill that existed within the business for IBM's reputation unfortunately.

    This hasn't been a disaster recovery - it has been a hastily planned and execute data center migration. We're on a new TM1 version so they haven't restored host servers. We're still waiting for the data tier to be restored - they could have driven the storage racks to Frankfurt in the time this is taking.

    It's so frustrating!



    ------------------------------
    Chris Davidson
    ------------------------------



  • 16.  RE: PAoC outage Amsterdam

    Posted 22 days ago

    Hi Chris,

    100% with you, we have Finance manically trying to do their reporting in Excel, which is exactly why we needed PA.   This is not giving the senior management confidence what so ever.   We've not been given a date except Saturday at 4:00 AM GMT.

    Jules 



    ------------------------------
    Julian Sedgewick
    ------------------------------