Open Source Development

 View Only
Expand all | Collapse all

Share your experiences porting an app to Linux on Power – what went well, what could have gone better

  • 1.  Share your experiences porting an app to Linux on Power – what went well, what could have gone better

    Posted Wed January 04, 2023 06:14 PM
    Share your experiences porting an app to Linux on Power – what went well, what could have gone better


    ------------------------------
    Linda Alkire
    IBM
    Minneapolis MN
    ------------------------------


  • 2.  RE: Share your experiences porting an app to Linux on Power – what went well, what could have gone better

    Posted Thu January 05, 2023 01:37 PM
    Hi Linda,

    The following are some challenges I faced:
    1. RedHat Core OS images are available as qcow2 format which would need to converted to ova for making them available to import for PowerVS instance. It would be nice to have the ova images made available along with qcow2. Even better would be to have the RHCOS images imported images available in the system images list. In order for me to get an OVA file to create a PowerVS, I need another PowerVS instance to convert images to ova. Minor inconvenience but in terms for user experience this little detail could help developer experience.

    2. For the larger open source community, some dependencies e.g. cryptography don't have ppc64e equivalents in its own dependencies. so the whole package fails on Power due to underlying dependencies not available for ppc64e. It is very difficult to recompile all open source equivalents on ppc64e but if IBM offers a broken repo site where developers can report such dependencies. IBM could make an effort to reach out to open source developers and have their builds targeted for power as well.  Just an idea.

    ------------------------------
    Sambasiva Andaluri
    ------------------------------

    Original Message


  • 3.  RE: Share your experiences porting an app to Linux on Power – what went well, what could have gone better

    Posted Thu January 05, 2023 01:57 PM
    Hi Sambasiva,

    You have two good questions!  I've asked someone else to handle the first one because it is tied into CoreOS and PowerVS which another team provides/delivers.  But for your second point, do you have examples of those packages beyond the cryptography example (a good one, btw).  With that particular example, we face a common challenge - we were able to partner with Travis CI to provide hardware for open source development and we have an ongoing relationship with them to keep it current.  Some of the problems in that chain you identified have been resolved (Travis is FREE for open source developers/projects on ppc64le, and the performance  issues have been resolved).  However, like many communities, adding another CI and maintaining code is difficult.  Further, it is challenging in areas like cryptography where often additional deep technical skills are required if there are any issues.

    We have been working to port and maintain as broad of a set of packages for ppc64le as possible, with more coming all the time, but as you might guess, they all take commitment and headcount.  That said, I'll see if we can look into this one but as you might see from the issue discussion, they are already hesitant on adding any additional packages.

    As far as where to submit packages that are broken or need to be ported, this is a reasonable place to mention them.  The hope over time is that this is a broader community which can come together to resolve these types of issues.

    ------------------------------
    Gerrit
    ------------------------------

    Original Message


  • 4.  RE: Share your experiences porting an app to Linux on Power – what went well, what could have gone better

    Posted Thu January 05, 2023 02:06 PM
    BTW, and thanks to Linda, the other place you can report these that comes almost directly to my team is here:  https://www.ibm.com/it-infrastructure/resources/power-open-source/porting-help

    where you can make specific requests.  Ideally, some level of justification and timeline helps in these requests as we prioritize them against other ongoing work, but we are here to support people who want to use open source on Linux on Power.

    ------------------------------
    Gerrit
    ------------------------------

    Original Message


  • 5.  RE: Share your experiences porting an app to Linux on Power – what went well, what could have gone better

    Posted Thu January 05, 2023 02:58 PM
    Hi Sambasiva,

    Gerrit pinged me about your first question. First of all, thank you so much for providing us with feedback.

    As Gerrit said, we're always trying to get more packages on Power. To add to what he said: any time you find a package you want someone to port to Power, it helps if let that community know, too. Sometimes a community is reluctant to accept our changes to support ppc64le because that they have never seen any customer ask for that support. Vocal partners are so helpful to us.

    --
    Some good news for you -- since you've started using RHCOS on Power, we've been working closely with Red Hat's OpenShift team, and now have OVA bundles specifically for Power VS starting with 4.11. You can get them from their mirror [1] or even import them from regional COS buckets in IBM Cloud. To do this via the Web UI isn't supported yet, so you'll need to use the ibmcloud cli.

    After you've installed the ibmcloud cli power-iaas plugin [2], use the following as an example import:

    ibmcloud pi image-import --bucket rhcos-powervs-images-eu-de --image-file-name rhcos-411-86-202212072103-0-ppc64le-powervs.ova.gz --region eu-de --bucket-access public --job --os-type rhel --disk-type tier1 rhcos-411-86-202212072103-0-ppc64le-powervs

    You can set affinity instead of a disk type, and use a different bucket and region, but you have to use these flags as-is:
    --bucket-access public --job --os-type rhel

    You can find the available (4.11 and later) OVA images on the mirror [1]. Keep in mind also that there are pre-release images there. You'll want to stick to supported releases.

    The regional COS buckets available aren't yet documented, but an easy way to find them is to look in the installer repo [3]. Search for `rhcos-powervs-images-`. The pattern is `rhcos-powervs-images-<region>`. These are COS region names, which are different than the Power VS region names.

    HTH,

    - Christy

    [1] https://mirror.openshift.com/pub/openshift-v4/ppc64le/dependencies/rhcos/
    [2] https://cloud.ibm.com/docs/cli?topic=cli-plug-ins
    [3] https://github.com/openshift/installer/blob/release-4.11/data/data/coreos/rhcos.json#L286-L341

    ------------------------------
    Christy Norman
    ------------------------------

    Original Message


  • 6.  RE: Share your experiences porting an app to Linux on Power – what went well, what could have gone better

    Posted Fri January 06, 2023 10:22 AM
    Hello everyone, nice to finally meet you and I hope you are having a great time. Be patient with me if you would. I'm starting to come out of my shell for group chats. Hope to hear from you soon. The whole environment is very well done. Easy to manage all of my creative needs. 


                              




    Sent from my LG Stylo 6, an AT&T 5G Evolution capable smartphone



    Original Message


  • 7.  RE: Share your experiences porting an app to Linux on Power – what went well, what could have gone better

    Posted Fri January 06, 2023 11:20 AM

    Back in 2019, I started porting IBM Qiskit to Power Linux.

    It was difficult, as many Python packages were difficult to build and weren't in the Power Linux package repository.

    But the most problematic aspect of porting Qiskit to Power Linux was that it was impossible at that time (and may still be) to get reliable and comprehensive information on optimizing the Power platform for scientific computation.  My $800 Intel Xeon box beat out Power Linux at quantum computing simulation by a wide margin.



    ------------------------------
    Jack Woehr
    IBM Champion 2021 -2022
    IBM Qiskit Advocate
    ------------------------------

    Original Message


  • 8.  RE: Share your experiences porting an app to Linux on Power – what went well, what could have gone better

    Posted Fri January 06, 2023 11:29 AM
    Details, please!  :-)  Let's find the gaps and work to fill them.  Maybe the capabilities are there, but there is a dearth of information for finding or using them?  Or, maybe there are capabilities missing?

    What did you try, where did you find dead ends?  There are lots of possibilities where code is written for and optimized on x86, but contains hidden/unintentional x86 bias that causes other platforms to suffer needlessly.

    ------------------------------
    Paul Clarke
    ------------------------------

    Original Message


  • 9.  RE: Share your experiences porting an app to Linux on Power – what went well, what could have gone better

    Posted Fri January 06, 2023 12:48 PM
    There are lots of possibilities where code is written for and optimized on x86, but contains hidden/unintentional x86 bias

    This is true @Paul Clarke ... but there were 2 of us working on this and we did not find much help on working with our performance issues. We found this strange as at the time there was still (ISTR) an IBM supercomputing offering based on Power.
    ​

    ------------------------------
    Jack Woehr
    IBM Champion 2021 -2022
    IBM Qiskit Advocate
    ------------------------------

    Original Message


  • 10.  RE: Share your experiences porting an app to Linux on Power – what went well, what could have gone better

    Posted Fri January 06, 2023 01:15 PM
    I hope you'll tolerate more pestering, because I think this could be fruitful...

    There have been a fair number of articles on porting, optimizing, and tuning applications for Linux on Power.  I wrote a few myself.  I confess that the "center of activity" in which articles like that have been located has changed a few too many times over the years (Knowledge Center, DeveloperWorks, IBM Developer, etc.), but we have tried to keep the content viable in each of the new incantations. Until this relatively new community was established, we also didn't have an effective means of two-way communications.

    I wrote a couple of articles that may have been helpful in the era in question:
    Were you aware of these, and if so, did they help?

    My apologies for repeating an earlier question, but were did you look, what did you try, and where did you hit dead ends? This sort of feedback is immensely helpful. If you hit roadblocks, others likely will hit the same, and it's hard to remove them if we don't know about them!  :-)  Keep the feedback coming!

    ------------------------------
    Paul Clarke
    ------------------------------

    Original Message