I agree.
"Properly fund" brings to mind the phrases like "you get what you pay for," "there is no such thing as a free lunch," and many more,
Original Message:
Sent: Wed September 21, 2022 05:18 AM
From: José Pina Coelho
Subject: Where can I find the JayDeBeApi, JSON, XML, excel2json python libraries?
The only solution to this is two-way approach:
- Demand proper FOSS support&enablement from your vendors.
- Properly fund FOSS projects.
(Do pick a comfortable chair)
------------------------------
José Pina Coelho
IT Specialist at Kyndryl
Original Message:
Sent: Tue September 20, 2022 03:04 PM
From: Donald Williams
Subject: Where can I find the JayDeBeApi, JSON, XML, excel2json python libraries?
I expect that our security team knows that the various distros may not explicitly scan for vulnerabilities. However, each distro will prefer to distribute Open Source Software that works. Therefore they are likely to test to ensure that they "work" in that distros environment. I assume our security team is hoping that the testers will need to modify a package to get it to work, which would improve the odds of a tester detecting malware. Until Open Source Software can be publically and formally certified (assuming that is possible), each company will have a home-grown process to vet Open Source Software.
------------------------------
Donald Williams
Original Message:
Sent: Mon September 19, 2022 10:07 AM
From: Stephen Ulmer
Subject: Where can I find the JayDeBeApi, JSON, XML, excel2json python libraries?
What code review and static analysis is required for the internally written applications? Would it be acceptable to download the Python module sources and subject them to the same tools as the locally written stuff? It is absolutely possible to fetch the module source code into your own repo, and then use that self-same code to deploy with (or at least cryptographically identical code).
Does security understand that there is no additional code review provided by *any* of the entities that package Python modules into RPMs? Red Hat is not examining code that they aren't themselves modifying, and neither is IBM. They are all retrieving the module code from PYPI. These is no reason to trust IBM more than PYPI when it comes to a random Python module. The convince of it being managed by RPM, and tested with the rest of the Toolbox is the value that's delivered by the Toolbox.
Note that I don't speak for IBM, my employer, or even myself if my wife is in the room.
--
Stephen L. Ulmer
Enterprise Architect
Mainline Information Systems
"We are getting bombarded with requests to integrate data from multiple sources and I am seeing that connecting to those sources using python (with additional packages) seems to be the most efficient way. We have IBM Data Stage as our enterprise data integration tool but there are some limitations with the tool and it can't connect to some JDBC sources. We currently have a use case to connect to Infor HR system and IBM said that Data Stage can't connect and up on research I found that python with JayDeBeApi can easily connect to that source."I agree that with so many packages, IBM needs some way to keep packages to a manageable number. José Pina Coelho's suggestion to have IBM's customers vote seems like a reasonable way to determine what to include.
------------------------------Donald WilliamsOriginal Message:Sent: Fri September 16, 2022 06:30 AMFrom: SANKET RATHISubject: Where can I find the JayDeBeApi, JSON, XML, excel2json python libraries?The current set of python packages those we host on AIX toolbox areneeded for Ansible, Salt and other packages.And some other very popular python and base modules needed for other modules. I was checking on RHEL and not all of these python packages are available on RHEL as well in rpm format. Today there is need of these 4 python modules in your environment but in future it might increase so it is difficult for us to keep adding them. I would request if you can find a way that is scalable.------------------------------SANKET RATHIOriginal Message:Sent: Thu September 15, 2022 11:16 PMFrom: Donald WilliamsSubject: Where can I find the JayDeBeApi, JSON, XML, excel2json python libraries?Hi SANKET,I do not work on the other operating systems in our company or with their developers. Therefore I'm not familiar with where they sourced their packages. However, they may not have known that a security review should have been completed before obtaining them.Looking at the massive number of Python packages, it is easy to see that providing all of them would be all but impossible. However, my developer only wants 3 or 4 additional packages. So how do customers suggest or influence which ones IBM will put into the AIX Toolbox?Thanks,Don Williams------------------------------Donald WilliamsOriginal Message:Sent: Wed September 14, 2022 06:30 AMFrom: SANKET RATHISubject: Where can I find the JayDeBeApi, JSON, XML, excel2json python libraries?Hi Don,Can you also find from where other operating system in your companies installing them ? As I see other distros also do not provide packages for all of these python modules. For AIX toolbox team it will be very difficult to provide all the python modules as RPMs, it will be difficult to maintain (update/security fixes etc)------------------------------SANKET RATHIOriginal Message:Sent: Wed September 14, 2022 01:28 AMFrom: Donald WilliamsSubject: Where can I find the JayDeBeApi, JSON, XML, excel2json python libraries? Thanks for the clarification. Currently, our security group will not approve.
I asked:
Is a security review required to use the pip installation method for python modules in a non-IBM repository?
Security replied:
No, we would not approve the use of PIP for installing/maintaining Python installs. That particular repository is pretty much wide-open, allowing anyone to upload packages. There are no gatekeepers vetting the validity of the packages uploaded to PIP so one of them could easily contain malicious code. We would need to stick to the official python repositories maintained by the particular *nix distro being used.
I replied:
It is my understanding that the packages Kiran is asking for have been installed and are in use on other platforms. Therefore, I assume Kiran needs/wants the same functionality on AIX. Would it be possible to download the modules Kiran asked for and have them scanned/vetted to allow them to be installed on AIX?
Security replied:
Even if those packages are installed, it doesn't mean they were sourced through PIP. The issue wouldn't necessarily be with the packages requested, but rather their source. It could be that on the other platforms, the packages were part of the supported distro where they are not with AIX.
Therefore, there would be no problem if the packages were available via the AIX Toolbox since security has already approved that source.
Thanks,
Don Williams | System Architect IT Computer Operations & Technical Support
UNC Health
WFH
p (984) 974-7148
Donald.Williams@unchealth.unc.edu
Sent from my office laptop
----- Confidentiality Notice -----The information contained in (or attached to) this electronic message may be legally privileged and/or confidential information. If you have received this communication in error, please notify the sender immediately and delete the message.
In retrospect, I realize that I was unclear: I meant that someone else in your enterprise was using Python for something
completely different, like on another OS (Windows, Linux, etc.) and then *they* may have already gotten approval... or they could totally be ignoring the issue. ;)
It sounds like you are on the right track.
Just some thoughts that have helped others in the past - you can obviously totally ignore these if they are contra to your predilections:
When making your case with the security team, you can note that this is the same problem that they will have with *any* of the Open Source languages, even more so with things like Node (which has a bazillion packages that all move in different directions at the speed of reckless), or Ruby, or Perl, or PHP, or anything that has a community-developed library. So if you have web or full-stack developers in your environment they are already facing this problem (or ignoring it).
Invocation of other silos in the enterprise may help the security people understand that you are asking for something fairly common.
Also, the developers using the downloads straight from PYPI is actually good from a testing/management point of view. You can pin the exact versions, you can install the modules on a per-account or per-app basis, etc. So they should have a better testing experience and it should be easier to communicate the environment when rolling from development to production.
Liberty,
--
Stephen L. Ulmer Enterprise Architect
Mainline Information Systems
Original Message:
Sent: 9/12/2022 11:01:00 PM
From: Donald Williams
Subject: RE: Where can I find the JayDeBeApi, JSON, XML, excel2json python libraries?
All of our AIX systems are rather vanilla. For the most part, we have only allowed commercial software to be installed on our AIX systems. Of course, many commercial software products use open-source components, but those modules would have been embedded in the product. Now our developers want to use open-source modules in their in-house apps. I expect many have just downloaded whatever they needed into their PCs/laptops. Since it worked great on their PCs, they want the same thing on the AIX systems (assuming that an AIX version is available). However, they will have to request that my team install their modules, but I will not bypass our Security department. I'll be ordering a security review of the AIX requests, even if I may not be popular if that triggers a security review of their PC software.
What is the URL for the pip repos (I assume that it is not an IBM site)? I'll need that to request a security review and to download or request firewall rules.
Thanks,
Don WIlliams
------------------------------
Donald Williams
Original Message:
Sent: Mon September 12, 2022 04:46 PM
From: Stephen Ulmer
Subject: Where can I find the JayDeBeApi, JSON, XML, excel2json python libraries?
You can use pip on another machine to download the sources or wheel files. Wheels are Python binary packages, so they are unique per OS and architecture. Pip will even calculate the prerequisites for you! There is another thread somewhere here where I give an example of that (I can dig that up or make it more complete if necessary). Note that this only grabs the Python bits, if some of those need a library of some kind, you'll still have to get the package for that from the Toolbox.
I would be surprised if nobody else in your enterprise uses the Python Package Index (so it may already be authorized). In general, trying to make the Internet responsible for your internal security will never work, so there may already be another policy that would apply. Either your security people should have a scanning process for those downloads or be a willing to "bless" some "official" sources.
Good luck, and let us know if we can provide more detail!
--
Stephen L. Ulmer
Enterprise Architect
Mainline Information Systems
Original Message:
Sent: 9/12/2022 1:37:00 PM
From: Donald Williams
Subject: RE: Where can I find the JayDeBeApi, JSON, XML, excel2json python libraries?
Our AIX systems are behind a firewall with only white-listed access to the Internet. I could request a firewall rule for a pip repo site or I could manually download modules. But in either case, our security dept would require the vendor/website owner to fill out a security questionnaire before authorizing either option. Since IBM is already approved, I can freely connect/download modules from the AIX Toolbox.
I did not see any pip repo sites listed in the output you posted. What are the pip repo sites?
Thanks,
Don Williams
------------------------------
Donald Williams
Original Message:
Sent: Mon September 12, 2022 09:25 AM
From: SANKET RATHI
Subject: Where can I find the JayDeBeApi, JSON, XML, excel2json python libraries?
Hi Donald,
There are many many python packages available and it is difficult to package all of them in RPM and manage.
We provide python package in RPMs those are most used.
Other distros also rely on the pip to install python modules.
If you have internet connection to the AIX system you can use pip command like following to install python packages those are not there on toolbox.
Otherwise also you can download source tar of package from pip repo and use pip command to install from that.
# /opt/freeware/bin/python3 -m pip install <module name>
You also need to install gcc and other packages.
I tried on my system and I could install these python packages using pip
# /opt/freeware/bin/python3 -m pip install JayDeBeApi
Collecting JayDeBeApi
Using cached JayDeBeApi-1.2.3-py3-none-any.whl (26 kB)
Collecting JPype1; python_version > "2.7" and platform_python_implementation != "Jython"
Using cached JPype1-1.4.0.tar.gz (1.2 MB)
Requirement already satisfied: typing_extensions in /opt/freeware/lib/python3.7/site-packages (from JPype1; python_version > "2.7" and platform_python_implementation != "Jython"->JayDeBeApi) (4.3.0)
Using legacy setup.py install for JPype1, since package 'wheel' is not installed.
Installing collected packages: JPype1, JayDeBeApi
Running setup.py install for JPype1 ... done
Successfully installed JPype1-1.4.0 JayDeBeApi-1.2.3
WARNING: You are using pip version 20.1.1; however, version 22.2.2 is available.
You should consider upgrading via the '/opt/freeware/bin/python3 -m pip install --upgrade pip' command.
# /opt/freeware/bin/python3 -m pip install jsons
Collecting jsons
Downloading jsons-1.6.3-py3-none-any.whl (60 kB)
|################################| 60 kB 1.1 MB/s
Collecting typish>=1.9.2
Downloading typish-1.9.3-py3-none-any.whl (45 kB)
|################################| 45 kB 701 kB/s
Installing collected packages: typish, jsons
Successfully installed jsons-1.6.3 typish-1.9.3
WARNING: You are using pip version 20.1.1; however, version 22.2.2 is available.
You should consider upgrading via the '/opt/freeware/bin/python3 -m pip install --upgrade pip' command.
# /opt/freeware/bin/python3 -m pip install excel2json
Collecting excel2json
Downloading excel2json-1.0.1.zip (15 kB)
Requirement already satisfied: six>=1.10.0 in /opt/freeware/lib/python3.7/site-packages (from excel2json) (1.13.0)
Collecting xlrd>=1.0.0
Downloading xlrd-2.0.1-py2.py3-none-any.whl (96 kB)
|################################| 96 kB 969 kB/s
Using legacy setup.py install for excel2json, since package 'wheel' is not installed.
Installing collected packages: xlrd, excel2json
Running setup.py install for excel2json ... done
Successfully installed excel2json-1.0.1 xlrd-2.0.1
WARNING: You are using pip version 20.1.1; however, version 22.2.2 is available.
You should consider upgrading via the '/opt/freeware/bin/python3 -m pip install --upgrade pip' command.
# /opt/freeware/bin/python3 -m pip install xml-python
Collecting xml-python
Downloading xml_python-0.4.3-py3-none-any.whl (9.2 kB)
Collecting attrs
Downloading attrs-22.1.0-py2.py3-none-any.whl (58 kB)
|################################| 58 kB 956 kB/s
Installing collected packages: attrs, xml-python
Successfully installed attrs-22.1.0 xml-python-0.4.3
WARNING: You are using pip version 20.1.1; however, version 22.2.2 is available.
You should consider upgrading via the '/opt/freeware/bin/python3 -m pip install --upgrade pip' command.
------------------------------
SANKET RATHI
Original Message:
Sent: Fri September 09, 2022 11:40 AM
From: Donald Williams
Subject: Where can I find the JayDeBeApi, JSON, XML, excel2json python libraries?
I do not know how to deal with Python packages using python. I used YUM to install python. That appeared to work, but I do not know how to test it.
Note: We have not figured out how to replace YUM with DNF yet.
The developer also asked for the following libraries:
JayDeBeApi | Not on AIX Toolbox |
Pandas | |
Requests ( for REST API calls) | |
numpy | |
JSON | Not on AIX Toolbox |
XML | Not on AIX Toolbox |
excel2json | Not on AIX Toolbox |
I assume I can use YUM to install Pandas, Requests, and numpy from the AIX Toolbox.
However, I do not know where to obtain the other ones nor where to obtain pip.
Also, our developer asked, "I wonder why not all libraries are available."
So, how does IBM decide which packages to include in AIX Toolbox?
dbwillia@iswhsenga-m1d1 631$ oslevel -s
7200-05-03-2136
[/home/dbwillia] RC=0
Thanks,
Don Williams | System Architect
IT Computer Operations & Technical Support
UNC Health
WFH
p (984) 974-7148
Donald.Williams@unchealth.unc.edu
Sent from my office laptop
----- Confidentiality Notice -----
The information contained in (or attached to) this electronic message may be legally privileged and/or confidential information. If you have received this communication in error, please notify the sender immediately and delete the message.
Original Message:
Sent: 9/8/2022 9:12:00 PM
From: Stephen Ulmer
Subject: RE: Where can I find the JayDeBeApi, JSON, XML, excel2json python libraries?
I think that most of these will install with pip, though you may need to install some prerequisites with RPM first. It's been a while, so I might have to hit my lab and see about current versions.
Do you know how to deal with Python packages using python, or do you need some help with that? Also, what level of AIX, and do you need other toolbox packages, or just some python modules?
--
Stephen L. Ulmer
Enterprise Architect
Mainline Information Systems
Original Message:
Sent: 9/7/2022 4:22:00 PM
From: Donald Williams
Subject: Where can I find the JayDeBeApi, JSON, XML, excel2json python libraries?
One of our developers asked for the following python libraries:
JayDeBeApi | Not on AIX Toolbox |
Pandas | |
Requests ( for REST API calls) | |
numpy | |
JSON | Not on AIX Toolbox |
XML | Not on AIX Toolbox |
excel2json | Not on AIX Toolbox |
How do we request additional python libraries be added to the AIX Toolbox?
Or can I obtain those libraries from a diffrent source?
------------------------------
Donald Williams
------------------------------
#AIXOpenSource