AIX Open Source

AIX Open Source

Share your experiences and connect with fellow developers to discover how to build and manage open source software for the AIX operating system

 View Only
  • 1.  ELK Stack support for AIX toolbox

    Posted Sun October 30, 2022 12:04 PM
    Team,

    I've mistakenly posted this into another community (Power). 

    Are there any updates for having a Logstash rpm in the toolbox compatible with AIX 7.1 ? Ultimately, we are looking for the complete ELK stack.

    Checking to see the porting effort / road map. 

    @SANKET RATHI

    Regards.​

    ------------------------------
    Shady Baseely
    ------------------------------


  • 2.  RE: ELK Stack support for AIX toolbox

    Posted Wed November 02, 2022 01:00 PM
    Hi Shady,
    Right now there is no roadmap/plan for ELK stack in AIX toolbox. 
    We will have to study to come up with effort but I am sure it is not easy port.

    ------------------------------
    SANKET RATHI
    ------------------------------



  • 3.  RE: ELK Stack support for AIX toolbox

    Posted Fri November 04, 2022 06:29 AM

    Hello @SANKET RATHI 

    Thank you for the feedback.​



    ------------------------------
    Shady Baseely
    ------------------------------



  • 4.  RE: ELK Stack support for AIX toolbox

    Posted Mon November 07, 2022 07:08 AM
    Hi Shady,

    AFAIR ElasticSearch is a Java application. Kibana is a Node.JS application. Both Java and Node.JS work on AIX. More than a year ago I wrote an article how to start ElasticSearch and Kibana on Linux on Power - https://www.power-devops.com/post/installing-elasticsearch-and-kibana-on-linux-on-power-ppc64le. The same (or similar) way it worked for me on AIX. Never tried with Logstash, but if it is Java why shouldn't it work?

    ------------------------------
    Andrey Klyachkin

    https://www.power-devops.com
    ------------------------------



  • 5.  RE: ELK Stack support for AIX toolbox

    Posted Mon November 07, 2022 07:14 AM
    Hello Andrey,
    Thanks for sharing your input. Actually, I thought about the same since it is a JVM based application, so it should work providing that we have a compatible java with the system itself.

    I'm struggling with running Logstash 8.x on older AIX 7.1 but facing some JRuby native libraries files that need to put somewhere on the system.

    Similar issue: https://github.com/elastic/logstash/issues/3127

    ------------------------------
    Shady Baseely
    ------------------------------



  • 6.  RE: ELK Stack support for AIX toolbox

    Posted Mon November 07, 2022 09:45 AM
    Hi Shady,

    I'm not an expert in Logstash and unfortunately I don't have an AIX 7.1 right now. I tested it on AIX 7.2 TL1 SP6 - this is the oldest AIX I have right now.

    I downloaded the latest logstash from ElasticSearch site and installed it into /opt/logstash:

    # curl -L -O https://artifacts.elastic.co/downloads/logstash/logstash-8.5.0-linux-x86_64.tar.gz
    # gunzip logstash*gz
    # tar xf logstash*tar
    # mv logstash-8.5.0 /opt/logstash​

    Because it is delivered with JDK for x86_64, I have to switch it to IBM JDK for AIX. Logstash uses JDK 17, that's why I decided to use JDK 17 too.

    # curl -L -O https://github.com/ibmruntimes/semeru17-certified-binaries/releases/download/jdk-17.0.4.1%2B1_openj9-0.33.1/ibm-semeru-certified-jdk_ppc64_aix_17.0.4.1.tar.gz
    # rm -rf /opt/logstash/jdk
    # gunzip ibm-semeru-certified-jdk_ppc64_aix_17.0.4.1.tar.gz
    # tar xf ibm-semeru-certified-jdk_ppc64_aix_17.0.4.1.tar -C /opt/logstash
    # cd /opt/logstash
    # ln -s jdk-17.0.4.1+1 jdk​


    To run logstash scripts, you need bash (unless you want to port it to ksh):

    # dnf -y install bash


    You need to amend the call to bash in the first line of /opt/logstash/bin/logstash. Just change #!/bin/bash to #!/opt/freeware/bin/bash. I added /opt/freeware/bin and /opt/logstash/bin to PATH in case I missed something:

    # export PATH=$PATH:/opt/freeware/bin:/opt/logstash/bin


    Starting logstash I've got the following error:

    Dependent module libc++.a(shr_64.o) could not be loaded.


    The reason for the error is old XLC Runtime on the system. I downloaded XLC Runtime 16.1.0.9 from IBM Fix Central and installed additionally the fileset libc++.rte.

    After that I started logstash as it proposed in the documentation for testing and successfully tested it, by entering 'hello world' and getting the message back:

    bin/logstash -e 'input { stdin { } } output { stdout {} }'
    Using bundled JDK: /opt/logstash/jdk
    Sending Logstash logs to /opt/logstash/logs which is now configured via log4j2.properties
    [2022-11-07T15:34:22,823][INFO ][logstash.runner          ] Log4j configuration path used is: /opt/logstash/config/log4j2.properties
    [2022-11-07T15:34:22,834][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"8.5.0", "jruby.version"=>"jruby 9.3.8.0 (2.6.8) 2022-09-13 98d69c9461 Eclipse OpenJ9 VM openj9-0.33.1 on 17.0.4.1+1 +indy +jit [ppc64-aix]"}
    [2022-11-07T15:34:22,856][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xoptionsfile=/opt/logstash/jdk-17.0.4.1+1/lib/options.default, -Xlockword:mode=default,noLockword=java/lang/String,noLockword=java/util/MapEntry,noLockword=java/util/HashMap$Entry,noLockword=org/apache/harmony/luni/util/ModifiedMap$Entry,noLockword=java/util/Hashtable$Entry,noLockword=java/lang/invoke/MethodType,noLockword=java/lang/invoke/MethodHandle,noLockword=java/lang/invoke/CollectHandle,noLockword=java/lang/invoke/ConstructorHandle,noLockword=java/lang/invoke/ConvertHandle,noLockword=java/lang/invoke/ArgumentConversionHandle,noLockword=java/lang/invoke/AsTypeHandle,noLockword=java/lang/invoke/ExplicitCastHandle,noLockword=java/lang/invoke/FilterReturnHandle,noLockword=java/lang/invoke/DirectHandle,noLockword=java/lang/invoke/ReceiverBoundHandle,noLockword=java/lang/invoke/DynamicInvokerHandle,noLockword=java/lang/invoke/FieldHandle,noLockword=java/lang/invoke/FieldGetterHandle,noLockword=java/lang/invoke/FieldSetterHandle,noLockword=java/lang/invoke/StaticFieldGetterHandle,noLockword=java/lang/invoke/StaticFieldSetterHandle,noLockword=java/lang/invoke/IndirectHandle,noLockword=java/lang/invoke/InterfaceHandle,noLockword=java/lang/invoke/VirtualHandle,noLockword=java/lang/invoke/PrimitiveHandle,noLockword=java/lang/invoke/InvokeExactHandle,noLockword=java/lang/invoke/InvokeGenericHandle,noLockword=java/lang/invoke/VarargsCollectorHandle,noLockword=java/lang/invoke/ThunkTuple, -Xjcl:jclse29, -Dcom.ibm.oti.vm.bootstrap.library.path=/opt/logstash/jdk-17.0.4.1+1/lib/default:/opt/logstash/jdk-17.0.4.1+1/lib, -Dsun.boot.library.path=/opt/logstash/jdk-17.0.4.1+1/lib/default:/opt/logstash/jdk-17.0.4.1+1/lib, -Djava.library.path=/opt/logstash/jdk-17.0.4.1+1/lib/default:/opt/logstash/jdk-17.0.4.1+1/lib:/opt/logstash/jdk-17.0.4.1+1/lib/server:/opt/logstash/jdk-17.0.4.1+1/lib:/opt/logstash/jdk-17.0.4.1+1/../lib:/usr/lib64:/usr/lib, -Djava.home=/opt/logstash/jdk-17.0.4.1+1, -Duser.dir=/opt/logstash, -Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED, -Djava.class.path=/opt/logstash/vendor/jruby/lib/jruby.jar:/opt/logstash/logstash-core/lib/jars/checker-qual-3.12.0.jar:/opt/logstash/logstash-core/lib/jars/commons-codec-1.15.jar:/opt/logstash/logstash-core/lib/jars/commons-compiler-3.1.0.jar:/opt/logstash/logstash-core/lib/jars/commons-logging-1.2.jar:/opt/logstash/logstash-core/lib/jars/error_prone_annotations-2.11.0.jar:/opt/logstash/logstash-core/lib/jars/failureaccess-1.0.1.jar:/opt/logstash/logstash-core/lib/jars/google-java-format-1.15.0.jar:/opt/logstash/logstash-core/lib/jars/guava-31.1-jre.jar:/opt/logstash/logstash-core/lib/jars/httpclient-4.5.13.jar:/opt/logstash/logstash-core/lib/jars/httpcore-4.4.14.jar:/opt/logstash/logstash-core/lib/jars/j2objc-annotations-1.3.jar:/opt/logstash/logstash-core/lib/jars/jackson-annotations-2.13.3.jar:/opt/logstash/logstash-core/lib/jars/jackson-core-2.13.3.jar:/opt/logstash/logstash-core/lib/jars/jackson-databind-2.13.3.jar:/opt/logstash/logstash-core/lib/jars/jackson-dataformat-cbor-2.13.3.jar:/opt/logstash/logstash-core/lib/jars/jackson-dataformat-yaml-2.13.3.jar:/opt/logstash/logstash-core/lib/jars/janino-3.1.0.jar:/opt/logstash/logstash-core/lib/jars/javassist-3.29.0-GA.jar:/opt/logstash/logstash-core/lib/jars/jsr305-3.0.2.jar:/opt/logstash/logstash-core/lib/jars/jvm-options-parser-8.5.0.jar:/opt/logstash/logstash-core/lib/jars/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/opt/logstash/logstash-core/lib/jars/log4j-1.2-api-2.17.1.jar:/opt/logstash/logstash-core/lib/jars/log4j-api-2.17.1.jar:/opt/logstash/logstash-core/lib/jars/log4j-core-2.17.1.jar:/opt/logstash/logstash-core/lib/jars/log4j-jcl-2.17.1.jar:/opt/logstash/logstash-core/lib/jars/log4j-slf4j-impl-2.17.1.jar:/opt/logstash/logstash-core/lib/jars/logstash-core.jar:/opt/logstash/logstash-core/lib/jars/reflections-0.10.2.jar:/opt/logstash/logstash-core/lib/jars/slf4j-api-1.7.32.jar:/opt/logstash/logstash-core/lib/jars/snakeyaml-1.30.jar, -Dsun.java.command=org.logstash.Logstash -e input { stdin { } } output { stdout {} }, -Dsun.java.launcher=SUN_STANDARD]
    [2022-11-07T15:34:24,041][INFO ][logstash.settings        ] Creating directory {:setting=>"path.queue", :path=>"/opt/logstash/data/queue"}
    [2022-11-07T15:34:24,074][INFO ][logstash.settings        ] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/opt/logstash/data/dead_letter_queue"}
    [2022-11-07T15:34:24,515][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
    [2022-11-07T15:34:24,597][INFO ][logstash.agent           ] No persistent UUID file found. Generating new UUID {:uuid=>"4d293e30-4cb4-47a3-8f06-9338f3685a88", :path=>"/opt/logstash/data/uuid"}
    [2022-11-07T15:34:30,110][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
    [2022-11-07T15:34:31,127][INFO ][org.reflections.Reflections] Reflections took 688 ms to scan 1 urls, producing 125 keys and 438 values
    [2022-11-07T15:34:32,939][INFO ][logstash.javapipeline    ] Pipeline `main` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
    [2022-11-07T15:34:33,217][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["config string"], :thread=>"#<Thread:0x9bac893d run>"}
    [2022-11-07T15:34:34,901][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>1.68}
    The stdin plugin is now waiting for input:
    [2022-11-07T15:34:35,156][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
    [2022-11-07T15:34:35,312][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
    hello world
    {
        "@timestamp" => 2022-11-07T14:35:06.432137Z,
              "host" => {
            "hostname" => "aixdev"
        },
           "message" => "hello world",
          "@version" => "1",
             "event" => {
            "original" => "hello world"
        }
    }
    [2022-11-07T15:35:13,895][INFO ][logstash.javapipeline    ][main] Pipeline terminated {"pipeline.id"=>"main"}
    [2022-11-07T15:35:14,314][INFO ][logstash.pipelinesregistry] Removed pipeline from registry successfully {:pipeline_id=>:main}
    [2022-11-07T15:35:14,400][INFO ][logstash.runner          ] Logstash shut down.





    ​​

    ------------------------------
    Andrey Klyachkin

    https://www.power-devops.com
    ------------------------------



  • 7.  RE: ELK Stack support for AIX toolbox

    Posted Mon November 07, 2022 09:57 AM
    I got the error, you've mentioned:

    NotImplementedError: stat.st_dev unsupported or native support failed to load; see https://github.com/jruby/jruby/wiki/Native-Libraries​


    ------------------------------
    Andrey Klyachkin

    https://www.power-devops.com
    ------------------------------



  • 8.  RE: ELK Stack support for AIX toolbox

    Posted Mon November 07, 2022 10:37 AM
    As far as I could dig into the problem, I think the problem is somewhere in Logstash Java settings. The reason for all failures is, that it can't load JRuby's FFI library:

    Failed to load native POSIX impl; falling back on Java impl. Stacktrace follows.
    java.lang.UnsatisfiedLinkError: could not load FFI provider jnr.ffi.provider.jffi.Provider​

    It searches on a wrong place with a wrong name:

    Caused by: java.lang.UnsatisfiedLinkError: could not get native definition for type `POINTER`, original error message follows: java.lang.UnsatisfiedLinkError: could not locate stub library in jar file.  Tried [jni/ppc64-AIX/libjffi-1.2.so, /jni/ppc64-AIX/libjffi-1.2.so]


    But even if I try to put it there, it didn't find it. The library per se works perfectly:

    require 'ffi'
    
    module POSIX
      extend FFI::Library
      # this line isn't really necessary since libc is always linked into JVM
      ffi_lib 'libc.a(shr_64.o)'
      
      attach_function :getuid, :getuid, [], :uint
      attach_function :getpid, :getpid, [], :uint
    end
    
    puts "Process #{POSIX.getpid} running as user #{POSIX.getuid}"


    Output:

    /opt/logstash/vendor/jruby/bin/jruby ffi.rb
    Process 7536902 running as user 0





    ------------------------------
    Andrey Klyachkin

    https://www.power-devops.com
    ------------------------------