Data and AI on Power

 View Only
Expand all | Collapse all

Question about MMA, GPU, CPU and types of LLMs

  • 1.  Question about MMA, GPU, CPU and types of LLMs

    Posted yesterday

    Hello,
    As part of our AI learning, we ran Red hat (ppc64le) on one of our Power10 machine. We are at the beginning of our journey with AI and it is difficult for us to understand some things, so a few questions come to my mind:
    1) How does MMA in Power10 processors compare to technologies such as GPU, TPU? What does it look like in terms of performance? (e.g. I see that a Macbook with an m1 processor generates text faster)
    2) We downloaded two models 8B-SPPO-Iter3-Q8_0.gguf and 8B-SPPO-Iter3-Q6_K.gguf and 8B-SPPO-Iter3-Q8_0.gguf is clearly faster even though it is larger(in theory it should be slower), why is this so? Should we choose a specific type of LLM from huggingface.co for MMA technology?

    We use llama.cpp from this manual https://community.ibm.com/community/user/powerdeveloper/blogs/vaibhav-shandilya/2024/05/07/prepare-ibm-power10-for-inferencing-with-llms



    ------------------------------
    Kamil
    ------------------------------