Architecture:        ppc64le
Model name:          POWER9
OS: AlmaLinux 8.10
---------------
I have 4x Tesla V100-SXM2-32GB gpus
NVIDIA-SMI 550.54.15         
Driver Version: 550.54.15   
CUDA Version: 12.4
Cuda compilation tools, release 12.4, V12.4.131
------------------
---------- vLLM try [https://community.ibm.com/community/user/powerdeveloper/blogs/manjunath-kumatagi/2024/06/27/run-vllm-on-ppc64le-architecture]
I installed vllm via docker. It runs well on cpu.
When I try to run it via gpu, it crashes.

----------- Ollama try

Same story with Ollam.

Does anyone know the reason for this? Why does this happen?

------------------------------
Bekbolat Kurmetbek
------------------------------

"It crashes" is not a very detailed problem report. Can you post some error messages?
Furthermore, there is a long discussion on the blog you show a link to. Perhaps you can join that discussion.

------------------------------
Jack Woehr
Senior Consultant
Seiden Group LLC
Beulah CO
3038478442
------------------------------

Hi Bekbolat,
I'm trying to run similar workloads on POWER9 (ppc64le) as well.
Were you able to find the cause of the GPU crash with vLLM or Ollama?
If so, how did you solve it?

I'm especially curious if the issue was related to CUDA compatibility, Docker configuration, or GPU support on ppc64le.

------------------------------
Maria Luísa Gomes
------------------------------