LLAMA-CPP-PYTHON on RTX4060 GPU

LLAMA-CPP-PYTHON on NVIDIA RTX4060 GPU

We try to use llama-cpp-python library with many OS.

Windows 11

Ubuntu 22.04

and Colab ( ubuntu 22.04 also )



Llama-cpp-python library with RTX4060 GPU on Windows11

Install NVDIA GPU Driver.

we use driver version 537.24


then check nvidia-smi command for check your GPU.


Install CUDA Toolkit.  https://developer.nvidia.com/cuda-downloads

we use CUDA 12.1 version


make sure your cuda is work fine with nvcc command.




We use Python 3.11.7


check python and pip with command line.


Install Pytorch https://pytorch.org/get-started/locally/


check pip list and check cuda available on your GPU with python shell.

if false it cannot use with NVIDIA GPU. find your problem. 



Install llama-cpp-python https://pypi.org/project/llama-cpp-python/


with CUDA 12.1 use this command.

on Jul 2,2024 version 0.2.81 but we use only 0.2.75 it's work fine. ( 0.2.81 not test on Windows11 yet ) 

pip install llama-cpp-python \
  --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu121

pip list your library.


How to check your llama-cpp-python lib work with CUDA?

you use LLM code and load your LLM model to test llama-cpp-python lib.

my code test on jupyter notebook in VS code.

we try to load Llama3-Typhoon1.5-8b 4bit GGUF format from Hugging Face scb10x/llama-3-typhoon-v1.5-8b-instruct






When load model done. Check BLAS = 1 or not
if BLAS = 0 it's not use with cuda yet. find your problem.
But your can use with CPU only so it is very slow inference.

Now you can use llama-cpp-python library on your windows11.

Llama-cpp-python library with RTX4060 GPU on Ubuntu 22.04.03 LTS


Install NVDIA GPU Driver.

we use Nvidia-driver version 535



then check nvidia-smi command for check your GPU.

Install CUDA Toolkit.  https://developer.nvidia.com/cuda-downloads

we use CUDA 11.5 version



Install Pytorch https://pytorch.org/get-started/locally/



check pip list and check cuda available on your GPU with python shell.

if false it cannot use with NVIDIA GPU. find your problem. 



Install llama-cpp-python https://pypi.org/project/llama-cpp-python/

on Jul 2,2024 version 0.2.81 but we use only 0.2.32 it's work fine. ( 0.2.81 not test on Ubuntu yet )

Install  pip Command
CMAKE_ARGS="-DLLAMA_CUDA=on" pip install llama-cpp-python==0.2.32

pip list your library.


How to check your llama-cpp-python lib work with CUDA?

you use LLM code and load your LLM model to test llama-cpp-python lib.

my code test on jupyter notebook in VS code.

we try to load Llama3-Typhoon1.5-8b 4bit GGUF format from Hugging Face scb10x/llama-3-typhoon-v1.5-8b-instruct




When load model done. Check BLAS = 1 or not
if BLAS = 0 it's not use with cuda yet. find your problem.
But your can use with CPU only so it is very slow inference.

Now you can use llama-cpp-python library on your Ubuntu OS.

https://huggingface.co/Adun

Adun Nantakaew อดุลย์ นันทะแก้ว

LINE : adunnan

ความคิดเห็น

แสดงความคิดเห็น

โพสต์ยอดนิยมจากบล็อกนี้

Llama3 Typhoon v1.5 (scb10x) LLM

รูปภาพ
Llama3 Typhoon v1.5 (scb10x) LLM   Typhoon-1.5 models come in 8B and 72B sizes These models are built on the 8B Llama3 and 72B Qwen base models,  8B weights released under the Meta Llama 3 Community License ; 72B weights released under the Tongyi Qianwen License Performance To gain insight into Typhoon’s performance, we evaluated it using multiple-choice exam: Language & Knowledge Capabilities: We assessed Typhoon on multiple-choice question answering datasets, including ThaiExam , M3Exam , and MMLU . The ThaiExam dataset was sourced from standard examinations in Thailand, including ONET, TGAT, TPAT, and A-Level. M3Exam is a benchmark for Southeast Asian countries, including Thailand. MMLU is a standard benchmark for language models in English. Typhoon-1.5X is an eXperimental model designed for application use cases, featuring improved capabilities in Retrieval-Augmented Generation (RAG) , constrained generation , and reasoning in order to achieve competitive performance in in
อ่านเพิ่มเติม

Eval Llama v3.1 8B ,70B Model with Thai Language

รูปภาพ
Eval Llama v3.1 8B ,70B  ด้วย ข้อสอบภาษาไทย   เทียบกับ llama3-typhoon ทดสอบ การใช้งาน llama v3.1 ที่ว่า support ภาษาไทย จะทำข้อสอบ ภาษาไทยได้ดีแค่ไหน มาดูกัน ข้อสอบ • 04_investment_consult • 05_facebook_beleble_th_200 • 06_xcopa_th_200 • 07_xnli2.0_th_200 • 08_onet_m3_thai • 09_onet_m3_social • 10_onet_m3_math • 11_onet_m3_science • 12_onet_m3_english • 13_onet_m6_thai • 14_onet_m6_math • 15_onet_m6_social • 16_onet_m6_science • 17_onet_m6_english Code ทดสอบ มาจาก  OpenThaiGPT - Thai Exams Eval https://github.com/OpenThaiGPT/openthaigpt_eval การทดสอบด้วยการทำข้อสอบแบบ Multiple Choice ภาษาไทย ทดสอบบนชุดข้อสอบที่ไม่เคยเห็นมาก่อน ทดสอบแบบ Zero-shot Learning, โค้ดการทดสอบและเนื้อหาข้อสอบสามารถทำตามได้ที่นี่ ข้อมูลของ ข้อสอบ https://docs.google.com/spreadsheets/d/1ZtP5Jkx0IvCWNPQhMKitZszGnLKqvEDEf0OKdmQiXjA/edit#gid=1181424412 ผลการทดสอบ ผลการทดสอบ Average Code การทดสอบ code การ plot Graph Adun Nantakaew อดุลย์ นันทะแก้ว LINE : adunnan Facebook : https://www.facebook.com/softpowergroup Y
อ่านเพิ่มเติม