Everything you need to know to start fine-tuning LLMs in the privacy of your home

Got a modern Nvidia or AMD graphics card? Custom Llamas are only a few commands and a little data prep away

Setting up Axolotl for Nvidia GPUs

If you're running an Nvidia graphics card in your system, the process is actually pretty straightforward. 

To begin with, you'll want to make sure you've got git installed on your system as we'll be using it in a bit.

sudo apt install git -y

Install proprietary Nvidia drivers, CUDA

Next, we will install Nvidia's proprietary GPU drivers and CUDA toolkit.

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt-get update
sudo apt-get -y install cuda-drivers cuda-toolkit-12-6
sudo reboot

You can then check that CUDA is configured correctly by running:

nvcc --version

If you get an error about the program not being found, you need to add the location of the CUDA binaries to your PATH variable. For Bash users, the following command will do that; adjust .bashrc to your shell's configuration file if you're not using Bash.

echo "export PATH=/usr/local/cuda-12.6/bin${PATH:+:${PATH}}" >> ~/.bashrc

Close and reopen your terminal after running the above to pick up the change.

If you're reading this a few months after this article was published, you'll want to consult Nvidia's documentation for the latest installation instructions.

Install Miniconda

For this guide, we'll need Python 3.10. Unfortunately, this version of Python doesn't ship with Ubuntu 24.04 and isn't available in its repos. Instead, we'll use Miniconda to create and manage a Python 3.10 environment for us.

You can find instructions on installing and configuring Miniconda here.

Once configured, create and activate a new environment called "axolotl."

conda create -n axolotl python=3.10
conda activate axolotl

Install PyTorch

Next, we'll install PyTorch. At the time of writing, we ran into some trouble getting Axolotl to play nice with PyTorch 2.5 so we'll be using 2.4.1 instead by running the one-liner below:

pip install torch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1 --index-url https://download.pytorch.org/whl/cu124

Installing Axolotl

With our Python 3.10 environment up and running, we should now clone Axolotl from GitHub and begin the installation. This is accomplished by running the following commands:

git clone https://github.com/OpenAccess-AI-Collective/axolotl
cd axolotl
pip install packaging ninja
pip3 install -e '.[flash-attn,deepspeed]'

After a few minutes Axolotl will be ready to go.