1. Installation

1. Clone the repository. git clone https://github.com/lm-sys/FastChat.git

2. Navigate into the project directory. cd FastChat

3. Create a new conda environment. conda create -n fastchat python==3.10.0

4. Activate the environment. conda activate fastchat

5. Verify the pip and Python paths to ensure that packages are installed in the correct environment. which pip or which python

6. Install FastChat
pip3 install --upgrade pip
pip3 install -e ".[model_worker,webui]"
To speed up the installation, you can use the Tsinghua source by appending
-i https://pypi.tuna.tsinghua.edu.cn/simple to the command. For example,
pip3 install ".[model_worker,webui]" -i https://pypi.tuna.tsinghua.edu.cn/simple

7. Optional packages
• For inference using fine-tuned models: pip install -e ".[model_worker,llm_judge]"
• For memory efficiency with FlashAttn:
• Ensure you have:
• CUDA 11.6 and above. Use the command nvcc --version to check the CUDA version.
• PyTorch 1.12 and above. Use the import torch print(torch.__version__)
• Install FlashAttn with the command pip install flash-attn
• If you want to use lora or qlora, install the required packages with the command pip install deepspeed bitsandbytes scipy

2. Download the model weights

1. We use Llama2 models for fine-tuning. Start by filling out this form to obtain access.

2. After downloading the weights, they will need to be converted to the Hugging Face Transformers format using the conversion script. The script can be called with the following (example) command:

3. Fine-tune Llama2

1. nproc_per_node : your GPU_number;

2. model_name_or_path : the path of fine-tune model

3. data_path: the path of fine-tune datasets

1. If you want to modify the prompt, you can edit get_conversation_template("vicuna") in file fastchat/train/train.py

2. If you want to use lora, use the following script:

Tips

1. Quick Test Run: You can reducing the number of layers of model to make it simpler, ensuring a faster execution to verify if everything is set up correctly without waiting for a full-scale training.
1. Open the config.json file located in the model directory.
2. modify num_hidden_layers to 2 or other small number.

Bugs??

1. Version Mismatch: If the outputs of nvcc -V and nvidia-smi don't align, run:
Note: Replace cuda-12.2 with your version, ensuring it exists in /usr/local/.