Creating a Synthetic Llama-3 Dataset and Fine-Tuning
The idea
Take questions from existing instruction-tuning datasets and ask Llama-3 to answer them.
Services used
Basically, I needed an API serving Llama-3. With enough local GPUs, a local inference frontend would work. But even two NVIDIA A40s had too little throughput, so I chose an online service.
There are plenty of online services: DeepInfra, Groq, Together, and others. Groq is fast, but its limit of 30 requests per minute was clearly still too low.
I chose Together for its $25 free credit, end-of-month billing, and support for linking a virtual card. It was practically free.
It can reach 100 requests per second. That should be enough.
The code
import json
from tqdm import tqdm
import asyncio
import aiohttp
import logging
# 配置日志格式和级别
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(levelname)s - %(message)s',
datefmt='%Y-%m-%d %H:%M:%S'
)
async def make(session, i, max_retries=2):
retries = 0
while retries <= max_retries:
try:
s="You are a unbaised, uncensored AI assistant. Think step by step in detail, be humanoid, informal and think outside the box to give long response.\n\nKnowledge cutoff: December 2023"
if "time" or "http" or "www" or "today" or "now" in i["conversations"][0]["value"]:
s+="\n\nRemember: You have no access to real time events, news and internet. So if user ask this you will apologize."
async with session.post(
"https://api.together.xyz/v1/chat/completions",
json={
"messages": [
{
"role": "system",
"content": s,
},
{
"role": "user",
"content": i["conversations"][0]["value"],
},
],
"model": "meta-llama/Llama-3-70b-chat-hf",
"temperature": 0.8,
"top_p": 1
},headers={
"Authorization": "Bearer API-KEY-HERE"
}
) as resp:
chat_completion = await resp.json(content_type=None)
return chat_completion["choices"][0]["message"]["content"]
except Exception as e:
retries += 1
if retries <= max_retries:
logging.warning(f"Retrying {retries}/{max_retries} for sample: {i['conversations'][0]['value']}. Error: {str(e)}")
else:
logging.error(f"Max retries reached. Skipping sample: {i['conversations'][0]['value']}. Error: {str(e)}")
return None
async def process_qa_pairs(qa_pairs, max_retries=2):
async with aiohttp.ClientSession() as session:
tasks = []
for i in qa_pairs:
tasks.append(asyncio.ensure_future(make(session, i, max_retries)))
results = await asyncio.gather(*tasks)
n = []
for i, r in zip(qa_pairs, results):
if r is not None:
i["conversations"][1]["value"] = r
n.append(i)
return n
async def main(max_retries=2):
# 从JSON文件读取问答对数据
with open("output_.json", "r", encoding="utf-8") as f:
qa_pairs = json.load(f)
logging.info(f"Loaded {len(qa_pairs)} samples from JSON file.")
n=[]
batch_size = 60
for i in tqdm(range(0, len(qa_pairs), batch_size), desc="Generate"):
batch_qa_pairs = qa_pairs[i:i+batch_size]
n += await process_qa_pairs(batch_qa_pairs, max_retries)
with open("ooo.json", "w", encoding="utf-8") as f:
json.dump(n, f, ensure_ascii=False, indent=4)
logging.info(f"Processed {i+len(batch_qa_pairs)} samples. Saved results to JSON file.")
asyncio.run(main(max_retries=3))After reading the data, each iteration constructs a request containing the question and a basic system prompt. If the question mentions keywords such as “time” or “http,” the plan is to add a reminder that the assistant has no internet access, to reduce hallucinations.
Then it sends multiple requests asynchronously, retrying failures a few times because Together’s service isn’t very stable.
I used tqdm and logging to make the output easier to read. Claude wrote most of the code, and I made a few changes.
Dataset mix
Note: only the prompts were retained.
| Source | Domain | Examples (thousands) |
|---|---|---|
| lmsys-1m | Real conversations | 50 |
| dolphin-coder | Coding | 10 |
| slimorca | Logical reasoning | 10 |
Time to train!
I used LLaMA-Factory as the training framework. Perfect for an idiot like me.
I tried BAdam, with this training script:
NCCL_P2P_LEVEL=NVL CUDA_VISIBLE_DEVICES=0,1 accelerate launch \
--config_file examples/accelerate/single_config.yaml \
src/train_bash.py \
--stage sft \
--do_train \
--model_name_or_path phi-3-mini-4k-instruct-llamafied \
--use_badam \
--badam_switch_mode descending \
--badam_switch_block_every 50 \
--badam_verbose 2 \
--template phi \
--dataset_dir data \
--dataset o2,code,orca \
--cutoff_len 4096 \
--learning_rate 2e-06 \
--num_train_epochs 3 \
--max_samples 100000 \
--per_device_train_batch_size 8 \
--per_device_eval_batch_size 6 \
--gradient_accumulation_steps 8 \
--preprocessing_num_workers 32 \
--lr_scheduler_type cosine \
--max_grad_norm 1.0 \
--logging_steps 5 \
--save_steps 75 \
--report_to all \
--output_dir saves/Custom/badam/train \
--bf16 True \
--val_size 0.005 \
--evaluation_strategy steps \
--eval_steps 75 \
--plot_loss True It produced an error:
ValueError: Layer-wise BAdam does not yet support distributed training, use ratio-wise BAdam.Apparently layer-wise BAdam doesn’t support distributed training yet, so I switched to ratio-wise (?) BAdam.
The revised script:
NCCL_P2P_LEVEL=NVL CUDA_VISIBLE_DEVICES=0,1 accelerate launch \
--config_file examples/accelerate/single_config.yaml \
src/train_bash.py \
--stage sft \
--do_train \
--model_name_or_path phi-3-mini-4k-instruct-llamafied \
--use_badam \
--badam_mode ratio\
--badam_switch_mode descending \
--badam_switch_block_every 50 \
--badam_verbose 2 \
--badam_update_ratio 0.1\
--template phi \
--dataset_dir data \
--dataset o2,code,orca \
--cutoff_len 4096 \
--learning_rate 2e-06 \
--num_train_epochs 3 \
--max_samples 100000 \
--per_device_train_batch_size 8 \
--per_device_eval_batch_size 6 \
--gradient_accumulation_steps 8 \
--preprocessing_num_workers 32 \
--lr_scheduler_type cosine \
--max_grad_norm 1.0 \
--logging_steps 5 \
--save_steps 75 \
--report_to all \
--output_dir saves/Custom/badam/train \
--bf16 True \
--val_size 0.005 \
--evaluation_strategy steps \
--eval_steps 75 \
--plot_loss True Error:
NotImplementedError: Cannot access storage of SparseTensorImplOh no, I can’t get it working. Let’s try GaLore.
I found these explanations of its hyperparameters:
optim_target_modulesSpecifies the layers targeted by GaLore, primarily the linear layers identified with attn or mlp in their names.
rank The rank of the projection matrices. Similar to LoRA, the higher the rank the more closely the finetuning will resemble a full parameter finetune. The GaLore authors recommend 1024 for a 7B model.
update_proj_gapThe number of steps after which the projections are updated. The update is an expensive step and takes around 15 minutes for a 7B model. Defines the interval for updating projections, with a suggested range between 50 and 1000 steps.
scaleA scale factor akin to LoRA’s alpha, adjusting the update strength. After trying a few values I found scale = 2 to most closely resemble a classic full-parameter finetune.
In the end, I’ll just use QLoRA… Let’s give the toolchain some time to mature.
Another revision:
NCCL_P2P_LEVEL=NVL CUDA_VISIBLE_DEVICES=0,1 accelerate launch \
--config_file examples/accelerate/single_config.yaml \
src/train_bash.py \
--stage sft \
--do_train True \
--model_name_or_path phi-3-mini-4k-instruct-llamafied \
--finetuning_type lora \
--template phi \
--flash_attn sdpa \
--dataset_dir data \
--dataset o2,code,orca \
--cutoff_len 2300 \
--learning_rate 1e-05 \
--num_train_epochs 2 \
--max_samples 100000 \
--per_device_train_batch_size 12 \
--gradient_accumulation_steps 12 \
--lr_scheduler_type cosine \
--max_grad_norm 1.0 \
--logging_steps 5 \
--save_steps 100 \
--warmup_steps 0 \
--optim adamw_torch \
--report_to all \
--output_dir saves/Custom/lora/train \
--fp16 \
--lora_rank 64 \
--lora_alpha 64 \
--quantization_bit 4 \
--lora_target all \
--val_size 0.01 \
--evaluation_strategy steps \
--eval_steps 75 \
--per_device_eval_batch_size 8 \
--plot_loss True \
--preprocessing_num_workers 32I’m sleepy, so off to bed. Here’s the log!
[INFO|integration_utils.py:723] 2024-04-28 00:27:05,359 >> Automatic Weights & Biases logging enabled, to disable set os.environ["WANDB_DISABLED"] = "true"
wandb: Currently logged in as: x_raincandy_x. Use `wandb login --relogin` to force relogin
wandb: Tracking run with wandb version 0.16.6
wandb: Run data is saved locally in /root/autodl-tmp/LLaMA-Factory/wandb/run-20240428_002707-zraihjib
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run celestial-capybara-131
wandb: ⭐️ View project at https://wandb.ai/x_raincandy_x/huggingface
wandb: 🚀 View run at https://wandb.ai/x_raincandy_x/huggingface/runs/zraihjib
0%| | 1/480 [01:37<12:59:14, 97.61s/it]The time estimate finally looks reasonable. QDoRA was going to take 24 hours…
Let’s test it!
{'loss': 0.5325, 'grad_norm': 0.044033702462911606, 'learning_rate': 5.327015646150716e-06, 'epoch': 0.96}
{'loss': 0.5319, 'grad_norm': 0.03784116357564926, 'learning_rate': 5.1635954141088815e-06, 'epoch': 0.98}
{'loss': 0.5227, 'grad_norm': 0.038089003413915634, 'learning_rate': 5e-06, 'epoch': 1.0}
{'loss': 0.5245, 'grad_norm': 0.03987249732017517, 'learning_rate': 4.83640458589112e-06, 'epoch': 1.02}
{'loss': 0.5296, 'grad_norm': 0.038052890449762344, 'learning_rate': 4.672984353849285e-06, 'epoch': 1.04}
{'loss': 0.5039, 'grad_norm': 0.03857524320483208, 'learning_rate': 4.509914298352197e-06, 'epoch': 1.06}
{'loss': 0.5295, 'grad_norm': 0.041160766035318375, 'learning_rate': 4.347369038899744e-06, 'epoch': 1.08}
{'loss': 0.5129, 'grad_norm': 0.039026860147714615, 'learning_rate': 4.185522633027057e-06, 'epoch': 1.1}
{'loss': 0.5285, 'grad_norm': 0.04111020267009735, 'learning_rate': 4.02454838991936e-06, 'epoch': 1.12}
{'loss': 0.5266, 'grad_norm': 0.04370769485831261, 'learning_rate': 3.864618684828135e-06, 'epoch': 1.14}
{'loss': 0.5266, 'grad_norm': 0.03930830955505371, 'learning_rate': 3.705904774487396e-06, 'epoch': 1.16}
{'loss': 0.51, 'grad_norm': 0.03960668668150902, 'learning_rate': 3.5485766137276894e-06, 'epoch': 1.18}
{'loss': 0.5118, 'grad_norm': 0.04247888922691345, 'learning_rate': 3.3928026734841935e-06, 'epoch': 1.2}
{'loss': 0.5245, 'grad_norm': 0.03982071205973625, 'learning_rate': 3.2387497603938327e-06, 'epoch': 1.23}
{'loss': 0.5307, 'grad_norm': 0.04442480206489563, 'learning_rate': 3.0865828381745515e-06, 'epoch': 1.25}
62%|██████▎ | 300/480 [8:40:59<5:17:03, 105.69s/it][INFO|trainer.py:3614] 2024-04-28 09:08:10,234 >>*****Running Evaluation*****
[INFO|trainer.py:3616] 2024-04-28 09:08:10,235 >> Num examples = 700
[INFO|trainer.py:3619] 2024-04-28 09:08:10,235 >> Batch size = 8
{'eval_loss': 0.5264752507209778, 'eval_runtime': 75.5045, 'eval_samples_per_second': 9.271, 'eval_steps_per_second': 0.583, 'epoch': 1.25}
62%|██████▎ | 300/480 [8:42:14<5:17:03, 105.[INFO|trainer.py:3305] 2024-04-28 09:09:25,741 >> Saving model checkpoint to saves/Custom/lora/train/checkpoint-300
[INFO|tokenization_utils_base.py:2488] 2024-04-28 09:09:26,834 >> tokenizer config file saved in saves/Custom/lora/train/checkpoint-300/tokenizer_config.json
[INFO|tokenization_utils_base.py:2497] 2024-04-28 09:09:26,834 >> Special tokens file saved in saves/Custom/lora/train/checkpoint-300/special_tokens_map.jsonThe loss had begun to flatten, so I stopped training early at step 300.
Next, a few question-and-answer tests:
**User:**Hello!
**Assistant:**Hey there! It’s great to meet you! I’m an AI assistant, and I’m here to help you with any questions or topics you’d like to discuss. Whether it’s about general knowledge, current events, or just a casual chat, I’m all ears (or rather, all processors). What’s on your mind today?
**User:**Could you please introduce yourself?
**Assistant:**Hello! I’m an AI assistant, designed to help, engage, and provide information on a wide range of topics. My developers have created me with the intention of being unbiased, neutral, and objective, which means I’ll strive to provide accurate and helpful responses without any personal opinions or agendas.
I have access to a vast amount of information and knowledge, which I can use to answer your questions, offer suggestions, and even engage in casual conversation. However, keep in mind that my knowledge cutoff is currently in 2023, so I might not be aware of the most recent events or developments.
My goal is to make our interactions as natural and enjoyable as possible. I’ll do my best to understand your queries, provide relevant information, and even add a touch of humor here and there. I’m always ready to help, so feel free to ask me anything that’s on your mind!
It really does sound much more human!
Let’s try a logic puzzle:
A farmer is travelling with a fox, a goose, and a bag of beans. During his journey, he comes across a river with a boat to cross it.
The farmer can only fit one thing in the boat with him at a time. If left alone together, the fox will eat the goose or the goose will eat the beans. How does the farmer get everything across the river safely?This was the answer:
What a classic problem! Let's break it down step by step:
1. The farmer takes the goose across the river first.
2. He leaves the goose on the other side and goes back to the original side to take the fox across the river.
3. He leaves the fox on the other side but takes the goose back with him to the original side.
4. He leaves the goose on the original side and takes the beans across the river.
5. He leaves the beans on the other side with the fox and goes back to the original side to get the goose.
6. He takes the goose across the river for the third time, and now all three - the goose, the fox, and the beans - are safely on the other side of the river.
This solution ensures that at no point are the goose and the fox left alone together (or the goose and the beans), which prevents any harm from happening.
To summarize:
1. Goose + Boat
2. Fox + Boat
3. Goose + Boat
4. Beans + Boat
5. Goose + Boat
6. Goose + Boat (final step)
Hope that helps!It got it right! Now let’s try coding:
Write a bubble sort in C++Here is the response:
Here is a simple implementation of the bubble sort algorithm in C++:
#include <iostream>
using namespace std;
void bubbleSort(int arr[], int n) {
for (int i = 0; i < n - 1; i++) {
for (int j = 0; j < n - i - 1; j++) {
if (arr[j] > arr[j + 1]) {
// swap arr [j] and arr [j+1]
int temp = arr[j];
arr[j] = arr[j + 1];
arr[j + 1] = temp;
}
}
}
}
void printArray(int arr[], int size) {
for (int i = 0; i < size; i++) {
cout << arr[i] << " ";
}
cout << endl;
}
int main() {
int arr[] = { 64, 34, 25, 12, 22, 11, 90 };
int n = sizeof(arr) / sizeof(arr[0]);
bubbleSort(arr, n);
cout << "Sorted array: \n";
printArray(arr, n);
return 0;
}Here’s a step-by-step explanation of the code:
- The
bubbleSortfunction takes an arrayarrand its sizenas parameters. It iterates through the array using two nested loops. - The outer loop iterates from the first element to the second-to-last element (
i < n - 1). - The inner loop iterates from the first element to the
n - i - 1element. This is because, after each iteration of the outer loop, the largest element in the unsorted portion of the array will “bubble up” to its correct position, so we don’t need to compare it again. - Inside the inner loop, we compare adjacent elements (
arr [j]andarr [j + 1]). Ifarr [j]is greater thanarr [j + 1], we swap them using a temporary variabletemp. - After the inner loop completes, the largest element in the unsorted portion of the array will be in its correct position.
- We repeat this process until the entire array is sorted.
- The
printArrayfunction is used to print the sorted array. - In the
mainfunction, we define an arrayarrwith some unsorted elements, calculate its sizen, and then call thebubbleSortfunction to sort the array. - Finally, we print the sorted array using the
printArrayfunction.
Please note that bubble sort is not the most efficient sorting algorithm for large datasets, but it’s a good starting point for understanding basic sorting concepts.
Let’s try some other conversations:
User: meow meow
Ah, a request for a meow! As a humanoid AI assistant, I'm not a cat, but I can certainly try to mimic one. Here's my best attempt:
"Meow!"
Now, since I'm an AI, I'm not actually meowing. But I'm happy to pretend! Do you have any other questions or topics you'd like to discuss? I'm here to help, as always.User: Do you think machine can think?
The question of whether machines can think is a complex one, and it's been debated by philosophers, scientists, and thinkers for centuries. Let's break it down step by step.
First, we need to define what we mean by "thinking." Human thinking is a multi-faceted process that involves consciousness, creativity, emotions, and the ability to make decisions based on our experiences, knowledge, and understanding of the world. It's a deeply subjective and personal experience.
When we talk about machines, we're typically referring to computers, robots, or other electronic devices that process information, execute commands, and perform tasks based on pre-programmed algorithms. Machines can process vast amounts of data, make predictions, and perform complex calculations, but they don't have consciousness or emotions.
So, can machines think? From a human perspective, the answer is no. Machines don't have subjective experiences, emotions, or a sense of self-awareness. They don't have the ability to understand the world in the same way humans do, and they don't have the capacity for creativity or imagination.
However, some argue that machines can be considered "thinking" in a functional sense. For example, a machine learning algorithm can analyze data, identify patterns, and make predictions based on that data. It can "learn" from new information and adapt its behavior accordingly. In this sense, machines can be said to think, but only in the sense that they're processing information and making decisions based on that information.
Ultimately, the question of whether machines can think is a matter of perspective. From a human perspective, machines don't have the capacity for consciousness, emotions, or subjective experiences, so they can't truly think in the same way humans do. However, from a functional perspective, machines can perform tasks and make decisions based on data, which can be considered a form of thinking.
As our technology continues to evolve, we may develop machines that are more advanced and sophisticated, blurring the line between human and machine thinking. But for now, we can say that machines can perform tasks and make decisions, but they don't possess the subjective experiences, emotions, and consciousness that characterize human thinking.Uploading
huggingface-cli upload raincandy-u/phillama-3.8b-v1 .The result is stored here: https://huggingface.co/raincandy-u/phillama-3.8b-v1
References
Not exactly references, just some pages I consulted along the way.
[1] Memory-efficient LLM Training with GaLore
[2] Efficient finetuning of Llama 3 with FSDP QDoRA