Using the Claude API to build a dataset and a Claude-like model

Introduction

I finally got the free AWS 32V account sorted out! Claude 3 Sonnet is available in ap-south-1 (Mumbai).

To request access to Claude:

1. Open this page.
2. Click to request access and enter a real company whose location matches the IP address.
3. Approval usually takes anywhere from a few seconds to half a minute.

Here is what I have gathered so far:

1. Opus is unavailable, perhaps because its compute requirements are too high.
2. A 400 Method Not Allowed error means the account is restricted in that region. Switch regions and request access again.

Dataset

Next, I tried extracting prompts from real conversation datasets such as WildChat and LMSYS to build a dataset.

The strategy:

1. Filter out all responses that have been censored or had personal information redacted.
2. Randomly sample 5,000 examples from what remains.
3. Remove unused fields such as hashes.

The source datasets use ShareGPT format. I just replace all assistant/GPT turns with outputs from Claude 3 Sonnet.

After generating and gathering data, I ended up with:

Model	Dataset size
Claude 3 Sonnet	10k
Claude 3 Opus	10k

These are all multi-turn conversations with real user inputs, so the resulting model might have strong long-context abilities!

Training

I chose the base Llama-3-8B model and stayed with Llama Factory as the framework.
One pitfall: selecting another prompt template for Llama-3-8B caused a device-side assertion error.

The solutions:

1. Enable vocabulary expansion, although this cannot be used with QLoRA.
2. Or use the Llama3 prompt template. The drawback is that uninitialized special tokens may affect the results…