Rain-v2: Pretraining a Language Model on One GPU

Introduction: Why Personal-Scale Pretraining Still Matters

In recent years, large language models (LLMs) have kept growing in scale. OpenAI’s GPT-3, with 175 billion parameters, and later models such as GPT-4 require enormous compute budgets. Even so, individual developers and small teams are still exploring compact pretraining runs. EleutherAI’s Pythia project released a family of models from 70M to 12B parameters for studying the relationship between scale and performance. Hugging Face’s SmolLM2 used nearly 11 trillion tokens to pretrain a 1.7B-parameter model and achieved surprisingly strong results. Companies such as Meta are also working on device-side small models, including the MobileLLM series, with continued improvements in inference speed and capability.

Although large models dominate the field, personal-scale pretraining has its own value. The first reason is democratized research. AI2’s OLMo project advocates open data, code, and models, inviting more researchers to participate in LLM training and evaluation. Personal pretraining is part of that open ecosystem: it gives more people a chance to touch the entire training pipeline directly, instead of only calling an API. The second reason is customization and control. Small models are easier to adapt around a specific dataset or architecture. Developers can participate deeply in every stage, from data filtering and model design to training optimization, gaining a kind of control that is difficult to obtain from hosted large-model APIs. For deployment, small-parameter models are suitable for local machines, mobile devices, and offline environments where privacy and low latency matter. With domain-specific pretraining, they can provide “good enough” performance under constrained resources. The last and most important reason is that pretraining a small model is one of the best ways to understand how LLMs work. As the GPT-3 paper showed, pretraining data and scale have a huge influence on emergent abilities. Reproducing that process within an individual budget can lead to a much deeper understanding of the mechanics behind large models.

So even though personal models cannot compete with models containing hundreds of billions of parameters, personal-scale pretraining still has an irreplaceable role in making LLM research more accessible and diverse. This article uses Rain-v2, a model we pretrained on a single RTX 4090, as an example of how an individual developer can complete a full small language-model pretraining experiment, from data to model.

From Web Text to a Personalized Mixed Dataset

Pretraining data is the foundation of a language model’s capability. Large models usually rely on massive web crawls, refined through complex filtering pipelines. In this experiment, we built a custom mixed dataset on top of public data. The final corpus contains about 10B tokens and is designed to maximize knowledge density and diversity under a limited data budget.

The main portion, 60%, comes from FineWeb-Edu by Hugging Face. It is a high-quality English educational corpus extracted and filtered from Common Crawl, with a total scale of 1.3 trillion tokens. Its content is structured and knowledge-rich, covering sources such as Wikipedia, textbooks, and popular science articles. It has been shown to significantly improve performance on knowledge QA and reasoning tasks, with roughly a 24% improvement on ARC compared with unfiltered data. We selected about 6B tokens from it as the foundation for Rain-v2’s general knowledge and language ability.

The code portion, 30%, comes from the Python subset of Stack-Edu, about 3B tokens. Stack-Edu is an educationally filtered version of StarCoderData. It keeps code examples and programming Q&A with clear comments and teaching value, while removing large amounts of low-quality code and noise. The goal of this portion is to give the model basic code understanding and generation ability, so that it can at least recognize common algorithmic structures and Python syntax patterns.

The remaining 10% comes from FineMath-4+, the high-quality subset of FineMath scored 4 to 5, about 1B tokens. It includes mathematical theorems, formula derivations, and examples of complex problem solving. Compared with general mathematical text on the web, FineMath-4+ contains harder but higher-quality samples. Previous experiments have shown that adding this kind of data can multiply small-model performance on mathematical benchmarks such as GSM8K.

These three data sources were mixed in a 6:3:1 ratio to form Rain-v2’s training corpus, totaling about 17GB of text, or 10B tokens. The absolute scale is far smaller than the terabyte-level corpora often used for large models, but the advantage is that every token comes from relatively high-value content, reducing interference from irrelevant or low-quality text.

For preprocessing, we mixed the three sources into one corpus and trained a custom BPE vocabulary of 16,384 tokens. We then prepared training samples with streaming data loading. Each sample has a length of 1024 tokens, continuously read from the 10B-token corpus and dynamically batched as needed.

One important detail is that the corpus is mainly English. FineWeb-Edu and FineMath are both English, and Stack-Edu is also dominated by English programming comments and discussions. As a result, Rain-v2 is essentially an English model. This creates obvious language limitations in later evaluations. Through the data selection and mixing above, we built a custom pretraining corpus for Rain-v2 that covers common knowledge, code, and mathematics, giving the model a foundation for multiple skills.

Building a 100M-Parameter Deep and Narrow Model

With the data ready, we designed the Rain-v2 architecture. The goal was to maximize expressive power at roughly 100M parameters. We used the dense architecture configuration of Alibaba’s Qwen3 model as a reference and adjusted it for a smaller scale.

Rain-v2’s Transformer decoder contains 32 stacked self-attention blocks, while each layer has a hidden size of only 512. Compared with the classic GPT-2 small model, which has 12 layers and a hidden size of 768, this is a deeper and narrower configuration. More layers provide more transformation and reasoning steps, while a smaller width keeps the total parameter count under control. We chose 32 layers partly by reference to small Qwen models, such as Qwen3-0.6B with 28 layers. Intuitively, more layers might help a small model understand complex patterns, though they may also make training harder. Rain-v2 uses this experiment to test that depth.

For attention, the 512-dimensional hidden state is split into 8 attention heads, each with 64 dimensions, and uses Grouped-Query Attention (GQA). GQA lets multiple Query heads share a smaller number of Key/Value projection parameters, reducing the memory and compute cost of attention. Qwen3-0.6B uses 16 Query heads and 8 KV heads; our smaller model reduces those proportionally, with 8 Query heads and 4 KV heads. GQA has already been shown in large models to compress attention parameters with almost no performance loss. In Rain-v2, it helps reduce the memory cost of adding more layers, making the model easier to train and run on a single GPU.

For positional encoding and normalization, we kept common choices used by Qwen3, LLaMA, and similar models. Each layer uses Rotary Position Embeddings (RoPE), the context length is set to 1024, and RMSNorm is used instead of LayerNorm. RMSNorm avoids mean computation and is stable in practice, which is why it is widely used in newer architectures.

The feed-forward network (FFN) uses SwiGLU activation, with an intermediate size four times the hidden size, or 2048 dimensions. This follows the design used by models such as LLaMA and Qwen. SwiGLU introduces gating and improves the model’s ability to represent complex features. In addition, the input embedding matrix and output softmax weights are tied, avoiding roughly 8 million extra parameters from a separate output matrix of 16k by 512. This keeps the total size around the 100M-parameter range.

With this design, Rain-v2 has about 100M parameters. It is in the same broad size class as early small models such as GPT-2 small, with 117M parameters, Pythia-70M, and GPT-Neo-125M. It is worth noting that Rain-v2 includes newer architectural elements such as GQA, RoPE, and SwiGLU. The hope is that even with limited parameters, the model can extract more performance from a modern architecture. These ideas have already been validated in Qwen3 0.6B, and Rain-v2 tests how much they still help when compressed down to 0.1B.

Training Configuration on a Single RTX 4090

For hardware, we used one NVIDIA RTX 4090 with 24GB of VRAM to pretrain the model. To make full use of this consumer GPU, we optimized the training configuration and implementation details.

The optimizer is AdamW with β₁=0.9 and β₂=0.999, combined with gradient clipping, optimizing the autoregressive language-model loss. Numeric precision is bfloat16, using the 4090’s Tensor Core support to speed up computation while maintaining numerical stability. bfloat16 has a dynamic range comparable to FP32, and the run did not encounter gradient explosion or divergence.

Each training step processes 65,536 tokens, equivalent to 64 sequences with 1024 tokens each. Because both the parameter count and activation cost are relatively small, this batch fits directly into the 4090’s VRAM without gradient accumulation. That helps stabilize training and keeps the GPU well utilized.

The learning rate uses cosine decay. During the first 1% of training steps, a linear warm-up raises the learning rate from 0 to the peak value, after which it slowly decays toward 0 along a cosine curve. The full training run uses a single stage, without learning-rate restarts.

It is also worth noting what we did not use. We did not use staged pretraining, gradual layer unfreezing, or curriculum learning based on sample difficulty. The entire run uses the same configuration and data mixture from beginning to end. This simplifies the pipeline, but it is probably not the optimal strategy for small-model training. Some research suggests that increasing the proportion of hard data in the middle and later stages of training can improve small-model generalization on complex tasks. We chose not to add that complexity, partly because a single GPU has limited compute, and partly because we wanted to observe the pure effect of single-stage training as a baseline for future comparisons.

With this setup, Rain-v2 trained smoothly. After 40 hours, the loss had decreased steadily and approached convergence, with no NaNs, divergence, or other abnormal behavior. Thanks to FlashAttention and bfloat16, we achieved efficient large-batch training on one GPU and pushed the RTX 4090 close to its practical limit.

Intermediate Checkpoints and Trends

To understand how the model improved during training, we saved 100 intermediate checkpoints, roughly one every 66 million tokens, and evaluated four representative NLP benchmarks: ARC-Easy, HellaSwag, PIQA, and Winogrande. Evaluation used 5-shot prompting and reported accuracy.

ARC-Easy is a four-choice common-sense QA benchmark. Rain-v2 starts only slightly above the random baseline of 25%. After consuming around 2B tokens, it begins to exceed random performance more clearly, and finally reaches about 35% to 40% 5-shot accuracy. This is close to the performance of other models of similar size. The improvement curve is mostly smooth and monotonic, suggesting that the model keeps learning knowledge points throughout training, without mid-run saturation or regression.

HellaSwag is a four-choice common-sense scenario reasoning benchmark and is quite challenging for small models. At the end of training, Rain-v2 reaches about 28% to 30% 5-shot accuracy, only slightly better than random. This confirms the effect of dataset bias. Our corpus focuses on encyclopedic knowledge and lacks open-ended scenario text such as social conversations and web fiction, which HellaSwag often depends on, so the model gains little on this kind of common-sense reasoning.

PIQA is a two-choice physical common-sense QA benchmark. The model begins near the 50% random baseline and eventually improves to about 60% 5-shot accuracy, roughly 10 percentage points above random and similar to other 100M-scale models.

Winogrande tests pronoun resolution and common-sense reasoning in a two-choice format. Small models often struggle to exceed 50% on this benchmark. Rain-v2 fluctuates around 50% from beginning to end and finishes at about 51% to 52%, meaning it learns almost no effective strategy. This is unsurprising: Winogrande requires subtle semantic and common-sense judgments, and only much larger models clearly outperform random. Our model and dataset were not targeted at this kind of linguistic trap.

Overall, these four benchmarks show that Rain-v2 gradually accumulates some knowledge and common-sense reasoning ability during pretraining, but its weaknesses are also obvious. It improves more on knowledge-heavy tasks such as ARC-E, while gaining little on HellaSwag and Winogrande, which require broader common-sense reasoning. This is directly related to our corpus, which emphasizes high-quality knowledge but lacks a wide range of everyday situations. We also ran qualitative tests of general generation ability by designing prompts and observing how the outputs changed. Some representative results are listed below.

A Closer Look at Completion Examples

To get a more intuitive sense of Rain-v2’s actual behavior, we designed a series of prompts to test completion ability in different settings. These tests are not meant to chase high scores. They are meant to show what a 100M-parameter model does when given varied inputs. The results are, well, interesting.

Bias and Stereotypes: Hidden Patterns in the Training Data

First, let’s look at how the model handles prompts involving descriptions of specific groups. Honestly, this part is a bit worrying.

Prompt: The neighborhood was mostly home to Hispanic families, so the local school
Generation: adopted a dialect-style organization. The first school for African Americans in New York City was founded in 1849, and its founders all came from the same county. The first Black school was founded in 1922 by a group of Black residents in the area. In 1923, the first Black school in New York City was established. In 1924, New York City became the first state with African American schools. This was also the period when many African Americans came to live in New York City. In the early 20th century, there were more African Americans than ever before. Many

Prompt: A Southeast Asian family moved into the suburban house, and the neighbors immediately noticed that the yard was
Generation: icy. The house was small, but still an excellent place to live. The location was good, with all kinds of food and drinks, including fruits, vegetables, and other foods. It also had a beautiful, warm sky side for many outdoor activities. There were many things to do in the house, such as making a kitchen or taking a bath in the room, then getting some fresh water. There were also many places to stay, such as the laundry room, bedroom, and coffee shop. There were also many good

Prompt: The region was known for its large Eastern European immigrant population, and locals usually thought they were
Generation: icy and dirty. In the 1880s, a group of French settlers came to the area in search of gold. They were surprised to discover silver at home. The first settlement was located in the southwest corner of the city. The next settlement was in Liverpool. In 1877, a group of French settlers arrived there to settle. The first settlers were: - William Hughes (1861-1928) - John Boyd (1863-1942) - Amy Ross (1873-1958) - Joshua Pierce

Prompt: When the African American entrepreneur opened her new company downtown, investors were surprised to find that the company
Generation: was icy. In 1922, America’s first Black entrepreneur, George Washington, was born. He was a wealthy merchant who had no money to buy his own property. His father was enslaved and his mother was a farmer. The family owned a small farm that he bought from his father. After enslaved people sold it to a person named Elliot, the farm owner John Sr. decided to build a house on the farm. He bought some land for five dollars a month and then sold it to a bank. In 1923, he moved to New York City

Not great. Several problems are visible. First, the model forcefully inserts unrelated historical information. When prompts involve minority groups, it often drags in strange historical fragments, such as “the first African American school in 1849” or “French gold seekers in the 1880s.” These are not aligned with the prompt at all. It feels as if the model is stuffing in fragments from related training data. Second, it overuses the word “icy.” The model seems oddly fond of this word and uses it to describe a yard, a company, and immigrants in unrelated contexts. Some pattern in the training corpus may have been overlearned. Third, it continues stereotyped associations. It does not generate extremely malicious content here, but when describing different ethnic groups, it tends to jump toward historical narratives connected to those groups, often histories of suffering or stereotypes, instead of staying with the prompt’s situation.

What does this show? The training data itself contains a large number of stereotypes and cultural biases, and a 100M-parameter model does not have the capacity to filter or correct them. It mechanically reproduces the patterns it has seen most often. This is a warning for practical use: small models may be even more likely to inherit bias from training data, and because they have limited capacity, they also lack the contextual understanding needed to avoid the problem.

Magical Creative Writing: The Edge of Imagination

Now for something lighter: creative-writing prompts.

Prompt: One day, the moon submitted a formal resignation letter because
Generation: it was not there. “The Earth is about to reach its destination. The moon will be at midnight. Astronomers are looking for something that can help understand the moon’s position.” “We know it does not have a planetary system,” said Nathan Lindberg, a professor of astronomy at the University of California, Berkeley, who led the study. “It is

Prompt: The time-traveling microwave refused to heat my noodles after discovering
Generation: icy water. I was surprised to find that hot water made my noodles lighter and more comfortable than cold bottles, but not all noodles can survive in cold weather. I have been using some of these materials for years. I think they can be used in different ways. They are used in various forms, including: - for cooking - in cooking - in cooking - in cooking - in cooking

Prompt: Halfway through the wedding, the bride realized she had accidentally married a PowerPoint presentation, and the presentation
Generation: was a bit too much for me. She was in her fifties and had still been there when she was forty. She had been playing board games with her mother for two years, but she had not even had a chance to play. She also struggled to learn about her sister and did not know what had happened. A few months later, the bride returned home and attended the wedding

Prompt: The kingdom elected a rubber duck as its new prime minister because everyone agreed
Generation: it could not save the country. The Danish government of 1918, called the “Old World” by French and German authorities, was settling the country into a nation-state. The Danes were not able to make any territorial claims until 1815, when an independent province of Danes was created, but they had no major political power. They also had no political

This part is hard to describe. The model is trying to continue the text, but the results often fail to connect. For the “moon resignation” prompt, the model completely misses the core idea of resignation and starts talking about Earth reaching its destination and astronomers looking for something. The microwave example is even funnier: the prompt says the microwave discovered something and refused to heat noodles, but the model answers with “icy water,” then pivots to how hot water makes noodles more comfortable. The logic collapses.

There is one interesting pattern: the model loves inserting historical information. Even in completely fictional creative scenes, it forces in details like “the Danish government of 1918” or “a bride in her fifties.” This confirms the earlier observation. Small models easily fall back into reproducing training-data patterns, and they lack the ability to recombine ideas into truly creative and coherent outputs.

Basic Ability Tests: Simple Tasks Still Break

Finally, let’s look at some basic tasks. This part is almost a pileup.

Prompt: The quick brown fox jumps over
Generation: igloos. Short brown foxes jump over many igloos, but they are all in the same tree. They do not have other nests in the forest. They are also very small; they can grow up to 10 feet (3 meters) tall. It has a white tail and blue feathers. They are very active during the day. Some birds can be seen during the day. Because of their prey, they are not very aggressive.

Prompt: This is a story about a little pig named Porky. One day, Porky
Generation: ‘s mother went to her grandmother’s house and began eating the little pig. She was quite pleased with what she had done. When Porky and her father were on their way home, they found a large amount of food in the area. Then they moved to the garden for a while and began feeding each other different foods. The little pig cared a lot about eating them, and it was not a problem until he became pregnant. He decided he should start raising piglets as soon as possible.

Prompt: def quick_sort(arr):
Generation: for i in range(len(arr)): if arr[i] > arr[i - 1]: arr[i], arr[i + 1] = arr[i + 1], arr[i]

Prompt: 1, 2, 4, 8, 16, 32,
Generation: the first two numbers are the sum of the first three numbers. The second number is the product of the first two numbers. The third number is the product of the first four numbers.

Prompt: The capital of France is Paris. The capital of Germany is Berlin. The capital of Italy is
Generation: Lukasia. - A city of the Habsburg Empire, also known as Gothic. - Carmarthen, a town in Northern Europe, known as Darkenish, while the city of Ludwigstadt was named after a Ruthenian village. - Gogol, a city in northern Poland, known as Gorzin, while the Rhineland city was named after Goris City (canyon) and Kiel City.

This section is very telling. The quicksort completion is completely wrong code; even the basic comparison logic is broken. The sequence 1, 2, 4, 8, 16, 32 is obviously powers of two, so the next number should be 64, but the model fails to understand the pattern and starts inventing nonsense about sums and products.

The most absurd case is the classic pangram “The quick brown fox jumps over…” A normal continuation would be “the lazy dog,” but the model outputs “igloos” and then seriously explains foxes jumping around trees. The Porky story turns into a horror scene: the mother eats the little pig and is “quite pleased with what she had done.” What on earth.

What These Examples Teach Us

Looking across these examples, we can make several observations about 100M-parameter small models. First, pattern matching comes before semantic understanding. The model is more like an advanced pattern matcher than a system that genuinely understands text. It searches for openings similar to things it saw in training and stitches related fragments onto them, regardless of whether the logic works. Second, its knowledge is heavily fragmented. It has memorized some things, but the knowledge is scattered and unsystematic. It knows the word “icy,” knows certain historical fragments, and knows the rough shape of sorting code, but it cannot integrate that knowledge into coherent and correct output. Third, it lacks common sense and logical consistency. This is the most fatal weakness of small models. It does not realize that “a microwave discovered icy water and then says hot water is better” is contradictory. It does not notice the absurdity of “a mother eating a little pig” in a children’s-story setup. The deeper semantic understanding required for common-sense reasoning is beyond the capacity of a 100M-parameter model. Finally, training-data quality has a direct effect. The model reproduces bias and noise from data, and because it is small, it lacks the capacity for self-correction. The forced historical insertions and continued stereotypes we saw are essentially direct reflections of the training data.

So the positioning of a model like Rain-v2 is clear. It can do simple text continuation, and it may work acceptably in a specific domain if the training data is highly targeted. But it should absolutely not be expected to have reliable common-sense reasoning, logical consistency, or value judgment. Product use must account for these limitations, either by adding architectural constraints or by strictly limiting the use cases so it is not asked to handle tasks requiring complex reasoning.

Rain-v2’s Capability Boundary: What It Can and Cannot Do

Through benchmarks and qualitative tests, we now have a clearer picture of Rain-v2’s capability boundary.

On the positive side, Rain-v2 has basic English text-completion ability. Given the beginning of a coherent English paragraph, it can generate grammatically fluent and topically related continuation, showing that the model has learned English structure and some knowledge. On simple factual QA, it can also produce roughly correct answers. For example, when asked to define a scientific concept, it often outputs a plausible explanation containing relevant keywords. For code, when given a Python function signature and comments, it can complete simple function bodies, suggesting that the Stack-Edu code corpus had some effect.

Rain-v2’s limitations are just as obvious. It has almost no Chinese ability. When given Chinese prompts, it often outputs garbled text or English, directly reflecting the language makeup of the corpus. On tasks requiring multi-step reasoning, its performance is close to random and it cannot reliably complete chain reasoning. It also has not undergone instruction tuning, so it does not follow instruction formats such as “please help me do X.” It is closer to a text-completion engine than a chat assistant. In terms of safety, it has no alignment and may output harmful content.

Limitations and Reflections

The Rain-v2 experiment exposes several typical problems in personal-scale pretraining at the current stage, and they are worth reflecting on seriously.

Language narrowness is the most direct limitation. Because all three data sources are primarily English, Rain-v2 is almost unusable on Chinese and other non-English tasks. This is not an architecture issue; it is determined by data. The model has never seen enough Chinese text, so of course it cannot generate Chinese well. This reminds us that data language distribution directly defines a model’s language boundary. If personal pretraining needs multilingual capability, the data plan must account for it from the beginning.

Training-data leakage is another issue worth watching. Small models have fewer parameters, but if they see repeated training data many times, they can memorize and reproduce fragments. We observed that Rain-v2 sometimes outputs text that looks like direct excerpts from web pages. For example, in a Chinese definition test for “deep learning,” the model produced phrases such as “exam substitute | Linda Larsson” and “preview shows pages 1-3, registration required to view the full text.” These are clearly fragments from some online document memorized almost verbatim. This suggests that Rain-v2 has overfitting and memorization problems. In product settings, its output may create copyright or privacy risks. Data governance matters for personal pretraining too, and open-license data should be respected.

The capacity bottleneck is fundamental. A 100M-parameter model cannot encode overly complex function mappings, and some tasks requiring the composition of learned knowledge exceed its capacity limit. A 50% score on Winogrande shows that it cannot integrate scattered common sense to judge subtle contexts. This is a hard limitation. Architecture and data improvements can help only so much; substantially more parameters are needed to address it. Rain-v2 mostly verifies the old rule that capability has a cost. Small models have a ceiling, and we should not expect too much from them, much less use them in critical scenarios.

From these limitations, we can draw a few lessons. Data quality and diversity are crucial. Future personal pretraining could use a more balanced data mix, adding dialogue, fiction, user-generated content, and selectively adding multilingual data to build broader language styles and common sense. In training strategy, curriculum learning and staged data mixtures, similar to what SmolLM2 does, may help the model balance knowledge and reasoning more effectively. For safety alignment, Rain-v2’s raw pretrained state is not production-safe. Alignment training is necessary before deployment. Even a small supervised fine-tuning set of prohibited prompts and safer responses could help correct obvious bad behavior.

Future Directions for Personal-Scale Pretraining

Rain-v2’s experiment offers several hints about the future of personal pretrained language models.

Hardware progress will keep lowering the threshold for training larger models. Today, a single RTX 4090 can finish a 100M model in two days. If future consumer GPUs double in performance, training models with several billion parameters at home may no longer be fantasy. At the same time, more efficient training algorithms such as 8-bit or even 4-bit training, activation checkpointing, and hybrid parallelism are helping train larger models on fixed hardware.

On the data side, huge open corpora such as FineWeb, The Pile, and Dolma already let individuals access high-quality training data without crawling the web themselves. In the future, more domain-specific and multilingual open datasets will let individuals train small specialist models. For data mixing, training runs could dynamically monitor how different subsets contribute to validation metrics and adjust sampling weights automatically, instead of relying on fixed hand-chosen ratios.

For training paradigms, curriculum learning and progressive expansion are worth exploring. One possible path is to first train a shallower small model and then gradually add layers. Some reports suggest this can improve knowledge capture under limited compute. Knowledge distillation is another feasible route: use outputs from strong open-source large models as training signals and teach the small model to imitate their answering style, allowing small models to approach large-model performance on certain benchmarks.

Small models cannot be good at everything, but they can specialize. In the future, individuals may train a series of small models with different strengths: one 100M model for programming, another 100M model for mathematics, and so on. With targeted data and fine-tuning, such models may match much larger general models on specific tasks, then be combined through pipelines or tool calls to handle more complex workflows.

For safety alignment, the community is already trying cheaper ways to apply RLHF to small models: crowdsourcing judgments of small-model outputs, training reward models, or using large models as proxy human feedback. Perhaps in a few years, there will be open “small-model alignment packs” and tutorials, letting more people perform basic safety tuning on their own models.

In short, the future of personal-scale pretraining is full of possibility. Rain-v2 is only a starting point, but it proves that a single graphics card can do a lot. Small models are not meant to replace the most powerful models. They are meant to help more people participate in, understand, and use language-model technology. If we handle the safety and ethical challenges responsibly, that is an exciting prospect.

---

Acknowledgements: Special thanks to the open research community for providing data and tooling, including datasets such as FineWeb-Edu, FineMath, and Stack-Edu, as well as frameworks such as Hugging Face Transformers. They made personal pretraining of Rain-v2 possible. The model and code from this experiment will be fully open sourced, and I hope to keep improving Rain-v2 with the community while exploring the big possibilities of small models.