Finding My Favorite VOCALOID Songs with Few-Shot Preference Learning

Before we begin

I just wanted to do something with my VOCALOID collection.

About half a year ago, maybe? I tried building my own music library away from streaming platforms. The plan was to save every song locally and manage them through some magical method. It… failed. I can’t remember exactly how. Probably laziness (cross that out), or perhaps I realized the maintenance cost was too high. Anyway, the thousand-plus VOCALOID tracks I’d batch-downloaded from NetEase Cloud Music sat on my hard drive gathering dust.

Then a few days ago I was idly browsing folders, spotted that VOCALOID/ directory, and suddenly felt interested again.

The idea

Well, if I’m rebuilding a music library, it needs a central feature. I actually had a specific goal: use an algorithm to find the songs I really love among those thousand-plus tracks.

I started by asking ChatGPT, “What’s the BERT of music?” It told me about MERT. Hey, even their names match: BERT is Bidirectional Encoder Representations from Transformers; MERT is Musical Encoder Representations from Transformers. Quite a nice pair.

The technical plan was simple:

Use MERT to encode each track into a song-level embedding. Sample eight evenly spaced 12-second clips, pass each through the model to get its hidden states, mean-pool over time, average the eight clip vectors, then L2-normalize. That gives one vector representing the whole song.

Like/dislike labels felt a little inefficient, so I wanted pairwise comparison: randomly choose songs A and B, listen, then tell the model “I prefer A.”

Once the preference model is trained, score all thousand-plus songs and listen to my favorites.

I chose m-a-p/MERT-v1-330M: 330 million parameters, available directly through Hugging Face’s from_pretrained. Everything runs locally, without an API. With over a thousand songs, API calls would cost too much.

Let’s do it.

Preprocessing

The first pitfall

At first I used the script GPT gave me. Roughly:

• Scan the directory.
• For each song, extract eight clips, run inference on them one at a time, then save the embedding.

I ran it on my 5070 laptop… oh no, why is it so slow?

It felt like serial processing was to blame: eight clips meant eight separate passes…

Time to optimize.

After discussing it with Kimi, I settled on a few changes:

Optimization	How it works	Expected benefit
Batch the clips	Send all eight clips of a song to the model in one batch	Fully utilize the GPU
Automatic mixed precision (AMP)	Use FP16 through torch.cuda.amp.autocast	20–30% faster; less VRAM
Multiprocess data loading	DataLoader with num_workers	CPU preprocessing doesn’t block the GPU
Model compilation	torch.compile (PyTorch 2.0+)	Another 10–20% faster

I ran the revised code, and hey! It really was much faster! The progress bar went from “stare at it blankly” to “whoosh, whoosh, whoosh.”

The second pitfall

Just as I was happily watching that progress bar race ahead, the PowerShell window suddenly vanished. The mouse froze too, then recovered after a while.

…Huh?

Task Manager showed that RAM had filled up, causing the freeze. Strange, though: the 5070 has 8 GB of VRAM, and the computer has 32 GB of RAM. I’m investigating a possible memory leak, or whether DataLoader worker processes aren’t being released.

Sleepy, so I’ll stop here and continue once I’ve found the cause.

The third pitfall, and the fix

A short nap.

Back to investigating after waking up. I asked GPT to examine the code and found the problem: DataLoader’s num_workers setting!

I’d set num_workers=4, but the worker processes weren’t being released correctly after processing data. Memory kept accumulating until all 32 GB were exhausted. The PowerShell window disappeared because the system killed the process using too much memory.

The solution was simple: stop using DataLoader altogether! The audio-file list was already in memory. A Python tqdm iterator with manual batching would do. The revised script is extract_mert_embeddings_final.py.

A few other small optimizations:

• Read audio in segments: use soundfile seeking to read only the necessary clips instead of loading whole files.
• Clear VRAM promptly: call torch.cuda.empty_cache() after processing a set number of items.
• More conservative batches: set batch_size to 8 to avoid giving the GPU too much at once.

Another run… everything finished in twenty minutes! Got it right!

This produced two key outputs:

• manifest.jsonl: each song’s metadata and embedding path.
• embeddings/*.npy: 1,199 vector files, each with 1,024 dimensions.

Preference training

From MLP to XGBoost

With the embeddings ready, the next step was to train a preference model.

GPT’s first plan used an MLP (multilayer perceptron), an ordinary neural network taking an embedding and returning a score. But it didn’t work very well: neural networks overfit too easily with so few examples.

Then GPT mentioned XGBoost.

XGBoost is a decision-tree ensemble algorithm, especially good at tabular data. With small datasets it is much steadier than a neural network, trains quickly, and can report feature importance so you can see which embedding dimensions matter most to preference predictions.

The central idea is pairwise ranking:

1. Pick songs A and B.
2. I listen and say, “I prefer A.”
3. The model treats (A, B) as a training example, learning which combinations of features mean “better” among similar songs.
4. Optimize using XGBoost’s rank:pairwise objective.

GPT wrote preference_pairwise_webui.py, a simple Gradio web interface:

python preference_pairwise_webui.py --server_port 7860 --inbrowser

The interface has:

• A search box at the top to find seed songs by title.
• Two songs in the middle, ready to play and compare.
• “Prefer A,” “Prefer B,” and “Skip” buttons below.
• The current top 30 recommendations on the right, updating live.

Bored after 30 labels…

The plan was to label more than 100 pairs so the model could really learn my preferences. In reality, I got fed up after about 30. (?)

Pairwise labeling is exhausting! Constantly choosing between two songs, listening until I started questioning my life. I felt like a data-labeling factory girl.

So I gave up trying: downloaded XGBoost’s ranking.csv and copied the top 100 into a favorites/ folder.

The result… sounded strangely good!

I played a few at random. The top-ranked songs really were in styles I liked: MIMI, DECO*27… even a few old songs whose names I’d forgotten, but whose intros immediately made me go “Ohhh, this one!”

Apparently 30 labeled pairs were enough, with good seed songs—I picked about 15 I definitely liked—and XGBoost’s stability on small datasets. The results were already pretty good.

What I’ve done, and what’s next

The whole process so far

Step	Tool / script	Time	Output
Extract embeddings	extract_mert_embeddings_final.py	~20 minutes	1,199 vectors
Train preferences	preference_pairwise_webui.py	~30 minutes of labeling	ranking.csv
Make a playlist	Have an OpenCode Agent copy the top 100 songs	2 minutes	favorites/ folder

In under an hour, I selected the 100 songs I was “most likely to love” from 1,199 tracks.

Possible next steps…

I’m already very happy with the results, but plenty could be improved.

The current pairwise strategy randomly picks songs with nearby rankings. It could use uncertainty sampling instead: choose the pairs the model is most unsure about, meaning those with the smallest score difference, so each label provides the greatest information gain.

MERT encodes only audio. It could be combined with lyrics encoded by BERT and metadata such as artist, album, and year for multimodal fusion. Sometimes I love a song not just for its melody, but because a line of lyrics hits a particular feeling.

But… let me finish listening to these 100 songs first!

One last thing

This project brought me back to that dusty VOCALOID collection. The algorithm’s playlist really is excellent. It even recovered a few treasures I used to play on repeat but had forgotten the names of.

The biggest technical surprise was that XGBoost is so much easier to use than neural networks when you have very little data. I used to think deep learning could do everything. This experiment showed that sometimes a simple tree ensemble is more dependable.

Also, pairwise labeling was far more tiring than I expected. Unless you have lots of spare time, similarity to seed songs plus 10–20 preference comparisons may be enough; the results are already quite good.

Anyway, the playlist is ready. Headphones on, repeat mode engaged!

---

Code repository: Coming sooooooooooooon