Fine-tuned Qwen improved the previous best held-out MRR by 30.0% relative.
Closer Look
A machine learning study in meaning.
Slangle turns a word game into a retrieval problem: every guess is embedded or scored, compared against slang concept profiles, and converted into heat feedback that helps players move through meaning instead of spelling.
The correct concept appeared in the top ten for nearly 42% of frozen test queries.
INT8 ONNX keeps live semantic inference inside the serverless deployment envelope.
Every benchmark query ranks its correct slang concept against the complete candidate bank.
Problem framingRetrieval over meanings
Why this is not a standard classifier
The game does not need one label prediction from a fixed input. It needs a smooth semantic ranking for arbitrary player guesses. A useful scorer should know that “flirting ability” is close to “rizz,” that “lying” is close to “cap,” and that a miss can still be meaningfully warmer than an unrelated word.
Player guessfree-text word or phrase
Concept profiletarget, aliases, definitions, contexts
Similarity scoreembedding score blended with lexical fallback
Game feedbackcold, cool, warm, hot, burning, correct
Data sourceMLBtrio + Linguabase
Dataset design
The research pipeline starts from MLBtrio/genz-slang-dataset. The dataset was transformed into concept-level records: canonical slang terms, aliases, definitions, examples, and contextual phrases. Linguabase enrichment was tested as additional definition and association data.
- Normalized slang targets and aliases so equivalent forms map to the same concept.
- Extracted descriptions, context fields, and example sentences into anchor-positive pairs.
- Masked examples where the answer appeared directly, preventing trivial memorization.
- Filtered sensitive or insufficient rows before building training and evaluation splits.
- Kept raw downloaded data out of the repository while tracking derived fingerprints.
Leakage controlSource-row grouping
Corrected split policy
The first phrase-level split looked usable, but it leaked related fields from the same upstream row across train and test. That would inflate metrics. The corrected v2 split groups every (target, source_row) before splitting.
| Split family | Train | Validation | Test | Overlap |
|---|---|---|---|---|
| closed-set v2 | 4,779 | 146 | 222 | 0 |
| zero-shot-target v2 | 3,327 | 794 | 1,026 | 0 |
TrainingSentence embeddings
How the models were trained
The strongest checkpoints use sentence-transformers/all-MiniLM-L6-v2 as the base model. Training examples are anchor-positive pairs: a meaning phrase is the anchor, and the matching slang concept is the positive. The main training objective was MultipleNegativesRankingLoss, which uses other items in the batch as contrastive negatives.
Base modelall-MiniLM-L6-v2
Clean v2 training rows4,779
Balanced Linguabase rows6,312
Epochs3
Batch size32
Selection metricvalidation MRR@10
The Linguabase MiniLM checkpoint was selected for the live scorer because it balanced corrected retrieval quality with a compact deployment envelope. Its quantized ONNX export is served in production. Fine-tuned Qwen is retained as the stronger offline teacher and quality reference rather than placed in the live request path.
EvaluationCorrected benchmark
What improved
| Model | MRR | Recall@1 | Recall@5 | Recall@10 | Latency |
|---|---|---|---|---|---|
| Hybrid baseline | 0.1484 | 9.01% | 19.82% | 23.42% | 259.59 ms |
| Linguabase MiniLM | 0.1883 | 13.06% | 24.32% | 29.73% | 1.32 ms |
| v2 profile MiniLM | 0.1796 | 11.26% | 23.42% | 31.98% | 1.44 ms |
| Linguabase ensemble + profile feature | 0.1944 | 13.51% | 25.23% | 28.83% | 251.68 ms |
Impact: the embedding approach moved the scorer from lexical approximation toward semantic retrieval. Linguabase MiniLM raised test MRR by 0.0399 absolute, and v2 profile MiniLM raised Recall@10 by 8.56 percentage points. The calibrated ensemble reached 0.1944 test MRR, the strongest result in the compact MiniLM study.
Frontier experimentOpen-weight local teacher
Fine-tuning a larger semantic model
Model size alone did not solve the task. Frozen Qwen3-Embedding-0.6B and BGE-M3 both trailed the domain-trained MiniLM on validation MRR. Qwen improved only after the retrieval task was formulated asymmetrically: player-like meaning phrases received a retrieval instruction, while slang targets and concept profiles remained unprompted.
Teacher modelQwen3-Embedding-0.6B
Training objectiveCached MNRL
Validation seeds42, 43, 44
Mean validation MRR0.3234 ± 0.0061
Frozen test MRR0.2526
Frozen test Recall@1041.89%
Training ran locally with four-example encoder micro-batches and a 32-example cached contrastive pool. Explicit seeded row shuffling ensured that the three runs used genuinely different training orders. The best validation checkpoint was selected before the test split was opened once.
Impact: fine-tuned Qwen raised frozen-test MRR by 0.0582 over the previous best result and improved Recall@10 by 9.91 percentage points. It established a stronger quality ceiling, but its approximately 1.1 GB artifact is not suitable for the current serverless runtime.
Knowledge distillationQwen teacher → MiniLM student
Transferring quality into a compact model
The first distillation study tested whether Qwen’s domain knowledge could improve a MiniLM-sized student without placing the larger model in the live request path. Teacher supervision used training text only. Validation selected configurations, and the frozen test remained closed because the final student did not cross the promotion gate.
| Validation configuration | Seeds | MRR | Recall@10 | Latency |
|---|---|---|---|---|
| Supervised MiniLM baseline | 3 | 0.2078 ± 0.0062 | 30.14% ± 0.68 pp | ~1.99 ms |
| Fine-tuned Qwen teacher | 3 | 0.3234 ± 0.0061 | 49.32% ± 0.68 pp | ~17.07 ms |
| Direct coordinate transfer | 1 | 0.1150 | 17.12% | 0.88 ms |
| Relation-only KL transfer | 1 | 0.1666 | 25.34% | 0.75 ms |
| Joint KL + supervised, 1:1 | 1 | 0.2139 | 32.19% | 0.79 ms |
| Joint KL + supervised, 1:2 | 3 | 0.2151 ± 0.0055 | 32.19% ± 1.19 pp | 0.75 ± 0.04 ms |
Directly forcing MiniLM to copy Qwen’s first 384 embedding coordinates damaged the student’s existing geometry. Relation-only KL divergence also caused task forgetting. Alternating teacher-relation batches with two original supervised batches was the strongest approach.
Decision: joint distillation improved mean Recall@10 by 2.05 percentage points, showing partial transfer of Qwen’s broader semantic neighborhoods. The MRR gain was only 0.0072 and remained inside observed seed variation, so the student was not tested, exported, or promoted. Production stays on the verified compact checkpoint.
CalibrationGameplay layer
Why keep the hybrid scorer?
Embedding models are stronger on corrected retrieval metrics, but short guesses, spelling-adjacent guesses, aliases, and exact-answer detection still benefit from lexical features. Slangle blends semantic and lexical signals so the game remains stable when a model checkpoint is unavailable or when a guess is unusually short.
0.78 * embedding_score + 0.22 * hybrid_scoreThe visible scorer uses a 78/22 embedding-to-hybrid blend. An 85/15 gameplay candidate remains available for shadow comparison. A profile-size feature improved offline candidate ranking but inflated live heat scores, so it was explicitly rejected for visible gameplay.
Negative resultsResearch judgment
What did not work
- Adding ungraded Linguabase associations improved some validation numbers but did not beat the prior leaders on test.
- Graded CoSENT with sparse hard negatives underperformed the hybrid baseline.
- Increasing MNRL batch size from 32 to 64 lowered validation MRR.
- Scaling frozen Qwen from 0.6B to 4B parameters reduced retrieval quality, confirming that parameter count is not a substitute for task alignment.
- Direct embedding-coordinate distillation erased useful MiniLM geometry faster than it transferred teacher quality.
The conclusion is practical: better supervision, reviewed confusions, and retrieval-objective alignment are more valuable than simply adding parameters, generic rows, or more epochs.