Custom GISTEmbed Approach: Pre-computed Embeddings and Hard Negative Mining for Legal Retrieval

By NewMind AI Team

•  This work builds on the encoder-focused findings of our paper, Mecellem Models, where we demonstrate that strong legal retrieval performance is not driven by pre-training scale alone, but is substantially influenced by how encoder representations are post-trained, optimized, and evaluated for retrieval-specific objectives.

•  Training effective retrieval models for legal document search requires sophisticated negative sampling strategies that go beyond simple in-batch negatives. The GIST (Generative Hard Negative Sampling using In-batch Negatives and Semi-hard Negatives from a Teacher) approach provides a framework for leveraging teacher model embeddings to mine challenging negatives. However, traditional implementations rely on BGE-M3 as the guide model—selected for its reasonable performance under low computational requirements—which must be loaded at each training step, leading to memory overhead and training-time bottlenecks.

• To address this limitation, we implement a custom GISTEmbed approach that pre-computes teacher embeddings offline using Qwen3-Embedding-8B, selected for its superior legal retrieval performance (Legal Average 53.38 vs 51.16 for BGE-M3). This enables efficient hard negative mining through a three-way strategy—Query-to-Query, Positive-to-Positive, and Query-to-Corpus—while preserving GIST-compliant filtering constraints. By moving teacher inference entirely offline, we eliminate the need for teacher model loading during training, allowing us to exploit stronger teacher representations without training-time computational constraints, while generating more diverse and more challenging negatives that improve model discrimination.

• Experiments on the MSMARCO-TR dataset demonstrate that pre-mined hard negatives consistently outperform regular MSMARCO triplet training. The best-performing configuration achieves 47.87% Legal Average on the Mizan legal retrieval benchmark, representing a 0.77% absolute improvement over baseline models trained with standard MSMARCO triplets.

Custom GISTEmbed Pipeline Overview

Our implementation consists of three distinct stages that work together to create an efficient training pipeline:

Stage 1: Pre-computing Teacher Embeddings - Offline computation of Qwen3-8B embeddings for all queries, positives, and negatives in the dataset, enabling parallel processing and eliminating teacher model overhead during training.

Stage 2: Hard Negative Mining - A 3-way mining strategy that combines Query-to-Query (Q2Q), Positive-to-Positive (P2P), and Query-to-Corpus (Q2C) approaches with GIST filtering to create diverse, challenging negatives.

Stage 3: Training with Explicit Negatives - Training using MultipleNegativesRankingLoss with explicitly included hard negatives, in-batch negatives, and random negatives, providing fine-grained control over negative sampling.

Stage 1: Pre-computing Teacher Embeddings

Motivation

Traditional GIST approaches require loading the teacher model (~16GB) during training, creating memory bottlenecks. By pre-computing embeddings offline, we eliminate this overhead while enabling full contrastive learning.

Teacher Model Selection

We selected Qwen3-Embedding-8B for pre-computation due to its superior performance on legal retrieval benchmarks. While regular GIST approaches use BGE-M3 as the guide model during training (due to its decent performance and low computation requirements), Qwen3-Embedding-8B's superior performance makes it ideal for offline pre-computation, despite being too large for training-time use:

Qwen3-Embedding-8B achieves state-of-the-art performance on MTEB multilingual leaderboard (70.58 score as of June 2025) [2] and provides superior legal retrieval performance (53.38 vs 51.16 Legal Average), making it ideal for offline embedding pre-computation where computational constraints are less critical.

Implementation

We pre-compute embeddings for all 920,006 MSMARCO-TR triplets using Qwen3-Embedding-8B on 4× H100 GPUs at Barcelona Supercomputing Center [7]. Processing uses parallel GPU processing (230K samples per GPU), batch size 64 per GPU, and produces three NumPy arrays (~15 GB each) storing query, positive, and negative embeddings. Runtime: ~2-3 hours.

Benefits

No teacher model needed during training - Eliminates 16GB memory overhead  

Full contrastive learning - Enables in-batch negatives without teacher inference  

Maximum distillation performance - Pre-computed embeddings ensure consistent teacher signals  

Scalability - 4× speedup through parallel processing

Stage 2: Hard Negative Mining with 3-Way Strategy

Mining Strategy

Inspired by GIST [1], we employ a 3-way mining strategy using pre-computed Qwen3-Embedding-8B embeddings:

1. Q2Q (Query → Query): Mine from similar queries' positives (~3 negatives)

• Find queries with similar intent but different correct answers

• GIST Filter: Skip if sim(query, candidate) >= sim(query, positive)

• Forces model to distinguish between semantically similar queries with different answers

2. P2P (Positive → Positive): Mine from similar positives (~2 negatives)

• Find documents semantically related but answer different queries

• GIST Filter: Skip if sim(query, candidate) >= sim(query, positive)

• Challenges model to understand query-document relationships

3. Q2C (Query → Corpus): Traditional corpus mining (~3 negatives)

• Find documents similar to query but not the positive

• GIST Filter: Skip if sim(query, candidate) >= sim(query, positive)

• Standard hard negative mining approach

Total: 8 diverse hard negatives per query, all filtered using GIST rule to remove false negatives.

Implementation Details

Input: Pre-computed Qwen3-8B embeddings (45 GB)

Processing:

• Parallel GPU processing: 4× H100 GPUs (230K queries each) at Barcelona Supercomputing Center

• Chunked processing: Load 100K samples at a time (memory-efficient)

• Batch saving: Save every 10K queries (progress persistence)

• Checkpoint/Resume: Each GPU saves independently, can recover from failures

Memory Management:

• Memory per GPU: ~32GB peak (corpus + 1 search batch) < 80GB VRAM

• Q2Q/P2P: Batched search (100K embeddings per batch = ~1.6GB)

• Runtime: ~3-4 hours (all GPUs in parallel)

Stage 3: Training with Explicit Negative Sampling

Training Configuration

We train models using MultipleNegativesRankingLoss with explicitly included negatives in each training example:

Negative Structure per Example:

• Hard negative from mining: 1 (pre-mined using Qwen3-8B)

• In-batch negatives: 7-15 (from other samples in the same batch)

• Random negatives: 4-12 (from other rows in dataset)

• Total: 18-28 texts per example (1 query + 1 positive + 16-26 negatives)

Loss Function: MultipleNegativesRankingLoss with temperature=0.05

Training Hyperparameters:

• Base models: Batch size 12, Gradient accumulation 8 (Effective batch: 384)

• Large models: Batch size 4, Gradient accumulation 16 (Effective batch: 256)

• Learning rate: 2e-5 (default) or 4e-5 (ablation)

• Max sequence length: 1024

• Epochs: 1-10 (depending on configuration)

• Optimizer: AdamW with weight decay 0.01, max grad norm 1.0

Ablation Studies

We conducted systematic ablation studies varying in-batch negatives, random negatives, and learning rates. See the Hyperparameter Lookup Table section for detailed configuration mappings.

Experimental Setup

Hardware Configuration

All experiments were performed on the Barcelona Supercomputing Center (BSC) HPC infrastructure with:

• GPU: 4× NVIDIA H100 GPUs (80GB VRAM each) per node

• Interconnects: High-bandwidth interconnects for distributed training

• Environment: Singularity containers for reproducible training environments

Base Models

1. Mursit-Base-TR-Retrieval

   - Base model: Mursit-Base (Pretrained MLM Model)

2. Mursit-Large-TR-Retrieval: ModernBERT v2 Large (last checkpoint)

   - Base model: Mursit-Large

Dataset

MSMARCO-TR Mined Hard Negatives:

• Source Dataset: [parsak/msmarco-tr](https://huggingface.co/datasets/parsak/msmarco-tr) - Original MSMARCO Turkish dataset

• Triplets Dataset: [newmindai/ms-marco-turkish-triplets](https://huggingface.co/datasets/newmindai/ms-marco-turkish-triplets) - Constructed query-positive-negative triplets (920,006 samples)

• Final Dataset: [newmindai/msmarco-tr-mined-hard-negatives](https://huggingface.co/datasets/newmindai/msmarco-tr-mined-hard-negatives) - Mined dataset with 8 hard negatives per query

• Mining Strategy: 3-way strategy (Q2Q, P2P, Q2C) with Qwen3-Embedding-8B embeddings

• Format: HuggingFace datasets with explicit negative lists

Evaluation Framework

Mizan Legal Retrieval Benchmark:

• Contract retrieval (NDCG@10)

• Caselaw retrieval (NDCG@10)

• Regulation retrieval (NDCG@10)

• Legal Average (mean of three datasets)

MTEB Benchmark:

• TurHistQuad (NDCG@10)

• WebFAQ (NDCG@10)

• MKQA (NDCG@10)

• XQuAD (NDCG@10)

• MTEB Average (mean of four datasets)

All models were evaluated automatically during and after training, with results logged to centralized leaderboards.

Results

Baseline Comparison: Mined vs Regular MSMARCO

Our mined MSMARCO approach demonstrates consistent improvements over regular MSMARCO training:

Key Findings:

• Contract retrieval: +0.94% improvement

• Regulation retrieval: +1.34% improvement

• Overall Legal Average: +0.51% improvement

• Pre-mined hard negatives provide consistent benefits for legal retrieval

Base Model Ablation Results (Mined Experiments)

Best Configuration: Exp5 (ib8-r8-lr4e5) achieves 47.87% Legal Average (+0.26% over Exp1 baseline)

Observations:

• Higher learning rate (4e-5) improves Regulation retrieval (+1.17%)

• In-batch negative variations show limited impact (47.45-47.69% range)

• Pre-mined hard negatives are the primary performance driver

Large Model Results (Mined Experiments)

Higher learning rate (4e-5) dramatically improves large model performance (+7.78% Legal Average in Exp3 vs Exp1).

MTEB Evaluation Results

Observations:

• Consistent MTEB performance across configurations (~50%)

• XQuAD performance is strongest (88%)

• MKQA performance is weakest (~4%), indicating room for improvement on multilingual QA

Performance Analysis

Pre-computation eliminates the 16GB teacher model overhead during training, enabling larger batch sizes and efficient GPU utilization. Processing times: pre-computation ~2-3 hours, mining ~3-4 hours (both on 4× H100 GPUs), with checkpoint/resume capability. Training runs ~8 hours per epoch for base models and ~12-24 hours for large models.

Conclusion

Our custom GISTEmbed approach demonstrates that pre-computing teacher embeddings and employing a 3-way hard negative mining strategy significantly improves retrieval model performance while maintaining training efficiency.

Key Achievements:

• Performance: +0.51% Legal Average over regular MSMARCO training; best configuration (Exp5: ib8-r8-lr4e5) achieves 47.87% Legal Average

• Efficiency: Pre-computed embeddings eliminate 16GB teacher model overhead during training, enabling larger batch sizes and 4× speedup through parallel processing

• Methodology: 3-way mining (Q2Q, P2P, Q2C) creates more diverse negatives than traditional approaches; learning rate tuning (4e-5) proves more impactful than negative ratio variations, especially for large models (+7.78% improvement)

The results show that pre-mined hard negatives provide consistent benefits for legal retrieval tasks, with explicit negative control enabling fine-grained optimization.

Hyperparameter Lookup Table

References

3. Muennighoff, N., et al. (2024). GIST: Generative hard negative sampling using In-batch Negatives and Semi-hard Negatives from Teacher. arXiv preprint arXiv:2402.16829. https://arxiv.org/pdf/2402.16829

4. Zhang, Y., et al. (2025). Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models. arXiv preprint arXiv:2506.05176. https://arxiv.org/pdf/2506.05176

5. Qwen Team. (2024). Qwen3-Embedding-8B: A Large-Scale Multilingual Embedding Model. Hugging Face Model Card. https://huggingface.co/Qwen/Qwen3-Embedding-8B

6. Uğur, Ö., et al. (2026). Mecellem Models: Turkish Models Trained from Scratch and Continually Pre-trained for the Legal Domain. arXiv preprint arXiv:2601.16018. https://arxiv.org/abs/2601.16018

7. Parsak. (2024). msmarco-tr: MSMARCO Turkish Dataset. Hugging Face Datasets. https://huggingface.co/datasets/parsak/msmarco-tr

8. NewMind AI. (2025). ms-marco-turkish-triplets: MSMARCO Turkish Triplets Dataset. Hugging Face Datasets. https://huggingface.co/datasets/newmindai/ms-marco-turkish-triplets

9. NewMind AI. (2025). msmarco-tr-mined-hard-negatives: MSMARCO Turkish Mined Hard Negatives Dataset. Hugging Face Datasets. https://huggingface.co/datasets/newmindai/msmarco-tr-mined-hard-negatives

10. Barcelona Supercomputing Center. (2025). BSC HPC Infrastructure. https://www.bsc.es/

11. Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. Proceedings of EMNLP-IJCNLP.

12. Mizan Legal Retrieval Benchmark. (2025) https://huggingface.co/spaces/newmindai/Mizan

13. Muennighoff, N., et al. (2023). MTEB: Massive Text Embedding Benchmark. arXiv preprint arXiv:2210.07316.

14. PyTorch FSDP2 Documentation. (2024). Fully Sharded Data Parallel (FSDP2). https://pytorch.org/tutorials/intermediate/FSDP_tutorial.html