In the rapidly evolving world of Large Language Models (LLMs), techniques like In-Context Learning (ICL) and Retrieval-Augmented Generation (RAG) have significantly changed how these models adapt to new tasks and domains. As we leverage retrieval methods to enhance LLM performance, it’s crucial to reassess whether our current strategies are truly adapted to these advanced learning techniques.
In this blog post, we explore the current state of retrieval in RAG and ICL, uncover the limitations of the traditional evaluation of retrieval methods, and introduce our novel benchmark, the In-Context Learning Embedding and Reranker Benchmark (ICLERB).
More details and findings can be found in our paper.
In-Context Learning (ICL) enables LLMs to perform new tasks by providing examples or relevant information directly within the prompt. Instead of updating the model’s weights through fine-tuning, the model is conditioned with context that guides its response.
One form of ICL is few-shot learning. Suppose we want the model to translate English sentences into French. We can craft a prompt that includes examples:
[Prompt]
Translate the following English sentences to French:
“It’s raining cats and dogs.” → “Il pleut des cordes.”
“Bite the bullet.” → “Prendre son courage à deux mains.”
“Under the weather.” → “Ne pas être dans son assiette.”
“Once in a blue moon.” →
[Model output]
“Tous les trente-six du mois.”
By providing these examples through prompt engineering, we enable the model to infer the pattern and generate the correct translation. Another powerful ICL technique is Chain-of-Thought prompting, which encourages the model to “think” through the problem step by step.
For example:
[Prompt]
Question: Emma thinks of a number. She doubles it, adds 10, and the result is 30.
What number did Emma think of?
Answer: Let the original number be “x”. After doubling and adding 10: 2x + 10 = 30.
Subtract 10 and divide by 2: x = 20 / 2 = 10. Emma thought of the number “10”.
Question: A number is multiplied by 3, then decreased by 4, giving 11.
What was the original number?
[Model output]
Answer: Let the original number be "x". After multiplying by 3 and subtracting 4: 3x - 4 = 11.
Add 4 and divide by 3: x = 15 / 3 = 5. The original number was "5".
By prompting the model to break down the problem, we achieve more accurate and interpretable results.
In the previous examples of ICL, we provided pre-selected static examples within the prompt. While effective, this approach relies on the examples we manually choose, which may not be as effective for specific questions, especially if they differ from the patterns or content of these examples.
Retrieval-Augmented Generation (RAG) builds upon ICL by integrating dynamically selected external information directly into the model's prompt.
Imagine the LLM is being asked: “What’s the schedule for the 2024 Summer Olympics?”. As an LLM with knowledge only up to a specific cutoff date, it wouldn’t provide an accurate response. RAG systems allow dynamic retrieval of up-to-date or domain-specific information from external sources to augment the model’s knowledge.
RAG operates through:
While some RAG systems still use traditional retrieval methods like BM25 due to their simplicity and efficiency, these term-based methods often underperform because they rely on exact term matching. This fails to capture the complex semantics of natural language, making it challenging to retrieve relevant documents that don’t share the exact wording of the query.
More advanced retrieval methods have thus become prevalent, notably embedding models and rerankers:
To overcome this challenge, retrieval systems often adopt a two-stage approach: an initial retrieval using embedding models quickly narrows down the pool of candidate documents, and then a reranker refines the list by reordering the top candidates. This two-stage approach combines the scalability of embedding models with the nuanced understanding of rerankers.
When implementing retrieval methods in RAG systems, practitioners have various options for embedding models and rerankers, including API-based services and locally-hosted models.
Many organizations provide these models as API services, simplifying integration. For example:
These APIs eliminate the need to manage underlying infrastructure but come with considerations such as recurring costs per API call (refer to pricing of OpenAI, Cohere, Voyage AI), potential latency due to network communication, and data privacy concerns.
Alternatively, platforms like Hugging Face offer a vast range of embedding models and rerankers that can be downloaded and run locally. At the time of writing, there are 10.8k models labeled for the task of “Feature Extraction” in NLP and 7.2k for “Sentence Similarity,” both of which are relevant to this context. Examples include:
Several rerankers are also available, such as mixedbread-ai/mxbai-rerank-base-v1 and jinaai/jina-reranker-v2-base-multilingual. There are actually 415 models that have the keyword “rerank” in their name.
Running models locally gives practitioners full control over their data and can reduce long-term costs by eliminating per-call fees. However, it requires investment in hardware and expertise in deploying models.
With so many embedding models and rerankers available, each varying in aspects such as fine-tuning datasets, architectural complexities, and model sizes, understanding how these retrieval methods influence RAG performance is essential.
When it comes to choosing a retrieval method for RAG, practitioners often turn to benchmarks like the Massive Text Embedding Benchmark (MTEB). MTEB evaluates models based on their ability to retrieve or rank documents in tasks resembling traditional search scenarios. Despite being first introduced in 2022 before the widespread adoption of LLMs and the emergence of RAG, it remains a common reference point in industry discussions on optimizing retrieval for RAG, as seen in several industry blogs.
However, MTEB primarily assesses how well models retrieve semantically similar documents or rank them according to relevance in search tasks. The datasets used for evaluation include DBPedia Entity, MS MARCO, and Quora, which are meant to evaluate search and semantic similarity (more on these datasets in BEIR here).
This traditional approach frames retrieval as a search problem. While effective for search engines, it was not designed with the concept of RAG in mind. Moreover, many retrieval models are fine-tuned on these search-oriented datasets, like ColBERT on MS MARCO. This fine-tuning reinforces the emphasis on semantic similarity, which may not align with the objectives of RAG.
To better serve RAG and ICL, we propose reframing the retrieval task in RAG as a recommendation problem. In recommender systems, the goal is to select items that optimize downstream metrics, directly enhancing overall performance.
Applying this to RAG, the retrieval process should prioritize documents that will most significantly boost the LLM’s performance to the task at hand.
To illustrate this point, consider the following example:
User query: “What are the best practices for securing REST APIs?”
Document A: "The best practice to secure a REST API is to use authentication."
Document B: "For robust security of REST APIs, implement advanced measures like
utilizing OAuth 2.0 with scopes for granular access control, applying rate limiting to mitigate
DDoS attacks, and adopting mutual TLS for client and server authentication to prevent
man-in-the-middle attacks."
While Document A is closer semantically to the query, it provides a well-known principle that might already be within the LLM's knowledge. Document B, though less similar to the query in meaning, offers advanced strategies that can enhance the LLM’s response. A good retrieval approach should retrieve Document B instead of Document A.
To address existing limitations, we introduce ICLERB — the In-Context Learning Embedding and Reranker Benchmark. ICLERB evaluates retrieval methods based on how effectively the retrieved documents enhance the LLM’s performance in ICL tasks.
Key features of ICLERB include:
Calculating DPO values requires LLMs that expose probability outputs for the provided prompts, as we need to access the probabilities of the answers that we include in the prompt. Models without accessible probability distributions of the input, such as most proprietary LLMs, cannot be evaluated using ICLERB.
In the initial release of ICLERB, focusing on few-shot learning scenarios, we observe several noteworthy findings regarding the performance of prominent embedding models and rerankers for RAG and ICL.
Contrary to expectations, rerankers from Cohere and Voyage AI underperform compared to their embedding model counterparts. This trend is also seen with NVIDIA's retriever and embedding models, despite both being competitive overall.
This suggests that rerankers, often fine-tuned for semantic similarity or search relevance, may not be as effective in selecting documents that enhance LLM performance in ICL tasks. Embedding models seem to capture a wider range of contextual information beneficial for improving LLM outputs.
This finding has significant implications for practitioners employing the two-stage retrieval strategy of initially retrieving documents using embeddings and then refining the ordering with rerankers. Adopting this strategy is not only unnecessary, but could actually degrade performance in ICL tasks when using these specific rerankers (though this might not be the case with other rerankers like cm-rerank-mxbai-rlaif-v0.1).
An interesting observation from ICLERB is that larger model size doesn't necessarily correlate with better performance in ICL tasks. While many top-ranking models are indeed large, our model, cm-rerank-mxbai-rlaif-v0.1 (more details here), with only 335M parameters, outperforms several larger models. This indicates that models specifically optimized for utility in ICL can achieve superior results without the need for massive parameter counts.
Similarly, within the Snowflake model family, the smallest model evaluated, snowflake-arctic-embed-s (33M parameters), surpasses its larger counterparts in ICL performance. This underscores the importance of targeted optimization over sheer model size.
.png)
Comparing ICLERB results with those from MTEB reveals important differences in embedding model rankings. Some models performing well on traditional semantic similarity benchmarks don’t necessarily excel in ICL tasks.
While models like BAAI’s bge-en-icl and NVIDIA’s NV-Embed-v2 maintain strong positions across both benchmarks, other models exhibit notable shifts:
These discrepancies highlight that traditional benchmarks like MTEB may not fully capture models' effectiveness in RAG and ICL contexts, emphasizing the need for specialized benchmarks like ICLERB.
ICLERB results also reveal that some models exhibit significant performance variability across different datasets. While NVIDIA's NV-Embed-v2 and our cm-rerank-mxbai-rlaif-v0.1 maintain consistently high performance, BAAI's bge-en-icl shows fluctuations depending on the dataset used. This suggests certain models may be sensitive to specific data characteristics, impacting their generalizability across diverse ICL tasks.
ICLERB represents a step forward in evaluating retrieval methods for RAG and ICL by focusing on the utility of retrieved documents in enhancing LLM performance. By reframing retrieval as a recommendation problem, we align the evaluation process with the core objectives of these advanced systems.
Future developments for ICLERB include expanding the benchmark to cover a broader range of LLMs and datasets, including more varied document types. This will provide deeper insights into retrieval methods across different RAG scenarios and contribute to a more robust and comprehensive assessment of retrieval strategies.
As we refine ICLERB, our goal is to facilitate the development of retrieval methods tailored for utility in ICL, ultimately advancing the capabilities of LLMs in providing accurate and contextually rich responses.
