Is it better to distill or finetune language models?​

Arcee did a bunch of experiments comparing model distillation to finetuning, base vs instruct model distillation and more.

Main Takeaways:​

♦️ Both logit-based and hidden states-based distillation methods consistently outperform standard SFT across various benchmarks.

♦️ General-Purpose Performance Gains: Significant improvements across datasets like OpenHermes, WebInstruct-Sub, and FineTome, particularly in MMLU and MMLU-Pro benchmarks, indicating enhanced knowledge absorption.

♦️ Domain-Specific Performance Gains: Distilling models for domain-specific tasks, especially when using the same training data as the teacher model leads to performance improvements.

Experiments​

♦️Experiment 1: What's better Supervised-Finetune(SFT) or Distill+SFT?

Three models—Hermes-Distilled (logit-based), Hermes-Hidden-States, and Hermes Vanilla (SFT-only)—were evaluated, all distilled from Arcee Spark using a subset of the Teknium's OpenHermes-2.5 dataset (200k examples). Both distillation methods were better than SFT-only model across major benchmarks such as BBH, MUSR, and MMLU-PRO. The logit-based approach was better then the hidden-states-based distillation.

♦️Experiment 2: Effectiveness of Logit-based Distillation in a Generic Domain

The 1.5B Distilled model, trained on a 200k subset of WebInstruct-Sub, demonstrated performance improvements over the baseline Qwen2-1.5B-Instruct model across all metrics. Its performance was also comparable to the teacher model, Arcee Spark, particularly on MUSR and GPQA benchmarks.

♦️Experiment 3: Distillation on Instruct vs. Base Student Models

The 1.5B-Instruct-Distilled model (logit-based), trained on WebInstruct-Sub, showed performance improvements over the vanilla Qwen2-1.5B-Instruct model on the MMLU benchmark, showing benefits of distillation for enhancing knowledge retrieval.

♦️Experiment 4: Effectiveness of Domain-specific Distillation

Distilling Arcee Agent, a 7B parameter model specialized in function calling, into Qwen2-1.5B-Instruct using the same dataset that trained the teacher model resulted in performance gains. This approach underscores the potential of using the same training data for both teacher and student models to achieve even greater improvements, particularly in domain-specific tasks.

More resources: DistillKit by Arcee AI

🔗 arXiv Link

📜 Download paper

Ready to start building?​

Check out the Quickstart tutorial, or sign up for a free Weaviate Cloud account.

Distillation has become popular recently due to its ability to efficiently compress the knowledge of larger LLMs into smaller ones. Here’s how it works, why it’s useful, and examples of how you can perform distillation

What is distillation?​

Distillation is a model compression technique where a smaller "student" model is trained to mimic the behavior of a larger "teacher" model. This is achieved by transferring knowledge from the teacher to the student, usually through methods like logit-based or hidden states-based distillation. These methods are designed to help the student model replicate the teacher's output distribution or internal representations, often leading to a more efficient model with comparable performance.

When would we use this?​

Distillation is commonly used when deploying large models is impractical due to resource constraints, such as in real-time applications or edge devices. For instance, a smaller student model can be distilled from a powerful teacher model like Llama3.1 405B, retaining much of the original model’s capability but with significantly lower computational demands. Distillation is also useful when adapting models to specific tasks or domains, as seen in domain-specific distillation cases like "function calling," where specialized knowledge from a teacher model is transferred to a smaller model for specific use cases.

What’s the benefit?​

Distillation offers a significant reduction in model size and computational requirements while maintaining a high level of performance. This is especially valuable in scenarios where memory and processing power are limited. Moreover, distillation allows for flexibility in model architecture choices; for example, distilling knowledge from a Llama-3.1-70B model into a much smaller StableLM-2-1.6B model. Distillation methods like those provided in Arcee-AI's DistillKit, including logit-based and hidden states-based distillation, can lead to substantial performance gains over traditional training routines without requiring additional data.

Examples of Distillation Techniques:​

(1) Logit-based Distillation:​

This method involves transferring knowledge by using both the hard targets (actual labels) and soft targets (teacher logits) to guide the student model. The student is trained to minimize the difference between its output distribution and the teacher’s output, typically using Kullback-Leibler (KL) divergence. This method is particularly effective for maintaining performance close to the teacher model while improving the student’s generalization abilities.

(2) Hidden States-based Distillation:​

Here, the focus is on aligning the intermediate layer representations of the student with those of the teacher. This layer-wise guidance helps the student model capture similar features and improves its performance and generalization. This method also allows for cross-architecture distillation, enabling knowledge transfer between different model architectures, such as distilling from a Llama-3.1-70B model into a StableLM-2-1.6B model.

More resources: DistillKit by Arcee AI

🔗 arXiv Link

📜 Download paper

Ready to start building?​

Check out the Quickstart tutorial, or sign up for a free Weaviate Cloud account.

The Problem LoRA Solves:​

• In early 2021, Microsoft partnered with OpenAI to explore the commercial viability of GPT-3.
• They found that prompting was insufficient for production tasks like natural language to code generation.
• Fine-tuning was necessary but prohibitively expensive due to the large size of model checkpoints.

How It Works:​

• LoRA generalizes full fine-tuning(updating every single parameter) by asking two questions:
  1. Do we need to fine-tune all parameters?
  2. For the weight matrices we fine-tune, how expressive should the updates be in terms of matrix rank?
• These questions define a 2D plane where full fine-tuning is the top-right corner(full rank and full parameter updates) and the origin represents the original model.
• Any point in this plane is a valid LoRA configuration.

The chosen rank of the update matrix controls the expressivity of the finetuning process.

• A d x d matrix can represent any linear transformation in a d-dimensional vector space.
• By first transforming the input to a lower-dimensional space and then back to the original space, we can restrict the kind of linear transformations that can be represented.
• This reduces the number of parameters that need to be stored from (dxd) to (dxr + dxr) where r << d.
• A point near the origin often performs as well as full fine-tuning. - because often Neural Networks are over-parametrized and thus the weight matrices are full of linearly dependent
• This suggests that we can start with a low-rank configuration and gradually increase the rank if needed.

Common practices when using LoRA:​

• How to choose the rank R of the update matrix: Start with a low rank and increase it if needed.
• When to use full fine-tuning?: When finetuning on data that is completely new and absent from the pretraining of the base model (for example if you are tuning an English model on Martian then full fine-tuning may be necessary).
• Can I use LoRA for any model architecture?: As long as the model uses matrix multiplication, LoRA can be applied. So basically pretty much every model architecture can use LoRA!

Benefits of LoRA:​

• Reduced checkpoint sizes: On GPT-3, checkpoint size was reduced from 1TB to 25MB.
• No additional inference latency: LoRA updates can be merged with the original parameters during inference. W_new = W_old + AxB
• Ability to quickly switch between tasks: LoRA modules can be loaded and unloaded efficiently.(A_frenchxB_french),(A_germanxB_german),(A_spanishxB_spanish)

Some interesting engineering ideas enabled by LoRA:​

• Caching LoRA modules in RAM for faster model switching and routing between different finetunes.
• Training multiple LoRA modules in parallel on different batches of the training set.
• Creating a tree of adaptive models where each node is a LoRA module.

🔗 arXiv Link

📜 Download paper

Ready to start building?​

Check out the Quickstart tutorial, or sign up for a free Weaviate Cloud account.

Training LLMs to write solutions such that smaller models can better check them. This makes them easier to check for humans, too.​

Key Findings​

1. Performance vs. Legibility Trade-Off:

   • A model with only supervised fine-tuning on human-written derivations achieved high legibility but poor accuracy.
   • Optimizing solely for correctness led to high accuracy but poor legibility.
   • They propose method balanced both, retaining high legibility with moderate accuracy.

2. They Train Three Models:

   • Robust Verifier: Accurately distinguishes correct from incorrect solutions, even when misleading.
   • Helpful Prover: Generates human-legible solutions, reducing human error.
   • Sneaky Prover: Produces subtly incorrect solutions, initially confusing human evaluators and highlighting improvement areas.

How It Works​

To train the prover models to produce human-legible text, they use a scheme where the prover's solutions must be easily verifiable by a weaker model.

They use models from the GPT-4 family with significant differences in pretraining compute, they train using a GAN like(generator and discriminator) alternating training process as follows:

1. Generate multiple solutions to math problems from the prover, some of which are naturally incorrect.
2. Alternate between two training phases:
   • Verifier Training: Train the weak verifier to predict the correctness of the prover’s solutions.
   • Prover Training: Instruct the prover to be either "helpful" or "sneaky," rewarding correct solutions in the former role and incorrect solutions that fool the verifier in the latter.

Initially, the sneaky prover easily deceives the verifier. However, as incorrect solutions are added to verifier training, the prover must continuously develop new strategies. Training concludes when the sneaky prover can no longer deceive the verifier.

Training Framework: Multi-agent setup with honest prover (correct solutions) and sneaky prover (deceptive solutions), both checked by a smaller model(robust verifier).

Details:​

• Unique, effective loss function.
• Honest prover's solutions become detailed and reasoned over time.
• Sneaky prover's solutions develop subtler flaws.
• Legibility transfer: Honest prover’s solutions become easier for humans to verify with training.
• Humans perform better (speed and accuracy) with more trained models in time-constrained tasks.
• Traditional reinforcement learning for correctness leads to less legible solutions.
• Approach helps in training models to explain actions comprehensibly, reducing deception.
• "Legibility tax" affects large model performance. - as legibility increases performance decreases!

🔗 arXiv Link

📜 Download paper

Ready to start building?​

Check out the Quickstart tutorial, or sign up for a free Weaviate Cloud account.

You don't need a 2 trillion parameter model to tell you the capitol of France is Paris.​

Be smart and route between a panel of models according to query difficulty and model specialty!

New paper proposes a framework to train a router that routes queries to the appropriate LLM to optimize the trade-off b/w cost vs. performance.

Model inference cost varies significantly: Per one million output tokens: Llama-3-70b ($1) vs. GPT-4-0613 ($60), Haiku ($1.25) vs. Opus ($75)

Overview:​

The RouteLLM paper propose a router training framework based on human preference data and augmentation techniques, demonstrating over 2x cost saving on widely used benchmarks.

They define the problem as having to choose between two classes of models: (1) strong models - produce high quality responses but at a high cost (GPT-4o, Claude3.5)

(2) weak models - relatively lower quality and lower cost (Mixtral8x7B, Llama3-8b)

A good router requires a deep understanding of the question’s complexity as well as the strengths and weaknesses of the available LLMs.

Explore different routing approaches:

• Similarity-weighted (SW) ranking
• Matrix factorization
• BERT query classifier
• Causal LLM query classifier

Neat Ideas to Build From:​

• Users can collect a small amount of in-domain data to improve performance for their specific use cases via dataset augmentation.

• Can expand this problem from routing between a strong and weak LLM to a multiclass model routing approach where we have specialist models(language vision model, function calling model etc.)

• Larger framework controlled by a router - imagine a system of 15-20 tuned small models and the router as the n+1'th model responsible for picking the LLM that will handle a particular query at inference time.

• MoA architectures: Routing to different architectures of a Mixture of Agents would be a cool idea as well. Depending on the query you decide how many proposers there should be, how many layers in the mixture, what the aggregate models should be etc.

• Route based caching: If you get redundant queries that are slightly different then route the query+previous answer to a small model to light rewriting instead of regenerating the answer

🔗 arXiv Link

📜 Download paper

Ready to start building?​

Check out the Quickstart tutorial, or sign up for a free Weaviate Cloud account.

How do you get the retrieval quality of a cross-encoder/re-ranker and the efficiency of a bi-encoder?​

Typically people choose to do this with the trusty old retrieve-and-re-rank approach.

This new paper from DeepMind proposes Adaptive Cross-Encoder Nearest Neighbor Search, an alternative which approximates the re-ranker query-document similarities while still using a bi-encoder setup.

High-level Overview:​

You can think of this as an efficient way to train an adaptor for the query vector that transforms the query vector in such a way that makes the similarity scores b/w query-documents more like the cross-encoder similarity scores.

• Once you pass the query vector through the trained adopter then you can simply use cosine similarity with the document embeddings
• Can use existing bi-encoder models to initialize the item and query embeddings
• In an offline indexing step -> compute query/item embeddings to index a given set of items from a target domain making sure the similarity scores are similar to cross encoder scores
• At test time -> compute the test query embedding to approximate cross-encoder scores of the given test query for a small set of adaptively-chosen items
• Perform retrieval with the approximate cross-encoder scores

Details:​

At test time, they keep item embeddings fixed and perform retrieval over multiple rounds, alternating between:

estimating the test query embedding by minimizing error in approximating CE scores of items retrieved thus far, and

using the updated test query embedding for retrieving more items in the next round.

In the last round once the test query embedding is fully refined, this test query embedding can now be used to retrieve items using cosine similarity.

Proposed k-NN search method can achieve up to 5% and 54% improvement in k-NN recall for k = 1 and 100 respectively over the widely-used DE-based retrieve-and-rerank approach.

🔗 arXiv Link

📜 Download paper

Ready to start building?​

Check out the Quickstart tutorial, or sign up for a free Weaviate Cloud account.

Should you finetune your LLM or just give relevant examples in the prompt? How many examples should you give for best performance?? If you give more will it hurt perf?? Does order of the examples matter!??​

New paper from Deepmind answers all these questions and more, so much to take away from this one, lets dig in!

Main Takeaways:​

• Large performance jumps when going from providing very few(1-5) examples(few-shot incontext learning(ICL) to providing many(100s-1000s) examples(many-shot ICL) - The harder the task the more it benefits from more examples in the prompt!
• Propose using synthetically generated examples(as opposed to human labelled ones) and find that works quite well
• Propose providing only questions and no answers, in the examples, and find this also works quite well!!
• Show that many-shot ICL can overcome pre-training biases, perform comparably to supervised fine-tuning, and learn non-NLP prediction tasks.

Juicy Details:​

• Full supervised/instruction fine-tuning only slightly outperforms many-shot ICL for many tasks

• They mainly test Gemini 1.5 but also try GPT4 and Claude 3.5 and show that different LLMs have varying degrees of success when using many-shot ICL - not a model agnostic trick

• They show that if you provide encough examples in the prompt it can adapt to unseen non-lingual tasks and even in domains that might be misaligned with an LLM’s training data

• Surprisingly, the order of examples in the prompt also influences many-shot performance - would be interesting to see how optimization systems like DSPy can help with this

• Adding more examples, then optimal, in the prompt can also sometimes degrade performance for certain tasks - weird finding - opportunity for DSPy to do its thing here aswell

• Many-shot ICL achieves comparable or superior performance when using only problems compared to using problems with solutions - signals that providing solutions with many-shot ICL might just be redundant

• Many shot ICL also shows an improvement in out-of-distribution general problem-solving abilities from in-context learning - Math tasks and etc.

• Biases instilled in the model during pre-training can also be overcome with many shot ICL - a small number of shots leads to the model giving into the bias but with enough examples this eventually diminishes as task learning takes effect in the many shot regime.

🔗 arXiv Link

📜 Download paper

Ready to start building?​

Check out the Quickstart tutorial, or sign up for a free Weaviate Cloud account.

🏹Multi-vector retrieval approaches, like ColBERT, have great retrieval quality but vector count can balloon, AnswerAI propose a solution!​

Below is an explanation of how ColBERT works and AnswerAI's proposed modification!

Breakdown of different types of encoders:​

Cross-encoders:

• Document text & query text strings concatenated and passed into a cross-encoder which then outputs a rank/score.

Bi-encoders:

• Document text passed into an encoder and generates a document embedding
• Query text separately passed into an encoder and generates a query embedding
• Similarity of query and doc embedding calculated
• Retrieval performance can suffer especially on Out-Of-Domain data

Multi-vector bi-encoder - (eg. ColBERT):

• Functions as a bi-encoder: all documents representations are pre-computed in isolation
• Similarity computation occurs between individual query and document token vectors, as opposed to the full document.

Main weakness of multi-vector approaches:​

1. Storage and memory usage balloons up, each token in a document requires a vector(as opposed to one document = one vector)

2. Complicated to efficiently search through multiple vectors

AnswerAI's Proposed Token Pooling Solution:​

• Main Idea: a lot of the tokens are likely to carry somewhat redundant semantic information, we can semantically cluster them!
• Requires no model modification whatsoever, nor any complex processing, while greatly improving the scalability of easily updatable indexing methods - like HNSW, which are typically harder to use with ColBERT.

Approach:

• Token pooling by clustering similar tokens within a given document and averaging (mean pooling) their representation.
• After being pooled, the vectors are then quantised to 2-bits using the ColBERTv2 quantisation approach
• Each cluster is represented by a single token by averaging the values of contained tokens

Results:​

• All results compared to non-pooled ColBERT vector approach
• Pooling by a factor 2 achieves a 50% vector count reduction and 100.6% retrieval performance on average.
• Pool factor = 3 achieves 66% reduction while reaching 99% performance.
• Pool factor = 4 achieves 75% reduction while reaching 97% performance.

Code

Blog

🔗 Blog Link

Ready to start building?​

Check out the Quickstart tutorial, or sign up for a free Weaviate Cloud account.

🤖Can multiple smaller open-source LLMs be combined to outperform larger monolithic LLMs?​

New paper shows that LLMs tend to generate better responses when presented with outputs from other models, even if less capable.

They use this to build a Mixture of Agents(MoA) architecture where multiple LLMs are used to iteratively enhance the generation quality.

LLMs in deeper layers are presented responses from LLMs in earlier layers and iteratively improve the response; mitigates individual model deficiencies and enhance overall response.

MoA Architecture​

The complete architecture consists of LLM agents playing one of two roles:

1. Proposers: These models generate initial reference responses.

2. Aggregators: These models synthesize the different responses from the proposers into one high-quality response.

• Models used: Qwen1.5-110B-Chat, Qwen1.572B-Chat, WizardLM-8x22B, LLaMA-3-70B-Instruct, Mixtral-8x22B-v0.1, dbrx-instruct

• 3 MoA layers and use the same set of models in each MoA layer

• Qwen1.5-110BChat as the aggregator in the last layer

• Some models work better as proposers and others as both proposers and aggregators.

How do you choose which models to include in the MoA??​

Two metrics are used to assess which models are included in the mixture:

• Performance: The average win rate of models in layer i decides if they are included in layer i + 1.

• Diversity: The diversity of model outputs is important - using heterogeneous models across layers is better then using the same model

Details:​

• MoA achieves a new SOTA win rate of 65.8% on AlpacaEval 2.0 compared to the previous best of 57.5% achieved by GPT-4 Omni.

• Overall performance comparable to GPT-4 Turbo while being 2× more cost-effective.

• No finetuning required only utilizes the interface of prompting and generation of LLMs.

• Extends the MoE concept to the model level by operating at the model level rather than at the activation level.

• You can swap the final aggregator to any LLM of your choice (Gemini, GPT-4o, Claude3.5) and this improves performance!

Demo

Code

Blog

🔗 arXiv Link

📜 Download paper

Ready to start building?​

Check out the Quickstart tutorial, or sign up for a free Weaviate Cloud account.

Using Metadata Filters to Improve Recall in RAG​

Filtered search using metadata filtering is a simple technique that can significantly improve retrieval quality in a RAG pipeline, but how do you extract metadata from chunks if your data doesn't already come with it??

GLiNER is a powerful model that allows you to extract arbitrary entities such as names, times, places, etc. from any text chunk. It outperforms decoder models like ChatGPT and others at zero-shot identification of names entities in text chunks.

How it Works:​

GLiNER operates by taking in text chunks and entity labels, that you want to identify in the chunks. Both inputs are concatenated, encoded, and projected into the same latent space and fed into a classifier that predicts the entity labels per word in the text input.

This method allows the model to generalize across different NER tasks and labels passed in at query time.

The fact that entity vectors and the text chunk is concatenated allows the entity labels to attend to the text chunks and vice versa in the encoder step which allows GLiNER to work very well OOD.

Architechture:​

GLiNER consists of three joined components:

1. An encoder backbone(DeBERTa) that generates token-level representations of the entity labels and text tokens.

2. A simple feedforward network that takes in entity token representations from the encoder and embeds them into vectors.

3. A Span layer that embeds groups of words (ie. "McGill University" -> vector) from the text into vectors

The entity vectors and span vectors are then combined and used to train a classifier that identifies which text spans correctly paired with entity labels.

Effectiveness:​

• Generalization: The model demonstrates SoTA performance across various NER benchmarks, outperforming traditional task-specific models.

• Adaptability: GLiNER is super easy to use, pass in any text and any labels you want to extract and it simply works making it a flexible solution to add to your RAG pipeline.

• Scalability: The unified approach simplifies the deployment process, as a single model can handle multiple NER tasks.

Demo Code

🔗 arXiv Link

📜 Download paper

Ready to start building?​

Check out the Quickstart tutorial, or sign up for a free Weaviate Cloud account.

How do you train a Large Language Model without it memorizing training data?​

This paper proposes a technique called Goldfish Loss that is now used to mitigate the risk of LLMs memorizing copyrighted or private training data.

In Short:​

The paper introduces Goldfish Loss, a method where the model does not compute the loss on every token but excludes (e.g.) 1 in 4 tokens from loss computation. This makes it difficult for the model to memorize the training data.

How it Works:​

Goldfish Loss works by omitting a portion of tokens from loss computation during training. When the model encounters these excluded tokens at test time, it has to guess, reducing its ability to reproduce training samples exactly.

Effectiveness:​

In standard training on Wikipedia articles, about 85% of them get perfectly memorized after 100 updates. With Goldfish Loss, the model usually diverges from the training data within the first 5 tokens it generates.

Trade-off:​

The model learns slower because it does not get credit for the dropped tokens. Training on N tokens with Goldfish Loss is equivalent to standard training on 0.75N tokens.

Benefits:​

Goldfish training is scalable and helps avoid the need for unlearning methods, which are often not scalable. This makes it possible to prevent the memorization of copyrighted text/code during training.

Results:​

The paper validates Goldfish Loss by pre-training a model for 200B tokens, showing that it effectively prevents memorization without significantly compromising the learning rate.

Details in the Paper:​

• Explanation of the Goldfish Loss technique
• Comparison of memorization rates with standard training
• Analysis of the trade-offs between learning rate and memorization prevention
• Validation experiments and results

Code

🔗 arXiv Link

📜 Download paper

Ready to start building?​

Check out the Quickstart tutorial, or sign up for a free Weaviate Cloud account.

How do you teach a Large Language Model to see? Here's a breakdown!

This paper proposes a technique called Visual Instruction Tuning that is now used by many of the language vision models we see in the field such as LLaVA, GPT4-Vision and Gemini etc.

In Short:​

The paper introduces a method to generate multimodal language-image instruction-following data using a language-only GPT-4 model. This data is then used to train LLaVA, a model that combines a vision encoder and a large language model (LLM) for general-purpose visual and language understanding.

How it Works:​

VIT works by using GPT-4 to generate instructions for corresponding images and captions. This dataset is used to train LLaVA to learn to follow instructions and understand images. A vision encoder (CLIP ViT 40) is combined with an LLM (Vicuna) to process text and images and generate text.

Architecture:​

LLaVA consists of two main components:

1. Vision Encoder (VE): A pre-trained vision encoder (e.g. CLIP) that takes an image as input and generates a visual embedding.

2. Large Language Model (LLM): A pre-trained LLM (Vicuna) that takes a text prompt as input and generates a language embedding.

The VE and LLM are combined through a series of layers and mechanisms to enable multimodal understanding and generation:

Benefits:​

The combination of the VE and LLM enables LLaVA to understand and generate text and images in a unified framework, leverage the strengths of both visual and language models, generalize to unseen images and text prompts.

Results:​

LLaVA achieves a 85.1% relative score compared to GPT-4 on a synthetic multimodal instruction-following dataset

Details in the Paper:​

• The VIT generation process, including prompt engineering and filtering

• The LLaVA architecture, including the vision encoder and LLM components

• Experimental results, including comparisons to GPT-4 and other baselines

• Ablation studies and analysis of the effectiveness of different components and training strategies

Demo

Code

🔗 arXiv Link

📜 Download paper

Ready to start building?​

Check out the Quickstart tutorial, or sign up for a free Weaviate Cloud account.

Can we finetune our LLM and retriever together to improve RAG performance? This paper proposes a technique to do exactly that!

RAG Basics:​

When you prompt an LLM, RAG supplies relevant documents. A separate retrieval model computes the probability of each text chunk being relevant and provides the top chunks to the LLM. The LLM generates tokens based on the chunks, prompt, and previous tokens.

In Short:​

Fine-tuning LLMs and retrieval models together improves performance without extensive data processing, enabling better retrieval-augmented generation. LLMs aren't exposed to retrieval-augmented inputs during pretraining, limiting their ability to use retrieved text effectively. Fine-tuning the LLM and retrieval model together can improve performance without requiring extensive data processing.

How it Works:​

Authors from Meta fine-tuned Llama 2 (65B parameters) and DRAGON+, a retriever, to create RA-DIT 65B. They fine-tuned Llama 2 on prompts with retrieved text and questions, and fine-tuned DRAGON+ to retrieve more relevant chunks. Fine-tuning was supervised for tasks like question-answering and self-supervised for text chunk completion.

Results:​

RA-DIT 65B achieved 49.1% accuracy on average across four question datasets, outperforming LLaMA 2 65B with DRAGON+ (45.1%) and LLaMA 2 65B alone (32.9%). With five example inputs, RA-DIT 65B reached 51.8% accuracy. RA-DIT offers an efficient way to enhance LLM performance with RAG, making it a valuable technique for developers.

Details:​

RA-DIT fine-tunes Llama 2 and DRAGON+ to work together effectively, leveraging the strengths of both models to generate better output. By fine-tuning the LLM to better use retrieved knowledge and the retrieval model to select more relevant text, RA-DIT achieves improved performance without requiring extensive data processing.

🔗 arXiv Link

📜 Download paper

Ready to start building?​

Check out the Quickstart tutorial, or sign up for a free Weaviate Cloud account.

Can you reliably tell apart fake, LLM-generated, text from human-written text?🤖⚖️👱

Binoculars is a technique that requires no training and can 0-shot detect 90% of LLM-generated content at a 0.01% false positive rate.

In Short⏩:​

Human tokens are, on average more surprising to LLMs than other LLM tokens. They use this insight to identify a classification threshold.

Given two LLMs, M1 and M2. Their main insight is that human text should diverge from M1 more than M2 diverges from M1, provided the LLMs M1 and M2 are more similar to each other than they are to a human.

Details🔎:​

They look at the text in question through the lenses of two different LMs and calculate two perplexity scores:

1. Perplexity of the text using an "observer" LLM(M1).

2. Compute all the next-token predictions that a "performer" LLM(M2) would make at each position in the string, and compute their perplexity according to the "observer" LLM(M1).

Then, they take the ratio of the two PPL scores: PPL1/PPL2.

If the string is written by a machine, we should expect these two perplexities to be similar. If it is written by a human they should be different.

They find that if PPL1/PPL2 > 0.9 then text is human generated; otherwise it's LLM generated.

Works on detecting fake multilingual text as well.

They think of PPL as how surprising the next token is - human tokens are, on average more surprising to LLMs than LLM tokens.

They use PPL2, what they call cross perplexity, to account for the increase in perplexity due to the prompt; normalizing the observed perplexity by the expected perplexity of a machine acting on the same text, we can arrive at a detection metric that is fairly invariant to the prompt

They use Falcon-7b model (M1) and the Falcon-7b-instruct (M2)

🧑‍💻Code

🤗HuggingFace Demo

🔗 arXiv Link

📜 Download paper

Ready to start building?​

Check out the Quickstart tutorial, or sign up for a free Weaviate Cloud account.

A recent survey on Retrieval-Augmented Generation (RAG) mentions an evolving paradigm: Modular RAG.

Modular RAG is comprised of various functional modules. Thus, modular RAG is not standalone. Instead, different RAG patterns are composed of different modules.

For example, the following animation shows: 🥚 The original naive RAG paradigm consists of the “Retrieval”, "Augmentation," and "Generation" modules.

🐣 After naive RAG has shown some limitations, advanced RAG has emerged as a new paradigm. A typical pattern of Advanced RAG builds upon the foundation of Naive RAG by adding “Rewrite” and “Rerank” modules.

🐓 Different RAG patterns, such as DSP, can be composed of entirely different modules.

The modular RAG paradigm is slowly becoming the norm in the RAG domain due to its versatility and flexibility, allowing:

• the adaption of modules within the RAG process to suit your specific problem,
• for a serialized pipeline or an end-to-end training approach across multiple modules.

I definitely recommend checking out the full survey if you want to catch up on recent advancements in the RAG domain.

🔗 arXiv Link

📜 Download paper

Ready to start building?​

Check out the Quickstart tutorial, or sign up for a free Weaviate Cloud account.

An Overview of OpenAI's New Truncatable - Matryoshka Embeddings🪆​

OpenAI recently announced embeddings that you can simply use chunks of (say the first 8, 16, 32, 64, 128 or 256 ... dimensions of the total 2048d vector) they use Matryoshka representation learning(MRL).

This is how they work, In Short⏩:

• MLR allows you to use a subset of the dimensions of the embedding vector - earlier dimensions store more information than dimensions later on in the vector, which simply add more details

• You can understand how this works by the analogy of trying to classify an image at multiple resolutions - the lower res give high-level info and the higher res add details - Human perception of the natural world also has a naturally coarse-to-fine granularity

• This is done by modifying the loss function which is optimized. If previously the loss function was L, for MRL we break down the Loss function into the sum of the losses on individual vector dimension ranges: Loss_Total = L(upto 8d) + L(upto 16d) + L(upto 32d) + ... + L(upto 2048d) - Now there is incentive for the model to capture information in each sub-section of the vec.

• After modifying the loss you get these truncatable vectors for free/no additional costs - this works on almost all loss functions and pre-existing models can be finetuned to output MRL vectors! - super easy-to-adopt technique

• You can actually use any slice of dimensions, not just 8, 16,32 ... - b/c information is diffused in an interpolative fashion; so you can choose an arbitrary-sized chunk dimension that falls between the chosen granularity of the representations

🔗 arXiv Link

📜 Download paper

Ready to start building?​

Check out the Quickstart tutorial, or sign up for a free Weaviate Cloud account.

A Simple Overview of the LLM Training Steps:🔡

1. Unsupervised Pretraining:

High quantity, low quality data The model is trained to predict the next token for trillions of tokens. Produces what is called the foundation or base model.

2. Supervised Finetuning:

Low quantity, high quality {prompt, response} Enables the model to be finetuned for dialogue - turning the base model into a chatbot Often referred to as instruction tuning

3. Reinforcement Learning from Human Feedback (RLHF): - lots of innovation going on here (will cover DPO, PTO, and KTO soon)

This is a two-step process:

a. Train a reward model to act as a scoring function:

This model will take in a prompt + response and provide a score of how good it is. Human labelers are asked to pick good vs. bad responses and this data is used to train a model.

b. Optimize LLM to generate responses for which the reward model will give high scores.

Use an iterative procedure to update a part of the model such that:

1. Produces outputs with higher score
2. Outputs that are not too far away from the SFT model from Step 2
3. Outputs that aren't getting worse a text completion

Specifically for this phase it is better to think of this as learning an optimal strategy/policy for predicting a probability distribution over tokens and we want to tweak this distribution to produce higher quality text, here the:

The policy is a language model that takes in a prompt and returns a probability distribution over text. The action space of this policy is all the tokens corresponding to the vocabulary of the language model (~50k tokens) The observation space: distribution of possible input token sequences The reward Model is a combination of the preference model(score higher) and a constraint on policy shift(don't change too much, get worse at text completion).

RLHF Learning Resources:

1. InstructGPT Paper

2. RLHF Paper Anthropic

3. OpenAI Blog

4. RLHF Blog Chip Huyen

5. RLHF Nathan Lambert

6. Karpathy Talk

Ready to start building?​

Check out the Quickstart tutorial, or sign up for a free Weaviate Cloud account.

Using a 7B Model + RAG to Identify and Edit Word-level Hallucinations in LLMs better then GPT-4:​

In Short⏩:​

Train a model that consists of a Retreiver and a Language Model:

The retriever, Mret, takes the original output you want to check to hallucination (y) and optionally input prompt (x) and retrieves top relevant documents (C). So C = Mret(x, y). This can be a vector database like Weaviate for example.

The detector and editor, Medit, takes in the context - (C), input - (x) and output - (y) and detects (and if possible also edits/corrects) factual errors in (y) given the retrieved context (C): y* = Medit(x, y, C).

🏋️Training:​

Create a synthetic hallucination dataset of 35k C = context, y=incorrect output, y*=annotated fixed output -> (C, y, y*)

Magic Synthetic Dataset Creation:

GPT-4 is few-shot prompted to add different types of errors to a passage

It is also instructed to mark phrases or sentences for deletion along with their error type and insert phrases and sentences along with insertion tags

Start off with Llama2-Chat 7B to initialize Medit and then train on (C, y, y∗)

Medit takes in (C, y) as input and learns to predict the edited outputs with tags to represent error type y∗ using standard language modeling objective.

The model, once trained, can identify different types of hallucination and mark which words they come from - it also suggests edits to improve factuality.

Result 📈:​

The model has a fine-grained hallucination detection accuracy 46.5% while it's binary acc.{hallucination, no hallucination} is 79%.

For comparison ChatGPT has a fine-grained hallucination detection acc. of 21.5% (59% binary acc) w/o RAG and 26%(68.5% binary hall detect) w/ RAG

💻Code

🔷Data

🏗️Model

🔗 arXiv Link

📜 Download paper

Ready to start building?​

Check out the Quickstart tutorial, or sign up for a free Weaviate Cloud account.

A breakdown of the different types of hallucinations from AI2:🍄​

1. Verifiably Factually Wrong ❌

• Entity: an entity in a statement is incorrect (eg. Christmas falls on Nov. 25th)

• Relation: semantic relationship in a statement is incorrect (eg. The mouse ate the cat.)

• Contradictory: statements that entirely contradict relevant evidence from the web (eg. Raptors are yet to win an NBA final.)

2. Unverifiable Types of Hallucinations ⁉️

• Invented: statements of concepts that do not exist in world knowledge (eg. MJ created the sideways somersault)

• Subjective: Statement that lacks universal validity - basically an opinion (eg. The Raptors are the best NBA team)

• Unverifiable: potentially factual statement but cannot be grounded in world evidence(eg. Jensen sleeps in a leather jacket.)

🔍Word vs. Sentence Level:​

Entity and Relation are usually word level, and so can be fixed with small edits if you know where they occur.

Contradictory, Invented, Subjective, and Unverifiable are often sentence level and thus need to be removed completely to fix the issue.

💻Code

🔗 arXiv Link

📜 Download paper

Ready to start building?​

Check out the Quickstart tutorial, or sign up for a free Weaviate Cloud account.

A breakdown of the Long Context Retrieval Embedding Models from Stanford!💥

In Short⏩:​

1. They release 3 long context(2k/8k/32k) BERT-like encoder embedding models on HuggingFace

2. The models are only 80M params and outperform MUCH larger models (4-85x larger)

3. Accessible via @togethercompute endpoints and integrated into @llama_index and @LangChainAI

4. They also release LoCo a long context retrieval benchmark.

🏗️Architechtural Details:​

1. They replace the Attention and MLP blocks in the transformer architecture with diagonal block matrix (Monarch Matrices -M2) operations which are hardware optimized and subquadratic in the sequence length - O(N^(1.5))

2. This enables scaling sequence length and model parameters better.

🪃Training Details:​

1. These M2 models are trained for long context retrieval on a mixture of long and short context tasks data - surprisingly only training on long context doesn't work.

2. Use a cosine similarity loss instead of the trusty supervised contrastive training loss.

   This loss function. can be computed independently per datapoint in a batch instead of needing to sum over all negative examples in a batch.

   Thus training can be scaled for large batch sizes of long context inputs without OOM'ing

📜Blog

🧑‍💻Code

🔷Models

🔗 arXiv Link

📜 Download paper

Ready to start building?​

Check out the Quickstart tutorial, or sign up for a free Weaviate Cloud account.