My Favorite AI Papers of 2023
LLM reasoning, truthfulness, correctness – a short summary of my favorite AI papers of 2023
With the proliferation of LLMs into many industries and use cases, and with new models coming out nearly every month, it is becoming quite evident that reasoning, truthfulness, and correctness will be some of the most critical aspects of application development with LLMs and one of the main trends in AI in 2024.
That said, my shortlist consists mostly of papers and methods that focus on addressing this particular problem.
RA-DIT: Retrieval-Augmented Dual Instruction Tuning
https://arxiv.org/abs/2310.01352
This paper introduces a novel approach for enhancing language models (LMs) using retrieval-augmented instruction tuning. Here's a summary of the key aspects:Methodology: RA-DIT involves fine-tuning pre-trained language models in two distinct steps: one focuses on enhancing the LM's use of retrieved information (LM-ft), and the other improves the retriever's relevance (R-ft). This process allows the LM to better utilize external knowledge and ignore irrelevant content while aligning the retriever’s outputs with the LM’s requirements.
Overall Architecture: The architecture comprises two main components: a pre-trained auto-regressive language model (like LLAMA) and a dual-encoder-based retriever (like DRAGON+). The LM is fine-tuned to better integrate retrieved external knowledge, and the retriever is fine-tuned to provide more contextually relevant results.
Retriever Functionality: The retriever operates by mapping queries and text chunks to embeddings, then retrieves the most relevant text chunks based on the similarity of these embeddings. During fine-tuning, the retriever is adjusted to align with the preferences of the language model, aiming to return more useful results for the LM.
Key Highlights and Conclusions: RA-DIT significantly outperforms existing in-context retrieval-augmented language model (RALM) approaches in both zero-shot and few-shot settings on various knowledge-intensive benchmarks.
It demonstrates that language models and retrievers when fine-tuned independently and then fused, can effectively compete against extensively pre-trained RALMs.
Summary of Key Results: The largest model, RA-DIT 65B, shows state-of-the-art performance in both zero-shot and few-shot scenarios, surpassing un-tuned in-context RALM approaches and competing effectively against methods requiring extensive pre-training.
Comparison with RAG Approach: Unlike the Retrieve-and-Generate (RAG) approach, which integrates retrieval during the pre-training phase, RA-DIT applies a dual fine-tuning process to pre-trained models, separating the fine-tuning of the language model and the retriever. This method offers a more flexible and efficient way to retrofit existing large language models with retrieval capabilities.
I recommend also checking out SAIL (Search-Augmented Instruction Learning), as proposed by a team from MIT Computer Science and Artificial Intelligence Lab and CUHK Centre for Perceptual and Interactive Intelligence, is an innovative method designed to enhance the performance of language models, particularly addressing the limitations of transparency and obsolescence in traditional models.
Fine-tuning Language Models for Factuality
https://arxiv.org/abs/2311.08401
The paper discusses a strategy to improve the factuality of long-form text generated by language models. The authors propose two approaches to estimate the truthfulness of the generated text: reference-based and reference-free. They use preference-based learning to optimize the models for factuality. The experiments show that fine-tuning the models with either criterion reduces the number of incorrect facts generated. The reference-free approach eliminates the need for external knowledge or annotations. The results suggest potential avenues for future work in improving factuality in dialogue models.
The authors leverage two key innovations in natural language processing (NLP) to achieve factuality tuning. Firstly, they use methods that measure the factuality of open-ended text by assessing consistency with an external knowledge base (reference-based) or a large model's confidence scores (reference-free). Secondly, they employ the direct preference optimization algorithm, which allows fine-tuning of language models based on preference rankings over possible model responses.
The results show a significant reduction in factual error rate, with improvements of 58% and 40% for biographies and medical questions, respectively.
Deliberate Problem Solving with Large Language Models
https://arxiv.org/abs/2305.10601
A fantastic paper describing a ToT approach, which enables a much better LLM problem-solving capabilities. The main characteristic of this approach is that the algorithm can find the most optimal solution over a tree of possible solutions where LLMs are used to produce thoughts which are then represented as nodes in the tree and branches of the tree are heuristics the sum of which lead to the most optimal solution.
As such, any graph exploration algorithm is sufficient to find the most optimal chain of thoughts. Notably, the paper mentions BFS or DFS as the most simple algorithms (A* could be used too).
The paper also compares different approaches, such as iterative-refine and classical search methods, and evaluates their effectiveness in generating coherent passages.
In my opinion, a combination of these techniques will be the key to achieving better reasoning capabilities, especially in custom fine-tuned models which are smaller and more efficient to deploy for solving specific narrow tasks or creating smarter and more helpful assistants using existing larger models.
Teaching Small Language Models How to Reason
https://arxiv.org/abs/2311.11045
This paper in particular is going to be the most useful for those fine-tuning and deploying smaller more cost-efficient custom models.
The objective of Orca 2, a method proposed in this paper, is two-fold. Firstly, it aims to teach smaller models a suite of reasoning techniques such as step-by-step processing, recall-then-generate, recall-reason-generate, extract-generate, and direct-answer methods. Secondly, it attempts to help these models determine the most effective reasoning strategy for each task, allowing them to perform at their best. The paper emphasizes the importance of teaching smaller models to select the most suitable solution strategy based on the problem at hand at runtime, rather than simply imitating the reasoning behavior of larger models.
The paper uses ACI-BENCH, QMSum, and MS MARCO datasets for the evaluation grounding in abstractive summarization with the primary objective of measuring the quality of generated summaries and the hallucination rate of different models.
DSPy: Compiling Declarative Language Model Calls Into Self-Improving Pipelines
https://arxiv.org/pdf/2310.03714.pdf
DSPy framework is gaining popularity in the community and provides a convenient way of assembling complex LLM pipelines (mostly involving RAG techniques) using a simple declarative Python code. I highly recommend checking it out!
The key concepts described in this paper focus on solving a question-answering task using interleaving Thought, Action, and Observation steps, which form a pipeline.
The Thought step allows reasoning about the current step, while the Action step can be of three types: Search[entity], Lookup[keyword], and Finish[answer].
The Observation step provides the result of the Action taken. The paper provides examples of how this approach can be applied to answer specific questions.
The paper is full of extensive Python code examples, making it easy to jump into experimenting with the proposed techniques fairly quickly.
DSPy is already being used by several LLM/RAG Python frameworks under the hood, and I predict it’s going to be a key part of high-level LLM programming in 2024.
This concludes the short list of my most favorite papers of 2023. As I mentioned earlier, I believe that LLM reasoning, truthfulness, and correctness will be the key topics discussed in 2024, as the models get bigger they are not necessarily getting smarter, especially when applied in narrow domains with proprietary data.
If you liked this article, please consider subscribing to my blog, and I would appreciate it if you could share it with your colleagues and friends.