LLM frameworks

This is an incomplete doc about an assortment of things loosely grouped for now? These are external tools that support interacting with multiple LLMs for querying, evaluating, or developing.

ChainForge - chainforge.ai
- “ChainForge is an open-source visual programming environment for prompt engineering, LLM evaluation and experimentation.”
- HT: Ian Arawjo (website, Twitter
- citation: Arawjo et al. (2023)
Comet - comet.com/
- “Comet provides a meta machine learning platform that enables data scientists & teams to track, compare, explain, & optimize their experiment tracking.”
- HT: @omarsar0
  
  I have been really impressed with the new prompting tools by @Cometml. I already use their tools for tracking and managing my fine-tuned LLMs, so it’s cool to see that they also enable LLMOps and prompting tools to easily track and debug prompts at scale.
Cursive - github.com/meistrari/cursive-py
- HT: @mattshumer_
GodMode - github.com/smol-ai/GodMode
- HT: @swyx
LangChain - github.com/langchain-ai/langchain
ragas - explodinggradients/ragas
- “Evaluation framework for your Retrieval Augmented Generation (RAG) pipelines”
- HT: Aaron Tay (website | Twitter)

@aarontay via Twitter on Sep 8, 2023

My anecdotal experiences trying academic search + generative AI that generate direct answers with citations using RAG (retrieval Augmented Generation) method (1)

References

Arawjo, I., Vaithilingam, P., Swoopes, C., Wattenberg, M., & Glassman, E. (2023). ChainForge. https://www.chainforge.ai/. [arawjo2023chainforge]