This is an incomplete doc about an assortment of things loosely grouped for now? These are external tools that support interacting with multiple LLMs for querying, evaluating, or developing.
- ChainForge - [chainforge.ai](https://chainforge.ai/)
- "ChainForge is an open-source visual programming environment for prompt engineering, LLM evaluation and experimentation."
- HT: Ian Arawjo ([website](https://ianarawjo.therottingcartridge.com/), [Twitter](https://twitter.com/IanArawjo)
- citation: @arawjo2023chainforge
- Comet - [comet.com/](https://www.comet.com/)
- "Comet provides a meta machine learning platform that enables data scientists & teams to track, compare, explain, & optimize their experiment tracking."
- HT: [\@omarsar0](https://twitter.com/omarsar0/status/1702327692363018361)
> I have been really impressed with the new prompting tools by \@Cometml. I already use their tools for tracking and managing my fine-tuned LLMs, so it's cool to see that they also enable LLMOps and prompting tools to easily track and debug prompts at scale.
- Cursive - [github.com/meistrari/cursive-py](https://github.com/meistrari/cursive-py)
- HT: [\@mattshumer_](https://twitter.com/mattshumer_/status/1693664769910227167)
- GodMode - [github.com/smol-ai/GodMode](https://github.com/smol-ai/GodMode/)
- HT: [\@swyx](https://twitter.com/swyx/status/1693697843863683354)
- LangChain - [github.com/langchain-ai/langchain](https://github.com/langchain-ai/langchain)
- ragas - [explodinggradients/ragas](https://github.com/explodinggradients/ragas)
- "Evaluation framework for your Retrieval Augmented Generation (RAG) pipelines"
- HT: Aaron Tay ([website](https://musingsaboutlibrarianship.blogspot.com/) | [Twitter](https://twitter.com/aarontay))
{% include weblinks.html url="https://twitter.com/aarontay/status/1700103826987188685" %}