EVAL #006: LLM Evaluation Tools — RAGAS vs DeepEval vs Braintrust vs LangSmith vs Arize Phoenix

EVAL #006: LLM Evaluation Tools — RAGAS vs DeepEval vs Braintrust vs LangSmith vs Arize Phoenix By Ultra Dune | EVAL Newsletter You shipped the RAG pipeline. The demo worked. The CEO nodded. Then production happened. Users started asking questions your retriever never anticipated. The LLM hallucinated a return policy that doesn't exist. Your "95% accuracy" metric turned out to measure nothing useful. Welcome to the actual hard part of building LLM applications: evaluation. Here's the uncomfortable truth most AI engineering teams discover around month three: building the LLM app was the easy part. Knowing whether it actually works — consistently, at scale, across edge cases — is where projects go to die. Evals are the difference between a demo and a product. And yet most teams are still vibes-checking their outputs manually, or worse, not evaluating at all. The tooling landscape for LLM evaluation has exploded in the past year. We now have open-source frameworks, managed platforms, and