Autoevals is Braintrust's library for scoring LLM outputs — LLM-as-a-judge classifiers, heuristic string comparisons, embedding similarity, RAGAS metrics, all behind one Score object with a .score between 0 and 1.
What it does
You call a scorer with output, expected, and sometimes input, and get back a Score(name, score, metadata). Some scorers are pure Python (Levenshtein distance, exact match), some hit an LLM to grade subjectively (factuality, toxicity, security review), some hit an embeddings endpoint. It ships for both Python (pip install autoevals) and TypeScript (npm install autoevals).
Why I starred it
I've written this kind of code before — a wrapper around an OpenAI call that asks "is this answer correct, yes or no" and parses the response. It's always messier than it should be: prompt templating, forcing structured output, retrying on rate limits, handling both sync and async callers. Autoevals has clearly been through that pain multiple times and hardened every seam. It's also honest about what it is — a scoring library, not an eval framework — so it composes cleanly with whatever harness you already run evals in.
How it works
The part that made me stop scrolling is SpecFileClassifier in py/autoevals/llm.py:499. Classes like Factuality, Battle, Humor, and Security are all one-liners:
class Factuality(SpecFileClassifier):
"""Check factual accuracy against a reference."""
passThere's no logic in the class body. SpecFileClassifier.__new__ converts the class name to snake_case with a regex (FooBar → foo_bar), loads templates/foo_bar.yaml, and builds an LLMClassifier from it:
cls_name = cls.__name__
template_name = re.sub(r"(?<!^)(?=[A-Z])", "_", cls_name).lower()
template_path = os.path.join(SCRIPT_DIR, "templates", template_name + ".yaml")templates/factuality.yaml is just a prompt and a choice_scores map:
prompt: |-
Compare the factual content of the submitted answer with the expert answer...
choice_scores:
"A": 0.4
"B": 0.6
"C": 1
"D": 0
"E": 1So adding a new built-in scorer is: write a YAML file, add an empty class. The actual grading discipline comes from build_classification_tools (llm.py:115), which forces the model to call a select_choice function with an enum constrained to the YAML's choice keys — no regex-parsing "the answer is (C)" out of free text, which is how I'd have built this a few years ago.
The other thing worth reading is oai.py. is_gpt5_model() (oai.py:133) checks model.startswith("gpt-5") and routes those calls through the Responses API instead of Chat Completions, then convert_responses_to_chat_completion (oai.py:221) reshapes the response back into the old choices[0].message format so the rest of the scoring pipeline doesn't need to know the difference. That's the kind of shim you only write after a model launch breaks your assumptions — a nice trace of how fast this library has to move to track the frontier.
run_cached_request (oai.py:620) is misleadingly named — despite the name there's no caching in this function itself (that comes from Braintrust's wrap_openai tracing layer if it's installed). What it does do is retry on rate limits with backoff that grows by 1.5x per attempt, up to 100 tries, which is a lot of patience to bake into a library default.
ListContains in py/autoevals/list.py is the sharpest piece of engineering here: to score how well a generated list matches an expected list, it computes pairwise similarity between every output/expected pair (Levenshtein by default, swappable) and then runs scipy.optimize.linear_sum_assignment — the Hungarian algorithm — to find the optimal one-to-one matching before scoring. Most homegrown list-comparison code I've seen just does greedy nearest-neighbor matching, which can produce suboptimal pairs. This does it properly.
Using it
from autoevals import Factuality, init
from openai import OpenAI
init(OpenAI())
result = Factuality().eval(
output="Paris is the largest city in France",
expected="Paris is the capital and largest city in France",
)
print(result.score, result.metadata["rationale"])Local, no API key needed, for the heuristic scorers:
from autoevals import Levenshtein
Levenshtein().eval(output="hello wrld", expected="hello world").score
# 0.9090909090909091polyleven does the actual distance computation — a C extension, not a pure-Python edit-distance loop, so it doesn't fall over on longer strings.
Rough edges
Score.__post_init__ in py/autoevals/score.py:36 has a bug in the deprecation warning for the error field:
print(
"The error field is deprecated, as errors are now propagated to the caller...",
sys.stderr,
)sys.stderr is passed as a second positional argument to print, not as file=sys.stderr. Python treats it as another value to print, separated by a space, and writes the whole thing to stdout — the opposite of what the comment intends. Harmless (it's a deprecation notice nobody depends on), but it's proof the deprecated error field isn't exercised by CI.
ragas.py is 1,476 lines reimplementing RAGAS metrics (faithfulness, context precision/recall) — a serious chunk of surface area for one file, and it's one of the least-tested corners relative to its size. The test suite overall is solid (over 2,100 lines across test_*.py files), but coverage skews toward the LLM classifiers and thread-utils, less toward ragas.py and the litellm.py gateway integration, both of which are newer additions per the commit history (0278eff, 9eba0fe).
Bottom line
If you're building eval harnesses for LLM apps and don't want to hand-roll LLM-as-a-judge scaffolding, structured-output parsing, and rate-limit retries yourself, autoevals is worth pulling in as a dependency rather than reimplementing badly. If you just need string similarity or exact match, it's overkill — grab polyleven directly.