In a recent project, I wanted to generate structured output according to a highly-nested hierarchy of Pydantic models (see My Desired Structured Output). The ultimate goal is to generate objects of type N, but I knew that’s probably asking too much, so I decided to see if LLMs can generate objects of type E or I. This quest sent me down a rabbit whole, and it became clear to me that many open-source LLM projects are on shaky grounds and need much improvement.

SOSO (as of January 26, 2024)

❌ outlines

• Looked promising at first: support for CFGs, JSON schemas, Pydantic models, etc. and integration with llama-cpp-python. However, all the outputs were nonsense.
• Turns out, the llama-cpp-python integration was pretty recent and lots of things (e.g., tokenization) were not handled correctly.

❌ guidance

• Maybe one of the most starred repos in this field, but they revamped the syntax and opted for something that doesn't make sense (e.g., concatenating a lm LLM object such as llama-cpp-python model with a string!).
• They expose parts of their syntax to generate CFGs on the go, but they don't offer a cleaner approach to generating grammars (such as converting a Pydantic model or JSON schema to grammar).

❌ LMQL

• Despite being a bit unknown in this field, it has a nicer syntax and better integration with Python. However, LMQL only supports dataclasses, not Pydantic models or JSON schema (last I checked). And AFAIK it doesn't handle nested models very well (or at all). We can constrain the output using Python code in LMQL, but similar to guidance, this process is far from being clean and portable.

❌ JSONFormer, ReML, etc.

• Abandoned projects.

❌ llama.cpp (GBNF)

• There's a JSON-schema-to-grammar converter, but it can't handle the type of JSON schema that I needed.
• llama.cpp server kept crashing every time it couldn't parse the grammar string.

❌ LMFE

• A simple and small project made by someone for their own needs. Yet, this is the only one that works at least for objects E and I . However, I was still not able to generate object N with this.

✅ llama-cpp-agent

• The only tool that’s capable of generating all the objects, including N[0] Whether you actually want that complex of a grammar is another matter.
• [0]: The only caveat is that currently it doesn’t support Literal, so you need to use Enum instead, which is not a big deal.