The speaker's team evaluates customer-specific systems by understanding the use case, building and repeatedly testing the system, and using an LLM judge to assess performance and reliability.

“We have to really understand their use cases and at the end.”

“At the end of the day, just build it and then test it a whole bunch of times and then use LLM as a judge to tell us, you know, did it work?”

“Was it reliable or do we need to add a little bit more preprocessing and context optimization ahead of time in order to make it reliable?”