The OpenAI RAG system struggled with multiple documents, showing inconsistent performance with our evaluation framework. However, performance improved markedly when all documents were uploaded as a single document. Despite current limitations, such as a 20-file limit per assistant and challenges in handling multiple documents, there is significant potential for improvement. Enhancing the Assistants API to match GPT quality and reducing restrictions could make it a leading RAG solution.
https://www.tonic.ai/blog/rag-evaluation-series-validating-openai-assistants-rag-performance
Its weird for me. I fried my brain since they released it on this matter and so far, whenever I want it to process, I explicitly tell it to try opening the files no matter what.
I believe there is an issue in their “os” or file management system when you’re uploading files. I believe uploading the files (so not using the file ids) fixes it. Accessing these files via file ids generated at a much earlier time is not as effective as uploading the same file although its the exact same file.
I don’t know as it’s kind of a black box right now. It might be something else and what I’m getting might be coincidental.
Very varied responses here, so we need deterministic responses (seeds) for assistants too, which I think will be available one day.
If you’re struggling, add a feedback loop and have 2 agents, one will be your agent, the other will be quality control. I think then you can generate a json response to see if the quality checker approved the output of the main agent. Works for me although conversations are sometimes very long (not always, 10% of the time) but they do figure it out eventually.