The OpenAI RAG system struggled with multiple documents, showing inconsistent performance with our evaluation framework. However, performance improved markedly when all documents were uploaded as a single document. Despite current limitations, such as a 20-file limit per assistant and challenges in handling multiple documents, there is significant potential for improvement. Enhancing the Assistants API to match GPT quality and reducing restrictions could make it a leading RAG solution.
https://www.tonic.ai/blog/rag-evaluation-series-validating-openai-assistants-rag-performance
Surprising that OpenAI would release something so haphazardly. Must’ve wanted to get it in before dev day