I have about a half million pdfs I need to summarize. Very wide range of types: invoices, diagrams, contracts, emails, letters, pictures, schedules, notices, data sheets, manuals, more.

Which is… woof. Something else. I’ve been trying for many hours now to figure out a service/combination thereof that can get me there, but I’m seriously struggling. The ideal solution would be to throw the pdfs in and have it return a csv with dates and summaries, maybe parsed out email heading info.

I’m currently running these pdfs through Acrobat OCR now, which its own special hell.

I’ve tried myriad local and webhosted solutions. The BEST results in what is almost the perfect system for this I found on https://docalysis.com/. Good text results, works in batches, BUT I can only upload a single document at a time. They have a service to do batch processing and so I’m waiting to hear from them now. I imagine at the scale I need it’s expensive.

I also got this solution working: https://github.com/mayooear/gpt4-pdf-chatbot-langchain. Seemed solid, I was able to upload a thousand pdfs in a single go, but it would keep returning information from only 2-3 documents. Upload 5? Results for 2-3. Upload a thousand? Results for 2-3. My uneducated guess is that it’s hitting the OpenAI API token limit, but maybe not?

I know it’s possible, just not whether it’s feasible for an end user.