Well check out gpt4all.

There are a number of different models to download and run.

I think your real challenge is to find the right model.

Do you know any programming languages?

Gpt4all is easy to use

Raspberry Pi

This is a famously high-compute problem and you want to chuck it on a pi? Most llm models require a good GPU.

I think you’ll struggle with the coherent part

Most LLMs can do a few paragraphs and stay on topic but after thet they need better guidance, usually by changing the prompt to stay relevant. 10k+ words can be hard for normal authors to stay coherent on a single prompt, let alone a GPT-3 model

How much time do you have? Because even small models will take alot of time on that kind of hardware to spit out a long text…
And the small models arent that great. I think the current best and economic model would be a mistral, mixtral or dolphin.
If you got the power, nous-capybara is very good and “only” 34B parameters (loading alone needs like 40GB of memory).

1. you need a gpu or gpu resources to run a GPT level model

2. stop thinking of it as “make me a story” and start thinking of it as “let’s make a story together”

First talk back and forth over the basic idea.
Then get a short outline.
Then work on each chapter of the outline.
Then repeat and refine.

Try to keep overall queries under a certain character limit. Google what some good ranges are.

Make sure to save each conversation, and use the outline and previous chapter as the start of your conversation for making the next chapter.

The AI won’t write it for you, but it can be an amazing accelerator if you’re willing to put in the work.

If you want something local and open source, I think your main problem will be the number of parameters (the b thing). ChatGPT-3 is (was?) noticeably big and open source models are usually smaller. There is, of course, an exchange about how much the size of the model matters and how the quality of the training data affects the results. But when I did a non-scientific comparison ~half a year ago, there was a noticeable difference between smaller models and bigger ones.

Having said all of that, check out https://huggingface.co/ it aims to be like GitHub for AIs. Most of the models are more or less open source, you will only need to figure out how to run one and if you have some bottlenecks on PI

I use koboldAI

I use machine learning/ai pretty much daily and I run stuff at home locally when I do it. What you’re asking is possible, but might require some experimentation on your side, and you might have to really consider what’s important in your project because there will be some serious trade-offs.

If you’re adamant about running locally on a Rasberry Pi, then you’ll want a RPi 4 or 5, preferably an RPi 5. You’ll also want as much RAM as you can get (I think 8gb is the current max). You’re not going to have much VRAM since RPi’s don’t have a dedicated graphics card, so you’ll have to use it’s CPU and normal RAM to do the work. This will be a slow process, but if you don’t mind waiting a couple minutes per paragraph of text, then it may work for your use case. Because of the limited memory of Pis in general you’ll want to limit what size LLM models you use. Something specialized like a 7b story telling LLM, or a really good general purpose model like Mistral Open Orca 7b is a good place to start. You aren’t going to be able to run much larger models than that, however, and that could be a bit creatively limiting. As good as I think Mistral Open Orca 7b is, it lacks a lot of content that would make it interesting as a story teller.

Alternatively, you could run your LLM on a desktop and then use an RPi to connect to it over a local network. If you’ve got a decent graphics card with like 24gb of VRAM you could run a 30b model locally, and get decent results fairly fast.

As for the 10k words prompt, that’s going to be tricky. Most LLMs have a certain number of tokens they can spit out before they have to start up again. I think some of the 30b models I use have a context length of 4096 tokens… so no matter what you do you’ll have to tell your LLM to do multiple jobs.

Personally, I’d use LM Studio (not open source) to see if the results you get from running locally are acceptable. If you decide that its not performing as well as you had hoped, LM studio also generates python code so you could send commands to an LLM on a local network.

Perhaps if you broke your story up into chapters, each of those would be a more digestible chunk for the LLM?

Is this something that a TPU module could accelerate?

Install ollama and create an alias for ollama run for ease of access is what I did