Why we are training our own models
Everything Nostyss generates today comes from a general-purpose model behind an API. That was the only sensible way to start, and it is not where we intend to stay. This is a progress note on work that is genuinely unfinished.
Writing it now rather than at the end is deliberate. Announcing a finished thing is marketing. Describing an unfinished one, including the parts that are not working, is the only version of this that is worth reading, and it is the version we would want from someone else.
What a general model is optimised for, and why it is not this
A general-purpose model is trained to be helpful, safe, and broadly acceptable across every task anyone might bring it. Those are the right goals for an assistant. They are a strange fit for fiction.
The tendencies show up quickly once you read enough generated chapters. It reaches for the tidy resolution. It explains what it just showed you. It smooths conflict rather than letting it sit. It has a house style that surfaces no matter which voice you asked for, and after forty chapters that house style is the only voice left in the book.
Prompting can ask a model to stop doing something. It cannot make not doing it the model's default.
That gap is the whole argument. A prompt is a request made at generation time against a lifetime of training that says otherwise, and it holds for a few chapters before the training wins. Changing the default is a training problem, not a prompting one.
The other reason: you do not own an API
There is a less romantic argument, and it is the one that actually forces the schedule.
Models are retired. The model this product was built on has a published end-of-life, and a date on a vendor's page becomes a migration on your calendar whether or not you had plans that quarter. Everything downstream moves with it: the writing changes, so every quality measurement taken before the switch describes a system that no longer exists, and the comparison you were carefully running has to start again.
That is survivable once. It is a bad foundation for a product whose entire promise is that it behaves consistently over long stretches. A model we host does not get deprecated out from under us, and its cost per chapter is a number we control rather than one we receive.
What is actually hard
Three things, in ascending order of how much they have cost us.
The data is most of the work. Not gathering it — deciding what a training example even is. A novel is far longer than anything you train on directly, so it has to be cut, and where you cut and what context travels with each piece changes the result more than most choices about the model itself. Nearly all the effort here is upstream of anything that looks like machine learning.
Evaluation is harder than training. Standard metrics for text generation compare the output to a reference answer and score how close it landed. Fiction has no reference answer. There is no correct next chapter, so a metric that rewards proximity to one specific chapter is measuring conformity, which is close to the opposite of what we want. Perplexity tells you the model is confident, not that the chapter is good. Building an honest way to say this run is better than that one has taken more of our time than getting runs to finish.
Knowing when to stop. With no clean metric, it is easy to keep tuning toward whatever your eye happens to like that week. Guarding against that means fixing what you measure before you look at the results, which is unglamorous and is the difference between improvement and drift.
Where it stands
Unfinished, and we are not going to put a date on it. The honest status is that the training side runs and the measurement side is what gates it: we can produce candidate models faster than we can responsibly decide whether they are better, and shipping one we cannot evaluate would trade a known quantity for an unknown one on a hunch.
Nothing you read on Nostyss today comes from a model we trained. When that changes, it will be because we could show it was better, not because it was ours.
Consistency over long stretches is the whole subject of this blog; the place to start is why AI stories forget.
Questions
Why fine-tune a model instead of prompting it?
Because a prompt is a request made against everything the model was trained to do, and it holds for a few chapters before the training wins. Changing what a model does by default, such as reaching for tidy resolutions or a house style, is a training problem.
Does Nostyss use its own model today?
No. Everything Nostyss generates today comes from a general-purpose model behind an API. Our own models are in training, and one will replace it only when we can show that it writes better.
Why is it so hard to evaluate a fiction model?
Standard text metrics compare the output with a reference answer, and fiction has none: there is no correct next chapter. A metric that rewards closeness to one chapter measures conformity, not quality, so an honest evaluation has to be built for the purpose.
See whether it actually remembers
Ten worlds you can start without an account. Play a few chapters, contradict something you established earlier, and watch what the world does about it. That test is the honest way to judge any of this.
Choose a world