Paid AI Calls With TaskFused, One Model Call Billed When Three Parts of an App Ask

A user uploads a 40-page contract and the app wants a summary. The browser asks for it when the page opens. The search indexer asks for it a few seconds later, because summaries make good snippets. And the user, impatient, hits Refresh. Three requests, one document, and each one would send the whole contract to a language model and pay for every token of it, three times.

AI calls are the clearest case for doing work once. They’re slow, often ten or twenty seconds for a long document, so duplicates overlap a lot. They’re billed per token. And the answer for the same document, model and prompt doesn’t need to be made twice.

The Name Is the Prompt

The key has to change whenever the answer could change. That means the document, the model and the prompt template all belong in it. A content hash of the document is a good start, so a re-uploaded copy of the same file still shares. Keep the input to what defines the answer, too: an upload ID would differ between two uploads of the same file, and the second would be refused as a conflict.

key := fmt.Sprintf("summary/%s/%s/%s", docSHA256, "model-2026-09", "prompt-v4")
input, _ := json.Marshal(SummaryJob{DocSHA256: docSHA256, MaxWords: 200})

sub, err := store.Submit(ctx, taskfused.Request{
    Namespace: "ai", Handler: "summarize", Version: "v1",
    Key: key, Input: input, JoinExisting: true,
    MaxAttempts: 4, RetryDelay: 5 * time.Second,
})

Change the prompt and bump prompt-v4 to prompt-v5, and every document gets a fresh summary the next time it’s asked for. The old ones stay stored under their old names until you prune them.

Rate Limits and Retries

AI providers answer busy periods with “too many requests”. The handler returns that as an error, and TaskFused schedules another attempt, waiting 5 seconds, then 10, then 20 in the example above. Every caller has to submit the same retry settings, because they’re part of what the operation is, and a different setting under the same key is refused with ErrPolicyConflict. Keep them in one place in your code. Everyone waiting keeps waiting on the same operation, so a burst of rate limits produces a few spaced retries for one call, instead of three callers each hammering the API on their own.

If all four attempts fail, every waiter sees the operation as failed with the provider’s last error. Once the provider is healthy again, taskfused retry reopens it with a few more attempts, and the original callers’ next request picks up the result.

What It Saves

The saving is whatever share of your calls are duplicates. For a page that loads a summary on open and on every refresh, that share can be large. For a chat where every message is new, it’s close to zero, and TaskFused won’t help there. A quick way to find out is to log the key you’d use for a week and count how many requests repeat one.

There’s a second saving that’s easy to miss. When the same summary is wanted again tomorrow, a joined request finds the stored result at once. The second user doesn’t wait twenty seconds, and nobody pays for the call again.

The Honest Part

TaskFused runs each operation at least once. If the worker crashes after the provider has answered but before the result is saved, another worker runs the call again, and you pay for it twice. That’s rare, and much cheaper than paying for every duplicate request, but it isn’t zero. Where a provider accepts an idempotency key, pass the operation ID (claim.Operation.ID) so a repeat after a crash can be recognised on their side.

Results are stored up to 1 MiB each, which fits any summary or answer. Keep the full prompt and response logs in your own storage if you need them for audits; TaskFused keeps the result and the history of attempts.

See the reference for the retry settings, and how TaskFused was tested for what happens when a worker dies mid-call.