Video Rendering Through a Deploy With TaskFused, the Job Moves to a New Worker

A video app turns each upload into a 1080p preview. A render takes about four minutes of CPU. Deploys happen several times a day, and every deploy restarts the render workers. With a simple job runner, a render that was three minutes in when the deploy started is just gone. The user’s preview never appears, or someone has to notice and start it again.

The same render is also asked for more than once. The uploader’s browser asks for the preview, the share page asks for it when the link is opened, and the mobile app asks when it syncs. They all want the same file.

How the Job Moves

Each attempt is held on a lease. The worker renews it every third of the lease while the handler runs, so with a 60-second lease it checks in every 20 seconds. When a deploy stops the worker, the handler’s context is canceled, so the render stops, and the worker stops renewing. That doesn’t cancel or fail the operation. It stays running until its lease runs out, and then the first worker of the new release that asks for preview-1080p v2 work takes it over under the same operation ID.

worker, err := taskfused.NewWorker(store, taskfused.WorkerOptions{
    Namespace: "media", Concurrency: 2, LeaseDuration: 60 * time.Second,
    Handlers: []taskfused.Registration{{
        Name: "preview-1080p", Version: "v2",
        Handle: func(ctx context.Context, c taskfused.Claim) ([]byte, error) {
            out := "previews/" + c.Operation.ID + ".mp4"
            if err := render(ctx, c.Operation.Input, out+".part"); err != nil {
                return nil, err
            }
            return []byte(out), os.Rename(out+".part", out)
        },
    }},
})

The new attempt starts the render again from the beginning. TaskFused moves the job, and the render starts over. Writing to a .part file and renaming it at the end means a reader never finds a half-written preview, and the second attempt simply overwrites the first one’s leftovers.

If the old worker somehow comes back and tries to save its own result, it’s turned away: it holds an older token than the attempt that replaced it.

Waiting Through the Deploy

The three callers aren’t attached to the worker. They wait on the operation in the SQLite file, from whatever process they’re in, so the deploy doesn’t interrupt them. Each one gets the preview path when the second attempt finishes, about four minutes plus the lease later. The stored result then answers the next request for that video straight away.

Every interrupted attempt counts against the operation’s attempts, three by default. If deploys can land several times during one render, submit renders with a higher MaxAttempts, say 10, so a busy deploy day doesn’t fail them.

The version in the name matters during a deploy too. A job is only taken over by a worker that registers the same handler and version, so the new release has to keep registering v2 for as long as old v2 renders might be unfinished. If the new release changes how previews are rendered, register the new code as v3 beside it. New requests make v3 renders, old ones finish as v2, and once no v2 work is left, the v2 handler can go.

Stopping a Render on Purpose

If the uploader deletes the video halfway through, cancel the operation:

taskfused cancel -db media.db -namespace media -id op_...

The worker learns at its next lease renewal, within 20 seconds here, and the render’s context is canceled, so render should check it and stop. The operation ends as canceled, and no later result can overwrite that.

Choosing the Lease

A shorter lease means a quicker takeover after a crash, and more renewal writes while things are healthy. A longer one means a longer wait after a crash. For four-minute jobs, 60 seconds is a fair middle: a stopped render starts again on a new worker about a minute later, and a healthy one renews three times a minute. The tested recovery in how TaskFused was tested used a one-second lease and finished about 1.1 seconds after the kill; with a 60-second lease, expect about a minute.

All workers share one machine and one SQLite file in version 0.1, so this pattern suits one render box whose worker processes restart. A fleet of separate machines is beyond what 0.1 does.