Queue Fairness: Who Gets the Next Worker?

00:00:02 — Maya
Welcome to Development Radio. I'm Maya, and Alex is here with me. Today we're talking about queue fairness, through a very ordinary shop task: changing a product photo.

00:00:13 — Alex
I like a task with a visible finish. Same product, nicer picture. You can stand back and admire your contribution.

00:00:21 — Maya
Would you take the picture yourself?

00:00:23 — Alex
I would confidently offer. Then somebody would point out that you can see me reflected in the kettle.

00:00:28 — Maya
And now the kettle comes with a tiny Alex.

00:00:31 — Alex
At no extra charge. I'd probably stick to choosing between the photographs somebody else took.

00:00:36 — Maya
That can still take a while. I want the one where the colour looks right. Then I notice the background. Then I'm comparing two pictures that everybody else thinks are identical.

00:00:46 — Alex
Are they identical?

00:00:48 — Maya
One has a slightly warmer handle. I am prepared to care about that.

00:00:52 — Alex
I think that's fair. It's a shop window, even if the window is a screen. Somebody has to choose what people see.

00:00:58 — Maya
Yes. And there's something satisfying about finally choosing. One photograph, one little improvement, and you can get on with the rest of the day. I'd send you the page, too. Not the picture on its own. I want you to see how the whole thing looks together.

00:01:13 — Alex
With me safely cropped out of the kettle.

00:01:16 — Maya
Let's build a shop where the editor has chosen that photograph. Its processing job is next, but all four workers are importing the catalogue. Here's our question: how do we get that photo a worker without needlessly holding up the import?

00:01:30 — Alex
We need a little room between choosing the picture and seeing it on the page. The service prepares the image before the page can show it. A worker does that processing. We have four workers, and each can do one job at a time.

00:01:43 — Maya
The catalogue import needs those workers too. The shop is adding a collection of products, and all their pictures need preparing. That's where our four workers are now. Each is in the middle of a catalogue job.

00:01:55 — Alex
Those jobs can run for tens of seconds. Once one starts, this service can't pause it to process something else. The worker finishes that job before taking another.

00:02:05 — Maya
And the editor has just sent her kettle picture. I'd already put edits ahead of waiting catalogue work. She's there, looking at the screen. The catalogue can keep moving in the background.

00:02:15 — Alex
That's the priority rule our dispatcher uses. When a worker becomes available, it takes a waiting edit before another catalogue job. But right now, every worker is still occupied.

00:02:25 — Maya
So there's nobody to hand the photo to. We shouldn't tell her we've fixed the wait just because we've moved her to the front.

00:02:32 — Alex
Being next chooses who gets a free worker. It doesn't make a worker free.

00:02:36 — Maya
She'd have every reason to be impatient. We gave the last free worker to the catalogue before she arrived. How would you limit that without stopping the import altogether?

00:02:46 — Alex
There's a useful passage in the AWS Builders' Library article, Avoiding insurmountable queue backlogs. David Yanacek writes that their asynchronous systems “prevent one workload from using more than its fair share of threads.” He's talking about protecting the resources doing the work.

00:03:03 — Maya
I like that it says threads. We'd been looking at the order of the waiting jobs. He's asking how much running work one workload gets to hold. But what's a fair share for our shop? Both jobs belong to it.

00:03:16 — Alex
The article doesn't choose that split for us. We could let the catalogue occupy at most three of our four workers. That leaves capacity for editing, while three workers still make progress on the import.

00:03:27 — Maya
That sounds useful for the next arrival. It doesn't release any of the four jobs already running, though. Our editor is still waiting for the kettle picture.

00:03:35 — Alex
She is. Let's finish that moment before we change the rule. One of the catalogue jobs finishes. Its worker becomes free. The dispatcher sees the waiting edit and gives that worker the kettle photo.

00:03:47 — Maya
So the priority rule helps now. She gets that worker before the next catalogue job. She doesn't have to wait for the whole import to finish.

00:03:55 — Alex
Her photo is processed, and the new picture appears on the page. That's the end of this wait. The catalogue is still being imported by the other workers.

00:04:03 — Maya
Good. Now she can see whether the picture works with the page. That's the satisfying part. I'd probably send you the link at this point and ask whether I'm imagining the colour difference.

00:04:14 — Alex
Now we're choosing the rule for the next import. Three workers can do catalogue work. We keep the fourth available for an editor's photo. Nobody is editing at the moment, so that fourth worker stays empty.

00:04:25 — Maya
And there's still a pile of catalogue pictures waiting. I want to defend using that worker, because this is the same shop. The editor wants the kettle photo, but she also wants the rest of her products ready to sell.

00:04:37 — Alex
Yes. Getting the catalogue finished is useful to her too. What's the case for lending the worker while she's choosing the next picture?

00:04:45 — Maya
She might spend several minutes on the crop. During that time, the fourth worker could finish more of her catalogue. If we never let it help, we're giving up that progress even when she sends us nothing.

00:04:56 — Alex
I would notice that empty worker too. But when we say we're lending it, when does the loan end?

00:05:01 — Maya
When the catalogue job finishes. We can't call it back in the middle. So the question is whether the editor can live with the wait if she comes back before it does.

00:05:11 — Alex
Let's have her come back just after we lend it. Three workers were already busy with the catalogue. The fourth has just started another long catalogue job. Now she sends the next photo.

00:05:21 — Maya
She's next again. All four workers are occupied again. Giving her priority was never going to recall that job. We knowingly traded the empty slot for more catalogue progress.

00:05:33 — Alex
And she waits until one of those jobs finishes. It doesn't have to be the borrowed worker. But we need a reason to expect some worker back soon enough, if we're going to sell this as a quick editing screen.

00:05:43 — Maya
What would count as a reason? Say most catalogue jobs finish quickly. Would you lend on that basis?

00:05:50 — Alex
It would help me understand the usual experience. It wouldn't tell me how long these running jobs could keep her waiting. A quick average can include jobs that take much longer.

00:05:59 — Maya
Then I need to decide what I'm promising before I decide whether that evidence is enough. If we accept an occasional longer pause, that's one product. If the editing screen has to stay responsive during an import, that's another.

00:06:12 — Alex
And the pause doesn't end when a worker becomes free. Her photo still needs processing. We can't spend the whole acceptable pause waiting to start.

00:06:21 — Maya
For this shop, I'd keep the fourth worker available. We want someone to adjust a picture, look at the result, and try again. A catalogue job that takes tens of seconds can break that little rhythm before her photo even starts.

00:06:34 — Alex
What would you tell her when she sees the catalogue still importing, and one of our workers isn't doing anything?

00:06:39 — Maya
That we've kept room for her next edit. The other three workers are still preparing her catalogue. It may finish later because we've chosen to keep that room. I wouldn't pretend the empty worker was free of cost.

00:06:50 — Alex
She could reasonably say the catalogue matters more today. She might be happy to leave the editor alone and come back when the import is done.

00:06:58 — Maya
She could. If that's the experience we're offering, lending can make sense. But our editing screen currently invites her to carry on while the import runs. I'd rather support that invitation than quietly change the bargain halfway through.

00:07:12 — Alex
I'd keep the capacity too, for that version of the product. And I'd be careful about what it promises. Another edit could already be using the protected worker. Keeping bulk off that slot doesn't make every possible editing load instant.

00:07:25 — Maya
It removes the particular wait we've been following: the catalogue occupying every worker. If the next photo arrives while the protected worker is empty, that worker can take it. We still have to do the image processing.

00:07:38 — Alex
And during the next quiet gap, the worker stays empty. The catalogue keeps moving on three. That's the cost we have to keep accepting, not just the moment when an editor arrives and the choice looks good.

00:07:50 — Maya
I find the quiet gap harder to defend than the busy moment. You can point to the catalogue pictures we could have finished. You can't point to the next edit until it arrives.

00:08:00 — Alex
No. You'd still be comparing those two photographs. From the worker's side, there's nothing to do. From your side, you're halfway through choosing.

00:08:08 — Maya
Right. She cares about seeing the picture on the page. The worker starting is only our partway marker. I'd want to try the edit at this awkward moment, just after the loan begins, and follow it all the way through.

00:08:21 — Alex
So would I. One quick result wouldn't settle it. We could have happened to arrive just before one of the other catalogue jobs finished. We'd want to know which running job released the worker, and whether that return was something we could rely on.

00:08:34 — Maya
So I'd leave borrowing off for now, and use that test to challenge the choice. Show me the editor can get the picture back within the pause we're willing to accept, including an awkward arrival. Then I'd have a reason to give the catalogue that spare capacity.

00:08:49 — Alex
And if the result is an occasional longer pause, we can decide whether to accept it. We have to say that plainly. The label high priority doesn't make that choice for us.

00:08:59 — Maya
For your own service, try one edit just after bulk work takes the last available worker. Follow it from submission to the visible result. Compare that pause with what you want the person at the screen to experience before you turn borrowing on.

00:09:14 — Alex
Back at our shop, the editor has her photo, and the catalogue is still making progress. Next time we keep a worker available, we know what we're buying with the idle time.

00:09:24 — Maya
A chance to carry on with the page. Which I would still like you to look at before I send it to anyone else.

00:09:30 — Alex
I'll check the kettle for reflections.