TL;DR: This is the queueing model perspective of the “paper pool” conference reviewing model with math and numbers based on Little’s Law. Think of it as supplementary material to the post on David’s blog on current ML/AI conferences; you might want to read that one first.

Written by Sebastian Pokutta in collaboration with David Martínez-Rubio.

Little’s law for queueing systems

Little’s law states that the long-run average number $L$ of jobs in a stable system equals the arrival/throughput rate $\lambda$ times the average time $W$ a job spends in the system:

\[\tag{Little'sLaw} L = \lambda W.\]

Interpretation.

  • $L$: average work-in-progress (jobs “in the system”)
  • $\lambda$: average completion rate for the flow considered (arrivals = departures in steady state)
  • $W$: average time in system per job

Assumptions. The assumptions are minimal: stability (finite averages, arrivals balance departures), clearly defined entry/exit points, and stationarity/ergodicity. Crucially, Little’s law is insensitive to the arrival/service distributions and scheduling discipline, and it also holds for stable subflows (e.g.,when splitting into different classes of jobs).

Note. We stick to the notion of hazard as the per-step completion (conditional) probability as customary in queueing theory.

In our setting, the “system” is the pool of papers under review across calls; arrivals are new submissions per call; exits are final decisions (accept or abandonment). We model each conference call as one time step and authors may resubmit their papers to the next call if rejected, i.e., these papers effectively stay in the system. If $N$ new submissions arrive per call and each paper has an independent per-call accept/complete hazard $p$, then Little’s law gives:

No giving up (infinite patience). In this case, each paper is accepted if it is completed, which happens at some point in time. Given hazard $p$, we have the mean time in system as $W = \frac{1}{p}$ and the expected pool level scales as $L = N/p$:

\[W=\frac{1}{p}\ \text{calls},\qquad L=\frac{N}{p},\qquad \text{accepted per call}=\lambda=N.\]

Key insight: the throughput (= number of accepted papers per call) does not change with $p$. Lowering $p$ just increases the in-system population $L$ (and thus reviewer load).

Finite patience with papers being abandoned after $T$ calls. This is the case where papers are abandoned after $T$ calls if not accepted. We still have only one “class” of papers, but now there are two exit modes: accept or abandon.

\[W_{\text{all}}=\frac{1-(1-p)^T}{p},\quad L_{\text{all}}=N\frac{1-(1-p)^T}{p},\quad \lambda_{\text{acc}}=N\big[1-(1-p)^T\big],\quad \lambda_{\text{abn}}=N(1-p)^T.\] \[W_{\text{acc}}=\frac{1}{p}-\frac{T(1-p)^T}{1-(1-p)^T},\quad L_{\text{acc}}=N\!\left(\frac{1-(1-p)^T}{p}-T(1-p)^T\right),\quad L_{\text{abn}}=N\,T(1-p)^T.\]

Key insight: acceptances rise only modestly with $p$, but backlog and review volume drop sharply. This model is more realistic but still has only one class of papers.

Quality buckets: class-dependent hazards and patience. When submissions split into quality classes (e.g., “great/average/bad”) with different per-call completion hazards $p_i$ and potentially different patience limits $T_i$, Little’s law applies to each class separately:

\[W_i=\frac{1-(1-p_i)^{T_i}}{p_i},\quad L_i=\lambda_i W_i,\quad \lambda^{\text{acc}}_i=\lambda_i\big[1-(1-p_i)^{T_i}\big],\]

where $\lambda_i = N\pi_i$ is the arrival rate for class $i$ with share $\pi_i$. The backlog and acceptance shares become

\[\text{BacklogShare}_i=\frac{L_i}{\sum_j L_j},\quad \text{AcceptShare}_i=\frac{\lambda^{\text{acc}}_i}{\sum_j \lambda^{\text{acc}}_j}.\]

Key insight: reweighting hazards across classes mainly shifts the composition of acceptances and backlog, while total throughput remains roughly fixed at $N$ (since $\sum_i \lambda_i = N$).

Slow growth in arrivals ($g$ multiplicative increase per call). We can immediately also estimate the impact of growth in the arrival rate. Over a window roughly equal to the mean time in system $W\approx 1/p$, a sustained arrival growth $g$ compounds into an approximate $g \cdot W \approx g/p$ increase in the pool level (rule-of-thumb).

Caveats, Assumptions, and Gotchas.

  • The hazard $p$ bundles many realities (desk reject, area-chair filtering, author revision time). If those change, so does $W$.
  • Little’s law reveals immediately that steady-state acceptances follow arrivals, not the per-call acceptance hazard.
  • If resubmissions between different venues happen with delays, one can model them as additional stages; Little’s law still applies stage-wise.
  • If arrivals are highly seasonal (e.g., co-located deadlines), replace the $g/p$ heuristic with the exact weighted-sum solution or simulate.

Little’s Law in Action

Below we visualize Little’s law with a simple funnel model, where papers arrive at the top and then go through the system, eventually being accepted or abandoned.

Details and Further Discussion

We will now discuss the above model in more detail and provide additional insights. At the very end we also present a proof of Little’s law.

Assumptions

We make the assumptions from above more precise.

Time unit: one conference call (one submission cycle).

Arrivals: each call injects $N$ new submissions, hence the arrival rate per call is $\lambda=N$.

Completion (“service”): each paper present in a call independently completes (i.e., gets a final accept/abandon decision) with hazard $p$ in that call.

  • “No giving up” = a paper continues until accepted;
  • “Finite patience” = a paper is abandoned after $T$ unsuccessful calls.

State (queue length): Let $L_t$ denote number of papers currently “in the system” (awaiting final disposition). Then the stock–flow equation is given by

\[L_{t+1}=(1-p)L_t+N,\]

which is exactly a discrete-time queue with batch arrivals $N$ and per-step departure probability $p$.

No-give-up world: geometric time in system

If authors never give up, each paper’s time in system is distributed as Geometric($p$) on ${1,2,\dots}$ with mean

\[W=\mathbb{E}[G]=\frac{1}{p}\quad \text{(calls)}.\]

With $\lambda=N$ per call, Little’s law yields

\[L=\lambda W = N\cdot\frac{1}{p}=\frac{N}{p}.\]

As such, steady-state throughput (accepted papers per call) equals the arrival rate:

\[\text{accepted per call}=\lambda_{\text{acc}}=\lambda=N.\]

Finite patience $T$: truncated geometric & subflows

Let each paper give up after $T$ calls if not accepted. Now there are two exit modes: accept or abandon.

Probability of eventual acceptance. Let us first compute the probability that a paper is eventually accepted (within $T$ calls):

\[q_{\text{acc}}=1-(1-p)^T.\]

All-exits. Next we compute the time in system irrespective of the exit mode (accept or abandon):

\[W_{\text{all}}=\mathbb{E}[\min(G,T)] =\sum_{k=1}^{T}\Pr(G\ge k)=\frac{1-(1-p)^T}{p}.\]

Now we first apply Little’s law to the whole system (all exit modes), to obtain

\[L_{\text{all}}=\lambda\,W_{\text{all}} =N\cdot\frac{1-(1-p)^T}{p}.\]

And next, we apply Little’s law to the subflows arising from the exit modes accept and abandon:

Accepted subflow.

Throughput is given by

\[\lambda_{\text{acc}}=N\,q_{\text{acc}}=N\big[1-(1-p)^T\big].\]

Conditional mean time given acceptance is given by

\[W_{\text{acc}}=\mathbb{E}[G\mid G\le T] =\frac{1}{p}-\frac{T(1-p)^T}{1-(1-p)^T}.\]

Population of papers currently in-system that are destined to be accepted:

\[L_{\text{acc}}=\lambda_{\text{acc}}\,W_{\text{acc}} =N\!\left(\frac{1-(1-p)^T}{p}-T(1-p)^T\right).\]

Abandonment subflow.

Throughput is given by

\[\lambda_{\text{abn}}=N(1-p)^T.\]

Time in system is given by

\[W_{\text{abn}}=T.\]

Population of papers currently in-system that are destined to be abandoned:

\[L_{\text{abn}}=\lambda_{\text{abn}}\,W_{\text{abn}}=N\,T(1-p)^T.\]

Check. $L_{\text{acc}}+L_{\text{abn}}=L_{\text{all}}$.

Observation. Increasing $p$ modestly boosts $\lambda_{\text{acc}}$ (since $1-(1-p)^T$ saturates), but it significantly lowers $W_{\text{all}}$ and hence $L_{\text{all}}$: far fewer papers sitting in the pool at any time.

Quality buckets: class-dependent hazards and patience

If submissions are split into classes (e.g., “great/average/bad”) and decisions prioritize higher quality, you have class-dependent hazards $p_i$ and potentially class-dependent patience $T_i$. Let $\pi_i$ denote the arrival share of class $i$, so per call

\[\lambda_i = N\,\pi_i \quad (\text{arrivals of class } i).\]

Class-wise Little’s law. For each stable class $i$,

\[L_i = \lambda_i\,W_i,\]

where $W_i$ is the mean time in system for class $i$.

Finite patience (class $i$). If class $i$ abandons after $T_i$ calls with per-call accept hazard $p_i$, then

\[W^{\text{all}}_{i}=\frac{1-(1-p_i)^{T_i}}{p_i},\qquad L^{\text{all}}_{i}=\lambda_i\,W^{\text{all}}_{i}=\lambda_i\,\frac{1-(1-p_i)^{T_i}}{p_i}.\]

Accepted throughput (class $i$). The per-call accepted rate is

\[\lambda^{\text{acc}}_{i}=\lambda_i\Big(1-(1-p_i)^{T_i}\Big).\]

Mixes (shares). The backlog and accepted shares of class $i$ are

\[\text{BacklogShare}_i=\frac{L^{\text{all}}_i}{\sum_j L^{\text{all}}_j} =\frac{\lambda_i\frac{1-(1-p_i)^{T_i}}{p_i}}{\sum_j \lambda_j\frac{1-(1-p_j)^{T_j}}{p_j}},\]

and

\[\text{AcceptShare}_i=\frac{\lambda^{\text{acc}}_{i}}{\sum_j \lambda^{\text{acc}}_{j}} =\frac{\lambda_i\left[1-(1-p_i)^{T_i}\right]}{\sum_j \lambda_j\left[1-(1-p_j)^{T_j}\right]}.\]

Special case (equal hazard). If $p_i\equiv p$ for all classes, then for two classes $g, b$ with $\pi_g+\pi_b=1$,

\[\text{AcceptShare}_g =\frac{\pi_g\left[1-(1-p)^{T_g}\right]} {\pi_g\left[1-(1-p)^{T_g}\right]+\pi_b\left[1-(1-p)^{T_b}\right]},\]

and $\text{BacklogShare}_g = \text{AcceptShare}_g$.

Interpretation. Because $\sum_i \lambda_i = N$ per call is fixed, reweighting hazards mainly shifts the composition of acceptances; total throughput changes little, while total backlog still scales with the average hazard via $W$.

What a small arrival growth $g$ does to the pool

Little’s law says the instantaneous pool level tracks a windowed average of recent arrivals with window length $\approx W$ (see proof below). Hence a sustained per-call growth $g$ in arrivals translates into an approximate level increase of

\[\Delta L \;\approx\; (g \cdot \text{current arrivals}) \cdot W \;\approx\; (gN)\cdot \frac{1}{p},\]

thus as a rule-of-thumb we obtain:

\[\frac{\Delta L}{L}\;\approx\; \frac{g}{p}\ \text{(over }\approx W\text{ calls)}.\]

Note. For precise dynamics with time-varying arrivals $N_t$, the exact solution is a geometrically weighted sum of recent $N_t$’s.

Converting to yearly units

We considered calls as the time unit. However, we can easily convert/rescale to yearly units. If the field has $c$ calls/year:

  • Mean time in system:
    • $W_{\text{years}}=\dfrac{1}{pc}$ for the case of no-give-up
    • $W_{\text{all,years}}=\dfrac{1-(1-p)^T}{pc}$ for the case of finite $T$.
  • Backlog level in
    • papers: multiply $L$ above by $1$ (dimensionless)
    • paper-years: multiply by $1/c$.

A short proof of Little’s law

Let $Q(t)$ be the number of items in the system at time $t$. Let $a_k$ and $d_k$ be the arrival times and departure times of the $k$-th job respectively, and $W_k \doteq d_k-a_k$ its sojourn times (time in system). We assume that the system is stable/stationary with long-run arrival rate $\lambda$ and finite mean sojourn time

\[\tag{meanSojournTime} W \doteq \mathbb E[W_k] < \infty.\]

Let

  • $A(T) \doteq \abs{\set{k: a_k\le T}}$ denote arrivals by $T$ and,
  • $D(T) \doteq \abs{\set{k: d_k\le T}}$ denote departures by $T$.

We further assume rate stability

\[\tag{rateStability} \lim_{T\to\infty}\tfrac{A(T)}{T}=\lim_{T\to\infty}\tfrac{D(T)}{T}=\lambda.\]

Area–interval identity. We start with a simple observation. Given any time $t$, let $Q(t) \doteq\sum_k \mathbf 1_{\set{a_k \le t < d_k}}$ denote the number of jobs in the system at time $t$. We can now integrate and swap sum and integral to obtain:

\[\int_0^T Q(t)\,dt = \sum_k \int_0^T \mathbf 1_{\set{a_k \le t < d_k}} \,dt = \sum_k \abs{[0,T]\cap[a_k,d_k)}.\]

Therefore, the “area” under $Q(\cdot)$ over $[0,T]$ equals the sum of the lengths of the intersections of $[0,T]$ with the jobs’ sojourn intervals. We can now split the jobs into three disjoint groups:

  1. jobs that arrive and depart within $[0,T]$: contribute exactly $W_k$;
  2. jobs present at time $0$ (arrived before $0$): contribute at most their remaining times inside $[0,T]$;
  3. jobs still present at time $T$: contribute at most the elapsed times since arrival up to $T$.

Letting $S(T) \doteq \int_0^T Q(t)\,dt$, with the three groups we now have:

\[\sum_{k:\,0\le a_k<d_k\le T} W_k \;\le\; S(T) \;\le\; \sum_{k:\,a_k\le T} W_k \;+\; B_0 \;+\; B_T,\]

where $B_0$ is the total residual time inside $[0,T]$ of the finitely many jobs present at $t=0$, and $B_T$ is the total elapsed time inside $[0,T]$ of the finitely many jobs present at $t=T$. Note that both $B_0, B_T$ are finite constants, i.e., they do not grow with $T$.

Divide by $T$ and pass to limits. Similar to convergence proofs of, e.g., (stochastic) gradient descent, we now divide by $T$:

\[\frac{1}{T}\sum_{k:\,0\le a_k<d_k\le T} W_k \;\le\; \frac{S(T)}{T} \;\le\; \frac{1}{T}\sum_{k:\,a_k\le T} W_k \;+\; \frac{B_0+B_T}{T},\]

and then take the limit as $T\to\infty$. In particular, as $T\to\infty$ we have $\tfrac{B_0+B_T}{T}\to 0$. Note, this also measures how long it takes for the system to reach stability. Now for the sums:

The left sum counts departures within $[0,T]$. We rewrite it suggestively as

\[\frac{D(T)}{T}\cdot \frac{1}{D(T)}\sum_{k=1}^{D(T)} W_k \;\xrightarrow[T\to\infty]{}\; \lambda\cdot W,\]

by rate stability and the (ergodic/renewal-reward) law of large numbers for ${W_k}$.

The right sum counts arrivals by $T$ and it can similarly be rewritten as

\[\frac{A(T)}{T}\cdot \frac{1}{A(T)}\sum_{k=1}^{A(T)} W_k \;\xrightarrow[T\to\infty]{}\; \lambda\cdot W.\]

As such we obtain for $\lim_{T\to\infty}\frac{S(T)}{T}$ that

\[\lim_{T\to\infty}\frac{1}{T}\int_0^T Q(t)\,dt \;=\; \lambda\,W,\]

as it is sandwiched between the two limits from above.

Finally, observe that the left-hand side is the time average of the number of jobs in system:

\[L \doteq \lim_{T\to\infty} \frac{1}{T}\int_0^T Q(t)\,dt.\]

Therefore,

\[L = \lambda W.\]

A few remarks are in order.

Remarks.

  1. No assumptions on service-time distribution, number of servers, or service order were used. We only needed stability and finiteness of the relevant limits.
  2. The same argumentation can be applied to subflows (e.g., “accepted only”, or splitting into “great/average/bad” papers), yielding $L_i=\lambda_i W_i$ for each stable class $i$.