Steady-State Behavior

Long-Run Behavior

A key question for Markov chains: What happens in the long run? Does the system settle into a steady state?

Intuition

After many transitions, the probability of being in each state may converge to fixed values, regardless of the starting state.

Steady-State Distribution

A steady-state distribution (also called stationary distribution) $\pi$ satisfies:

$$\pi_j = \sum_i \pi_i \cdot p_{ij}$$

In matrix form: $\pi = \pi P$

Interpretation

If the system starts in the steady-state distribution, it stays in that distribution forever. The probabilities don’t change over time.

Balance Equations

For a steady state, the “flow” into each state must equal the “flow” out:

$$\sum_{i \neq j} \pi_i \cdot p_{ij} = \pi_j \sum_{k \neq j} p_{jk}$$

Intuition

• Left side: Total probability flowing INTO state $j$ from all other states
• Right side: Total probability flowing OUT of state $j$ to all other states

In steady state, these must balance.

Finding the Steady State

Method

1. Write the balance equations
2. Add the normalization condition: $\sum_j \pi_j = 1$
3. Solve the system of linear equations

Example: Two-State Chain

States: 0 and 1. Transition probabilities:

• $p_{01} = a$ (probability of going from 0 to 1)
• $p_{10} = b$ (probability of going from 1 to 0)

Balance equations:

• $\pi_0 \cdot a = \pi_1 \cdot b$ (flow from 0→1 equals flow from 1→0)
• $\pi_0 + \pi_1 = 1$ (normalization)

Solution:

$$\pi_0 = \frac{b}{a + b}, \quad \pi_1 = \frac{a}{a + b}$$

Convergence to Steady State

Conditions

For convergence to a unique steady state:

1. Irreducible: Can reach any state from any other state
2. Aperiodic: Don’t get stuck in cycles

Result

If the chain is irreducible and aperiodic, then:

$$\lim_{n \to \infty} P(X_n = j | X_0 = i) = \pi_j$$

Regardless of the starting state $i$, the long-run probability of being in state $j$ is $\pi_j$.

Example: Queue with Finite Capacity

For a supermarket queue with capacity 10:

• Arrivals with probability $p$
• Departures with probability $q$ (when non-empty)

The steady-state distribution tells us:

• Long-run probability of having $k$ customers in queue
• Average queue length: $E[\text{queue}] = \sum_k k \cdot \pi_k$
• Probability of full queue (customers turned away)