Reinforcement Learning Decisions Before You Buy

When building decision-making agents, especially in sales, trading, or resource allocation, you often face a critical question: **when should I act, sell, or stop?** This is the optimal stopping problem — a classic mathematical challenge that arises when uncertain outcomes must be evaluated against potential rewards.

In reinforcement learning (RL), this translates to deciding whether to continue exploring or exploit known options. The agent needs a decision playbook to avoid costly mistakes and maximize long-term value.

The Math Behind Optimal Stopping

The core idea is simple: you observe a sequence of values, but only one is optimal. You must decide when to stop observing and commit to the current best option. This is not just about "when to buy" — it's about **when to act** under uncertainty.

We can model this with a threshold strategy. For a sequence of n observations, the agent should observe the first k observations (exploration phase) and then select the next observation that exceeds the maximum of those k.

The optimal k is approximately n/e, where e ≈ 2.718. For n=100, k ≈ 37. This is known as the **Secretary Problem** — a classic in probability theory.

Let’s code it:

import numpy as np

def optimal_stopping_strategy(observations):
    n = len(observations)
    k = int(n / np.e)  # Exploration phase size
    
    if k == 0:
        return observations[0]
    
    # Find max in exploration phase
    exploration_max = max(observations[:k])
    
    # Look for first value exceeding exploration max
    for i in range(k, n):
        if observations[i] > exploration_max:
            return observations[i]
    
    # If not found, take last option
    return observations[-1]

# Example: 100 random observations
obs = np.random.rand(100)
decision = optimal_stopping_strategy(obs)
print(f"Decision value: {decision:.3f}")

This strategy ensures you're more likely to select the best available option without over-exploring.

Practical RL Use Cases

In RL, this logic applies to:

- **Sales agents**: When to offer a deal or wait for better conditions.

- **Trading bots**: When to sell an asset or hold.

- **Resource allocation**: When to commit a resource or stop observing.

You can integrate this into your agent’s policy by computing thresholds dynamically based on current state, reward estimates, and uncertainty levels.

FAQ

Q: How does this approach handle noisy data?

A: The threshold strategy is robust to noise. It's not about perfect prediction but selecting the best option within a sequence. If observations are noisy, it still selects the first value exceeding the exploration phase max — which inherently filters out early outliers.

Q: Can I adapt this for non-sequential data?

A: Yes, but you'll need to define a meaningful sequence. For example, in pricing decisions, you might iterate through price points or time windows, treating each as an observation. The key is ensuring your "sequence" reflects real-world decision points.

Q: What if I don't know the total number of observations?

A: In dynamic environments, you can use **adaptive thresholding**. Start with a small k, and increase it as more data arrives — but always ensure k ≤ n/2 to maintain effectiveness.

Get it

Ready to implement optimal stopping in your decision agents? Try our **Optimal-Stopping Decisions Playbook for Agents**, a build-once tool that gives you the math and code patterns to decide when to act, sell, or stop under uncertainty — without reinventing the wheel. Get it here.