Experiment: Building a Resilient Data Retrieval Agent with Circuit Breakers and Fallbacks

In a perfect world, every API call succeeds, every database responds instantly, and every data source is always available. In the real world, distributed systems are prone to transient failures, network latency, and service outages. For a business that relies on aggregating data from multiple sources, a single failing service can cause a cascade of errors, leading to a poor user experience and unreliable applications.

This is where building for resiliency isn't just a best practice—it's a necessity. Your Data Retrieval API is a critical piece of infrastructure. How do you ensure it remains robust even when its dependencies are not?

At Searches.do, we believe the answer lies in treating your data retrieval logic as first-class business logic, or Business-as-Code. By building an intelligent agentic search layer, you can encapsulate complex resiliency patterns, making them simple to manage and infinitely reusable.

Let's run an experiment. We'll build a data retrieval agent that starts fragile and progressively make it more resilient using two powerful patterns: fallbacks and circuit breakers.

The Scenario: The All-in-One Customer Profile Agent

Imagine we need a searchId called get-customer-dashboard. This agent's job is to create a unified customer profile by fetching data from three different microservices:

User Service: Provides core user information (name, email).
Billing Service: Provides the current subscription status.
Support Service: Provides a list of recent support tickets.

Our initial, optimistic implementation of the search agent handler might look like this:

// searches/get-customer-dashboard-v1.ts
// A fragile agent that assumes all services are always available.

import { createSearch } from '@searches.do/sdk';
import { userService, billingService, supportService } from '../lib/apis';

export default createSearch({
  id: 'get-customer-dashboard',
  // ...input parameter definitions
  handler: async ({ parameters }) => {
    // 1. Get core user data (critical)
    const user = await userService.findByEmail(parameters.email);
    if (!user) throw new Error('User not found');

    // 2. Get subscription status (important)
    const subscription = await billingService.getStatus(user.id);

    // 3. Get recent tickets (nice-to-have)
    const tickets = await supportService.getRecentTickets(user.id);

    return {
      ...user,
      subscriptionStatus: subscription.status,
      recentTickets: tickets,
    };
  },
});

When a user calls this search, the Searches.do platform executes this logic. But what happens if the supportService is down for maintenance? The entire request fails. The user gets an error, even though the critical user and billing information was successfully retrieved. We can do better.

Pattern 1: Adding a Fallback for Non-Critical Data

A simple way to increase resilience is to implement a fallback. Instead of letting a non-critical error terminate the entire process, we catch it and return a default or partial state. The support tickets are a "nice-to-have," so let's wrap that call in a try...catch block.

// searches/get-customer-dashboard-v2.ts
// An agent with a simple fallback for the support service.

// ...imports and search definition
  handler: async ({ parameters }) => {
    const user = await userService.findByEmail(parameters.email);
    if (!user) throw new Error('User not found');

    const subscription = await billingService.getStatus(user.id);

    let tickets = [];
    try {
      // Attempt to get tickets, but don't fail the whole search
      tickets = await supportService.getRecentTickets(user.id);
    } catch (error) {
      console.error(`Support service failed for user ${user.id}:`, error);
      // Fallback: return an empty array. The UI can handle this.
    }

    return {
      ...user,
      subscriptionStatus: subscription.status,
      recentTickets: tickets,
    };
  },
// ...

This is a significant improvement. Our API as a Service is now more robust. The caller gets a successful response with the most important data, even if a downstream dependency has a problem.

Pattern 2: Protecting Your System with a Circuit Breaker

The fallback works well for non-critical data, but what if the billingService is experiencing a major outage and timing out on every request? Our agent will patiently wait for the timeout on every call, tying up resources and slowing down the entire system. If traffic is high, this could lead to a cascading failure.

This is the perfect use case for the Circuit Breaker pattern. Here's how it works:

Closed: The circuit is closed, and requests flow normally. The breaker monitors for failures.
Open: After a configured number of failures, the circuit "opens." All subsequent calls fail immediately without attempting to contact the failing service. This gives the troubled service time to recover.
Half-Open: After a timeout, the circuit becomes "half-open" and allows a single test request through. If it succeeds, the circuit closes. If it fails, the circuit opens again.

Let's integrate a standard circuit breaker library (like opossum for Node.js) into our search agent handler to protect the call to the billingService.

// searches/get-customer-dashboard-v3.ts
// The most resilient agent with fallbacks and a circuit breaker.

import { createSearch } from '@searches.do/sdk';
import { userService, billingService, supportService } from '../lib/apis';
import CircuitBreaker from 'opossum';

// Configure a circuit breaker for the critical billing service call
const billingBreaker = new CircuitBreaker(
  (userId) => billingService.getStatus(userId), 
  {
    timeout: 3000, // If call takes longer than 3s, it's a failure
    errorThresholdPercentage: 50, // Open circuit if 50% of calls fail
    resetTimeout: 30000, // Try again after 30s
  }
);

// Define a fallback for when the circuit is open
billingBreaker.fallback(() => ({ status: 'unknown', error: 'Billing service is currently unavailable.' }));

export default createSearch({
  id: 'get-customer-dashboard',
  // ...
  handler: async ({ parameters }) => {
    const user = await userService.findByEmail(parameters.email);
    if (!user) throw new Error('User not found');
    
    // Fire the request through the breaker
    const subscription = await billingBreaker.fire(user.id);
    
    // ... same fallback logic for support tickets
    let tickets = [];
    try {
      tickets = await supportService.getRecentTickets(user.id);
    } catch (error) { /* ... */ }
    
    return {
      ...user,
      subscriptionStatus: subscription.status,
      billingError: subscription.error, // Pass along the error message
      recentTickets: tickets,
    };
  },
});

The Searches.do Advantage: Resiliency as Code

With this final version, we've built a truly resilient agentic search. Let's review the benefits of encapsulating this logic within Searches.do:

Abstraction: The consumer of your API simply calls {"searchId": "get-customer-dashboard", ...}. They don't need to know about the complex orchestration, fallbacks, or circuit breakers happening behind the scenes. They just get a fast, predictable, and reliable response. This is Headless Search in action.
Reusability: This resilient logic is now a standardized, reusable business asset. Any application, from your front-end web app to an internal admin tool, can use this search agent and instantly gain all of its resiliency benefits without rewriting a single line of code.
Centralized Management: Instead of scattering resiliency logic across multiple applications, it lives in one place: the search agent's definition. If you need to tweak the circuit breaker's threshold, you change it once, and it's updated everywhere.

By defining your data retrieval logic as code within the Searches.do platform, you elevate it from a simple query to an intelligent, robust, and scalable microservice. You're not just fetching data; you're delivering powerful, resilient Services-as-Software.

Ready to stop worrying about downstream failures? Turn your complex data queries into resilient, reusable APIs.