Building Resilient Microservices: Strategies for Handling Failures

The goal of our system is to monitor changes in stock prices and promptly notify users of drastic price changes using email notification. In order to ensure the necessary level of fault tolerance, we opted for a microservices architecture based on Apache Kafka as a distributed event streaming platform and Kubernetes as the orchestration tool. The communication between services is facilitated by Kafka topics, allowing for seamless data flow and event-driven interactions. Our system collects data from the exchange through the Market Data Handler and then distributes it to subscribers of the Price Change topic via Kafka. Among these subscribers is a notification service responsible for sending notifications via email.

Let’s outline a scenario where fault tolerance and circuit breaker mechanisms are needed, along with approaches to make the system resilient.

Scenario 1: Dependency Service Outage: The Market Data Handler service experiences a temporary outage due to an issue with the stock exchange’s API or infrastructure.

Challenge: Without data from the Market Data Handler, the entire system may fail to provide accurate notifications, impacting user experience and trust.

Approaches:

• Timeouts and Retries: Implement timeouts and retry mechanisms when calling external dependencies to…