Solving service discovery in Node.js with SWIM

[rjrodger]

Title

Solving service discovery in Node.js with SWIM

Description

The microservice architecture is a powerful way to structure large scale Node.js systems. Microservices bring many benefits, enabling rapid development, making continuous delivery easier, and making fine-grained scaling practical. But this architecture is a distributed system, and brings with it all the associated challenges of such systems, not least of which is the need for individual services to find each other. Deciding how to solving the service discovery problem is a key decision point for any architect building microservice systems.

A common approach is to use a service registry. Unfortunately service registries bring their own problems, such as ensuring consistency while scaling, or increasing deployment complexity with the need to run agents. All this could be forgiven, but for a greater sin - registries encourage strong coupling between microservices by providing a concept of service identity. The very purpose of the registry, service discovery, is poisonous to one of the key benefits of microservices - that the system is dynamic and flexible.

There is a better way. Services can find each other dynamically, using gossip protocol algorithms. Highly efficient algorithms such SWIM (Scalable Weakly-consistent Infection-style Process Group Membership Protocol) mean that microservices can do away with service discovery altogether, relying instead on an constantly evolving, dynamic map of their local network. By pushing intelligence to the edges of the network, not only does this approach make failure detection orders of magnitude faster (no more waiting for heartbeats to timeout), but it also makes deployment much easier, as the network configures itself.

This talk will examine the use of the SWIM algorithm to build a Twitter clone in Node.js, one microservice at a time.

Learning objectives

• Understand the problems of running O(10^2+) Node services
• Understand why service discovery is not as simple as it looks
• Understand how newer algorithms like SWIM can overcome these issues
• Review and build on a working code example

City of residence

Waterford, Ireland