Scale with NSQ: a realtime distributed messaging platform

Scale
06/02/2015 - 17:20 to 17:40
Stage 1
short talk (20 min)
Intermediate

Session abstract: 

NSQ is an open source realtime distributed messaging platform designed to operate at scale and handle billions of messages per day. It is built entirely in Go and promotes distributed and decentralized topologies without single points of failure. This enables fault tolerance and high availability coupled with a reliable message delivery guarantee.

NSQ was built by the engineering team at Bitly to support its high volume even stream processing architecture. It powers our near real-time analytics systems and is now used by a range of companies including Docker, Stripe and BuzzFeed.

During this talk we will answer questions such as why was NSQ built and how it fits into the larger picture. We will also take a look at some code examples. By using real-life scenarios that we faced at Bitly, we will discusss how NSQ deals with concepts of availability, performance, latency, fault tolerance and how it's enabled us to scale effectively. We will also discuss why Go was an excellent language to use for building NSQ. 

Video: 

Slide: