Designing Large-Scale Real-Time Data Distribution Networks with ØMQ
Introduction
- Real-time data feeds?
- Stock trading
- TV/radio
- Weather Reports
- Social Networks
- Basically whole state of the world can be turned into real-time feed
- We often use pull model (req/rep) where push (pub/sub) would be more appropriate because we lack the technology to do so.
- We still think of computers and networks as glorified hard-disks (serving historical data instead of real-time data).
- With getting close to 24/7 access to the net the world is changing
What's wrong with real-time feeds?
- We don't want to saturate the network by a large amount of data that nobody needs
- We don't care much about load on publishers themselves though
- The bandwidth problem. How can we eliminate messages nobody is interested in from the network?
Scaling
- Using devices in the middle
End-to-end nature of filtering
- Allows for HW intergration
- Allows for extensible matching algorithms