Build Efficient Data Processing Pipelines in Elixir Using Broadway
Announced in February 2019, Broadway is a new open source tool developed by Plataformatec that aims to streamline data processing pipelines with Elixir. It allows developers to build complex GenStage topologies that can consume and process data efficiently from different sources, including Amazon SQS and RabbitMQ. In this talk, we'll discuss some of the main concepts behind Broadway, how it leverages OTP to achieve fault-tolerance, sharing implementation details and architectural aspects. You'll learn how to build efficient data processing pipelines and how to optimize them using real-time metrics based on Telemetry events.
Marlus Saraiva is a Software Engineer with over 20 years of experience building systems and leading teams to deliver products in different domains and technologies. His work ranges from enterprise finance/accounting systems to open-source tools. Marlus currently enjoys being part of the R&D team at Plataformatec building exciting things like Broadway.