Manu Ajith
Building a Distributed, Parallel ETL Pipeline Using GenStage, Flow, Mnesia
Course Description

The will be about a tale of building a data segregation system built using GenStage, Flow, and Mnesia.

We will explore how we used GenStage for various pipelines to feed the data into the system, and ran some computation on it in parallel using Elixir Flow, and how we leveraged Mnesia's fast and distributed nodes and clustering capabilities to make the data available for each step in the flow, and finally writing the segregated result back to a PostgreSQL database.

We will also cover some of the mistakes made, lessons learned while working with Mnesia, and how we recovered from Mnesia crashes and other pitfalls.

Manu is a tech entrepreneur and software engineer with a passion for building scalable web systems. He is a polyglot developer who primarily uses Ruby, Elixir, and Go at work, but also keeps a keen eye on other functional paradigms like Haskell. These days, he is into building systems using Domain Driven Design, CQRS, and Event Sourcing.

Manu is also a travel enthusiast, an aspiring nomad, who has roamed 26 countries so far and plans to visit the rest soon.

see all the speakers and trainers