Category: data-engineering

Spark 3 with Scala Notes

EtLT Some examples of the type of transformation that fits into the EtLT subpattern include the following: Deduplicate records in

Origins Originally, we relied on pre-aggregated graphs and a carefully managed, hand-coded, set of scripts over a MySQL database of

Scuba Not all the applications need joins Deep systems Systems must provide SLO (Service Level Objectives) to define OKRs (you

Kafka Streams operates at an exciting layer of the Kafka ecosystem: the place where data from many sources converges. This