Zero - ETL ?
- AWS announced ETL-free integration between Aurora and Redshift, and between Redshift and Apache Spark
- Google announced the Zero-ETL approach with big Query
- Databricks announced direct queries on external systems using JDBC
- Microsoft Lake storage, in combination with Snowflake / Snowpipe, is close to the Zero-ETL approach
As Data Engineer, We well-know a question “What is Zero ETL”?
So what is Zero ETL
-
The Zero ETL is ideated to eliminate these problems by providing a secure way for data to move between different systems without any manual intervention. It ensures that all data is up-to-date through continuous federation between all connected systems.
-
In Zero ETL, engineers will learn how to process data in the data lakes, lake houses, warehouses, or Spark clusters and Python scripts, so on, and they will rely on low-code platforms for many of their needs.
-
This approach can be useful in situations where data needs to be transferred quickly and efficiently between systems, to enable near real-time analytics and machine learning (ML) without the need for complex data transformation or manipulation.
-
With near real-time access to transactional data, you can leverage this for analytics and capabilities such as built-in ML, materialized views, data sharing, and federated access to multiple data stores and data lakes to derive insights from transactional and other data.