Data Engineering Weekly

JULY 2, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork, and SQL grunt work out of building complete customer profiles, so you can quickly ship actionable, enriched data to every downstream team. Write SQL queries without learning SQL? See how it works today.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Ascend.io

MAY 31, 2023

In the modern world of data engineering, two concepts often find themselves in a semantic tug-of-war: data pipeline and ETL. Fast forward to the present day, and we now have data pipelines. Data Ingestion Data ingestion is the first step of both ETL and data pipelines.

Data Pipeline 52

Data Pipeline ETL Tools Pipeline-centric Data Warehouse 52

Knowledge Hut

JUNE 20, 2023

In addition, they are responsible for developing pipelines that turn raw data into formats that data consumers can use easily. Languages Python, SQL, Java, Scala R, C++, Java Script, and Python Tools Kafka, Tableau, Snowflake, etc. Examples Pull daily tweets from the data warehouse hive spreading in multiple clusters.

Machine Learning 52

Machine Learning Data Engineering Data Engineer Engineering 52

Data Engineering Weekly

JANUARY 8, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. Pipelines for data in motion can quickly turn into DAG hell.

Data Engineering 40

Data Engineering Data Engineer Engineering Pipeline-centric 40

Cloudera

OCTOBER 17, 2023

As organizations shift from the modernization of data-driven applications via Kafka towards delivering real-time insight and/or powering smart automated systems, Flink At Current, adoption of Flink was a hot topic and many of the vendors (Cloudera included) use Flink as the engine to power their stream processing offerings as well.

Database-centric 77

Database-centric Kafka Pipeline-centric Database 77

Knowledge Hut

FEBRUARY 27, 2023

Engineers work with Data Scientists to help make the most of the data they collect and have deep knowledge of distributed systems and computer science. In large organizations, data engineers concentrate on analytical databases, operate data warehouses that span multiple databases, and are responsible for developing table schemas.

Data Engineering 52

Data Engineering Data Engineer Database-centric Pipeline-centric 52

phData: Data Engineering

JULY 18, 2023

With the birth of cloud data warehouses, data applications, and generative AI , processing large volumes of data faster and cheaper is more approachable and desired than ever. This typically results in long-running ETL pipelines that cause decisions to be made on stale or old data.

Data Warehouse 52

Data Warehouse Pipeline-centric Government Data 52

Maxime Beauchemin

JANUARY 20, 2017

Data modeling is changing Typical data modeling techniques — like the star schema  — which defined our approach to data modeling for the analytics workloads typically associated with data warehouses, are less relevant than they once were.

Data Engineering 130

Data Engineering Data Engineer Engineering ETL Tools 130

Ascend.io

OCTOBER 20, 2023

Instead of burning through credits on inefficient SQL queries, Snowflake would rather you channel those resources into higher-impact projects. The Nuances of Snowflake Costing Snowflake’s pricing strategy is an exemplification of its user-centric approach: pay for what you use. No one wins with wasteful spending.

Pipeline-centric 52

Pipeline-centric IT Data Pipeline Bytes 52

Monte Carlo

NOVEMBER 10, 2022

Cookies unfortunately also enabled more nefarious actors to capture and share customer data that didn’t have consumers’ best interests in mind. Instead, it’s the enterprise data warehouse /lakehouse, and enriched first-party data, not the CDP nor the CRM, that must occupy the center of our marketing analytics kingdom.

Pipeline-centric 52

Pipeline-centric Data Warehouse Data Data Ingestion 52

Knowledge Hut

JANUARY 30, 2024

This data can be structured, semi-structured, or entirely unstructured, making it a versatile tool for collecting information from various origins. The extracted data is then duplicated or transferred to a designated destination, often a data warehouse optimized for Online Analytical Processing (OLAP).

ETL Tools 52

ETL Tools Database-centric Data Mining Data Cleanse 52

Knowledge Hut

SEPTEMBER 26, 2023

Microsoft Azure's Azure Synapse, formerly known as Azure SQL Data Warehouse, is a complete analytics offering. Designed to tackle the challenges of modern data management and analytics, Azure Synapse brings together the worlds of big data and data warehousing into a unified and seamlessly integrated platform.

Data Lake 52

Data Lake Database-centric Pipeline-centric Machine Learning 52

Grouparoo

OCTOBER 6, 2021

This provided a nice overview of the breadth of topics that are relevant to data engineering including data warehouses/lakes, pipelines, metadata, security, compliance, quality, and working with other teams. Open question: how to seed data in a staging environment? Test system with A/A test. Be adaptable.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

DataKitchen

NOVEMBER 4, 2021

DataOps works on top of your existing data, allowing you to use your favorite tools , whether you prefer SQL, Alteryx, Tableau, Looker or whatever else. DataOps is an open, secure platform that enables anyone granted permission to access data and update the processes and workflows that operate on data.

Process 98

Process Data Process Pharmaceutical Data Lake 98

DareData

JULY 29, 2020

One paper suggests that there is a need for a re-orientation of the healthcare industry to be more "patient-centric". Furthermore, clean and accessible data, along with data driven automations, can assist medical professionals in taking this patient-centric approach by freeing them from some time-consuming processes.

Healthcare 52

Healthcare Data Pipeline Medical Pipeline-centric 52

Monte Carlo

JANUARY 9, 2024

Tip #2: Accept that data quality is a war, not a battle — but we may be at a turning point Our data experts know that data downtime is an ancient enemy — relative to the age of the modern data stack, in any case. I think monitoring for software is a no-brainer, and I feel the same way about monitoring for data.

Pipeline-centric 52

Pipeline-centric Database-centric Data Manufacturing 52

Databand.ai

DECEMBER 20, 2022

Follow Ken on LinkedIn 7) Michael Dillon Senior Business Intelligence Developer Michael is a Senior BI Developer at an English football club and the author of How to Get a Job in Data Analytics. Michael is certified in text mining and analytics, SQL, Python, Tableau, and more.

Data Analytics 52

Data Analytics Google Cloud Data Science Data Mining 52

Databand.ai

DECEMBER 29, 2022

Benjamin shares similar advice on LinkedIn, posting regularly about big data, data infrastructure, data science, data engineering, and data warehousing. He provides AI strategy, data product strategy, transformation, and data organizational build-out services to clients like Airbus, Siemens, Walmart, and JPMC.

BI 52

BI Consulting Data Science Data Governance 52