Christophe Blefari

SEPTEMBER 28, 2023

An easy-to-manage central storage and querying and transforming layer in SQL. When you put the things like this it opens the doors and does not limit the modern data stack to 4 vendors. When it comes to incrementally change a data platform this is a bit different, you need to find what is going wrong and what could be improved.

Cloud Storage 147

Cloud Storage Big Data Hadoop SQL 147

Data Engineering Podcast

JANUARY 1, 2022

In the same way that application performance monitoring ensures reliable software and keeps application downtime at bay, Monte Carlo solves the costly problem of broken data pipelines. Start trusting your data with Monte Carlo today! Hightouch is the easiest way to sync data into the platforms that your business teams rely on.

Data Warehouse 100

Data Warehouse Data Lake SQL Hadoop 100

Data Engineering Podcast

DECEMBER 18, 2022

In this episode Abe Gong brings his experiences with the Great Expectations project and community to discuss the technical and organizational considerations involved in implementing these constraints to your data workflows. Atlan is a collaborative workspace for data-driven teams, like Github for engineering or Figma for design teams.

Metadata 130

Metadata Business Intelligence Data Lake BI 130

Data Engineering Podcast

OCTOBER 22, 2021

In this episode Oliver Laslett describes why dashboards aren’t sufficient for business analytics, how Lightdash promotes the work that you are already doing in your data warehouse modeling with dbt, and how they are focusing on bridging the divide between data teams and business teams and the requirements that they have for data workflows.

Engineering 100

Engineering Business Intelligence BI Data Warehouse 100

Monte Carlo

MAY 14, 2024

It was designed to support high-volume data exchange and compatibility across different system versions, which is essential for streaming architectures such as Apache Kafka. This development was crucial for enabling both batch and streaming data workflows in dynamic environments, ensuring consistency and durability in big data processing.

Data Lake 52

Data Lake Metadata Hadoop Data Governance 52

Data Engineering Podcast

NOVEMBER 27, 2021

In the same way that application performance monitoring ensures reliable software and keeps application downtime at bay, Monte Carlo solves the costly problem of broken data pipelines. Start trusting your data with Monte Carlo today! Hightouch is the easiest way to sync data into the platforms that your business teams rely on.

Data Warehouse 100

Data Warehouse Data Lake BI Business Intelligence 100

Workfall

JULY 4, 2023

Reading Time: 8 minutes In the world of data engineering, a mighty tool called DBT (Data Build Tool) comes to the rescue of modern data workflows. Imagine a team of skilled data engineers on an exciting quest to transform raw data into a treasure trove of insights.

Data Warehouse 69

Data Warehouse SQL PostgreSQL Database 69

Data Engineering Podcast

NOVEMBER 14, 2021

Atlan is a collaborative workspace for data-driven teams, like Github for engineering or Figma for design teams. Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows. What are the tradeoffs that you see in each approach?

Data Warehouse 100

Data Warehouse BI Data Workflow Data 100

Data Engineering Podcast

OCTOBER 15, 2021

Hightouch is the easiest way to sync data into the platforms that your business teams rely on. The data you’re looking for is already in your data warehouse and BI tools. Connect your warehouse to Hightouch, paste a SQL query, and use their visual mapper to specify how data should appear in your SaaS systems.

Metadata 100

Metadata BI Data Warehouse Government 100

François Nguyen

MARCH 22, 2021

TL;DR After setting up and organizing the teams, we are describing 4 topics to make data mesh a reality. With this 3rd platform generation, you have more real time data analytics and a cost reduction because it is easier to manage this infrastructure in the cloud thanks to managed services. This is really the dark side of the data.

Architecture 130

Architecture Technology Google Cloud Metadata 130

Data Engineering Podcast

OCTOBER 29, 2021

In this episode Ahmed Elsamadisi shares his journey to find a more flexible and universal data model in the form of the "activity schema" that is powering the Narrator platform, and how it has allowed his customers to perform self-service exploration of their business domains without being blocked by schema evolution in the data warehouse.

Data Warehouse 100

Data Warehouse BI Data Workflow Data Pipeline 100

dbt Developer Hub

APRIL 24, 2023

One example of a popular drag-and-drop transformation tool is Alteryx which allows business analysts to transform data by dragging and dropping operators in a canvas. In this sense, dbt may be a more suitable solution to building resilient and modular data pipelines due to its focus on data modeling.

BI 83

BI Data Workflow SQL Data Pipeline 83

Data Engineering Podcast

JULY 3, 2022

In order to quickly identify if and how two data systems are out of sync Gleb Mezhanskiy and Simon Eskildsen partnered to create the open source data-diff utility. The Ascend Data Automation Cloud provides a unified platform for data ingestion, transformation, orchestration, and observability.

Data Integration 100

Data Integration MongoDB Scala MySQL 100

Ascend.io

DECEMBER 19, 2023

You’re extracting and loading data first, then transforming it in Snowflake’s cloud data warehouse. Snowpipe : Automates data loading in near real-time, significantly reducing the latency in data availability for processing and analysis. that you can combine to create custom data workflows.

ETL Tools 52

ETL Tools IT Data Pipeline Data Warehouse 52

Data Engineering Podcast

JULY 17, 2022

Datafold built automated regression testing to help data and analytics engineers deal with data quality in their pull requests. Datafold shows how a change in SQL code affects your data, both on a statistical level and down to individual rows and values before it gets merged to production.

Data Management 130

Data Management Management Metadata MongoDB 130

Knowledge Hut

SEPTEMBER 25, 2023

Skill Requirements for Azure Data Engineer Job Description Here are some important skill requirements that you may find in a job description for Azure Data Engineers: 1. Azure Data Engineers work with these and other solutions. They guarantee that the data is efficiently cleaned, converted, and loaded.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Lake 52

Meltano

OCTOBER 5, 2022

Meltano is based on the extract, load, transform (ELT) principle to simplify the extraction of data from various sources and load it into data warehouses or databases, using Singer taps and targets. Data Orchestration Data orchestration allows businesses to streamline and automate the way they derive insights from data.

Data Engineering 52

Data Engineering Data Engineer Engineering Raw Data 52

Monte Carlo

APRIL 28, 2023

Here’s who’s using DuckDB and what for: Data Analysts : Analysts working with large datasets and complex analytical queries benefit from DuckDB’s performance and columnar storage. Data Engineers : Data engineers who work with data pipelines and ETL processes use DuckDB to enhance the performance of their data processing tasks.

Data Warehouse 52

Data Warehouse Programming Language Datasets Data Pipeline 52

AltexSoft

MARCH 14, 2023

As the volume and complexity of data continue to grow, organizations seek faster, more efficient, and cost-effective ways to manage and analyze data. In recent years, cloud-based data warehouses have revolutionized data processing with their advanced massively parallel processing (MPP) capabilities and SQL support.

IT 59

IT Data Warehouse Data Governance Data Lake 59

dbt Developer Hub

MARCH 23, 2023

Auditing tables is a major part of analytics engineers’ daily tasks, especially when refactoring tables that were built using SQL Stored Procedures or Alteryx Workflows. Easy and intuitive code : Because audit_helper relies on dbt macros, it was designed to be an intuitive tool that runs on simple SQL queries.

Data Warehouse 64

Data Warehouse Generalist SQL Coding 64

ProjectPro

JANUARY 20, 2016

“I want to work with big data and hadoop. ” How much SQL is required to learn Hadoop? In our previous posts, we have answered all the above questions in detail except “How much SQL is required to learn Hadoop?” Studies found that the de facto language for analysts was SQL.

Hadoop 52

Hadoop SQL Java Big Data 52

Monte Carlo

JULY 27, 2023

Interestingly, only 43% specifically mention resolving the data quality issues that do arise. The skills, languages and tools of a data quality engineer Data quality engineers need to be highly skilled in multiple programming languages such as SQL (mentioned in 61% of postings), Python (56%), and Scala (13%).

Engineering 52

Engineering Healthcare Scala Data Warehouse 52

Data Engineering Weekly

JANUARY 15, 2023

SiliconANGLE theCUBE: Analyst Predictions 2023 - The Future of Data Management By far one of the best analyses of trends in Data Management. The author writes an exciting article about switching to Github actions for dbt to save $65,000 [link] Sponsored: Upsolver - Write a SQL Query, Get a Data-in-Motion Pipeline!

Data Engineering 52

Data Engineering Data Engineer Engineering Metadata 52

Data Engineering Podcast

SEPTEMBER 4, 2022

Select Star’s data discovery platform solves that out of the box, with an automated catalog that includes lineage from where the data originated, all the way to which dashboards rely on it and who is viewing them every day. Go to dataengineeringpodcast.com/ascend and sign up for a free trial. (e.g.

MongoDB 100

MongoDB Scala MySQL Data Ingestion 100

Knowledge Hut

NOVEMBER 2, 2023

Top 10 Azure Data Engineering Project Ideas for Beginners For beginners looking to gain practical experience in Azure Data Engineering, here are 10 Azure Data engineer real time projects ideas that cover various aspects of data processing, storage, analysis, and visualization using Azure services: 1.

Data Engineering 52

Data Engineering Data Engineer Coding Project 52

Monte Carlo

DECEMBER 13, 2023

Thanks to Monte Carlo, we can detect these issues very quickly — most of the time, even before our data workflow fails, so users aren’t impacted. Jessica recommends setting up domains according to how data ownership is distributed within your organization. “We Each of those teams are going to look at different tables.

BI 52

BI Data Ingestion SQL Data Warehouse 52

Meltano

OCTOBER 5, 2022

In the same way, a DataOps engineer designs the data assembly line that enables data scientists to derive insights from data analytics faster and with fewer errors. DataOps engineers improve the speed and quality of the data development process by applying DevOps principles to data workflow, known as DataOps.

Engineering 52

Engineering Raw Data Data Pipeline ETL Tools 52

Workfall

JUNE 12, 2023

CDC also plays a crucial role in data integration and ETL processes. It captures incremental changes from transactional databases or other sources, efficiently loading them into data warehouses or data lakes. This table will serve as the source of data, and any changes made here will be reflected in your data warehouse.

Telecommunication 80

Telecommunication Metadata Healthcare Finance 80

Ascend.io

MAY 31, 2023

Data Transformation Because of the many variations of source systems, the data collected during the ingestion phase is often raw, messy, and unstructured. In the ETL world, data transformation is intended to change the structure of the source data to match a specific target database schema, usually in the context of a data warehouse.

Data Pipeline 52

Data Pipeline ETL Tools Pipeline-centric Data Warehouse 52

Monte Carlo

JUNE 14, 2023

The modern data stack era , roughly 2017 to present data, saw the widespread adoption of cloud computing and modern data repositories that decoupled storage from compute such as data warehouses, data lakes, and data lakehouses. They also recently acquired Apache Flink , another streaming solution.

Data Pipeline 52

Data Pipeline Architecture Data Lake Data Warehouse 52

Knowledge Hut

DECEMBER 26, 2023

Data Engineering is typically a software engineering role that focuses deeply on data – namely, data workflows, data pipelines, and the ETL (Extract, Transform, Load) process. This job requires a handful of skills, starting from a strong foundation of SQL and programming languages like Python , Java , etc.

Data Engineering 98

Data Engineering Data Engineer Engineering Pipeline-centric 98

Ripple Engineering

MAY 27, 2021

Our data analysts used to schedule queries on BigQuery for transformation workflows and test the transformed data manually. We did not have a single tool that would automate the building, compiling, testing and documenting of SQL models, so we had no way to scale the process. SQL Models A model is a single.sql file.

Building 52

Building Raw Data SQL Data 52

Knowledge Hut

DECEMBER 28, 2023

This certification, often referred to as the Azure Data Engineer Associate certification, validates the competency of individuals in implementing Azure data solutions. It’s a testament to their ability to create scalable, efficient and secure data pipelines. What is the Azure Data Engineer Certification?

Certification 52

Certification Data Engineering Data Engineer Engineering 52

Snowflake

MARCH 27, 2023

Introducing CARTO Workflows Snowflake’s powerful data ingestion and transformation features help many data engineers and analysts who prefer SQL. Workflows is integrated with Snowflake, so users can design, execute, and automate hundreds of analysis components and push these queries to the Snowflake platform.

Generalist 52

Generalist Raw Data Business Analyst SQL 52

Netflix Tech

OCTOBER 18, 2022

With the high growth of workflows in the past few years?—?increasing increasing at > 100% a year, the need for a scalable data workflow orchestrator has become paramount for Netflix’s business needs. CockroachDB is used for persisting workflow definitions and instance state.

Java 81

Java Data Machine Learning Systems 81

DataKitchen

APRIL 13, 2021

Piperr.io — Pre-built data pipelines across enterprise stakeholders, from IT to analytics, tech, data science and LoBs. Prefect Technologies — Open-source data engineering platform that builds, tests, and runs data workflows. Genie — Distributed big data orchestration service by Netflix.

Consulting 123

Consulting Machine Learning Data Science Data Pipeline 123

DataKitchen

JANUARY 3, 2022

If the IT or data engineering team can’t respond with an enabling data platform in the required time frame, the business analyst does the necessary data work themselves. This ad hoc data engineering work often means coping with numerous data tables and diverse data sets using Alteryx, SQL, Excel or similar tools. .

Business Analyst 98

Business Analyst Data Lake Consulting Data Analytics 98