Data Engineering Weekly

JUNE 29, 2023

I’m at the Databricks summit as I’m writing my thoughts on the Snowflake and Databricks announcements. Databricks and Snowflake are better places to index the data and its metadata to enable natural language query capabilities. Snowflake is a DataLake Platform Snowflake is moving beyond a SQL data warehouse.

Data Engineering 76

Data Engineering Data Engineer Google Cloud Engineering 76

Monte Carlo

APRIL 24, 2023

By accommodating various data types, reducing preprocessing overhead, and offering scalability, data lakes have become an essential component of modern data platforms , particularly those serving streaming or machine learning use cases. Databricks Data Catalog and AWS Lake Formation are examples in this vein.

Data Lake 52

Data Lake Google Cloud Data Warehouse AWS 52

Knowledge Hut

NOVEMBER 17, 2023

Becoming an Azure Data Engineer in this data-centric landscape is a promising career choice. Also, the role of an Azure Data Engineer is highly relevant and in demand. In the post, we will investigate how to become an Azure data engineer, the skills required, the roles and responsibilities of an Azure data engineer, and much more.

Data Engineering 52

Data Engineering Data Engineer Engineering Scala 52

Data Engineering Podcast

JUNE 12, 2022

Another category of unstructured data that every business deals with is PDFs, Word documents, workstation backups, and countless other types of information. In this episode Rod Christensen shares the story behind Aparavi and how you can use it to cut costs and gain value for the long tail of your unstructured data. In fact, while only 3.5%

Unstructured Data 130

Unstructured Data MongoDB Scala MySQL 130

Data Engineering Podcast

OCTOBER 30, 2022

Datafold shows how a change in SQL code affects your data, both on a statistical level and down to individual rows and values before it gets merged to production. Datafold shows how a change in SQL code affects your data, both on a statistical level and down to individual rows and values before it gets merged to production.

Engineering 100

Engineering MongoDB Scala MySQL 100

Snowflake

MARCH 21, 2023

In February, Snowflake launched new features around streaming data ingestion and data governance and improved SQL experience and performance, with enhancements to Search Optimization Service and more. Read on to learn about everything new announced in February. Learn more here. Learn more here.

Retail 72

Retail Healthcare Data Ingestion Consulting 72

Snowflake

MAY 31, 2023

Today, we’re excited to announce the Snowflake Connector for Microsoft Power Platform is now available. This connector provides instant access to up-to-date data within your Snowflake instance without manually integrating against API endpoints. How does the Snowflake Connector for Power Platform work?

BI 57

BI Coding Machine Learning Data Science 57

Knowledge Hut

APRIL 25, 2023

The tremendous growth in data generation, then the rise in data engineer jobs - there’s no arguing the fact that the big data industry is at its best pace and you, as an aspiring data engineer, have a lot to learn and make out of it - including some tools! While they go about it - enter big data data engineer tools.

Data Engineering 52

Data Engineering Data Engineer Engineering Google Cloud 52

Monte Carlo

MARCH 6, 2024

If you’ve been keeping track of our integrations page you might have noticed Monte Carlo is adding support for different databases faster than a nested loop server crash. By the end of 2023, we added Azure SQL database to the list as well as enterprise databases such as Oracle DB , SAP HANA and Teradata.

Database 52

Database MySQL Data Warehouse AWS 52

Data Engineering Podcast

NOVEMBER 13, 2022

Datafold shows how a change in SQL code affects your data, both on a statistical level and down to individual rows and values before it gets merged to production. Datafold shows how a change in SQL code affects your data, both on a statistical level and down to individual rows and values before it gets merged to production.

MongoDB 100

MongoDB Scala MySQL Google Cloud 100

Data Engineering Podcast

JUNE 5, 2022

The team at Skyflow decided that the second best way is to build a storage system dedicated to securely managing your sensitive information and making it easy to integrate with your applications and data systems. Summary The best way to make sure that you don’t leak sensitive data is to never have it in the first place.

Data Security 100

Data Security Metadata MongoDB Scala 100

Data Engineering Podcast

JULY 3, 2022

Summary The perennial challenge of data engineers is ensuring that information is integrated reliably. Using universal data connectors and a flexible API, Tonic integrates seamlessly into your existing pipelines and allows you to shape and size your data to the scale, realism, and degree of privacy that you need. does exactly that.

Data Integration 100

Data Integration MongoDB Scala MySQL 100

Data Engineering Podcast

OCTOBER 23, 2022

Summary The database market has seen unprecedented activity in recent years, with new options addressing a variety of needs being introduced on a nearly constant basis. Despite that, there are a handful of databases that continue to be adopted due to their proven reliability and robust features. Data teams are increasingly under pressure to deliver.

Database 100

Database MySQL Cloud MongoDB 100

Data Engineering Podcast

OCTOBER 16, 2022

In this episode Jason Hughes explains what it means for a lakehouse to be "open" and describes the different components that the Dremio team build and contribute to. Go to dataengineeringpodcast.com/linode today and get a $100 credit to launch a database, create a Kubernetes cluster, or take advantage of all of their other services.

Data Lake 100

Data Lake Food MongoDB Scala 100

Christophe Blefari

MAY 6, 2023

If you already have access to OpenAI in Azure you can now use GPT-4 —only in preview yet. As a reminder the Lakehouse is the merge of the datalake and the data warehouse, which is mainly a way to add a SQL friendly processing engine on top of a datalake with ACID transactions. Meta ones). Internet found this ironical.

Programming Language 100

Programming Language Data Python Data Warehouse 100

Data Engineering Podcast

SEPTEMBER 25, 2022

Summary Data integration from source systems to their downstream destinations is the foundational step for any data product. In this episode Mark Van de Wiel explains how they integrated CDC functionality into their existing product, discusses the nuances of different approaches to change data capture from various sources.

Food 100

Food MongoDB Scala MySQL 100

Knowledge Hut

MARCH 20, 2024

Example 1 X [company's name] seeks a proficient AI engineer who understands deep learning, neuro-linguistic programming, computer vision, and other AI technologies. Typical roles and responsibilities include the following: Ability to create and evaluate AI models using neural networks, ML algorithms, deep learning, etc.

Engineering 52

Engineering NoSQL Programming Language Deep Learning 52

AltexSoft

AUGUST 29, 2023

If your organization fits into one of these categories and you’re considering implementing advanced data management and analytics solutions, keep reading to learn how data lakes work and how they can benefit your business. What is a data lake? A data lake allows to store data before a specific use case has been identified.

Data Lake 52

Data Lake Architecture IT Amazon Web Services 52

Data Engineering Podcast

JULY 24, 2022

This complicates the work involved in making end-to-end workflows visible and integrated. Ascend users love its declarative pipelines, powerful SDK, elegant UI, and extensible plug-in architecture, as well as its support for Python, SQL, Scala, and Java. And don’t forget to thank them for their continued support of this show!

MongoDB 100

MongoDB Scala MySQL Data Lake 100

Data Engineering Podcast

SEPTEMBER 11, 2022

Summary Any business that wants to understand their operations and customers through data requires some form of pipeline. Building reliable data pipelines is a complex and costly undertaking with many layered requirements. And don’t forget to thank them for their continued support of this show! Data stacks are becoming more and more complex.

Data Pipeline 130

Data Pipeline Building MongoDB Scala 130

Knowledge Hut

MARCH 26, 2024

Continuous Learning: Technology is constantly evolving, providing Solutions Architects with opportunities to learn new tools, frameworks, and methodologies to stay updated in their field. Solution architects work with developers, stakeholders, and other IT specialists to build unified and effective systems.

Amazon Web Services 52

Amazon Web Services Google Cloud Computer Science AWS 52

Data Engineering Podcast

OCTOBER 2, 2022

Summary Data lineage is something that has grown from a convenient feature to a critical need as data systems have grown in scale, complexity, and centrality to business. Alvin is a platform that aims to provide a low effort solution for data lineage capabilities focused on simplifying the work of data engineers.

IT 100

IT Food PostgreSQL MongoDB 100

Data Engineering Podcast

OCTOBER 9, 2022

In this episode Purvi Shah, the VP of Enterprise Big Data Platforms at American Express, explains how they have invested in the cloud to power this visibility and the complex suite of integrations they have built and maintained across legacy and modern systems to make it possible. Data teams are increasingly under pressure to deliver.

Food 100

Food MongoDB Scala MySQL 100

Data Engineering Podcast

SEPTEMBER 18, 2022

Summary In order to improve efficiency in any business you must first know what is contributing to wasted effort or missed opportunities. When your business operates across multiple locations it becomes even more challenging and important to gain insights into how work is being done. Data teams are increasingly under pressure to deliver.

Hospitality 100

Hospitality Food MongoDB Scala 100

Snowflake

AUGUST 16, 2023

At Snowflake Summit, we announced a wave of product innovations: Snowpark ML Modeling API, Snowflake Native App Framework, Dynamic Tables and more. Applications Snowflake Native App Framework now available in AWS – public preview Snowflake Native Apps are an entirely new way to put data to work. Learn more here.

Scala 52

Scala Transportation Kafka Data Lake 52

Data Engineering Podcast

JULY 12, 2021

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management You listen to this show to learn about all of the latest tools, patterns, and practices that power data engineering projects across every domain. Go to dataengineeringpodcast.com/97things today to get your copy!

Designing 100

Designing Data Warehouse Data Governance Consulting 100

Databand.ai

DECEMBER 13, 2022

Whether you’re a data engineering pro looking to stay up to date on the latest trends or new to the space and want to learn more, following the right leaders and joining the right conversations can make all the difference when it comes to plugging into the data engineering community. You’ve come to the right place.

Data Engineering 52

Data Engineering Data Engineer Engineering AWS 52

Striim

NOVEMBER 30, 2022

We are excited to launch Striim Cloud on AWS: a real-time data integration and streaming platform that connects clouds, data and applications with unprecedented speed and simplicity. When you run Striim on AWS, it lets you create real-time data pipelines for Redshift, S3, Kinesis, Databricks, Snowflake and RDS for enterprise workloads.

AWS 52

AWS Cloud Management MySQL 52

Data Engineering Podcast

AUGUST 13, 2022

In this episode founder Shayan Mohanty explains how he and his team are bringing software best practices and automation to the world of machine learning data preparation and how it allows data engineers to be involved in the process. And don’t forget to thank them for their continued support of this show!

Machine Learning 100

Machine Learning Pipeline-centric Database-centric MongoDB 100

Knowledge Hut

JUNE 20, 2023

A novice data scientist prepared to start a rewarding journey may need clarification on the differences between a data scientist and a machine learning engineer. Many people are learning data science for the first time and need help comprehending the two job positions. Apache Spark, Microsoft Azure, Amazon Web services, etc.

Machine Learning 52

Machine Learning Data Engineering Data Engineer Engineering 52

Data Engineering Podcast

SEPTEMBER 4, 2022

Summary The global climate impacts everyone, and the rate of change introduces many questions that businesses need to consider. Getting answers to those questions is challenging, because the climate is a multidimensional and constantly evolving system. And don’t forget to thank them for their continued support of this show!

MongoDB 100

MongoDB Scala MySQL Data Ingestion 100

Data Engineering Podcast

JUNE 26, 2022

Using universal data connectors and a flexible API, Tonic integrates seamlessly into your existing pipelines and allows you to shape and size your data to the scale, realism, and degree of privacy that you need. And don’t forget to thank them for their continued support of this show! does exactly that.

Data Management 100

Data Management Management MongoDB Scala 100

Data Engineering Podcast

MAY 19, 2019

And for your machine learning workloads, they just announced dedicated CPU instances. You only need to maintain one integration to instrument your code and get a future-proof way to send data to over 250 services with the flip of a switch. As a result, the tooling for those transformations needs to be reimagined.

Data Analytics 100

Data Analytics Building Engineering Data Warehouse 100

ProjectPro

DECEMBER 7, 2021

Data Pipeline Tools AWS Data Pipeline Azure Data Pipeline Airflow Data Pipeline Learn to Create a Data Pipeline FAQs on Data Pipeline What is a Data Pipeline? It can also consist of simple or advanced processes like ETL (Extract, Transform and Load) or handle training datasets in machine learning applications.

Data Pipeline 52

Data Pipeline Architecture Kafka AWS 52

phData: Data Engineering

SEPTEMBER 19, 2023

Data modeling helps understand the data requirements, ensure data integrity, and facilitate efficient data storage and retrieval. Want to learn more about data governance? Check out our Data Governance on Snowflake blog! They provide a framework for organizing and representing data elements, attributes, and relationships.

Data Lake 52

Data Lake Process Metadata Data Warehouse 52

Monte Carlo

FEBRUARY 6, 2023

But the advent of cloud data warehouses like Snowflake has changed the game. But in reality, a data warehouse migration to cloud solutions like Snowflake and Redshift requires a tremendous amount of preparation to be successful—from schema changes and data validation to a carefully executed QA process. A worthy quest to be sure.

Data Warehouse 52

Data Warehouse AWS Data Validation Data 52

Rockset

JULY 29, 2022

This consistency promotes data integrity, so you can trust the insights to make informed decisions. Often though, they learn that data warehouses are too slow and too expensive to run low latency, high concurrency workloads on real-time data. Additionally, data warehouses are great at offering historical intelligence.

Data Analytics 52

Data Analytics Data Warehouse Datasets Google Cloud 52