Rockset

AUGUST 4, 2021

Organizations that depend on data for their success and survival need robust, scalable data architecture, typically employing a data warehouse for analytics needs. Snowflake is often their cloud-native data warehouse of choice. Data ingestion must be performant to handle large amounts of data.

Data Ingestion 40

Data Ingestion Cloud Storage Data Warehouse Data Lake 40

Data Engineering Podcast

OCTOBER 23, 2022

Mention the podcast to get a free "In Data We Trust World Tour" t-shirt. RudderStack helps you build a customer data platform on your warehouse or data lake. Sign up free… or just get the free t-shirt for being a listener of the Data Engineering Podcast at dataengineeringpodcast.com/rudder.

Database 100

Database MySQL Cloud MongoDB 100

Christophe Blefari

JANUARY 20, 2024

Data engineering inherits from years of data practices in US big companies. Hadoop initially led the way with Big Data and distributed computing on-premise to finally land on Modern Data Stack — in the cloud — with a data warehouse at the center. Understand Change Data Capture — CDC.

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Data Engineering Podcast

OCTOBER 16, 2022

Summary The "data lakehouse" architecture balances the scalability and flexibility of data lakes with the ease of use and transaction support of data warehouses. Mention the podcast to get a free "In Data We Trust World Tour" t-shirt. Data teams are increasingly under pressure to deliver.

Data Lake 100

Data Lake Food MongoDB Scala 100

Data Engineering Podcast

JUNE 12, 2022

Another category of unstructured data that every business deals with is PDFs, Word documents, workstation backups, and countless other types of information. In this episode Rod Christensen shares the story behind Aparavi and how you can use it to cut costs and gain value for the long tail of your unstructured data.

Unstructured Data 130

Unstructured Data MongoDB Scala MySQL 130

Knowledge Hut

NOVEMBER 2, 2023

Who is Azure Data Engineer? An Azure Data Engineer is a professional who is in charge of designing, implementing, and maintaining data processing systems and solutions on the Microsoft Azure cloud platform. Azure SQL Database, Azure Data Lake Storage).

Data Engineering 52

Data Engineering Data Engineer Project Coding 52

Christophe Blefari

MARCH 4, 2023

How to run dbt with BigQuery in GitHub Actions — When you're starting with dbt you don't need any orchestrator or dbt Cloud, a CI/CD do it for sure. Tabular is a cloud offer using Apache Iceberg. Some company also call it a lakehouse or a data lake, but the word shift is enough interesting to notice.

Machine Learning 130

Machine Learning AWS Data Data Lake 130

Knowledge Hut

MARCH 28, 2024

Azure Data Engineer Career Demands & Benefits Azure has become one of the most powerful platforms in the industry, where Microsoft offers a variety of data services and analytics tools. As a result, organizations are looking to capitalize on cloud-based data solutions.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Warehouse 52

Snowflake

SEPTEMBER 14, 2023

The Snowflake Data Cloud gives you the flexibility to build a modern architecture of choice to unlock value from your data. Snowflake was built from the ground up in the cloud. Snowflake’s platform provides industry-leading features that ensure the highest standards of governance for your account, users, and data.

Data Architecture 84

Data Architecture Architecture Hadoop Data Warehouse 84

Monte Carlo

JANUARY 5, 2024

Data lakehouse architecture combines the benefits of data warehouses and data lakes, bringing together the structure and performance of a data warehouse with the flexibility of a data lake. The data lakehouse’s semantic layer also helps to simplify and open data access in an organization.

Architecture 52

Architecture Data Lake Metadata Unstructured Data 52

Monte Carlo

JANUARY 5, 2024

Data lakehouse architecture combines the benefits of data warehouses and data lakes, bringing together the structure and performance of a data warehouse with the flexibility of a data lake. The data lakehouse’s semantic layer also helps to simplify and open data access in an organization.

Architecture 52

Architecture Data Lake Metadata Unstructured Data 52

Data Engineering Podcast

NOVEMBER 6, 2022

RudderStack helps you build a customer data platform on your warehouse or data lake. Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control. In fact, while only 3.5%

MongoDB 130

MongoDB Scala MySQL Data Lake 130

Data Engineering Podcast

NOVEMBER 13, 2022

Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows. RudderStack helps you build a customer data platform on your warehouse or data lake. How have you approached the design and growth of your data platform? In fact, while only 3.5%

MongoDB 100

MongoDB Scala MySQL Google Cloud 100

Data Engineering Podcast

JULY 24, 2022

In this episode Nick Schrock discusses the importance of orchestration and a central location for managing data systems, the road to Dagster’s 1.0 release, and the new features coming with Dagster Cloud’s general availability. RudderStack helps you build a customer data platform on your warehouse or data lake.

MongoDB 100

MongoDB Scala MySQL Data Lake 100

Data Engineering Podcast

SEPTEMBER 25, 2022

Mention the podcast to get a free "In Data We Trust World Tour" t-shirt. RudderStack helps you build a customer data platform on your warehouse or data lake. Sign up free… or just get the free t-shirt for being a listener of the Data Engineering Podcast at dataengineeringpodcast.com/rudder.

Food 100

Food MongoDB Scala MySQL 100

Data Engineering Podcast

JULY 17, 2022

RudderStack helps you build a customer data platform on your warehouse or data lake. Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control. In fact, while only 3.5%

Data Engineering 100

Data Engineering Data Engineer Engineering MongoDB 100

Data Engineering Podcast

OCTOBER 9, 2022

In this episode Purvi Shah, the VP of Enterprise Big Data Platforms at American Express, explains how they have invested in the cloud to power this visibility and the complex suite of integrations they have built and maintained across legacy and modern systems to make it possible. In fact, while only 3.5%

Food 100

Food MongoDB Scala MySQL 100

Data Engineering Podcast

SEPTEMBER 18, 2022

RudderStack helps you build a customer data platform on your warehouse or data lake. Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control. In fact, while only 3.5%

Hospitality 100

Hospitality Food MongoDB Scala 100

Knowledge Hut

MARCH 19, 2024

AWS is the gold standard of Cloud Computing and has some reasons for it. market share, while all of its rivals combined, Microsoft Azure (29.4%), Google Cloud (3.0%), and IBM (2.6%), do not even reach that percentage. That shows how much AWS has to offer, and you must know about it if you’re a cloud computing enthusiast.

AWS 52

AWS Cloud Computing Amazon Web Services Relational Database 52

Databand.ai

AUGUST 30, 2023

DataOps is a collaborative approach to data management that combines the agility of DevOps with the power of data analytics. It aims to streamline data ingestion, processing, and analytics by automating and integrating various data workflows.

Architecture 52

Architecture Data Ingestion Data Cleanse Data Governance 52

Data Engineering Podcast

SEPTEMBER 11, 2022

RudderStack helps you build a customer data platform on your warehouse or data lake. Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control. In fact, while only 3.5%

Data Pipeline 130

Data Pipeline Building MongoDB Scala 130

Data Engineering Podcast

JULY 10, 2022

RudderStack helps you build a customer data platform on your warehouse or data lake. Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control. In fact, while only 3.5%

Data Engineering 100

Data Engineering Data Engineer Engineering MongoDB 100

Data Engineering Podcast

JULY 3, 2022

report having current investments in automation, 85% of data teams plan on investing in automation in the next 12 months. The Ascend Data Automation Cloud provides a unified platform for data ingestion, transformation, orchestration, and observability. In fact, while only 3.5% That’s where our friends at Ascend.io

Data Integration 100

Data Integration MongoDB Scala MySQL 100

Data Engineering Podcast

AUGUST 6, 2022

RudderStack helps you build a customer data platform on your warehouse or data lake. Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control. In fact, while only 3.5%

Machine Learning 100

Machine Learning Database MySQL PostgreSQL 100

Knowledge Hut

SEPTEMBER 25, 2023

Microsoft Azure is a cloud computing platform that gives businesses fantastic services. This demonstrates how in-demand Microsoft Certified Data Engineers are becoming. They are moving their servers and on-premises data to Azure Cloud. What does all of this mean for Data Engineering professionals?

Data Engineering 52

Data Engineering Data Engineer Engineering Data Lake 52

Data Engineering Podcast

OCTOBER 2, 2022

Mention the podcast to get a free "In Data We Trust World Tour" t-shirt. RudderStack helps you build a customer data platform on your warehouse or data lake. Sign up free… or just get the free t-shirt for being a listener of the Data Engineering Podcast at dataengineeringpodcast.com/rudder.

IT 100

IT Food PostgreSQL MongoDB 100

Data Engineering Podcast

OCTOBER 30, 2022

Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows. RudderStack helps you build a customer data platform on your warehouse or data lake. RudderStack helps you build a customer data platform on your warehouse or data lake.

Engineering 100

Engineering MongoDB Scala MySQL 100

Snowflake

AUGUST 16, 2023

The Snowflake Native App Framework remains available in private preview on Google Cloud Platform and Azure. That’s why we built Snowpipe Streaming, now generally available to handle row-set data ingestion. See Snowflake documentation on how to set up Cross-Cloud Auto-Fulfillment on your listings. Learn more here.

Scala 52

Scala Transportation Kafka Data Lake 52

Data Engineering Podcast

JUNE 26, 2022

Data Engineering Podcast listeners can sign up for a free 2-week sandbox account, go to dataengineeringpodcast.com/tonic today to give it a try! RudderStack helps you build a customer data platform on your warehouse or data lake. Data teams are increasingly under pressure to deliver. In fact, while only 3.5%

Data Management 100

Data Management Management MongoDB Scala 100

Ascend.io

JANUARY 2, 2024

The sources of data can be incredibly diverse, ranging from data warehouses, relational databases, and web analytics to CRM platforms, social media tools, and IoT device sensors. Regardless of the source, data ingestion, which usually occurs in batches or as streams, is the critical first step in any data pipeline.

Data Pipeline 52

Data Pipeline Building Raw Data Data Warehouse 52

Edureka

FEBRUARY 7, 2023

An Azure Data Engineer is responsible for designing, implementing, and maintaining data management and data processing systems on the Microsoft Azure cloud platform. They work with large and complex data sets and are responsible for ensuring that data is stored, processed, and secured efficiently and effectively.

Data Engineering 52

Data Engineering Data Engineer Engineering Programming Language 52

Data Engineering Weekly

OCTOBER 30, 2022

link] Dagster: Build a poor man’s data lake from scratch with DuckDB The value of the data is directly proportional to the recency of the data. Part of me wonders, shouldn’t it be a solved problem where the cloud providers can give these solutions out-of-the-box?

Data Engineering 52

Data Engineering Data Engineer Engineering Data Ingestion 52

Cloudera

AUGUST 26, 2020

All of these happen continuously and repetitively on a daily basis, amounting to petabytes worth of information and data. This requires massive amounts of data ingestion, messaging, and processing within a data-in-motion context. From a data ingestion standpoint, NiFi is designed for this purpose.

Banking 80

Banking Data Ingestion Kafka Data Lake 80

Cloudera

FEBRUARY 16, 2021

In the following sections, we see how the Cloudera Operational Database is integrated with other services within CDP that provide unified governance and security, data ingest capabilities, and expand compatibility with Cloudera Runtime components to cater to your specific use cases. . Integrated across the Enterprise Data Lifecycle .

Database 81

Database Machine Learning Data Lake Kafka 81

Knowledge Hut

APRIL 25, 2023

Here, we'll take a look at the top data engineer tools in 2023 that are essential for data professionals to succeed in their roles. These tools include both open-source and commercial options, as well as offerings from major cloud providers like AWS, Azure, and Google Cloud. What are Data Engineering Tools?

Data Engineering 52

Data Engineering Data Engineer Engineering Google Cloud 52

WeCloudData

OCTOBER 19, 2021

By leveraging data engineering techniques combined with a cloud toolchain, WeCloudData helped a client achieve a continuous flow of current job market data with analytical capabilities and dashboards to drive the business forward and stay competitive.

Consulting 52

Consulting Raw Data Data Lake Data Pipeline 52

WeCloudData

OCTOBER 19, 2021

By leveraging data engineering techniques combined with a cloud toolchain, WeCloudData helped a client achieve a continuous flow of current job market data with analytical capabilities and dashboards to drive the business forward and stay competitive.

Consulting 52

Consulting Raw Data Data Lake Data Pipeline 52