phData: Data Engineering

APRIL 4, 2023

Fivetran today announced support for Amazon Simple Storage Service (Amazon S3) with Apache Iceberg data lake format. Amazon S3 is an object storage service from Amazon Web Services (AWS) that offers industry-leading scalability, data availability, security, and performance.

Data Lake 52

Data Lake Amazon Web Services Data Cleanse Data Warehouse 52

Data Engineering Podcast

NOVEMBER 11, 2018

Summary A data lake can be a highly valuable resource, as long as it is well built and well managed. In this episode Yoni Iny, CTO of Upsolver, discusses the various components that are necessary for a successful data lake project, how the Upsolver platform is architected, and how modern data lakes can benefit your organization.

Data Lake 100

Data Lake Building Kafka Cloud 100

Ascend.io

APRIL 22, 2024

As the world of data continues to evolve at a breakneck pace, we are thrilled to announce the next revolutionary step in our journey – Project Inception. Ascend has always been at the forefront of innovation, and with Project Inception, we’re setting a new standard.

Project 52

Project Metadata Data Pipeline Data Engineering 52

Knowledge Hut

NOVEMBER 2, 2023

As an AWS Cloud Practitioner with experience in delivering multiple AWS cloud practitioner projects, I vividly recall assisting a startup to prove the scalability of their AI solution on AWS during one of my early projects. In this dynamic profession, there is no alternative to project-based learning.

AWS 52

AWS Coding Cloud Project 52

Knowledge Hut

NOVEMBER 2, 2023

Azure Data engineering projects are complicated and require careful planning and effective team participation for a successful completion. While many technologies are available to help data engineers streamline their workflows and guarantee that each aspect meets its objectives, ensuring that everything works properly takes time.

Data Engineering 52

Data Engineering Data Engineer Coding Project 52

Monte Carlo

JUNE 2, 2021

As companies leverage more and more data to drive decision-making and maintain their competitive edge, it’s crucial that this data is accurate and reliable. Over the last few years, data lakes have emerged as a must-have for the modern data stack. What is data observability?

Data Lake 52

Data Lake Machine Learning Unstructured Data Datasets 52

Knowledge Hut

JUNE 26, 2023

If you're aspiring to be a data engineer and seeking to showcase your skills or gain hands-on experience, you've landed in the right spot. Get ready to delve into fascinating data engineering project concepts and explore a world of exciting data engineering projects in this article.

Data Engineering 98

Data Engineering Data Engineer Coding Project 98

Data Engineering Podcast

FEBRUARY 19, 2023

Because of their complete ownership of your data they constrain the possibilities of what data you can store and how it can be used. Can you describe what Iceberg is and its position in the data lake/lakehouse ecosystem? What are the characteristics of Iceberg that lead teams to adopt it for their lakehouse projects?

IT 147

IT Data Lake Metadata Data Warehouse 147

Data Engineering Podcast

MAY 22, 2022

Summary Machine learning has become a meaningful target for data applications, bringing with it an increase in the complexity of orchestrating the entire data flow. Flyte is a project that was started at Lyft to address their internal needs for machine learning and integrated closely with Kubernetes as the execution manager.

Machine Learning 130

Machine Learning Data Engineering Data Engineer Cloud 130

ProjectPro

OCTOBER 6, 2021

Flexera’s State of Cloud report highlighted that 41% of the survey respondents showed the most interest in using Google Cloud Platform for their future cloud computing projects. Beginner Level GCP Sample Projects Ideas 1. Google Cloud Platform is an online vendor of multiple cloud services which can be used publicly.

Google Cloud 52

Google Cloud Project Data Lake Healthcare 52

Edureka

MAY 31, 2023

Introduction Azure Stream Analytics is a powerful and versatile service offered by Microsoft Azure, providing real-time data processing and analytics capabilities. With its seamless integration with various Azure services, it has become a preferred choice for both industrial projects and streaming OTT platforms.

Project 52

Project Manufacturing Retail Data Lake 52

Data Engineering Podcast

DECEMBER 11, 2022

Summary One of the most critical aspects of software projects is managing its data. Managing the operational concerns for your database can be complex and expensive, especially if you need to scale to large volumes of data, high traffic, or geographically distributed usage. or any other destination you choose.

Database 100

Database MySQL Data Lake MongoDB 100

Data Engineering Podcast

OCTOBER 16, 2022

Summary The "data lakehouse" architecture balances the scalability and flexibility of data lakes with the ease of use and transaction support of data warehouses. Mention the podcast to get a free "In Data We Trust World Tour" t-shirt. Dremio has its ancestry in the Drill project.

Data Lake 100

Data Lake Food MongoDB Scala 100

ProjectPro

JULY 22, 2016

For the same cost, organizations can now store 50 times as much data as in a Hadoop data lake than in a data warehouse. Data lake is gaining momentum across various organizations and everyone wants to know how to implement a data lake and why.

Data Lake 40

Data Lake Data Warehouse Hadoop Unstructured Data 40

Data Engineering Podcast

DECEMBER 4, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. Pricing for SQLake is simple.

Data Lake 100

Data Lake MongoDB MySQL Data Warehouse 100

Data Engineering Podcast

AUGUST 20, 2021

Summary Data lakes have been gaining popularity alongside an increase in their sophistication and usability. Despite improvements in performance and data architecture they still require significant knowledge and experience to deploy and manage. Can you describe what Cuelake is and the story behind it?

Building 100

Building Data Lake Data Warehouse SQL 100

Data Engineering Podcast

JANUARY 28, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. And Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data.

Data Lake 147

Data Lake Building High Quality Data AWS 147

Cloudera

JUNE 18, 2022

Cloudera customers run some of the biggest data lakes on earth. These lakes power mission critical large scale data analytics, business intelligence (BI), and machine learning use cases, including enterprise data warehouses. On data warehouses and data lakes.

Data Lake 94

Data Lake Data Warehouse BI SQL 94

Data Engineering Podcast

SEPTEMBER 7, 2020

In this episode he explains how it is designed to allow for querying and combining data where it resides, the use cases that such an architecture unlocks, and the innovative ways that it is being employed at companies across the world. What are the tradeoffs of using Presto on top of a data lake vs a vertically integrated warehouse solution?

Architecture 100

Architecture Data Architecture SQL Engineering 100

Data Engineering Podcast

NOVEMBER 6, 2022

In this episode she shares the story behind the project, the details of how it is implemented, and how you can use it for your own data projects. RudderStack helps you build a customer data platform on your warehouse or data lake. And don’t forget to thank them for their continued support of this show!

MongoDB 130

MongoDB Scala MySQL Data Lake 130

Data Engineering Weekly

MARCH 3, 2024

These improvements, part of Project Lightspeed, aim to reduce latency and increase throughput for stateful streaming queries, making them more efficient and cost-effective. The article also highlights sink-specific improvements and operator-specific enhancements that contribute to the overall performance boost.

Data Engineering 67

Data Engineering Data Engineer Pipeline-centric Engineering 67

Knowledge Hut

MARCH 28, 2024

Role Level Intermediate Responsibilities Design and develop data pipelines to ingest, process, and transform data. Implemented and managed data storage solutions using Azure services like Azure SQL Database , Azure Data Lake Storage, and Azure Cosmos DB.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Warehouse 52

Data Engineering Podcast

MAY 8, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Database 100

Database Data Lake BI Business Intelligence 100

Data Engineering Podcast

MARCH 27, 2022

The growing prominence of cloud and hybrid environments in data management adds additional stress to an already complex endeavor. Privacera is an enterprise grade solution for cloud and hybrid data governance built on top of the robust and battle tested Apache Ranger project.

Data Governance 130

Data Governance Government Cloud Building 130

François Nguyen

MARCH 22, 2021

With this 3rd platform generation, you have more real time data analytics and a cost reduction because it is easier to manage this infrastructure in the cloud thanks to managed services. The data domain Discovery portal with all the metadata on the data life cycle 4.Federated The number of subjects to automatize is not short.

Architecture 130

Architecture Technology Google Cloud Metadata 130

Data Engineering Podcast

OCTOBER 14, 2018

Unfortunately, with no formal specification, each project works slightly different which increases the difficulty of integration across systems. The Hive format is also built with the assumptions of a local filesystem which results in painful edge cases when leveraging cloud object storage for a data lake.

Data Lake 100

Data Lake Big Data Cloud Hadoop 100

Data Engineering Podcast

OCTOBER 30, 2022

It’s hard to have a conversation about data engineering or analysis without mentioning it. Despite its widespread adoption there are still rough edges in its workflow that cause friction for data analysts. To help simplify the adoption and management of dbt projects Nandam Karthik helped create Optimus.

Engineering 100

Engineering MongoDB Scala MySQL 100

Cloudera

OCTOBER 6, 2022

A major goal of these projects is cost reduction; it’s not sexy, it’s pragmatic. Finding opportunities for monetary savings offers the benefit of reducing costs, but more importantly, it enables a reallocation of budgets towards innovation projects. . These data lakes house much of the data needed to also support other use cases.

Data Lake 90

Data Lake Machine Learning Data Storage Cloud Computing 90

Data Engineering Podcast

FEBRUARY 9, 2020

If you’re struggling with unwieldy dimensional models, slow moving projects, or challenges integrating new data sources then listen in on this conversation and then give data vault a try for yourself. What are some of the foundational skills and knowledge that are necessary for effective modeling of data warehouses?

Data Lake 100

Data Lake Data Warehouse Hadoop NoSQL 100

Data Engineering Podcast

JUNE 12, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Unstructured Data 130

Unstructured Data MongoDB Scala MySQL 130

Christophe Blefari

MARCH 4, 2023

Fast News ⚡️ Writing well: a data engineer’s advantage — This is probably a leftover part of the data engineer toolkit, but writing is an essential skill. Qwak proposes a all-in-one platform to manage all operations in a machine learning project. Are the ducks the new elephants? At least for me.

Machine Learning 130

Machine Learning AWS Data Data Lake 130

Knowledge Hut

NOVEMBER 17, 2023

They enhance data pipelines, transform data, and guarantee the accuracy, integrity, and compliance of the data. Their job entails Azure data engineer skills like using big data, databases, data lakes, and analytics to help firms make efficient data-driven decisions.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Lake 52

Data Engineering Podcast

MARCH 31, 2024

To bring observability to dbt projects the team at Elementary embedded themselves into the workflow. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. What are the challenges/shortcomings associated with those approaches?

Project 130

Project Data Lake High Quality Data Data Workflow 130

Data Engineering Podcast

AUGUST 31, 2020

I’m working with O’Reilly on a project to collect the 97 things that every data engineer should know, and I need your help. When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Data Warehouse 100

Data Warehouse Cloud Building Data Lake 100

ProjectPro

JULY 23, 2021

AWS (Amazon Web Services) is the world’s leading and widely used cloud platform, with over 200 fully featured services available from data centers worldwide. This blog presents some of the most unique and innovative AWS projects from beginner to advanced levels. Applications of AWS Projects Why should you work on AWS Projects?

AWS 52

AWS Project Amazon Web Services Cloud Computing 52

Data Engineering Podcast

OCTOBER 23, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Database 100

Database MySQL Cloud MongoDB 100

Cloudera

MARCH 17, 2023

The table format provides the necessary structure for the unstructured data that is missing in a data lake, using a schema or metadata definition, to bring it closer to a data warehouse. Some of the popular table formats are Apache Iceberg, Delta Lake, Hudi, and Hive ACID.

Education 62

Education Unstructured Data Data Lake Data Warehouse 62