Data Engineering Podcast

OCTOBER 30, 2022

In this episode he shares his experiences working with organizations to adopt analytics engineering patterns and the ways that Optimus and dbt were combined to let data analysts deliver insights without the roadblocks of complex pipeline management. Interview Introduction How did you get involved in the area of data management?

Engineering 100

Engineering MongoDB Scala MySQL 100

Data Engineering Podcast

OCTOBER 23, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. That’s where our friends at Ascend.io

Database 100

Database MySQL Cloud MongoDB 100

Data Engineering Podcast

OCTOBER 16, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. That’s where our friends at Ascend.io

Data Lake 100

Data Lake Food MongoDB Scala 100

Data Engineering Podcast

SEPTEMBER 25, 2022

In this episode Mark Van de Wiel explains how they integrated CDC functionality into their existing product, discusses the nuances of different approaches to change data capture from various sources. The Ascend Data Automation Cloud provides a unified platform for data ingestion, transformation, orchestration, and observability.

Food 100

Food MongoDB Scala MySQL 100

Data Engineering Podcast

SEPTEMBER 18, 2022

He discusses the challenges of making data collection purposeful and efficient without distracting employees from their primary duties and how business owners can use the provided analytics to support their staff in their duties. That’s where our friends at Ascend.io Can you describe what OpsAnalitica is and the story behind it?

Hospitality 100

Hospitality Food MongoDB Scala 100

Data Engineering Podcast

JULY 24, 2022

Summary The current stage of evolution in the data management ecosystem has resulted in domain and use case specific orchestration capabilities being incorporated into various tools. In this episode Nick Schrock discusses the importance of orchestration and a central location for managing data systems, the road to Dagster’s 1.0

MongoDB 100

MongoDB Scala MySQL Data Lake 100

Data Engineering Podcast

AUGUST 28, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. That’s where our friends at Ascend.io

Building 100

Building MongoDB Scala MySQL 100

Data Engineering Podcast

SEPTEMBER 11, 2022

In this episode he shares his journey from building a consumer product to launching a data pipeline service and how his frustrations as a product owner have informed his work at Hevo Data. The Ascend Data Automation Cloud provides a unified platform for data ingestion, transformation, orchestration, and observability.

Data Pipeline 130

Data Pipeline Building MongoDB Scala 130

Data Engineering Podcast

JUNE 5, 2022

In this episode Sean Falconer explains the idea of a data privacy vault and how this new architectural element can drastically reduce the potential for making a mistake with how you manage regulated or personally identifiable information. That’s where our friends at Ascend.io Can you describe what Skyflow is and the story behind it?

Data Security 100

Data Security Metadata MongoDB Scala 100

Data Engineering Podcast

JULY 10, 2022

In this episode Chris Riccomini shares his experiences building and scaling data operations at WePay and LinkedIn, as well as the lessons he has learned working with other teams as they automated their own systems. What are the pieces of data platforms and processing that have been most difficult to scale in an organizational sense?

Data Engineering 100

Data Engineering Data Engineer Engineering MongoDB 100

Data Engineering Podcast

JULY 3, 2022

In this episode they explain how the utility is implemented to run quickly and how you can start using it in your own data workflows to ensure that your data warehouse isn’t missing any records from your source systems. Can you describe what the data diff tool is and the story behind it? That’s where our friends at Ascend.io

Data Integration 100

Data Integration MongoDB Scala MySQL 100

Data Engineering Podcast

OCTOBER 9, 2022

In this episode Purvi Shah, the VP of Enterprise Big Data Platforms at American Express, explains how they have invested in the cloud to power this visibility and the complex suite of integrations they have built and maintained across legacy and modern systems to make it possible. That’s where our friends at Ascend.io

Food 100

Food MongoDB Scala MySQL 100

Data Engineering Podcast

OCTOBER 2, 2022

In this episode co-founder Martin Sahlen explains the impact that easy access to lineage information can have on the work of data engineers and analysts, and how he and his team have designed their platform to offer that information to engineers and stakeholders in the places that they interact with data.

IT 100

IT Food PostgreSQL MongoDB 100

Data Engineering Podcast

AUGUST 6, 2022

In this episode Frank Liu explains how the open source Milvus vector database is implemented to speed up machine learning development cycles, how to think about proper storage and scaling of these vectors, and how data engineering and machine learning teams can collaborate on the creation and maintenance of these data sets.

Machine Learning 100

Machine Learning Database MySQL PostgreSQL 100

Data Engineering Podcast

AUGUST 13, 2022

In this episode founder Shayan Mohanty explains how he and his team are bringing software best practices and automation to the world of machine learning data preparation and how it allows data engineers to be involved in the process. That’s where our friends at Ascend.io Can you describe what Watchful is and the story behind it?

Machine Learning 100

Machine Learning Pipeline-centric Database-centric MongoDB 100

Data Engineering Podcast

SEPTEMBER 4, 2022

In this episode Gopal Erinjippurath discusses the data engineering challenges of building and serving those data sets, and how they are distilling complex climate information into consumable facts so you don’t have to be an expert to understand it. That’s where our friends at Ascend.io What audience(s) are you focused on?

MongoDB 100

MongoDB Scala MySQL Data Ingestion 100

Ascend.io

OCTOBER 4, 2023

Snowpark stands out as a game-changer for data engineers. It empowers them to tap into the familiar terrain of languages like Scala, Java, and Python, but with the unique advantage of not having to move data out of Snowflake. Snowflake is renowned for its vast capabilities, and Snowpark is no exception.

Scala 52

Scala Data Ingestion Python Java 52

Cloudera

APRIL 20, 2021

The developers must understand lower-level languages like Java and Scala and be familiar with the streaming APIs. A modern streaming architecture consists of critical components that provide data ingestion, security and governance, and real-time analytics. What is modern streaming architecture?

Hospitality 94

Hospitality Kafka Retail Data Ingestion 94

Snowflake

AUGUST 16, 2023

Data comes in a continuous manner, and often a separate architecture is required to handle streaming data. What remains challenging is how streaming data is brought together with batch data. That’s why we built Snowpipe Streaming, now generally available to handle row-set data ingestion. Learn more here.

Scala 52

Scala Transportation Kafka Data Lake 52

Edureka

FEBRUARY 7, 2023

An Azure Data Engineer is responsible for designing, implementing, and maintaining data management and data processing systems on the Microsoft Azure cloud platform. They work with large and complex data sets and are responsible for ensuring that data is stored, processed, and secured efficiently and effectively.

Data Engineering 52

Data Engineering Data Engineer Engineering Programming Language 52

ProjectPro

FEBRUARY 21, 2023

Due to the enormous amount of data being generated and used in recent years, there is a high demand for data professionals, such as data engineers, who can perform tasks such as data management, data analysis, data preparation, etc. Basic understanding of Microsoft Azure.

Certification 52

Certification Data Engineering Data Engineer Engineering 52

Knowledge Hut

OCTOBER 3, 2023

Additionally, for a job in data engineering, candidates should have actual experience with distributed systems, data pipelines, and related database concepts. First and foremost, the training institute needs to have a solid reputation for offering high-quality instruction in data engineering and cloud computing.

Data Engineering 52

Data Engineering Data Engineer Engineering Cloud Computing 52

Knowledge Hut

SEPTEMBER 29, 2023

By combining data from various structured and unstructured data systems into structures, Microsoft Azure Data Engineers will be able to create analytics solutions. Why Should You Get an Azure Data Engineer Certification? You can browse the data lake files with the interactive training material.

Certification 52

Certification Data Engineering Data Engineer Engineering 52

Data Engineering Podcast

NOVEMBER 20, 2022

In this episode CEO and founder Salma Bakouk shares her views on the causes and impacts of "data entropy" and how you can tame it before it leads to failures. The Ascend Data Automation Cloud provides a unified platform for data ingestion, transformation, orchestration, and observability.

Data Lake 130

Data Lake Data Ingestion MongoDB Scala 130

Data Engineering Podcast

JUNE 19, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. That’s where our friends at Ascend.io

Metadata 130

Metadata MongoDB Scala MySQL 130

Data Engineering Podcast

JULY 31, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. That’s where our friends at Ascend.io

Data Analysis 100

Data Analysis MongoDB Scala Algorithm 100

Snowflake

NOVEMBER 29, 2023

The data is there, it’s just not FAIR: Findable, Accessible, Interoperable and Reusable. Defining FAIR data and it’s applications for life sciences FAIR was a term coined in 2016 to help define good data management practices within the scientific realm. The principles emphasize machine-actionability (i.e.,

Metadata 84

Metadata Healthcare Medical Data Storage 84

Knowledge Hut

SEPTEMBER 26, 2023

Microsoft Azure's Azure Synapse, formerly known as Azure SQL Data Warehouse, is a complete analytics offering. Designed to tackle the challenges of modern data management and analytics, Azure Synapse brings together the worlds of big data and data warehousing into a unified and seamlessly integrated platform.

Data Lake 52

Data Lake Database-centric Pipeline-centric Machine Learning 52

Data Engineering Podcast

MAY 29, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. That’s where our friends at Ascend.io

Architecture 100

Architecture Database Data Architecture PostgreSQL 100

Edureka

FEBRUARY 9, 2023

As the demand for data engineers grows, having a well-written resume that stands out from the crowd is critical. Azure data engineers are essential in the design, implementation, and upkeep of cloud-based data solutions. It is also crucial to have experience with data ingestion and transformation.

Data Engineering 52

Data Engineering Data Engineer Engineering Amazon Web Services 52

ProjectPro

FEBRUARY 8, 2023

Jupyter Notebook – Those comfortable and familiar with creating ETL jobs using jupyter notebook can choose this option to create a new Python or Scala ETL job script using this notebook. You can also have the option of scripting the Python or Scala code in a script editor window or uploading an existing script locally.

AWS 98

AWS Scala Metadata Data Lake 98

Snowflake

MARCH 30, 2023

3EJHjvm Once a business need is defined and a minimal viable product ( MVP ) is scoped, the data management phase begins with: Data ingestion: Data is acquired, cleansed, and curated before it is transformed. ML workflow, ubr.to/3EJHjvm

Engineering 83

Engineering Raw Data Data Science Scala 83

AltexSoft

JULY 29, 2022

To facilitate data ingestion, there are Apache Flume aggregating log data from multiple servers and Apache Sqoop designed to transport information between Hadoop and relational (SQL) databases. If you’re going to create applications for the Hadoop ecosystem, get familiar with Scala, which is the default language of Apache Spark.

Hadoop 59

Hadoop Big Data Google Cloud NoSQL 59

ProjectPro

JANUARY 31, 2023

Define Big Data and Explain the Seven Vs of Big Data. Big Data is a collection of large and complex semi-structured and unstructured data sets that have the potential to deliver actionable insights using traditional data management tools. Data Storage: Spark stores data using the HDFS file system.

Big Data 52

Big Data Hadoop AWS Relational Database 52

AltexSoft

MARCH 30, 2023

What is Databricks Databricks is an analytics platform with a unified set of tools for data engineering, data management , data science, and machine learning. It combines the best elements of a data warehouse, a centralized repository for structured data, and a data lake used to host large amounts of raw data.

Scala 64

Scala Data Lake BI Google Cloud 64