Snowflake

APRIL 18, 2024

Legacy SIEM cost factors to keep in mind Data ingestion: Traditional SIEMs often impose limits to data ingestion and data retention. Snowflake allows security teams to store all their data in a single platform and maintain it all in a readily accessible state, with virtually unlimited cloud data storage capacity.

Data Lake 80

Data Lake Data Ingestion Bytes Cloud Computing 80

Databand.ai

AUGUST 30, 2023

DataOps is a collaborative approach to data management that combines the agility of DevOps with the power of data analytics. It aims to streamline data ingestion, processing, and analytics by automating and integrating various data workflows.

Architecture 52

Architecture Data Ingestion Data Governance Data Cleanse 52

Data Engineering Podcast

NOVEMBER 6, 2022

In this episode she shares the story behind the project, the details of how it is implemented, and how you can use it for your own data projects. report having current investments in automation, 85% of data teams plan on investing in automation in the next 12 months. In fact, while only 3.5% That’s where our friends at Ascend.io

MongoDB 130

MongoDB Scala MySQL Data Lake 130

Data Engineering Weekly

MARCH 24, 2024

[link] Freshworks: Modernizing analytics data ingestion pipeline from legacy engine to distributed processing engine The article discusses Freshworks' journey in modernizing its analytics data platform to handle increasing volumes of data efficiently.

Data Engineering 62

Data Engineering Data Engineer Engineering Metadata 62

Cloudera

NOVEMBER 1, 2023

The connector makes it easy to update the LLM context by loading, chunking, generating embeddings, and inserting them into the Pinecone database as soon as new data is available. High-level overview of real-time data ingest with Cloudera DataFlow to Pinecone vector database.

Machine Learning 108

Machine Learning Data Ingestion Database Architecture 108

Data Engineering Podcast

JUNE 12, 2022

In this episode Rod Christensen shares the story behind Aparavi and how you can use it to cut costs and gain value for the long tail of your unstructured data. report having current investments in automation, 85% of data teams plan on investing in automation in the next 12 months. In fact, while only 3.5% In fact, while only 3.5%

Unstructured Data 130

Unstructured Data MongoDB Scala MySQL 130

Databand.ai

AUGUST 30, 2023

DataOps , short for data operations, is an emerging discipline that focuses on improving the collaboration, integration, and automation of data processes across an organization. These tools help organizations implement DataOps practices by providing a unified platform for data teams to collaborate, share, and manage their data assets.

Data Cleanse 52

Data Cleanse Data Pipeline Data Ingestion Data Validation 52

Data Engineering Podcast

JULY 17, 2022

Summary Data engineering is a large and growing subject, with new technologies, specializations, and "best practices" emerging at an accelerating pace. report having current investments in automation, 85% of data teams plan on investing in automation in the next 12 months. In fact, while only 3.5% Links Podcast.__init__

Data Engineering 100

Data Engineering Data Engineer Engineering MongoDB 100

Knowledge Hut

APRIL 23, 2024

The history of big data takes people on an astonishing journey of big data evolution, tracing the timeline of big data. The Emergence of Data Storage and Processing Technologies A data storage facility first appeared in the form of punch cards, developed by Basile Bouchon to facilitate pattern printing on textiles in looms.

Big Data 52

Big Data Amazon Web Services Media Cloud Computing 52

Data Engineering Podcast

OCTOBER 30, 2022

Summary One of the most impactful technologies for data analytics in recent years has been dbt. It’s hard to have a conversation about data engineering or analysis without mentioning it. Despite its widespread adoption there are still rough edges in its workflow that cause friction for data analysts.

Engineering 100

Engineering MongoDB Scala MySQL 100

Snowflake

MARCH 21, 2023

In February, Snowflake launched new features around streaming data ingestion and data governance and improved SQL experience and performance, with enhancements to Search Optimization Service and more. Check out Felipe Hoffa’s video on how to use Snowsight to get from data to decision faster.

Retail 72

Retail Healthcare Data Ingestion Consulting 72

Data Engineering Podcast

AUGUST 21, 2022

In this episode Shruti Bhat gives her view on the state of the ecosystem for real-time data and the work that she and her team at Rockset is doing to make it easier for engineers to build those experiences. report having current investments in automation, 85% of data teams plan on investing in automation in the next 12 months.

Lambda Architecture 100

Lambda Architecture MongoDB Scala MySQL 100

Data Engineering Podcast

JUNE 19, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Metadata 130

Metadata MongoDB Scala MySQL 130

Data Engineering Podcast

JULY 31, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Data Analysis 100

Data Analysis MongoDB Scala Algorithm 100

Lyft Engineering

NOVEMBER 29, 2023

Druid at Lyft Apache Druid is an in-memory, columnar, distributed, open-source data store designed for sub-second queries on real-time and historical data. Druid enables low latency (real-time) data ingestion, flexible data exploration and fast data aggregation resulting in sub-second query latencies.

Kafka 104

Kafka Data Ingestion Datasets Architecture 104

Cloudera

FEBRUARY 8, 2021

Adding more wires and throwing more compute hardware to the problem is simply not viable considering the cost and complexities of today’s connected cars or the additional demands designed into electric cars (like battery management systems and eco-trip planning).

Manufacturing 108

Manufacturing Machine Learning Data Ingestion Electronics 108

Databand.ai

AUGUST 30, 2023

The DataOps framework is a set of practices, processes, and technologies that enables organizations to improve the speed, accuracy, and reliability of their data management and analytics operations. This can be achieved through the use of automated data ingestion, transformation, and analysis tools.

Data Governance 52

Data Governance Data Pipeline Government Data Cleanse 52

Data Engineering Podcast

NOVEMBER 13, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

MongoDB 100

MongoDB Scala Google Cloud MySQL 100

Snowflake

NOVEMBER 29, 2023

Data cloud technology can accelerate FAIRification of the world’s biomedical patient data. Next-generation sequencing (NGS) technology has dramatically dropped the price of genomic sequencing, from about $1 million in 2007 to $600 today per whole genome sequencing (WGS).

Metadata 84

Metadata Healthcare Medical Data Storage 84

Precisely

MAY 13, 2024

Key Takeaways: Data integration is vital for real-time data delivery across diverse cloud models and applications, and for leveraging technologies like generative AI. The right data integration solution helps you streamline operations, enhance data quality, reduce costs, and make better data-driven decisions.

Data Integration 52

Data Integration Metadata Amazon Web Services Data Governance 52

Data Engineering Podcast

AUGUST 13, 2022

In this episode Shinji Kim discusses the challenges of data discovery and how to collect and preserve additional context about each piece of information so that you can find what you need when you don’t even know what you’re looking for yet. In fact, while only 3.5% That’s where our friends at Ascend.io In fact, while only 3.5%

Metadata 100

Metadata MongoDB Scala MySQL 100

Data Engineering Podcast

OCTOBER 23, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Database 100

Database MySQL Cloud MongoDB 100

Data Engineering Podcast

OCTOBER 16, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Data Lake 100

Data Lake Food MongoDB Scala 100

Data Engineering Podcast

JULY 24, 2022

Summary The current stage of evolution in the data management ecosystem has resulted in domain and use case specific orchestration capabilities being incorporated into various tools. In this episode Nick Schrock discusses the importance of orchestration and a central location for managing data systems, the road to Dagster’s 1.0

MongoDB 100

MongoDB Scala MySQL Data Lake 100

Data Engineering Podcast

SEPTEMBER 25, 2022

In this episode Mark Van de Wiel explains how they integrated CDC functionality into their existing product, discusses the nuances of different approaches to change data capture from various sources. report having current investments in automation, 85% of data teams plan on investing in automation in the next 12 months.

Food 100

Food MongoDB Scala MySQL 100

Data Engineering Podcast

SEPTEMBER 18, 2022

He discusses the challenges of making data collection purposeful and efficient without distracting employees from their primary duties and how business owners can use the provided analytics to support their staff in their duties. In fact, while only 3.5% That’s where our friends at Ascend.io Closing Announcements Thank you for listening!

Hospitality 100

Hospitality Food MongoDB Scala 100

Data Engineering Podcast

SEPTEMBER 11, 2022

In this episode he shares his journey from building a consumer product to launching a data pipeline service and how his frustrations as a product owner have informed his work at Hevo Data. report having current investments in automation, 85% of data teams plan on investing in automation in the next 12 months.

Data Pipeline 130

Data Pipeline Building MongoDB Scala 130

Data Engineering Podcast

AUGUST 28, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Building 100

Building MongoDB Scala MySQL 100

Data Engineering Podcast

JUNE 5, 2022

In this episode Sean Falconer explains the idea of a data privacy vault and how this new architectural element can drastically reduce the potential for making a mistake with how you manage regulated or personally identifiable information. In fact, while only 3.5% That’s where our friends at Ascend.io

Data Security 100

Data Security Metadata MongoDB Scala 100

DareData

JANUARY 30, 2023

Our goal is to help data scientists better manage their models deployments or work more effectively with their data engineering counterparts, ensuring their models are deployed and maintained in a robust and reliable way. Airflow is written in Python and has a web-based user interface for managing and monitoring pipelines.

Data Engineering 52

Data Engineering Data Engineer NoSQL Engineering 52

Data Engineering Podcast

MAY 29, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Architecture 100

Architecture Database Data Architecture PostgreSQL 100

Edureka

FEBRUARY 9, 2023

Azure Data Engineering is a rapidly growing field that involves designing, building, and maintaining data processing systems using Microsoft Azure technologies. As the demand for data engineers grows, having a well-written resume that stands out from the crowd is critical.

Data Engineering 52

Data Engineering Data Engineer Engineering Amazon Web Services 52

Data Engineering Podcast

JULY 10, 2022

In this episode Chris Riccomini shares his experiences building and scaling data operations at WePay and LinkedIn, as well as the lessons he has learned working with other teams as they automated their own systems. report having current investments in automation, 85% of data teams plan on investing in automation in the next 12 months.

Data Engineering 100

Data Engineering Data Engineer Engineering MongoDB 100

Data Engineering Podcast

JULY 3, 2022

In this episode they explain how the utility is implemented to run quickly and how you can start using it in your own data workflows to ensure that your data warehouse isn’t missing any records from your source systems. Can you describe what the data diff tool is and the story behind it? In fact, while only 3.5%

Data Integration 100

Data Integration MongoDB Scala MySQL 100

Snowflake

APRIL 24, 2023

Today, the asset servicing industry is constrained by highly manual, as well as time-intensive processes from poorly integrated legacy technology platforms and data silos. Faced with these challenges, asset servicers have acquired numerous technologies over time to meet their order management, risk management, and fund analytics.

Management 52

Management Portfolio Data Ingestion Data Warehouse 52

Ascend.io

AUGUST 29, 2023

Just as every play and position in baseball is key to a win, each component of a data pipeline is integral to effective data management. Similarly , in data, every step of the pipeline, from data ingestion to delivery, plays a pivotal role in delivering impactful results.

Data Pipeline 52

Data Pipeline Pipeline-centric Database-centric Data Ingestion 52

Data Engineering Podcast

AUGUST 6, 2022

In this episode Frank Liu explains how the open source Milvus vector database is implemented to speed up machine learning development cycles, how to think about proper storage and scaling of these vectors, and how data engineering and machine learning teams can collaborate on the creation and maintenance of these data sets.

Machine Learning 100

Machine Learning Database MySQL PostgreSQL 100