Towards Data Science

OCTOBER 25, 2023

As a data or analytics engineer, you knew where to find all the transformation logic and models because they were all in the same codebase. You probably work closely with the colleague who builds the data pipeline that you were consuming. There was only one data team, two at most.

Data Pipeline 82

Data Pipeline SQL Data Architecture Data 82

Data Engineering Podcast

SEPTEMBER 30, 2019

Summary Building an end-to-end data pipeline for your machine learning projects is a complex task, made more difficult by the variety of ways that you can structure it. We have partnered with organizations such as O’Reilly Media, Dataversity, Corinium Global Intelligence, and Data Council.

Data Pipeline 100

Data Pipeline Machine Learning Media Data 100

Data Engineering Podcast

JANUARY 26, 2020

Summary Data pipelines are complicated and business critical pieces of technical infrastructure. You listen to this show to learn and stay up to date with what’s happening in databases, streaming platforms, big data, and everything else you need to know about modern data management.

Data Pipeline 100

Data Pipeline PostgreSQL Media Data Validation 100

Data Engineering Podcast

JUNE 26, 2022

Summary The most complicated part of data engineering is the effort involved in making the raw data fit into the narrative of the business. Master Data Management (MDM) is the process of building consensus around what the information actually means in the context of the business and then shaping the data to match those semantics.

Data Management 100

Data Management Management MongoDB Scala 100

Ascend.io

AUGUST 15, 2023

This same principle holds true in data management. It’s no surprise, then, that the quest for Fivetran alternatives is on the rise as organizations set their sights on a more holistic data approach. Defense: Saving Money with Intelligent Data Refresh In football, a solid defense does more than just stop goals.

Data Management 52

Data Management Management Data Ingestion Data Pipeline 52

Ascend.io

MARCH 2, 2023

It won with ease of use by our analysts, the comprehensive scope of its automation, and the speed of end-to-end pipeline development. Terry Dougal Intelligent Data Pipelines at Audacy Instead of hiring more data engineers to patch over all the tooling with more bespoke code, Terry shifted gears and looked for a new solution.

Data Pipeline 52

Data Pipeline Media Portfolio Data Ingestion 52

Data Engineering Podcast

DECEMBER 16, 2018

As the organization grows and gains more customers, the requirements for that pipeline will change. In this episode Christian Heinzmann, Head of Data Warehousing at Grubhub, discusses the various requirements for data pipelines and how the overall system architecture evolves as more data is being processed.

Data Pipeline 100

Data Pipeline Data Lake Data Warehouse Python 100

Ascend.io

APRIL 12, 2023

From this research, we developed a framework with a sequence of stages to implement data integrity quickly and measurably via data pipelines. Table of Contents Why does data integrity matter? At every level of a business, individuals must trust the data, so they can confidently make timely decisions. Let’s explore!

Data Pipeline 52

Data Pipeline Data Integration Datasets Data 52

ProjectPro

DECEMBER 7, 2021

Data pipelines are a significant part of the big data domain, and every professional working or willing to work in this field must have extensive knowledge of them. Table of Contents What is a Data Pipeline? The Importance of a Data Pipeline What is an ETL Data Pipeline?

Data Pipeline 52

Data Pipeline Architecture Kafka AWS 52

Data Engineering Podcast

JUNE 17, 2017

utm_source=rss&utm_medium=rss and get a $20 credit to try out their fast and reliable Linux virtual servers for running your data pipelines or trying out the tools you hear about on the show. Can you start by giving an overview of your pipeline and the type of workload that you are optimizing for?

Data Pipeline 100

Data Pipeline Kafka Business Intelligence Architecture 100

Data Engineering Podcast

MAY 27, 2018

Preamble Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline you’ll need somewhere to deploy it, so check out Linode. What is Alooma and what is the origin story? How is the Alooma platform architected?

Data Pipeline 100

Data Pipeline MongoDB Scala Google Cloud 100

Hevo

MAY 30, 2023

The healthcare industry has seen an exponential growth in the use of data management and integration tools in recent years to leverage the data at their disposal. Unlocking the potential of “Big Data” is imperative in enhancing patient care quality, streamlining operations, and allocating resources optimally.

Healthcare 52

Healthcare Data Pipeline Building Big Data 52

ProjectPro

FEBRUARY 16, 2023

With so much riding on the efficiency of ETL processes for data engineering teams, it is essential to take a deep dive into the complex world of ETL on AWS to take your data management to the next level. This is particularly useful for companies that need to process data in near-real-time.

AWS 52

AWS Data Management ETL Tools Management 52

Databand.ai

AUGUST 30, 2023

The DataOps framework is a set of practices, processes, and technologies that enables organizations to improve the speed, accuracy, and reliability of their data management and analytics operations. This can be achieved through the use of automated data ingestion, transformation, and analysis tools.

Data Governance 52

Data Governance Data Pipeline Government Data Cleanse 52

Data Engineering Podcast

JULY 2, 2023

In this episode Razi Raziuddin shares how data engineering teams can support the machine learning workflow through the development and support of systems that empower data scientists and ML engineers to build and maintain their own features. How is this distinct from other forms of data pipeline development and delivery?

Machine Learning 130

Machine Learning Data Engineering Data Engineer Engineering 130

Data Engineering Podcast

OCTOBER 30, 2022

In this episode Amir Orad discusses the Sisense platform and how it facilitates the embedding of analytics and data insights in every aspect of organizational and end-user experiences. Especially once they realize 90% of all major data sources like Google Analytics, Salesforce, Adwords, Facebook, Spreadsheets, etc.,

Business Intelligence 130

Business Intelligence Metadata MongoDB MySQL 130

Data Engineering Podcast

NOVEMBER 13, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Building 130

Building Metadata MongoDB MySQL 130

Data Engineering Podcast

DECEMBER 11, 2022

Summary One of the most critical aspects of software projects is managing its data. Managing the operational concerns for your database can be complex and expensive, especially if you need to scale to large volumes of data, high traffic, or geographically distributed usage. Closing Announcements Thank you for listening!

Database 100

Database MySQL Data Lake MongoDB 100

Databand.ai

AUGUST 30, 2023

Besides these categories, specialized solutions tailored specifically for particular domains or use cases also exist, such as ETL (Extract-Transform-Load) tools for managing data pipelines, data integration tools for combining information from disparate sources/systems, and more.

Data Cleanse 52

Data Cleanse Data Validation Data Pipeline Datasets 52

Data Engineering Podcast

DECEMBER 4, 2022

In this episode Arjun Narayan explains how the technical barriers to adopting real-time data in your analytics and applications have become surmountable by organizations of all sizes. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Data Lake 100

Data Lake MongoDB MySQL Data Warehouse 100

Data Engineering Podcast

NOVEMBER 6, 2022

He also describes the considerations involved in bringing behavioral data into your systems, and the ways that he and the rest of the Snowplow team are working to make that an easy addition to your platforms. Especially once they realize 90% of all major data sources like Google Analytics, Salesforce, Adwords, Facebook, Spreadsheets, etc.,

Building 130

Building IT Metadata MongoDB 130

Data Engineering Podcast

DECEMBER 4, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Business Intelligence 100

Business Intelligence Metadata BI MongoDB 100

Databand.ai

JULY 17, 2023

DataOps , short for Data Operations, is an emerging discipline that focuses on improving the collaboration, integration, and automation of data management processes. It aims to streamline the entire data lifecycle—from ingestion and preparation to analytics and reporting.

Data Pipeline 52

Data Pipeline Machine Learning High Quality Data BI 52

Databand.ai

JULY 11, 2023

By automating many of the processes involved in data quality management, data quality platforms can help organizations reduce errors, streamline workflows, and make better use of their data assets. Support and services: Finally, consider the level of support and services offered by the data quality platform vendor.

Data Cleanse 52

Data Cleanse Telecommunication High Quality Data BI 52

Towards Data Science

FEBRUARY 6, 2024

Order snapshots are stored in my own development area (image by the author) To prevent my extractions from impacting performance on the operational side, I queried this data regularly and stored it in a persistent staging area (PSA) within my data warehouse.

Systems 83

Systems Raw Data Metadata Data Cleanse 83

Databand.ai

JULY 19, 2023

This feature is critical in today’s data-driven business environment, where data may originate from a variety of sources and undergo numerous transformations before reaching its final destination. Metadata Management Metadata, or ‘data about data’, is a crucial component of data management.

Pipeline-centric 52

Pipeline-centric Data Governance Metadata Government 52

Hevo

APRIL 12, 2024

About the Author Mona Rakibe is the co–founder, and CEO of Telmai, a low-code data reliability platform designed for open architecture, i.e., any batch/streaming source of your data pipeline. As we hurtle […]

Data Pipeline 40

Data Pipeline Architecture Data Cloud 40

Snowflake

APRIL 8, 2024

Many entries also used Snowpark , taking advantage of the ability to work in the code they prefer to develop data pipelines, ML models and apps, then execute in Snowflake. Our sincere thanks go out to everyone who participated in this year’s competition.

Pipeline-centric 125

Pipeline-centric Food Healthcare Unstructured Data 125

Ascend.io

OCTOBER 20, 2023

While Snowflake offers comprehensive guidance on optimizing warehouse sizes, queries, and tables, among others, there’s a significant area they touch upon less: the optimization of data pipelines. These pipelines form the very backbone of how businesses process, analyze, and act on their data.

Pipeline-centric 52

Pipeline-centric IT Data Pipeline Bytes 52

Ascend.io

JANUARY 10, 2024

Visual representation of Conway’s Law ( source ) Read More: The Chief AI Officer: Avoid The Trap of Conway’s Law Process: Ensuring Data Readiness The backbone of successful AI implementation is robust data management processes. AI models are only as good as the data they consume, making continuous data readiness crucial.

Data Pipeline 52

Data Pipeline Government Data Governance Raw Data 52

Data Engineering Podcast

OCTOBER 16, 2022

In this episode Adrian Kosowski explains how the Pathway product got started, how its design simplifies the creation of data products that support supply chain operations, and how developers can help to build an ecosystem of applications that allow businesses to accelerate their time to insight. Closing Announcements Thank you for listening!

Database 100

Database Metadata MongoDB MySQL 100

Data Engineering Podcast

OCTOBER 2, 2022

In this episode Roambee CEO, Sanjay Sharma, shares the types of questions that companies are asking about their logistics, the technical work that they do to provide ways to answer those questions, and how they approach the challenge of data quality in its many forms. Can you describe what Roambee is and the story behind it?

Metadata 100

Metadata Electronics MongoDB MySQL 100

Data Engineering Podcast

JANUARY 23, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Building 100

Building Management Data Pipeline Metadata 100

Data Engineering Podcast

NOVEMBER 6, 2022

In this episode she shares the story behind the project, the details of how it is implemented, and how you can use it for your own data projects. Their SDKs make event streaming from any app or website easy, and their extensive library of integrations enable you to automatically send data to hundreds of downstream tools.

MongoDB 130

MongoDB Scala MySQL Data Lake 130

Data Engineering Podcast

SEPTEMBER 25, 2022

In this episode Tom Baeyens explains their reasons for creating a new syntax for expressing and validating checks for data assets and processes, as well as how to incorporate it into your own projects. Especially once they realize 90% of all major data sources like Google Analytics, Salesforce, Adwords, Facebook, Spreadsheets, etc.,

Building 100

Building Metadata MongoDB MySQL 100

Data Engineering Podcast

FEBRUARY 20, 2022

He also explains how he has architected the systems that ingest, process, and distribute the data that he is responsible for and the requirements that are introduced when collaborating with researchers, domain experts, and machine learning developers. RudderStack’s smart customer data pipeline is warehouse-first.

Systems 100

Systems Software Engineer Software Engineering Data Warehouse 100

Jesse Anderson

NOVEMBER 14, 2023

First of all, in data science, data discovery means finding patterns in data using database query languages to test hypotheses. This kind of data discovery can be subdivided into several steps, as e.g. suggested by Piethein Strengholt in Data Management at Scale.

Metadata 147

Metadata Data Science Big Data Data 147