Data Engineering Podcast

SEPTEMBER 3, 2023

Summary Cloud data warehouses and the introduction of the ELT paradigm has led to the creation of multiple options for flexible data integration, with a roughly equal distribution of commercial and open source options. What are the interfaces available for extending/customizing/integrating with dlt?

Data Integration 162

Data Integration BI SQL Python 162

Edureka

APRIL 24, 2024

These best DevOps practices will give you a starting point of how to approach DevOps integration into your processes, too! You will understand how a DevOps pipeline can be imagined for existing infrastructure. These pipelines provide better feedback and faster releases, enabling teams to perform better.

Coding 52

Coding Project Process Architecture 52

Netflix Tech

NOVEMBER 14, 2023

Furthermore, we’ll delve into the inner workings of Psyberg, its unique features, and how it integrates into our data pipelining workflows. By the end of this series, we hope you will gain an understanding of how Psyberg transformed our data processing, making our pipelines more efficient, accurate, and timely. Let’s dive in!

Data Engineering 105

Data Engineering Data Engineer Engineering Metadata 105

Precisely

DECEMBER 28, 2023

That’s where streaming data pipelines come into play. This article explores what streaming data pipelines are, how they work, and how to build this data pipeline architecture. What is a streaming data pipeline? How do streaming data pipelines work?

Data Pipeline 91

Data Pipeline Building Kafka Big Data 91

Snowflake

APRIL 4, 2024

The Snowflake Data Cloud includes powerful capabilities for transforming data and orchestrating data pipelines, and we partner with best-in-class providers to give customers a choice in the data transformation technologies they use. The Coalesce platform is uniquely built for Snowflake.

Cloud 98

Cloud Management Data Pipeline SQL 98

Workfall

SEPTEMBER 18, 2023

This is where the power of data integration comes into play. Meet Airbyte, the data magician that turns integration complexities into child’s play. Airbyte is an open-source data integration platform that allows you to collect and move data from various sources to a destination of your choice.

Data Pipeline 93

Data Pipeline Raw Data Data Schemas Healthcare 93

KDnuggets

DECEMBER 21, 2023

With Scikit-LLM, we could integrate the LLM easily into our ML pipeline. LLM is a powerful model that could improve our text analysis.

115

115

Knowledge Hut

FEBRUARY 6, 2024

The DevOps Pipeline is the sequence of activities that flow from a customer's idea to the delivery of software and services. The DevOps pipeline aims to improve software products' quality and delivery speed. It will also explore the tools available for building and maintaining an effective DevOps pipeline.

Building 52

Building AWS Coding Certification 52

Ascend.io

APRIL 22, 2024

As companies scale their data and analytics capabilities, their data pipelines and operations have grown increasingly complex. But as they add more data sources, implement real-time streaming pipelines, and build out a modern data lake, the complexity compounds quickly. It’s time we stop Data Plumbing, and get back to Data Engineering.

Project 52

Project Metadata Data Pipeline Data Engineering 52

Edureka

APRIL 17, 2024

CI/CD Pipeline using Azure In this project, you will use the live Azure DevOps environment. Here, you’ll set up a CI/CD pipeline. You’ll set up source control, do build, and release pipeline configurations and automation that will fully package your application for deployment. 2: What is the main goal of DevOps?

Project 52

Project Java AWS Coding 52

LinkedIn Engineering

OCTOBER 19, 2023

Authors: Bingfeng Xia and Xinyu Liu Background At LinkedIn, Apache Beam plays a pivotal role in stream processing infrastructures that process over 4 trillion events daily through more than 3,000 pipelines across multiple production data centers. Additionally, they needed the ability to experiment with streaming pipelines in batch mode.

Process 119

Process Lambda Architecture Kafka Machine Learning 119

Netflix Tech

NOVEMBER 14, 2023

By Abhinaya Shetty , Bharath Mummadisetty In the inaugural blog post of this series, we introduced you to the state of our pipelines before Psyberg and the challenges with incremental processing that led us to create the Psyberg framework within Netflix’s Membership and Finance data engineering team.

Data Process 90

Data Process Process Metadata Finance 90

ThoughtSpot

MARCH 5, 2024

Managing complex data pipelines is a major challenge for data-driven organizations looking to accelerate analytics initiatives. Snowflake's new dynamic tables feature redefines how BI and analytics teams approach data transformation pipelines. Now, that’s changing. Now, that’s changing.

BI 94

BI Datasets Raw Data SQL 94

Knowledge Hut

JANUARY 11, 2024

To meet this demand, implementing a robust CI/CD ( Continuous Integration and Continuous Deployment ) pipeline has become essential. This blog aims to provide a comprehensive understanding of CI/CD pipelines, their components, and their functioning. What is the CI/CD Pipeline?

Coding 52

Coding AWS Utilities Programming Language 52

Knowledge Hut

MARCH 28, 2024

The DevOps specialists are essential to modern software development pipelines because of their wide range of tools and services for creating, testing, and launching apps. These positions cover various duties, such as configuration management, continuous integration and deployment, infrastructure management, and development team cooperation.

Amazon Web Services 52

Amazon Web Services Computer Science Consulting Google Cloud 52

Snowflake

APRIL 8, 2024

The list of Top 10 semi-finalists is a perfect example: we have use cases for cybersecurity, gen AI, food safety, restaurant chain pricing, quantitative trading analytics, geospatial data, sales pipeline measurement, marketing tech and healthcare.

Pipeline-centric 117

Pipeline-centric Food Healthcare Unstructured Data 117

Monte Carlo

JANUARY 3, 2024

And in the long run, a quick integration with the OpenAI API won’t cut it. And because it’s early, the generative AI features that have been integrated are just “ChatGPT but over here.” Sure, you could integrate your AI model more deeply into your organization’s processes, but that feels risky. It’s GenAI, but where’s the moat?

Technology 113

Technology Database Data Governance Data Engineering 113

Monte Carlo

MARCH 4, 2024

This article will define DataOps and the DataOps lifecycle, and dive into how data observability fits into DataOps for comprehensive data pipeline management. DataOps is a strategic initiative that merges data engineering and data science teams to build and manage data pipelines. (And when is Beyonce’s next album coming out anyway?)

Data Pipeline 52

Data Pipeline Data Data Science Software Engineer 52

Snowflake

OCTOBER 3, 2023

We are excited to announce the availability of data pipelines replication, which is now in public preview. In the event of an outage, this powerful new capability lets you easily replicate and failover your entire data ingestion and transformations pipelines in Snowflake with minimal downtime.

Data Pipeline 76

Data Pipeline Management Data Ingestion Data 76

Snowflake

DECEMBER 5, 2023

Here are highlights from the collaborations, integrations and product enhancements that we were proud to dig in to throughout the week. By integrating Snowflake with LLMs from external services like Amazon Bedrock, developers can choose from a wide array of LLMs to plug-and-play based on their specific use cases.

AWS 90

AWS Amazon Web Services Government Cloud Computing 90

Knowledge Hut

MARCH 27, 2024

Release Management Tools automate repetitive tasks, reduce manual errors, and provide visibility into the release pipeline. Azure Pipelines Features: Azure Pipelines builds and deploys code in any language on any platform. Users can initiate Continuous Integration (CI) and Continuous Delivery (CD) release pipelines.

Management 52

Management Certification Utilities Coding 52

Precisely

APRIL 11, 2024

As a Technical Architect at Precisely, I’ve had the unique opportunity to lead the AWS Mainframe Modernization Data Replication for IBM i initiative, a project that not only challenged our technical capabilities but also enriched our understanding of cloud integration complexities.

AWS 72

AWS Amazon Web Services Cloud Cloud Computing 72

Towards Data Science

OCTOBER 25, 2023

You probably work closely with the colleague who builds the data pipeline that you were consuming. Decentralization allows organizations to scale data management but brings new challenges in ensuring the coordination of different components, such as data products and pipelines developed and managed by various teams.

Data Pipeline 81

Data Pipeline SQL Data Architecture Data 81

Knowledge Hut

MARCH 19, 2024

I Integrated AWS CloudWatch for monitoring and AWS Simple Notification Service (SNS) for effective alert notifications. Solution: Integration of Amazon OpenSearch Service: Dream11 strategically chose Amazon OpenSearch Service to address its performance and cost optimization goals.

Cloud Computing 52

Cloud Computing Cloud Amazon Web Services AWS 52

DataKitchen

APRIL 11, 2024

DataOps and Its Benefits: The concept of DataOps was introduced, drawing parallels with DevOps in terms of its emphasis on automation, continuous integration, and providing a high-quality data production pipeline. The goal is to reduce errors and operational overhead, allowing data teams to focus on delivering value.

Manufacturing 62

Manufacturing High Quality Data Data Pipeline Data 62

Data Engineering Weekly

MAY 16, 2023

Instead, Let’s examine the current data pipeline architecture and ask why data quality is expensive. Instead of looking at the implementation of the data quality frameworks, Let's examine the architectural patterns of the data pipeline. But before doing that, let's revisit some of the basic theories of the data pipeline.

Engineering 110

Engineering Kafka Data Pipeline Data Warehouse 110

Ascend.io

DECEMBER 19, 2023

Our focus is to make your decision-making process smoother, helping you understand how to best integrate ETL into your data strategy. That’s what we call a data pipeline. The question of whether to use a separate ETL tool when Snowflake provides a suite of native capabilities for data pipeline management is a nuanced one.

ETL Tools 52

ETL Tools IT Data Pipeline Data Warehouse 52

Snowflake

SEPTEMBER 7, 2023

Snowpark is the set of libraries and runtimes that enables data engineers, data scientists and developers to build data engineering pipelines, ML workflows, and data applications in Python, Java, and Scala. Snowpark External Access is leveraged to build a Ingest and Reverse ETL data pipeline for production workload.

Amazon Web Services 108

Amazon Web Services AWS Government Python 108

Snowflake

OCTOBER 23, 2023

Python’s popularity has more than doubled in the past decade¹ and it is quickly becoming the preferred language for development across machine learning, application development, pipelines, and more. One of our goals at Snowflake is to ensure we continue to deliver a best-in-class platform for Python developers.

Python 138

Python Cloud Data Science Data Architecture 138

Monte Carlo

JANUARY 29, 2024

Improve coverage with automated anomaly detection Monte Carlo uses machine learning detectors to monitor the health of data pipelines across dimensions like: Data freshness : Did the data arrive when we expected? This is a visual representation of the data as it moves through your pipelines. Volume: Are there too many or too few rows?

Metadata 64

Metadata High Quality Data Data Pipeline Machine Learning 64

Monte Carlo

MARCH 25, 2024

.” Justin Swenson Data, Analytics, and AI Governance Lead, Sub-Zero That means Monte Carlo needs to support and integrate across those same solutions and architectures. Deepening our integrations with modern orchestration tools like dbt and Airflow. More pipelines, more data sources, and more consumers.

MySQL 64

MySQL Government Data Governance Kafka 64

Workfall

AUGUST 14, 2023

Reading Time: 7 minutes In today’s data-driven world, efficient data pipelines have become the backbone of successful organizations. These pipelines ensure that data flows smoothly from various sources to its intended destinations, enabling businesses to make informed decisions and gain valuable insights. What is Airflow?

Data Pipeline 62

Data Pipeline Raw Data Data Database 62

Ascend.io

NOVEMBER 28, 2023

The terms ‘data orchestration’ and ‘data pipeline orchestration’ are often used interchangeably, yet they diverge significantly in function and scope. In contrast, data pipeline orchestration is a more targeted approach. It zeroes in on the specific tasks required to build, operate, and manage data pipelines.

Data Pipeline 52

Data Pipeline IT Data Metadata 52

Monte Carlo

FEBRUARY 27, 2024

On the other, business leaders are realizing that slapping a quick ChatGPT integration into their product isn’t going to cut it — and they’re looking to data engineers for a better solution. They need robust data pipelines, high-quality data, well-guarded privacy, and cost-effective scalability.

Data Engineering 59

Data Engineering Data Engineer Engineering High Quality Data 59

Monte Carlo

MARCH 28, 2024

DevOps focuses on service uptime, continuous integration, continuous deployment without breaking, container orchestration, security, and more. Develop This phase consists of coding, writing, unit testing, reviewing, and integrating code with the existing system. This is where continuous deployment and continuous integration arises.

Data Pipeline 52

Data Pipeline Coding Machine Learning Data Warehouse 52

Monte Carlo

JANUARY 19, 2024

Just because it looked good yesterday doesn’t mean it’ll hold up tomorrow – and that’s why we’re talking about data integrity testing today. Data integrity testing is the process of ensuring data is fit for the task at hand and available to only those who should have access. Well, the same goes for your data.

Data Integration 52

Data Integration Retail SQL Data Pipeline 52

Monte Carlo

FEBRUARY 12, 2024

Lineage : A map of how and where data is flowing and changing across your pipelines. Our Vision for Vector Data Observability Just like any database, vector databases involve data pipelines that ingest and consume data. If the pipeline is disrupted for whatever reason then there is a risk of unavailable or inaccurate information.

Database 72

Database Unstructured Data Data Pipeline Metadata 72