Ascend.io

MARCH 17, 2023

Theoretically, data and analytics should be the backbones of decision-making in business. Like mitochondria power a cell, data powers a business. Today, there are no intelligent systems that deliver data at the pace, and with the impact, leaders need to power the business. But for most companies, that’s not the reality.

Data Pipeline 52

Data Pipeline Datasets Data Software Engineer 52

Precisely

MAY 13, 2024

Key Takeaways: Data integration is vital for real-time data delivery across diverse cloud models and applications, and for leveraging technologies like generative AI. As enterprise technology landscapes grow more complex, the role of data integration is more critical than ever before.

Data Integration 52

Data Integration Metadata Amazon Web Services Data Governance 52

Cloudera

FEBRUARY 8, 2021

You can read part 1, here: Digital Transformation is a Data Journey From Edge to Insight. The first blog introduced a mock connected vehicle manufacturing company, The Electric Car Company (ECC), to illustrate the manufacturing data path through the data lifecycle. 1 The enterprise data lifecycle.

Data Pipeline 116

Data Pipeline Building Manufacturing Data Warehouse 116

Christophe Blefari

JANUARY 20, 2024

Learn data engineering, all the references ( credits ) This is a special edition of the Data News. But right now I'm in holidays finishing a hiking week in Corsica 🥾 So I wrote this special edition about: how to learn data engineering in 2024. The idea is to create a living reference about Data Engineering.

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Data Engineering Podcast

NOVEMBER 20, 2022

Sifflet is a platform that brings your entire data stack into focus to improve the reliability of your data assets and empower collaboration across your teams. In this episode CEO and founder Salma Bakouk shares her views on the causes and impacts of "data entropy" and how you can tame it before it leads to failures.

Data Lake 130

Data Lake Data Ingestion MongoDB Scala 130

Towards Data Science

NOVEMBER 4, 2023

Platform Specific Tools and Advanced Techniques Photo by Christopher Burns on Unsplash The modern data ecosystem keeps evolving and new data tools emerge now and then. In this article, I want to talk about crucial things that affect data engineers. Are your data pipelines efficient? Data warehouse exmaple.

Data Engineering 83

Data Engineering Data Engineer Engineering BI 83

Data Engineering Weekly

MAY 16, 2023

In the first part of this series, we talked about design patterns for data creation and the pros & cons of each system from the data contract perspective. In the second part, we will focus on architectural patterns to implement data quality from a data contract perspective. Why is Data Quality Expensive?

Engineering 110

Engineering Kafka Data Pipeline Data Warehouse 110

Data Engineering Podcast

JUNE 5, 2022

Summary The best way to make sure that you don’t leak sensitive data is to never have it in the first place. The team at Skyflow decided that the second best way is to build a storage system dedicated to securely managing your sensitive information and making it easy to integrate with your applications and data systems.

Data Security 100

Data Security Metadata MongoDB Scala 100

Ascend.io

FEBRUARY 6, 2023

If you’ve read this week’s release notes , you’ll have seen “Added ability to configure or override connection details at the Data Service level in BigQuery and Databricks Data Planes.” Data Services Listing with BigQuery, Snowflake, and Databricks. Data Service – is a group of Data Flows.

Cloud 52

Cloud Data Ingestion Raw Data Data Pipeline 52

Ascend.io

FEBRUARY 6, 2023

If you’ve read this week’s release notes , you’ll have seen “Added ability to configure or override connection details at the Data Service level in BigQuery and Databricks Data Planes.” Data Services Listing with BigQuery, Snowflake, and Databricks. Data Service – is a group of Data Flows.

Cloud 52

Cloud Data Ingestion Raw Data Data Pipeline 52

DataKitchen

OCTOBER 20, 2023

In today’s data-driven landscape, Data and Analytics Teams i ncreasingly face a unique set of challenges presented by Demanding Data Consumers who require a personalized level of Data Observability. Data Observability platforms often need to deliver this level of customization.

Insurance 67

Insurance Pharmaceutical Data Data Ingestion 67

Monte Carlo

MARCH 28, 2023

If you don’t like change, data engineering is not for you. The most prominent, recent examples are Snowflake and Databricks disrupting the concept of the database and ushering in the modern data stack era. As part of this movement, Fivetran and dbt fundamentally altered the data pipeline from ETL to ELT.

Data Warehouse 126

Data Warehouse Raw Data Data Pipeline Software Engineer 126

Monte Carlo

APRIL 4, 2023

The modern data stack consists of multiple interdependent data systems all working together. Data anomalies and incidents can be introduced across any point of this pipeline, which can make root cause analysis a fragmented and frustrating multi-tab process. Was it a query changed within Snowflake? A modified dbt model?

BI 64

BI Data Ingestion Metadata Data Pipeline 64

Ascend.io

JUNE 8, 2023

Because of its transformative potential, data has graduated from being merely a by-product of business operations to a critical asset in its own right. Is it possible to treat data not just as a necessary operational output, but as a product that holds immense strategic value? Without them, data products can’t exist.

Pipeline-centric 52

Pipeline-centric Database-centric Data Ingestion Data Pipeline 52

Netflix Tech

JANUARY 25, 2023

Scalable Annotation Service — Marken by Varun Sekhri , Meenakshi Jindal Introduction At Netflix, we have hundreds of micro services each with its own data models or entities. For example, we have a service that stores a movie entity’s metadata or a service that stores metadata about images. Allows to annotate any entity.

Algorithm 113

Algorithm Media Metadata Data Ingestion 113

Cloudera

JUNE 16, 2022

The ability to perform analytics on data as it is created and collected (a.k.a. real-time data streams) and generate immediate insights for faster decision making provides a competitive edge for organizations. . CSP was recently recognized as a leader in the 2022 GigaOm Radar for Streaming Data Platforms report.

Kafka 86

Kafka Manufacturing Data Lake SQL 86

Cloudera

DECEMBER 15, 2022

Since we announced the general availability of Apache Iceberg in Cloudera Data Platform (CDP), Cloudera customers, such as Teranet , have built open lakehouses to future-proof their data platforms for all their analytical workloads. Only metadata will be regenerated. Only metadata will be regenerated.

Cloud 78

Cloud Metadata Google Cloud Data Warehouse 78

Monte Carlo

NOVEMBER 14, 2023

Databricks announced that Delta tables metadata will also be compatible with the Iceberg format, and Snowflake has also been moving aggressively to integrate with Iceberg. How data observability can increase data trust in Iceberg tables What is Apache Iceberg? How Apache Iceberg tables structure metadata. Let’s dive in.

Data Lake 64

Data Lake Metadata Data Warehouse SQL 64

Data Engineering Weekly

OCTOBER 30, 2022

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. The highlights are that 59% of folks think data catalogs are sometimes helpful.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Ingestion 52

Cloudera

DECEMBER 20, 2021

The right set of tools helps businesses utilize data to drive insights and value. But balancing a strong layer of security and governance with easy access to data for all users is no easy task. Another option — a more rewarding one — is to include centralized data management, security, and governance into data projects from the start.

Government 89

Government Data Security Banking Metadata 89

Databand.ai

AUGUST 30, 2023

DataOps is a collaborative approach to data management that combines the agility of DevOps with the power of data analytics. It aims to streamline data ingestion, processing, and analytics by automating and integrating various data workflows. As a result, they can be slow, inefficient, and prone to errors.

Architecture 52

Architecture Data Ingestion Data Governance Data Cleanse 52

Databand.ai

AUGUST 30, 2023

DataOps , short for data operations, is an emerging discipline that focuses on improving the collaboration, integration, and automation of data processes across an organization. These tools help organizations implement DataOps practices by providing a unified platform for data teams to collaborate, share, and manage their data assets.

Data Cleanse 52

Data Cleanse Data Pipeline Data Ingestion Data Validation 52

Towards Data Science

APRIL 3, 2023

The post-modern data stack is coming. If you don’t like change, data engineering is not for you. The most prominent, recent examples are Snowflake and Databricks disrupting the concept of the database and ushering in the modern data stack era. The answer is, yes of course we will have to rebuild our data systems.

Data Engineering 98

Data Engineering Data Engineer Engineering Data Warehouse 98

Booking.com Engineering

DECEMBER 2, 2022

From data ingestion, data science, to our ad bidding[2], GCP is an accelerant in our development cycle, sometimes reducing time-to-market from months to weeks. Data Ingestion and Analytics at Scale Ingestion of performance data, whether generated by a search provider or internally, is a key input for our algorithms.

Systems 52

Systems Cloud MySQL Relational Database 52

Ascend.io

JUNE 21, 2023

Creating business value from the onslaught of data can feel like captaining a high-tech vessel through uncharted waters. Data teams across business areas are cranking out data sets in response to impatient business requests, while simultaneously trying to build disparate DataOps processes from scratch.

Pipeline-centric 52

Pipeline-centric Data Pipeline Metadata Data 52

Cloudera

MARCH 31, 2021

The addition of support for Google Cloud enables Cloudera to deliver on its promise to offer its enterprise data platform at a global scale. In this first Google Cloud release, CDP Public Cloud provides built-in Data Hub definitions (see screenshot for more details) for: Data Ingestion (Apache NiFi, Apache Kafka).

Google Cloud 92

Google Cloud Cloud Amazon Web Services Cloud Storage 92

AltexSoft

AUGUST 29, 2023

In 2010, a transformative concept took root in the realm of data storage and analytics — a data lake. The term was coined by James Dixon , Back-End Java, Data, and Business Intelligence Engineer, and it started a new era in how organizations could store, manage, and analyze their data. What is a data lake?

Data Lake 52

Data Lake Architecture IT Amazon Web Services 52

Data Engineering Podcast

AUGUST 6, 2022

Summary The optimal format for storage and retrieval of data is dependent on how it is going to be used. For analytical systems there are decades of investment in data warehouses and various modeling techniques. Data stacks are becoming more and more complex. Sifflet also offers a 2-week free trial.

Machine Learning 100

Machine Learning Database MySQL PostgreSQL 100

Netflix Tech

DECEMBER 21, 2020

By Anupom Syam Background At Netflix, our current data warehouse contains hundreds of Petabytes of data stored in AWS S3 , and each day we ingest and create additional Petabytes. Some of the optimizations are prerequisites for a high-performance data warehouse. Iceberg plans to enable this in the form of delta files.

Data Warehouse 81

Data Warehouse Metadata Algorithm Data 81

DataKitchen

SEPTEMBER 21, 2023

Bridging the Gap: How ‘Data in Place’ and ‘Data in Use’ Define Complete Data Observability In a world where 97% of data engineers report burnout and crisis mode seems to be the default setting for data teams, a Zen-like calm feels like an unattainable dream. What is Data in Place?

Raw Data 67

Raw Data Data Business Intelligence High Quality Data 67

Netflix Tech

MARCH 25, 2019

Building and Scaling Data Lineage at Netflix to Improve Data Infrastructure Reliability, and Efficiency By: Di Lin , Girish Lingappa , Jitender Aswani Imagine yourself in the role of a data-inspired decision maker staring at a metric on a dashboard about to make a critical business decision but pausing to ask a question?—?“Can

Building 87

Building Metadata Transportation Data Ingestion 87

Ascend.io

MAY 19, 2023

Picture a data engineering leader, like a captain at the helm of a ship, navigating the vast ocean of data. The modern data stack promised a vessel with cutting-edge navigation tools to sail smoothly through this ocean. Enter the post-modern data stack. This is the modern data stack as we know it today.

Data Pipeline 52

Data Pipeline Data Engineering Data Engineer Media 52

RandomTrees

SEPTEMBER 6, 2020

Snowflake Overview A data warehouse is a critical part of any business organization. Lot of cloud-based data warehouses are available in the market today, out of which let us focus on Snowflake. Snowflake is an analytical data warehouse that is provided as Software-as-a-Service (SaaS).

Cloud Storage 52

Cloud Storage Data Ingestion Data Cleanse Data Warehouse 52

Rockset

AUGUST 28, 2023

Lessons learned from @scale deployments I managed and scaled Facebook's online data infrastructure from 2007, when it had 30-40 million MAUs, to 2015 when it had 1.5 I saw first-hand the transformative power of having the right data stack. We believe any engineer in the world should be able to quickly build powerful data apps.

Metadata 78

Metadata Unstructured Data SQL Database 78

Monte Carlo

AUGUST 19, 2023

A data platform, often referred to as a ‘modern data stack,’ is the central processing hub for an organization’s data ecosystem. A comprehensive data platform solution powers data acquisition, storage, preparation, delivery, governance, and even the robust security needs of users and applications.

Building 52

Building BI Data Lake Data Governance 52

Silectis

JANUARY 3, 2021

If you’re new to data engineering or are a practitioner of a related field, such as data science, or business intelligence, we thought it might be helpful to have a handy list of commonly used terms available for you to get up to speed. Big Data Large volumes of structured or unstructured data.

Data Engineering 40

Data Engineering Data Engineer Engineering Hadoop 40

Rockset

NOVEMBER 28, 2022

Today, I’m excited to share a few product updates we’ve been working on related to real-time Change Data Capture (CDC), including early access for popular templates and 3rd-party CDC platforms. Because all databases make technical tradeoffs, it’s common to move data from a source to a destination based on how the data will be used.

Data Warehouse 52

Data Warehouse PostgreSQL MongoDB Data Pipeline 52