Christophe Blefari

OCTOBER 9, 2023

Tech and data engineering stuff ⚙️ CRON jobs at Slack scale — Why do you need an orchestrator when you can run CRONs? Data modeling is dead! Long live data modeling! — Joe Reis Keynote at Big Data London about his next book topic: data modeling. Goodbye Spark.

Python 130

Python Data Database Java 130

Data Engineering Podcast

MAY 22, 2022

Summary Machine learning has become a meaningful target for data applications, bringing with it an increase in the complexity of orchestrating the entire data flow. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform.

Machine Learning 130

Machine Learning Data Engineering Data Engineer Cloud 130

Data Engineering Podcast

JULY 24, 2022

Summary The current stage of evolution in the data management ecosystem has resulted in domain and use case specific orchestration capabilities being incorporated into various tools. In this episode Nick Schrock discusses the importance of orchestration and a central location for managing data systems, the road to Dagster’s 1.0

MongoDB 100

MongoDB Scala MySQL Data Lake 100

Lyft Engineering

OCTOBER 3, 2023

High-quality data is necessary for the success of every data-driven company. It is now the norm for tech companies to have a well-developed data platform. This makes it easy for engineers to generate, transform, store, and analyze data at the petabyte scale. What and Where is Data Quality?

Big Data 84

Big Data Metadata Data Warehouse Data 84

Snowflake

MARCH 5, 2024

Because human-machine interaction using natural language is now possible with large language models (LLMs), more data teams and developers can bring AI to their daily workflows. All that an analyst or any other user familiar with SQL needs to do is point the specific function below to a column of a table containing text data and voila!

Government 94

Government AWS Data Preparation Data Governance 94

Data Engineering Weekly

FEBRUARY 18, 2024

RudderStack is the Warehouse Native CDP, built to help data teams deliver value across the entire data activation lifecycle, from collection to unification and activation. Our hope is only with the amazing community of data practitioners who constantly support us. We are so over the Big Data Era to Modern Data Stack.

Data Engineering 62

Data Engineering Data Engineer Engineering Data 62

Data Engineering Podcast

JULY 17, 2022

Summary There are extensive and valuable data sets that are available outside the bounds of your organization. Whether that data is public, paid, or scraped it requires investment and upkeep to acquire and integrate it with your systems. Atlan is the metadata hub for your data ecosystem.

Data Management 130

Data Management Management Metadata MongoDB 130

DataKitchen

SEPTEMBER 20, 2021

Data organizations don’t always have the budget or schedule required for DataOps when conceived as a top-to-bottom, enterprise-wide transformational change. DataOps can and should be implemented in small steps that complement and build upon existing workflows and data pipelines. Production DataOps. Source: DataKitchen.

Data Pipeline 98

Data Pipeline Data Cleanse Process Architecture 98

Christophe Blefari

MARCH 24, 2023

I really liked the presentations from Mage and Kestra and even if I was focus on hosting the event it was great to see 2 other visions about the future of orchestration. Out of the box Mage provide all-in-one web editor to write data pipelines with a great UX. Kestra is a YAML-based data pipeline tool mixed with string templating.

Lambda Architecture 130

Lambda Architecture Data Pipeline Data SQL 130

Christophe Blefari

OCTOBER 20, 2023

Still dbt Labs announcements were mainly towards dbt Cloud with great features to drive adoption of the paid product. They announced dbt Mesh a product enabling cross-project dependencies for teams with multiple dbt projects. This post gives great insights about the impact on the data platform team.

Generalist 130

Generalist Entertainment NoSQL Datasets 130

Cloudera

JULY 13, 2021

After the launch of CDP Data Engineering (CDE) on AWS a few months ago, we are thrilled to announce that CDE, the only cloud-native service purpose built for enterprise data engineers, is now available on Microsoft Azure. . CDP data lifecycle integration and SDX security and governance. Key features of CDP Data Engineering.

Data Engineering 129

Data Engineering Data Engineer Engineering Pipeline-centric 129

DataKitchen

AUGUST 9, 2021

Below is our third post (3 of 5) on combining data mesh with DataOps to foster greater innovation while addressing the challenges of a decentralized architecture. We’ve talked about data mesh in organizational terms (see our first post, “ What is a Data Mesh? ”) and how team structure supports agility. Source: Thoughtworks.

Pharmaceutical 98

Pharmaceutical Raw Data Data Lake Data 98

Monte Carlo

MARCH 4, 2024

Well, we can liken the Data Observability vs. DataOps relationship to Beyonce (there’s a sentence I never expected to write!). Just as every Beyonce song is a hit, but not every hit song is by Beyonce, data observability is one key component of DataOps, but DataOps is more than just data observability. What is DataOps?

Data Pipeline 52

Data Pipeline Data Data Science Software Engineer 52

DataKitchen

APRIL 28, 2021

Industry analysts who follow the data and analytics industry tell DataKitchen that they are receiving inquiries about “data fabrics” from enterprise clients on a near-daily basis. Gartner included data fabrics in their top ten trends for data and analytics in 2019. What is a Data Fabric?

Data Pipeline 98

Data Pipeline Data Data Analytics Architecture 98

Cloudera

DECEMBER 21, 2021

Since the release of Cloudera Data Engineering (CDE) more than a year ago , our number one goal was operationalizing Spark pipelines at scale with first class tooling designed to streamline automation and observability. We also introduced Apache Airflow on Kubernetes as the next generation orchestration service.

Data Engineering 120

Data Engineering Data Engineer Engineering Data Pipeline 120

Netflix Tech

MAY 4, 2023

Behind the scenes, a myriad of systems and services are involved in orchestrating the product experience. These backend systems are consistently being evolved and optimized to meet and exceed customer and product expectations. It provides a good read on the availability and latency ranges under different production conditions.

Utilities 135

Utilities Systems Architecture Coding 135

Snowflake

JANUARY 31, 2024

Because Snowflake Native Apps run within the customer’s Snowflake account, the customer doesn’t need to move or copy their data, which increases security and helps maintain governance controls. Browse Cybersyn’s data products on Snowflake Marketplace. “Maxa’s mission is to reinvent the way ERP insights are made and consumed.

AWS 99

AWS Government Programming Data Science 99

Edureka

APRIL 19, 2024

The personnel in this role may be the first ones to incorporate cutting-edge technology to help enhance operational efficiency and reduce the time taken to develop new software products. On the other hand, according to the data collected by Google , 86% of companies deem DevOps valuable in the efficient production of innovative software.

Portfolio 52

Portfolio Recruitment AWS Electronics 52

DataKitchen

FEBRUARY 23, 2024

DataKitchen Resource Guide To Data Journeys & Data Observability & DataOps Data (and Analytic) Observability & Data Journey – Ideas and Background Data Journey Manifesto and Why the Data Journey Manifesto?

Data Validation 48

Data Validation Data Data Pipeline Data Solutions 48

Cloudera

JUNE 2, 2021

Data pipelines are in high demand in today’s data-driven organizations. As critical elements in supplying trusted, curated, and usable data for end-to-end analytic and machine learning workflows, the role of data pipelines is becoming indispensable.

Data Pipeline 93

Data Pipeline Data Warehouse Machine Learning Data Architect 93

Data Engineering Weekly

MARCH 17, 2024

Compliance is mandatory, with strict penalties for violations, emphasizing the importance of data scientists familiarizing themselves with the law to avoid prohibited AI uses and ensure ethical, safe AI development. It discusses the significance of data governance, sharing history, and generative AI's impact on data economy standards.

Data Engineering 70

Data Engineering Data Engineer Engineering Data Governance 70

Mutt Data

OCTOBER 19, 2023

About The Company MODO is a product of Play Digital S.A., Challenge Modo’s expanding operations and data demands required an enhanced data platform that could manage the increasing volume, variety, and necessary features of the business. It offers three services: money transfers, money requests, and payments through QR codes.

Banking 52

Banking Healthcare Designing Data Engineering 52

Mutt Data

OCTOBER 19, 2023

About The Company MODO is a product of Play Digital S.A., Challenge Modo’s expanding operations and data demands required an enhanced data platform that could manage the increasing volume, variety, and necessary features of the business. It offers three services: money transfers, money requests, and payments through QR codes.

Banking 52

Banking Healthcare Designing Data Engineering 52

Yelp Engineering

JANUARY 29, 2023

Amongst many, one case is to eliminate defective products automatically from reaching the finished goods inventory. The same principles of these systems can be adopted to filter out malformed data from datastores. Yelp orchestrates Cassandra clusters on Kubernetes. Many different features on Yelp are powered by Cassandra.

Data Pipeline 75

Data Pipeline NoSQL Manufacturing Data 75

Workfall

AUGUST 14, 2023

Reading Time: 7 minutes In today’s data-driven world, efficient data pipelines have become the backbone of successful organizations. These pipelines ensure that data flows smoothly from various sources to its intended destinations, enabling businesses to make informed decisions and gain valuable insights. What is Airflow?

Data Pipeline 62

Data Pipeline Raw Data Data Database 62

Pinterest Engineering

OCTOBER 31, 2023

In this article, we discuss the PinCompute primitives, architecture, control plane and data plane capabilities, and showcase the value that PinCompute has delivered for innovation and efficiency at Pinterest. Architecture PinCompute is a regional Platform-as-a-Service (PaaS) that builds on top of Kubernetes.

Architecture 96

Architecture Pipeline-centric Accessible Accessibility 96

Cloudera

SEPTEMBER 26, 2023

Organizations increasingly rely on streaming data sources not only to bring data into the enterprise but also to perform streaming analytics that accelerate the process of being able to get value from the data early in its lifecycle.

Kafka 95

Kafka Technology IT Government 95

Data Engineering Weekly

AUGUST 6, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles, so you can quickly ship actionable, enriched data to every downstream team. See how it works today. Editor’s Note: DewCon.ai Registration is Now Open Great news!

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

Ascend.io

AUGUST 15, 2023

This same principle holds true in data management. You require a comprehensive solution that addresses every facet, from ingestion and transformation to orchestration and reverse ETL. It’s no surprise, then, that the quest for Fivetran alternatives is on the rise as organizations set their sights on a more holistic data approach.

Data Management 52

Data Management Management Data Ingestion Data Pipeline 52

Knowledge Hut

MARCH 14, 2024

Blue-Green Deployment: Definition: Blue-Green deployment maintains two identical production environments, allowing seamless version switching. Advantages: Data-Driven Decisions: Helps make data-driven decisions based on user feedback and behavior. Use Case: Ideal for applications with complex integrations and dependencies.

Utilities 52

Utilities Software Engineer Software Engineering Systems 52

DataKitchen

AUGUST 19, 2021

Below is our fourth post (4 of 5) on combining data mesh with DataOps to foster innovation while addressing the challenges of a decentralized architecture. We’ve covered the basic ideas behind data mesh and some of the difficulties that must be managed. Below is a discussion of a data mesh implementation in the pharmaceutical space.

Pharmaceutical 98

Pharmaceutical Data Lake Data Warehouse Raw Data 98

Monte Carlo

MARCH 25, 2024

When we launched the data observability category in 2020, we set out to solve a very real problem: data trust. In the preceding months, I met with hundreds of data leaders about what kept them up at night. What if there was a way to ensure that your data was trustworthy and reliable? The list goes on.

MySQL 64

MySQL Government Data Governance Kafka 64

Cloudera

OCTOBER 19, 2021

Airflow has been adopted by many Cloudera Data Platform (CDP) customers in the public cloud as the next generation orchestration service to setup and operationalize complex data pipelines. This makes our pipeline engine flexible to support multitude of orchestration services. Easing development friction.

Coding 119

Coding Data Engineering Data Engineer Engineering 119

DataKitchen

JANUARY 25, 2022

Analytics are prone to frequent data errors and deployment of analytics is slow and laborious. When internal resources fall short, companies outsource data engineering and analytics. There’s no shortage of consultants who will promise to manage the end-to-end lifecycle of data from integration to transformation to visualization. .

Consulting 97

Consulting Recruitment Data Lake Data Engineering 97

Knowledge Hut

MARCH 27, 2024

Release Management Tools play a vital role in orchestrating and automating the release process, ensuring smooth deployment. In our Release Management Tools List of 2024, we're here to help you minimize release failures and increase productivity to meet business demands. Customer Rating: G2: 4.2/5 5 (4,490+ reviews) Capterra: 4.4/5

Management 52

Management Certification Utilities Coding 52

The Modern Data Company

APRIL 15, 2024

The elegance of Data Products is undeniable, but many leaders question the efficacy of their data strategies: Why does the return on data investments often disappoint? Why is proving data’s value becoming harder? Why do data models become more cumbersome than beneficial? What Exactly Is a Data Product?

Government 52

Government Metadata Data Data Lake 52

DareData

NOVEMBER 28, 2023

In today's data-driven world, machine learning has emerged as a transformative force, empowering organizations to extract valuable insights from vast amounts of data. As the scope of the models and the data continues to scale, the role of a Data Scientist has evolved accordingly in the last years. But that is not all!

Data Science 52

Data Science Medical Machine Learning Data 52