Christophe Blefari

MARCH 24, 2023

📺 Watch the full replay Here are my takeaways about the event: Mage and Kestra have been both developed with Airflow flaws in mind, especially about deployment complexity, reusability and data sharing between tasks. Out of the box Mage provide all-in-one web editor to write data pipelines with a great UX. led to the outage.

Lambda Architecture 130

Lambda Architecture Data Pipeline Data SQL 130

Towards Data Science

APRIL 26, 2023

An open-source implementation of a Data Lake with DuckDB and AWS Lambdas A duck in the cloud. Photo by László Glatz on Unsplash In this post we will show how to build a simple end-to-end application in the cloud on a serverless infrastructure. A lightinign fast analytics app built with our system.

Engineering 69

Engineering Data Lake AWS BI 69

Christophe Blefari

JANUARY 7, 2023

The blog crossed the 2000 members mark (❤️) and I won the best data science newsletter award. I talked in Robin's podcast about the newsletter and my data engineering journey. And you know what, this is the first time in 7 years of teaching that more than 80% of the class wants to become data engineer.

Data Science 130

Data Science Data BI Kafka 130

Data Engineering Podcast

AUGUST 27, 2021

Summary The reason that so much time and energy is spent on data integration is because of how our applications are designed. By making the software be the owner of the data that it generates, we have to go through the trouble of extracting the information to then be used elsewhere. No more scripts, just SQL.

Data Integration 100

Data Integration Architecture Data Warehouse Data Lake 100

Data Science Blog: Data Engineering

NOVEMBER 15, 2023

In addition to Business Intelligence (BI), Process Mining is no longer a new phenomenon, but almost all larger companies are conducting this data-driven process analysis in their organization. The Event Log Data Model for Process Mining Process Mining as an analytical system can very well be imagined as an iceberg.

Architecture 52

Architecture Database-centric Process BI 52

Cloudera

MARCH 2, 2021

However, in the typical enterprise, only a small team has the core skills needed to gain access and create value from streams of data. This data engineering skillset typically consists of Java or Scala programming skills mated with deep DevOps acumen. A rare breed. The difficulty with querying streams.

SQL 112

SQL Data Lake Java Scala 112

Data Engineering Podcast

MARCH 20, 2022

Summary Data and analytics are permeating every system, including customer-facing applications. The introduction of embedded analytics to an end-user product creates a significant shift in requirements for your data layer. So now your modern data stack is set up.

Datasets 100

Datasets Kafka Analytics Application SQL 100

Cloudera

AUGUST 13, 2021

Every enterprise is trying to collect and analyze data to get better insights into their business. Whether it is consuming log files, sensor metrics, and other unstructured data, most enterprises manage and deliver data to the data lake and leverage various applications like ETL tools, search engines, and databases for analysis.

Data Pipeline 109

Data Pipeline Data Lake ETL Tools Unstructured Data 109

Rockset

MARCH 10, 2022

This is the first post in a series by Rockset's CTO Dhruba Borthakur on Designing the Next Generation of Data Systems for Real-Time Analytics. He was an engineer on the database team at Facebook, where he was the founding engineer of the RocksDB data store. More on the downsides of this later.

Data Analytics 52

Data Analytics Data Warehouse Medical MySQL 52

Cloudera

JUNE 16, 2022

The ability to perform analytics on data as it is created and collected (a.k.a. real-time data streams) and generate immediate insights for faster decision making provides a competitive edge for organizations. . CSP was recently recognized as a leader in the 2022 GigaOm Radar for Streaming Data Platforms report.

Kafka 86

Kafka Manufacturing Data Lake SQL 86

Cloudera

SEPTEMBER 1, 2020

It is hard to believe if you have had previous experience with setting up, sizing, and deploying a distributed search engine service that this is possible. Imagine how many times IT has lost valuable time spending hours trying to understand Apache Solr application requirements and map them into how to best size and deploy the Solr service.

Cloud Storage 68

Cloud Storage Unstructured Data AWS Analytics Application 68

Knowledge Hut

JUNE 16, 2023

Do you know, " What is data science course ?" " Data science courses illustrate techniques for extracting insights and knowledge from data. Data analysis is a fundamental component of data science, focusing on exploring and understanding data through statistical and visual methods.

Data Analytics 52

Data Analytics Business Analyst Business Intelligence Consulting 52

Rockset

DECEMBER 9, 2019

As a data engineer, my time is spent either moving data from one place to another, or preparing it for exposure to either reporting tools or front end users. As data collection and usage have become more sophisticated, the sources of data have become a lot more varied and disparate, volumes have grown and velocity has increased.

Data Engineering 40

Data Engineering Data Engineer Raw Data Engineering 40

Cloudera

NOVEMBER 22, 2021

In legacy analytical systems such as enterprise data warehouses, the scalability challenges of a system were primarily associated with computational scalability, i.e., the ability of a data platform to handle larger volumes of data in an agile and cost-efficient way. Introduction. CRM platforms). CRM platforms).

Government 92

Government Hadoop Data Security Data Warehouse 92

Striim

AUGUST 31, 2023

Treating batch and streaming as separate pipelines for separate use cases drives up complexity, cost, and ultimately deters data teams from solving business problems that truly require data streaming architectures. Stream processors, storage layers, message brokers, and databases make up the basic components of this architecture.

Data Integration 69

Data Integration Architecture Amazon Web Services ETL System 69

Cloudera

NOVEMBER 2, 2020

Users today are asking ever more from their data warehouse. As an example of this, in this post we look at Real Time Data Warehousing (RTDW), which is a category of use cases customers are building on Cloudera and which is becoming more and more common amongst our customers. Ingest 100s of TB of network event data per day .

Data Warehouse 94

Data Warehouse Kafka Lambda Architecture Telecommunication 94

phData: Data Engineering

JULY 18, 2023

The Ultimate Modern Data Stack Migration Guide phData Marketing July 18, 2023 This guide was co-written by a team of data experts, including Dakota Kelley, Ahmad Aburia, Sam Hall, and Sunny Yan. Imagine a world where all of your data is organized, easily accessible, and routinely leveraged to drive impactful outcomes.

Data Warehouse 52

Data Warehouse Pipeline-centric Government Data 52

Cloudera

NOVEMBER 7, 2023

Are you struggling to manage the ever-increasing volume and variety of data in today’s constantly evolving landscape of modern data architectures? Each volume can contain multiple buckets of data. Keys store data inside buckets. Most traditional analytics applications like Hive, Spark, Impala, YARN etc.

Systems 103

Systems Hadoop Unstructured Data Media 103

ProjectPro

FEBRUARY 11, 2021

Spoiler Alert: Becoming a machine learning engineer can sound like a hard-to-reach goal but let us tell you the truth – it isn’t as hard as it seems. Image Credit: Makeameme.org So you are considering learning machine learning skills , and you’ve heard that becoming a machine learning engineer is the way to go.

Machine Learning 40

Machine Learning Engineering Programming Language Portfolio 40

Rockset

OCTOBER 26, 2023

JetBlue is the data leader in the airline industry using data to offer industry-leading customer experiences and disruptive low fares to popular destinations around the world. The complexity of the data and situation can be hard to quickly comprehend and take action on without the assistance of machine learning.

Machine Learning 64

Machine Learning Data Science Architecture Data Warehouse 64

Snowflake

MAY 4, 2023

The market volatility we’ve seen in recent years has only underscored the value of using data to model and analyze market behavior and avoid to manage risk for the Street. In the end, we want all of DTCC’s data securely accessible to our internal and external stakeholders. This presents what I like to call an “efficiency opportunity.”

Portfolio 52

Portfolio Cloud Analytics Application Data Security 52

Monte Carlo

MAY 14, 2024

As organizations seek greater value from their data, data architectures are evolving to meet the demand — and table formats are no exception. But while the modern data stack , and how it’s structured, may be evolving, the need for reliable data is not — and that also has some real implications for your data platform.

Data Lake 52

Data Lake Metadata Hadoop Data Governance 52

Rockset

MAY 3, 2023

Rockset is a database used for real-time search and analytics on streaming data. In scenarios involving analytics on massive data streams, we’re often asked the maximum throughput and lowest data latency Rockset can achieve and how it stacks up to other databases. Why measure streaming data ingestion?

Data Ingestion 59

Data Ingestion Kafka Database Architecture 59

Snowflake

JULY 13, 2023

ESO is the largest software and data solutions provider to emergency medical services (EMS) agencies and fire departments in the U.S. With a mission to improve community health and public safety through the power of data, ESO makes software that helps save lives. This data is also used for legal and regulatory purposes.

Medical 93

Medical Hospitality Scala Transportation 93

AltexSoft

JULY 18, 2023

These seemingly unrelated terms unite within the sphere of big data, representing a processing engine that is both enduring and powerfully effective — Apache Spark. Maintained by the Apache Software Foundation, Apache Spark is an open-source, unified engine designed for large-scale data analytics.

Big Data 52

Big Data Data Process Process Hadoop 52

Rockset

MARCH 3, 2022

The amount of data companies generate, transform, store and query is growing exponentially. This data has material financial value when it’s both fresh and easy to access, however, customers commonly face scalability challenges running both transactional and analytical applications on the same database. This makes sense.

Amazon Web Services 52

Amazon Web Services Data Warehouse SQL AWS 52

dbt Developer Hub

NOVEMBER 28, 2021

Before entering the field of data, I spent my entire career as a product manager working alongside user experience designers and engineers on cross-functional product teams. The more friction there is in analytics and the less we focus on the user, the less our output will be used. Stop at the data visualization.

Designing 52

Designing Analytics Application Building Data Analytics 52

phData: Data Engineering

MAY 11, 2023

Practices centered on software engineering principles can create a barrier to entry for teams with skilled data wranglers looking to take their infrastructure to the next level with cloud-native tools like Matillion for the Snowflake Data Cloud. Cyclic graphs” are unheard of in analytics.

Coding 52

Coding Software Engineer Software Engineering Government 52

Rockset

JANUARY 30, 2024

In 2023, Rockset announced a new cloud architecture for search and analytics that separates compute-storage and compute-compute. With this architecture, users can separate ingestion compute from query compute, all while accessing the same real-time data. minutes to batch load the data. minutes Ingestion compute saving (1-(( 16.7

Data Ingestion 64

Data Ingestion Utilities Architecture SQL 64

Rockset

DECEMBER 9, 2021

Not Just Modern, But Real Time The modern data stack emerged a decade ago, a direct response to the shortcomings of big data. Companies that undertook big data projects ran head-long into the high cost, rigidity and complexity of managing complex on-premises data stacks. The inevitable next stage?

Transportation 52

Transportation BI SQL Data Warehouse 52

Rockset

MAY 17, 2022

This is the fourth post in a series by Rockset's CTO Dhruba Borthakur on Designing the Next Generation of Data Systems for Real-Time Analytics. For instance, customer personalization systems need to combine historic data sets with real-time data streams to instantly provide the most relevant product recommendations to customers.

SQL 52

SQL NoSQL Hadoop MongoDB 52

Analytics Vidhya

FEBRUARY 28, 2023

Introduction Azure Databricks is a fast, easy, and collaborative Apache Spark-based analytics platform that is built on top of the Microsoft Azure cloud. A collaborative and interactive workspace allows users to perform big data processing and machine learning tasks easily.

Big Data 310

Big Data Machine Learning Cloud Data Process 310

Knowledge Hut

MAY 3, 2024

One of the most important decisions for Big data learners or beginners is choosing the best programming language for big data manipulation and analysis. Java does not support Read-Evaluate-Print-Loop (REPL), which is a major deal-breaker when choosing a programming language for big data processing.

Scala 52

Scala Java Python Programming Language 52

Rockset

FEBRUARY 24, 2023

Introduction Let’s get this out of the way at the beginning: understanding effective streaming data architectures is hard, and understanding how to make use of streaming data for analytics is really hard. What Is Streaming Data? We’re going to start with a basic question: what is streaming data? Kafka or Kinesis ?

Kafka 52

Kafka AWS Amazon Web Services Programming Language 52

Knowledge Hut

MARCH 15, 2024

These services are intended to assist people and organizations in developing, implementing, and maintaining applications via Microsoft's global data centers. It teaches the fundamentals of data principles and how to use Microsoft data services. It will help if you know the basics of programming languages and Azure.

Cloud Computing 52

Cloud Computing Algorithm Certification SQL 52

U-Next

AUGUST 26, 2022

R is available as an open language of programming for statistical computing and data analytics, and R often has a command-line API. What do the data types R mean? A programming language must understand what else can and can be done to a certain value to handle data correctly. How Do Data Types Operate?

R (Programming) 52

R (Programming) Programming Language Algorithm Programming 52