Data Engineering Podcast

JULY 6, 2020

Summary There are an increasing number of use cases for real time data, and the systems to power them are becoming more mature. Once you have a streaming platform up and running you need a way to keep an eye on it, including observability, discovery, and governance of your data. That’s what the Lenses.io

Systems 100

Systems Kafka SQL Government 100

Cloudera

JANUARY 20, 2021

Most of what is written though has to do with the enabling technology platforms (cloud or edge or point solutions like data warehouses) or use cases that are driving these benefits (predictive analytics applied to preventive maintenance, financial institution’s fraud detection, or predictive health monitoring as examples) not the underlying data.

Manufacturing 105

Manufacturing Data Warehouse Kafka Retail 105

Knowledge Hut

SEPTEMBER 6, 2023

Taking Big Data Certification has multifold benefits. It is a well-known fact that we inhabit a data-rich world. Businesses are generating, capturing, and storing vast amounts of data at an enormous scale. This influx of data is handled by robust big data systems which are capable of processing, storing, and querying data at scale.

Big Data 52

Big Data Certification Hadoop Scala 52

Confluent

FEBRUARY 6, 2019

Building a scalable, reliable and performant machine learning (ML) infrastructure is not easy. It takes much more effort than just building an analytic model with Python and your favorite machine learning framework. It allows real-time data ingestion, processing, model deployment and monitoring in a reliable and scalable way.

Machine Learning 107

Machine Learning Python Kafka Java 107

Cloudera

JUNE 7, 2021

Cloudera DataFlow (CDF) is a comprehensive real-time streaming data platform that enables enterprises to address key streaming use cases across a wide range of streaming sources. The Forrester Wave calls out CDF’s edge management features that enable such IoT use cases and leverage streaming analytics. It’s too late.

Kafka 101

Kafka Data Ingestion Architecture Cloud 101

Data Engineering Podcast

JULY 27, 2021

They explain how Pulsar is being used by their customers, the work that they have done to scale the administrative workload for multi-tenant environments, and the challenges of operating such a data intensive service at large scale. Go to dataengineeringpodcast.com/97things today to get your copy!

Building 100

Building Management Kafka Data Warehouse 100

Lyft Engineering

JUNE 28, 2023

In early 2022, Lyft already had a comprehensive Machine Learning Platform called LyftLearn composed of model serving , training , CI/CD, feature serving , and model monitoring systems. Lyft is a real-time marketplace and many teams benefit from enhancing their machine learning models with real-time signals.

Machine Learning 115

Machine Learning Building Metadata Kafka 115

Rockset

AUGUST 16, 2022

Streams for Everyone If you have come this far it means you have already considered or are considering using event streaming in your data architecture for the wide variety of benefits it can offer. In either case, both Amazon Kinesis and Apache Kafka can help but which one is the right fit for you and your goals.

Kafka 52

Kafka AWS Cloud Java 52

Cloudera

JANUARY 26, 2021

Here we explore three common use cases where a CFM cluster can assume an HDF cluster’s dataflows with minimal to no downtime. Use Case 1: NiFi pulling data from Kafka and pushing it to a file system (like HDFS). Use Case 1: NiFi pulling data from Kafka and pushing it to a file system (like HDFS).

Kafka 77

Kafka Hadoop Data Ingestion Utilities 77

Rockset

JULY 28, 2022

MongoDB has grown from a basic JSON key-value store to one of the most popular NoSQL database solutions in use today. This can be used to create data lakes, populate data warehouses or for specific use cases like offloading analytics and text search. So, using CDC avoids these problems.

MongoDB 52

MongoDB Kafka NoSQL Data Lake 52

Knowledge Hut

JUNE 30, 2023

It refers to a series of operations to convert raw data into a format suitable for analysis, reporting, and machine learning which you can learn from data engineer books. You can pace your learning by joining data engineering courses such as the Bootcamp Data Engineer. What is Data Engineering? Who are Data Engineers?

Data Engineering 52

Data Engineering Data Engineer Engineering Data Warehouse 52

Data Engineering Podcast

NOVEMBER 4, 2019

In this episode he explains the technical implementation of the Ascend platform, the challenges that he has faced in the process, and how you can use it to simplify your dataflow automation. And for your machine learning workloads, they just announced dedicated CPU instances.

Programming Language 100

Programming Language Kafka Media Data Engineering 100

Knowledge Hut

JULY 3, 2023

In today's fast-paced and data-driven world, users increasingly depend on real-time intuition to get an aggressive side and define a plan of action. Conventional batch processing techniques seem incomplete in fulfilling the demand of driving the commercial environment. This is where real-time data ingestion comes into the picture.

Data Ingestion 52

Data Ingestion Pipeline-centric Google Cloud Media 52

Knowledge Hut

APRIL 25, 2023

The tremendous growth in data generation, then the rise in data engineer jobs - there’s no arguing the fact that the big data industry is at its best pace and you, as an aspiring data engineer, have a lot to learn and make out of it - including some tools! While they go about it - enter big data data engineer tools.

Data Engineering 52

Data Engineering Data Engineer Engineering Google Cloud 52

Big Data Tools

DECEMBER 7, 2021

The holiday season is almost upon us! News A lot of what we do in engineering involves learning new things and keeping a finger on the pulse of new technologies. Apache Arrow 6.0.1 – Apache Arrow presents itself as a cross-language development platform for in-memory analytics. Nevertheless, caches have their uses.

Data Engineering 52

Data Engineering Data Engineer Engineering Big Data Tools 52

Big Data Tools

DECEMBER 7, 2021

The holiday season is almost upon us! News A lot of what we do in engineering involves learning new things and keeping a finger on the pulse of new technologies. Apache Arrow 6.0.1 – Apache Arrow presents itself as a cross-language development platform for in-memory analytics. Nevertheless, caches have their uses.

Data Engineering 52

Data Engineering Data Engineer Engineering Big Data Tools 52

Data Engineering Podcast

OCTOBER 14, 2019

In this episode CTO and co-founder of Dataform Lewis Hemens joins the show to explain his motivation for creating the platform and company, how it works under the covers, and how you can start using it today to get your data warehouse under control. And for your machine learning workloads, they just announced dedicated CPU instances.

Data Warehouse 100

Data Warehouse PostgreSQL AWS Programming Language 100

Confluent

APRIL 18, 2019

And Confluent Cloud and Apache Kafka ® are at the center of this transformation. Kafka is business critical for us. So working with Confluent Cloud and the Kafka experts was an obvious choice,” says Stefan Frehse, Software Architect, Digitalization, for food processing machinery company BAADER.

Food 83

Food Cloud Retail Kafka 83

AltexSoft

AUGUST 29, 2023

This article explains what a data lake is, its architecture, and diverse use cases. A data lake allows to store data before a specific use case has been identified. Instead of relying on traditional hierarchical structures and predefined schemas, as in the case of data warehouses, a data lake utilizes a flat architecture.

Data Lake 52

Data Lake Architecture IT Amazon Web Services 52

Data Engineering Podcast

OCTOBER 9, 2018

MemSQL is a distributed database built to support concurrent use by transactional, application oriented, and analytical, high volume, workloads on the same hardware. In this episode the CEO of MemSQL describes how the company and database got started, how it is architected for scale and speed, and how it is being used in production.

PostgreSQL 100

PostgreSQL BI Data Warehouse Machine Learning 100

Big Data Tools

OCTOBER 10, 2022

News A lot of engineering is about learning new things and keeping a finger on the pulse of new technologies. Brooklin 4.1.0 – Once again, I learned something new while preparing this article. This time I learned about Brooklin, a LinkedIn service for streaming data in a heterogeneous environment. Greetings from sunny Berlin!

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

Big Data Tools

OCTOBER 10, 2022

News A lot of engineering is about learning new things and keeping a finger on the pulse of new technologies. Brooklin 4.1.0 – Once again, I learned something new while preparing this article. This time I learned about Brooklin, a LinkedIn service for streaming data in a heterogeneous environment. Greetings from sunny Berlin!

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

Towards Data Science

APRIL 3, 2023

credit: @psalms original_photo Introduction In many cases, processing data in-stream, or as it becomes available, can help reduce an enormous data problem (due to the volume and scale of the flow of data) into a more manageable one. You be the judge of what future you’d rather embrace. You be the judge of what future you’d rather embrace.

Process 87

Process Data Lake Systems Bytes 87

Data Engineering Podcast

DECEMBER 16, 2019

Even if you already have a metadata repository this is worth a listen to learn more about the value that visibility of your data can bring to your organization. And for your machine learning workloads, they just announced dedicated CPU instances. And for your machine learning workloads, they just announced dedicated CPU instances.

Metadata 100

Metadata PostgreSQL Datasets Data Warehouse 100

Data Engineering Podcast

OCTOBER 7, 2019

And for your machine learning workloads, they just announced dedicated CPU instances. You listen to this show to learn and stay up to date with what’s happening in databases, streaming platforms, big data, and everything else you need to know about modern data management. This week’s episode is also sponsored by Datacoral.

Structured Data 100

Structured Data Cloud SQL Programming Language 100

Knowledge Hut

APRIL 24, 2024

Why Project-based Learning is Important? Project-based learning is a cornerstone of software engineering education, fostering key competencies and hands-on experience that surpass traditional theoretical learning. cvtColor(image, cv2.COLOR_BGR2GRAY) COLOR_BGR2GRAY) _, thresh = cv2.threshold(gray_image, findContours(thresh, cv2.RETR_TREE,

Software Engineer 52

Software Engineer Software Engineering Coding Project 52

Knowledge Hut

OCTOBER 27, 2023

Why Project-based Learning is Important? Project-based learning is a cornerstone of software engineering education, fostering key competencies and hands-on experience that surpass traditional theoretical learning. cvtColor(image, cv2.COLOR_BGR2GRAY) COLOR_BGR2GRAY) _, thresh = cv2.threshold(gray_image, findContours(thresh, cv2.RETR_TREE,

Software Engineer 98

Software Engineer Software Engineering Coding Project 98

Monte Carlo

APRIL 24, 2023

Data lakes are useful, flexible data storage repositories that enable many types of data to be stored in its rawest state. This trend has enabled data teams to leverage the data lakehouse for more analytical use cases traditionally reserved for the data warehouse. See our post: Data Lakes vs. Data Warehouses.

Data Lake 52

Data Lake Google Cloud Data Warehouse AWS 52

AltexSoft

FEBRUARY 11, 2023

What’s more, investing in data products, as well as in AI and machine learning was clearly indicated as a priority. Data architecture is the organization and design of how data is collected, transformed, integrated, stored, and used by a company. machine learning and deep learning models; and business intelligence tools.

Data Architect 87

Data Architect Certification Generalist Big Data 87

Rockset

MARCH 10, 2022

He was also a contributor to the open source Apache HBase project. A platform such as Apache Kafka/Confluent , Spark or Amazon Kinesis for publishing that stream of event data. Immutable data stores have been useful in certain analytics scenarios. Event streaming/stream processing has been around for almost a decade.

Data Analytics 52

Data Analytics Data Warehouse Medical MySQL 52

ProjectPro

JUNE 18, 2021

Java for Data Science - Should data scientists learn Java? According to Popularity of Programming Languages (PYPL) , Python and Java are two of the most popular programming languages in use as of June 2021. They are used by various enterprises and developers across the globe today. renamed to Java.

Java 52

Java Data Science Python Programming Language 52

Knowledge Hut

DECEMBER 27, 2023

Software developers are the great mastermind behind video games with amazing graphics or mobile and web applications that you use smoothly. These skills can be learned with any Software Developer Training program. Software developers design and develop applications for various technologies using several tools and talents.

Medical 98

Medical Programming Language Amazon Web Services Entertainment 98

Knowledge Hut

DECEMBER 21, 2023

Machine learning is a subset of artificial intelligence, which stresses the analysis and identification of patterns and structure of data interpretation. The use of ML can help to analyze the humongous amounts of data and arrive at conclusions that are in favor of both customers as well as businesses. Read on to find out.

Machine Learning 52

Machine Learning Engineering Computer Science Portfolio 52

Netflix Tech

OCTOBER 15, 2019

Part I: Overview Andreas Andreakis , Falguni Jhaveri , Ioannis Papapanagiotou , Mark Cho , Poorna Reddy , Tongliang Liu Overview It is a commonly observed pattern for applications to utilize multiple datastores where each is used to serve a specific need such as storing the canonical form of data (MySQL etc.), caching (Memcached etc.),

Transportation 77

Transportation MySQL Kafka Data 77

Zalando Engineering

NOVEMBER 29, 2017

Second in our series about the use of Apache Kafka’s Streams API by Zalando This is the second in a series about the use of Apache Kafka’s Streams API by Zalando, Europe’s leading online fashion platform. See Ranking Websites in Real-time with Apache Kafka’s Streams API for the first post in the series.

Kafka 40

Kafka AWS Amazon Web Services Utilities 40

Monte Carlo

JANUARY 25, 2023

There exists a lot of infrastructure around the data warehouse (tools like dbt, Monte Carlo , Great Expectations, and Apache Airflow) that are already established products and can help to implement contracts. In the last couple of posts , I’ve focused on implementing data contracts in production services.

Data Warehouse 52

Data Warehouse Data High Quality Data Metadata 52

Rockset

FEBRUARY 7, 2020

In this IoT example, we examine how to enable complex analytic queries on real-time Kafka streams from connected car sensors. Building Real-Time Analytics on Connected Car IoT Data For our example, we have a fleet of connected vehicles that send the sensor data they generate to a Kafka cluster.

Kafka 40

Kafka Transportation Data SQL 40