Data Engineering Podcast

JUNE 3, 2018

ArangoDB is a storage engine that supports documents, dey/value, and graph data formats, as well as being fast and scalable. Preamble Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline you’ll need somewhere to deploy it, so check out Linode.

Data Storage 100

Data Storage PostgreSQL Architecture Database 100

Christophe Blefari

AUGUST 4, 2023

Summer Edition ( credits ) This is the first article of the Data News Summer Edition: how to build a data platform. The modern data stack has been criticised a lot, a few are saying it's dead other are saying we are in the post-modern era. In this article we will try to design the fancy data stack for a batch usage.

Google Cloud 225

Google Cloud MongoDB NoSQL Data 225

Data Engineering Weekly

JUNE 10, 2024

Experience Enterprise-Grade Apache Airflow Astro augments Airflow with enterprise-grade features to enhance productivity, meet scalability and availability demands across your data pipelines, and more. Databricks and Snowflake offer a data warehouse on top of cloud providers like AWS, Google Cloud, and Azure.

Data Engineering 103

Data Engineering Data Engineer Engineering Kafka 103

KDnuggets

OCTOBER 30, 2023

A comparative overview of data warehouses, data lakes, and data marts to help you make informed decisions on data storage solutions for your data architecture.

Data Lake 122

Data Lake Data Warehouse Data Storage Data 122

Seattle Data Guy

DECEMBER 19, 2022

By Reseun McClendon Today, your enterprise must effectively collect, store, and integrate data from disparate sources to both provide operational and analytical benefits. Whether its helping increase revenue by finding new customers or reducing costs, all of it starts with data.

Data Lake 130

Data Lake Data Warehouse Database Data Storage 130

Christophe Blefari

SEPTEMBER 28, 2023

Make your data stack take-off ( credits ) Hello, another edition of Data News. This week, we're going to take a step back and look at the current state of data platforms. What are the current trends and why are people fighting around the concept of the modern data stack. Is the modern data stack dying?

Cloud Storage 147

Cloud Storage Big Data Hadoop SQL 147

Christophe Blefari

FEBRUARY 24, 2023

Data engineering team moving data manually ( credits ) Dear readers, I hope you had a great week. The current data state is heavily dependent on infrastructure, wether it's cloud, on-premise or semi-related we need to understand where the data lands and where the code runs. Often they are right.

Kafka 130

Kafka Data Lake Data Storage Data 130

Start Data Engineering

MAY 9, 2024

Analyze usage and optimize table data storage 3.2.1. Snowflake pricing and settings inheritance model 3. Strategies to reduce Snowflake cost 3.1. Quick wins by changing settings 3.1.1. Update warehouse settings 3.2. Identify expensive queries and optimize them 3.2.1.1. Identify expensive queries with query_history 3.2.1.2.

Data Storage 130

Data Storage Accessible Accessibility Management 130

Monte Carlo

MAY 28, 2024

A data ingestion architecture is the technical blueprint that ensures that every pulse of your organization’s data ecosystem brings critical information to where it’s needed most. Ensuring all relevant data inputs are accounted for is crucial for a comprehensive ingestion process.

Data Ingestion 52

Data Ingestion Architecture Designing Hadoop 52

Knowledge Hut

APRIL 25, 2024

Big data in information technology is used to improve operations, provide better customer service, develop customized marketing campaigns, and take other actions to increase revenue and profits. It is especially true in the world of big data. It is especially true in the world of big data. What Are Big Data T echnologies?

Big Data 98

Big Data Technology NoSQL Hadoop 98

Christophe Blefari

SEPTEMBER 25, 2023

This is a super late Data News, I wanted to send it earlier but I was travelling then enjoying time with friends and family. At the same time Microsoft leaked 38To of data — through a Github repository containing a link to an Azure storage with public access open. Providing more control over data storage.

Data 130

Data Data Warehouse Data Storage Data Pipeline 130

Christophe Blefari

SEPTEMBER 25, 2023

This is a super late Data News, I wanted to send it earlier but I was travelling then enjoying time with friends and family. At the same time Microsoft leaked 38To of data — through a Github repository containing a link to an Azure storage with public access open. Providing more control over data storage.

Data 130

Data Data Warehouse Data Storage Data Pipeline 130

Christophe Blefari

MAY 12, 2023

Sorting the news ( credits ) Hey you, new Friday means Data News. Let's start with with my newly created podcast Minds of Data. In Minds of Data I'll met people from the data ecosystem in order to learn more about them. In Tableau Pulse you'll have access to auto-generated insights on your data.

Data 130

Data Data Storage SQL Coding 130

Knowledge Hut

APRIL 23, 2024

Data handling became tedious only after huge amounts of it started to accumulate. For example, in 1880, the US Census Bureau needed to handle the 1880 Census data. They realized that compiling this data and converting it into information would take over 10 years without an efficient system.

Big Data 52

Big Data Amazon Web Services Media Cloud Computing 52

Christophe Blefari

JANUARY 20, 2024

Learn data engineering, all the references ( credits ) This is a special edition of the Data News. But right now I'm in holidays finishing a hiking week in Corsica 🥾 So I wrote this special edition about: how to learn data engineering in 2024. The idea is to create a living reference about Data Engineering.

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Cloudera

MAY 9, 2024

The Gartner Data and Analytics Summit in London is quickly approaching on May 13 th to 15 th , and the Cloudera team is ready to hit the show floor! With the rapid pace of AI innovations, today’s leaders must connect, share, and learn from each other on how to best leverage AI technology to effectively analyze data to our advantage.

Banking 102

Banking Data Storage Data Analytics Cloud 102

Knowledge Hut

MARCH 27, 2024

On the other hand, data structures are like the tools that help organize and arrange data within a computer program. In simpler terms, a database is where information is neatly stored, like books on shelves, while data structures are the behind-the-scenes helpers, ensuring data is well-organized and easy to find.

Database 52

Database Relational Database Algorithm PostgreSQL 52

Hevo

MAY 10, 2024

Amazon S3 is a prominent data storage platform with multiple storage and security features. Integrating data stored in Amazon S3 to a data warehouse like Databricks can enable better data-driven decisions. Integrating data from Amazon S3 to Databricks […]

Data Integration 52

Data Integration Data Warehouse Data Storage Machine Learning 52

Start Data Engineering

OCTOBER 22, 2021

Data modeling 2.1.1. Data storage 2.2. Data transformation 2.2.1. Data pipeline 2.4. Data analytics 3. Introduction SQL is the bread and butter of data engineering. Introduction 2. SQL skills 2.1. Gathering requirements 2.1.2. Exploration 2.1.3. Modeling 2.1.4. Transformation types 2.2.1.1.

SQL 130

SQL Data Engineering Data Engineer Engineering 130

Analytics Vidhya

FEBRUARY 27, 2023

Due to its lack of POSIX conformance, some believe it to be data storage instead. Introduction The Hadoop Distributed File System (HDFS) is a Java-based file system that is Distributed, Scalable, and Portable. HDFS and […] The post Top 10 Hadoop Interview Questions You Must Know appeared first on Analytics Vidhya.

Hadoop 233

Hadoop Java Data Storage Programming 233

KDnuggets

JULY 11, 2022

Learn about the modern tools for data orchestration, data storage, analytical engineering, batch processing, and data streaming.

Data Engineering 127

Data Engineering Data Engineer Engineering Data Storage 127

Analytics Vidhya

MARCH 7, 2023

Introduction Apache Flume is a tool/service/data ingestion mechanism for gathering, aggregating, and delivering huge amounts of streaming data from diverse sources, such as log files, events, and so on, to centralized data storage. Flume is a tool that is very dependable, distributed, and customizable.

Data Ingestion 216

Data Ingestion Data Storage Hadoop Data 216

Christophe Blefari

JUNE 16, 2023

I'm now under the Berlin rain with 20° When I write in these conditions I feel like a tortured author writing a depressing novel while actually today I'll speak about the AI Act, Python, SQL and data platforms. Mainly he unit tests macros (the logic) with his framework and test data with soda and dbt contracts.

Programming Language 130

Programming Language SQL PostgreSQL Data 130

Knowledge Hut

JANUARY 29, 2024

Some techniques add to the development of technology in the business sectors, including Data Science and Cloud Computing, essential aspects of the technology industry. With the help of data science, one can gather all the critical analyses from vast chunks of data stored in clouds. In this model, the data is not 100% secure.

Cloud Computing 98

Cloud Computing Data Science Cloud Amazon Web Services 98

Hevo

MAY 3, 2024

Data is a powerful tool for organizational success today. However, businesses may face data storage and processing challenges in a data-rich world. When used effectively, it provides valuable insights into everyday operations to maximize business value.

Data Storage 52

Data Storage Structured Data Data Process 52

Christophe Blefari

JANUARY 20, 2023

Summer in coming ( credits ) Hey, new Friday, new Data News edition. Thank you for every recommendation you do about the blog or the Data News. The current state of data This week Benjamin Rogojan livestreamed an online conference featuring awesome data voices: state of data infra.

Google Cloud 130

Google Cloud Data Hadoop Machine Learning 130

Hevo

APRIL 20, 2024

Effective data migration is the key to overcoming the challenges associated with today’s data-driven world. The AWS Aurora Postgres to Databricks integration offers data storage and analytics solutions that help unlock the full potential of your organization’s operational data.

AWS 52

AWS Data Integration Data Storage Database 52

Cloudera

MARCH 5, 2024

AI, and any analytics for that matter, are only as good as the data upon which they are based. Struggling to access and collect, oftentimes disparate and siloed, data across environments that are required to power AI, many organizations are unable to achieve the business insight and value they had hoped for.

Data Lake 87

Data Lake Data Storage Government Kafka 87

Data Engineering Weekly

DECEMBER 2, 2023

AWS S3 Express One Zone sparks some delight in the data infrastructure. link] Amazon S3 Express One Zone is a high-performance, single-availability Zone storage class purpose-built to deliver consistent single-digit millisecond data access for your most frequently accessed data and latency-sensitive applications.

IT 85

IT BI AWS Kafka 85

Knowledge Hut

JANUARY 18, 2024

Data Science has risen to become one of the world's topmost emerging multidisciplinary approaches in technology. Recruiters are hunting for people with data science knowledge and skills these days. Data Scientists collect, analyze, and interpret large amounts of data. Choose data sets.

Data Science 98

Data Science Business Analyst ETL Method Data Architect 98

Christophe Blefari

NOVEMBER 11, 2022

On a social note, today I've joined data-folks Mastodon server, you can follow me there. I'll speak about "How to build the data dream team" Let's jump onto the news. Ingredients of a Data Warehouse Going back to basics. Ian describes how Riot Games uses data and what machine learning means.

BI 130

BI Data Warehouse Data Database 130

Engineering at Meta

JUNE 12, 2024

This means we need to regularly checkpoint our training state and efficiently store and retrieve training data. Optimal connectivity between GPUs: Large-scale model training involves transferring vast amounts of data between GPUs in a synchronized fashion. requires revisiting trade-offs made for other types of workloads.

Algorithm 114

Algorithm Data Storage Technology Building 114

Cloudera

APRIL 22, 2022

Part of this emphasis extends to helping enterprises deal with their data and overall cloud connectivity as well as local networks. At the same time, operators are also becoming more data- and cloud-centric themselves. Networked data and cloud are not just “edge computing”. This poses both opportunities and challenges.

Data Solutions 134

Data Solutions Amazon Web Services Data Storage Google Cloud 134

Knowledge Hut

JANUARY 18, 2024

Data Science is an amalgamation of several disciplines, including computer science, statistics, and machine learning. As the world on the internet is becoming our second home, Big Data has exploded. Data Science is the study of this big data to derive a meaningful pattern.

Data Science 98

Data Science Amazon Web Services Finance Big Data 98

Towards Data Science

DECEMBER 15, 2023

For a data scientist, there’s no such thing as too much data. Photo by Trnava University on Unsplash Data Science vs Security/IT: A Battle for the Ages Acquiring and keeping data is the focus of a huge amount of our mental energy as data scientists. If you ask a data scientist “Can we solve this problem?”

Machine Learning 88

Machine Learning Data Science Data Security Data Storage 88

phData: Data Engineering

SEPTEMBER 19, 2023

With the amount of data companies are using growing to unprecedented levels, organizations are grappling with the challenge of efficiently managing and deriving insights from these vast volumes of structured and unstructured data. What is a Data Lake?

Data Lake 52

Data Lake Process Metadata Data Warehouse 52

Data Engineering Weekly

MARCH 24, 2024

link] Kai Waehner: The Data Streaming Landscape 2024 This is a comprehensive overview of the state of the data streaming landscape in 2024. link] Mercado Libre Tech: Data Mesh @ MELI - Building Highways for Thousands of Data Producers Ok, Data Mesh is still alive!!

Data Engineering 62

Data Engineering Data Engineer Engineering Metadata 62