Grouparoo

OCTOBER 26, 2021

In today’s data-driven business world, organizations are looking for more efficient ways to leverage data from a variety of sources. For example, businesses often need to evaluate their performance based on large volumes of customer and sales data that might be stored in a variety of locations and formats.

Data Pipeline 52

Data Pipeline ETL Tools ETL System Data Warehouse 52

Christophe Blefari

JANUARY 20, 2024

Learn data engineering, all the references ( credits ) This is a special edition of the Data News. But right now I'm in holidays finishing a hiking week in Corsica 🥾 So I wrote this special edition about: how to learn data engineering in 2024. The idea is to create a living reference about Data Engineering.

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

ProjectPro

DECEMBER 7, 2021

Data pipelines are a significant part of the big data domain, and every professional working or willing to work in this field must have extensive knowledge of them. As data is expanding exponentially, organizations struggle to harness digital information's power for different business use cases.

Data Pipeline 52

Data Pipeline Architecture Kafka AWS 52

Data Engineering Podcast

DECEMBER 16, 2018

Summary Every business needs a pipeline for their critical data, even if it is just pasting into a spreadsheet. As the organization grows and gains more customers, the requirements for that pipeline will change. Can you start by sharing your definition of a data pipeline? What are your thoughts on that approach?

Data Pipeline 100

Data Pipeline Data Lake Data Warehouse Python 100

Ascend.io

APRIL 12, 2023

Right now, at this moment, are you prepared to act on your company’s data? So when we talk about making data usable, we’re having a conversation about data integrity. Data integrity is the overall readiness to make confident business decisions with trustworthy data, repeatedly and consistently. If not, why?

Data Pipeline 52

Data Pipeline Data Integration Datasets Data 52

Data Engineering Podcast

MAY 22, 2022

Summary Machine learning has become a meaningful target for data applications, bringing with it an increase in the complexity of orchestrating the entire data flow. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform.

Machine Learning 130

Machine Learning Data Engineering Data Engineer Cloud 130

Data Engineering Podcast

DECEMBER 4, 2022

Summary The term "real-time data" brings with it a combination of excitement, uncertainty, and skepticism. In this episode Arjun Narayan explains how the technical barriers to adopting real-time data in your analytics and applications have become surmountable by organizations of all sizes.

Data Lake 100

Data Lake MongoDB MySQL Data Warehouse 100

Data Engineering Weekly

MARCH 3, 2024

RudderStack is the Warehouse Native CDP, built to help data teams deliver value across the entire data activation lifecycle, from collection to unification and activation. Editor’s Note: Chennai, India Meetup - March-08 Update We are thankful to Ideas2IT to host our first Data Hero’s meetup.

Data Engineering 67

Data Engineering Data Engineer Pipeline-centric Engineering 67

Towards Data Science

MAY 3, 2023

Data Engineering Learn about slow change dimensions (SCD) and how to implement SCD Type 2 in VDK Photo by Joshua Sortino on Unsplash Data is the backbone of any organization, and in today’s fast-paced world, it is crucial to keep track of its versions. They store and manage current and historical data in a data warehouse.

Data Lake 87

Data Lake Data SQL Data Warehouse 87

Towards Data Science

OCTOBER 20, 2023

An End-To-End Tutorial for Beginners Continue reading on Towards Data Science »

Data Pipeline 76

Data Pipeline MySQL Building Data Science 76

Christophe Blefari

FEBRUARY 24, 2023

Data engineering team moving data manually ( credits ) Dear readers, I hope you had a great week. The current data state is heavily dependent on infrastructure, wether it's cloud, on-premise or semi-related we need to understand where the data lands and where the code runs. Often they are right.

Kafka 130

Kafka Data Lake Data Storage Data 130

François Nguyen

MARCH 22, 2021

TL;DR After setting up and organizing the teams, we are describing 4 topics to make data mesh a reality. TL;DR After setting up and organizing the teams, we are describing 4 topics to make data mesh a reality. How do we build data products ? How can we interoperate between the data domains ?

Architecture 130

Architecture Technology Google Cloud Metadata 130

Ascend.io

AUGUST 31, 2023

In the dynamic world of data, many professionals are still fixated on traditional patterns of data warehousing and ETL, even while their organizations are migrating to the cloud and adopting cloud-native data services. Modern platforms like Redshift , Snowflake , and BigQuery have elevated the data warehouse model.

Data Lake 52

Data Lake ETL Tools Data Warehouse Data Pipeline 52

Data Engineering Podcast

NOVEMBER 6, 2022

Summary Despite the best efforts of data engineers, data is as messy as the real world. Entity resolution and fuzzy matching are powerful utilities for cleaning up data from disconnected sources, but it has typically required custom development and training machine learning models.

MongoDB 130

MongoDB Scala MySQL Data Lake 130

Data Engineering Podcast

DECEMBER 11, 2022

Summary One of the most critical aspects of software projects is managing its data. Managing the operational concerns for your database can be complex and expensive, especially if you need to scale to large volumes of data, high traffic, or geographically distributed usage.

Database 100

Database MySQL Data Lake MongoDB 100

Ascend.io

APRIL 22, 2024

At Ascend, we believe it’s time to rethink data engineering from the ground up. As the world of data continues to evolve at a breakneck pace, we are thrilled to announce the next revolutionary step in our journey – Project Inception. Read More: The State of Data Engineering in 2023: Does Your Data Program Stack Up?

Project 52

Project Metadata Data Pipeline Data Engineering 52

Data Engineering Podcast

DECEMBER 4, 2022

Summary Business intelligence is the foremost application of data in organizations of all sizes. Zing Data is building a mobile native platform for business intelligence. Atlan is the metadata hub for your data ecosystem. Data engineers don’t enjoy writing, maintaining, and modifying ETL pipelines all day, every day.

Business Intelligence 100

Business Intelligence Metadata BI MongoDB 100

Knowledge Hut

MARCH 28, 2024

Data science has become one of the most trending fields today. Data engineering is one of them. According to AnalytixLabs , the data science market is expected to be worth USD 230.80 This demonstrates the increasing need for Microsoft Certified Data Engineers. That’s where data engineers are on the go.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Warehouse 52

Towards Data Science

FEBRUARY 6, 2024

This generalisation makes their data models complex and cryptic and require domain expertise. Even harder to manage, a common setup within large organisations is to have several instances of these systems with some underlaying processes in charge of transmitting data among them, which could lead to duplications, inconsistencies, and opacity.

Systems 83

Systems Raw Data Metadata Data Cleanse 83

Knowledge Hut

SEPTEMBER 25, 2023

This demonstrates how in-demand Microsoft Certified Data Engineers are becoming. They are moving their servers and on-premises data to Azure Cloud. What does all of this mean for Data Engineering professionals? In order to manage big data and other operational services, businesses are continuously in need of data engineers.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Lake 52

Knowledge Hut

NOVEMBER 17, 2023

Azure Data Engineers play an important role in building efficient, secure, and intelligent data solutions on Microsoft Azure's powerful platform. The position of Azure Data Engineers is becoming increasingly important as businesses attempt to use the power of data for strategic decision-making and innovation.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Lake 52

Data Engineering Podcast

MAY 8, 2022

Summary Many of the events, ideas, and objects that we try to represent through data have a high degree of connectivity in the real world. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Struggling with broken pipelines?

Database 100

Database Data Lake BI Business Intelligence 100

Monte Carlo

AUGUST 31, 2023

Data is a priority for your CEO, as it often is for digital-first companies, and she is fluent in the latest and greatest business intelligence tools. But what happens when the data is wrong? What about a frantic email from your CTO about “duplicate data” in a business intelligence dashboard? The root of data downtime?

Data Lake 52

Data Lake Data Pipeline Unstructured Data Data Warehouse 52

Knowledge Hut

NOVEMBER 17, 2023

The demand for data-related professions, including data engineering, has indeed been on the rise due to the increasing importance of data-driven decision-making in various industries. Becoming an Azure Data Engineer in this data-centric landscape is a promising career choice. Who is an Azure Data Engineer?

Data Engineering 52

Data Engineering Data Engineer Engineering Scala 52

Data Engineering Podcast

AUGUST 21, 2022

Summary The position of Chief Data Officer (CDO) is relatively new in the business world and has not been universally adopted. In this episode Tracy Daniels, CDO of Truist, shares her journey into the position, her responsibilities, and her relationship to the data professionals in her organization.

Metadata 100

Metadata MongoDB MySQL Data Lake 100

Data Engineering Podcast

DECEMBER 4, 2021

Summary Hiring data professionals is challenging for a multitude of reasons, and as with every interview process there is a potential for bias to creep in. The whole process of hiring is an important organizational skill to cultivate and this is an interesting exploration of the specific challenges involved in finding data professionals.

Data Warehouse 100

Data Warehouse Data Lake BI Business Intelligence 100

Data Engineering Podcast

OCTOBER 9, 2022

Summary The core of any data platform is the centralized storage and processing layer. For many that is a data warehouse, but in order to support a diverse and constantly changing set of uses and technologies the data lakehouse is a paradigm that offers a useful balance of scale and cost, with performance and ease of use.

Metadata 100

Metadata AWS MongoDB MySQL 100

Cloudera

JUNE 2, 2022

Since 2015, the Cloudera DataFlow team has been helping the largest enterprise organizations in the world adopt Apache NiFi as their enterprise standard data movement tool. This need has generated a market opportunity for a universal data distribution service. Why does every organization need it when using a modern data stack?

Systems 102

Systems Data Lake Google Cloud Data Collection 102

Knowledge Hut

NOVEMBER 2, 2023

Azure Data engineering projects are complicated and require careful planning and effective team participation for a successful completion. While many technologies are available to help data engineers streamline their workflows and guarantee that each aspect meets its objectives, ensuring that everything works properly takes time.

Data Engineering 52

Data Engineering Data Engineer Coding Project 52

Data Engineering Podcast

JULY 3, 2022

Summary The perennial challenge of data engineers is ensuring that information is integrated reliably. In order to quickly identify if and how two data systems are out of sync Gleb Mezhanskiy and Simon Eskildsen partnered to create the open source data-diff utility. Data teams are increasingly under pressure to deliver.

Data Integration 100

Data Integration MongoDB Scala MySQL 100

Databand.ai

JULY 5, 2023

What Are Data Observability Tools? Data observability tools are software solutions that oversee, analyze, and improve the performance of data pipelines. In this article: Why Are Data Observability Tools Important? Faster Troubleshooting Data pipeline failures are expensive and damaging to organizations.

Data Pipeline 52

Data Pipeline Data Lake Data Warehouse Datasets 52

Data Engineering Podcast

NOVEMBER 13, 2022

Summary CreditKarma builds data products that help consumers take advantage of their credit and financial capabilities. To make that possible they need a reliable data platform that empowers all of the organization’s stakeholders. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

MongoDB 100

MongoDB Scala Google Cloud MySQL 100

Data Engineering Podcast

AUGUST 28, 2022

For data engineers, this is a monumental undertaking. Atlan is the metadata hub for your data ecosystem. RudderStack helps you build a customer data platform on your warehouse or data lake. The only thing worse than having bad data is not knowing that you have it.

Data Engineering 100

Data Engineering Data Engineer MongoDB Metadata 100

Data Engineering Podcast

JANUARY 28, 2024

Summary Monitoring and auditing IT systems for security events requires the ability to quickly analyze massive volumes of unstructured log data. Cliff Crosland co-founded Scanner to provide fast querying of high scale log data for security auditing. SIEM) A query engine is useless without data to analyze.

Data Lake 147

Data Lake Building High Quality Data AWS 147

Edureka

FEBRUARY 7, 2023

An Azure Data Engineer is responsible for designing, implementing, and maintaining data management and data processing systems on the Microsoft Azure cloud platform. They work with large and complex data sets and are responsible for ensuring that data is stored, processed, and secured efficiently and effectively.

Data Engineering 52

Data Engineering Data Engineer Engineering Programming Language 52

Monte Carlo

MAY 2, 2023

It’s that time of year where we announce the results of our annual The State of Data Quality survey. The headline for this year was, without a doubt, the fact that data downtime nearly doubled year over year , driven by a 166% increase in time to resolution for data quality issues. ?

Data Lake 80

Data Lake Data Software Engineer Software Engineering 80

Data Engineering Weekly

JUNE 25, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. Join Monte Carlo, dbt, and Shiftkey's VP of Data & Analytics, John Steinmetz.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Lake 52