Monte Carlo

JANUARY 23, 2023

When it comes to the data community, there’s always a debate broiling about something— and right now “data mesh vs data lake” is right at the top of that list. In this post we compare and contrast the data mesh vs data lake to illustrate the benefits of each and help discover what’s right for your data platform.

Data Lake 52

Data Lake Unstructured Data Business Intelligence Architecture 52

Data Engineering Podcast

MARCH 27, 2022

In this episode Balaji Ganesan shares how his experiences building and maintaining Ranger in previous roles helped him understand the needs of organizations and engineers as they define and evolve their data governance policies and practices. Acryl]([link] The modern data stack needs a reimagined metadata management platform.

Data Governance 130

Data Governance Government Cloud Building 130

Data Engineering Podcast

FEBRUARY 19, 2023

Because of their complete ownership of your data they constrain the possibilities of what data you can store and how it can be used. It's supposed to make building smarter, faster, and more flexible data infrastructures a breeze. If you're fed up with the 'Modern Data Stack', give TimeXtender a try.

IT 147

IT Data Lake Metadata Data Warehouse 147

Data Engineering Podcast

FEBRUARY 5, 2023

Sign up now for early access to Materialize and get started with the power of streaming data with the same simplicity and low implementation cost as batch cloud data warehouses. . -> Airflow and Luigi -> Dagster, Prefect, Lyft, etc. . -> Airflow and Luigi -> Dagster, Prefect, Lyft, etc.

Data Engineering 147

Data Engineering Data Engineer Engineering PostgreSQL 147

Monte Carlo

APRIL 1, 2021

Over the past few years, data lakes have emerged as a must-have for the modern data stack. But while the technologies powering our access and analysis of data have matured, the mechanics behind understanding this data in a distributed environment have lagged behind. Who has access to it?

Data Lake 52

Data Lake Unstructured Data Data Warehouse Metadata 52

ProjectPro

AUGUST 11, 2021

“Data Lake vs Data Warehouse = Load First, Think Later vs Think First, Load Later” The terms data lake and data warehouse are frequently stumbled upon when it comes to storing large volumes of data. Data Warehouse Architecture What is a Data lake?

Data Lake 52

Data Lake Data Warehouse Cloud Hadoop 52

François Nguyen

MARCH 22, 2021

TL;DR After setting up and organizing the teams, we are describing 4 topics to make data mesh a reality. How do we build data products ? With this 3rd platform generation, you have more real time data analytics and a cost reduction because it is easier to manage this infrastructure in the cloud thanks to managed services.

Technology 130

Technology Architecture Google Cloud Metadata 130

Monte Carlo

JANUARY 5, 2024

Data lakehouse architecture combines the benefits of data warehouses and data lakes, bringing together the structure and performance of a data warehouse with the flexibility of a data lake. Metadata layer 4. …ok, so maybe they don’t say that. But they should! Storage layer 3. API layer 5.

Architecture 52

Architecture Data Lake Metadata Unstructured Data 52

Monte Carlo

JANUARY 5, 2024

Data lakehouse architecture combines the benefits of data warehouses and data lakes, bringing together the structure and performance of a data warehouse with the flexibility of a data lake. Metadata layer 4. …ok, so maybe they don’t say that. But they should! Storage layer 3. API layer 5.

Architecture 52

Architecture Data Lake Metadata Unstructured Data 52

Data Engineering Podcast

DECEMBER 4, 2022

Summary Business intelligence is the foremost application of data in organizations of all sizes. The typical conception of how it is accessed is through a web or desktop application running on a powerful laptop. Zing Data is building a mobile native platform for business intelligence.

Business Intelligence 100

Business Intelligence Metadata BI MongoDB 100

Snowflake

MAY 21, 2024

This will enable our joint customers to experience bidirectional data access between Snowflake and Microsoft Fabric, with a single copy of data with OneLake in Fabric. Organizations using both platforms will be able to do so more cost-effectively, rather than building pipelines or maintaining copies of data in each platform.

Metadata 123

Metadata Cloud Accessible Accessibility 123

Data Engineering Podcast

AUGUST 21, 2022

In this episode Tracy Daniels, CDO of Truist, shares her journey into the position, her responsibilities, and her relationship to the data professionals in her organization. Atlan is the metadata hub for your data ecosystem. RudderStack helps you build a customer data platform on your warehouse or data lake.

Metadata 100

Metadata MongoDB MySQL Data Lake 100

Monte Carlo

AUGUST 19, 2023

A comprehensive data platform solution powers data acquisition, storage, preparation, delivery, governance, and even the robust security needs of users and applications. In today’s data-driven landscape, building a data platform is no longer a nice-to-have, but a necessity for most organizations. It depends.

Building 52

Building BI Data Lake Data Governance 52

Data Engineering Podcast

NOVEMBER 27, 2022

Summary The data ecosystem has been growing rapidly, with new communities joining and bringing their preferred programming languages to the mix. This has led to inefficiencies in how data is stored, accessed, and shared across process and system boundaries. Atlan is the metadata hub for your data ecosystem.

Data Process 100

Data Process Process Metadata Business Intelligence 100

Data Engineering Podcast

AUGUST 28, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Data Engineering 100

Data Engineering Data Engineer MongoDB Metadata 100

ProjectPro

FEBRUARY 8, 2023

It offers a simple and efficient solution for data processing in organizations. It offers users a data integration tool that organizes data from many sources, formats it, and stores it in a single repository, such as data lakes, data warehouses, etc., being data exactly matches the classifier, and 0.0

AWS 98

AWS Scala Metadata Data Lake 98

Precisely

JUNE 6, 2023

Atif Salam noted that as recently as a year ago, the primary focus in many organizations was on ingesting data and building data lakes. Today, they’re doing everything they can to prevent their data lakes from turning into data swamps.

Data Architecture 52

Data Architecture Architecture Metadata Data Lake 52

Towards Data Science

FEBRUARY 6, 2024

As you do not want to start your development with uncertainty, you decide to go for the operational raw data directly. Accessing Operational Data I used to connect to views in transactional databases or APIs offered by operational systems to request the raw data. Does it sound familiar?

Systems 80

Systems Raw Data Metadata Data Cleanse 80

Data Engineering Podcast

FEBRUARY 27, 2022

Summary Building a data platform is a complex journey that requires a significant amount of planning to do well. In this episode Tobias Macey, the host of the show, reflects on his plans for building a data platform and what he has learned from running the podcast that is influencing his choices. That’s Timescale.

Designing 100

Designing Metadata Data Lake Relational Database 100

LinkedIn Engineering

JULY 19, 2023

Co-Authors: Sumedh Sakdeo , Lei Sun , Sushant Raikar , Stanislav Pak , and Abhishek Nath Introduction At LinkedIn, we build and operate an open source data lakehouse deployment to power Analytics and Machine Learning workloads. This allows organizations to enforce constraints on data models, compliance annotations, and other metadata.

Big Data 78

Big Data Data Management Management Metadata 78

Data Engineering Podcast

SEPTEMBER 4, 2022

In this episode founders Barr Moses and Lior Gavish rejoin the show to reflect on the evolution and adoption of data observability technologies and the capabilities that are being introduced as the broader ecosystem adopts the practices. Atlan is the metadata hub for your data ecosystem. A data observability platform?

IT 100

IT Metadata MongoDB MySQL 100

Snowflake

AUGUST 31, 2023

Catalog Integration: Our newly developed Catalog Integration feature allows you to seamlessly plug Snowflake into other Iceberg catalogs tracking table metadata. Since 2021, Snowflake has had External Tables for the purpose of read-only querying external data lakes.

Metadata 85

Metadata AWS Data Lake Datasets 85

Data Engineering Weekly

MARCH 24, 2024

al6z: 16 Changes to the Way Enterprises Are Building and Buying Generative AI This report has a lot of interesting insight into the enterprise adoption of Gen AI. The APIs support emitting unstructured log lines and typed metadata key-value pairs (per line). The extracted key-value pairs are written to the line’s metadata.

Data Engineering 62

Data Engineering Data Engineer Engineering Metadata 62

The Modern Data Company

MARCH 1, 2024

For today’s Chief Data Officers (CDOs) and data teams, the struggle is real. We’re drowning in data yet thirsting for actionable insights. We need a new approach, a paradigm shift that delivers data with the agility and efficiency of a speedboat – enter Data Products.

Database-centric 40

Database-centric Data Lake Government Retail 40

AltexSoft

AUGUST 22, 2022

In this post, we’ll attempt to explain the idea behind a data fabric, its architectural building blocks, the benefits it brings, and ways to approach its implementation. What is data fabric? to provide a unified view of all enterprise data. Data fabric architecture example. Unified data access.

Architecture 52

Architecture Metadata Data Lake Machine Learning 52

Cloudera

JUNE 7, 2023

By now, almost everyone across the tech landscape has heard of the Zero Trust (ZT) security model, which assumes that every device, application, or user attempting to access a network is not to be trusted (see NIST definitions below). The data is protected but it is also accessible by the people who need it.

Government 81

Government Metadata Data Lake Data 81

Monte Carlo

JANUARY 25, 2023

In an evolving data landscape, the explosion of new tooling solutions—from cloud-based transforms to data observability —has made the question of “build versus buy” increasingly important for data leaders. Hint: it’s not a one-size-fits-all answer, but answering a few critical questions can help. Let’s jump in!

Data Warehouse 40

Data Warehouse Building Data Lake Data Storage 40

Monte Carlo

MAY 18, 2023

Depending on your requirements (more on that in the next section), it is prudent to evaluate open source data lineage tools alongside the several other types of platforms that provide data lineage capabilities. You may even already have access to one of them. The Spark data lake lineage example is helpful for context here.

Metadata 64

Metadata Data Lake Government Data Governance 64

Precisely

JANUARY 25, 2024

There are three potential approaches to mainframe modernization: Data Replication creates a duplicate copy of mainframe data in a cloud data warehouse or data lake, enabling high-performance analytics virtually in real time, without negatively impacting mainframe performance. Best Practice 2. Best Practice 3.

Metadata 67

Metadata Data Governance Relational Database Government 67

Towards Data Science

JULY 24, 2023

Why are these things related, and more importantly, why should data leaders care? Well, the connective tissue that powers and integrates these services is frequently found in the metadata of table formats (storage) and query/job logs (compute). Take data governance for example. Again, that answer will depend on who you are.

Metadata 96

Metadata Data Warehouse Data Lake Data Science 96

Cloudera

JUNE 30, 2022

With Cloudera’s vision of hybrid data , enterprises adopting an open data lakehouse can easily get application interoperability and portability to and from on premises environments and any public cloud without worrying about data scaling. Why integrate Apache Iceberg with Cloudera Data Platform?

Data Lake 86

Data Lake Business Intelligence Metadata Data Warehouse 86

Cloudera

MARCH 17, 2023

I love the enthusiasm surrounding ChatGPT and the eagerness to learn about modern data architectures such as data lakehouses, data meshes, and data fabrics. ChatGPT is an excellent resource for gaining high-level insights and building awareness of any technology.

Education 62

Education Unstructured Data Data Lake Data Warehouse 62

AltexSoft

NOVEMBER 10, 2021

The pun being obvious, there’s more to that than just a new term: Data lakehouses combine the best features of both data lakes and data warehouses and this post will explain this all. What is a data lakehouse? Data warehouse vs data lake vs data lakehouse: What’s the difference.

Architecture 52

Architecture Data Lake Data Warehouse Metadata 52

Snowflake

JANUARY 22, 2024

As part of that transition, Fidelity has consolidated its analytics data into its Enterprise Analytics Platform, which is engineered using the Snowflake Data Cloud, making it easier for teams and departments across the company to access the data they need. Ready to start building models of your own with Snowpark ML?

Engineering 63

Engineering Data Lake Unstructured Data Metadata 63

AltexSoft

DECEMBER 23, 2022

Whether your goal is data analytics or machine learning , success relies on what data pipelines you build and how you do it. But even for experienced data engineers, designing a new data pipeline is a unique journey each time. Data engineering in 14 minutes. ELT allows them to work with the data directly.

Process 52

Process Building Raw Data Data Lake 52

Cloudera

JANUARY 21, 2021

Building a Shadow IT organization with separate, disconnected data repositories that only serve your line of business’ needs and introduce compliance and security risk for your company, lose your organization’s focus and participation in core business, and see your company ending up paying much more in the end? Must you be: .

IT 93

IT Data Lake Data Warehouse Cloud Storage 93

Monte Carlo

JUNE 28, 2022

To help organizations realize the full potential of their data lake and lakehouse investments, Monte Carlo, the data observability leader, is proud to announce integrations with Delta Lake and Databricks’ Unity Catalog for full data observability coverage. billion in 2020 to 17.60 billion in 2020 to 17.60

Data Lake 52

Data Lake Metadata AWS Data Warehouse 52