Precisely

MAY 13, 2024

Key Takeaways: Data integration is vital for real-time data delivery across diverse cloud models and applications, and for leveraging technologies like generative AI. The right data integration solution helps you streamline operations, enhance data quality, reduce costs, and make better data-driven decisions.

Data Integration 52

Data Integration Metadata Amazon Web Services Data Governance 52

Cloudera

MARCH 14, 2023

We just announced the general availability of Cloudera DataFlow Designer , bringing self-service data flow development to all CDP Public Cloud customers. In this blog post we will put these capabilities in context and dive deeper into how the built-in, end-to-end data flow life cycle enables self-service data pipeline development.

Data Pipeline 83

Data Pipeline Designing Kafka Metadata 83

Christophe Blefari

SEPTEMBER 25, 2023

This is a super late Data News, I wanted to send it earlier but I was travelling then enjoying time with friends and family. At the same time Microsoft leaked 38To of data — through a Github repository containing a link to an Azure storage with public access open. Data Economy 💰 Cisco acquired Splunk for $28b in cash.

Data 130

Data Data Warehouse Data Storage Data Pipeline 130

Christophe Blefari

SEPTEMBER 25, 2023

This is a super late Data News, I wanted to send it earlier but I was travelling then enjoying time with friends and family. At the same time Microsoft leaked 38To of data — through a Github repository containing a link to an Azure storage with public access open. Data Economy 💰 Cisco acquired Splunk for $28b in cash.

Data 130

Data Data Warehouse Data Storage Data Pipeline 130

Data Engineering Weekly

NOVEMBER 26, 2023

It’s the end of the year, and there will be a lot of buzz about what the next five years in data engineering might bring. 🕵️‍♂️📈 Before we delve into the patterns, it's important to remember that data infrastructure maturity model. Who knows what insights we might uncover?

Data Engineering 70

Data Engineering Data Engineer Engineering Kafka 70

Snowflake

OCTOBER 12, 2023

Our Digital Carbon Footprint Optimization catalyst enables CSPs to lower Scope 3 emissions in their product offerings by empowering customers to make lower carbon choices. Reduce carbon footprint: By pinpointing areas for improvement, the platform assists telecom companies in reducing the carbon footprint of their product offerings.

Telecommunication 79

Telecommunication Utilities Consulting Cloud 79

ThoughtSpot

FEBRUARY 9, 2023

Two years ago, we announced our Databricks partnership —including the launch of ThoughtSpot for Databricks, which gives joint customers the ability to run ThoughtSpot search queries directly on the Databricks Lakehouse without the need to move any data. That’s why we are excited to announce our new integration with Databricks Unity Catalog.

Government 82

Government SQL Machine Learning Cloud 82

Christophe Blefari

SEPTEMBER 1, 2023

I've taken an unplanned 3-week break since the last Data News, let's be honest, it was necessary! I spent a few hours working on the fancy data stack project and articles are in the works, but it was idealistic to produce quality code and content while enjoying the summer. dbt tests: How to write fewer and better data tests?

Food 130

Food Data SQL Python 130

Data Engineering Weekly

JUNE 29, 2023

But hey, I met my friends after a long time and got my copy of “ Fundamentals of Data Engineering ” signed by Joe Reis & Matt Housely. If you’re starting data engineering, I highly recommend reading it. There are many exciting announcements from both the data conferences, and I want to highlight a few themes here.

Data Engineering 76

Data Engineering Data Engineer Google Cloud Engineering 76

Data Engineering Podcast

AUGUST 13, 2022

Summary Data is useless if it isn’t being used, and you can’t use it if you don’t know where it is. Data catalogs were the first solution to this problem, but they are only helpful if you know what you are looking for. Data stacks are becoming more and more complex.

Metadata 100

Metadata MongoDB Scala MySQL 100

François Nguyen

MARCH 22, 2021

TL;DR After setting up and organizing the teams, we are describing 4 topics to make data mesh a reality. TL;DR After setting up and organizing the teams, we are describing 4 topics to make data mesh a reality. How do we build data products ? How can we interoperate between the data domains ?

Architecture 130

Architecture Technology Google Cloud Metadata 130

Christophe Blefari

FEBRUARY 23, 2024

My ideas these days ( credits ) Hey, fresh Data News edition. This week I've participated to a round table about data and did a cool presentation about Engines. — Apache Arrow is an awesome library that powers a lot of innovations in the data space in the recent years. Is the modern data stack disappearing?

Data Lake 130

Data Lake PostgreSQL MongoDB Scala 130

Data Engineering Podcast

NOVEMBER 9, 2020

Summary A data catalog is a critical piece of infrastructure for any organization who wants to build analytics products, whether internal or external. While there are a number of platforms available for building that catalog, many of them are either difficult to deploy and integrate, or expensive to use at scale.

Building 100

Building PostgreSQL BI Metadata 100

Pinterest Engineering

MAY 13, 2024

These systems together enabled numerous use cases that allowed Pinterest to significantly scale its business as we continued to grow our user base and evolve the products over the past 10 years. Pinterest hosted one of the largest production deployments of HBase in the world. A typical HBase deployment in production.

NoSQL 69

NoSQL MySQL Database Systems 69

Zalando Engineering

JUNE 25, 2023

Viewing their product details page will also enable it. These intentions are identified by our backend systems with specific business domain rules, i.e. the Search backend will have different rules from the Product backend. To date, the Fashion Store was based solely on domain-specific data.

Retail 57

Retail Architecture Engineering Systems 57

Christophe Blefari

JANUARY 20, 2023

Summer in coming ( credits ) Hey, new Friday, new Data News edition. Thank you for every recommendation you do about the blog or the Data News. The current state of data This week Benjamin Rogojan livestreamed an online conference featuring awesome data voices: state of data infra.

Google Cloud 130

Google Cloud Data Hadoop Machine Learning 130

Cloudera

DECEMBER 9, 2022

Cloudera has been providing enterprise support for Apache NiFi since 2015, helping hundreds of organizations take control of their data movement pipelines on premises and in the public cloud. Developers need to onboard new data sources, chain multiple data transformation steps together, and explore data as it travels through the flow.

Designing 99

Designing Coding Google Cloud AWS 99

Data Engineering Podcast

AUGUST 21, 2022

Summary Data has permeated every aspect of our lives and the products that we interact with. In this episode Shruti Bhat gives her view on the state of the ecosystem for real-time data and the work that she and her team at Rockset is doing to make it easier for engineers to build those experiences.

Lambda Architecture 100

Lambda Architecture MongoDB Scala MySQL 100

Netflix Tech

NOVEMBER 13, 2023

Like semantic segmentation for audio, SMAD separately tracks the amount of speech and music in each frame in an audio file and is useful in content understanding tasks during the audio production and delivery lifecycle. Likewise, authoring accessibility-features like Audio Description (AD) involves music and speech segmentation.

Datasets 89

Datasets Metadata Algorithm Architecture 89

Data Engineering Weekly

MARCH 30, 2023

In this episode of Data Engineering Weekly Radio, we delve into modern data stacks under pressure and the potential consolidation of the data industry. We discussed that the siloed nature of many data products has led to industry consolidation, ultimately benefiting customers.

Machine Learning 52

Machine Learning Government AWS Data Engineering 52

Data Engineering Weekly

DECEMBER 29, 2022

Data catalogs are the most expensive data integration systems you never intended to build. Data Catalog as a passive web portal to display metadata requires significant rethinking to adopt modern data workflow, not just adding “modern” in its prefix. How happy are you with your data catalogs?

Metadata 62

Metadata Data Warehouse ETL Tools Data Workflow 62

Data Engineering Podcast

AUGUST 28, 2022

Summary AirBnB pioneered a number of the organizational practices that have become the goal of modern data teams. With their new managed database service you can launch a production ready MySQL, Postgres, or MongoDB cluster in minutes, with automated backups, 40 Gbps connections from your application hosts, and high throughput SSDs.

Building 100

Building MongoDB Scala MySQL 100

Knowledge Hut

JANUARY 31, 2024

Service Catalog Management Service catalog management ensures that a service catalog or a list of all services is created or maintained for all those involved in IT service management or who have access to it. Products - Items or tools that aid in the delivery of IT services. It also upholds and enforces security policy.

Designing 52

Designing Process IT Portfolio 52

Cloudera

JUNE 7, 2021

Since then, we have been working hard to expose the full power of Apache Flink SQL and the existing Data Warehousing tools in CDP to combine it into a state-of-the-art real-time analytics platform. Flink SQL DDL and Catalog support. Catalog Integrations. This step significantly speeds up query development and data exploration.

Kafka 104

Kafka SQL Accessible Accessibility 104

Monte Carlo

JUNE 22, 2023

At our latest IMPACT summit, Roche shared their data mesh strategy for creating reliable data products. The goal of the Roche data team is to maximize the outcomes of customers and patients through data and analytics products. So they agreed to leverage Data Vault 2.0

Building 52

Building Data Ingestion Data Business Analyst 52

Cloudera

JANUARY 6, 2021

Python is used extensively among Data Engineers and Data Scientists to solve all sorts of problems from ETL/ELT pipelines to building machine learning models. Apache HBase is an effective data storage system for many workflows but accessing this data specifically through Python can be a struggle. Using Catalogs.

Machine Learning 133

Machine Learning Data Science Database Building 133

Data Engineering Weekly

APRIL 7, 2024

Poor data quality and unlcear data ownership remains the top challenges for the data teams. Data Mesh continuously gaining popularity among the enterprises. It is a stark difference from the Gartner report about data mesh. I guess only the time will tell who wins in the data mesh vs data fabric war.

Data Engineering 70

Data Engineering Data Engineer Engineering Unstructured Data 70

Cloudera

JUNE 26, 2023

Here are some tips and tricks of the trade to prevent well-intended yet inappropriate data engineering and data science activities from cluttering or crashing the cluster. For data engineering and data science teams, CDSW is highly effective as a comprehensive platform that trains, develops, and deploys machine learning models.

ETL Tools 76

ETL Tools Programming Language Datasets Data Pipeline 76

Data Engineering Weekly

MARCH 11, 2023

We are back in our Data Engineering Weekly Radio for edition #120. We will take 2 or 3 articles from each week's Data Engineering Weekly edition and go through an in-depth analysis. We discuss an article by Colin Campbell highlighting the need for a data catalog and the market scope for data contract solutions.

Data Engineering 52

Data Engineering Data Engineer Engineering High Quality Data 52

Pinterest Engineering

NOVEMBER 7, 2023

Jeff Xiang | Software Engineer, Logging Platform Vahid Hashemian | Software Engineer, Logging Platform Jesus Zuniga | Software Engineer, Logging Platform At Pinterest, data is ingested and transported at petabyte scale every day, bringing inspiration for our users to create a life they love.

Kafka 99

Kafka Java Software Engineer Software Engineering 99

Cloudera

JULY 18, 2022

In part 1 of this blog we discussed how Cloudera DataFlow for the Public Cloud (CDF-PC), the universal data distribution service powered by Apache NiFi, can make it easy to acquire data from wherever it originates and move it efficiently to make it available to other applications in a streaming fashion. Data decays!

Process 89

Process Kafka Scala SQL 89

Towards Data Science

JUNE 16, 2023

How to Identify Your Business-Critical Data Practical steps to identifying business-critical data models and dashboards and drive confidence in your data Source: synq.io This article has been co-written with Lindsay Murphy Not all data is created equal. There’s a good reason for this. Source: synq.io

BI 79

BI Data ETL Tools Machine Learning 79

Precisely

DECEMBER 6, 2023

The European Union’s General Data Protection Regulation (commonly known as GDPR) came into effect on the 25th May 2018. It is important to any user or processor of personally identifiable information (PII) as this new law marks a seismic change in the way that organizations must protect consumers’ data.

Retail 64

Retail Data Systems Data Integration 64

Data Engineering Weekly

SEPTEMBER 24, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. Airflow isn’t meant to process the data. See how it works today.

Data Engineering 67

Data Engineering Data Engineer Engineering Kafka 67

Ascend.io

MARCH 23, 2023

Below, I draw on my prior knowledge, discussions with customers, and conversations with other leaders to outline six key data engineering trends to be aware of (and capitalize on) in 2023. Efficiency and productivity will be paramount in 2023 Teams will get extremely cost-conscious if they haven’t already.

Software Engineer 52

Software Engineer Software Engineering Data Ingestion Data Engineering 52

ThoughtSpot

APRIL 2, 2024

Over the past couple of years, ThoughtSpot and Google have collaborated on a series of seamless user experiences—enabling deployments on Google Cloud Platform, creating the ability to live query entire Google BigQuery analytics catalogs, and integrating key Looker Modeling functionality just to name a few.

Google Cloud 59

Google Cloud Cloud Accessible Accessibility 59

Cloudera

MAY 1, 2023

Businesses everywhere have engaged in modernization projects with the goal of making their data and application infrastructure more nimble and dynamic. These are tables from other projects, or tables that are accessed through a Catalog. brokers, trust store) Catalog properties (e.g.

SQL 78

SQL Project Kafka Accessible 78