Marc Lamberti

SEPTEMBER 24, 2023

An Airflow TaskGroup helps make a complex DAG easier to organize and read. Airflow taskgroups are meant to replace SubDAGs, the historical way of grouping your tasks. Indeed, SubDAGs are too complicated only for grouping tasks. They bring a lot of complexity as you must create a DAG in a DAG, import the SubDagOperator (which is a sensor), define the parameters correctly, and so on.

Coding 130

Coding Python Process Data Pipeline 130

databricks

SEPTEMBER 24, 2023

Product matching is an essential function in many retail and consumer goods organizations. Incoming products are compared to items in the existing product.

Metadata 84

Metadata Retail 84

Data Engineering Weekly

SEPTEMBER 24, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. See how it works today. Thoughtworks: Measuring the Value of a Data Catalog The cost & effort value proportion for a Data Catalog implementation is always questionable in a large-scale data infrastructure.

Data Engineering 67

Data Engineering Data Engineer Engineering Kafka 67

Hevo

SEPTEMBER 24, 2023

SAP is a popular tool among large-scale organizations for optimizing daily operations. These include financial management, inventory management, human resources, sales CRM, and more. Within the SAP ERP framework, SAP ERP data integration ensures data uniformity. This will provide real-time insights for decision-making.

Data Integration 52

Data Integration Data Management 52

Advertisement

Generative AI is upending the way product developers & end-users alike are interacting with data. Despite the potential of AI, many are left with questions about the future of product development: How will AI impact my business and contribute to its success? What can product managers and developers expect in the future with the widespread adoption of AI?

Data

Mutt Data

SEPTEMBER 24, 2023

Introduction If you’ve been at least a little bit online during these past few months then you’re probably aware of a new “AI-boom”. We’ve all played with ChatGPT or generative models , and some have even deployed these models in real environments. Things might have calmed down a bit since that initial shock, but that doesn’t mean the field stops at these amazing applications.

Machine Learning 40

Machine Learning Software Engineer Software Engineering Database 40

Hevo

SEPTEMBER 24, 2023

According to HG Data, 1.5 million companies are slated to spend $133.6 billion on modern data infrastructure in 2023.[1] It’s no surprise that the volume of data in our hands has grown exponentially over the past few years. To keep up with it, the data infrastructure had to evolve at a similar pace.

Data 52

Data IT 52

Hevo

SEPTEMBER 24, 2023

Integrating PostgreSQL on Google Cloud SQL to Redshift is an essential step in unlocking the power of data for modern businesses. By centralizing data in Redshift, a fully managed data warehousing service that provides high-performance analytical capabilities, you can expedite the analysis of voluminous datasets.

Google Cloud 52

Google Cloud PostgreSQL SQL Cloud 52

Hevo

SEPTEMBER 24, 2023

MySQL has remained the most popularly used open-source relational database for many years and continues to maintain its dominant position in the industry. Its robustness, reliability, and flexibility for a wide range of applications, from small-scale projects to vast enterprise systems, justifies its widespread adoption.

Google Cloud 52

Google Cloud MySQL PostgreSQL SQL 52

Hevo

SEPTEMBER 24, 2023

Migrating data between two platforms is a critical process for organizations to leverage the power of advanced analytics. The migration from PostgreSQL on Amazon RDS to Firebolt is one such example of how businesses can unlock the full potential of their data.

PostgreSQL 52

PostgreSQL Data Storage Data Process 52

Hevo

SEPTEMBER 24, 2023

In a digital data landscape, data keeps changing due to evolving user requirements. Manually managing such dynamic data from different sources is no longer sufficient to get the latest information. You need a solution that can coordinate and seamlessly integrate data in a centralized storage system.

Data 52

Data Systems Management IT 52

Advertisement

Many organizations today are unlocking the power of their data by using graph databases to feed downstream analytics, enahance visualizations, and more. Yet, when different graph nodes represent the same entity, graphs get messy. Watch this essential video with Senzing CEO Jeff Jonas on how adding entity resolution to a graph database condenses network graphs to improve analytics and save your analysts time.

Database