The Pragmatic Engineer

NOVEMBER 16, 2022

7 November Domino Data Lab (data workflow platform, Series E). In Mexico, the company had a large initiative with private drivers and a fleet of Tesla cars. That operation is in talks of selling to other investors. Meta - 13% layoffs. Widely-reported, e.g. here. 25% of their workforce laid off. Verified.

Healthcare 130

Healthcare Software Engineer Software Engineering Data Workflow 130

KDnuggets

JANUARY 4, 2023

These Python packages would improve your data workflow.

Python 116

Python Data Workflow Data Data Science 116

Data Engineering Podcast

AUGUST 28, 2022

What are the different concerns that need to be included in a stack that supports fully automated data workflows? There was recently an interesting article suggesting that the "left-to-right" approach to data workflows is backwards.

Data Engineering 100

Data Engineering Data Engineer MongoDB Metadata 100

François Nguyen

MARCH 22, 2021

TL;DR After setting up and organizing the teams, we are describing 4 topics to make data mesh a reality. As you can see, this is in the code part where you are building your data pipelines, a misnomer because this is an over simplification. The other benefit is you can also use parameters and build a generic workflows to be re-used.

Technology 130

Technology Architecture Google Cloud Metadata 130

ThoughtSpot

SEPTEMBER 6, 2023

The pathway from ETL to actionable analytics can often feel disconnected and cumbersome, leading to frustration for data teams and long wait times for business users. And even when we manage to streamline the data workflow, those insights aren’t always accessible to users unfamiliar with antiquated business intelligence tools.

Data Workflow 91

Data Workflow Raw Data Data Lake Business Intelligence 91

Databand.ai

AUGUST 30, 2023

DataOps is a collaborative approach to data management that combines the agility of DevOps with the power of data analytics. It aims to streamline data ingestion, processing, and analytics by automating and integrating various data workflows.

Architecture 52

Architecture Data Ingestion Data Governance Data Cleanse 52

Hevo

FEBRUARY 28, 2023

As raw data comes in various forms and sizes, it is essential to have a proper system to handle big data. One of the significant challenges is referencing data points as the complexities increase in data workflows.

Raw Data 52

Raw Data Data Workflow Big Data Systems 52

Cloudyard

FEBRUARY 19, 2024

The data, originating from different formats and sources, requires consolidation into Snowflake tables for comprehensive analysis. Therefore, Snowpark, with its capabilities in simplifying complex data workflows, becomes instrumental in achieving this objective. The journey begins with customer invoice data stored in a CSV file.

Data Process 52

Data Process Process Data Workflow Data 52

Preset

FEBRUARY 21, 2022

CrateDB is a distributed SQL database that excels at IoT and Time Series data workflows. In this post, we'll showcase how CrateDB and Superset can be used together.

Data Workflow 52

Data Workflow SQL Database Data 52

Data Engineering Weekly

DECEMBER 25, 2023

The Rising Impact of AI and Large Language Models 2023 witnessed a substantial impact of AI and large language models in data engineering. These technologies are increasingly automating processes like ETL, improving data quality management, and evolving the landscape of data tools.

Data Engineering 52

Data Engineering Data Engineer Engineering Unstructured Data 52

Data Engineering Weekly

DECEMBER 29, 2022

Data catalogs are the most expensive data integration systems you never intended to build. Data Catalog as a passive web portal to display metadata requires significant rethinking to adopt modern data workflow, not just adding “modern” in its prefix. How happy are you with your data catalogs?

Metadata 62

Metadata Data Warehouse ETL Tools Data Workflow 62

Christophe Blefari

SEPTEMBER 28, 2023

When it comes to incrementally change a data platform this is a bit different, you need to find what is going wrong and what could be improved. A lot of companies started with Postgres + dbt + Metabase, don't be ashamed.

Cloud Storage 147

Cloud Storage Big Data Hadoop SQL 147

Data Engineering Podcast

FEBRUARY 20, 2022

Summary The life sciences as an industry has seen incredible growth in scale and sophistication, along with the advances in data technology that make it possible to analyze massive amounts of genomic information.

Systems 100

Systems Software Engineer Software Engineering Data Warehouse 100

Ascend.io

DECEMBER 19, 2023

However, there is a critical aspect to consider: using Snowflake’s native features to build data pipelines is akin to constructing a bespoke data platform. that you can combine to create custom data workflows. But it also requires a lot of integration work to create a cohesive data management process.

ETL Tools 52

ETL Tools IT Data Pipeline Data Warehouse 52

Data Engineering Podcast

JULY 17, 2022

Shipyard]([link] Shipyard is an orchestration platform that helps data teams build out solid data operations from the get-go by connecting data tools and streamlining data workflows.

Data Management 130

Data Management Management Metadata MongoDB 130

Data Engineering Podcast

OCTOBER 22, 2021

In this episode Oliver Laslett describes why dashboards aren’t sufficient for business analytics, how Lightdash promotes the work that you are already doing in your data warehouse modeling with dbt, and how they are focusing on bridging the divide between data teams and business teams and the requirements that they have for data workflows.

Engineering 100

Engineering Business Intelligence BI Data Warehouse 100

Workfall

JULY 18, 2023

Additionally, they offer a way to keep your transformed data up to date without redoing the entire transformation process. By using DBT materializations, you can streamline your data workflows and focus on gaining insights rather than spending excessive time on repetitive calculations.

Datasets 77

Datasets Entertainment Data Workflow Data 77

Data Engineering Podcast

JULY 31, 2022

Your host is Tobias Macey and today I’m interviewing Ernie Ostic about Manta, an automated data lineage service for managing visibility and quality of your data workflows Interview Introduction How did you get involved in the area of data management? Can you describe what Manta is and the story behind it?

IT 100

IT Metadata MongoDB MySQL 100

Netflix Tech

NOVEMBER 14, 2023

Next Steps and Conclusion Integrating Psyberg into our operations has improved our data workflows and opened up exciting possibilities for the future. As we continue to innovate, Netflix’s data platform team is focused on creating a comprehensive solution for incremental processing use cases.

Metadata 97

Metadata Data Pipeline Scala Data Workflow 97

Data Engineering Podcast

JULY 3, 2022

In order to quickly identify if and how two data systems are out of sync Gleb Mezhanskiy and Simon Eskildsen partnered to create the open source data-diff utility. In order to quickly identify if and how two data systems are out of sync Gleb Mezhanskiy and Simon Eskildsen partnered to create the open source data-diff utility.

Data Integration 100

Data Integration MongoDB Scala MySQL 100

Data Engineering Podcast

SEPTEMBER 11, 2022

If you’re a data engineering podcast listener, you get credits worth $5,000 when you become a customer. Can you describe what Hevo Data is and the story behind it? If you’re a data engineering podcast listener, you get credits worth $5,000 when you become a customer. Can you describe what Hevo Data is and the story behind it?

Data Pipeline 130

Data Pipeline Building MongoDB Scala 130

Data Engineering Podcast

JANUARY 30, 2022

Pandas is a tool that spans data processing and data science. What are some of the ways that data engineers should think about writing their code to make it accessible to data scientists for supporting collaboration across data workflows? Pandas is a tool that spans data processing and data science.

Data Engineering 100

Data Engineering Data Engineer Engineering Python 100

Knowledge Hut

SEPTEMBER 25, 2023

Data Engineer Design, implement, and maintain data pipelines for data ingestion, processing, and transformation in Azure. Work together with data scientists and analysts to understand the needs for data and create effective data workflows.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Lake 52

Data Engineering Podcast

JANUARY 1, 2022

Over the past couple of months, we’ve seen the resurgence of “benchmark wars” between major data warehousing platforms. To what extent do speed benchmarks inform decisions for modern data teams? How important is query speed in a modern data workflow? How has the way data teams work been changing?

Data Warehouse 100

Data Warehouse Data Lake SQL Hadoop 100

Monte Carlo

FEBRUARY 8, 2024

Here are some strategies to keep things firing on all cylinders: Parallelize data processing: Divide large data tasks into smaller, manageable chunks that can be processed concurrently. Identify parts of your data workflow that can be parallelized and adjust your pipeline configuration accordingly.

Data Pipeline 52

Data Pipeline AWS Datasets Data 52

Databand.ai

AUGUST 30, 2023

One key aspect of data orchestration is the automation of data pipeline tasks. By automating repetitive tasks, such as data extraction, transformation, and loading (ETL), organizations can streamline their data workflows and reduce the risk of human error.

Data Governance 52

Data Governance Data Pipeline Government Data Cleanse 52

Data Engineering Podcast

OCTOBER 15, 2021

What are some of the challenges that teams are facing when integrating metadata management and analysis into their data workflows? What are some of the challenges that teams are facing when integrating metadata management and analysis into their data workflows? How is the governance of DataHub being managed?

Metadata 100

Metadata BI Data Warehouse Government 100

Data Engineering Weekly

OCTOBER 30, 2022

Editor’s Note: The current state of the Data Catalog The results are out for our poll on the current state of the Data Catalogs. The highlights are that 59% of folks think data catalogs are sometimes helpful. We saw in the Data Catalog poll how far it has to go to be helpful and active within a data workflow.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Ingestion 52

Meltano

JULY 7, 2022

Those best practices, such as version control, a unified development environment, and continuous, built-in testing, can also be brought to data workflows. Now it’s Data’s Turn The data profession is about five years behind software engineering. Data is just very different from dev.

BI 52

BI Software Engineer Software Engineering Data Workflow 52

Data Engineering Podcast

SEPTEMBER 4, 2022

What are some examples of the types of data workflows and transformations that are necessary to maintain your customer-facing services? How have you approached the question of modeling for the data that you provide to end-users to make it straightforward to integrate and analyze the information? (e.g.

MongoDB 100

MongoDB Scala MySQL Data Ingestion 100

DataKitchen

FEBRUARY 27, 2024

Embracing DataOps for Enhanced Data Journey Management The complexity of managing Data Journeys, especially in RAG and LLMs, underscores the importance of embracing DataOps principles. DataOps provides a framework for automating and optimizing data workflows, emphasizing collaboration, monitoring, and continuous improvement.

Data Engineering 48

Data Engineering Data Engineer Engineering Unstructured Data 48

Data Engineering Podcast

MAY 3, 2021

Summary The Data industry is changing rapidly, and one of the most active areas of growth is automation of data workflows. Taking cues from the DevOps movement of the past decade data professionals are orienting around the concept of DataOps.

Data Warehouse 100

Data Warehouse Data Pipeline BI Metadata 100

Data Engineering Podcast

APRIL 5, 2021

Your host is Tobias Macey and today I’m interviewing Prukalpa Sankar about Atlan, a modern data workspace that makes collaboration among data stakeholders easier, increasing efficiency and agility in data projects Interview Introduction How did you get involved in the area of data management?

Data Warehouse 100

Data Warehouse Data Pipeline BI Metadata 100

Meltano

FEBRUARY 22, 2023

Testing a data application is similar to testing any software application in many ways, just with a strong focus on testing data-related issues. But testing problems like failing data workflows, mismatches in data reconciliation after ETL, and data quality issues means that you’re not only testing the code but also the data itself.

Data 52

Data Transportation Data Workflow Data Lake 52

DataKitchen

JANUARY 25, 2022

DataOps improves the robustness, transparency and efficiency of data workflows through automation. For example, DataOps can be used to automate data integration. Previously, the consulting team had been using a patchwork of ETL to consolidate data from disparate sources into a data lake.

Consulting 97

Consulting Recruitment Data Lake Data Engineering 97

Data Engineering Weekly

MARCH 11, 2023

[link] Data Engineering Weekly Data Catalog - A Broken Promise Data catalogs are the most expensive data integration systems you never intended to build.

Data Engineering 52

Data Engineering Data Engineer Engineering High Quality Data 52

Data Engineering Podcast

AUGUST 5, 2019

Finding the data that you need is tricky, and Amundsen will help you solve that problem. And as your data grows in volume and complexity, there are foundational principles that you can follow to keep data workflows streamlined. Finding the data that you need is tricky, and Amundsen will help you solve that problem.

PostgreSQL 100

PostgreSQL MongoDB Metadata Media 100