Snowflake

APRIL 17, 2024

In this Quickstart, you’ll learn how to perform key actions — from installing the Snowflake Python API to retrieving object data and managing Snowpark Container Services. Dive in to experience how the enhanced Python API streamlines your data workflows and unlocks the full potential of Python within Snowflake.

Data Pipeline 105

Data Pipeline Python Data Engineering Data Engineer 105

Knowledge Hut

SEPTEMBER 25, 2023

They guarantee that the data is efficiently cleaned, converted, and loaded. Big Data and Analytics: To handle and analyze enormous volumes of data, Azure Data Engineers use big data technologies like Azure Databricks and Apache Spark.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Lake 52

Monte Carlo

FEBRUARY 8, 2024

Here are some strategies to keep things firing on all cylinders: Parallelize data processing: Divide large data tasks into smaller, manageable chunks that can be processed concurrently. Identify parts of your data workflow that can be parallelized and adjust your pipeline configuration accordingly.

Data Pipeline 52

Data Pipeline AWS Datasets Data 52

DataKitchen

FEBRUARY 27, 2024

Just as pilots rely on auto-pilot technologies because they understand how they work and trust the data fed into them, data engineers and business leaders must foster a similar level of trust in LLMs. This trust hinges on the quality of the input data and the transparency of the Data Journey.

Data Engineering 46

Data Engineering Data Engineer Engineering Unstructured Data 46

Data Engineering Podcast

JULY 17, 2022

Contact Info Email LinkedIn Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? Shipyard]([link] Shipyard is an orchestration platform that helps data teams build out solid data operations from the get-go by connecting data tools and streamlining data workflows.

Data Management 130

Data Management Management Metadata MongoDB 130

Data Engineering Podcast

OCTOBER 22, 2021

In this episode Oliver Laslett describes why dashboards aren’t sufficient for business analytics, how Lightdash promotes the work that you are already doing in your data warehouse modeling with dbt, and how they are focusing on bridging the divide between data teams and business teams and the requirements that they have for data workflows.

Engineering 100

Engineering Business Intelligence BI Data Warehouse 100

Databand.ai

AUGUST 30, 2023

The DataOps framework is a set of practices, processes, and technologies that enables organizations to improve the speed, accuracy, and reliability of their data management and analytics operations. One key aspect of data orchestration is the automation of data pipeline tasks.

Data Governance 52

Data Governance Data Pipeline Government Data Cleanse 52

Data Engineering Podcast

JULY 31, 2022

Your host is Tobias Macey and today I’m interviewing Ernie Ostic about Manta, an automated data lineage service for managing visibility and quality of your data workflows Interview Introduction How did you get involved in the area of data management? Can you describe what Manta is and the story behind it?

IT 100

IT Metadata MongoDB MySQL 100

Knowledge Hut

MARCH 28, 2024

Apart from the demand, pursuing Azure data engineer jobs has numerous advantages, such as high salaries, opportunities for career advancement, and the possibility to work with the most advanced technologies in the field of data innovation. Implement data ingestion, processing, and analysis pipelines for large-scale data sets.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Warehouse 52

Data Engineering Podcast

MAY 3, 2021

Summary The Data industry is changing rapidly, and one of the most active areas of growth is automation of data workflows. Taking cues from the DevOps movement of the past decade data professionals are orienting around the concept of DataOps. Closing Announcements Thank you for listening!

Data Warehouse 100

Data Warehouse Data Pipeline BI Metadata 100

Data Engineering Podcast

JULY 3, 2022

In order to quickly identify if and how two data systems are out of sync Gleb Mezhanskiy and Simon Eskildsen partnered to create the open source data-diff utility. In order to quickly identify if and how two data systems are out of sync Gleb Mezhanskiy and Simon Eskildsen partnered to create the open source data-diff utility.

Data Integration 100

Data Integration MongoDB Scala MySQL 100

Data Engineering Podcast

SEPTEMBER 11, 2022

If you’re a data engineering podcast listener, you get credits worth $5,000 when you become a customer. Can you describe what Hevo Data is and the story behind it? Contact Info LinkedIn @ManishJethani on Twitter Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today?

Data Pipeline 130

Data Pipeline Building MongoDB Scala 130

Data Engineering Podcast

JANUARY 30, 2022

Pandas is a tool that spans data processing and data science. What are some of the ways that data engineers should think about writing their code to make it accessible to data scientists for supporting collaboration across data workflows? Pandas is a tool that spans data processing and data science.

Data Engineering 100

Data Engineering Data Engineer Engineering Data Pipeline 100

Data Engineering Podcast

JANUARY 1, 2022

Over the past couple of months, we’ve seen the resurgence of “benchmark wars” between major data warehousing platforms. To what extent do speed benchmarks inform decisions for modern data teams? How important is query speed in a modern data workflow? How has the way data teams work been changing?

Data Warehouse 100

Data Warehouse Data Lake SQL Hadoop 100

ThoughtSpot

SEPTEMBER 6, 2023

And 86% of respondents stated that they needed better technology to enable those in-the-moment decisions. Those coveted insights live at the end of a process lovingly known as the data pipeline.

Data Workflow 91

Data Workflow Raw Data Data Lake Business Intelligence 91

Data Engineering Podcast

OCTOBER 15, 2021

What are some of the challenges that teams are facing when integrating metadata management and analysis into their data workflows? What are some of the challenges that teams are facing when integrating metadata management and analysis into their data workflows? How is the governance of DataHub being managed?

Metadata 100

Metadata BI Data Warehouse Government 100

Data Engineering Podcast

OCTOBER 19, 2020

Interview Introduction How did you get involved in the area of data management? What is "data downtime"? Can you start by giving your definition of observability in the context of data workflows? What are some of the contributing factors that lead to poor data quality at the different stages of the lifecycle?

Machine Learning 100

Machine Learning Data Engineering Data Engineer Data 100

Data Engineering Podcast

SEPTEMBER 4, 2022

What are some examples of the types of data workflows and transformations that are necessary to maintain your customer-facing services? How have you approached the question of modeling for the data that you provide to end-users to make it straightforward to integrate and analyze the information? When is Sust the wrong choice?

MongoDB 100

MongoDB Scala MySQL Data Ingestion 100

Workfall

JULY 18, 2023

Additionally, they offer a way to keep your transformed data up to date without redoing the entire transformation process. By using DBT materializations, you can streamline your data workflows and focus on gaining insights rather than spending excessive time on repetitive calculations.

Datasets 77

Datasets Entertainment Data Workflow Data 77

Data Engineering Podcast

APRIL 5, 2021

Your host is Tobias Macey and today I’m interviewing Prukalpa Sankar about Atlan, a modern data workspace that makes collaboration among data stakeholders easier, increasing efficiency and agility in data projects Interview Introduction How did you get involved in the area of data management?

Data Warehouse 100

Data Warehouse Data Pipeline BI Metadata 100

Knowledge Hut

NOVEMBER 2, 2023

Azure Data engineering projects are complicated and require careful planning and effective team participation for a successful completion. While many technologies are available to help data engineers streamline their workflows and guarantee that each aspect meets its objectives, ensuring that everything works properly takes time.

Data Engineering 52

Data Engineering Data Engineer Coding Project 52

Data Engineering Weekly

MARCH 11, 2023

We also touch on the idea that data creation will be a decentralized process and the role of tools like data contracts in enabling successful decentralized data modeling. We emphasize the importance of creating high-quality data and the need for technological and organizational solutions to achieve this goal.

Data Engineering 52

Data Engineering Data Engineer Engineering High Quality Data 52

Data Engineering Podcast

AUGUST 5, 2019

Finding the data that you need is tricky, and Amundsen will help you solve that problem. And as your data grows in volume and complexity, there are foundational principles that you can follow to keep data workflows streamlined. Links Amundsen Data Council Presentation Strata Presentation Blog Post Lyft Airflow Podcast.__init__

PostgreSQL 100

PostgreSQL MongoDB Metadata Media 100

Data Engineering Weekly

OCTOBER 30, 2022

Editor’s Note: The current state of the Data Catalog The results are out for our poll on the current state of the Data Catalogs. The highlights are that 59% of folks think data catalogs are sometimes helpful. We saw in the Data Catalog poll how far it has to go to be helpful and active within a data workflow.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Ingestion 52

Monte Carlo

JULY 27, 2023

These applications & technologies may include but are not limited to: PowerBI, Tableau, and other BI tools, our external corporate website, intranet site, proprietary employee management and other HR systems, and data feeds that may flow between these and other systems.

Engineering 52

Engineering Healthcare Scala Data Warehouse 52

Knowledge Hut

SEPTEMBER 29, 2023

Why Should You Get an Azure Data Engineer Certification? Becoming an Azure data engineer allows you to seamlessly blend the roles of a data analyst and a data scientist. One of the pivotal responsibilities is managing data workflows and pipelines, a core aspect of a data engineer's role.

Certification 52

Certification Data Engineering Data Engineer Engineering 52

Knowledge Hut

DECEMBER 28, 2023

Human society in 2023 is a digital world, and its fuel - its currency - is data. Today, organizations seek skilled professionals who can harness data’s power to drive informed decisions. As technology evolves, cloud platforms have emerged as the cornerstone of modern data management.

Certification 52

Certification Data Engineering Data Engineer Engineering 52

Knowledge Hut

NOVEMBER 17, 2023

Data Architecture and Design: These experts excel in creating effective data structures that meet scalability requirements, ensure optimal data storage, processing, and retrieval, and correspond with business demands. Azure Data Factory stands at the forefront, orchestrating data workflows.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Lake 52

Data Engineering Podcast

APRIL 14, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management This episode is brought to you by Datafold – a testing automation platform for data engineers that prevents data quality issues from entering every part of your data workflow, from migration to dbt deployment.

Non-relational Database 147

Non-relational Database Relational Database Database Designing 147

Data Engineering Podcast

MARCH 31, 2024

Identifying and accounting for those errors is a critical piece of building trust in the organization that your data is accurate and up to date. While there are numerous products available to provide that visibility, they all have different technologies and workflows that they focus on. Your first 30 days are free!

Project 130

Project Data Lake High Quality Data Data Workflow 130

Workfall

JUNE 12, 2023

Source: Records in the final table after updating and inserting new rows: We can also create data workflows in the form of data pipelines using apache airflow to know more, you can check our blog at Easily Build ETL Pipeline Using Python And Airflow Conclusion In this blog, we demonstrated implementing a CDC workflow.

Telecommunication 80

Telecommunication Metadata Healthcare Finance 80

Cloudera

AUGUST 21, 2020

Finally, having the right platforms and skillsets within the business to be able to process data and get the right insights in real-time is critical to leveraging the power of data to drive meaningful change. Can you talk about some of the technology that helps make managing live streaming data possible?

Banking 60

Banking Kafka Cloud Storage Government 60

Workfall

JULY 4, 2023

Reading Time: 8 minutes In the world of data engineering, a mighty tool called DBT (Data Build Tool) comes to the rescue of modern data workflows. Imagine a team of skilled data engineers on an exciting quest to transform raw data into a treasure trove of insights.

Data Warehouse 69

Data Warehouse SQL PostgreSQL Database 69

Data Engineering Podcast

MAY 18, 2024

In Code Comments, host Jamie Parker, Red Hatter and experienced engineer, shares the journey of technologists from across the industry and their hard-won lessons in implementing new technologies. Data lakes are notoriously complex. Code Comments Podcast Logo]([link] Putting new technology to use is an exciting prospect.

Building 130

Building Data Lake High Quality Data Business Intelligence 130

Data Engineering Podcast

MAY 12, 2024

In Code Comments, host Jamie Parker, Red Hatter and experienced engineer, shares the journey of technologists from across the industry and their hard-won lessons in implementing new technologies. Data lakes are notoriously complex. Code Comments Podcast Logo]([link] Putting new technology to use is an exciting prospect.

Management 130

Management Data Lake High Quality Data Machine Learning 130

Data Engineering Podcast

APRIL 21, 2024

Summary Generative AI has rapidly transformed everything in the technology sector. Data lakes are notoriously complex. Contact Info LinkedIn Blog Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? Data lakes are notoriously complex.

Data Lake 182

Data Lake High Quality Data Data Pipeline Machine Learning 182

Cloudera

FEBRUARY 11, 2019

“Disruption slows as cloud and nonrelational technology take their place beside traditional approaches , the leaders extend their lead, and distributed data approaches solidify their place as a best practice for DMSA.” Established, mature technologies represent an integral part of DMSA.

Data Management 42

Data Management Management Metadata Government 42