Snowflake

APRIL 8, 2024

In 2020, Snowflake announced a new global competition to recognize the work of early-stage startups building their apps — and their businesses — on Snowflake, offering up to $250,000 in investment as the top prize. Our sincere thanks go out to everyone who participated in this year’s competition. SignalFlare.ai

Pipeline-centric 126

Pipeline-centric Food Healthcare Unstructured Data 126

The Modern Data Company

DECEMBER 29, 2022

If they can streamline their data pipelines to provide actionable insights, they can usher in the next generation of financial services and survive the next global disruption. In addition, business users can leverage drag-and-drop engineering to build pipelines without extensive coding.

Banking 52

Banking Pipeline-centric Building Finance 52

Confluent

NOVEMBER 20, 2020

Building data pipelines isn’t always straightforward. The gap between the shiny “hello world” examples of demos and the gritty reality of messy data and imperfect formats is sometimes all too […].

Cloud 139

Cloud Data Pipeline Data Building 139

Data Engineering Podcast

DECEMBER 4, 2022

In this episode Arjun Narayan explains how the technical barriers to adopting real-time data in your analytics and applications have become surmountable by organizations of all sizes. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Data Lake 100

Data Lake MongoDB MySQL Data Warehouse 100

Data Engineering Podcast

DECEMBER 11, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Database 100

Database MySQL Data Lake MongoDB 100

Data Engineering Podcast

OCTOBER 30, 2022

In this episode he shares his experiences working with organizations to adopt analytics engineering patterns and the ways that Optimus and dbt were combined to let data analysts deliver insights without the roadblocks of complex pipeline management. Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold.

Engineering 100

Engineering MongoDB Scala MySQL 100

Meltano

JULY 19, 2022

Use Cases of Custom Extractors The use cases of custom extractors fall into one of the following three categories: Unconventional data sources: You may need to build a custom extractor to deal with custom databases, queues, files, internal APIs, etc. You should enter a name like meltano-demo.

Building 52

Building Python Database Project 52

Data Engineering Podcast

JULY 17, 2022

In this episode Crux CTO Mark Etherington discusses the different costs involved in managing external data, how to think about the total return on investment for your data, and how the Crux platform is architected to reduce the toil involved in managing third party data. Tired of deploying bad data?

Data Management 130

Data Management Management Metadata MongoDB 130

Data Engineering Podcast

NOVEMBER 6, 2022

In this episode she shares the story behind the project, the details of how it is implemented, and how you can use it for your own data projects. RudderStack helps you build a customer data platform on your warehouse or data lake. Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold.

MongoDB 130

MongoDB Scala MySQL Data Lake 130

Monte Carlo

JUNE 6, 2023

But is there more to generative AI than a fancy demo on Twitter? And how will it impact data? How generative AI will disrupt data With the advent of OpenAI’s DALL-E and ChatGPT, large language models became much more useful to the vast majority of humans. Let’s assess.

Data Engineering 84

Data Engineering Data Engineer Engineering Business Intelligence 84

Snowflake

DECEMBER 21, 2023

Welcome to Snowflake’s Startup Spotlight, where we feature awesome companies building businesses on Snowflake. As we prototyped solutions, we found that there was a considerable amount of pipeline setup, database and cache tuning, and API development involved. Instead, they’ll import a data package and start coding right away.

Software Engineer 90

Software Engineer Software Engineering Database Data Pipeline 90

Data Engineering Podcast

JULY 10, 2022

Summary Building a data platform is a journey, not a destination. Beyond the work of assembling a set of technologies and building integrations across them, there is also the work of growing and organizing a team that can support and benefit from that platform. Tired of deploying bad data? What does Riskified do?

Metadata 100

Metadata MongoDB MySQL Data Pipeline 100

The Modern Data Company

JANUARY 31, 2023

Another year, another chance to learn more about the world of data. In 2023, The Modern Data Company (Modern) hopes to reach more companies and organizations with our data operating system, build incredible value from existing and upcoming data assets, and share insights into major shifts in what it means to be data-driven.

Retail 98

Retail Healthcare Data Architecture Data Pipeline 98

Grouparoo

MAY 11, 2021

In practice, one does not simply connect Sheets and Mailchimp: There's a disconnect here; it's not so point-and-shoot to get data flowing across this pipeline. Let's build him a pipeline to get data from Google Sheets to flow automatically into Mailchimp. Why don't we help Dylan out a bit?

Data Pipeline 52

Data Pipeline Data Project Utilities 52

Data Engineering Podcast

JULY 3, 2022

She also discusses her views on the role of the data lakehouse as a building block for these architectures and the ongoing influence that it will have as the technology matures. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code. Tired of deploying bad data?

Architecture 100

Architecture Metadata MongoDB MySQL 100

Data Engineering Podcast

NOVEMBER 13, 2022

Summary CreditKarma builds data products that help consumers take advantage of their credit and financial capabilities. To make that possible they need a reliable data platform that empowers all of the organization’s stakeholders. Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold.

MongoDB 100

MongoDB Scala Google Cloud MySQL 100

Monte Carlo

NOVEMBER 9, 2023

And in many ways, LLMs are going to make data engineers more valuable – and that’s exciting! Still, it’s one thing to show your boss a cool demo of a data discovery tool or text-to-SQL generator – it’s another thing to use it with your company’s proprietary data, or even more concerning, customer data.

Unstructured Data 86

Unstructured Data Database Data Pipeline Architecture 86

Data Engineering Podcast

JUNE 19, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Metadata 130

Metadata Unstructured Data MongoDB MySQL 130

Precisely

SEPTEMBER 12, 2023

As a proud member of the Connect with Confluent program , we help organizations going through digital transformation and IT infrastructure modernization break down data silos and power their streaming data pipelines with trusted data.

Kafka 52

Kafka Data Integration Data Pipeline Datasets 52

Data Engineering Podcast

JUNE 5, 2022

Summary The best way to make sure that you don’t leak sensitive data is to never have it in the first place. The team at Skyflow decided that the second best way is to build a storage system dedicated to securely managing your sensitive information and making it easy to integrate with your applications and data systems.

Data Security 100

Data Security Metadata MongoDB Scala 100

Data Engineering Podcast

FEBRUARY 24, 2020

Summary Misaligned priorities across business units can lead to tensions that drive members of the organization to build data and analytics projects without the guidance or support of engineering or IT staff. Are you spending too much time maintaining your data pipeline?

IT 100

IT Data Lake Data Pipeline Media 100

Data Engineering Podcast

JUNE 12, 2022

Summary Building a well rounded and effective data team is an iterative process, and the first hire can set the stage for future success or failure. Trupti Natu has been the first data hire multiple times and gone through the process of building teams across the different stages of growth.

Metadata 100

Metadata Unstructured Data Business Intelligence MongoDB 100

Towards Data Science

JUNE 19, 2023

Photo by CHUTTERSNAP on Unsplash This article is about Meerschaum Compose , a tool for defining ETL pipelines in YAML and a plugin for the data engineering framework Meerschaum. Docker was a game-changer, revolutionizing the way we design, build, and run our cloud applications.

PostgreSQL 80

PostgreSQL SQL Python Project 80

Data Engineering Weekly

JULY 2, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork, and SQL grunt work out of building complete customer profiles, so you can quickly ship actionable, enriched data to every downstream team. Now every team can build a customer360 in Snowflake with RudderStack Profiles.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Data Engineering Podcast

MARCH 2, 2020

Summary Building applications on top of unbounded event streams is a complex endeavor, requiring careful integration of multiple disparate systems that were engineered in isolation. The ksqlDB project was created to address this state of affairs by building a unified layer on top of the Kafka ecosystem for stream processing.

Kafka 100

Kafka Process PostgreSQL MySQL 100

Monte Carlo

OCTOBER 4, 2022

In November of 2020, O’Reilly Media first approached us with the idea to author Data Quality Fundamentals: A Practitioner’s Guide to Building More Trustworthy Data Pipelines. It was an inflection point for a fledgling company that had only just begun to establish the category of data observability.

High Quality Data 52

High Quality Data Data Pipeline Media Data 52

Monte Carlo

MAY 26, 2023

A better measurement is the data downtime formula (above), which more comprehensively measures the amount of time the data was inaccurate, missing, or otherwise erroneous. A proactive approach for measuring data freshness is to create service level agreements or SLAs for specific data pipelines.

Data Pipeline 98

Data Pipeline Data Data Warehouse Machine Learning 98

Data Engineering Podcast

DECEMBER 21, 2020

I’m working with O’Reilly on a project to collect the 97 things that every data engineer should know, and I need your help. When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Data Governance 130

Data Governance Government Data Lake Banking 130

Snowflake

JUNE 27, 2023

For example, data science teams can accelerate the execution of machine learning (ML) Python libraries used in training and inference jobs. App developers can build and deploy front-end user interfaces using React and other popular web development frameworks. See this in action in the demo from Snowflake Summit 2023 keynote.

Scala 61

Scala Programming Language Data Science Python 61

Workfall

SEPTEMBER 18, 2023

It’s designed to be user-friendly, customizable, and extensible, making it a valuable tool for data engineers, analysts, and data-driven organizations looking to streamline their data pipelines. With its drag-and-drop interface, creating data pipelines becomes as easy as arranging blocks in a puzzle.

Data Pipeline 93

Data Pipeline Raw Data Data Schemas Healthcare 93

Monte Carlo

SEPTEMBER 27, 2023

When it comes to building data and AI products you can trust, Monte Carlo, the leader of the data observability category, and Alation, the data intelligence leader, are better together. Why is my data late?” “How How did this pipeline break?” “Can Can I trust the data feeding this model?

Data Governance 69

Data Governance Government Data Pipeline Data 69

The Modern Data Company

FEBRUARY 15, 2023

No matter how much data they collect, what tools they buy, and what talent they manage to lure away from companies like Google and Facebook (if they can), data remains a source of powerful but hidden potential. For example, imagine a company invests heavily in building an experienced data science team.

Government 98

Government Data Governance Raw Data Data 98

The Modern Data Company

JULY 31, 2023

By omitting the complexity and debt of data pipelines resulting from multiple fragmented tools in the ‘MAD Ecosystem,’ these loops can be efficiently maintained Process Angle : Administering Data Products involves a combination of technology, culture, and process.

Government 98

Government Data Governance Architecture Data Pipeline 98

Data Engineering Podcast

JANUARY 4, 2021

In order to make those efforts sustainable, the core capability they need is for data scientists and analysts to be able to build and deploy features in a self service manner. As a result the feature store is becoming a required piece of the data platform. What types of data can feature stores manage?

Python 100

Python Data Lake Machine Learning Computer Science 100

Data Engineering Podcast

JANUARY 11, 2021

In this episode he explains how the TerminusDB system is architected to provide a versioned graph storage engine that allows for branching and merging of data sets, how that opens up new possibilities for individuals and teams to work together on building new data repositories. When is TerminusDB the wrong choice?

PostgreSQL 100

PostgreSQL Python Data Lake Computer Science 100

Ascend.io

MAY 23, 2023

The data engineering world is full of tips and tricks on how to handle specific patterns that recur with every data pipeline. In general, the root of the orphaned data problem lies in the interim storage of data between the individual processing steps that make up a data pipeline.

Datasets 52

Datasets Data Pipeline Metadata Database 52

Precisely

JULY 21, 2023

Confluent data streams help accelerate innovation for data and analytics initiatives – but only when sourcing from data you can trust. Precisely – the global leader in data integrity and a proud Connect with Confluent program member – helps you build trust in your data to derive the insights your business users need.

Data Integration 52

Data Integration Kafka Bytes Banking 52