Data Engineering Podcast

AUGUST 13, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. Data stacks are becoming more and more complex.

Machine Learning 100

Machine Learning Pipeline-centric Database-centric MongoDB 100

Meltano

JANUARY 25, 2023

SQL Jinja isn’t enough SQL + Jinja isn’t enough – why we need dataframes by Furcy Pin “In October 2021, Max Beauchemin, creator of Apache Airflow, wrote an excellent article about 12 trends that are currently affecting data engineers.

Pipeline-centric 52

Pipeline-centric Database-centric Data Warehouse SQL 52

Data Engineering Weekly

JULY 2, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork, and SQL grunt work out of building complete customer profiles, so you can quickly ship actionable, enriched data to every downstream team. LinkedIn write about Hoptimator for auto generated Flink pipeline with multiple stages of systems.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Knowledge Hut

JUNE 20, 2023

In addition, they are responsible for developing pipelines that turn raw data into formats that data consumers can use easily. Languages Python, SQL, Java, Scala R, C++, Java Script, and Python Tools Kafka, Tableau, Snowflake, etc. A machine learning engineer or ML engineer is an information technology professional.

Machine Learning 52

Machine Learning Data Engineering Data Engineer Engineering 52

Ripple Engineering

JANUARY 29, 2024

Ripple Data Consumers query the data from the lake storage using the SQL strategy.   A lack of a centralized system makes building a single source of high-quality data difficult. It can lead to expensive, slow, and unmaintainable systems.

Database-centric 52

Database-centric Pipeline-centric NoSQL High Quality Data 52

Knowledge Hut

NOVEMBER 17, 2023

Becoming an Azure Data Engineer in this data-centric landscape is a promising career choice. The main duties of an Azure Data Engineer are planning, developing, deploying, and managing the data pipelines. Master data integration techniques, ETL processes, and data pipeline orchestration using tools like Azure Data Factory.

Data Engineering 52

Data Engineering Data Engineer Engineering Scala 52

Snowflake

JUNE 28, 2023

At this year’s Summit, we are excited to announce a series of advancements to Snowpark runtimes and libraries, making the deployment and processing of non-SQL code in Snowflake even simpler, faster, and more secure. Snowpark — Set of libraries and runtimes for secure deployment and processing of non-SQL code on the Snowflake Data Cloud.

Python 52

Python Accessible Accessibility Pipeline-centric 52

Snowflake

JUNE 28, 2023

A key highlight at this year’s Summit, is all the innovation enhancing and expanding Snowpark libraries and runtimes developers can use to deploy and process your non-SQL code even easier and more secure. Simplified streaming pipelines in Snowflake We are expanding our streaming capabilities with Dynamic Tables (public preview).

Pipeline-centric 57

Pipeline-centric Programming Language Python Government 57

Data Engineering Weekly

JANUARY 8, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. Pipelines for data in motion can quickly turn into DAG hell. Sign up free to test out the tool today.

Data Engineering 40

Data Engineering Data Engineer Engineering Pipeline-centric 40

Ascend.io

OCTOBER 20, 2023

Instead of burning through credits on inefficient SQL queries, Snowflake would rather you channel those resources into higher-impact projects. The Nuances of Snowflake Costing Snowflake’s pricing strategy is an exemplification of its user-centric approach: pay for what you use. No one wins with wasteful spending.

Pipeline-centric 52

Pipeline-centric IT Data Pipeline Bytes 52

François Nguyen

MARCH 22, 2021

To illustrate that, let’s take Cloud SQL from the Google Cloud Platform that is a “Fully managed relational database service for MySQL, PostgreSQL, and SQL Server” It looks like this when you want to create an instance. You are starting to be an operation or technology centric data team.

Architecture 130

Architecture Technology Google Cloud Metadata 130

Knowledge Hut

SEPTEMBER 26, 2023

Microsoft Azure's Azure Synapse, formerly known as Azure SQL Data Warehouse, is a complete analytics offering. This cloud-centric approach ensures scalability, flexibility, and cost-efficiency for your data workloads. SQL: Enables users to query and manipulate data using standard SQL, making it accessible to a broad audience.

Data Lake 52

Data Lake Database-centric Pipeline-centric Machine Learning 52

Knowledge Hut

FEBRUARY 27, 2023

Data engineering builds data pipelines for core professionals like data scientists, consumers, and data-centric applications. A data engineer can be a generalist, pipeline-centric, or database-centric. Who is Data Engineer, and What Do They Do?

Data Engineering 52

Data Engineering Data Engineer Database-centric Pipeline-centric 52

Ascend.io

MAY 31, 2023

In the modern world of data engineering, two concepts often find themselves in a semantic tug-of-war: data pipeline and ETL. Fast forward to the present day, and we now have data pipelines. Data Ingestion Data ingestion is the first step of both ETL and data pipelines. However, they are not just an upgraded version of ETL.

Data Pipeline 52

Data Pipeline ETL Tools Pipeline-centric Data Warehouse 52

Ascend.io

SEPTEMBER 14, 2023

Here’s how Python stacks up against SQL, Java, and Scala based on key factors: Feature Python SQL Java Scala Performance Offers good performance which can be enhanced using libraries like NumPy and Cython. Interpreter / Compiler Interpreted Executed by a database engine, interpreting and executing SQL statements.

Data Engineering 52

Data Engineering Data Engineer Python Engineering 52

Knowledge Hut

JANUARY 30, 2024

This can involve SQL queries or ETL (Extract, Transform, Load) processes. Customer Interaction Data: In customer-centric industries, extracting data from customer interactions (e.g., Database Queries: When dealing with structured data stored in databases, SQL queries are instrumental for data extraction.

ETL Tools 52

ETL Tools Database-centric Data Mining Data Cleanse 52

Snowflake

AUGUST 21, 2023

The four data mesh principles Fundamentally, data mesh architectures are built on four core principles: Domain-centric ownership and architecture: The teams that work within/most closely with each respective domain are responsible for its functionality and upkeep.

Pipeline-centric 52

Pipeline-centric Architecture Government Data Architect 52

Monte Carlo

NOVEMBER 14, 2023

Data orchestration tools minimize manual intervention by automating the movement of data within data pipelines. Or are there other data orchestration tools that could be a better fit for your pipeline needs? Each block of code in your pipeline produces data that can be versioned, partitioned, and cataloged for future reference.

Pipeline-centric 40

Pipeline-centric Google Cloud Data Workflow Python 40

Data Engineering Weekly

APRIL 2, 2023

Meta: Presto - A Decade of SQL Analytics at Meta Presto and Kafka are the two systems that greatly impacted data infrastructure in the last decade. Watch On-demand Picnic: Using Change Data Capture for Warehouse Analytics Picnic writes about its Change Data Capture pipeline and the lessons learned while integrating Debezium with Postgres.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Grouparoo

OCTOBER 6, 2021

This provided a nice overview of the breadth of topics that are relevant to data engineering including data warehouses/lakes, pipelines, metadata, security, compliance, quality, and working with other teams. 7 Be Intentional About the Batching Model in Your Data Pipelines Different batching models. Test system with A/A test.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Monte Carlo

JANUARY 23, 2024

Gen AI can whip up serviceable code in moments — making it much faster to build and test data pipelines. Once ChatGPT launched, they quickly started exploring how to use OpenAI’s API to bring AI inside their BI tools, such as enabling their customers to use plain-language text-to-SQL queries. Can I see the pipeline?

Pipeline-centric 52

Pipeline-centric Database-centric Metadata Unstructured Data 52

DataKitchen

NOVEMBER 4, 2021

DataOps works on top of your existing data, allowing you to use your favorite tools , whether you prefer SQL, Alteryx, Tableau, Looker or whatever else. These limited-term databases can be generated as needed from automated recipes (orchestrated pipelines and qualification tests) stored and managed within the process hub. .

Process 98

Process Data Process Pharmaceutical Data Lake 98

Rockset

NOVEMBER 1, 2022

Elasticsearch has become ubiquitous as an index centric datastore for search and rose in tandem with the popularity of the internet and Web2.0. Furthermore, Rockset supports SQL joins so there is no data denormalization required upstream. SQL has become the lingua franca for expressing queries on databases of all varieties.

Database-centric 52

Database-centric Pipeline-centric SQL Aggregated Data 52

DareData

JULY 29, 2020

One paper suggests that there is a need for a re-orientation of the healthcare industry to be more "patient-centric". Furthermore, clean and accessible data, along with data driven automations, can assist medical professionals in taking this patient-centric approach by freeing them from some time-consuming processes.

Healthcare 52

Healthcare Data Pipeline Medical Pipeline-centric 52

DataKitchen

FEBRUARY 21, 2023

As a result, a less senior team member was made responsible for modifying a production pipeline. ” Marcus will not fix his challenges by helping his team write SQL faster. To add fuel to the fire, a few weeks ago, one of the most talented people in the group left.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Maxime Beauchemin

JANUARY 20, 2017

Data is simply too centric to the company’s activity to have limitation around what roles can manage its flow. Required Skills SQL mastery: if english is the language of business, SQL is the language of data. While generations of technologies age and fade, SQL is still standing strong as the lingua franca of data.

Data Engineering 130

Data Engineering Data Engineer Engineering ETL Tools 130

Monte Carlo

NOVEMBER 10, 2022

That means you have more API integrations and data pipelines that can fail. While many legacy CDPs are starting to offer ways to reach back to the data warehouse, not too many marketers are going to be writing SQL queries to pull operational data nor are too many data professionals diving that deep into the weeds of the marketing stack.

Pipeline-centric 52

Pipeline-centric Data Warehouse Data Data Ingestion 52

Databand.ai

DECEMBER 20, 2022

Michael is certified in text mining and analytics, SQL, Python, Tableau, and more. Eric is certified in Lean Six Sigma and experienced in Python, SQL, and machine learning. On LinkedIn, he posts regularly about Excel, dashboards, data science, data analytics, and SQL.

Data Analytics 52

Data Analytics Google Cloud Data Science Data Mining 52

Picnic Engineering

DECEMBER 19, 2023

Combining efficient incident handling, establishing resilience by design, and strict adherence to SLOs are pivotal in ensuring our services remain resilient, reliable, stable, and user-centric. Now we use simulation-driven automated testing as a stage of our CI/CD pipeline, totaling over 900 pull requests since the release!

Transportation 52

Transportation Pipeline-centric Database-centric Python 52

Monte Carlo

JANUARY 9, 2024

“It gets hard when you’re trying to apply data quality the same way we think about storage and pipelines,” says Chad. Now, as more companies become more data-centric and data becomes more front and center, ROI becomes much more clear.” As long as the trend is fine, stakeholders don’t really care if it’s correct.

Pipeline-centric 52

Pipeline-centric Database-centric Data Manufacturing 52

Monte Carlo

JULY 15, 2021

As data pipelines become increasingly complex, investing in a data quality solution is becoming an increasingly important priority for modern data teams. And as data pipelines become increasingly complex, reactive approaches to solving data quality issues are not enough. But should you build it—or buy it?

Building 40

Building Pipeline-centric Data Engineering Data Engineer 40

ProjectPro

OCTOBER 3, 2016

has expanded its analytical database support for Apache Hadoop and Spark integration and also to enhance Apache Kafka management pipeline. To compete in a field of diverse data tools, Vertica 8.0 Vertical analytic platform could access hadoop data before but with Vertica 8.0

Hadoop 52

Hadoop Database-centric Pipeline-centric Big Data 52

ProjectPro

DECEMBER 16, 2021

The job of a Machine Learning Engineer is to maintain the software architecture, run data pipelines to ensure seamless flow in the production environment. They are also expected to collaborate with appropriate business stakeholders to identify and analyze the requirements, scope, and hint resolution.

Machine Learning 52

Machine Learning Engineering Pipeline-centric Database-centric 52

Databand.ai

DECEMBER 29, 2022

Seth champions exponential change by combining existing technologies and data science to create industrial scale processes including innovative automation, IT systems and analysis pipelines to support these. He is active on LinkedIn, posting about data strategy, data analytics, data science, career advice, SQL, and Python.

BI 52

BI Consulting Data Science Data Governance 52

Knowledge Hut

NOVEMBER 15, 2023

Looking for a position to test my skills in implementing data-centric solutions for complicated business challenges. Example 6: A well-qualified Cloud Engineer is looking for a position responsible for developing and maintaining automated CI/CD and deploying pipelines to support platform automation.

Finance 98

Finance Business Intelligence Database-centric Certification 98

U-Next

AUGUST 17, 2022

SQL – A database may be used to build data warehousing, combine it with other technologies, and analyze the data for commercial reasons with the help of strong SQL abilities. The job description for Data Engineers may require them to eventually specialize in one or more SQL kinds (such as advanced modeling, big data, etc.).

Data Engineering 52

Data Engineering Data Engineer Pipeline-centric Database-centric 52

Confluent

JUNE 18, 2019

Who has never seen an application use RDBMS SQL statements to run searches? Using SQL to run your search might be enough for your use case, but as your project requirements grow and more advanced features are needed—for example, enabling synonyms, multilingual search, or even machine learning—your relational database might not be enough.

Architecture 96

Architecture Building Kafka Database-centric 96