Data Engineering Podcast

OCTOBER 15, 2023

Datafold integrates with dbt, the modern data stack, and seamlessly plugs in your data CI for team-wide and automated testing. Datafold integrates with dbt, the modern data stack, and seamlessly plugs in your data CI for team-wide and automated testing. How have you worked to address that in the Decodable platform and interfaces?

Process 182

Process Building SQL BI 182

Start Data Engineering

JULY 18, 2020

Given Source database connection details (endpoint, port, username, password and database name) Source table to replicate destination schema name run frequency can be set to 10min We are assuming the destination data warehouse is already setup in stitch. here Click on Add Integration button on your dashboard.

Data Warehouse 100

Data Warehouse Database PostgreSQL Data 100

Christophe Blefari

JULY 8, 2023

Aujourd'hui il s'occupe du produit chez Kestra, un orchestrateur open-source développé en France. The new dbt Semantic Layer Following the acquisition of Transform by dbt Labs a few months ago, dbt Core integrates MetricsFlow. PS : est-ce qu'une version française de mon contenu t'intéresse ?

Kafka 130

Kafka PostgreSQL Transportation Data 130

Knowledge Hut

MAY 31, 2023

Whether you are a beginner or an experienced programmer, these top database projects with source codes can help you improve your skills and add value to your portfolio. If you want to learn more about Inventory Control Management, you can check the source code for the project. So, Let's get started!

Database 52

Database Coding MongoDB Project 52

Zalando Engineering

FEBRUARY 2, 2016

Last week members of Zalando’s database engineering team spoke at FOSDEM PGDay , an event hosted by the PostgreSQL community the day before the FOSDEM conference in Brussels. I presented on Patroni , Zalando’s Python-based PostgreSQL controller to provide automatic failover functionality for PostgreSQL.

PostgreSQL 40

PostgreSQL Python Data Integration Database 40

Knowledge Hut

DECEMBER 28, 2023

There are numerous open-source SQL tools available, along with paid ones. Adminer Adminer is a free, open-source database management tool that allows users to manage databases using a web interface. If you have learned SQL, you might think you do not need a tool. It is mostly used for SQL and PL/SQL development.

SQL 52

SQL MySQL PostgreSQL Database 52

Christophe Blefari

JUNE 16, 2023

Change Data Capture (CDC) with PostgreSQL and ClickHouse — This is a nice vendor post about CDC with Kafka as movement layer (using Debezium). This year ~90k people answered, they also integrate a small AI category to measure impact on dev work. The post explains well the architecture you need to make it work.

Programming Language 130

Programming Language SQL PostgreSQL Data 130

Knowledge Hut

MARCH 27, 2024

Relationships Allows the establishment of relationships between different tables, supporting data integrity and normalization. Examples MySQL, PostgreSQL, MongoDB Arrays, Linked Lists, Trees, Hash Tables Scaling Challenges Scales well for handling large datasets and complex queries.

Database 52

Database Algorithm Relational Database PostgreSQL 52

Striim

JULY 7, 2023

Analytical workloads aren’t meant for transactional databases, which are optimized for high latency reads, writes, and data integrity. Examples include PostgreSQL and MySQL, which scale to 20 thousand transactions per second. Make sure archive logs are stored on the source based on the CDC tool’s current specifications.

PostgreSQL 52

PostgreSQL MySQL Data Warehouse Data Pipeline 52

Knowledge Hut

NOVEMBER 17, 2023

You can use Amazon ECS to plan the location of containers and integrate your own scheduler to address business requirements and reminders. Amazon RDS allows access to several acquainted database engines, including Amazon Aurora, MySQL, PostgreSQL, MariaDB, Oracle, and SQL Server. Plus, you only pay for what you consume.

AWS 52

AWS Amazon Web Services PostgreSQL Relational Database 52

Christophe Blefari

APRIL 28, 2023

This is mainly a roadmap of the MeticFlow integration within dbt ecosystem. At the moment we have a dbt Semantic Layer that correspond to YAML definitions and MetricFlow—which was Transform open-source project—that is able to understand the semantics to generates SQL. A lot of changes will happen to MetricFlow incl.

SQL 100

SQL Food PostgreSQL Data 100

Data Engineering Weekly

APRIL 30, 2023

Watch On-demand Trivago: Implementing Data Validation with Great Expectations in Hybrid Environments The article by Trivago discusses the integration of data validation with Great Expectations. link] WTTJ Tech: From PostgreSQL to Snowflake - A data migration story WTTJ Tech has an interesting story to share about data migration.

Data Engineering 62

Data Engineering Data Engineer Engineering PostgreSQL 62

Data Engineering Podcast

APRIL 19, 2020

This was an interesting inside look at building a business on top of open source stream processing frameworks and how to reduce the burden on end users. How do you handle versioning of the queries and integration with the software development lifecycle? What is the workflow for developing and deploying different SQL jobs?

Building 100

Building PostgreSQL MongoDB SQL 100

Edureka

FEBRUARY 13, 2024

It supports multiple database engines, inclusive of MySQL, PostgreSQL, Oracle, and Microsoft SQL Server. It can be integrated with different AWS offerings, inclusive of S3 and EC2, to supply static and dynamic content material efficaciously.

AWS 52

AWS Cloud Computing Amazon Web Services Relational Database 52

Data Engineering Podcast

MAY 10, 2021

Their SDKs and plugins make event streaming easy, and their integrations with cloud applications like Salesforce and ZenDesk help you go beyond event streaming. For users of Tinybird, can you talk through the workflow of getting it integrated into their platform and designing an API from their data?

PostgreSQL 100

PostgreSQL Data Warehouse Data Pipeline Kafka 100

Ascend.io

SEPTEMBER 14, 2023

Plug-and-Play: Many of these libraries are designed to be integrated seamlessly , reducing development time and increasing compatibility across tasks. Integration with Spark: When paired with platforms like Spark, Python’s performance is further amplified. csv') data_excel = pd.read_excel('data2.xlsx')

Data Engineering 52

Data Engineering Data Engineer Python Engineering 52

Data Engineering Podcast

OCTOBER 2, 2022

Ascend automates workloads on Snowflake, Databricks, BigQuery, and open source Spark, and can be deployed in AWS, Azure, or GCP. What are some of the other sources/pieces of information that you integrate into the lineage graph? What are some of the other sources/pieces of information that you integrate into the lineage graph?

IT 100

IT Food PostgreSQL MongoDB 100

Data Engineering Podcast

NOVEMBER 9, 2020

While there are a number of platforms available for building that catalog, many of them are either difficult to deploy and integrate, or expensive to use at scale. Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows. How is that represented?

Building 100

Building PostgreSQL BI Metadata 100

Workfall

JULY 4, 2023

They are responsible for verifying the quality and integrity of the data within the models and transformations. Now, let’s explore how DBT integrates with databases and data warehouses. DBT seamlessly integrates with popular databases and data warehouses, such as PostgreSQL , BigQuery, Redshift , and Snowflake.

Data Warehouse 69

Data Warehouse SQL PostgreSQL Database 69

Striim

JANUARY 4, 2023

Similarly, an engineer might change something in the schema of the source system. A data contract is a formal agreement between the users of a source system and the data engineering team that is extracting data for a data pipeline. Who are the points of contact for the source system and ingestion? What are data contracts?

PostgreSQL 52

PostgreSQL Data Warehouse Data Lake Data 52

Knowledge Hut

JULY 26, 2023

Database applications have become vital in current business environments because they enable effective data management, integration, privacy, collaboration, analysis, and reporting. Amazon RDS (Relational Database Service) is a service provided by AWS for maintaining relational databases such as MySQL, PostgreSQL, SQL Server, and Oracle.

Database 52

Database NoSQL Telecommunication MongoDB 52

Data Engineering Podcast

JANUARY 26, 2020

Your host is Tobias Macey and today I’m interviewing James Campbell about Great Expectations, the open source test framework for your data pipelines which helps you continually monitor and validate the integrity and quality of your data Interview Introduction How did you get involved in the area of data management?

Data Pipeline 100

Data Pipeline PostgreSQL Media Data Validation 100

Data Engineering Podcast

JANUARY 5, 2020

Debezium is an open source platform for reliable change data capture that you can use to build supplemental systems for everything from maintaining audit trails to real-time updates of your data warehouse. What are the data sources that are supported by Debezium? Pulsar, Bookkeeper, Pravega)?

Database 100

Database Kafka PostgreSQL MySQL 100

Knowledge Hut

OCTOBER 4, 2023

Understanding Power BI Requirements As I have mentioned before, Power BI is a revolutionary, remarkable program that enables high-speed data integration and the creation of plenty of reports. Some of the supported data sources are, 1. It includes the manually-entered data or the queers collected from external data sources.

BI 52

BI Systems Raw Data Business Intelligence 52

Data Engineering Podcast

APRIL 1, 2018

With their monitoring, metrics, and log collection agent, including extensive integrations and distributed tracing, you’ll have everything you need to find and fix performance bottlenecks in no time. What was lacking in the existing options (services and self-hosted/open source) that ThreatStack solves for?

Amazon Web Services 100

Amazon Web Services Cloud PostgreSQL Kafka 100

Data Engineering Podcast

NOVEMBER 5, 2023

In this episode Lukas Fittl shares some hard-won wisdom about the causes and solution of many performance bottlenecks and the work that he is doing to shine some light on PostgreSQL to make it easier to understand how to keep it running smoothly.

PostgreSQL 147

PostgreSQL Data Lake High Quality Data SQL 147

Data Engineering Podcast

AUGUST 5, 2019

As organizations grow and data sources proliferate it becomes difficult to keep track of everything, particularly for analysts and data scientists who are not involved with the collection and management of that information. For someone who wants to run their own instance of Amundsen what is involved in getting it deployed and integrated?

PostgreSQL 100

PostgreSQL MongoDB Metadata Media 100

Datakin

SEPTEMBER 24, 2021

Gain visibility of your entire ecosystem Datakin integrates with your data processing and scheduling platforms to give you full visibility on the flow of data across teams and systems. We continuously work on the development of additional OpenLineage integrations , and plan to make new ones available regularly.

PostgreSQL 52

PostgreSQL Datasets Data Pipeline Data Process 52

Data Engineering Podcast

MARCH 2, 2020

Summary Building applications on top of unbounded event streams is a complex endeavor, requiring careful integration of multiple disparate systems that were engineered in isolation. The ksqlDB project was created to address this state of affairs by building a unified layer on top of the Kafka ecosystem for stream processing.

Kafka 100

Kafka Process PostgreSQL MySQL 100

Knowledge Hut

DECEMBER 21, 2023

In the database ecosystem, Postgres is one of the top open-source databases, and one of the most widely used PSQL tools for managing PostgreSQL is pgAdmin. To run PostgreSQL instances on the Azure cloud, Azure offers Azure Database for PostgreSQL. What are PostgreSQL Tools?

PostgreSQL 52

PostgreSQL MySQL MongoDB SQL 52

Data Engineering Podcast

OCTOBER 9, 2018

And the team at Metis Machine has shipped a proof-of-concept integration between the Skafos machine learning platform and the Tableau business intelligence tool, meaning that your BI team can now run the machine learning models custom built by your data science team. What are the typical use cases for customers running MemSQL?

PostgreSQL 100

PostgreSQL BI Data Warehouse Machine Learning 100

Data Engineering Podcast

NOVEMBER 2, 2020

Along with the benefits there are some additional complexities to consider, including how to safely integrate new data sources or test out changes to existing pipelines. Summary Data lakes are gaining popularity due to their flexibility and reduced cost of storage.

Data Lake 100

Data Lake PostgreSQL Datasets Machine Learning 100

Data Engineering Podcast

DECEMBER 16, 2019

The metadata repository serves as a data catalog and a means of reporting on the health and status of your datasets when it is properly integrated into the rest of your tools. What are some of the tools or platforms that are currently integrated with Marquez and what additional integrations would you like to see?

Metadata 100

Metadata PostgreSQL Datasets Data Warehouse 100

Zalando Engineering

NOVEMBER 8, 2023

Introduction This blog post describes a recent contribution from Zalando to the Postgres JDBC driver to address a long-standing issue with the driver’s integration with Postgres’ logical replication that resulted in runaway Write-Ahead Log (WAL) growth. So what is happening at a low level during this event-streaming process?

PostgreSQL 139

PostgreSQL Java Database Bytes 139

Data Engineering Podcast

OCTOBER 14, 2019

Attendees will hear from companies including Walmart, Netflix, Google, and DBS Bank on how they leveraged technologies such as Alluxio, Presto, Spark, Tensorflow, and you will also hear from creators of open source projects including Alluxio, Presto, Airflow, Iceberg, and more!

Data Warehouse 100

Data Warehouse PostgreSQL AWS Programming Language 100

Knowledge Hut

APRIL 3, 2023

They also work closely with front-end developers to ensure the seamless integration of the application's user interface and functionality. Backend developers work with programming languages such as Java, Python, Ruby, and PHP, as well as databases such as MySQL, MongoDB, and PostgreSQL. Who is a Backend Developer?

Programming Language 52

Programming Language Computer Science Portfolio PostgreSQL 52

Rockset

JUNE 1, 2021

MySQL and PostgreSQL are widely used as transactional databases. By integrating MySQL and PostgreSQL with Rockset, you can easily scale out to handle demanding analytics. we did an integration with RDS MySQL on Rockset. This mitigates the need to tune these relational databases to handle heavy analytical workloads.

MySQL 40

MySQL PostgreSQL AWS Relational Database 40