Data Engineering Podcast

JULY 3, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold. Atlan is the metadata hub for your data ecosystem.

Architecture 100

Architecture Metadata MongoDB MySQL 100

ThoughtSpot

JANUARY 31, 2024

Since GPT’s release, LLMs have made it easier to translate natural language into SQL and other forms of code. If you’re ready to spark your own Data Renaissance with ThoughtSpot , I invite you to request a demo. As a result, end users expect faster response times and easier access to meaningful insights.

BI 89

BI Business Intelligence Metadata Machine Learning 89

Towards Data Science

JUNE 19, 2023

Commands When you encounter a new Compose project (using the Tech Slam ‘N Eggs demo project in the screenshots below), try these commands to get your bearings: Note: see this repository if you’d like to run Compose in a Docker container. This may be as simple as a SQL query or may include custom keys for use in your plugins.

PostgreSQL 80

PostgreSQL SQL Python Project 80

Rockset

AUGUST 28, 2023

Typical examples of repeating tasks include scaling compute resources to optimize their usage from a cost and performance perspective, sending automated e-mails or Slack messages with results of a SQL query, materializing views or doing periodic copies of data for development purposes, exporting data to S3 buckets, and so on.

AWS 52

AWS SQL Datasets Metadata 52

Data Engineering Podcast

DECEMBER 16, 2019

The solution to discoverability and tracking of data lineage is to incorporate a metadata repository into your data platform. The metadata repository serves as a data catalog and a means of reporting on the health and status of your datasets when it is properly integrated into the rest of your tools.

Metadata 100

Metadata PostgreSQL Datasets Data Warehouse 100

Monte Carlo

FEBRUARY 13, 2023

Automatically track data reliability for your key tables Table Health Dashboard, available today, provides an automatic, all-in-one view of table health across several key pillars of data observability, including freshness, volume, and schema, as well as specific SQL rules. Reach out to Brandon and the rest of the team for a demo!

BI 52

BI Metadata Data Engineering Data Engineer 52

Cloudera

JULY 14, 2023

They simply read the underlying data (not even full read, they just read the parquet headers) and create corresponding Iceberg metadata files. Query engines (Impala, Hive, Spark) might mitigate some of these problems by using Iceberg’s metadata files. Hive creates Iceberg’s metadata files for the same exact table.

Metadata 70

Metadata Data Warehouse Big Data Ecosystem Java 70

Cloudera

FEBRUARY 14, 2024

A bloated metadata.json file could increase both read/write times because a large metadata file needs to be read/written every time. Regularly expiring snapshots is recommended to delete data files that are no longer needed, and to keep the size of table metadata small. In addition the procedure also deletes the dangling deletes.

Bytes 57

Bytes Metadata Data Lake SQL 57

Ascend.io

DECEMBER 19, 2022

Improved Support for Databricks To highlight our improved Databricks capabilities, our re:Invent booth was next to theirs, and we chose to power our demos with their Lakehouse. Instead, it is a Sankey diagram driven by the same dynamic metadata that runs the Ascend control plane. If you were wondering why, here is your answer.

Data Ingestion 52

Data Ingestion Data Pipeline Metadata AWS 52

Data Engineering Weekly

JULY 2, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork, and SQL grunt work out of building complete customer profiles, so you can quickly ship actionable, enriched data to every downstream team. Write SQL queries without learning SQL? See how it works today.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Monte Carlo

APRIL 24, 2023

However, one of the biggest trends in data lake technologies, and a capability to evaluate carefully, is the addition of more structured metadata creating “lakehouse” architecture. If not paired with Glue, or another metastore/catalog solution, S3 will also lack some of the metadata structure required for more advanced data management tasks.

Data Lake 52

Data Lake Google Cloud Data Warehouse AWS 52

Cloudera

DECEMBER 10, 2020

These techniques range from distributing concurrent query for overall throughput to metadata caching, data caching, and results caching. CDW saves time and resources for end-user workloads but does not compromise metadata availability. 700 / month for security and metadata service . $90 These techniques directly reduce costs. .

Data Warehouse 98

Data Warehouse Cloud Storage Metadata Data 98

Silectis

DECEMBER 15, 2020

Magpie is an enterprise-ready solution built on the powerful Apache Spark, but with language support for SQL, Python, R, and Scala. Request your demo below and we’ll be in touch to get you set up. A Unified Data Engineering Solution Your journey with Magpie starts with a seamless integration with AWS, GCP, or Azure.

Data Engineering 52

Data Engineering Data Engineer Engineering ETL Tools 52

Cloudera

AUGUST 31, 2021

It has a consistent framework that secures and provides governance for all data and metadata on private clouds, multiple public clouds, or hybrid clouds. The data from your existing data warehouse is migrated to the storage option you choose, and all the metadata is migrated into SDX (Shared Data Experiences) layer of Cloudera Data Platform.

Data Warehouse 81

Data Warehouse Database-centric Metadata Cloud 81

Rockset

APRIL 18, 2023

Turn events into real-time features: Rockset turns events into features in real time with SQL ingest transformations. We will use one of the files for the demo but the steps below can be applied to all files. Efficiently compute time-windowed aggregation features, within 1-2 seconds of when the data was generated.

Machine Learning 52

Machine Learning Datasets Database Data Pipeline 52

Monte Carlo

JULY 10, 2023

Data profiling Data profiling uses a tool or programming language to surface metadata about your data. Observability also understands actions like SQL query history for missing tables, relevant GitHub pull requests, and dbt models that may be impacted by incomplete data.

Data Collection 40

Data Collection Data Governance Government Data 40

Data Engineering Podcast

JUNE 26, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold. Atlan is the metadata hub for your data ecosystem.

Datasets 130

Datasets Unstructured Data Metadata MongoDB 130

Rockset

NOVEMBER 7, 2023

As a result, indexing of newly ingested vectors and metadata does not negatively impact search performance. Rockset makes hybrid search as easy as a SQL WHERE clause. Rockset has exploited the power of the search index with an integrated SQL engine so your queries are always executed efficiently.

Cloud 64

Cloud Metadata Database SQL 64

Cloudera

FEBRUARY 8, 2024

It brings the reliability and simplicity of SQL tables to big data while enabling engines like Hive, Impala, Spark, Trino, Flink, and Presto to work with the same tables at the same time. The snapshotId of the source tables involved in the materialized view are also maintained in the metadata.

Metadata 89

Metadata Data Warehouse BI AWS 89

Snowflake

JANUARY 23, 2024

Snowpark ML Operations: Model management The path to production from model development starts with model management, which is the ability to track versioned model artifacts and metadata in a scalable, governed manner. The Snowpark Model Registry API provides simple catalog and retrieval operations on models.

Machine Learning 120

Machine Learning Metadata Python Telecommunication 120

Data Engineering Podcast

NOVEMBER 20, 2022

Datafold shows how a change in SQL code affects your data, both on a statistical level and down to individual rows and values before it gets merged to production. Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold. Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold.

Data Lake 130

Data Lake Data Ingestion MongoDB Scala 130

Cloudera

JULY 13, 2023

In this blog, we will discuss performance improvement that Cloudera has contributed to the Apache Iceberg project in regards to Iceberg metadata reads, and we’ll showcase the performance benefit using Apache Impala as the query engine. Impala can access Hive table metadata fast because HMS is backed by RDBMS, such as mysql or postgresql.

Metadata 70

Metadata Java PostgreSQL Data Warehouse 70

Snowflake

FEBRUARY 29, 2024

This grid, along with the metadata that represents the details of how the grid is constructed and used, is referred to as a spatial index. Caption: A demo Streamlit app visualizes H3 hexagons at different resolutions for the same data. There are 16 layers, or levels, of resolution in H3.

Insurance 85

Insurance SQL BI Consulting 85

Snowflake

NOVEMBER 1, 2023

Snowflake’s latest ML announcements Develop interactively with SQL and Python in Snowflake Notebooks Snowflake Notebooks, in private preview, is a new development interface that offers an interactive, cell-based programming environment for Python and SQL users to explore, process and experiment with data in Snowpark.

Building 105

Building Python SQL Programming Language 105

Snowflake

OCTOBER 2, 2023

To achieve a viable customer 360 solution at scale, a data team must: Build and maintain an identity graph Compute complex user traits Represent funnels as part of the journey Maintain metadata for predictive traits that change over time This work is complex and laborious. Click here to request a demo of Profiles.

Cloud 102

Cloud Building Insurance Data Engineering 102

Monte Carlo

JANUARY 23, 2024

Once ChatGPT launched, they quickly started exploring how to use OpenAI’s API to bring AI inside their BI tools, such as enabling their customers to use plain-language text-to-SQL queries. It can show me how it built that chart, which dataset it used, and show me the metadata.” Who built what for this number to get here? “We

Pipeline-centric 52

Pipeline-centric Database-centric Metadata Unstructured Data 52

Cloudera

NOVEMBER 23, 2021

Atlas provides open metadata management and governance capabilities to build a catalog of all assets, and also classify and govern these assets. Apache Phoenix is a RDBMS, an ANSI SQL interface. Phoenix provides: SQL and JDBC API support. Security and governance policies are set once and applied across all data and workloads.

Database 89

Database Non-relational Database NoSQL Government 89

Cloudera

SEPTEMBER 15, 2022

Structured data (such as name, date, ID, and so on) will be stored in regular SQL databases like Hive or Impala databases. Apache Ozone achieves this significant capability through the use of some novel architectural choices by introducing bucket type in the metadata namespace server. FILE_SYSTEM_OPTIMIZED Bucket (“FSO”).

Systems 87

Systems Hadoop Metadata Telecommunication 87

Monte Carlo

MARCH 1, 2022

With the Monte Carlo SDK, data engineers and scientists can leverage Python in addition to SQL to update settings, pull events to create custom dashboards, glean operational insights, trigger rules, and configure more advanced monitors, among other use cases normally handled by our API. Aggregating metadata.

Python 52

Python Metadata Business Intelligence Data Engineering 52

Rockset

AUGUST 26, 2022

There, the data is converted, tagged with metadata, cleaned, and de-duplicated in preparation for queries. They demoed but quickly eliminated Snowflake. Rockset also powerfully supports SQL, with which all of Meza’s developers were experts. When a change is detected, the data is ingested into a data lake hosted on Amazon S3.

IT 52

IT Food Data Lake Aggregated Data 52

Datakin

OCTOBER 14, 2021

Clone the project repository from GitHub and cd into the new directory: % git clone [link] % cd demo/dbt/stacko Install dbt and the OpenLineage integration inside a Python virtual environment: % python3 -m venv datakin-dbt % source datakin-dbt/bin/activate % pip3 install dbt openlineage-dbt Add an entry to ~/.dbt/profiles.yml That looks good.

Google Cloud 52

Google Cloud Datasets Bytes Metadata 52

Monte Carlo

AUGUST 7, 2023

In the beginning, data observability focused exclusively on detecting, resolving, and preventing data issues by leveraging both metadata and the data itself to piece together a picture of data health. Maybe your analytics engineer made a change to your SQL that filters out the most recent data? Request a demo today.

Coding 52

Coding Data Software Engineer Software Engineering 52

Cloudera

SEPTEMBER 6, 2018

SQL, Python, R, Java, and Scala are widely used in the platform. Cloudera Enterprise also ships with SDX , a unified metadata layer that provides a data catalog, security policies, access logs, and data lineage. It supports a variety of storage engines that can handle raw files, structured data (tables), and unstructured data.

Banking 40

Banking Machine Learning Big Data Scala 40

AltexSoft

OCTOBER 8, 2021

To make a more informed decision, enquire about demos and do your own in-depth research. Oracle Data Integrator has the functionality that automatically analyzes metadata from various data stores, detects patterns, generates, and then applies data quality rules to identify any issues among actual values. MongoDB), SQL databases (e.g.,

Data Integration 52

Data Integration Hadoop Data Warehouse Data Lake 52

Snowflake

MARCH 12, 2024

A business analyst now has the ability to run complex optimization algorithms with just a simple SQL statement. It also provides metadata on what assumptions and fixes were used to inform the address match. You can unlock the power of what Snowflake already has, and what Mapbox can bring to bear.”

Business Analyst 97

Business Analyst Retail Business Intelligence Data Analysis 97

Data Engineering Weekly

JULY 23, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles, so you can quickly ship actionable, enriched data to every downstream team. Write SQL queries without learning SQL? See how it works today.

Data Engineering 52

Data Engineering Data Engineer Engineering Deep Learning 52

Snowflake

APRIL 11, 2024

Data is the lifeblood of modern businesses, but unlocking its true insights often requires complex SQL queries. We’re thrilled to announce the public preview of Snowflake Copilot, a new solution on the bleeding edge of text-to-SQL that simplifies data analysis while maintaining robust governance.

SQL 137

SQL AWS Data Analysis High Quality Data 137