Christophe Blefari

APRIL 5, 2024

DBRX, a new state-of-the-art open LLM — Databricks has to be an AI company ( bragging vs. Snowflake ). How we built Text-to-SQL at Pinterest — Pinterest open-sourced a tool called Querybook that they used to access Pinterest data every day. In order to boost usage they developed a text-to-SQL feature.

SQL 130

SQL Data Python Datasets 130

Christophe Blefari

FEBRUARY 3, 2024

The text-to-sql problem Every once in a while the people are trying to give a shot at the text-to-sql problem. Obviously when it comes to self-service we need a layer that does a text-to-sql conversion. The platform is built on-top of Databricks with a lot of other logos revolving around making sense of the lakehouse platform.

MongoDB 130

MongoDB SQL Data Data Warehouse 130

Data Engineering Podcast

FEBRUARY 11, 2024

For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst powers petabyte-scale SQL analytics fast, at a fraction of the cost of traditional methods, so that you can meet all your data needs ranging from AI to data applications to complete analytics. Don't forget to check out our other shows.

Data Lake 147

Data Lake High Quality Data Government Data Pipeline 147

Christophe Blefari

OCTOBER 9, 2023

Fast News ⚡️ Microsoft Fabric: should Databricks be worried? — Vantage did a price analysis between Microsoft Fabric and Databricks. Generally Fabric pricing is simpler because fresh and new but for more complex stuff Databricks still shine. Python 3.12 Contentsquare acquires Heap.

Python 130

Python Data Database Java 130

Data Engineering Podcast

AUGUST 13, 2023

Summary Data pipelines are the core of every data product, ML model, and business intelligence dashboard. The folks at Rivery distilled the seven principles of modern data pipelines that will help you stay out of trouble and be productive with your data. Can you start by defining what you mean by a "modern" data pipeline?

Data Pipeline 130

Data Pipeline BI SQL Business Intelligence 130

ThoughtSpot

FEBRUARY 9, 2023

Two years ago, we announced our Databricks partnership —including the launch of ThoughtSpot for Databricks, which gives joint customers the ability to run ThoughtSpot search queries directly on the Databricks Lakehouse without the need to move any data.

Government 82

Government SQL Machine Learning Cloud 82

Data Engineering Weekly

JUNE 29, 2023

I’m at the Databricks summit as I’m writing my thoughts on the Snowflake and Databricks announcements. The Race to Deprecate Data Analyst Databricks announced the support of Natural Language Query as part of LakeHouseIQ. Snowflake is a DataLake Platform Snowflake is moving beyond a SQL data warehouse.

Data Engineering 76

Data Engineering Data Engineer Google Cloud Engineering 76

Towards Data Science

OCTOBER 17, 2023

By: Jeff Chou, Stewart Bryson Image by Los Muertos Crew Databricks’ SQL warehouse products are a compelling offering for companies looking to streamline their production SQL queries and warehouses. What are Databricks’ SQL warehouse offerings? AWS EC2 bill).

SQL 52

SQL Cloud Data Science AWS 52

Data Engineering Podcast

JANUARY 22, 2023

Summary The most interesting and challenging bugs always happen in production, but recreating them is a constant challenge due to differences in the data that you are working with. Building your own scripts to replicate data from production is time consuming and error-prone. Don't forget to check out our other shows.

PostgreSQL 130

PostgreSQL MongoDB MySQL Machine Learning 130

Knowledge Hut

SEPTEMBER 26, 2023

The Microsoft Azure Synapse and Databricks platforms are two strong competitors in this space. To help data architects, engineers, and analysts choose the platform that best meets their objectives, we'll examine the essential characteristics, advantages, and use cases of Azure Synapse vs. Databricks in this thorough comparison.

Data Lake 52

Data Lake Database-centric Pipeline-centric Machine Learning 52

Scott Logic

APRIL 21, 2024

Apache Spark SQL has gone through a major evolution, now supporting ANSI SQL, and adding many new features and making many performance improvements. Improvements to Apache Spark SQL Apache Spark’s SQL functionality is a major part of the platform, being one of the two main ways of manipulating data. The latest V3.5.1

Scala 52

Scala Java SQL Python 52

Christophe Blefari

SEPTEMBER 28, 2023

Over the years Cloudera logo has been replaced by Snowflake and Databricks ones. An easy-to-manage central storage and querying and transforming layer in SQL. The central storage can be cloud storage, a warehouse, a real-time system, while the SQL engine can be a data warehouse or a dedicated processing engine.

Cloud Storage 147

Cloud Storage Big Data Hadoop SQL 147

Data Engineering Podcast

NOVEMBER 20, 2022

With their new managed database service you can launch a production ready MySQL, Postgres, or MongoDB cluster in minutes, with automated backups, 40 Gbps connections from your application hosts, and high throughput SSDs. By the time errors have made their way into production, it’s often too late and damage is done.

Data Lake 130

Data Lake Data Ingestion MongoDB Scala 130

Monte Carlo

OCTOBER 16, 2023

Monte Carlo and Databricks have been long-time partners , which is why we are excited to announce that, starting today, data teams can bring data observability to their Databricks environment in just a few clicks via Partner Connect. Step 2: Choose the SQL warehouse and schema you’d like to integrate.

High Quality Data 52

High Quality Data SQL Accessible Accessibility 52

Monte Carlo

MARCH 25, 2024

Beyond our recognition in the category, we were also named one of G2Crowd’s 100 Fastest Growing Products for 2024 , signifying our unrivaled customer adoption across the U.S., Supporting new table formats in Snowflake and Databricks. EMEA, and other markets. Adding capabilities for Azure Synapse Analytics and Microsoft Data Fabric.

MySQL 64

MySQL Government Data Governance Kafka 64

Christophe Blefari

SEPTEMBER 1, 2023

dbt tests 🧪 dbt Core proposition has been to bring software engineering practices to SQL development. Snowflake fine-tuning Code Llama for SQL generation — With these fine-tuning it seems they are close to GPT-4 accuracy in text-to-SQL. Enjoy the news recap. New features in Airflow — with 2.7

Food 130

Food Data SQL Python 130

Data Engineering Podcast

NOVEMBER 6, 2022

With their new managed database service you can launch a production ready MySQL, Postgres, or MongoDB cluster in minutes, with automated backups, 40 Gbps connections from your application hosts, and high throughput SSDs. By the time errors have made their way into production, it’s often too late and damage is done.

MongoDB 130

MongoDB Scala MySQL Data Lake 130

Monte Carlo

JANUARY 16, 2024

Data warehouses like Snowflake or Google BigQuery have made improvements integrating streaming data capabilities while Databricks has added ACID properties via delta tables and its new Unity Catalog. For example, Snowflake’s Cortex and Databricks’ LakehouseAI demonstrate this trend of integration and innovation.

Data Lake 52

Data Lake Data Warehouse Unstructured Data AWS 52

Monte Carlo

JULY 19, 2023

The first layer of your stack will generally fall into one of three categories: a data warehouse solution like Snowflake that handles predominantly structured data; a data lake that focuses on larger volumes of unstructured data; and a hybrid solution like Databricks’ Lakehouse that combines elements of both. Image courtesy of Databricks.

Building 96

Building Business Intelligence Cloud Storage BI 96

Towards Data Science

OCTOBER 7, 2023

Across industries, executives are pushing their data leaders to build AI-powered products that will save time, drive revenue, or give them a competitive advantage. And financial products and firms are ripe with potential use cases for generative AI. Image courtesy of author. And we see a lot of value there.”

Unstructured Data 75

Unstructured Data Finance SQL Media 75

Knowledge Hut

MARCH 28, 2024

Implemented and managed data storage solutions using Azure services like Azure SQL Database , Azure Data Lake Storage, and Azure Cosmos DB. Education & Skills Required Proficiency in SQL, Python, or other programming languages. Experience with Azure data services like Azure SQL Database, Azure Data Factory, and Azure Databricks.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Warehouse 52

Monte Carlo

OCTOBER 4, 2023

Across industries, executives are pushing their data leaders to build AI-powered products that will save time, drive revenue, or give them a competitive advantage. And financial products and firms are ripe with potential use cases for generative AI. The hype around generative AI is real, and data and ML teams are feeling the heat.

Unstructured Data 72

Unstructured Data Finance SQL Media 72

Monte Carlo

MARCH 4, 2024

And as crazy as it sounds at first blush, innovations in table formats got prime air-time at both Snowflake and Databricks annual conferences in 2023. Now, Delta or Iceberg tables are now sufficient in many environments that leverage Databricks, Snowflake, or other means of compute. Our promise: we will show you the product.

Metadata 52

Metadata Data Lake Data Warehouse Data Engineering 52

Ascend.io

FEBRUARY 6, 2023

If you’ve read this week’s release notes , you’ll have seen “Added ability to configure or override connection details at the Data Service level in BigQuery and Databricks Data Planes.” Data Services Listing with BigQuery, Snowflake, and Databricks. Let’s first brush up on Ascend product terminology. So what do we mean by that?

Cloud 52

Cloud Data Ingestion Raw Data Data Pipeline 52

Ascend.io

FEBRUARY 6, 2023

If you’ve read this week’s release notes , you’ll have seen “Added ability to configure or override connection details at the Data Service level in BigQuery and Databricks Data Planes.” Data Services Listing with BigQuery, Snowflake, and Databricks. Let’s first brush up on Ascend product terminology. So what do we mean by that?

Cloud 52

Cloud Data Ingestion Raw Data Data Pipeline 52

Monte Carlo

AUGUST 25, 2023

Teams using a data warehouse usually leverage SQL queries for analytics use cases. With pre-built functionalities and robust SQL support, data warehouses are tailor-made to enable swift, actionable querying for data analytics teams working primarily with structured data. Let’s dive in.

Data Lake 52

Data Lake Data Warehouse Unstructured Data Raw Data 52

Monte Carlo

JANUARY 27, 2024

The first layer of your stack will generally fall into one of three categories: a data warehouse solution like Snowflake that handles predominantly structured data; a data lake that focuses on larger volumes of unstructured data; and a hybrid solution like Databricks’ Lakehouse that combines elements of both. Image courtesy of Databricks.

Building 52

Building Business Intelligence Cloud Storage BI 52

Data Engineering Podcast

JUNE 12, 2022

With their new managed database service you can launch a production ready MySQL, Postgres, or MongoDB cluster in minutes, with automated backups, 40 Gbps connections from your application hosts, and high throughput SSDs. Open Source DataHub is running in production at several companies like Peloton, Optum, Udemy, Zynga and others.

Unstructured Data 130

Unstructured Data MongoDB Scala MySQL 130

Data Engineering Weekly

MARCH 31, 2024

Intuit: How Intuit data analysts write SQL 2x faster with the internal GenAI tool The productivity increase with GenAI is undeniable, and several startups are trying to solve the Text2SQL generation problem. link] Databricks: PySpark in 2023 - A Year in Review Can we safely say PySpark killed Scala-based data pipelines?

Data Engineering 62

Data Engineering Data Engineer Engineering Scala 62

Christophe Blefari

FEBRUARY 18, 2023

To me it stops here, all the marketing derivation of it saying we do data products using DataOps methodology is just marketing. Balancing quality and coverage with our data validation framework — Dropbox tech team developed a data validation framework in SQL. This is why DataOps is important.

Software Engineer 100

Software Engineer Software Engineering Data Validation Data 100

Ripple Engineering

JANUARY 29, 2024

For Ripple's product capabilities, the Payments team of Ripple, for example, ingests millions of transactional records into databases and performs analytics to generate invoices, reports, and other related payment operations.   Ripple Data Consumers query the data from the lake storage using the SQL strategy.

Database-centric 52

Database-centric Pipeline-centric NoSQL High Quality Data 52

Data Engineering Podcast

NOVEMBER 13, 2022

Summary CreditKarma builds data products that help consumers take advantage of their credit and financial capabilities. In this episode Vishnu Venkataraman shares the journey that he and his team have taken to build and evolve their systems and improve the product offerings that they are able to support.

MongoDB 100

MongoDB Scala MySQL Google Cloud 100

Data Engineering Podcast

OCTOBER 30, 2022

With their new managed database service you can launch a production ready MySQL, Postgres, or MongoDB cluster in minutes, with automated backups, 40 Gbps connections from your application hosts, and high throughput SSDs. By the time errors have made their way into production, it’s often too late and damage is done.

Engineering 100

Engineering MongoDB Scala MySQL 100

Data Engineering Podcast

OCTOBER 23, 2022

In this episode field CTO Manjot Singh shares his experiences as an early user of MySQL and MariaDB and explains how the suite of products being built on top of the open source foundation address the growing needs for advanced storage and analytical capabilities. What are the use cases/capabilities that you are targeting with those products?

Database 100

Database MySQL Cloud MongoDB 100

Data Engineering Podcast

OCTOBER 16, 2022

Dremio is one of the companies leading the development of products and services that support the open lakehouse. With their new managed database service you can launch a production ready MySQL, Postgres, or MongoDB cluster in minutes, with automated backups, 40 Gbps connections from your application hosts, and high throughput SSDs.

Data Lake 100

Data Lake Food MongoDB Scala 100

Monte Carlo

JANUARY 23, 2024

In 2024, data engineers can run big jobs very quickly thanks to the compute power of BigQuery, Snowflake, Firebolt, Databricks, and other cloud warehousing technologies. If I want to change a column name, it would be fairly hard to do, because you have to rerun your ETL and change your SQL,” said Maxime. One example?

Data Engineering 98

Data Engineering Data Engineer Engineering Data 98

Data Engineering Weekly

JANUARY 29, 2023

link] Sponsored: Upsolver - Write a SQL Query, Get a Data-in-Motion Pipeline! Upsolver SQLake lets you process fast-moving data by simply writing a SQL query. The author writes from a product manager perspective things to consider while adopting data-driven customer-facing data products.

Data Engineering 52

Data Engineering Data Engineer Engineering Deep Learning 52