Jesse Anderson

JANUARY 9, 2023

I’ve spent time trying to think of something Flink can’t do that Kafka Streams can do and I can’t. This addition means that Databricks and Confluent will go head-to-head more. It will put even more importance on Databricks’ Project Lightspeed to deliver on its goals.

Kafka 147

Kafka Coding Technology SQL 147

Christophe Blefari

JUNE 3, 2023

Sometimes I feel like I'm trying to find a needle in a stack. Data Economy 💰 Databricks acquires bit.io — bit.io When looking at the press release Databricks acquisition is a team acquisition to improve their own developper experience. Or several needles. dbt at Zendesk ; setting foundations for scalability.

Data Pipeline 130

Data Pipeline Data SQL Algorithm 130

Data Engineering Podcast

OCTOBER 15, 2023

Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack This episode is brought to you by Datafold – a testing automation platform for data engineers that finds data quality issues before the code and data are deployed to production. What was the process for adding full Java support in addition to SQL?

Process 182

Process Building SQL BI 182

Data Engineering Podcast

AUGUST 13, 2023

Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack This episode is brought to you by Datafold – a testing automation platform for data engineers that finds data quality issues before the code and data are deployed to production. Email hosts@dataengineeringpodcast.com ) with your story.

Data Pipeline 130

Data Pipeline BI SQL Business Intelligence 130

Christophe Blefari

APRIL 5, 2024

DBRX, a new state-of-the-art open LLM — Databricks has to be an AI company ( bragging vs. Snowflake ). I did not yet analysed the new map, but I'll try to do it soon. I guess there is a video associated with the slides, but I don't have it. This week they released a new open model that performs great.

SQL 130

SQL Data Python Datasets 130

Data Engineering Podcast

APRIL 2, 2023

As a venture capital investor Matt Turck has been trying to keep track of the main trends and has compiled his findings into the MAD (ML, AI, and Data) landscape reports each year. Summary The data ecosystem has been building momentum for several years now. Email hosts@dataengineeringpodcast.com ) with your story.

Hadoop 130

Hadoop Machine Learning Python Architecture 130

Towards Data Science

OCTOBER 17, 2023

By: Jeff Chou, Stewart Bryson Image by Los Muertos Crew Databricks’ SQL warehouse products are a compelling offering for companies looking to streamline their production SQL queries and warehouses. What are Databricks’ SQL warehouse offerings? From a cost perspective, both classic and pro run inside the user’s cloud environment.

SQL 52

SQL Cloud Data Science AWS 52

Data Engineering Weekly

NOVEMBER 26, 2023

Databricks is already doing a lot of these move-up stack products. Databricks announced a fleet of features for catalogs, governance, observability, and orchestration. It is Databricks vs. Snowflake, where both companies ran their data conference on the same day. The categories are merging.

Data Engineering 70

Data Engineering Data Engineer Engineering Kafka 70

Christophe Blefari

FEBRUARY 18, 2023

It was a try I'll see in the future how I can do it better. Data engineering is mainly software engineering applied to data, or at least we try. Still, what is a metrics store ? You can check out the post extracted from the newsletter. I personally do share this perspective.

Software Engineer 100

Software Engineer Software Engineering Data Validation Data 100

Monte Carlo

JUNE 26, 2023

Table of Contents Setting the stage Pre-conference Rumblings Thoughtspot Acquires Mode GenAI Strategies: Databricks Acquires MosaicML & Snowflake Announces Microsoft Partnership Lakehouse Apps = Steamlit Apps? Databricks likely has their eye on going public, and targeting Snowflake’s record as largest tech IPO of all time.

BI 59

BI Machine Learning Data Science Cloud 59

Monte Carlo

JANUARY 3, 2024

Image courtesy of Databricks. Here’s an oversimplification of RAG application architecture: RAG architecture combines information retrieval with a text generator model, so it has access to your database while trying to answer a question from the user. RAG application architecture. Snowflake data cloud.

Technology 113

Technology Database Data Governance Data Engineering 113

Monte Carlo

MARCH 25, 2024

Our developers didn’t know where to investigate and data engineering teams were trying to humpty dumpty fix dashboards. Supporting new table formats in Snowflake and Databricks. In 2021, we were flying blind. That was a huge waste of time. Deepening our integrations with modern orchestration tools like dbt and Airflow.

MySQL 64

MySQL Government Data Governance Kafka 64

Christophe Blefari

JANUARY 14, 2023

I want now to try a group by. sort_values(ascending=False) And lastly (because if I continue the newsletter gonna be too long for you to read), I just try to convert a str Series to datetime. I want now to try a group by. Enjoy the Data News. It looks like stuff I know with pandas. df.groupby("Station").agg([pl.count()]).sort("count",

Python 130

Python Kafka Data Scala 130

Data Engineering Podcast

APRIL 3, 2022

With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. Go to dataengineeringpodcast.com/linode today and get a $100 credit to try out a Kubernetes cluster of your own.

Designing 100

Designing Data Warehouse Data Engineering Data Engineer 100

Striim

FEBRUARY 8, 2024

Today with the combined power of Microsoft Azure, Databricks, and Striim, American Airlines TechOps has completed the vision of digital transformation and was able to cut over to production at a global scale within 12 weeks. The sky’s the limit when it comes to the potential of real-time data in transforming aviation.

MongoDB 52

MongoDB Data Analytics Data Data Integration 52

Knowledge Hut

JANUARY 18, 2024

Databricks Databricks is an American company that works on the system of Data Lakehouse. So, if you are looking for a complete course to teach you A to Z about Data Science at your own pace, give this course a try by searching for the best and google Knowledgehut Data Science Bootcamp near me.

Data Science 98

Data Science Amazon Web Services Finance Big Data 98

Data Engineering Weekly

FEBRUARY 4, 2024

It opens some old memory; try to solve this problem first with Presto-Kafka connector and then using OLAP engines like Druid & Apache Pinot. It is noteworthy to see the reasoning of Databricks over Snowflake with the data science workload.

Data Engineering 62

Data Engineering Data Engineer Engineering PostgreSQL 62

Data Engineering Podcast

MAY 1, 2022

With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. Go to dataengineeringpodcast.com/linode today and get a $100 credit to try out a Kubernetes cluster of your own.

Data Warehouse 100

Data Warehouse Data Lake Architecture Data 100

Christophe Blefari

JANUARY 20, 2024

Convert your PySpark code to Snowpark code using SnowConvert — Snowflake trying to attract Databricks customers. Integrating Airbyte with data orchestrators: Airflow, Dagster and Prefect — A orchestrators comparison and how Airbyte can be used as extract-and-load within them.

Data 130

Data Google Cloud Python Algorithm 130

Data Engineering Podcast

NOVEMBER 6, 2022

Ascend automates workloads on Snowflake, Databricks, BigQuery, and open source Spark, and can be deployed in AWS, Azure, or GCP. Ascend automates workloads on Snowflake, Databricks, BigQuery, and open source Spark, and can be deployed in AWS, Azure, or GCP. Go to dataengineeringpodcast.com/ascend and sign up for a free trial.

MongoDB 130

MongoDB Scala MySQL Data Lake 130

Monte Carlo

JANUARY 5, 2024

Data consumers downstream, like data analysts and data scientists, can try new analytical approaches and run their own reports without needing to move or copy data – reducing the workload for data engineers. Image courtesy of Databricks.

Architecture 52

Architecture Data Lake Metadata Unstructured Data 52

Monte Carlo

JANUARY 5, 2024

Data consumers downstream, like data analysts and data scientists, can try new analytical approaches and run their own reports without needing to move or copy data – reducing the workload for data engineers. Image courtesy of Databricks.

Architecture 52

Architecture Data Lake Metadata Unstructured Data 52

Data Engineering Podcast

AUGUST 20, 2021

With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. Go to dataengineeringpodcast.com/linode today and get a $100 credit to try out a Kubernetes cluster of your own.

Building 100

Building Data Lake Data Warehouse SQL 100

Data Engineering Podcast

OCTOBER 2, 2022

Sign up for a free-forever plan at dataengineeringpodcast.com/metaplane , or try out their most advanced features with a 14-day free trial. Ascend automates workloads on Snowflake, Databricks, BigQuery, and open source Spark, and can be deployed in AWS, Azure, or GCP. What is the core problem that you are trying to solve at Alvin?

IT 100

IT Food PostgreSQL MongoDB 100

Data Engineering Podcast

JUNE 12, 2022

Ascend automates workloads on Snowflake, Databricks, BigQuery, and open source Spark, and can be deployed in AWS, Azure, or GCP. Ascend automates workloads on Snowflake, Databricks, BigQuery, and open source Spark, and can be deployed in AWS, Azure, or GCP. Go to dataengineeringpodcast.com/ascend and sign up for a free trial.

Unstructured Data 130

Unstructured Data MongoDB Scala MySQL 130

Rockset

JANUARY 31, 2023

Real-Time Data Stack with Databricks and Rockset We needed a storage layer capable of large-scale ML processing for terabytes of new data per day. We compared Snowflake and Databricks, choosing the latter because of Databrick’s compatibility with more tooling options and support for open data formats.

MongoDB 59

MongoDB Pharmaceutical Recruitment Raw Data 59

Data Engineering Podcast

OCTOBER 23, 2022

Sign up for a free-forever plan at dataengineeringpodcast.com/metaplane , or try out their most advanced features with a 14-day free trial. Ascend automates workloads on Snowflake, Databricks, BigQuery, and open source Spark, and can be deployed in AWS, Azure, or GCP. Setup takes 30 minutes. Setup takes 30 minutes.

Database 100

Database MySQL Cloud MongoDB 100

Data Engineering Podcast

OCTOBER 16, 2022

Sign up for a free-forever plan at dataengineeringpodcast.com/metaplane , or try out their most advanced features with a 14-day free trial. Ascend automates workloads on Snowflake, Databricks, BigQuery, and open source Spark, and can be deployed in AWS, Azure, or GCP. Setup takes 30 minutes. Setup takes 30 minutes.

Data Lake 100

Data Lake Food MongoDB Scala 100

Data Engineering Podcast

SEPTEMBER 25, 2022

Sign up for a free-forever plan at dataengineeringpodcast.com/metaplane , or try out their most advanced features with a 14-day free trial. Ascend automates workloads on Snowflake, Databricks, BigQuery, and open source Spark, and can be deployed in AWS, Azure, or GCP. Setup takes 30 minutes. Setup takes 30 minutes.

Food 100

Food MongoDB Scala MySQL 100

Data Engineering Podcast

SEPTEMBER 18, 2022

Ascend automates workloads on Snowflake, Databricks, BigQuery, and open source Spark, and can be deployed in AWS, Azure, or GCP. Sign up for a free-forever plan at dataengineeringpodcast.com/metaplane , or try out their most advanced features with a 14-day free trial. Setup takes 30 minutes. Setup takes 30 minutes.

Hospitality 100

Hospitality Food MongoDB Scala 100

ProjectPro

FEBRUARY 21, 2023

Cloudera or Databricks? Proficient in building data processing solutions using Azure Data Factory , Azure Synapse Analytics , Azure Data Lake Storage, Azure Databricks , etc. Also, you can try out the free Practice Question Sets to have a better understanding of the exam format. AWS or Azure?

Certification 52

Certification Data Engineering Data Engineer Engineering 52

Data Engineering Podcast

SEPTEMBER 11, 2022

Ascend automates workloads on Snowflake, Databricks, BigQuery, and open source Spark, and can be deployed in AWS, Azure, or GCP. What is the core problem that you are trying to solve with the Hevo platform? Ascend automates workloads on Snowflake, Databricks, BigQuery, and open source Spark, and can be deployed in AWS, Azure, or GCP.

Data Pipeline 130

Data Pipeline Building MongoDB Scala 130

Data Engineering Podcast

JULY 3, 2022

Data Engineering Podcast listeners can sign up for a free 2-week sandbox account, go to dataengineeringpodcast.com/tonic today to give it a try! Ascend automates workloads on Snowflake, Databricks, BigQuery, and open source Spark, and can be deployed in AWS, Azure, or GCP. Data teams are increasingly under pressure to deliver.

Data Integration 100

Data Integration MongoDB Scala MySQL 100

Data Engineering Podcast

OCTOBER 9, 2022

Sign up for a free-forever plan at dataengineeringpodcast.com/metaplane , or try out their most advanced features with a 14-day free trial. Ascend automates workloads on Snowflake, Databricks, BigQuery, and open source Spark, and can be deployed in AWS, Azure, or GCP. Setup takes 30 minutes. Setup takes 30 minutes.

Food 100

Food MongoDB Scala MySQL 100

Ascend.io

MARCH 1, 2023

Additional Reading and Resources Improved Ascend for Databricks, New Lineage Visualization, and Better Incremental Data Ingestion New Data Plane Usage Report, plus Databricks Merge Link Multiple Data Clouds to Ascend The post Ascend Spotlight: Stop Data Problems in Real-Time appeared first on Ascend.io.

Data Pipeline 52

Data Pipeline Data Ingestion Data Data Integration 52

Ascend.io

FEBRUARY 20, 2023

Additional Reading and Resources Improved Ascend for Databricks, New Lineage Visualization, and Better Incremental Data Ingestion New Data Plane Usage Report, plus Databricks Merge Link Multiple Data Clouds to Ascend The post Ascend Spotlight: Stop Data Problems in Real-Time appeared first on Ascend.io.

Data Pipeline 52

Data Pipeline Data Ingestion Data Data Integration 52

Ascend.io

FEBRUARY 20, 2023

Additional Reading and Resources Improved Ascend for Databricks, New Lineage Visualization, and Better Incremental Data Ingestion New Data Plane Usage Report, plus Databricks Merge Link Multiple Data Clouds to Ascend The post Ascend Spotlight: Stop Data Problems in Real-Time appeared first on Ascend.io.

Data Pipeline 52

Data Pipeline Data Ingestion Data Data Integration 52