Towards Data Science

JUNE 6, 2023

It runs locally, has extensive SQL support and can run queries directly on Pandas data, Parquet, JSON data. The fact it’s insanely fast and does (mostly) all processing in memory make it a good choice for building my personal data warehouse. Extra points for its seamless integration with Python and R.

Data Warehouse 93

Data Warehouse Data SQL Data Analysis 93

Snowflake

NOVEMBER 2, 2023

Over the years, the technology landscape for data management has given rise to various architecture patterns, each thoughtfully designed to cater to specific use cases and requirements. These patterns include both centralized storage patterns like data warehouse , data lake and data lakehouse , and distributed patterns such as data mesh.

Data Lake 84

Data Lake Data Warehouse Cloud Unstructured Data 84

Data Engineering Podcast

MAY 13, 2021

Summary There is a lot of attention on the database market and cloud data warehouses. While they provide a measure of convenience, they also require you to sacrifice a certain amount of control over your data. Firebolt is the fastest cloud data warehouse. Visit dataengineeringpodcast.com/firebolt to get started.

PostgreSQL 100

PostgreSQL Data Warehouse Building MySQL 100

databricks

AUGUST 10, 2023

At this year's Data+AI Summit, Databricks SQL continued to push the boundaries of what a data warehouse can be, leveraging AI across the.

SQL 89

SQL Data Warehouse Data 89

Data Engineering Podcast

FEBRUARY 8, 2021

In this episode Zeeshan Qureshi and Michelle Ark share their experiences using DBT to manage the data warehouse for Shopify. Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows. What kinds of data sources are you working with?

Data Warehouse 100

Data Warehouse Building BI SQL 100

Data Science Blog: Data Engineering

SEPTEMBER 19, 2023

In the contemporary age of Big Data, Data Warehouse Systems and Data Science Analytics Infrastructures have become an essential component for organizations to store, analyze, and make data-driven decisions. So why using IaC for Cloud Data Infrastructures?

Data Warehouse 52

Data Warehouse Coding Systems Cloud 52

databricks

MARCH 6, 2024

This blog continues our series looking at advancements from 2023 to the serverless data warehouse Databricks SQL. The best data warehouse is.

SQL 52

SQL Data Warehouse Data 52

Data Engineering Podcast

JANUARY 1, 2022

In this episode Emily Riederer shares her work to create a controlled vocabulary for managing the semantic elements of the data managed by her team and encoding it in the schema definitions in her data warehouse. Atlan is a collaborative workspace for data-driven teams, like Github for engineering or Figma for design teams.

Data Warehouse 100

Data Warehouse BI Data Workflow Coding 100

Start Data Engineering

JANUARY 16, 2021

Introduction Setup Code Conditional logic to read from mock input Custom macro to test for equality Setup environment specific test Run ELT using dbt Conclusion Further reading Introduction With the recent advancements in data warehouses and tools like dbt most transformations(T of ELT) are being done directly in the data warehouse.

SQL 130

SQL Data Warehouse Coding IT 130

Cloudera

SEPTEMBER 24, 2020

Some of the most powerful results come from combining complementary superpowers, and the “dynamic duo” of Apache Hive LLAP and Apache Impala, both included in Cloudera Data Warehouse , is further evidence of this. Both Impala and Hive can operate at an unprecedented and massive scale, with many petabytes of data.

Data Warehouse 68

Data Warehouse SQL Engineering Metadata 68

Cloudera

JANUARY 19, 2024

As described in our recent blog post , an SQL AI Assistant has been integrated into Hue with the capability to leverage the power of large language models (LLMs) for a number of SQL tasks. This is a real game-changer for data analysts on all levels and will make SQL development faster, easier, and less error-prone.

SQL 71

SQL Data Warehouse AWS Architecture 71

Monte Carlo

AUGUST 25, 2023

Different vendors offering data warehouses, data lakes, and now data lakehouses all offer their own distinct advantages and disadvantages for data teams to consider. So let’s get to the bottom of the big question: what kind of data storage layer will provide the strongest foundation for your data platform?

Data Lake 52

Data Lake Data Warehouse Unstructured Data Raw Data 52

Data Engineering Podcast

MAY 1, 2022

Atlan is a collaborative workspace for data-driven teams, like Github for engineering or Figma for design teams. Datafold built automated regression testing to help data and analytics engineers deal with data quality in their pull requests. No more shipping and praying, you can now know exactly what will change in your database!

Data Warehouse 100

Data Warehouse Data Integration Cloud Google Cloud 100

Data Engineering Podcast

OCTOBER 14, 2019

Summary Managing a data warehouse can be challenging, especially when trying to maintain a common set of patterns. Raghu Murthy, founder and CEO of Datacoral built data infrastructures at Yahoo! and Facebook, scaling from mere terabytes to petabytes of analytic data. Visit Datacoral.com today to find out more.

Data Warehouse 100

Data Warehouse PostgreSQL AWS Programming Language 100

Monte Carlo

FEBRUARY 6, 2023

So, you’re planning a cloud data warehouse migration. But be warned, a warehouse migration isn’t for the faint of heart. As you probably already know if you’re reading this, a data warehouse migration is the process of moving data from one warehouse to another. A worthy quest to be sure.

Data Warehouse 52

Data Warehouse AWS Data Validation Data 52

Cloudera

JANUARY 15, 2021

Cloud data warehouses allow users to run analytic workloads with greater agility, better isolation and scale, and lower administrative overhead than ever before. The results demonstrate superior price performance of Cloudera Data Warehouse on the full set of 99 queries from the TPC-DS benchmark. Introduction.

Data Warehouse 90

Data Warehouse Cloud Consulting SQL 90

Data Engineering Podcast

DECEMBER 8, 2019

Summary Data warehouses have gone through many transformations, from standard relational databases on powerful hardware, to column oriented storage engines, to the current generation of cloud-native analytical engines. If you are evaluating your options for building or migrating a data platform, then this is definitely worth a listen.

Data Warehouse 100

Data Warehouse Cloud AWS Relational Database 100

Monte Carlo

JANUARY 25, 2023

In this article, Chad Sanderson , Head of Product, Data Platform , at Convoy and creator of Data Quality Camp , introduces a new application of data contracts: in your data warehouse. In the last couple of posts , I’ve focused on implementing data contracts in production services.

Data Warehouse 52

Data Warehouse Data High Quality Data Metadata 52

Data Engineering Podcast

AUGUST 31, 2020

Summary Data warehouse technology has been around for decades and has gone through several generational shifts in that time. The current trends in data warehousing are oriented around cloud native architectures that take advantage of dynamic scaling and the separation of compute and storage.

Data Warehouse 100

Data Warehouse Cloud Building Data Lake 100

Data Engineering Podcast

JUNE 25, 2023

You can collect, transform, and route data across your entire stack with its event streaming, ETL, and reverse ETL pipelines. You can Implement RudderStack SDKs once, then automatically send events to your warehouse and 150+ business tools, and you’ll never have to worry about API changes again.

Data Engineering 225

Data Engineering Data Engineer Python Engineering 225

Data Engineering Podcast

MAY 27, 2021

Summary The data warehouse has become the focal point of the modern data platform. With increased usage of data across businesses, and a diversity of locations and environments where data needs to be managed, the warehouse engine needs to be fast and easy to manage.

Data Warehouse 100

Data Warehouse Cloud PostgreSQL Kafka 100

Christophe Blefari

MARCH 1, 2023

dbt Core is an open-source framework that helps you organise data warehouse SQL transformation. dbt was born out of the analysis that more and more companies were switching from on-premise Hadoop data infrastructure to cloud data warehouses. This switch has been lead by modern data stack vision.

Data Warehouse 278

Data Warehouse SQL Metadata Raw Data 278

Data Engineering Podcast

OCTOBER 28, 2021

He also explains why he started Decodable to address that limitation and the work that he and his team have done to let data engineers build streaming pipelines entirely in SQL. Start trusting your data with Monte Carlo today! Hightouch is the easiest way to sync data into the platforms that your business teams rely on.

Data Pipeline 100

Data Pipeline SQL Data Warehouse Data Lake 100

Data Engineering Podcast

FEBRUARY 18, 2024

Summary A data lakehouse is intended to combine the benefits of data lakes (cost effective, scalable storage and compute) and data warehouses (user friendly SQL interface). Data lakes are notoriously complex. Data lakes are notoriously complex. Your first 30 days are free! Your first 30 days are free!

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Christophe Blefari

MARCH 15, 2024

With yato you give a folder with SQL queries and it guesses the DAG and runs the queries in the right order. BigQuery supports DELETE to delete partitions in a SQL query. Arrow doing a lot of the data operation heavy lifting. Give a lot of insights on the market. this is more common sense but always works.

Metadata 272

Metadata Datasets Data Data Warehouse 272

Data Engineering Podcast

JULY 8, 2019

Summary The market for data warehouse platforms is large and varied, with options for every use case. What are some of the advanced capabilities, such as SQL extensions, supported data types, etc. For someone getting started with Clickhouse can you describe how they should be thinking about data modeling?

Data Warehouse 100

Data Warehouse MySQL Data Lake Hadoop 100

Cloudera

DECEMBER 21, 2023

Now imagine you had a personal assistant who knew everything about your data sets and was an expert in SQL, sitting alongside you every step of the way to help you quickly problem solve, write optimized code, explain queries, and much more. Well imagine it no longer, as Cloudera’s SQL AI Assistant is exactly that!

SQL 66

SQL Data Warehouse Business Analyst Data Schemas 66

Monte Carlo

APRIL 16, 2022

The data warehouse is the foundation of the modern data stack, so it caught our attention when we saw Convoy head of data Chad Sanderson declare, “ the data warehouse is broken ” on LinkedIn. Treating data like an API. Immutable data warehouses have challenges too.

Data Warehouse 52

Data Warehouse Architecture Data Government 52

Engineering at Meta

NOVEMBER 30, 2022

UPM is our internal standalone library to perform static analysis of SQL code and enhance SQL authoring. UPM takes SQL code as input and represents it as a data structure called a semantic tree. A limiting type system: Initially, we used only the fixed set of built-in Hive data types ( string , integer , boolean , etc. )

SQL 74

SQL Data Warehouse Metadata Coding 74

Cloudera

SEPTEMBER 17, 2018

SQL development is not a new concept. However, as the data warehousing world shifts into a fast-paced, digital, and agile era, the demands to quickly generate reports and help guide data-driven decisions are constantly increasing. Cloudera recently launched Cloudera Data Warehouse, a modern data warehousing solution.

Data Warehouse 52

Data Warehouse SQL Portfolio MySQL 52

Hevo

AUGUST 20, 2023

If you’re looking to migrate your data from HubSpot to SQL Server in a secure and simple manner, you’ve come to the right place. There are many reasons why organizations might want to replicate data from their source systems to the destination of their choice.

SQL 52

SQL Data Warehouse Data Systems 52

Netflix Tech

OCTOBER 27, 2020

Usually Data scientists and engineers write Extract-Transform-Load (ETL) jobs and pipelines using big data compute technologies, like Spark or Presto , to process this data and periodically compute key information for a member or a video. The processed data is typically stored as data warehouse tables in AWS S3.

Data Warehouse 97

Data Warehouse Datasets Data Big Data 97

ProjectPro

AUGUST 11, 2021

“Data Lake vs Data Warehouse = Load First, Think Later vs Think First, Load Later” The terms data lake and data warehouse are frequently stumbled upon when it comes to storing large volumes of data. Data Warehouse Architecture What is a Data lake?

Data Lake 52

Data Lake Data Warehouse Cloud Hadoop 52

ProjectPro

FEBRUARY 16, 2023

At the heart of these data engineering skills lies SQL that helps data engineers manage and manipulate large amounts of data. Did you know SQL is the top skill listed in 73.4% of data engineer job postings on Indeed? Almost all major tech organizations use SQL. use SQL, compared to 61.7%

Data Engineering 52

Data Engineering Data Engineer SQL Engineering 52

Cloudera

FEBRUARY 9, 2021

Today’s customers have a growing need for a faster end to end data ingestion to meet the expected speed of insights and overall business demand. This ‘need for speed’ drives a rethink on building a more modern data warehouse solution, one that balances speed with platform cost management, performance, and reliability.

Data Warehouse 67

Data Warehouse Cloud Kafka Cloud Storage 67

dbt Developer Hub

JULY 12, 2022

What is a SQL function? Some common SQL functions are COALESCE , LOWER , and EXTRACT. Most, if not all, modern cloud data warehouses support some type of the DATE_TRUNC function. Below, we’ll outline some of the slight differences in the implementation between some of the data warehouses.

SQL 52

SQL IT Data Warehouse BI 52

Hevo

JANUARY 30, 2023

Organizations demand real-time data to fastrack their decision-making process based on the latest, most relevant data. Most businesses use 10+ applications and need to continuously update the data from their local data store to their data warehouses or databases like SQL Server.

SQL 52

SQL Data Warehouse Database Process 52