Data Engineering Podcast

AUGUST 3, 2021

By adding support for small, incremental inserts into large table structures, and building support for arbitrary update and delete operations the Hudi project brings the best of both worlds together. Go to dataengineeringpodcast.com/97things today to get your copy! And don’t forget to thank them for their continued support of this show!

Data Lake 130

Data Lake Data Warehouse Hadoop Architecture 130

dbt Developer Hub

FEBRUARY 7, 2022

You can follow along with Donny's data modeling technique for identity resolution in this dbt project repo. So, I thought it only apt to build on the collective reverence for these tasty, crunchy snacks to talk about customer 360 views. To do so, they want to build out their sales motion to target companies with active gaggles.

Data Warehouse 52

Data Warehouse Datasets Data SQL 52

dbt Developer Hub

MAY 16, 2022

Analytics engineers (AEs) are constantly navigating through the names of the models in their project, so naming is important for maintainability in your project in the way you access it and work within it. By default, dbt will use your model file name as the view or table name in the database.

BI 52

BI Data Warehouse Database SQL 52

Monte Carlo

SEPTEMBER 27, 2023

When it comes to building data and AI products you can trust, Monte Carlo, the leader of the data observability category, and Alation, the data intelligence leader, are better together. Can I trust the data feeding this model? Learn more by visiting the Monte Carlo docs site. What’s wrong with this metric?” “Why

Data Governance 69

Data Governance Government Data Pipeline Data 69

Edureka

APRIL 19, 2024

Midjourney is a generative AI model that combines art and technology, blended with human creativity, to generate stunning art images from text-based prompts. On July 12, 2022, it entered open beta, and since then, it’s been upgrading its algorithms and launching new model versions almost every few months. What Is Midjourney?

Algorithm 52

Algorithm Deep Learning Machine Learning Media 52

Data Engineering Weekly

OCTOBER 1, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. See how it works today. Dropbox: Is this a date?

Data Engineering 52

Data Engineering Data Engineer Engineering Data Pipeline 52

Knowledge Hut

MARCH 19, 2024

Understanding cloud types (private, public, community, and hybrid) and service models (SaaS, PaaS, IaaS) is crucial. This model is efficient when multiple entities want to benefit from a common cloud setup while retaining a degree of exclusivity. Think Google Docs or Microsoft 365 – they're SaaS because we use them online.

Cloud Computing 52

Cloud Computing Cloud Accessible Accessibility 52

dbt Developer Hub

JANUARY 23, 2023

At Tempus , a precision medicine company specializing in oncology, high quality data is a necessary component for high quality clinical models. Building views on top of the base table to split tests by owner or severity, and creating visualizations using our tool of choice. FROM metadata m LEFT JOIN failures f on m. test_alias = f.

Metadata 52

Metadata High Quality Data SQL Data Integration 52

Monte Carlo

DECEMBER 13, 2023

To answer that question, we asked data pros from a variety of industries, geographies, and company sizes who are building the infrastructure for these tools to share their unvarnished “hand on heart” opinions about the on-the-ground reality of GenAI…and they didn’t disappoint. Does it do things perfectly? Is it killing jobs?

BI 59

BI Metadata Data Technology 59

dbt Developer Hub

APRIL 30, 2023

I wanted to build prestige in my career. dbt brought code-based tests, integrated CI, efficient development with packages, and global docs. These features have been foundational to the way data teams work, and have allowed data engineers to focus on the most important part of their job: building data pipelines that power the business.

Data Engineering 64

Data Engineering Data Engineer Engineering Finance 64

Data Engineering Podcast

MAY 19, 2019

Summary In recent years the traditional approach to building data warehouses has shifted from transforming records before loading, to transforming them afterwards. The data build tool (dbt) is designed to bring battle tested engineering practices to your analytics pipelines. Can you talk through how the packaging system is implemented?

Data Analytics 100

Data Analytics Building Engineering Data Warehouse 100

dbt Developer Hub

MARCH 9, 2022

Special Thanks: Emilie Schario, Matt Winkler dbt has done a great job of building an elegant, common interface between data engineers, analytics engineers, and any data-y role, by uniting our work on SQL. Should we increase the SQL surface area to build ML models? I like to call this interoperability a “baton pass.”

Machine Learning 52

Machine Learning Raw Data SQL Python 52

Monte Carlo

DECEMBER 13, 2023

To answer that question, we asked data pros from a variety of industries, geographies, and company sizes who are building the infrastructure for these tools to share their unvarnished “hand on heart” opinions about the on-the-ground reality of GenAI…and they didn’t disappoint. Does it do things perfectly? Is it killing jobs?

BI 52

BI Metadata Data Technology 52

Data Engineering Weekly

OCTOBER 30, 2022

The author narrates the data contract, why it is the foundation for building data mesh, and how it glue data catalogs, glossary and data quality systems. link] Iris Sun: Anything and Everything about Generative Models The generative AI is on everyone’s lips. The current state of these systems is inherently passive systems.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Ingestion 52

Monte Carlo

MAY 18, 2023

We’ve written extensively on how to analyze the build vs buy decision across the modern data stack, but it’s important to speak specifically about open source data lineage options. Our team attempted to leverage Spline (one of the open source data lineage tools listed here) as a building block. Ok, that is enough caveat emptor.

Metadata 64

Metadata Data Lake Government Data Governance 64

Monte Carlo

MAY 17, 2023

Because GitHub monitoring requires information about the mapping between customers’ tables to dbt models and their file locations, a dbt integration is currently required to utilize our GitHub integration. Once you’ve got your dbt integration set up, installing your GitHub app for Monte Carlo is just a few clicks away. .”

Data Pipeline 52

Data Pipeline Coding Data Engineering Data Engineer 52

Monte Carlo

MAY 17, 2023

Because GitHub monitoring requires information about the mapping between customers’ tables to dbt models and their file locations, a dbt integration is currently required to utilize our GitHub integration. Once you’ve got your dbt integration set up, installing your GitHub app for Monte Carlo is just a few clicks away. .”

Data Pipeline 52

Data Pipeline Data Engineering Data Engineer Data 52

Data Engineering Weekly

MARCH 26, 2023

” [link] Rittman Analytics: ChatGPT, Large Language Models and the Future of dbt and Analytics Consulting Very fascinating to read about the potential impact of LLM in the future of dbt and analytical consulting. The BTYD model is excellent for building a recommendation engine and marketing personalization.

Data Engineering 52

Data Engineering Data Engineer Engineering Lambda Architecture 52

dbt Developer Hub

AUGUST 2, 2023

A enterprise customer I was working with, Jetblue, asked me for help running their dbt models every 2 minutes to meet a 5 minute SLA. Just like you would materialize your sql model as table or view today, you can use materialized_view in your model configuration, dbt_project.yml, and resources.yml files. In production? (If

Datasets 52

Datasets Kafka SQL Cloud 52

Monte Carlo

SEPTEMBER 6, 2022

Check out the docs to learn more. Build rule-based monitors that are easily configured and deployed on your organization’s most critical fields and tables for more refined anomaly detection. Take a look at the dbt (Cloud and Core) and Airflow integration docs to learn more. Learn how in the docs.

Data Lake 52

Data Lake Machine Learning Data Science Data 52

Rockset

SEPTEMBER 25, 2020

What you’ll notice is there is a clause that states while cursor.alive: This while statement allows your code to keep checking to see if your cursor is still alive and doc references the different documents that captured the change in the oplog. first = oplog.find().sort('$natural', sort('$natural', pymongo.DESCENDING).limit(-1).next()

MongoDB 52

MongoDB Kafka Database NoSQL 52

dbt Developer Hub

JULY 2, 2023

is a data modeling technique designed to help scale large data warehousing projects. If not, it might be hard to initially understand the benefits of Data Vault, and maybe Kimball modelling is better for you. They allow for more flexibility and extensibility and can be used to model complex processes in an agile way.

Cloud 52

Cloud Data Warehouse Data BI 52

dbt Developer Hub

JANUARY 14, 2023

dlt handles the data cleaning and normalisation automatically so I can focus on curating and modelling it in dbt. Using both a verified source in the gsheets connector and building my own from Rapid API endpoints seemed equally intuitive. package ( pipeline , "dbt/property_analytics" , venv = venv ) # and run it models = dbt.

Data Ingestion 52

Data Ingestion Google Cloud Python Cloud 52

Snowflake

JULY 10, 2023

Customers are looking for ways to better utilize unstructured data, which could be in any file format such as doc, pdf, images (jpeg, png), audio (mp3), video, e-mails, or industry-specific formats such as raster data for geospatial,las files for well log data, DICOM for medical, and many more.

Unstructured Data 40

Unstructured Data Python Process Scala 40

Picnic Engineering

FEBRUARY 1, 2023

Without a smooth build and test setup teams lose productivity and everything grinds to a halt. While doing that we standardized our pipelines, reduced the build/queue time, and took control of our build agent hardware and environment. to build our largest repo. Turns out, that was not the case. Quite impressive right?

AWS 64

AWS Cloud Project Building 64

Rockset

JUNE 13, 2019

Why build a new SQL development environment? And we built Rockset around two core principles: strong dynamic typing and the document object model. Document object model / Smart schemas (TLDR: Rockset ‘schemas' resemble more JSON objects than field lists): Fields can be nested within other fields or even within arrays.

SQL 52

SQL Structured Data Building Raw Data 52

Workfall

FEBRUARY 28, 2023

The language model has swept the internet since its initial release. OpenAI created ChatGPT, a large language model (LLM). LLM is a natural language processing-focused machine learning model (NLP). a version of GPT-3, serves as the language model for ChatGPT – which was trained on Azure AI supercomputing infrastructure.

Datasets 62

Datasets Coding Algorithm Technology 62

Knowledge Hut

MAY 3, 2023

Tools/Software Other equipment/data models, etc. Template 1 (PDF): resource-management-plan.pdf Template 2 (Doc): resource-management-plan.docx Benefits of Creating Resource Management Plan Offers Safety: In real life, we all operate under numerous constraints and Project Success could be difficult under certain circumstances.

Management 52

Management IT Certification Project 52

Cloudera

SEPTEMBER 28, 2020

Cloudera delivers an enterprise data cloud that enables companies to build end-to-end data pipelines for hybrid cloud, spanning edge devices to public or private cloud, with integrated security and governance underpinning it to protect customers data. The customer leverages Cloudera’s multi-function analytics stack in CDP.

Cloud 131

Cloud Kafka Professional Services Metadata 131

Rockset

OCTOBER 15, 2021

SaaS: Low Configuration, Immediate Productivity SaaS is easily the most familiar software delivery model for cloud applications: just log in and start working. We all use SaaS applications, sometimes dozens of them in a given day—ranging from Google Docs to Salesforce CRM.

Database 52

Database Cloud Cloud Computing MongoDB 52

dbt Developer Hub

NOVEMBER 28, 2021

So I challenge you after reading this article to test out your skillsets, think about the code that you find yourself reusing again and again, and build a package. It could contain models that help you model your SaaS dataset in a manner of minutes (I’m looking at you, Fivetran Salesforce package). Add in your package contents.

Building 52

Building SQL Coding Datasets 52

dbt Developer Hub

APRIL 30, 2019

As the maintainers of dbt, and analytics consultants, at Fishtown Analytics (now dbt Labs) we build a lot of dbt projects. Instead, use this as a guide once you’ve already got an idea of what you’re building for how you should break the transformations up into separate dbt models. Still confused? An example might help!

Project 40

Project Database-centric Raw Data Data Warehouse 40

dbt Developer Hub

FEBRUARY 12, 2024

dbt Explorer: an upgrade to data discovery ​ In the days of yore, answering a question about your data platform may have required a bit of cryptography, sifting through possibly-up-to-date documentation in your internal wiki, run logs to figure out when your models were executed, and slacking the data team member with the most tenure.

Metadata 59

Metadata Raw Data BI Project 59

Christophe Blefari

OCTOBER 19, 2023

In this core model you have an orders model. On the other side the finance data team wants to build a revenue model on top of the core.orders model. With cross-project references you can declare a model to be public on core to use it elsewhere. Or even more, versioning models. See the doc.

Project 264

Project Finance SQL Government 264

Knowledge Hut

JUNE 19, 2023

They can use them to build the classrooms of tomorrow that are tech-savvy and capable of preparing children for the ever-increasing competition. Google Workspace includes docs, slides, forms, lenses, and drawings that help students and teachers take learning to another level.

Education 52

Education Technology Coding Computer Science 52

Tweag

NOVEMBER 1, 2023

For example, we build Nickel with Nix, and we need to make sure the version of wasm-bindgen we pull from Nixpkgs is the same as the one set in Cargo.toml. The actual Nix code to build Nickel calls to a mysterious override function, which is a bespoke mechanism of Nixpkgs to simulate this kind of overriding. s t r i n g. s t r i n g.

Metadata 105

Metadata Programming Programming Language Coding 105

Christophe Blefari

MAY 29, 2023

Gen Ai 🤖 QLoRA: Efficient Finetuning of Quantized LLMs — 65B parameter model on a single 48GB GPU reaching 99.3% Data Modeling, architecture Pattern, tools and the future — part 3 of Simon's guide. Writing design docs for data pipelines. See you on Friday. of the performance level of ChatGPT on Vicuna.

BI 130

BI Data Warehouse Data Data Pipeline 130