Ascend.io

JUNE 22, 2023

Data-driven organizations are increasingly looking for ways to enable both centralized and distributed teams to build, share and collaborate on analytical data products. Get started with a free developer-tier Ascend Cloud environment and begin loading your data into MotherDuck today ( docs )!

NoSQL 52

NoSQL Google Cloud MySQL Data Pipeline 52

Data Engineering Weekly

MARCH 19, 2023

Now you can win $1,000 cash by contributing a Transformation to our open-source library. Try it free Netflix: Building a Media Understanding Platform for ML Innovations Netflix recently wrote a series of blogs about its media ML platform. Build guardrails into your prompt. link] Sanjeev Mohan: What Exactly is a Data Product?

Data Engineering 57

Data Engineering Data Engineer Engineering Media 57

Cloudera

JANUARY 31, 2024

It forms a foundational element for building robust and reliable distributed architectures. When SRM replicates a topic, it renames the topic in the target cluster by prefixing the name of the topic with the alias (name) of the source cluster. The process lists all of the topic names of a cluster, then detects the source cluster name.

Management 71

Management Kafka Big Data Cloud 71

dbt Developer Hub

FEBRUARY 7, 2022

So, I thought it only apt to build on the collective reverence for these tasty, crunchy snacks to talk about customer 360 views. To do so, they want to build out their sales motion to target companies with active gaggles. Below, I’ll break down the DAG at each step of our process so you can see how it builds all together.

Data Warehouse 52

Data Warehouse Datasets Data SQL 52

dbt Developer Hub

MAY 16, 2022

They will access the data via: Precomputed views/tables in a BI tool Read-only access to the dbt Cloud IDE docs Full list of tables and views in their data warehouse Precomputed views/tables in a BI tool ​ Here we have drag and drop functionality and a skin over top of the underlying database.schema.table where the database object is stored.

BI 52

BI Data Warehouse Database SQL 52

Monte Carlo

DECEMBER 13, 2023

To answer that question, we asked data pros from a variety of industries, geographies, and company sizes who are building the infrastructure for these tools to share their unvarnished “hand on heart” opinions about the on-the-ground reality of GenAI…and they didn’t disappoint. Does it do things perfectly? Is it killing jobs?

BI 59

BI Data Metadata Technology 59

Monte Carlo

MAY 18, 2023

The first instinct of many early stage companies or budget strapped data teams is to turn to open source data lineage tools. Depending on your requirements (more on that in the next section), it is prudent to evaluate open source data lineage tools alongside the several other types of platforms that provide data lineage capabilities.

Metadata 64

Metadata Data Lake Government Data Governance 64

Netflix Tech

MARCH 5, 2020

By Kevin Glisson, Marc Vilanova, Forest Monsen Netflix is pleased to announce the open-source release of our crisis management orchestration framework: Dispatch! Building on past incidents to speed up the resolution of future incidents. Getting Started Dispatch is available now on the Netflix Open Source site.

Metadata 81

Metadata AWS Management Architecture 81

Monte Carlo

DECEMBER 13, 2023

To answer that question, we asked data pros from a variety of industries, geographies, and company sizes who are building the infrastructure for these tools to share their unvarnished “hand on heart” opinions about the on-the-ground reality of GenAI…and they didn’t disappoint. Does it do things perfectly? Is it killing jobs?

BI 52

BI Data Metadata Technology 52

Cloudera

OCTOBER 28, 2020

Please note that grabbing logs from Solr is not the only solution, to completely reduce the amount of logs a cluster will hold, MiNiFi agents or Apache NiFi can grab the logs directly from their source. These foundational steps should get you started in building your own pipeline to help enrich the value of your log data.

Kafka 66

Kafka SQL Professional Services Consulting 66

Data Engineering Podcast

JANUARY 26, 2020

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Data Pipeline 100

Data Pipeline PostgreSQL Media Data Validation 100

DataKitchen

JANUARY 18, 2023

By automating your tests, then running them with each refresh, you build in safety valves for your data pipeline. . Data Engineers take similar risks when they over-complicate their code or tack on extraneous processes at the wrong points in a critical path build. Don’t over-tighten your fittings. .

Data Pipeline 52

Data Pipeline Data AWS Data Engineering 52

Rockset

SEPTEMBER 25, 2020

What you’ll notice is there is a clause that states while cursor.alive: This while statement allows your code to keep checking to see if your cursor is still alive and doc references the different documents that captured the change in the oplog. first = oplog.find().sort('$natural', sort('$natural', pymongo.DESCENDING).limit(-1).next()

MongoDB 52

MongoDB Kafka Database NoSQL 52

Monte Carlo

SEPTEMBER 6, 2022

Check out the docs to learn more. Build rule-based monitors that are easily configured and deployed on your organization’s most critical fields and tables for more refined anomaly detection. With automated lineage, data teams can determine if an upstream source table generated anomalous data. EDW-level (e.g.,

Data Lake 52

Data Lake Machine Learning Data Science Data 52

dbt Developer Hub

MARCH 9, 2022

Special Thanks: Emilie Schario, Matt Winkler dbt has done a great job of building an elegant, common interface between data engineers, analytics engineers, and any data-y role, by uniting our work on SQL. Should we increase the SQL surface area to build ML models? It wasn’t intuitive, but it was better than leaving it out of dbt docs.

Machine Learning 52

Machine Learning Raw Data SQL Python 52

Grouparoo

OCTOBER 26, 2020

That isn’t to say writing docs is the best approach for all instances, but it’s a solid default stance to take. Default to open When we decided to build an open-source product, we found ourselves asking over and over again “should this feature be open-source/free or paid?” Most recently we played some Settlers of Catan!

Programming 52

Programming Project Building Process 52

dbt Developer Hub

JULY 2, 2023

Have you experienced a data warehousing project with 50+ data sources, with 25+ data developers working on the same data platform, or data spanning 5+ years with two or more generations of source systems? Just find another water source and fill up the reservoir. So why Data Vault? A new development in the city? No problem!

Cloud 52

Cloud Data Warehouse Data BI 52

Data Engineering Weekly

MARCH 26, 2023

Now you can win $1,000 cash by contributing a Transformation to our open-source library. Redpoint ventures write about the top 25 open-source companies ranked by adoption, momentum, usage, and health. The BTYD model is excellent for building a recommendation engine and marketing personalization.

Data Engineering 52

Data Engineering Data Engineer Engineering Lambda Architecture 52

dbt Developer Hub

JANUARY 23, 2023

With roughly 1,000 dbt models, nearly a hundred data sources, and a dozen different data quality stakeholders, producing a framework that allows stakeholders to take action on test failures is challenging. Building views on top of the base table to split tests by owner or severity, and creating visualizations using our tool of choice.

Metadata 52

Metadata High Quality Data SQL Data Integration 52

Rockset

NOVEMBER 4, 2021

Apache Spark is an open-source project that was started at UC Berkeley AMPLab. Here are some examples of the things you can do in your apps with Apache Spark: Build continuous ETL pipelines for stream processing SQL BI and analytics Do machine learning, and much more! How do you build a data application with Apache Spark and Rockset?

Scala 52

Scala Java AWS Hadoop 52

Grouparoo

MAY 11, 2021

Let's build him a pipeline to get data from Google Sheets to flow automatically into Mailchimp. We just insert it between our data source and destination (Sheets and Mailchimp) and then it just works. Note that the open-source version of Grouparoo needs to run on a server. Why don't we help Dylan out a bit?

Data Pipeline 52

Data Pipeline Data Project Utilities 52

Picnic Engineering

FEBRUARY 1, 2023

Without a smooth build and test setup teams lose productivity and everything grinds to a halt. While doing that we standardized our pipelines, reduced the build/queue time, and took control of our build agent hardware and environment. to build our largest repo. Turns out, that was not the case. Quite impressive right?

AWS 64

AWS Cloud Project Building 64

Rockset

OCTOBER 21, 2022

It’s part of building a stellar customer experience, where customers want to keep coming back, and you start building customer champions. Customer experience is part of the journey of building loyal customers. When building a real-time customer 360 app, you’ll definitely need event data from a streaming data source, like Kafka.

Kafka 52

Kafka Building SQL Database 52

Data Engineering Weekly

OCTOBER 30, 2022

The author narrates the data contract, why it is the foundation for building data mesh, and how it glue data catalogs, glossary and data quality systems. LinkedIn writes about how it builds “Super Tables” [ Is that another name for the semantic layer? ] The current state of these systems is inherently passive systems.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Ingestion 52

dbt Developer Hub

JANUARY 14, 2023

The only data source we have is Idealista, which is a portal where real estate agencies post ads. A transformation layer taking the source data loaded by my dlt functions and creating the tables necessary, handled by dbt. file for the source). This is handled by dlt functions written in Python. dbt ​ No surprises there.

Data Ingestion 52

Data Ingestion Google Cloud Python Cloud 52

Ripple Engineering

FEBRUARY 2, 2021

The Ripple Data Engineering team is expanding, which means higher frequency changes to our data pipeline source code. This means we need to build better, more configurable and more collaborative tooling that prevents code collisions and enforces software engineering best practices. py}" -.gitlab-ci.yml

Building 52

Building Python Google Cloud Coding 52

Rockset

JANUARY 23, 2019

Redash is a powerful open source query and visualization tool that helps you make sense of your data. It connects to variety of data sources and also includes a native connector for Rockset. In this post we will demonstrate how to use Redash to build live dashboards on Rockset data sets.

SQL 40

SQL Building IT Data 40

Grouparoo

DECEMBER 13, 2021

This year we've watched dbt take off as the open source data transformation tool of choice and were excited for their 1.0 In this file, we used Jinja and SQL to build out a dbt_meta table. To learn more about the different ways Grouparoo can be run, check out our Orchestration Docs release this month !

Metadata 52

Metadata Data Warehouse SQL Utilities 52

Zalando Engineering

NOVEMBER 19, 2023

And even if you were to gather all the information from the docs, it's still not enough. Origami builds on top of Elasticsearch and our internal/Zalando-specific suite of APIs. We use an open-sourced solution called Skipper as an ingress controller, which gives us access to filters. The license is Apache License 2.0.

Scala 86

Scala Java Coding IT 86

dbt Developer Hub

AUGUST 2, 2023

For Postgres/Redshift/Databricks {{ config ( materialized = 'materialized_view' , ) }} For Snowflake: {{ config ( materialized = 'dynamic_table' , ) }} note We are only supporting dynamic tables on Snowflake, not Snowflake’s materialized views (for a comparison between Snowflake Dynamic Tables and Materialized Views, refer docs.

Datasets 52

Datasets Kafka SQL Cloud 52

Cloudera

SEPTEMBER 28, 2020

Cloudera delivers an enterprise data cloud that enables companies to build end-to-end data pipelines for hybrid cloud, spanning edge devices to public or private cloud, with integrated security and governance underpinning it to protect customers data. Data Science and machine learning workloads using CDSW.

Cloud 131

Cloud Kafka Professional Services Metadata 131

Snowflake

JULY 10, 2023

Data engineers and data scientists can take advantage of Snowflake’s fast engine with secure access to open source libraries for processing images, video, audio, and more. Using Snowpark runtimes and libraries, you can securely deploy and process Python, Java and Scala code to build pipelines, ML models, and applications in Snowflake.

Unstructured Data 40

Unstructured Data Python Process Scala 40

Rockset

MAY 27, 2021

Low Ops When you build a cloud-native system, you can enable serverless cloud scaling and the vectors we're optimizing for are both hardware efficiency and human efficiency in the cloud. Elasticsearch has an inverted index and it also has doc value storage built using Apache Lucene. That's one of the design goals.

MongoDB 40

MongoDB Kafka Cloud SQL 40

dbt Developer Hub

NOVEMBER 28, 2021

So I challenge you after reading this article to test out your skillsets, think about the code that you find yourself reusing again and again, and build a package. By using dbt, you are inherently part of our open source community. For more information on developing your package, check out our docs site.

Building 52

Building SQL Coding Datasets 52

Workfall

FEBRUARY 28, 2023

Source – OpenAI How Does ChatGPT Function? Source – Statista (Leading search engines’ desktop market shares globally from January 2015 to December 2022). This compatibility enables businesses to build on existing solutions to create unique AI-powered chatbots that can understand and respond to user needs in real-time.

Datasets 62

Datasets Coding Algorithm Technology 62

dbt Developer Hub

APRIL 30, 2019

As the maintainers of dbt, and analytics consultants, at Fishtown Analytics (now dbt Labs) we build a lot of dbt projects. Instead, use this as a guide once you’ve already got an idea of what you’re building for how you should break the transformations up into separate dbt models. There are entire books on this topic.

Project 40

Project Database-centric Raw Data Data Warehouse 40

Rockset

OCTOBER 15, 2021

Figure 1: Different levels of abstraction in cloud computing (Source: ComputerWeekly.com ) We can apply the principles behind this cloud computing taxonomy to cloud databases as well. We all use SaaS applications, sometimes dozens of them in a given day—ranging from Google Docs to Salesforce CRM.

Database 52

Database Cloud Cloud Computing MongoDB 52