Cloudera

JUNE 25, 2021

The Replication Manager support matrix is documented in our public docs. Source CDH cluster versions – CM 6.3.4, Configure the required ports to enable connectivity from CDH to CDP Public Cloud (see docs for details). This step involves checking the following configs on source: . Runtime 7.2.8. Click Next.

Cloud 71

Cloud Data Lake Cloud Storage Metadata 71

dbt Developer Hub

AUGUST 2, 2022

One potential solution is to leverage the open-source package pre-commit-dbt , created by dbt community members, that can be used to automatically run tests before committing files to git or as part of CI steps. As described before, we need to run a dbt docs generate in order to create updated JSON artifacts used in the pre-commit hooks.

Python 52

Python SQL Data Warehouse Project 52

Grouparoo

MAY 11, 2021

We just insert it between our data source and destination (Sheets and Mailchimp) and then it just works. Note that the open-source version of Grouparoo needs to run on a server. Let designate the sheetsapp as a source (I'm calling this src ), and the mailchimpapp as a destination (I'm calling this dest ).

Data Pipeline 52

Data Pipeline Data Project Utilities 52

Ripple Engineering

FEBRUARY 2, 2021

The Ripple Data Engineering team is expanding, which means higher frequency changes to our data pipeline source code. For example, you could set up your repo to be triggered by incoming changes to your branch source code, which is what we wanted to do for our code pipeline. hooks: - id: seed-isort-config - repo: [link] rev: v4.3.21

Building 52

Building Python Google Cloud Coding 52

Zalando Engineering

NOVEMBER 19, 2023

And even if you were to gather all the information from the docs, it's still not enough. Of course, a lot of these files are the configs and tests and test resources, so the actual number of Scala files is much lower. We use an open-sourced solution called Skipper as an ingress controller, which gives us access to filters.

Scala 86

Scala Java Coding IT 86

Cloudera

SEPTEMBER 28, 2020

This customer’s workloads leverage batch processing of data from 100+ backend database sources like Oracle, SQL Server, and traditional Mainframes using Syncsort. Atlas ships with a default heap size of 2GB as specified in this Config changes to increase memory configurations resolved the issue. Ranger, Ranger KMS and Atlas.

Cloud 131

Cloud Kafka Professional Services Metadata 131

Tweag

MAY 16, 2023

A REPL nickel repl , a markdown documentation generator nickel doc and a nickel query command to retrieve metadata, types and contracts from code. The absence of a single source of truth might lead to inconsistencies. Here, | not_exported indicates that config is an internal value that shouldn’t appear in the final exported YAML.

MySQL 131

MySQL Metadata Data Validation Coding 131

Tweag

NOVEMBER 1, 2023

If you control the source code, you can change the value provided to the configuration-as-a-function at the call-site. For example: { i n p u t s | not_exported = { f o o | String | doc "Doc of foo" , b a r | Number | doc "Doc of bar" , u n u s e d | Bool | optional , } , l o c a l | not_exported = { computed = s t d.

Metadata 105

Metadata Programming Programming Language Coding 105

Christophe Blefari

OCTOBER 19, 2023

But dbt is still open-source, so it's up to the community to adapt. See the doc. Join blef.fr for free Enters dbt-loom Obviously the community did not welcome well this announcement as it converged with the new pricing. And the community adapted. Or even more, versioning models. That's all for the core project.

Project 264

Project Finance SQL Government 264

Lyft Engineering

JANUARY 30, 2023

We have some internal fine-tuning on top of open-source Flask to optimize with consideration for the Envoy load balancer and the underlying Gunicorn web server. It also ensures the config files contain necessary details like runtime secrets and database entries required for correct operation.

Machine Learning 92

Machine Learning Systems Python Coding 92

Lyft Engineering

MARCH 6, 2024

It also has links to the upgrade docs, an FAQ, and our support Slack channel for any questions. We add support for the new Python version in our internal tools in parallel with the existing Python version(s), as well as updating our docs. It’s fairly mechanical (copy/paste) — enough said.

Python 72

Python Metadata Datasets Coding 72

Cloudera

FEBRUARY 28, 2024

This blog post walks you through how you can use prefixless replication with Streams Replication Manager (SRM) to aggregate Kafka topics from multiple sources. In pull mode, the SRM service (specifically the SRM driver role instances) replicates data by pulling from their sources. Set this to: 8. No need to specify full paths.

Kafka 71

Kafka Management Big Data Architecture 71

Towards Data Science

JUNE 2, 2023

Mocking dbt sources and refs With dbt-unit-testing you can test dbt models independently by mocking their dependencies. We have a dbt app called health-insights that takes weight and height data from upstream sources and calculates the metric body mass index. We test our dbt app as a whole mocking only its sources.

Coding 73

Coding SQL Software Engineer Software Engineering 73

dbt Developer Hub

NOVEMBER 28, 2021

Most organizations spend at least some of their time contributing to an open source project. 100% of them, though, depend in some way on the output of open source communities. Call me weird, but I think everyone should be interested in how open source communities grow and operate. In fact we all do.

Raw Data 52

Raw Data Metadata Database Datasets 52

Rockset

JANUARY 16, 2019

python3 -m virtualenv rockset-kafka-demo source rockset-kafka-demo/bin/activate pip install -r requirements.txt Open config.py Connecting Rockset to Kafka Kafka Connect , an open-source component of Apache Kafka, is a framework for connecting Kafka with external systems such as databases, key-value stores, search indexes, and file systems.

Kafka 40

Kafka SQL Python Data 40

DoorDash Engineering

MARCH 7, 2023

/src RUN npm install --frozen-lockfile # We have to install these dev dependecies as regular dependencies to get hot swapping to work RUN npm install nodemon ts-node @types/pg # Bundle app source COPY. Continuing the theme of automation via Docker Compose, we can now use Docker to automatically apply our Terraform config on startup.

AWS 119

AWS PostgreSQL Database SQL 119

Big Data Tools

FEBRUARY 9, 2022

SeaTunnel is a tool that addresses one of the pain points our plugin solves: data synchronization between different sources. While our plugin does this in the UI right inside the IDE, SeaTunnel works on a different scale and provides users with a way to describe synchronization config declaratively.

Data Engineering 52

Data Engineering Data Engineer Engineering Big Data Tools 52

Big Data Tools

FEBRUARY 9, 2022

SeaTunnel is a tool that addresses one of the pain points our plugin solves: data synchronization between different sources. While our plugin does this in the UI right inside the IDE, SeaTunnel works on a different scale and provides users with a way to describe synchronization config declaratively.

Data Engineering 52

Data Engineering Data Engineer Engineering Big Data Tools 52

dbt Developer Hub

JUNE 19, 2020

Are you tagging individual models in config blocks? Useful links Packages Docs Package Hub dbt utils package ✅ Code style ​ Do you have a clearly defined code style? Do you use sources? If so, do you use source freshness tests? Do you use refs and sources for everything? Are you using YAML selectors? If not, why?

Project 40

Project BI SQL Utilities 40

Cloudera

SEPTEMBER 13, 2021

Cruise Control is one of the earliest open source tools to provide a solution for the failure management problem but lately for the monitoring problem as well. The open source version of Cruise Control stores metrics back to Kafka or Prometheus. This config only marks the broker dead and triggers an alert. Metrics Reporting.

Kafka 72

Kafka Bytes Utilities Management 72

Mutt Data

APRIL 11, 2023

Prometheus Prometheus is an open-source systems monitoring and alerting toolkit originally built at SoundCloud. Read the docs and you will find most use cases explained. Which is why we used that config in the next test. But there are lots of packages that add features and can be seamlessly integrated. Utilization ✅ ✅ 71.3%

Utilities 40

Utilities Python Deep Learning AWS 40

ProjectPro

DECEMBER 10, 2021

Source: Microsoft Azure docs By launching a cloud service instance, Azure cloud services can be utilized to implement multi-tier web-based apps in Azure. Redis open-source (OSS Redis) and Redis Enterprise (from Redis Labs) are both available as managed services through Azure Redis Cache.

BI 52

BI Cloud Computing SQL Database 52