Knowledge Hut

MARCH 19, 2024

Think Google Docs or Microsoft 365 – they're SaaS because we use them online. PaaS (Platform as a Service): Providing a platform for application development and deployment. It provides tools and services needed for development and deployment, saving us from setting up everything from scratch.

Cloud Computing 52

Cloud Computing Cloud Accessible Accessibility 52

dbt Developer Hub

MAY 9, 2022

The main goal of this article is to spread awareness of these endpoints as the docs are being built & show you how to use them. Please keep in mind that the collection is not an evergreen doc. Building “starter projects” they can deploy as templates for new teams onboarding to dbt from a learning standpoint.

Cloud 52

Cloud Management Government Data Warehouse 52

Netflix Tech

MARCH 10, 2021

Access the AWS console ( docs , talk , demo ) ConsoleMe allows users to access the AWS console through the use of temporary IAM role credentials. Retrieve and serve short-lived AWS credentials through Weep ( docs , talk ) Weep is ConsoleMe’s CLI utility. Users have a number of ways they can log in to the AWS console.

AWS 100

AWS Accessible Accessibility Cloud 100

Monte Carlo

FEBRUARY 13, 2023

Historically, companies have leveraged data cataloging solutions with documentation to track table health, a tedious and time-intensive process that requires frequent updates when tables are refreshed, incidents occur, or new tests deployed. To dive into other new releases, check out our docs.

BI 52

BI Metadata Data Engineering Data Engineer 52

phData: Data Engineering

OCTOBER 12, 2022

Upon a quick trial and look at dbt Cloud, the primary things you might notice are the IDE as well as the ease of managing deployments. Hosted Doc Site for Documentation One of the most powerful features of dbt can be the documentation you generate. However, dbt Cloud offers you much more than that.

Cloud 52

Cloud Metadata SQL Data Warehouse 52

Monte Carlo

SEPTEMBER 6, 2022

To help organizations that have deployed Databricks Delta Lake ensure data quality, we announced new integrations with Delta Lake and Unity Catalog over the summer. Interested in deploying data observability for your Delta Lake? Check out the docs to learn more.

Data Lake 52

Data Lake Machine Learning Data Science Data 52

Cloudera

FEBRUARY 21, 2023

We could just add the following fragment describing the new field and its time into the schema under the “Schema Definition” tab: { "name" : "data_timestamp", "type": "long", "doc": "Injected from a custom data transformation" } The last step is to change the Kafka row time to use the new row that we just created.

SQL 108

SQL Kafka Raw Data Data 108

Zalando Engineering

NOVEMBER 19, 2023

And even if you were to gather all the information from the docs, it's still not enough. Because our Elasticsearch was a multi-cluster deployment, it would be practically impossible to upgrade in one go. First of all, we have a multi-cluster deployment, and the clusters are pretty large, so we're talking about some terabytes of data.

Scala 86

Scala Java Coding IT 86

Cloudera

JANUARY 31, 2024

For upgraded clusters, this feature will be disabled by default to ensure that existing SRM deployments will continue to work without changes in behavior. Better yet, prefixless replication is also a perfect tool to migrate that old Kafka deployment running on CDH, HDP, or HDF to CDP. This is the first part of a two-blog series.

Management 68

Management Kafka Big Data Cloud 68

Data Engineering Podcast

JANUARY 26, 2020

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Data Pipeline 100

Data Pipeline PostgreSQL Media Data Validation 100

Cloudera

OCTOBER 28, 2020

This is an optional step, once the logs have been filtered, what occurs to them after depends on the use case of this workflow’s deployment. xml version="1.0" encoding="UTF-8"?> Note that there are two ways to tackle this step, the first method is to convert it in a separate processor. Merging and Saving.

Kafka 64

Kafka SQL Professional Services Consulting 64

Cloudera

SEPTEMBER 28, 2020

Additionally, individual application teams contributed to test and deploy their jobs. Gather information on the current deployment. Perform the pre-transition steps for the components deployed in your clusters. Some customers may also need to include platform/infrastructure, network and information security resources.

Cloud 131

Cloud Kafka Professional Services Metadata 131

Data Engineering Podcast

JANUARY 14, 2017

I see that the docs mention using the map reduce pattern for analyzing the data in Pachyderm. What are some of the most interesting deployments and uses of Pachyderm that you have seen? I see that the docs mention using the map reduce pattern for analyzing the data in Pachyderm.

Data Lake 100

Data Lake Raw Data Kafka Data Pipeline 100

Data Engineering Podcast

MAY 19, 2019

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Data Analytics 100

Data Analytics Building Engineering Data Warehouse 100

Monte Carlo

DECEMBER 13, 2023

By that I mean feeding your GenAI engine the entirety of your confluence, Google, docs, policies, FAQ’s etc. I would think that most of the use cases I could see would have to work with some type of existing platform tool since the expertise to build and deploy one would take forever.

BI 59

BI Data Metadata Technology 59

Ripple Engineering

FEBRUARY 2, 2021

These are pretty standard DevOps requirements, and to achieve them, our team implemented a Continuous Integration Continuous Deployment (CI/CD) approach for our data applications in Google Cloud Platform (GCP). For more information about how to adjust the settings for the linters, see flake8-docs and isort-docs.

Building 52

Building Python Google Cloud Coding 52

Rockset

SEPTEMBER 25, 2020

What you’ll notice is there is a clause that states while cursor.alive: This while statement allows your code to keep checking to see if your cursor is still alive and doc references the different documents that captured the change in the oplog. first = oplog.find().sort('$natural', sort('$natural', pymongo.DESCENDING).limit(-1).next()

MongoDB 52

MongoDB Kafka Database NoSQL 52

dbt Developer Hub

JANUARY 23, 2023

This led us to have two main classes of tests — Data Integrity Tests (called Generic Tests in dbt docs) and Context Driven Tests (called Singular Tests in dbt docs), with varying levels of severity across both test classes.

Metadata 52

Metadata High Quality Data SQL Data Integration 52

dbt Developer Hub

AUGUST 2, 2022

As described before, we need to run a dbt docs generate in order to create updated JSON artifacts used in the pre-commit hooks. For that reason, we will need our CI step to execute this command, which will require setting up a profiles.yml file providing dbt the information to connect to the data warehouse.

Python 52

Python SQL Data Warehouse Project 52

Monte Carlo

DECEMBER 13, 2023

By that I mean feeding your GenAI engine the entirety of your confluence, Google, docs, policies, FAQ’s etc. I would think that most of the use cases I could see would have to work with some type of existing platform tool since the expertise to build and deploy one would take forever.

BI 52

BI Data Metadata Technology 52

dbt Developer Hub

MARCH 9, 2022

It wasn’t intuitive, but it was better than leaving it out of dbt docs. Even better, I could track versions of my ML models deployed over time in Modelbit + Hex and deploy ML external functions as dbt macros (by the way: how is this not more normal?!). What are the tradeoffs? ​

Machine Learning 52

Machine Learning Raw Data SQL Python 52

dbt Developer Hub

JANUARY 14, 2023

The builder: I’m a data freelancer who deploys end to end solutions, so when I have a data problem, I cannot just let it go. I developed the project locally, and to deploy to cloud functions I injected credentials to dbt via the dlt runner. dbt ​ No surprises there. This meant I could re-use the setup I did for the other dlt pipelines.

Data Ingestion 52

Data Ingestion Google Cloud Python Cloud 52

Grouparoo

AUGUST 15, 2021

There are lots of reasons why IPv6 might be better (including the fact that we are running out of public IPv4 Addresses ), but it’s rare to find a deployment environment that only has IPv6 addresses by default. server, you can to choose both the port to listen on and the hostname to bind to ( docs ). HTTP Server When starting a Node.js

IT 52

IT 52

Databand.ai

JUNE 14, 2023

dbt Teams use dbt Core and dbt Cloud to quickly deploy and test their analytics code. Great Expectations gives developers testing capabilities such as assertions about your data, automated data profiling, data validation, and Data Docs.

Amazon Web Services 52

Amazon Web Services Datasets High Quality Data ETL Tools 52

Rockset

OCTOBER 15, 2021

PaaS Pros You get similar scalability benefits as with an IaaS deployment. We all use SaaS applications, sometimes dozens of them in a given day—ranging from Google Docs to Salesforce CRM. However, as you might expect, the tradeoff is you may have fewer deployment and configuration options with a SaaS database.

Database 52

Database Cloud Cloud Computing MongoDB 52

Snowflake

JULY 10, 2023

Customers are looking for ways to better utilize unstructured data, which could be in any file format such as doc, pdf, images (jpeg, png), audio (mp3), video, e-mails, or industry-specific formats such as raster data for geospatial,las files for well log data, DICOM for medical, and many more.

Unstructured Data 40

Unstructured Data Python Process Scala 40

Rockset

NOVEMBER 5, 2021

Pattamatta was impressed by how easy it was to deploy Rockset. The docs were great.” “The fact that it could do analytics quickly was also important.” The serverless nature of Rockset made it incredibly simple to start on,” he said. “It

PostgreSQL 52

PostgreSQL Structured Data Data Lake Data Ingestion 52

dbt Developer Hub

APRIL 30, 2023

dbt brought code-based tests, integrated CI, efficient development with packages, and global docs. Join the #multi-project channel in the dbt Community Slack —start sparking up conversations with people like myself around the pains and gains of multi-deployment dbt projects. Comment directly in this post on thoughts of v1.5

Data Engineering 64

Data Engineering Data Engineer Engineering Finance 64

ProjectPro

JULY 15, 2021

Some of the popular examples of SaaS solutions are Google Docs and Dropbox. The service covers development tools, deployment tools, middleware, analytics solutions, etc. Cloud computing projects for final year engineering or cloud computing projects for MTech can be developed using cloud delivery and deployment models.

Cloud Computing 40

Cloud Computing Cloud Project Banking 40

DoorDash Engineering

FEBRUARY 27, 2024

The Search Engine supports all Apache Lucene fields, including text, numeric doc values, dimensional points, and KNN vectors. Users must be able to make changes in the index schema or fleet configuration and deploy them as soon as the changes do not have internal contradictions.

Engineering 130

Engineering Systems Designing Architecture 130

Tweag

MAY 16, 2023

The manifest of a web app, the configuration of an Apache virtual host, an Infrastructure-as-Code (IaC) cloud deployment (Terraform, Kubernetes, etc.). A REPL nickel repl , a markdown documentation generator nickel doc and a nickel query command to retrieve metadata, types and contracts from code. c o n t r a c t. s t r i n g.

MySQL 131

MySQL Metadata Data Validation Coding 131

dbt Developer Hub

FEBRUARY 12, 2024

In the past several years, dbt Docs helped centralize the documentation workflow and dramatically improved the documentation process. While useful, dbt Docs only ever provides a single point in time snapshot, and lacks any sense of your platform’s deployment and execution information. Look at that lineage! passing model!

Metadata 59

Metadata Raw Data BI Project 59

ProjectPro

SEPTEMBER 6, 2021

Recommended Web Scraping Tool: This project can be implemented by storing information in a google doc for analysis. Additionally, if you want the information to be sent to you periodically, you can deploy it on Raspberry Pi, which will run it after a specified time lag.

Project 52

Project Programming Language Entertainment Python 52

Tweag

NOVEMBER 1, 2023

For example: { i n p u t s | not_exported = { f o o | String | doc "Doc of foo" , b a r | Number | doc "Doc of bar" , u n u s e d | Bool | optional , } , l o c a l | not_exported = { computed = s t d. Combined with field metadata, records can come close to the notion of modules in other languages. s t r i n g.

Metadata 104

Metadata Programming Programming Language Coding 104

Cloudera

FEBRUARY 28, 2024

Additionally, the target cluster ( srm-target ) has SRM and SMM deployed on it. That is, this replication setup will be operating in pull mode, which is the Cloudera-recommended architecture for SRM deployments. You can find a more in-depth explanation of pull mode in the official docs. The topic will be created later.

Kafka 68

Kafka Management Big Data Architecture 68

Christophe Blefari

DECEMBER 19, 2022

To manage dbt you will have to answer multiple questions, but mainly dbt management is how the data team develop on dbt, how the project is validated/deployed, how you get alerted when something goes wrong, how you monitor. After development often comes deployment. Where dbt will run. Sometimes data teams forget it.

Management 130

Management Pipeline-centric Database-centric SQL 130

dbt Developer Hub

NOVEMBER 13, 2023

But as companies' dbt deployments have grown more complex, it doesn't make sense to assume that a single job is enough anymore. Make sure dbt Explorer always has the freshest information available ​ The old way : Your dbt docs site was based on a single job's run.

Cloud 52

Cloud Database-centric Metadata Data Warehouse 52