Rockset

MARCH 21, 2019

This makes turning any type of data—from JSON, XML, Parquet, and CSV to even Excel files—into SQL tables a trivial pursuit. We automatically build multiple general-purpose indexes on all data ingested into Rockset, so that we can eliminate the need for database administration and query tuning for a wide spectrum of applications.

SQL 52

SQL Data Management Management Engineering 52

Confluent

FEBRUARY 6, 2019

It allows real-time data ingestion, processing, model deployment and monitoring in a reliable and scalable way. This blog post focuses on how the Kafka ecosystem can help solve the impedance mismatch between data scientists, data engineers and production engineers. For now, we’ll focus on Kafka.

Machine Learning 107

Machine Learning Python Kafka Java 107

Snowflake

APRIL 19, 2023

Welcome to the third blog post in our series highlighting Snowflake’s data ingestion capabilities, covering the latest on Snowpipe Streaming (currently in public preview) and how streaming ingestion can accelerate data engineering on Snowflake. What is Snowpipe Streaming?

Data Ingestion 57

Data Ingestion Kafka Java Data Pipeline 57

Knowledge Hut

JUNE 26, 2023

Since there are numerous ways to approach this task, it encourages originality in one's approach to data analysis. Moreover, this project concept should highlight the fact that there are many interesting datasets already available on services like GCP and AWS. Source: Use Stack Overflow Data for Analytic Purposes 4.

Data Engineering 98

Data Engineering Data Engineer Coding Project 98

WeCloudData

OCTOBER 19, 2021

Conclusion WeCloudData helped a client build a flexible data pipeline to address the needs from multiple business units requiring different sets, views and timelines of job market data.

Consulting 52

Consulting Raw Data Data Lake Data Pipeline 52

WeCloudData

OCTOBER 19, 2021

Conclusion WeCloudData helped a client build a flexible data pipeline to address the needs from multiple business units requiring different sets, views and timelines of job market data.

Consulting 52

Consulting Raw Data Data Lake Data Pipeline 52

phData: Data Engineering

AUGUST 4, 2023

Data storage is a vital aspect of any Snowflake Data Cloud database. Within Snowflake, data can either be stored locally or accessed from other cloud storage systems. What are the Different Storage Layers Available in Snowflake? They are flexible, secure, and provide exceptional performance.

Cloud Storage 52

Cloud Storage Google Cloud Amazon Web Services Data Storage 52

DareData

JANUARY 30, 2023

Our goal is to help data scientists better manage their models deployments or work more effectively with their data engineering counterparts, ensuring their models are deployed and maintained in a robust and reliable way. AWS Glue: A fully managed data orchestrator service offered by Amazon Web Services (AWS).

Data Engineering 52

Data Engineering Data Engineer NoSQL Engineering 52

ProjectPro

AUGUST 24, 2021

Salary of Data Engineers Data Engineering Tools Skills Required to Become a Data Engineer Responsibilities of a Data Engineer FAQS on Data Engineering Projects Data Engineering Projects List There are a few data-related skills that most data engineering practitioners must possess.

Data Engineering 52

Data Engineering Data Engineer Coding Project 52

Cloudera

MARCH 31, 2021

In this first Google Cloud release, CDP Public Cloud provides built-in Data Hub definitions (see screenshot for more details) for: Data Ingestion (Apache NiFi, Apache Kafka). Data Preparation (Apache Spark and Apache Hive) . Analyze static (Apache Impala) and streaming (Apache Flink) data.

Google Cloud 95

Google Cloud Cloud Amazon Web Services Cloud Storage 95

Cloudera

OCTOBER 15, 2020

The scenario is the same as it was in the previous blog but the ingest pipeline differs. Spark as the ingest pipeline tool for Search (i.e. If you do not have a CDP AWS account, please contact your favorite Cloudera representative, or sign up for a CDP trial here. <my access key> This value is generated for AWS users.

AWS 118

AWS Data Cloud Cloud Storage 118

Knowledge Hut

SEPTEMBER 26, 2023

Born out of the minds behind Apache Spark, an open-source distributed computing framework, Databricks is designed to simplify and accelerate data processing, data engineering, machine learning, and collaborative analytics tasks. This flexibility allows organizations to ingest data from virtually anywhere.

Data Lake 52

Data Lake Database-centric Pipeline-centric Machine Learning 52

Monte Carlo

FEBRUARY 15, 2023

Since the inception of the cloud, there has been a massive push to store any and all data. On the surface, the promise of scaling storage and processing is readily available for databases hosted on AWS RDS, GCP cloud SQL and Azure to handle these new workloads. Snowflake is the pioneer in cloud data warehousing.

Data Warehouse 52

Data Warehouse Unstructured Data AWS Business Intelligence 52

Rockset

MARCH 1, 2023

Developers can spin up or down virtual instances based on the performance requirements of their streaming ingest or query workloads. In addition, Rockset provides fast data access through the use of more performant hot storage, while cloud storage is used for durability.

Architecture 59

Architecture AWS SQL Cloud Storage 59

Netflix Tech

DECEMBER 14, 2018

MDVS also serves as the storehouse and the manager for the data schema itself. As was noted in the previous post , data schema could itself evolve over time, but all the data, ingested hitherto, has to remain compliant with the latest schema. NMDB leverages a cloud storage service (e.g.,

Media 94

Media Database Metadata Data Schemas 94

AltexSoft

MAY 12, 2023

Tools and platforms for unstructured data management Unstructured data collection Unstructured data collection presents unique challenges due to the information’s sheer volume, variety, and complexity. The process requires extracting data from diverse sources, typically via APIs. Hadoop, Apache Spark).

Unstructured Data 59

Unstructured Data NoSQL Hadoop Data Lake 59

ProjectPro

MAY 31, 2021

Data Description: You will use the Covid-19 dataset(COVID-19 Cases.csv) from data.world , for this project, which contains a few of the following attributes: people_positive_cases_count county_name case_type data_source Language Used: Python 3.7 This project also uses DataBricks since it is compatible with AWS.

Big Data 52

Big Data Coding Project Hadoop 52

AltexSoft

MARCH 30, 2023

Databricks architecture Databricks provides an ecosystem of tools and services covering the entire analytics process — from data ingestion to training and deploying machine learning models. Besides that, it’s fully compatible with various data ingestion and ETL tools. Let’s see what exactly Databricks has to offer.

Scala 64

Scala Data Lake BI Google Cloud 64

Monte Carlo

AUGUST 19, 2023

We’ll cover: What is a data platform? Recently, there’s been a lot of discussion around whether to go with open source or closed source solutions (the dialogue between Snowflake and Databricks’ marketing teams really brings this to light) when it comes to building your data platform.

Building 52

Building BI Data Lake Data Governance 52

Towards Data Science

NOVEMBER 4, 2023

Indeed, why would we build a data connector from scratch if it already exists and is being managed in the cloud? Very often it is row-based and might become quite expensive on an enterprise level of data ingestion, i.e. big data pipelines. The downside of this approach is it’s pricing model though. Image by author.

Data Engineering 87

Data Engineering Data Engineer Engineering BI 87