ProjectPro

NOVEMBER 14, 2015

Apache Hadoop and Apache Spark fulfill this need as is quite evident from the various projects that these two frameworks are getting better at faster data storage and analysis. These Apache Hadoop projects are mostly into migration, integration, scalability, data analytics, and streaming analysis. Why Apache Spark?

Hadoop 52

Hadoop Project Big Data Healthcare 52

Towards Data Science

FEBRUARY 9, 2024

In the second phase, we’ll develop an application that uses a language model to interact with this database. To set-up and run these tools we will use Docker. This article focuses more on practical application rather than theoretical aspects of the tools discussed. Image by the author. Overview of the data pipeline.

Kafka 76

Kafka Data Engineering Data Engineer PostgreSQL 76

Towards Data Science

MARCH 6, 2023

It’s possible to go from simple ETL pipelines built with python to move data between two databases to very complex structures, using Kafka to stream real-time messages between all sorts of cloud structures to serve multiple end applications. I covered Spark in many other posts. Read them later using their “path”. data:/data -./src:/src

Google Cloud 85

Google Cloud Cloud Storage Data Pipeline Cloud 85

Confluent

FEBRUARY 21, 2019

HELK is a free threat hunting platform built on various components including the Elastic stack, Apache Kafka ® and Apache Spark. You can read a worked example here of simulating such behavior with the Empire Project. You can see an example of a simple alerting application in an article written by Robin Moffatt here.

Process 81

Process Kafka Datasets SQL 81

Workfall

JUNE 27, 2023

Reading Time: 7 minutes The world of creating real-time applications can get complex when we have to consider latency, fault tolerance, scalability, and possible data losses. Traditionally, web sockets were the go-to option when it came to real-time applications, but think of a situation whereby there’s server downtime.

Kafka 75

Kafka Architecture AWS Transportation 75

Towards Data Science

FEBRUARY 19, 2024

Image from Unsplash Building a Semantic Book Search: Scale an Embedding Pipeline with Apache Spark and AWS EMR Serverless Using OpenAI’s Clip model to support natural language search on a collection of 70k book covers In a previous post I did a little PoC to see if I could use OpenAI’s Clip model to build a semantic book search.

AWS 61

AWS Building Bytes Python 61

Towards Data Science

FEBRUARY 15, 2023

The problem was so big that the terms “data swamp”, a joke on very messy data lakes, and “WORN paradigm”, Write Once Read Never, were created. I was a child at the time, I read all this history recently from modern literature) Time has passed and, based on the hits and misses of the past, new architectures were proposed. data:/data -./src:/src

Data Lake 80

Data Lake Data Warehouse Hadoop Data Architecture 80

AltexSoft

JULY 18, 2023

To some, the word Apache may bring images of Native American tribes celebrated for their tenacity and adaptability. On the other hand, the term spark often brings to mind a tiny particle that, despite its size, can start a large fire. What is Apache Spark? Apache Spark components.

Big Data 52

Big Data Data Process Process Hadoop 52

ProjectPro

JANUARY 24, 2023

But what makes Amazon SageMaker an important part of the applications like SiteEye? Read this article till the end as we find out the answer to the question by delving into the details of how SageMaker works and how it can be used to train, evaluate and deploy ML models in your applications. Why use Amazon SageMaker?

Building 52

Building Algorithm Machine Learning AWS 52

ProjectPro

SEPTEMBER 16, 2021

All major cloud platforms and various open-source applications are all trying to solve for production-ready machine learning. Retrieving images/analyses from Jupyter Notebooks to a presentation can be a tedious process prone to errors. 1) Perfect Project Structure – Cookiecutter & readme.so

Project 52

Project Amazon Web Services Machine Learning Data Science 52

Knowledge Hut

DECEMBER 26, 2023

This image depicts a very gh-level pipeline for DS. A) Project for Data Visualization The ability to create appealing, simple-to-read visualizations is a programming and design challenge, but your analysis will be much more beneficial if you succeed. Your portfolio will stand out if a project includes attractive charts.

Data Science 92

Data Science Python Programming Language Portfolio 92

DataKitchen

APRIL 13, 2021

Read the complete blog below for a more detailed description of the vendors and their capabilities. Apache Oozie — An open-source workflow scheduler system to manage Apache Hadoop jobs. BMC Control-M — A digital business automation solution that simplifies and automates diverse batch application workloads. Telm.ai — Telm.ai

Consulting 123

Consulting Machine Learning Data Science Data Pipeline 123

Zalando Engineering

NOVEMBER 22, 2017

Using Apache and the Kafka Streams API with Scala on AWS for real-time fashion insights This piece was originally published on confluent.io It fits in naturally with the functional style of the rest of our application. And why not use tools like Apache Hadoop or Apache Spark that provide implementations of MapReduce?”

Kafka 40

Kafka Scala Hadoop Algorithm 40

ProjectPro

DECEMBER 10, 2021

Azure Cloud Services is a Paas (platform-as-a-service) product that intends to provide robust, efficient, and cost-effective applications. Azure cloud services aid an application's scalability by making it easier and more adaptable. Organizations avail SaaS applications through a service delivery mechanism. on demand.

BI 52

BI Cloud Computing SQL Database 52

ProjectPro

MAY 31, 2021

None of this would have been possible without the application of big data. A big data project is a data analysis project that uses machine learning algorithms and different data analytics techniques on a large dataset for several purposes, including predictive modeling and other advanced analytics applications.

Big Data 52

Big Data Coding Project Hadoop 52

Towards Data Science

NOVEMBER 4, 2023

Image by author It also might be a datalake in the center and it depends on the type of our data platform and tools we use. Image by author. I previously wrote about it in one of my stories on Apache Iceberg table format [2]. It’s always a good practice to use something like Terraform to deploy our data pipeline applications.

Data Engineering 86

Data Engineering Data Engineer Engineering BI 86