Data Engineering Weekly

MARCH 3, 2024

RudderStack is the Warehouse Native CDP, built to help data teams deliver value across the entire data activation lifecycle, from collection to unification and activation. Editor’s Note: Chennai, India Meetup - March-08 Update We are thankful to Ideas2IT to host our first Data Hero’s meetup.

Data Engineering 67

Data Engineering Data Engineer Pipeline-centric Engineering 67

Data Engineering Podcast

SEPTEMBER 11, 2022

Summary Any business that wants to understand their operations and customers through data requires some form of pipeline. Building reliable data pipelines is a complex and costly undertaking with many layered requirements. Data stacks are becoming more and more complex.

Data Pipeline 130

Data Pipeline Building MongoDB Scala 130

Knowledge Hut

FEBRUARY 6, 2024

The DevOps Pipeline is the sequence of activities that flow from a customer's idea to the delivery of software and services. It is a framework used by organizations to plan their DevOps initiatives. The DevOps pipeline aims to improve software products' quality and delivery speed.

Building 52

Building AWS Coding Certification 52

Data Engineering Podcast

MAY 22, 2022

Summary Machine learning has become a meaningful target for data applications, bringing with it an increase in the complexity of orchestrating the entire data flow. Flyte is a project that was started at Lyft to address their internal needs for machine learning and integrated closely with Kubernetes as the execution manager.

Machine Learning 130

Machine Learning Data Engineering Data Engineer Cloud 130

Edureka

MAY 6, 2024

In today’s data-driven world, machine learning models play a huge role in developing sectors like healthcare, finance, transport, e-commerce, and so on. However, building and deploying these models is just the beginning. Ensuring their accuracy, reliability, and performance over time is a significant challenge.

Machine Learning 52

Machine Learning Metadata Programming Language Healthcare 52

Data Engineering Podcast

AUGUST 6, 2022

Summary The optimal format for storage and retrieval of data is dependent on how it is going to be used. For analytical systems there are decades of investment in data warehouses and various modeling techniques. Data stacks are becoming more and more complex.

Machine Learning 100

Machine Learning Database MySQL PostgreSQL 100

Data Engineering Podcast

OCTOBER 23, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Database 100

Database MySQL Cloud MongoDB 100

Cloudera

DECEMBER 13, 2023

Implementing a cohesive data vision LGIM’s Global Chief Operating Office is responsible for the company’s technology, data, client servicing and the management of its strategic operating agenda. This includes delivering a transformation project to modernize LGIM, enable growth, and introduce more scalable and seamless services.

Pipeline-centric 88

Pipeline-centric Professional Services Datasets BI 88

Data Engineering Podcast

JULY 24, 2022

Summary The current stage of evolution in the data management ecosystem has resulted in domain and use case specific orchestration capabilities being incorporated into various tools. RudderStack helps you build a customer data platform on your warehouse or data lake.

MongoDB 100

MongoDB Scala MySQL Data Lake 100

Knowledge Hut

MARCH 20, 2024

Typical roles and responsibilities include the following: Ability to create and evaluate AI models using neural networks, ML algorithms, deep learning, etc. Typical roles and responsibilities include the following: Ability to create and evaluate AI models using neural networks, ML algorithms, deep learning, etc.

Engineering 52

Engineering NoSQL Programming Language Deep Learning 52

Data Engineering Podcast

JUNE 5, 2022

Summary The best way to make sure that you don’t leak sensitive data is to never have it in the first place. The team at Skyflow decided that the second best way is to build a storage system dedicated to securely managing your sensitive information and making it easy to integrate with your applications and data systems.

Data Security 100

Data Security Metadata MongoDB Scala 100

Knowledge Hut

JUNE 26, 2023

Welcome to the world of data engineering, where the power of big data unfolds. If you're aspiring to be a data engineer and seeking to showcase your skills or gain hands-on experience, you've landed in the right spot. Therefore, the greatest thing you can do as a novice is to work on some real-time data engineering initiatives.

Data Engineering 98

Data Engineering Data Engineer Coding Project 98

Towards Data Science

APRIL 6, 2023

Today’s post follows the same philosophy: fitting local and cloud pieces together to build a data pipeline. But, instead of GCP, we’ll be using AWS. AWS is, by far, the most popular cloud computing platform, it has an absurd number of products to solve every type of specific problem you imagine.

AWS 78

AWS Data Pipeline Amazon Web Services Python 78

Knowledge Hut

MARCH 22, 2024

Did you know that Amazon Web Services (AWS) has a 33% market share in cloud computing? With this leadership status in the domain, the job roles associated with AWS have also gained traction. AWS solutions architect career opportunities have grown multiplefold. Businesses in every sector realize cloud adoption.

AWS 52

AWS Amazon Web Services Cloud Computing Google Cloud 52

DoorDash Engineering

AUGUST 1, 2023

Unfortunately, this was a challenge at DoorDash because of peak traffic failures while using our legacy metrics infrastructure based on StatsD. Just when we most needed observability data, the system would leave us in the lurch. That’s why we decided to migrate our observability technology stack to Prometheus-based monitoring.

AWS 82

AWS Transportation Programming Language Government 82

Knowledge Hut

MAY 2, 2024

As per Apache, “ Apache Spark is a unified analytics engine for large-scale data processing ” Spark is a cluster computing framework, somewhat similar to MapReduce but has a lot more capabilities, features, speed and provides APIs for developers in many languages like Scala, Python, Java and R. billion (2019 - 2022).

Scala 52

Scala Hospitality Healthcare Retail 52

ProjectPro

DECEMBER 7, 2021

Data pipelines are a significant part of the big data domain, and every professional working or willing to work in this field must have extensive knowledge of them. As data is expanding exponentially, organizations struggle to harness digital information's power for different business use cases.

Data Pipeline 52

Data Pipeline Architecture Kafka AWS 52

Knowledge Hut

SEPTEMBER 21, 2023

Azure DevOps comprises components like Azure Boards, Azure Repos, Azure Pipelines, Azure Test Plans, and Azure Artifacts to streamline the software development process and the delivery of top-notch applications faster than ever before. Azure DevOps is an industry that has gained tremendous popularity in recent years. What is Azure DevOps?

Engineering 52

Engineering AWS Cloud Computing Certification 52

Knowledge Hut

NOVEMBER 29, 2023

Cloud computing is helping them get access to their business data and applications from anywhere at any time, and at a reasonable price. As more and more companies are using AWS, the demand for AWS professionals has increased. So, there's no denying that certified AWS experts are in high demand.

AWS 52

AWS Certification Big Data Cloud Computing 52

Knowledge Hut

DECEMBER 26, 2023

According to the World Economic Forum, the amount of data generated per day will reach 463 exabytes (1 exabyte = 10 9 gigabytes) globally by the year 2025. Thus, almost every organization has access to large volumes of rich data and needs “experts” who can generate insights from this rich data.

Data Science 97

Data Science BI Business Intelligence Data Mining 97

Data Engineering Podcast

MARCH 22, 2021

For analytical systems, the only way to provide this reliably is by implementing change data capture (CDC). Unfortunately, this is a non-trivial undertaking, particularly for teams that don’t have extensive experience working with streaming data and complex distributed systems.

Data Warehouse 100

Data Warehouse Metadata Data Lake Hadoop 100

Data Engineering Podcast

DECEMBER 14, 2020

Summary As a data engineer you’re familiar with the process of collecting data from databases, customer data platforms, APIs, etc. At YipitData they rely on a variety of alternative data sources to inform investment decisions by hedge funds and businesses.

Data Analytics 100

Data Analytics Building MySQL Data 100

Knowledge Hut

DECEMBER 25, 2023

JavaScript was also used earlier but nowadays, it is created with JavaScript frameworks like ReactJS, Angular, or Vue. We are also required to know about DevOps, which is a practice of harmonizing development and operations whereby the entire pipeline from development, testing, deployment, continuous integration, and feedback is automated.

Java 98

Java Database PostgreSQL Project 98

Workfall

MAY 2, 2023

This blog will go over how to launch your Yew Rust application on AWS EC2. We will utilize GitHub Actions to automate the deployment process by configuring a pipeline to generate artifacts that will be deployed to an EC2 instance. We are going to use Nginx as a web server to serve our static files.

AWS 55

AWS Cloud Certification Utilities 55

Data Engineering Podcast

OCTOBER 14, 2019

Summary Managing a data warehouse can be challenging, especially when trying to maintain a common set of patterns. With 200Gbit private networking, scalable shared block storage, and a 40Gbit public network, you’ve got everything you need to run a fast, reliable, and bullet-proof data platform.

Data Warehouse 100

Data Warehouse PostgreSQL AWS Programming Language 100

Data Engineering Podcast

DECEMBER 9, 2018

Summary Apache Spark is a popular and widely used tool for a variety of data oriented projects. With the large array of capabilities, and the complexity of the underlying system, it can be difficult to understand how to get started using it. What are some of the main use cases for Spark? Who uses Spark?

Scala 100

Scala MySQL Kafka Hadoop 100

Knowledge Hut

SEPTEMBER 26, 2023

Amazon Web Services (AWS) is today’s most adopted cloud system. Several customers use the AWS security certification path to become more agile, lower costs, and innovate their products and services faster. So, let us discuss how secure digital environments are possible through AWS.

AWS 52

AWS Amazon Web Services Healthcare Computer Science 52

Data Engineering Podcast

SEPTEMBER 22, 2019

Summary Object storage is quickly becoming the unifying layer for data intensive applications and analytics. Modern, cloud oriented data warehouses and data lakes both rely on the durability and ease of use that it provides. And for your machine learning workloads, they just announced dedicated CPU instances.

AWS 100

AWS Google Cloud Cloud Storage Data Lake 100

Data Engineering Podcast

AUGUST 31, 2020

Summary Data warehouse technology has been around for decades and has gone through several generational shifts in that time. The current trends in data warehousing are oriented around cloud native architectures that take advantage of dynamic scaling and the separation of compute and storage.

Data Warehouse 100

Data Warehouse Cloud Building Data Lake 100

ProjectPro

FEBRUARY 25, 2022

What is a Machine Learning Pipeline? A machine learning pipeline helps automate machine learning workflows by processing and integrating data sets into a model, which can then be evaluated and delivered. A well-built pipeline helps in the flexibility of the model implementation.

Machine Learning 52

Machine Learning Building Amazon Web Services AWS 52

Data Engineering Podcast

SEPTEMBER 7, 2020

For analytical use cases you often want to combine data across multiple sources and storage locations. This frequently requires cumbersome and time-consuming data integration. If you hand a book to a new data engineer, what wisdom would you add to it?

Architecture 100

Architecture Data Architecture SQL Engineering 100

Knowledge Hut

APRIL 25, 2023

The tremendous growth in data generation, then the rise in data engineer jobs - there’s no arguing the fact that the big data industry is at its best pace and you, as an aspiring data engineer, have a lot to learn and make out of it - including some tools! While they go about it - enter big data data engineer tools.

Data Engineering 52

Data Engineering Data Engineer Engineering Google Cloud 52

ProjectPro

FEBRUARY 21, 2023

AWS or Azure? With so many data engineering certifications available , choosing the right one can be a daunting task. There are over 133K data engineer job openings in the US, but how will you stand out in such a crowded job market? The answer is- by earning professional data engineering certifications!

Certification 52

Certification Data Engineering Data Engineer Engineering 52

Data Engineering Podcast

OCTOBER 21, 2019

Summary The scale and complexity of the systems that we build to satisfy business requirements is increasing as the available tools become more sophisticated. In order to bridge the gap between legacy infrastructure and evolving use cases it is necessary to create a unifying set of components.

Cloud 100

Cloud Data Lake Hadoop Programming Language 100

Data Engineering Podcast

OCTOBER 7, 2019

Summary The process of exposing your data through a SQL interface has many possible pathways, each with their own complications and tradeoffs. One of the recent options is Rockset, a serverless platform for fast SQL analytics on semi-structured and structured data. This week’s episode is also sponsored by Datacoral.

Structured Data 100

Structured Data Cloud SQL Programming Language 100

Data Engineering Podcast

APRIL 19, 2020

Summary Modern applications frequently require access to real-time data, but building and maintaining the systems that make that possible is a complex and time consuming endeavor. Eventador is a managed platform designed to let you focus on using the data that you collect, without worrying about how to make it reliable.

Building 100

Building PostgreSQL MongoDB SQL 100

Data Engineering Podcast

OCTOBER 28, 2019

Summary Despite the fact that businesses have relied on useful and accurate data to succeed for decades now, the state of the art for obtaining and maintaining that information still leaves much to be desired. In an effort to create a better abstraction for building data applications Nick Schrock created Dagster.

Building 100

Building Data Pipeline Programming Language Metadata 100