Workfall

SEPTEMBER 26, 2023

Why Stream Data Using Server-Sent Events (SSE)? Unlike other real-time communication methods that involve complex setups or polling, SSE relies on a unidirectional flow of data, where the server pushes events as text-based messages to the client. SSE is standardized in HTML5 and is widely supported by modern browsers.

Python 85

Python Bytes Coding Data 85

Data Engineering Podcast

OCTOBER 5, 2021

In streaming systems this is necessary to ensure that a set of messages or transformations are all executed together across different queues. What are the use cases for transactions in a pub/sub messaging system? In this episode Denis Rystsov explains how he added support for transactions to the Redpanda streaming engine.

Engineering 100

Engineering MongoDB Kafka Data Lake 100

Tweag

JULY 5, 2023

Buck2 is a from-scratch rewrite of Buck , a polyglot, monorepo build system that was developed and used at Meta (Facebook), and shares a few similarities with Bazel. We were thrilled to have the opportunity to work with Meta on Buck2 to help make the tool useful and successful in the open-source use case.

Systems 139

Systems Building Java Programming Language 139

Rock the JVM

APRIL 24, 2024

Introduction In modern application development, efficient database interactions are crucial for building scalable and maintainable systems. To run a PostgreSQL database, we will be using Docker as a containerized service. The application will make use of Skunk concepts to interact with a PostgreSQL database and manage user data.

Scala 52

Scala Database PostgreSQL SQL 52

Snowflake

SEPTEMBER 6, 2023

By using cloud data platforms and their martech stack, retailers and consumer goods brands can masterfully stitch together previously fragmented customer touchpoints to offer customers more personalized experiences and drive meaningful engagements. A plethora of modern data analytics tools can help organizations achieve this.

Retail 87

Retail Machine Learning Data Science Data Analytics 87

Snowflake

OCTOBER 16, 2023

In addition to policy changes, there are major modernization efforts underway. Both policy changes and modernization efforts present challenges, but they also create opportunities to build a solid data foundation to drive collaboration and break out of data silos in each agency. IESs are the technological backbone for U.S.

Healthcare 77

Healthcare Programming Hospitality Food 77

Tweag

APRIL 17, 2024

In other words, Nix is now known enough as to serve as a conversation starter: the company might not use it, but many people have heard about it and were very interested to hear insights from big contributors like Tweag, the Modus Create OSPO. Cybersecurity Many security products were represented.

Cloud 76

Cloud Consulting Portfolio Programming 76

Data Engineering Weekly

MARCH 24, 2024

Companies are more open to adopting Gen AI for their internal use cases but have reservations about rolling it out to their clients. MES framework evaluates model quality across various lifecycle stages—prototyping, training, deployment, and prediction—using Service Level Agreement (SLA) principles.

Data Engineering 62

Data Engineering Data Engineer Engineering Metadata 62

Scott Logic

NOVEMBER 14, 2023

This is especially true with the modern micro-services architecture. It’s generally accepted that in such cases we don’t want the source system directly doing HTTP requests to all interested parties, we prefer a low level of coupling between systems.

Technology 52

Technology Kafka AWS Data 52

Cloudera

OCTOBER 18, 2023

The use cases for these technologies continue to expand and improve, shaping the industry in ways we could only dream of. Although this list is not exhaustive, I’ve provided an at-a-glance aggregate of use cases for quick wins, quicker wins, and longer-term strategies. Quickest wins!

Portfolio 77

Portfolio Banking Algorithm Machine Learning 77

Rock the JVM

OCTOBER 1, 2023

In this article we will cover gRPC which is a modern Open Source RPC framework designed by Google that uses Protocol Buffers for data serialization and HTTP/2 as a transport layer. One of the main use cases for gRPC is communication between internal microservices. Setting Up. lazy val protobuf = project.

Scala 63

Scala Metadata Transportation Java 63

LinkedIn Engineering

JANUARY 26, 2023

Co-authors:�� Michele Ursino and Joe Xue Introduction At LinkedIn, we believe that an opportunity can arise from just one conversation, so having reliable and powerful messaging capabilities to enable people to have those meaningful and professional conversations is crucial.

Architecture 79

Architecture Coding Recruitment Media 79

Knowledge Hut

APRIL 25, 2024

The continuously transforming world of cybersecurity can leave you longing for an understanding of today's modern threats due to a lack of cybersecurity vocabulary words. Malware Malware is the short form for “malicious software”, describing a wide variety of malicious software or code used to infect and/or damage a system.

Electronics 52

Electronics Utilities Banking Programming 52

Cloudera

APRIL 20, 2021

It continuously processes data from multiple streams and performs simple calculations to complex event processing for delivering sophisticated use cases. Streaming Analytics can be used in many industries: Healthcare: Monitoring hospital patients to get the latest and most actionable data to inform patient interactions better.

Hospitality 93

Hospitality Retail Kafka Data Ingestion 93

ProjectPro

FEBRUARY 8, 2023

Did you know thousands of businesses, including over 80% of the Fortune 100, use Apache Kafka to modernize their data strategies? Apache Kafka is the most widely used open-source stream-processing solution for gathering, processing, storing, and analyzing large amounts of data. What is Apache Kafka Used For?

Kafka 52

Kafka Banking Medical Healthcare 52

ProjectPro

FEBRUARY 6, 2023

With over 10 million active subscriptions, 50 million active topics, and a trillion messages processed per day, Google Cloud Pub/Sub makes it easy to build and manage complex event-driven systems. Google Pub/Sub provides global distribution of messages making it possible to send and receive messages from across the globe.

Google Cloud 52

Google Cloud Cloud Cloud Storage Data Ingestion 52

Workfall

JANUARY 17, 2024

Improve reading, speed, and user happiness using pagination tricks. Pagination is a method used to organize and present content, whether in printed documents or on digital platforms, in a more manageable and user-friendly manner. Following that, we’ll establish a connection to our MySQL database using Python code.

Python 76

Python MySQL Coding Database 76

Knowledge Hut

APRIL 26, 2024

Machine learning uses algorithms that comb through data sets and continuously improve the machine learning model. A d ataset is used to draw better insights and get a clear picture of a particular problem statement. Datasets play a crucial role and are at the heart of all Machine Learning models.

Datasets 77

Datasets Machine Learning Deep Learning Finance 77

Christophe Blefari

SEPTEMBER 1, 2023

If you want a first glimpse of the Dagster code, you can look at it on Github , not yet documented but commits messages are clean. On September 1, I'm still getting used to the school rhythm. After the modern data stack: welcome back, data platforms. Like wine, it takes time to get it right. So let's see where it goes.

Food 130

Food Data SQL Python 130

Data Engineering Podcast

MAY 1, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. How is Matillion architected?

Data Warehouse 100

Data Warehouse Data Integration Cloud Google Cloud 100

Data Engineering Podcast

DECEMBER 28, 2022

Summary With all of the messaging about treating data as a product it is becoming difficult to know what that even means. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code. __init__ covers the Python language, its community, and the innovative ways it is being used.

Data Lake 130

Data Lake Data Warehouse Data Pipeline MongoDB 130

Cloudera

JUNE 27, 2023

During the COVID-19 pandemic, telcos made unprecedented use of data and data-driven automation to optimize their network operations, improve customer support, and identify opportunities to expand into new markets. There are savings to be made in modernizing the data stack itself, in addition to applying data across the enterprise.

Data Architecture 62

Data Architecture Architecture Data Governance Government 62

Cloudera

DECEMBER 1, 2022

Full-stack observability is a critical requirement for effective modern data platforms to deliver the agile, flexible, and cost-effective environment organizations are looking for. RI is a global leader in the design and deployment of large-scale, production-level modern data platforms for the world’s largest enterprises.

Data Lake 82

Data Lake Data Pipeline Data Governance Analytics Application 82

Engineering at Meta

SEPTEMBER 12, 2023

But to get a better understanding of how exploitation could happen on VROS we decided to use this opportunity to write an elevation-of-privilege exploit that could execute arbitrary native code in VR Runtime. As an Android variant, VROS has many of the same security features as other modern Android systems.

Bytes 129

Bytes Coding Programming Process 129

Knowledge Hut

APRIL 17, 2023

It is a powerful technology that can be used in many web applications to provide real-time, bidirectional communication between a client and a server. They are used in various applications, real-time analytics, remote monitoring, notifications, alerts, etc. What are WebSockets Used for? How Do WebSockets Work?

IT 52

IT Programming Language Java Technology 52

Data Engineering Weekly

MAY 16, 2023

The modern LakeHouse format, such as Apache Iceberg & Apache Hudi, support a Zero-Copy WAP pattern implementation. In the Iceberg case, it is a simple two-step config change. The One-Phase WAP is structurally similar to the Two-Phase WAP, except in the One-Phase WAP, there is no deep copy from staging to the production environment.

Engineering 110

Engineering Kafka Data Pipeline Data Warehouse 110

U-Next

JUNE 21, 2022

The distinction between hashing and encryption is that hashing refers to converting permanent data into message digests, but encryption operates in two ways: decoding and encoding the data. Hashing serves to maintain the information’s integrity, while md5 encryption and decryption are used to keep data out of the hands of third parties.

Algorithm 52

Algorithm Banking Utilities Data Security 52

Data Engineering Weekly

SEPTEMBER 24, 2023

The exciting part of the article is about the comparison of Metaflow with Airflow and the decision not to use Airflow for ML workload. Your app and its documentation should be very clear about: The models or third-party AI providers you're using. link] Leo Godin: Are You Using Elementary for DBT?

Data Engineering 67

Data Engineering Data Engineer Engineering Kafka 67

Towards Data Science

APRIL 3, 2023

credit: @psalms original_photo Introduction In many cases, processing data in-stream, or as it becomes available, can help reduce an enormous data problem (due to the volume and scale of the flow of data) into a more manageable one. An example sensor event (encapsulated in the Google Protocol Buffer message format) is shown for clarity.

Process 86

Process Data Lake Systems Bytes 86

Data Engineering Podcast

OCTOBER 23, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. When is MariaDB the wrong choice?

Database 100

Database MySQL Cloud MongoDB 100

Workfall

APRIL 18, 2023

Reading Time: 6 minutes Modern applications are getting more real-time. Notification updates, chat messaging applications, and financial market updates are all examples of real-time updates in online apps. Case Studies Hands-on Conclusion There are several ways in which we can architect real-time applications.

Transportation 66

Transportation BI Java Coding 66

Data Engineering Podcast

SEPTEMBER 9, 2019

For operational data this typically means collecting log messages and system metrics. Often a different tool is used for each class of data, increasing the overall complexity and number of moving parts. What strategy are you using for project governance and sustainability? What are the main use cases that Vector enables?

Building 100

Building Kafka Media Data 100

Data Engineering Podcast

NOVEMBER 27, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. database engines, compute frameworks, etc.)

Data Process 100

Data Process Process Metadata Business Intelligence 100

Snowflake

JUNE 8, 2023

Expectations from customers and business partners have changed, making modernization even more critical. And, siloed data isn’t easy to find and use, and even more difficult to derive value from. And, that hope starts with thinking about how we use the data. And that message seems to resonate with data leaders.

Aggregated Data 62

Aggregated Data Raw Data Telecommunication Data 62

Rockset

MAY 3, 2022

The second is out-of-order messages, the first is duplicate messages, the third is off-by-one errors, and the first is duplicate messages. The Duplication Problem In distributed systems, messages are passed back and forth between many workers, and it’s common for messages to be generated two or more times.

Kafka 52

Kafka Data Warehouse Database Data 52

Data Engineering Podcast

MARCH 22, 2021

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. What are the alternatives to CDC?

Data Warehouse 100

Data Warehouse Metadata Data Lake Hadoop 100

Knowledge Hut

DECEMBER 1, 2023

Trojan malware comes from various aspects like Email Messages Transferring files from other devices Malicious Websites Social Engineering How to Characterize a Trojan? For case, in some cases adware sneaks onto your computer as portion of a bundle of program. what is trojan software?"

IT 52

IT Utilities Programming Certification 52