Engineering at Meta

DECEMBER 19, 2023

But, because of space limitations, we will only give examples of two existing components that played an important role: ZippyDB , our distributed key/value datastore, and Async , our aptly named asynchronous serverless function platform. Once all logical shards have been migrated, resharding is complete and we remove the original mapping.

Programming Language 141

Programming Language Engineering Database Coding 141

Cloudera

MARCH 22, 2024

That’s why Cloudera and Red Hat , renowned for their open-source solutions, have teamed up to bring Red Hat Enterprise Linux ( RHEL ) to Cloudera on public cloud as the operating system for all of our public cloud platform images. Let’s dive into what this means and why it’s a game-changer for our customers.

Cloud 112

Cloud Certification Systems Architecture 112

Snowflake

APRIL 8, 2024

This year’s entries presented an impressively diverse set of use cases. The list of Top 10 semi-finalists is a perfect example: we have use cases for cybersecurity, gen AI, food safety, restaurant chain pricing, quantitative trading analytics, geospatial data, sales pipeline measurement, marketing tech and healthcare.

Pipeline-centric 117

Pipeline-centric Food Healthcare Unstructured Data 117

LinkedIn Engineering

OCTOBER 19, 2023

This robust framework empowers near real-time data processing for critical services and platforms, ranging from machine learning and notifications to anti-abuse AI modeling. With over 950 million members, ensuring that our platform is running smoothly is critical to connecting members to opportunities worldwide. hours to 25 minutes).

Process 119

Process Lambda Architecture Kafka Machine Learning 119

Snowflake

APRIL 17, 2024

Snowflake’s new Python API (in public preview) marks a significant leap forward, offering a more streamlined, powerful solution for using Python within your data pipelines — and furthering our vision to empower all developers, regardless of experience, with a user-friendly and approachable platform.

Data Pipeline 95

Data Pipeline Python Data Engineering Data Engineer 95

Cloudera

MARCH 26, 2024

Cloudera: Your Trusted Partner in AI With over 25 Exabytes of Data Under Management and hundreds of customers leveraging our platform for Machine Learning, Cloudera has a long and successful history as an industry leader. Choose from high value use cases like Chat Q&A, Copilot, or Text Summarization. said Dr.

Machine Learning 79

Machine Learning Banking Professional Services Architecture 79

Netflix Tech

MARCH 7, 2024

Berg , Romain Cledat , Kayla Seeley , Shashank Srikanth , Chaoying Wang , Darin Yu Netflix uses data science and machine learning across all facets of the company, powering a wide range of business applications from our internal infrastructure and content demand modeling to media understanding.

Systems 90

Systems Media Machine Learning Data Warehouse 90

Cloudera

JANUARY 30, 2024

Here are some of the key use cases: Predictive maintenance: With time series data (sensor data) coming from the equipment, historical maintenance logs, and other contextual data, you can predict how the equipment will behave and when the equipment or a component will fail.

Manufacturing 78

Manufacturing Data Lake Data Science Cloud 78

Knowledge Hut

MARCH 22, 2024

In my opinion, website developers can show new clients that they can create successful projects using case studies, which is a great concept. This case study on web application development can also help the website developer gain credibility by demonstrating their abilities and the benefits they offer to companies.

Healthcare 52

Healthcare Electronics Education Medical 52

Snowflake

FEBRUARY 21, 2024

We previously announced Snowflake’s Unistore workload , which continues Snowflake’s legacy of breaking down data silos by uniting transactional and analytical data in a consistent and governed platform. Hybrid Tables is a new table type that enables transactional use cases within Snowflake with fast, high-concurrency point operations.

Government 90

Government AWS Data Architect Architecture 90

Knowledge Hut

JANUARY 11, 2024

Data scientists have been using data science techniques to solve challenging real-world issues in healthcare, agriculture, manufacturing, automotive, and many more. An excellent way to achieve this is through reading industry data science case studies. Almost every industry uses data science in some way.

Data Science 98

Data Science Transportation Hospitality Banking 98

Data Engineering Weekly

APRIL 7, 2024

link] Sponsored: Cloud Academy's Solution to Enhanced Embedded Analytics Cloud Academy, a SaaS e-learning platform, needed to deliver a seamless, highly available embedded analytics experience for their enterprise customers. We index only top-tier tables, promoting the use of these higher-quality datasets.

Data Engineering 70

Data Engineering Data Engineer Engineering Unstructured Data 70

Netflix Tech

DECEMBER 14, 2023

Streaming SQL on Data Mesh using Apache Flink Mark Cho, Guil Pires and Sujay Jain, Engineers from the Netflix Data Platform talk about how a managed Streaming SQL using Apache Flink can help unlock new Stream Processing use cases at Netflix.

Data Engineering 103

Data Engineering Data Engineer Engineering Metadata 103

Tweag

APRIL 17, 2024

In other words, Nix is now known enough as to serve as a conversation starter: the company might not use it, but many people have heard about it and were very interested to hear insights from big contributors like Tweag, the Modus Create OSPO. Internal Developer Platforms (IDPs) IDPs were a hot topic at Kubecon.

Cloud 76

Cloud Consulting Portfolio Programming 76

Knowledge Hut

APRIL 25, 2024

Although the term is new, because of the benefits of big data , it has been used since 1990s. The banking sector uses big data to better understand its clients. They can make sure that there are no unauthorized transactions by using big data in banking. Therefore, it doesn't offer any useful insights into client behavior.

Big Data 96

Big Data Entertainment Transportation Banking 96

Netflix Tech

MARCH 4, 2024

Operational automation–including but not limited to, auto diagnosis, auto remediation, auto configuration, auto tuning, auto scaling, auto debugging, and auto testing–is key to the success of modern data platforms. Therefore, the operational cost increases linearly with the number of failed jobs. Multi-objective optimizations.

Machine Learning 84

Machine Learning Big Data Data Engineering 84

Lyft Engineering

NOVEMBER 29, 2023

Introduction At Lyft, we have used systems like Apache ClickHouse and Apache Druid for near real-time and sub-second analytics. This is crucial for use cases like market signaling and forecasting which benefit from, and depend upon, the most up-to-date information. An example of how we use Druid rollup at Lyft.

Kafka 104

Kafka Data Ingestion Datasets Architecture 104

Monte Carlo

AUGUST 21, 2023

Integrating that OpenAI API into your application wasn’t that difficult and it may even turn out to be useful. Come to think of it, now that the project is complete there really isn’t any platform infrastructure that can help scale the next round of LLM models or other Gen AI product features. But now what happens? Via shutterstock.

Data Science 98

Data Science Machine Learning Media Data 98

Pinterest Engineering

OCTOBER 31, 2023

Harry Zhang, Jiajun Wang, Yi Li, Shunyao Li, Ming Zong, Haniel Martino, Cathy Lu, Quentin Miao, Hao Jiang, James Wen, David Westbrook | Cloud Runtime Team Image Source: [link] Overview Modern compute platforms are foundational to accelerating innovation and running applications more efficiently.

Architecture 95

Architecture Pipeline-centric Accessible Accessibility 95

Lyft Engineering

JUNE 28, 2023

In early 2022, Lyft already had a comprehensive Machine Learning Platform called LyftLearn composed of model serving , training , CI/CD, feature serving , and model monitoring systems. However, streaming data was not supported as a first-class citizen across many of the platform’s systems — such as training, complex monitoring, and others.

Machine Learning 115

Machine Learning Building Metadata Software Engineer 115

Knowledge Hut

MARCH 19, 2024

In this article, I will take you across some of the cool cloud computing case study examples and highlight cloud implementation in those cases. Instead of relying on local servers or personal devices to run applications, organizations and individuals can use a remote "cloud" of " servers to store and process data."

Cloud Computing 52

Cloud Computing Cloud Amazon Web Services AWS 52

Cloudera

OCTOBER 4, 2023

Flexibility for different use cases. Different data streams will have different characteristics, and having a platform flexible enough to adapt, with things like flexible partitioning for example, will be essential in adapting to different source volume characteristics. The post Don’t Blink: You’ll Miss Something Amazing!

Telecommunication 85

Telecommunication Data Warehouse Java Manufacturing 85

Netflix Tech

NOVEMBER 14, 2023

Write-Audit-Publish (WAP) Process This is the general pattern we use in our ETL pipelines. Psyberg’s metadata session table is used to identify the partitions included in a batch run. In this case, the minimum hour to process the data is hour 2. In this case, the maximum hour to process the data is hour 6.

Metadata 97

Metadata Data Pipeline Scala Data Workflow 97

Edureka

APRIL 24, 2024

Today, we’ll explore about ten of the most prominent use cases of Gen AI to help you use them better. Creating Content: Once the model has a good grasp of the patterns, it can use that knowledge to generate entirely new content. However, one crucial aspect to consider is using prompts.

Pharmaceutical 52

Pharmaceutical Manufacturing Datasets Algorithm 52

Cloudera

MARCH 5, 2024

This marks a significant milestone for the platform: according to IDC, today about half of the world’s enterprise production data under management is on-prem. The latest release of the Cloudera platform delivers a one-of-a-kind set of capabilities to bring the same open data lakehouse functionality from the cloud into those data centers.

Data Lake 87

Data Lake Government Data Storage Kafka 87

Knowledge Hut

MARCH 22, 2024

Azure IoT by Microsoft is a comprehensive platform that helps organizations connect, monitor, and manage many IoT devices and assets. What is Azure IoT Used For? Cloud platforms are becoming a huge part of organizations today. IDC forecasts that spending on IoT worldwide will grow to over $1.1 trillion by 2026. So why lag?

Cloud Computing 52

Cloud Computing Utilities Cloud Transportation 52

Knowledge Hut

MARCH 19, 2024

It offers more than 170 AWS services to the developers so they can use them from anywhere when required. I will explore the top 10 AWS applications and their use cases in this blog. It is a scalable and feature-complete cloud platform that helps companies of all sizes to take advantage of cloud computing. Why Use AWS?

AWS 52

AWS Cloud Computing Amazon Web Services Relational Database 52

Snowflake

FEBRUARY 5, 2024

To learn more about how the financial services industry will use AI in 2024, we sat down with our in-house industry and technology experts. However, sharing such data easily and securely has been a challenge with traditional data platforms. Here are just a few of their exciting predictions for the year ahead.

Unstructured Data 89

Unstructured Data Banking Government Insurance 89

Netflix Tech

MARCH 10, 2023

By Meenakshi Jindal Overview At Netflix, we built the asset management platform (AMP) as a centralized service to organize, store and discover the digital media assets created during the movie production. Hence we built the data pipeline that can be used to extract the existing assets metadata and process it specifically to each new use case.

Management 94

Management Kafka Metadata Media 94

Tweag

JULY 5, 2023

Buck2 is a from-scratch rewrite of Buck , a polyglot, monorepo build system that was developed and used at Meta (Facebook), and shares a few similarities with Bazel. We were thrilled to have the opportunity to work with Meta on Buck2 to help make the tool useful and successful in the open-source use case.

Systems 139

Systems Building Java Programming Language 139

Cloudera

JANUARY 26, 2024

In this article, we will walk you through the process of implementing fine grained access control for the data governance framework within the Cloudera platform. Case Introduction In this article we will take the example of a data governance office that wants to control access to metadata objects in the company’s central data repository.

Metadata 73

Metadata Data Governance Government Management 73

Snowflake

OCTOBER 25, 2023

You’ve probably used ChatGPT yourself out of curiosity if nothing else. That said, in some cases there is early tangible value being offered by the marketing ecosystem. Question #1: What kind of data are you using to train your AI models? Or as a marketer, you may have considerable knowledge about its marketing applications.

Government 80

Government Accessible Accessibility Technology 80

Knowledge Hut

MARCH 14, 2024

Companies are using deployment models for continuous integration and continuous delivery (CI/CD pipelines) to automate the deployment process. Typically, when discussing a large network (more than 20 machines), the phrase "software deployment" is used. Use Case: Ideal for applications where uninterrupted service is critical.

Utilities 52

Utilities Software Engineer Software Engineering Systems 52

Rock the JVM

FEBRUARY 22, 2023

First, we need to use a version of Java that is at least 19. We used virtual.threads.playground , but we can use any name we want. The important thing is that we need to use the requires directive to enable the incubator module. We’ll use Slf4j to log something on the console. playground { requires jdk.

Java 145

Java Programming Coding Scala 145

Snowflake

JANUARY 4, 2024

External data sources can fill that gap, which is why there’s a huge market for intelligence from third parties like government agencies, international organizations, open source platforms and nonprofits. But you’ll likely need to spend time and effort to acquire, vet and transform external data into a useful resource that delivers clear ROI.

SQL 90

SQL Retail Data Government 90