Knowledge Hut

APRIL 25, 2024

If you pursue the MSc big data technologies course, you will be able to specialize in topics such as Big Data Analytics, Business Analytics, Machine Learning, Hadoop and Spark technologies, Cloud Systems etc. There are a variety of big data processing technologies available, including Apache Hadoop, Apache Spark, and MongoDB.

Big Data 98

Big Data Technology NoSQL Hadoop 98

Netflix Tech

MARCH 4, 2024

We have deployed Auto Remediation in production for handling memory configuration errors and unclassified errors of Spark jobs and observed its efficiency and effectiveness (e.g., These advantages have been verified by the production deployment for remediating Spark jobs’ failures.

Machine Learning 84

Machine Learning Big Data Data Engineering 84

Christophe Blefari

SEPTEMBER 28, 2023

The era of Big Data was characterised by Hadoop, HDFS, distributed computing (Spark), above the JVM. That's why big data technologies got swooshed by the modern data stack when it arrived on the market—excepting Spark. This era was necessary and opened doors to the future, fostering innovation.

Cloud Storage 147

Cloud Storage Big Data Hadoop SQL 147

Data Engineering Podcast

NOVEMBER 6, 2022

With their new managed database service you can launch a production ready MySQL, Postgres, or MongoDB cluster in minutes, with automated backups, 40 Gbps connections from your application hosts, and high throughput SSDs. By the time errors have made their way into production, it’s often too late and damage is done.

MongoDB 130

MongoDB Scala MySQL Data Lake 130

Knowledge Hut

MAY 6, 2024

Agile methods have gained mainstream acceptance in software development organizations for the development of solutions for enhancing existing products or creating new ones. At the first thought, the potential of Agile methodology to be used for product design does not seem like a logical extension of software development approach.

Manufacturing 98

Manufacturing Retail Designing Project 98

Cloudera

DECEMBER 21, 2021

Since the release of Cloudera Data Engineering (CDE) more than a year ago , our number one goal was operationalizing Spark pipelines at scale with first class tooling designed to streamline automation and observability. Figure 2 – CDE product launch highlights in 2021. Modernizing pipelines. CDP Airflow Operators.

Data Engineering 115

Data Engineering Data Engineer Engineering Data Pipeline 115

Christophe Blefari

SEPTEMBER 25, 2023

Fresh money will help them to add AI capabilities to the product and increasing monitoring capabilities. In total they raised $100m and announced that Motherduck product is open for everyone and not anymore behind a waitlist. Yet I think Iceberg is not yet ready to be widely used ( Python write support still missing, you need Spark).

Data 130

Data Data Warehouse Data Storage Data Pipeline 130

Christophe Blefari

SEPTEMBER 25, 2023

Fresh money will help them to add AI capabilities to the product and increasing monitoring capabilities. In total they raised $100m and announced that Motherduck product is open for everyone and not anymore behind a waitlist. Yet I think Iceberg is not yet ready to be widely used ( Python write support still missing, you need Spark).

Data 130

Data Data Warehouse Data Storage Data Pipeline 130

Snowflake

NOVEMBER 15, 2023

As always, BUILD is created by builders, for builders, so you’ll find plenty to spark your interest and boost your skill set. Featured speakers with thought-provoking views on AI expansion, open source, app dev with gen AI and more No BUILD conference is complete without some big-picture thinking that sparks discussion and debate.

Building 108

Building Accessible Accessibility IT 108

Data Engineering Podcast

OCTOBER 16, 2022

Dremio is one of the companies leading the development of products and services that support the open lakehouse. With their new managed database service you can launch a production ready MySQL, Postgres, or MongoDB cluster in minutes, with automated backups, 40 Gbps connections from your application hosts, and high throughput SSDs.

Data Lake 100

Data Lake Food MongoDB Scala 100

Cloudera

JANUARY 6, 2021

In this blog series, we will explain how to configure PySpark and HBase together for basic Spark use as well as for jobs maintained in CDSW. For more information about CDSW visit the Cloudera Data Science Workbench product page. For more information about CDSW visit the Cloudera Data Science Workbench product page.

Machine Learning 131

Machine Learning Data Science Database Building 131

Snowflake

SEPTEMBER 14, 2023

Its technology team then moved Marriott Data Platform to AWS, which integrated batch and streaming data into cloud object storage, and processed data with a variety of technologies including Spark, Terraform, and Snowflake as a data warehouse for users to consume data through Tableau.

Data Architecture 94

Data Architecture Architecture Hadoop Data Warehouse 94

Precisely

MAY 20, 2024

With technology landscapes transforming at a breakneck pace, your ability to leverage cloud computing could be the game changer needed to boost efficiency and spark innovation in your business. The pandemic sparked the emergence of a skills gap that has remained a persistent and pressing problem for organizations of all sizes.

Cloud Computing 64

Cloud Computing Cloud Data Integration Datasets 64

Cloudera

JULY 13, 2021

Delivered through the Cloudera Data Platform (CDP) as a managed Apache Spark service on Kubernetes, CDE offers unique capabilities to enhance productivity for data engineering workloads: Visual GUI-based monitoring, troubleshooting and performance tuning for faster debugging and problem resolution. Key features of CDP Data Engineering.

Data Engineering 125

Data Engineering Data Engineer Engineering Pipeline-centric 125

Christophe Blefari

SEPTEMBER 15, 2023

At the same time Github Research quantified GitHub Copilot’s impact on developer productivity and happiness — Developer productivity is a difficult measure to compute. Also productivity ≠ speed, but speed is important. It will become a nice product in the Collibra data governance ecosystem.

Data Warehouse 130

Data Warehouse Data SQL Python 130

Christophe Blefari

APRIL 5, 2024

A talker did a survey during his keynote about the query engines used by the audience and Spark is still leading to BigQuery/Snowflake/Athena. How we built a 19 PiB logging platform with ClickHouse — ClickHouse is a tech company and you can see it from the blogpost. When is the data model finished?

SQL 130

SQL Data Python Datasets 130

Christophe Blefari

JANUARY 20, 2024

Obviously as data is different than "traditional product" — in term of users for instance — a data engineer uses other tools. and transformation (Spark, dbt, Pandas, etc.) Main technologies around stream are bus messages like Kafka and processing framework like Flink or Spark on top of the bus. This is not.

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Data Engineering Weekly

JANUARY 28, 2024

[link] Atlassian: The future of automation at Atlassian: Generating Confluence automation rules with large language models It is strictly not product marketing for Atlassian features :-) LLM greatly impacts how we build and use the software. It is exciting to see Atlassian sharing such product features utilizing LLM.

Data Engineering 57

Data Engineering Data Engineer Engineering AWS 57

Data Engineering Weekly

MAY 26, 2024

The former can lead to bad decisions and sub-par experiences; the latter leads to a superb product and experience that enables the people who use our services to make quick, data-informed decisions. Zillow writes about its approach to building easy-to-use data that adopts certified datasets, aka data products.

Data Engineering 57

Data Engineering Data Engineer Engineering Google Cloud 57

Data Engineering Weekly

NOVEMBER 26, 2023

We’ve seen a fleet of tools like TextToSQL ; Slack bots to ask questions to your data warehouse , Chat interface for spreadsheets , and even the English SDK for Spark ! Databricks is already doing a lot of these move-up stack products. One of the engines I’m excited to watch is an alternative to Apache Spark.

Data Engineering 70

Data Engineering Data Engineer Engineering Kafka 70

Cloudera

FEBRUARY 14, 2024

It offers several benefits such as schema evolution, hidden partitioning, time travel, and more that improve the productivity of data engineers and data analysts. As of this writing, Spark doesn’t write equality deletes, but it is capable of reading them. Unless aligned properly, a Spark task might touch multiple partitions.

Bytes 56

Bytes Metadata Data Lake SQL 56

Data Engineering Weekly

MARCH 3, 2024

Here is the agenda, 1) Data Application Lifecycle Management - Harish Kumar( Paypal) Hear from the team in PayPal on how they build the data product lifecycle management (DPLM) systems. 4) Building Data Products and why should you?

Data Engineering 67

Data Engineering Data Engineer Pipeline-centric Engineering 67

Data Engineering Weekly

MAY 16, 2023

I posted this LinkedIn post that sparked some exciting conversation. If the Audit step succeeds, the data in the staging environment gets copied to the production. The One-Phase WAP is structurally similar to the Two-Phase WAP, except in the One-Phase WAP, there is no deep copy from staging to the production environment.

Engineering 110

Engineering Kafka Data Pipeline Data Warehouse 110

Cloudera

SEPTEMBER 28, 2020

They have dev, test, and production clusters running critical workloads and want to upgrade their clusters to CDP Private Cloud Base. Customer Environment: The customer has three environments: development, test, and production. Hive, Ranger, Atlas, Spark. Production. Hive, Ranger, Atlas, Spark. Production.

Cloud 130

Cloud Kafka Professional Services Metadata 130

Cloudera

JUNE 26, 2023

For data engineering teams, Airflow is regarded as the best in class tool for orchestration (scheduling and managing end-to-end workflow) of pipelines that are built using programming languages like Python and SPARK. Impala vs Spark Use Impala primarily for analytical workloads triggered by end users.

ETL Tools 75

ETL Tools Programming Language Datasets Data Pipeline 75

Yelp Engineering

AUGUST 3, 2022

In this blog post, we introduce Spark-Lineage, an in-house product to track and visualize how data at Yelp is processed, stored, and transferred among our services. What is Spark-Lineage? Spark-ETL is our inhouse wrapper around Spark, providing high-level APIs to run Spark batch jobs and abstracting away the complexity of Spark.

Data 52

Data Engineering Process 52

Cloudera

FEBRUARY 16, 2021

Apache Spark in the Cloudera Data Engineering experience can be used with Apache HBase and Apache Phoenix in COD. You can read or write data using the Apache Phoenix JDBC driver, the Phoenix-Spark connector, or the HBase-Spark connector. You can use the Phoenix-Spark connector plugin to perform READ and WRITE operations on COD.

Database 82

Database Machine Learning Data Lake Kafka 82

Cloudera

DECEMBER 13, 2021

Cloudera’s security and engineering teams have identified the impact of this CVE across our product suite, and Cloudera customers have been sent detailed updates through Cloudera’s Technical Support Bulletins (TSB) and My Cloudera support cases. What Cloudera products and versions are affected?

Java 126

Java Coding Project Engineering 126

Netflix Tech

DECEMBER 14, 2023

The talk uses Apache Spark as an example, but the concepts generalize regardless of your specific tools. Media Data for ML Studio Creative Production In the last 2 decades, Netflix has revolutionized the way video content is consumed, however, there is significant work to be done in revolutionizing how movies and tv shows are made.

Data Engineering 103

Data Engineering Data Engineer Engineering Metadata 103

LinkedIn Engineering

OCTOBER 19, 2023

Authors: Bingfeng Xia and Xinyu Liu Background At LinkedIn, Apache Beam plays a pivotal role in stream processing infrastructures that process over 4 trillion events daily through more than 3,000 pipelines across multiple production data centers.

Process 119

Process Lambda Architecture Kafka Machine Learning 119

U-Next

FEBRUARY 26, 2023

Today, SAS offers an extensive suite of products for data management, predictive analytics, risk management, and other areas related to business intelligence (BI). Microsoft Office (Word, Excel, Outlook) are some of its best-known products. Average Salary per annum: INR 10.7 lakhs H2O.ai H2O.ai is a platform for Data Science.

Data Science 98

Data Science Machine Learning Food Consulting 98

Precisely

FEBRUARY 2, 2024

This news sparked our curiosity to better understand whether Glendale’s investment drove sustained visit increases to hotels, even after the event. And to take your analyses to the next level, planners and marketers can leverage our location data & insights products to help predict how events impact businesses.

Entertainment 69

Entertainment Management Data IT 69

Data Engineering Weekly

OCTOBER 1, 2023

Register for IMPACT: The Data Observability Summit | November 8, 2023 Announcing keynotes from three data & AI pioneers, Moneyball expert Billy Beane, poker legend Annie Duke, and SVP of Product Management at Salesforce AI, Nga Phan!

Data Engineering 52

Data Engineering Data Engineer Engineering Data Pipeline 52

Cloudera

OCTOBER 4, 2023

For example: In manufacturing, fast-moving data provides the only way to detect — or even predict and prevent — defects in real time before they propagate across an entire production cycle. This will reduce defect rates, increasing product yield.

Telecommunication 84

Telecommunication Data Warehouse Java Manufacturing 84

Cloudera

OCTOBER 5, 2020

This integration will include NVIDIA’s data science software such as NVIDIA AI , NVIDIA RAPIDS and NVIDIA RAPIDS Accelerator for Apache Spark 3.0. By teaming up with NVIDIA, we can bring accelerated computing to the entire data lifecycle on any cloud,” said Arun Murthy, Chief Product Officer at Cloudera. with Spark 3.0

Cloud 87

Cloud Business Analyst Deep Learning Machine Learning 87

ThoughtSpot

JANUARY 31, 2024

In the paragraphs below, I’ll focus on our AI strategy for our core internal analytics product, ThoughtSpot Analytics, including advancements in search, SpotIQ, and mobile, with security as a core element across all three. If you’re ready to spark your own Data Renaissance with ThoughtSpot , I invite you to request a demo.

BI 89

BI Business Intelligence Machine Learning Metadata 89

Cloudera

MARCH 5, 2024

This marks a significant milestone for the platform: according to IDC, today about half of the world’s enterprise production data under management is on-prem. This approach brings new compute engines into the fold, adding Spark, Flink, Impala, and NiFi, enabling concurrent access and processing of datasets within Iceberg.

Data Lake 86

Data Lake Data Storage Government Kafka 86