Christophe Blefari

SEPTEMBER 28, 2023

The era of Big Data was characterised by Hadoop, HDFS, distributed computing (Spark), above the JVM. That's why big data technologies got swooshed by the modern data stack when it arrived on the market—excepting Spark. An easy-to-manage central storage and querying and transforming layer in SQL.

Cloud Storage 147

Cloud Storage Big Data Hadoop SQL 147

Marc Lamberti

OCTOBER 11, 2023

For example, running a SQL request on Postgres means creating a connection, and a cursor, instantiating and configuring some objects, running the SQL query, and so on. How to use the DockerOperator We will use the DockerOperator in this example to run a Spark job. jar /spark/jars/ && mv aws-java-sdk-bundle-1.11.1026.jar

AWS 130

AWS Python Hadoop SQL 130

databricks

SEPTEMBER 21, 2023

Introduction In this blog post, we'll explore a set of advanced SQL functions available within Apache Spark that leverage the HyperLogLog algorithm, enabling.

Algorithm 99

Algorithm SQL Data Engineering Data Engineer 99

databricks

SEPTEMBER 18, 2023

We are thrilled to introduce the support of a new SQL feature in Apache Spark and Databricks: Lateral Column Alias (LCA). This feature.

SQL 104

SQL Engineering Data 104

Snowflake

DECEMBER 4, 2023

Apache Iceberg’s ecosystem of diverse adopters, contributors and commercial support continues to grow, establishing itself as the industry standard table format for an open data lakehouse architecture. Your other engines that may be writing to the table, such as Apache Spark or Apache Flink, can continue to write, and Snowflake can read.

Building 85

Building Metadata Cloud Storage AWS 85

Netflix Tech

DECEMBER 14, 2023

Streaming SQL on Data Mesh using Apache Flink Mark Cho, Guil Pires and Sujay Jain, Engineers from the Netflix Data Platform talk about how a managed Streaming SQL using Apache Flink can help unlock new Stream Processing use cases at Netflix.

Data Engineering 103

Data Engineering Data Engineer Engineering Metadata 103

Christophe Blefari

MARCH 22, 2024

On my side I'll talk about Apache Superset and what you can do to build a complete application with it. Finally, xAI released Grok-1 in open — The weights are available in torrent / HF and everything is under Apache License. This is a nice way to mix SQL and Python code. Now give me the news.

Electronics 130

Electronics Media Data Python 130

Christophe Blefari

SEPTEMBER 25, 2023

Astronomer released Ask Astro — A LLM application that is able to understand Astro docs to answer most of the Apache Airflow questions. Use the new SQL commands MERGE and QUALIFY in Redshift — Redshift still exists and tries to catches with the competition. The source code is on Github. Tabular raised $26m Series B.

Data 130

Data Data Warehouse Data Storage Data Pipeline 130

Christophe Blefari

SEPTEMBER 25, 2023

Astronomer released Ask Astro — A LLM application that is able to understand Astro docs to answer most of the Apache Airflow questions. Use the new SQL commands MERGE and QUALIFY in Redshift — Redshift still exists and tries to catches with the competition. The source code is on Github. Tabular raised $26m Series B.

Data 130

Data Data Warehouse Data Storage Data Pipeline 130

Data Engineering Podcast

OCTOBER 16, 2022

Ascend users love its declarative pipelines, powerful SDK, elegant UI, and extensible plug-in architecture, as well as its support for Python, SQL, Scala, and Java. Ascend automates workloads on Snowflake, Databricks, BigQuery, and open source Spark, and can be deployed in AWS, Azure, or GCP.

Data Lake 100

Data Lake Food MongoDB Scala 100

Cloudera

OCTOBER 16, 2023

Cloudera recently released a fully featured Open Data Lakehouse , powered by Apache Iceberg in the private cloud, in addition to what’s already been available for the Open Data Lakehouse in the public cloud since last year. Please note, you can also leverage Flink and SQL Stream Builder in CSA 1.11 Cloudera Flow Management 2.1.6

Cloud 75

Cloud Kafka SQL Data 75

Cloudera

DECEMBER 21, 2021

Since the release of Cloudera Data Engineering (CDE) more than a year ago , our number one goal was operationalizing Spark pipelines at scale with first class tooling designed to streamline automation and observability. We also introduced Apache Airflow on Kubernetes as the next generation orchestration service.

Data Engineering 115

Data Engineering Data Engineer Engineering Data Pipeline 115

Cloudera

MARCH 2, 2023

Recently, we announced enhanced multi-function analytics support in Cloudera Data Platform (CDP) with Apache Iceberg. It allows multiple data processing engines, such as Flink, NiFi, Spark, Hive, and Impala to access and analyze data in simple, familiar SQL tables. Currently, Iceberg support in CSP is in technical preview mode.

Process 112

Process SQL Kafka Database 112

Jesse Anderson

DECEMBER 12, 2022

Doug Cutting took those papers and created Apache Hadoop in 2005. Hadoop was hard to program, and Apache Hive came along in 2010 to add SQL. Apache Pig in 2008 came too, but it didn’t ever see as much adoption. Apache HBase came in 2007, and Apache Cassandra came in 2008. We lacked a scalable pub/sub system.

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Cloudera

JUNE 18, 2022

Warehouses are vertically integrated for SQL Analytics, whereas Lakes prioritize flexibility of analytic methods beyond SQL. The first generation of the Hive Metastore attempted to address the performance considerations to run SQL efficiently on a data lake. Hive and Spark of course, but also Presto, Impala, and many more.

Data Lake 89

Data Lake Data Warehouse BI SQL 89

ProjectPro

FEBRUARY 21, 2023

With around 35k stars and over 26k forks on Github, Apache Spark is one of the most popular big data frameworks used by 22,760 companies worldwide. Apache Spark is the most efficient, scalable, and widely used in-memory data computation tool capable of performing batch-mode, real-time, and analytics operations.

Scala 52

Scala Programming Language Java Hadoop 52

Jesse Anderson

JANUARY 9, 2023

Since Kafka Streams is part of the Apache project, I don’t see it going away as quickly. It employed some of the top people in the Apache Flink project. Coalescing Spark Streaming vs. Flink The general purpose real-time compute is coalescing around Spark Streaming and Flink. In 2019, Alibaba bought Ververica.

Kafka 147

Kafka Coding Technology SQL 147

Knowledge Hut

DECEMBER 26, 2023

Most Popular Programming Certifications C & C++ Certifications Oracle Certified Associate Java Programmer OCAJP Certified Associate in Python Programming (PCAP) MongoDB Certified Developer Associate Exam R Programming Certification Oracle MySQL Database Administration Training and Certification (CMDBA) CCA Spark and Hadoop Developer 1.

Certification 98

Certification Programming MongoDB R (Programming) 98

ProjectPro

FEBRUARY 16, 2023

Apache Spark is an open-source, distributed computing system for big data processing and analytics. Today, the Apache Spark project has over 1,000 contributors from over 250 companies worldwide. Thus, there is a tremendous demand for spark professionals in the coming years. Indeed recently posted nearly 2.4k

Big Data 52

Big Data Scala Machine Learning Hadoop 52

Data Engineering Podcast

AUGUST 20, 2021

In this episode Vikrant Dubey discusses his work on the Cuelake project which allows data analysts to build a lakehouse with SQL queries. By building on top of Zeppelin, Spark, and Iceberg he and his team at Cuebook have built an autoscaled cloud native system that abstracts the underlying complexity. No more scripts, just SQL.

Building 100

Building Data Lake Data Warehouse SQL 100

Cloudera

FEBRUARY 9, 2021

In this blog post, we’ll look at both Apache HBase and Apache Phoenix concepts relevant to developing applications for Cloudera Operational Database. Cloudera Operational Database is powered by Apache HBase and Apache Phoenix. Apache Phoenix is a SQL layer that provides a programmatic ANSI SQL interface.

Database 101

Database Java Data Ingestion SQL 101

Christophe Blefari

JULY 3, 2023

Once you are happy with the quality of answer you'll be able to publish the model and use it in SQL queries and write pipelines on top of it to infer on new documents and send emails when needed. In the demo there is also how the SQL UI integrates LLMs generating SQL from a comment. That's all.

SQL 130

SQL Data Kafka AWS 130

Cloudera

AUGUST 26, 2021

Apache Ozone is a scalable distributed object store that can efficiently manage billions of small and large files. In addition to big data workloads, Ozone is also fully integrated with authorization and data governance providers namely Apache Ranger & Apache Atlas in the CDP stack. create_bucket(Bucket='spark-bucket').

Data Science 107

Data Science Cloud Hadoop Metadata 107

Knowledge Hut

DECEMBER 28, 2023

The Apache Hadoop framework provides tools for efficient analysis, resource management, and parallel processing of datasets. Apache Mahout is a text mining project built on Hadoop; it offers a library of methods for doing machine learning and data mining on massive datasets. Why Are Hadoop Projects So Important?

Hadoop 52

Hadoop Project Datasets Big Data 52

Marc Lamberti

SEPTEMBER 22, 2023

What are XCOMs in Apache Airflow? If you checked my course “Apache Airflow: The Hands-On Guide”, Aiflow XCom should not sound unfamiliar. One solution could be to store the accuracies in a database and fetch them back in the task choosing_model with an SQL request. It is not Spark or Flink. What is an Airflow XCOM?

MySQL 246

MySQL Data Pipeline Database Python 246

Data Engineering Podcast

JANUARY 7, 2022

Prophecy provides an easy-to-use visual interface to design & deploy data pipelines on Apache Spark & Apache Airflow. – You visually design the pipelines, and Prophecy generates clean Spark code with tests on git; then you visually schedule these pipelines on Airflow. No more scripts, just SQL.

BI 100

BI Data Warehouse Metadata SQL 100

Cloudera

AUGUST 3, 2022

Fine grained access control (FGAC) with Spark. One approach is to use 3rd party tools (such as Privacera ) that integrate with Spark. This makes CDP the only platform where customers can use Spark with fine grained access control automatically, without requiring any additional tools or integrations. Starting with CDP 7.1.7

Datasets 84

Datasets BI Machine Learning Accessible 84

Cloudera

AUGUST 8, 2022

In June 2022, Cloudera announced the general availability of Apache Iceberg in the Cloudera Data Platform (CDP). Iceberg is a 100% open-table format, developed through the Apache Software Foundation , which helps users avoid vendor lock-in and implement an open lakehouse. . Cloudera Data Engineering (Spark 3) with Airflow enabled.

Data Warehouse 87

Data Warehouse BI Machine Learning SQL 87

Knowledge Hut

JUNE 30, 2023

It covers popular technologies such as Apache Kafka, Apache Storm, and Apache Hadoop, giving users practical advice on developing and executing effective data pipelines. With helpful illustrations and thorough explanations, it assists readers in comprehending how to use Spark for big data processing and analytics applications.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Warehouse 52

Cloudera

JUNE 10, 2021

Apache Spark is a very popular analytics engine used for large-scale data processing. CDP Operational Database Experience Experience (COD) is a CDP Public Cloud service that lets you create and manage operational database instances and it is powered by Apache HBase and Apache Phoenix. . Prerequisites .

Database 87

Database Data Engineering Data Engineer Big Data 87

Data Engineering Weekly

AUGUST 13, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork, and SQL grunt work out of building complete customer profiles, so you can quickly ship actionable, enriched data to every downstream team. Gluten from Intel & Kyligence is taking an open-source attempt to accelerate Spark SQL queries.

Data Engineering 52

Data Engineering Data Engineer Engineering Food 52

Christophe Blefari

DECEMBER 31, 2023

At the same time they released Mixtral 8x7B, the first open-source model of this calibre under an Apache Licence. BigQuery now integrates DuetAI — to help you generate or complete SQL queries. Other reads The state of SQL-based observability , on ClickHouse blog. And the weight are open-source as well.

Data 100

Data Cloud Storage Datasets Python 100

Cloudera

JANUARY 6, 2021

Apache HBase is an effective data storage system for many workflows but accessing this data specifically through Python can be a struggle. In this blog series, we will explain how to configure PySpark and HBase together for basic Spark use as well as for jobs maintained in CDSW. Prerequisites: Have a CDP cluster with HBase and Spark.

Machine Learning 131

Machine Learning Data Science Database Building 131

ProjectPro

SEPTEMBER 9, 2021

Apache Hive and Apache Spark are the two popular Big Data tools available for complex data processing. Spark vs. Hive comparison elaborates on the two tools’ architecture, features, limitations, and key differences. Apache Spark , on the other hand, is an analytics framework to process high-volume datasets.

Hadoop 52

Hadoop Big Data Tools Java SQL 52

Christophe Blefari

APRIL 14, 2023

Castor announced Castor AI — Doing it the other way Castor released a feature that explains a SQL query in natural language. In a nutshell they replaced Spark (EMR) in-memory transformations by BigQuery. This time Spotify explains how they did it with Apache Beam.

Datasets 130

Datasets Data Deep Learning SQL 130

Cloudera

FEBRUARY 14, 2024

Introduction Apache Iceberg has recently grown in popularity because it adds data warehouse-like capabilities to your data lake making it easier to analyze all your data — structured and unstructured. As of this writing, Spark doesn’t write equality deletes, but it is capable of reading them. Another problem arises with partitions.

Bytes 56

Bytes Metadata Data Lake SQL 56

Scott Logic

MARCH 25, 2024

Within the Amazon AWS environment, more complex analyses can be carried out in Amazon Athena , using Apache Spark and Athena Notebooks. QuickSight integrates seamlessly with AWS services and a number of third-party tools often used in AWS, such as Apache Spark. We expect that more features will be added over time.

BI 52

BI Business Intelligence Cloud AWS 52