This site uses cookies to improve your experience. By viewing our content, you are accepting the use of cookies. To help us insure we adhere to various privacy regulations, please select your country/region of residence. If you do not select a country we will assume you are from the United States. View our privacy policy and terms of use.
apache-parquet 
Uber Engineering
JUNE 29, 2023
Experience the power of row-level secondary indexing in Apache Parquet, enabling 3-20X faster upserts and unlocking new possibilities for efficient table ACID operations in today’s Lakehouse architecture.
94 
databricks
JUNE 6, 2023
Apache Parquet is one of the most popular open source file formats in the big data world today. Being column-oriented, Apache Parquet allows.
74 This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
The Path to Product Excellence: Avoiding Common Pitfalls and Enhancing Communication
Peak Performance: Continuous Testing & Evaluation of LLM-Based Applications
From Developer Experience to Product Experience: How a Shared Focus Fuels Product Success
Understanding User Needs and Satisfying Them
Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You Need to Know
Towards Data Science
JANUARY 17, 2024
Parquet vs ORC vs Avro vs Delta Lake Photo by Viktor Talashuk on Unsplash The big data world is full of various storage systems, heavily influenced by different file formats. You’ll explore four widely used file formats: Parquet , ORC , Avro , and Delta Lake. These will be used for Parquet, Avro, ORC, and Delta Lake.
72 The Path to Product Excellence: Avoiding Common Pitfalls and Enhancing Communication
Peak Performance: Continuous Testing & Evaluation of LLM-Based Applications
From Developer Experience to Product Experience: How a Shared Focus Fuels Product Success
Understanding User Needs and Satisfying Them
Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You Need to Know
Christophe Blefari
MARCH 22, 2024
On my side I'll talk about Apache Superset and what you can do to build a complete application with it. Finally, xAI released Grok-1 in open — The weights are available in torrent / HF and everything is under Apache License. Now give me the news. Sometimes it looks like you speak to a child.
130 
Christophe Blefari
FEBRUARY 3, 2023
Microsoft Azure announced managed Airflow — Starting this week you'll be able to launch Apache Airflow within Azure Data Factory. Parquet best practices: the art of filtering — How to leverage Parquet filtering to save processing time. The feature is in public preview.
130 
Snowflake
DECEMBER 4, 2023
Apache Iceberg’s ecosystem of diverse adopters, contributors and commercial support continues to grow, establishing itself as the industry standard table format for an open data lakehouse architecture. Your other engines that may be writing to the table, such as Apache Spark or Apache Flink, can continue to write, and Snowflake can read.
85 
Netflix Tech
MARCH 7, 2024
Data: Fast Data Our main data lake is hosted on S3, organized as Apache Iceberg tables. For ETL and other heavy lifting of data, we mainly rely on Apache Spark. We use Apache Arrow to decode Parquet and to host an in-memory representation of data.
90 
Data Engineering Podcast
NOVEMBER 22, 2017
In this episode Doug Cutting, creator of Avro, and Julien Le Dem, creator of Parquet, dig into the different classes of serialization formats, what their strengths are, and how to choose one for your workload. You’ve each developed a new on-disk data format, Avro and Parquet respectively.
100 
Cloudera
JULY 14, 2023
Therefore, Apache Iceberg table format is poised to replace the traditional Hive table format in the coming years. They simply read the underlying data (not even full read, they just read the parquet headers) and create corresponding Iceberg metadata files. You can change the data format say from “orc” to “parquet.’’
69 
Cloudera
JANUARY 15, 2021
Cloudera Data Warehouse is a highly scalable service that marries the SQL engine technologies of Apache Impala and Apache Hive with cloud-native features to deliver best-in-class price-performance for users running data warehousing workloads in the cloud. CDW supports running queries on either Apache Hive or Apache Impala engines.
Data Engineering Podcast
AUGUST 3, 2021
Your host is Tobias Macey and today I’m interviewing Vinoth Chandar about Apache Hudi, a data lake management layer for supporting fast and incremental updates to your tables. Go to dataengineeringpodcast.com/census today to get a free 14-day trial. Interview Introduction How did you get involved in the area of data management?
130 
Cloudera
FEBRUARY 14, 2024
Introduction Apache Iceberg has recently grown in popularity because it adds data warehouse-like capabilities to your data lake making it easier to analyze all your data — structured and unstructured. File formats like Parquet work better if the underlying file size is large. Opening files is costly.
56 
Data Engineering Podcast
JANUARY 30, 2022
Prophecy provides an easy-to-use visual interface to design & deploy data pipelines on Apache Spark & Apache Airflow. Prophecy provides an easy-to-use visual interface to design & deploy data pipelines on Apache Spark & Apache Airflow. How does it work? How does it work?
100 
Data Engineering Podcast
MAY 22, 2022
That’s why Kyligence has built on top of the leading open source OLAP engine for data lakes, Apache Kylin. Kubeflow Airflow AWS Step Functions Protocol Buffers XGBoost MLFlow Dagster Podcast Episode Prefect Podcast Episode Arrow Parquet Metaflow Pytorch Podcast.__init__ __init__ Episode dbt FastAPI Podcast.__init__
130 
Big Data Tools
NOVEMBER 8, 2021
Apache Spark® has been released and there are a load of changes, including ANSI SQL support, Pandas API layer over PySpark, and lots and lots of other things. Apache Ranger 2.2.0 If you are curious about what Apache Ranger is – it’s the framework set up to maintain security over the whole Hadoop platform. Parquet support?”
Big Data Tools
NOVEMBER 8, 2021
Apache Spark® has been released and there are a load of changes, including ANSI SQL support, Pandas API layer over PySpark, and lots and lots of other things. Apache Ranger 2.2.0 If you are curious about what Apache Ranger is – it’s the framework set up to maintain security over the whole Hadoop platform. Parquet support?”
Data Engineering Podcast
DECEMBER 30, 2019
__init__ Episode Apache NiFi Podcast Episode Luigi Dagster Podcast Episode Prefect The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA Support Data Engineering Podcast Summary DataDog is one of the most successful companies in the space of metrics and monitoring for servers and cloud infrastructure.
100 
Cloudera
JUNE 18, 2022
An open lakehouse, and the birth of Apache Iceberg. Apache Iceberg was built from inception with the goal to be easily interoperable across multiple analytic engines and at a cloud-native scale. The cloud native table format was open sourced into Apache Iceberg by its creators. Apache Iceberg’s real superpower is its community.
89 
Data Engineering Podcast
APRIL 26, 2021
__init__ Episode Preset ASP (Active Server Pages) VBScript Data Warehouse Institute Ralph Kimball Bill Inmon Ubisoft Hadoop Tableau Looker Podcast Episode The Future of Business Intelligence Is Open Source Supercharging Apache Superset Redash Podcast.__init__
Uber Engineering
MARCH 12, 2017
With the evolution of storage formats like Apache Parquet and Apache ORC and query engines like Presto and Apache Impala , the Hadoop ecosystem has the potential to become a general-purpose, unified serving layer for workloads that can tolerate latencies … The post Hudi: Uber Engineering’s Incremental Processing Framework on Apache Hadoop appeared (..)
105 
Big Data Tools
JUNE 8, 2022
I’ve had some experience with Apache Atlas, and even with the help of my colleagues, I wasn’t able to make it do what I wanted it to. This new release brings exciting features like support for Apache Iceberg! Apache Pulsar takes a step in this direction and adds an official management UI! There are several solutions.
Big Data Tools
JUNE 8, 2022
I’ve had some experience with Apache Atlas, and even with the help of my colleagues, I wasn’t able to make it do what I wanted it to. This new release brings exciting features like support for Apache Iceberg! Apache Pulsar takes a step in this direction and adds an official management UI! There are several solutions.
Data Engineering Podcast
JANUARY 6, 2019
__init__ Episode Cloudera Manager Apache Sentry Collibra The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA Support Data Engineering Podcast Summary The Hadoop platform is purpose built for processing large, slow moving data in long-running batch jobs.
100 
Data Engineering Podcast
DECEMBER 8, 2019
Links SnowflakeDB Free Trial Stack Overflow Data Warehouse Oracle DB MPP == Massively Parallel Processing Shared Nothing Architecture Multi-Cluster Shared Data Architecture Google BigQuery AWS Redshift AWS Redshift Spectrum Presto Podcast Episode SnowflakeDB Semi-Structured Data Types Hive ACID == Atomicity, Consistency, Isolation, Durability 3rd Normal (..)
100 
Cloudera
JUNE 2, 2021
For example, all common databases (redshift, snowflake, mongo, hbase, … ), service integrations (salesforce) and popular file formats (avro, parquet, ORC, csv…) and dozens more listed here. Integrate with a wide swathe of 3rd party data sources with Spark to provide an extensive library that can be leveraged in CDE.
Data Engineering Podcast
OCTOBER 9, 2022
To help other people find the show please leave a review on Apple Podcasts and tell your friends and co-workers Links Iomete Fivetran Podcast Episode Airbyte Podcast Episode Snowflake Podcast Episode Databricks Collibra Podcast Episode Talend Parquet Trino Spark Presto Snowpark Iceberg Podcast Episode Iomete dbt adapter Singer Meltano Podcast Episode (..)
100 
Cloudera
AUGUST 8, 2022
In June 2022, Cloudera announced the general availability of Apache Iceberg in the Cloudera Data Platform (CDP). Iceberg is a 100% open-table format, developed through the Apache Software Foundation , which helps users avoid vendor lock-in and implement an open lakehouse. . Fine-grained access control by SDX integration (Ranger) .
Pipeline Data Engineering
NOVEMBER 4, 2021
ROAPI: An API Server for Static Datasets Mark Litwintschik, #bigdata Consultant ROAPI is an API Server that exposes CSV, JSON and Parquet files without the need to write any code. SeattleDataGuy There’s no better time to jump into the world of data engineering. Announcing Streamlit 1.0! ?
52 
Knowledge Hut
DECEMBER 21, 2023
HDFS HDFS is the abbreviated form of Hadoop Distributed File System and is a component of Apache Hadoop. Mahout Overview: Apache Mahout is an open-source ML library that helps leverage big data computation through Hadoop MapReduce. The outcome derived from Apache Pig is stored in HDFS. Pros: Apache Pig is very easy to learn.
52 
Data Engineering Podcast
SEPTEMBER 7, 2020
__init__ Episode Tensorflow Spark The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA Support Data Engineering Podcast Summary Databases are limited in scope to the information that they directly contain.
100 
Data Engineering Podcast
NOVEMBER 4, 2019
To help other people find the show please leave a review on iTunes and tell your friends and co-workers Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat Links Ascend Kubernetes BigQuery Apache Spark Apache Beam Go Language SHA Hashes PySpark Delta Lake Podcast Episode DAG == Directed Acyclic Graph PrestoDB MinIO (..)
Data Engineering Podcast
DECEMBER 16, 2019
To help other people find the show please leave a review on iTunes and tell your friends and co-workers Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat Links Marquez DataEngConf Presentation WeWork Canary Yahoo Dremio Hadoop Pig Parquet Podcast Episode Airflow Apache Atlas Amundsen Podcast Episode Uber DataBook (..)
100 
Big Data Tools
NOVEMBER 9, 2022
Apache Doris 1.1.3 – Here’s another interesting database for you. We aren’t aware of many MPP databases, and none of them are under the motley umbrella of the Apache Software Foundation. release supports Apache Parquet as an output file format. Here’s what’s happening in the world of data engineering right now.
Big Data Tools
NOVEMBER 9, 2022
Apache Doris 1.1.3 – Here’s another interesting database for you. We aren’t aware of many MPP databases, and none of them are under the motley umbrella of the Apache Software Foundation. release supports Apache Parquet as an output file format. Here’s what’s happening in the world of data engineering right now.
Snowflake
MARCH 21, 2023
Streaming Data Ingestion Snowpipe Streaming, Now in Public Preview Ingest rowsets from business application and IoT devices or from Apache Kafka topics directly into Snowflake at low latency. Check out Felipe Hoffa’s video on how to use Snowsight to get from data to decision faster.
64 
Data Engineering Podcast
OCTOBER 9, 2018
Petersburg University of Fine Mechanics And Optics C C++ In-Memory Database RAM (Random Access Memory) Flash Storage Oracle DB PostgreSQL Podcast Episode Kafka Kinesis Wealth Management Data Warehouse ODBC S3 HDFS Avro Parquet Data Serialization Podcast Episode Broadcast Join Shuffle Join CAP Theorem Apache Arrow LZ4 S2 Geospatial Library Sybase SAP (..)
100 
Monte Carlo
JANUARY 5, 2024
At this layer, an organization might use tools like Amazon Data Migration Service ( Amazon DMS ) for importing data from RDBMSs and NoSQL databases, Apache Kafka for data streaming, and many more. Metadata layer The metadata layer manages and organizes the metadata associated with the data that’s been ingested and stored.
52 
Monte Carlo
JANUARY 5, 2024
At this layer, an organization might use tools like Amazon Data Migration Service ( Amazon DMS ) for importing data from RDBMSs and NoSQL databases, Apache Kafka for data streaming, and many more. Metadata layer The metadata layer manages and organizes the metadata associated with the data that’s been ingested and stored.
52 
Big Data Tools
OCTOBER 10, 2022
Apache Pegasus 2.3.0 – Have you ever been in a situation where you were designing a storage architecture and all the solutions in some areas just seemed wrong, leaving you to choose between an unsuitable option and an even less suitable one? Apache Pegasus might be the alternative you are looking for, if not now, then in your next project.
Big Data Tools
OCTOBER 10, 2022
Apache Pegasus 2.3.0 – Have you ever been in a situation where you were designing a storage architecture and all the solutions in some areas just seemed wrong, leaving you to choose between an unsuitable option and an even less suitable one? Apache Pegasus might be the alternative you are looking for, if not now, then in your next project.
Cloudera
SEPTEMBER 24, 2020
Some of the most powerful results come from combining complementary superpowers, and the “dynamic duo” of Apache Hive LLAP and Apache Impala, both included in Cloudera Data Warehouse , is further evidence of this. Uses Parquet as the preferred file format. Aren’t two superheroes better than one?
Monte Carlo
JUNE 28, 2022
Over the past decade, Databricks and Apache Spark™ not only revolutionized how organizations store and process their data, but they also expanded what’s possible for data teams by operationalizing data lakes at an unprecedented scale across nearly infinite use cases. Complete data lake coverage no matter the metastore.
52 
Data Engineering Weekly
OCTOBER 30, 2022
DuckDB is gaining much attention on this promise, and the Dagster team writes about its experimental data warehouse built on top of DuckDB, Parquet, and Dagster. link] Bart Maertens: A new Execution Information Logging Platform in Apache Hop Profiling and auditing the workflow is essential for operating the data pipeline at scale.
Ascend.io
SEPTEMBER 14, 2023
Strong especially in big data, with tools like Apache Spark. For those venturing into data lakes and distributed storage, tools like Hadoop’s Pydoop and PyArrow for Parquet ensure that Python isn’t left behind. PySpark allows Python to interface with Apache Spark, making distributed data tasks more approachable.
Expert insights. Personalized for you.
We have resent the email to
Are you sure you want to cancel your subscriptions?
Let's personalize your content