Christophe Blefari

JULY 3, 2023

In the UI you pick the tables you want the app to run on. Dynamic Tables — Dynamic tables are streaming tables. Dynamic tables has been announced last year, but looks finally in preview. Also you might know Mode through Benn Stancil blog. Hopsworks raises $6.5m — Hopsworks is a feature store.

SQL 130

SQL Data Kafka AWS 130

Cloudera

JANUARY 19, 2024

As described in our recent blog post , an SQL AI Assistant has been integrated into Hue with the capability to leverage the power of large language models (LLMs) for a number of SQL tasks. This blog post aims to help you understand what you can do to get started with generative AI assisted SQL using Hue image version ​​2023.0.16.0

SQL 75

SQL Data Warehouse AWS Architecture 75

Cloudyard

JANUARY 24, 2024

In this ongoing exploration of Streamlit, today’s focus is on dynamically creating a table and populating it with a user-defined number of records. Users have the flexibility to choose the record count using a slider, with a maximum limit of 15,000 records per table.

Consulting 52

Consulting Utilities IT Data 52

Cloudera

AUGUST 8, 2022

Iceberg is a 100% open-table format, developed through the Apache Software Foundation , which helps users avoid vendor lock-in and implement an open lakehouse. . Exploratory data science and visualization: Access Iceberg tables through auto-discovered CDW connection in CML projects.

Data Warehouse 94

Data Warehouse BI Machine Learning SQL 94

DataKitchen

MAY 10, 2024

This blog post explores the challenges and solutions associated with data ingestion monitoring, focusing on the unique capabilities of DataKitchen’s Open Source Data Observability software. I Loaded Data Into A Table, But Only A Partial Load. Or did the schema change? Does The Data Conformity To Predefined Values?

Data Ingestion 40

Data Ingestion Transportation High Quality Data Data Schemas 40

Workfall

MARCH 28, 2023

This blog will demonstrate to you how Hasura and PostgreSQL can help you accelerate app development and easily launch backends. In this blog, we will cover: GraphQL Hasura PostgreSQL Hands-on Conclusion GraphQL GraphQL is an API query language and runtime for answering queries with existing data.

PostgreSQL 75

PostgreSQL Coding MySQL Database 75

Cloudera

JANUARY 6, 2021

In this blog series, we will explain how to configure PySpark and HBase together for basic Spark use as well as for jobs maintained in CDSW. For context, all example operations in this specific blog post are run with a CDSW deployment. 3) Open the Project, go to Settings -> Engine -> Environment Variables. builder. .appName(

Machine Learning 133

Machine Learning Data Science Database Building 133

Cloudera

MARCH 5, 2024

To power our customers’ data, AI, and analytics needs, we are unveiling the next phase of our open data lakehouse , featuring several enhancements built to quickly scale enterprise AI and deliver unprecedented business value. Cloudera is now the only provider to offer an open data lakehouse with Apache Iceberg for cloud and on-premises.

Data Lake 90

Data Lake Data Storage Government Kafka 90

Netflix Tech

NOVEMBER 14, 2023

By Abhinaya Shetty , Bharath Mummadisetty This blog post will cover how Psyberg helps automate the end-to-end catchup of different pipelines, including dimension tables. Solution : One potential approach here would be as follows Create two stateless fact tables : a. Account Plans Create one stateful fact table: a.

Metadata 97

Metadata Data Pipeline Scala Data Workflow 97

Christophe Blefari

DECEMBER 31, 2023

At the same time they released Mixtral 8x7B, the first open-source model of this calibre under an Apache Licence. And the weight are open-source as well. Other reads The state of SQL-based observability , on ClickHouse blog. See you this Friday with a post opening 2024 🎊 Easy GCP cost anomaly detection.

Data 100

Data Cloud Storage Datasets Python 100

Cloudera

FEBRUARY 14, 2024

However, you need to regularly maintain Iceberg tables to keep them in a healthy state so that read queries can perform faster. This blog discusses a few problems that you might encounter with Iceberg tables and offers strategies on how to optimize them in each of those scenarios. Iceberg doesn’t delete the old data files.

Bytes 57

Bytes Metadata Data Lake SQL 57

Cloudyard

JANUARY 16, 2024

Although there isn’t a particular use case driving this exploration, my curiosity led me to develop code for exporting the DDL of both Tables and Stored Procedures. Once Schema is selected it will ask you to select either TABLE or PROCEDURE for whom you want to export the DDL.

Consulting 52

Consulting Database Utilities Coding 52

Data Engineering Weekly

APRIL 7, 2024

The emerging of composable data stack, and open data stack is certainly an interesting trend to watch. The highlight for me is, There is an ongoing table standardization effort at Pinterest to add tiering for the tables. We index only top-tier tables, promoting the use of these higher-quality datasets.

Data Engineering 70

Data Engineering Data Engineer Engineering Unstructured Data 70

Data Engineering Weekly

FEBRUARY 4, 2024

[link] [link] Grab: Rethinking Stream Processing - Data Exploration Grab writes an excellent blog about data exploration on stream processing. The solution centered around Notebook opens a Flink Session for the Kafka stream and continues the exploration. It's an excellent read if you're a snowflake user.

Data Engineering 62

Data Engineering Data Engineer Engineering PostgreSQL 62

Workfall

JULY 18, 2023

In this blog, we’ll whisk you away on an enchanting journey through DBT materializations. In this blog, we will cover: What is DBT? It’s an open-source command-line tool, crafted with love by a community of data enthusiasts. Table Materializations This means storing the transformed data as tables in a database.

Datasets 77

Datasets Entertainment Data Workflow Data 77

LinkedIn Engineering

APRIL 11, 2023

is an open source REST framework built by and heavily used at LinkedIn for microservice development in order to cater to member requests. To continue using Rest.li’s serialization abstraction and retain PDL modeling, we came up with the concept of symbol tables, a two-way mapping from PDL field names/enum values to integer ordinals.

Programming Language 91

Programming Language Java Scala Programming 91

ThoughtSpot

OCTOBER 27, 2023

Enter the following parameters: Host: your Amazon Redshift serverless workgroup (refer to this blog for more detail). Select all tables and all columns except column name start with padb*. Your new connection to Amazon Aurora zero-ETL tables in Amazon Redshift is created. Click Continue. Click Create Connections. Click Create.

Building 70

Building MySQL Data Warehouse SQL 70

Christophe Blefari

JANUARY 20, 2023

Thank you for every recommendation you do about the blog or the Data News. Still open topics, so whatever you want to share—have to be related to Airflow at some point—we'll be happy to welcome you as speaker. by adding ingestion-time partitioned table support for BigQuery.

Google Cloud 130

Google Cloud Data Hadoop Business Intelligence 130

Cloudera

JULY 14, 2023

Introduction For more than a decade now, the Hive table format has been a ubiquitous presence in the big data ecosystem, managing petabytes of data with remarkable efficiency and scale. Therefore, Apache Iceberg table format is poised to replace the traditional Hive table format in the coming years.

Metadata 72

Metadata Data Warehouse Big Data Ecosystem Java 72

ThoughtSpot

OCTOBER 27, 2023

Enter the following parameters: Host: your Amazon Redshift serverless workgroup (refer to this blog for more detail). Select all tables and all columns except column name start with padb*. Your new connection to Amazon Aurora zero-ETL tables in Amazon Redshift is created. Click Continue. Click Create Connections. Click Create.

Building 69

Building MySQL Data Warehouse SQL 69

Cloudera

JUNE 18, 2022

In a rush to own this term, many vendors have lost sight of the fact that the openness of a data architecture is what guarantees its durability and longevity. It provided the concept of a database, schemas, and tables for describing the structure of a data lake in a way that let BI tools traverse the data efficiently.

Data Lake 94

Data Lake Data Warehouse BI SQL 94

Pinterest Engineering

MARCH 20, 2024

In this blog, we touch on: Technical design Key innovations Offline results Online results Technical Design Data Most Pins are associated with a landing page. Table 1: Recall of LinkSage across different serving dimensions. To explore and apply to open roles, visit our Careers page. Higher recall means a better model.

Machine Learning 73

Machine Learning Engineering Software Engineer Software Engineering 73

Lyft Engineering

NOVEMBER 29, 2023

In this particular blog post, we explain how Druid has been used at Lyft and what led us to adopt ClickHouse for our sub-second analytic system. Druid at Lyft Apache Druid is an in-memory, columnar, distributed, open-source data store designed for sub-second queries on real-time and historical data.

Kafka 104

Kafka Data Ingestion Datasets Architecture 104

Cloudera

MAY 1, 2023

In this blog post, we will take a look at how these new concepts and features can help you develop complex Flink SQL projects, manage jobs’ lifecycles, and promote them between different environments in a more robust, traceable and automated manner. These are tables from other projects, or tables that are accessed through a Catalog.

SQL 80

SQL Project Kafka Accessible 80

Cloudera

MARCH 17, 2023

Hopefully this blog will give ChatGPT an opportunity to learn and correct itself while counting towards my 2023 contribution to social good. The one key component that is missing is a common, shared table format, that can be used by all analytic services accessing the lakehouse data. I thought this was a fairly comprehensive list.

Education 62

Education Unstructured Data Data Lake Data Warehouse 62

Cloudera

JULY 18, 2022

In part 1 of this blog we discussed how Cloudera DataFlow for the Public Cloud (CDF-PC), the universal data distribution service powered by Apache NiFi, can make it easy to acquire data from wherever it originates and move it efficiently to make it available to other applications in a streaming fashion. Use case recap.

Process 92

Process Kafka Scala SQL 92

Workfall

APRIL 18, 2023

In this blog, we will discuss: Polling WebSockets GraphQL Subscriptions Apollo Client Features Community Integrations Why Apollo Client? The way HTTP works is that the connection is closed each time the request is finished and another connection is opened when the response is sent from the server to the client. index.tsx(2): App.

Transportation 66

Transportation BI Java Coding 66

Workfall

JULY 4, 2023

In this blog, we will cover: DBT Overview Snowflake Overview DBT With Snowflake Hands-On Understanding DBT Project Structure Creating Your First Model Conclusion DBT Overview In the magical world of DBT, there are three key concepts that bring order and clarity to the realm of data: models, transformations, and tests.

Data Warehouse 69

Data Warehouse SQL PostgreSQL Database 69

Workfall

SEPTEMBER 18, 2023

In this blog, we’re diving into the world of data integration with Airbyte, unraveling the mystery behind its simplicity, and uncovering how it seamlessly connects with Snowflake to transform your data into actionable insights. In this blog, we will cover: What is Airbyte? You can also check the Job History, Logs, etc.

Data Pipeline 93

Data Pipeline Raw Data Data Schemas Healthcare 93

Cloudera

JUNE 7, 2021

Improved Table Management. In addition to quickly connecting to Kafka data sources, users can now use Flink DDL statements to create Tables and Views with full flexibility. By registering a catalog in the streambuilder platform, users get instant access to all the tables and data without having to add them manually.

Kafka 106

Kafka SQL Accessible Accessibility 106

Data Engineering Weekly

MARCH 19, 2023

Now you can win $1,000 cash by contributing a Transformation to our open-source library. The blog discusses implementing Type-2 SCD modeling and strategies to generate surrogate keys and bridge tables to handle many-to-many relationships. Map table vs. using complex data structure? Is chatGPT a data product?

Data Engineering 57

Data Engineering Data Engineer Engineering Media 57

Cloudera

DECEMBER 21, 2020

In this blog we will take you through a persona-based data adventure, with short demos attached, to show you the A-Z data worker workflow expedited and made easier through self-service, seamless integration, and cloud-native technologies. He starts with browsing the available tables in the Data Catalog tool in CDP. Assumptions.

Banking 98

Banking Data Lake Data Data Warehouse 98

Knowledge Hut

DECEMBER 28, 2023

In this blog, we'll explore the top 10 popular SQL tools in the market in 2024. There are numerous open-source SQL tools available, along with paid ones. Adminer Adminer is a free, open-source database management tool that allows users to manage databases using a web interface. What are SQL Tools?

SQL 52

SQL MySQL PostgreSQL Database 52

Edureka

APRIL 19, 2024

The idea is to win targets over to the idea that you are trustworthy so they will let down their defenses and be more likely to engage in risky behavior, like disclosing personal information, opening potentially malicious attachments, or clicking on web links. Our blog post about social engineering in cyber security is now complete.

Engineering 52

Engineering Media Banking Education 52

Cloudera

MAY 11, 2023

During his time living in Latin America in early 2016, Alex saw what he describes as a “knowledge gap” —s eeing the way skills, content and expertise are shared in an open, friendly way at conferences in the US, Alex wanted to replicate that in Latin America. To address this gap, Alex started planning meetups.

Education 90

Education AWS Big Data Technology 90

Data Engineering Weekly

SEPTEMBER 24, 2023

The blog talks about the limitations of rule engines and how LLM can enrich additional context to make the rule engine more effective. The blog specifically focuses on the following areas of LLM implementations. The blog is an excellent read about the “Scale First, Learn Last” problem in the data processing world.

Data Engineering 67

Data Engineering Data Engineer Engineering Kafka 67

Data Engineering Weekly

AUGUST 20, 2023

👋🙂 Register Now → Chip Huyen: Open challenges in LLM research 2023 is where we see the best minds and the money spent on improving LLM. Though the promise of LLM is great, many operational challenges remain open. The author gives an excellent overview of open challenges in LLM as of now.

Data Engineering 52

Data Engineering Data Engineer Engineering Data 52