KDnuggets

FEBRUARY 24, 2023

This article will discuss SQL visualization, its role in augmenting the modern-day data engineer, and five categories of SQL visualization tools.

SQL 93

SQL Data Engineering Data Engineer Engineering 93

Analytics Vidhya

FEBRUARY 27, 2023

So, we are […] The post How to Normalize Relational Databases With SQL Code? If a corrupted, unorganized, or redundant database is used, the results of the analysis may become inconsistent and highly misleading. appeared first on Analytics Vidhya.

Relational Database 255

Relational Database Database SQL Coding 255

Analytics Vidhya

FEBRUARY 28, 2023

SQL is widely used in the field of data science and is considered an essential skill to have if you work with data. After being introduced in the 70s, it has become the standard querying language for relational databases. […] The post Step-by-Step Roadmap to Learn SQL in 2023 appeared first on Analytics Vidhya.

SQL 223

SQL Relational Database Data Science Database 223

Analytics Vidhya

FEBRUARY 2, 2023

Introduction SQL injection is an attack in which a malicious user can insert arbitrary SQL code into a web application’s query, allowing them to gain unauthorized access to a database. It occurs when user input is […] The post SQL Injection: The Cyber Attack Hiding in Your Database appeared first on Analytics Vidhya.

Database 204

Database SQL Coding Accessible 204

Confessions of a Data Guy

MAY 7, 2023

I mean you have a problem if you could use Polars Python, and you find yourself using […] The post Polars – Laziness and SQL Context. It’s gotten so bad, I’ve started to use it in my Rust code on the side, Polars that is. appeared first on Confessions of a Data Guy.

SQL 182

SQL Python Coding IT 182

KDnuggets

FEBRUARY 27, 2023

5 SQL Visualization Tools for Data Engineers • Free TensorFlow 2.0 Complete Course • The Importance of Probability in Data Science • 4 Ways to Rename Pandas Columns • 5 Statistical Paradoxes Data Scientists Should Know

SQL 119

SQL Data Engineering Data Engineer Engineering 119

Confessions of a Data Guy

DECEMBER 21, 2023

Thinking about and contemplating life and data engineering … something flitted across my […] The post Datafusion SQL CLI – Look Ma, I made a new ETL tool. A few days ago I was lying by the river next to a fire, with the cold air blowing on my face and the eagles soaring above.

ETL Tools 113

ETL Tools SQL Data Engineering Data Engineer 113

Christophe Blefari

JANUARY 20, 2024

Learn data engineering, all the references ( credits ) This is a special edition of the Data News. But right now I'm in holidays finishing a hiking week in Corsica 🥾 So I wrote this special edition about: how to learn data engineering in 2024. The idea is to create a living reference about Data Engineering.

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Data Engineering Podcast

NOVEMBER 12, 2023

Generative AI has accelerated the ability of developer tools to provide useful suggestions that speed up the work of engineers. Tabnine is one of the main platforms offering an AI powered assistant for software engineers. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products.

Software Engineer 147

Software Engineer Software Engineering Engineering Data Lake 147

Snowflake

APRIL 17, 2024

Yet while SQL applications have long served as the gateway to access and manage data, Python has become the language of choice for most data teams, creating a disconnect. Our previous Python connector API , primarily available for those who need to run SQL via a Python script, enabled a connection to Snowflake from Python applications.

Data Pipeline 93

Data Pipeline Python Data Engineering Data Engineer 93

Uber Engineering

JANUARY 15, 2020

As Uber’s operations became more complex and we offered additional features and … The post Engineering SQL Support on Apache Pinot at Uber appeared first on Uber Engineering Blog.

SQL 134

SQL Engineering Aggregated Data Data 134

Seattle Data Guy

FEBRUARY 26, 2024

SQL Server Integration Services (SSIS) comes with a lot of functionality useful for extracting, transforming, and loading data. It can also play important roles in application development and other projects. But SSIS is far from the only platform that can provide these services.

SQL 130

SQL Project Data Consulting 130

Confessions of a Data Guy

APRIL 16, 2023

I was wondering the other day … since Polars now has a SQL context and is getting more popular by the day, do I need DuckDB anymore? You might think […] The post DuckDB vs Polars for Data Engineering. These two tools are hot. appeared first on Confessions of a Data Guy.

Data Engineering 147

Data Engineering Data Engineer Engineering SQL 147

ProjectPro

FEBRUARY 16, 2023

The demand for skilled data engineers who can build, maintain, and optimize large data infrastructures does not seem to slow down any sooner. At the heart of these data engineering skills lies SQL that helps data engineers manage and manipulate large amounts of data. Did you know SQL is the top skill listed in 73.4%

Data Engineering 52

Data Engineering Data Engineer SQL Engineering 52

KDnuggets

DECEMBER 6, 2022

SQL is the essential language for developers, engineers, and data professionals. Intermediate knowledge in SQL gives you an edge in your data science career.

SQL 146

SQL Data Science Data Engineering 146

Data Engineering Weekly

APRIL 7, 2024

dbt: 2024 State of Analytics Engineering The 2024 dbt’s state of analytical engineering report is out. What will the future of software engineers be? I strongly believe the concept of Data Product will play a bigger role in data engineering. Data Mesh continuously gaining popularity among the enterprises.

Data Engineering 70

Data Engineering Data Engineer Engineering Unstructured Data 70

Confessions of a Data Guy

DECEMBER 29, 2023

The post SQL Bad, Reddit Mad appeared first on Confessions of a Data Guy.

SQL 100

SQL Data Data Engineering Data Engineer 100

Cloudera

FEBRUARY 21, 2023

SQL Stream Builder (SSB) is a versatile platform for data analytics using SQL as a part of Cloudera Streaming Analytics, built on top of Apache Flink. It enables users to easily write, run, and manage real-time continuous SQL queries on stream data and a smooth user experience. What is a data transformation?

SQL 107

SQL Kafka Raw Data Data 107

DoorDash Engineering

FEBRUARY 27, 2024

Two primary aspects of that search engine were causing the trouble: its document-replication mechanism and its lack of support for complex document relationships. We decided the best way to address these challenges was to move away from Elasticsearch to a homegrown search engine. latency reduction and a 75% hardware cost decrease.

Engineering 129

Engineering Systems Designing Architecture 129

Data Engineering Podcast

JULY 2, 2023

Summary Feature engineering is a crucial aspect of the machine learning workflow. In this episode Razi Raziuddin shares how data engineering teams can support the machine learning workflow through the development and support of systems that empower data scientists and ML engineers to build and maintain their own features.

Machine Learning 130

Machine Learning Data Engineering Data Engineer Engineering 130

Netflix Tech

DECEMBER 14, 2023

Engineers from across the company came together to share best practices on everything from Data Processing Patterns to Building Reliable Data Pipelines. The result was a series of talks which we are now sharing with the rest of the Data Engineering community! Learn more about how batch and streaming data pipelines are built at Netflix.

Data Engineering 103

Data Engineering Data Engineer Engineering Metadata 103

Monte Carlo

JANUARY 23, 2024

In the world of data engineering, Maxime Beauchemin is someone who needs no introduction. One of the first data engineers at Facebook and Airbnb, he wrote and open sourced the wildly popular orchestrator, Apache Airflow , followed shortly thereafter by Apache Superset , a data exploration tool that’s taking the data viz landscape by storm.

Data Engineering 98

Data Engineering Data Engineer Engineering Data 98

Data Engineering Podcast

DECEMBER 22, 2019

To address that need more completely the team at Materialize has created an engine that allows for building queryable views of your data as it is continually updated from the stream of changes being generated by your applications. In the list of features, you highlight full support for ANSI SQL 92.

SQL 100

SQL Building Engineering Java 100

Towards Data Science

NOVEMBER 4, 2023

In this article, I want to talk about crucial things that affect data engineers. I’d like to discuss some popular Data engineering questions: Modern data engineering (DE). The data engineering landscape is constantly changing but major trends seem to remain the same. What is it? Data warehouse exmaple.

Data Engineering 88

Data Engineering Data Engineer Engineering BI 88

Data Engineering Podcast

SEPTEMBER 7, 2020

To address this problem Martin Traverso and his colleagues at Facebook built the Presto distributed query engine. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management What are the pieces of advice that you wish you had received early in your career of data engineering?

Architecture 100

Architecture Data Architecture SQL Engineering 100

Data Engineering Podcast

FEBRUARY 5, 2023

In that time there have been a number of generational shifts in how data engineering is done. Materialize’s PostgreSQL-compatible interface lets users leverage the tools they already use, with unsurpassed simplicity enabled by full ANSI SQL support. Look no further than Materialize, the streaming database you already know how to use.

Data Engineering 147

Data Engineering Data Engineer Engineering PostgreSQL 147

Knowledge Hut

MARCH 12, 2024

SQL databases are one of the most widely used types of database systems available. SQL is a structured query language that these databases enable users to utilize for data management, retrieval, and storage. A number of SQL databases are available. What is SQL? Structured Query Language is what SQL stands for.

SQL 52

SQL Relational Database PostgreSQL MySQL 52

Jesse Anderson

DECEMBER 12, 2022

Hadoop was hard to program, and Apache Hive came along in 2010 to add SQL. Apache Spark came in 2009 and gave a unified batch and streaming engine. Apache Flink came in 2011 and gave us our first real streaming engine. It is prominently data engineering and all of the technical debt difficulties that come with data.

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Knowledge Hut

MARCH 14, 2024

Should that be the case, Azure SQL Database might be your best bet. Microsoft SQL Server's functionalities are fully included in Azure SQL Database, a cloud-based database service that also offers greater flexibility and scalability. In this article, I will cover the various aspects of Azure SQL Database.

Database 52

Database SQL Relational Database BI 52

Data Engineering Podcast

OCTOBER 28, 2021

In this episode Eric Sammer discusses the shortcomings of the current set of streaming engines and how they force engineers to work at an extremely low level of abstraction. Data engineers struggling with unreliable data need look no further than Monte Carlo, the world’s first end-to-end, fully automated Data Observability Platform!

Data Pipeline 100

Data Pipeline SQL Data Warehouse Data Lake 100

Analytics Vidhya

FEBRUARY 7, 2023

Introduction Data engineering is the field of study that deals with the design, construction, deployment, and maintenance of data processing systems. This includes designing and implementing […] The post Most Essential 2023 Interview Questions on Data Engineering appeared first on Analytics Vidhya.

Data Engineering 240

Data Engineering Data Engineer Engineering Data 240

databricks

NOVEMBER 13, 2023

Today, we introduce the new availability of named arguments for SQL functions. With this feature, you can invoke functions in more flexible ways.

SQL 83

SQL Data Engineering Data Engineer Engineering 83

Knowledge Hut

DECEMBER 26, 2023

Data Engineering is typically a software engineering role that focuses deeply on data – namely, data workflows, data pipelines, and the ETL (Extract, Transform, Load) process. According to reports by DICE Insights, the job of a Data Engineer is considered the top job in the technology industry in the third quarter of 2020.

Data Engineering 98

Data Engineering Data Engineer Engineering Pipeline-centric 98

Confluent

NOVEMBER 3, 2021

SQL has proven to be an invaluable asset for most software engineers building software applications. Yet, the world as we know it has changed dramatically since SQL was created in […].

SQL 131

SQL Database Process Software Engineer 131

KDnuggets

MARCH 22, 2023

SQL and Python Interview Questions for Data Analysts • 5 SQL Visualization Tools for Data Engineers • 5 Free Tools For Detecting ChatGPT, GPT3, and GPT2 • Top Free Resources To Learn ChatGPT • Free TensorFlow 2.0

SQL 114

SQL Python Data Engineering Data Engineer 114

Data Engineering Podcast

DECEMBER 24, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Knowledge Hut

DECEMBER 27, 2023

What will be the key skills for a software engineer in 2024? The top software developers stand out from the crowd thanks to their extensive technical skills as software engineers. Thus, to become a highly sought-after software developer, you must increase and update the technical skills required for software engineering.

Software Engineer 98

Software Engineer Software Engineering Engineering Programming Language 98