Data Engineering Weekly

MARCH 17, 2024

India recently announced that all AI apps require government approval and dropped the plan later. link] Meta: Building Meta’s GenAI Infrastructure The article discusses building Meta's GenAI infrastructure, a system designed to train large AI models like Llama 3 efficiently.

Data Engineering 70

Data Engineering Data Engineer Engineering Data Governance 70

Christophe Blefari

FEBRUARY 23, 2024

— Apache Arrow is an awesome library that powers a lot of innovations in the data space in the recent years. — An article I wrote 4 days ago as an answer to the trend. A newsletter about the streaming data space — Robin collected a lot of cool articles about the streaming ecosystem.

Data Lake 130

Data Lake PostgreSQL MongoDB Scala 130

Christophe Blefari

JUNE 16, 2023

From databases introduction to SQL writing. Fast News contains lighter news and broad articles. Malloy's Near Term Roadmap — I've shared recently Malloy demo , which was awesome. Experimenting at Scale, the Spotify Home way — Simple principles to run a good old' experiment at Spotify scale.

Programming Language 130

Programming Language SQL PostgreSQL Data 130

Christophe Blefari

APRIL 21, 2023

This article gives you an overview of the Kubernetes platform they implemented. This is a great article and they even included a flowchart to identify which role will suit you the most. Test Driven Development for SQL — A smal article that gives you a vanilla BigQuery framework with CTE to write unit tests.

Raw Data 130

Raw Data Data Datasets SQL 130

Christophe Blefari

MARCH 24, 2023

It seems that you really liked the recent editions, which is perfect because it was fun to write. I feel that this week all the articles I found relevant for the newsletter are either AI related or technical. Apache Arrow releases Arrow nanoarrow — Recently Arrow got a lot of light because of DuckDB or Pandas 2.0

Lambda Architecture 130

Lambda Architecture Data Pipeline Data SQL 130

Christophe Blefari

JANUARY 14, 2023

Also a few of you have sent me messages recently and I've not yet answered, I see you and I did not forget you. In the meantime you can still check my article Using Airflow the wrong way that summarize a bit the operators vs. containers debate. This article shows you what are the key concepts to understand it.

Python 130

Python Kafka Data Scala 130

Knowledge Hut

DECEMBER 22, 2023

There are different types of hacking, including: SQL injection: This type of attack involves injecting malicious code into a website's database in order to extract data or take control of the site. Breaking Bad Breaking Bad is an ethical hacking forum that has been gaining popularity in recent years. Exploit.in

Education 98

Education Database Media Certification 98

Christophe Blefari

APRIL 28, 2023

As Katia shares in the article a 9-months migration is a long tunnel where you encounter a lot of roadblocks and frustration but in the end everyone feels the difference: 10x performance gain—at least—on dashboard execution time. Recently Benoit joined Kestra as their PO. In this article he shares his vision.

SQL 100

SQL Food PostgreSQL Data 100

Christophe Blefari

FEBRUARY 3, 2024

In the current cycle of hype it can be done with LLMs, like DuckDB-NSQL-7B , the one MotherDuck provided recently. Even if you give your LLM access to the database, the codebase and the docs there is something the LLM does not have: the implicit (vocal) business rules that are written nowhere. conference. conference.

MongoDB 130

MongoDB SQL Data Data Warehouse 130

The Pragmatic Engineer

MAY 11, 2023

A quick summary of these technologies: Prometheus : a time series database. A fast and open-source column-oriented database management system, which is a popular choice for log management. But could recent layoffs change things? A follow-up to this week’s public tech company compensation article.

AWS 304

AWS Software Engineer Software Engineering Google Cloud 304

Christophe Blefari

MAY 6, 2023

But still you'll find awesome articles that has been written recently. Even Amazon can't make sense of serverless or microservices — PrimeVideo tech team wrote an article that could be summarised by: we migrated from functions based approach to a monolith in a VM. I hope this email finds you well.

Programming Language 100

Programming Language Data Python Data Warehouse 100

Data Engineering Weekly

JUNE 12, 2023

The recent frenzy around large language models (LLM) increases the dilemma around the open source vs. non-commercial use licensing model. The author highlights the recent trend of increasing non-commercial & restrictive licenses. Get the Guide Yingjun Wu: Why You Shouldn’t Invest In Vector Databases?

Data Engineering 57

Data Engineering Data Engineer Engineering AWS 57

Data Engineering Weekly

JUNE 25, 2023

The highlight for me from my thinking from the article; ChatGPT can write code and make you a document, but who is responsible for the outcome? link] Ziheng Wang: Open Vector Data Lakes We’ve seen the rise of vector databases in the market. The author thinks not and lists thought-provoking points about how these interfaces work.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Lake 52

Knowledge Hut

APRIL 25, 2024

This article will discuss big data analytics technologies, technologies used in big data, and new big data technologies. NoSQL databases are designed for scalability and flexibility, making them well-suited for storing big data. The most popular NoSQL database systems include MongoDB, Cassandra, and HBase.

Big Data 98

Big Data Technology NoSQL Hadoop 98

Knowledge Hut

DECEMBER 25, 2023

A person interested in becoming a Full-stack developer needs to have proficiency in both the front-end and back-end tools, just like I started as a Front-end developer and later on become a Full stack developer by mastering JavaScript backend technologies and databases.  In the backend, we have a lot of technologies and databases also.

Java 98

Java Database PostgreSQL Project 98

Knowledge Hut

MARCH 26, 2024

ChatGPT Prompts for Resumes Here are 5 best ChatGPT prompts for writing resumes: Write an impactful opening statement for a recent college graduate seeking an entry-level position in finance. Craft a witty and engaging response to a positive comment on a recent post. ChatGPT Prompts [For All Jobs] 1.

Media 52

Media Finance Medical Healthcare 52

Ascend.io

FEBRUARY 23, 2024

This means that you are fully aware of the recent developments in AI and Large Language Models ( LLMs ). In this article, we outline the essential prerequisites for an AI data platform. You are a data professional. You’ve identified several complex challenges within your business that AI seems tailor-made to tackle.

Cloud Storage 52

Cloud Storage Data Ingestion Machine Learning Algorithm 52

Knowledge Hut

DECEMBER 26, 2023

In this article, we will look at the top 10 professional courses in the world that you can apply to get your dream job. You will be able to gain a thorough understanding of how to deploy, create, secure, and scale programs, as well as become familiar with the user interface, database stacks, and business logic.

Medical 98

Medical Cloud Computing Finance Data Science 98

Meltano

MARCH 30, 2023

This article will introduce you to isolated test environments and their benefits, and show you how to implement an isolated test environment with Meltano and Snowflake. Additionally, these environments are used to isolate test users so that they don’t interfere with mission-critical systems like mail servers and financial record databases.

Metadata 59

Metadata Database Python Coding 59

Scott Logic

FEBRUARY 26, 2023

If this is a question that crossed your mind recently, then this article is for you. I’m hoping to share you my experience of writing integration tests to test your repository layer with the use of Testcontainers in this article. I have a PostgreSQL database in my production, and now you are asking me to test with a H2?

MongoDB 75

MongoDB PostgreSQL Consulting Java 75

Knowledge Hut

MAY 10, 2024

Django supports four main databases (PostgreSQL, MariaDB, MySQL, Oracle and SQLite) and community libraries support other popular SQL and NoSQL databases at various levels. Migration applies any changes to your database scheme that you have made to your Django models. To migrate the database use: (my_env) $ python manage.py

Python 52

Python PostgreSQL Database NoSQL 52

Knowledge Hut

MARCH 28, 2024

How would a person who recently secured an internship as a web developer spend their regular day? In this article, I will give you the most important advice on How to how a Web Development internship. Now, let’s go on to the most important part of this article: How do you get a web development internship?

Portfolio 52

Portfolio Recruitment Education Coding Skills 52

Christophe Blefari

APRIL 8, 2023

In the recent years dbt simplified and revolutionised the tooling to create data models. This week I discovered SQLMesh , a all-in-one data pipelines tool. ❤️ Reducing the lottery factor, for data teams — if you had to read only one article today you should read this one. But alternatives are coming. seed round.

Pipeline-centric 130

Pipeline-centric Database-centric Algorithm Data 130

Christophe Blefari

APRIL 8, 2023

In the recent years dbt simplified and revolutionised the tooling to create data models. This week I discovered SQLMesh , a all-in-one data pipelines tool. ❤️ Reducing the lottery factor, for data teams — if you had to read only one article today you should read this one. But alternatives are coming. seed round.

Pipeline-centric 130

Pipeline-centric Database-centric Algorithm Data 130

Knowledge Hut

DECEMBER 26, 2023

In this article, you will find the Top 10 tech trends in 2024. We have listed some of the most-recent new technology breakthroughs to keep an eye on. Blockchain is a distributed database that allows transactions to be safe and transparent without any central authority.

Technology 98

Technology Cloud Computing Healthcare Data Science 98

Data Engineering Weekly

JUNE 4, 2023

link] LakeFs: How to Implement Write-Audit-Publish (WAP) I wrote extensively about the WAP pattern in my latest article, An Engineering Guide to Data Quality - A Data Contract Perspective. The author writes an exciting article about adopting unit testing in the data pipeline by producing sample tables during the development.

Data Engineering 52

Data Engineering Data Engineer Engineering Medical 52

Monte Carlo

DECEMBER 4, 2023

Extraction ChatGPT ETL prompts can be used to help write scripts to extract data from different sources, including: Databases I have a SQL database with a table named employees. Can you help me craft a Python script that uses the tweepy library to get the most recent 100 tweets from a given user handle?

PostgreSQL 52

PostgreSQL Data Lake ETL Tools MySQL 52

Data Engineering Podcast

AUGUST 28, 2022

With their new managed database service you can launch a production ready MySQL, Postgres, or MongoDB cluster in minutes, with automated backups, 40 Gbps connections from your application hosts, and high throughput SSDs. What are some of the tasks that you have found to be common problems in that sense?

Data Engineering 100

Data Engineering Data Engineer MongoDB Metadata 100

Knowledge Hut

JANUARY 23, 2024

MongoDB is a popular NoSQL database that is open-source and document-oriented. 'NoSQL' 'NoSQL' here implies that it is a non-relational database, i.e., it stores the data in a different format other than the relational tables and therefore does not require a fixed schema. mongodb-org-database=6.0.1

MongoDB 52

MongoDB NoSQL Non-relational Database Database 52

Data Engineering Weekly

JULY 2, 2023

Watch the Atlan AI demo and join the Atlan AI waitlist → Ranjay Kumar: The Significance of First and Last Day of the Month in Data Engineering The article gave me a very interesting perspective, and I thought about how the oncall structure should be extra cautious during these business milestones.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Monte Carlo

MARCH 7, 2024

In this article, we’ll look at prompt engineering, what it is, and why it’s valuable—as well as a few other genAI skills that should be at the top of your list. Understand vector databases The infrastructure and technical roadmap for AI tooling is being developed as we speak, with new startups emerging every day to tackle various problems.

Engineering 59

Engineering High Quality Data Database Architecture 59

Knowledge Hut

MAY 3, 2024

The comparison between these two has been in debate in recent times. In this article, we will understand the differences between PHP and ASP.Net also, will discuss which is better ASP.NET or PHP. Both of these languages are used in large web-based applications.

Programming Language 52

Programming Language Certification Programming Coding 52

Data Engineering Weekly

NOVEMBER 26, 2023

The data industry clearly understands the power of blog storage, and using S3 as a database is not a new concept either. We saw back-to-back articles comparing the features and the performance of Delta Lake, Apache Hudi, and Iceberg. Snowflake recently acquired Ponder to bring some alternatives to Apache Spark.

Data Engineering 70

Data Engineering Data Engineer Engineering Kafka 70

Knowledge Hut

OCTOBER 29, 2023

Hospital Management System Type : Application development, Database management, Programming There is no shortage of computer science project topics out there. Creating a database to store patient information. As a result, VPNs have become increasingly popular in recent years. Programming a system to track medical appointments.

Computer Science 111

Computer Science Coding Project Hospitality 111

Monte Carlo

JANUARY 5, 2024

In this article, we’ll peel back the 5 layers that make up data lakehouse architecture: data ingestion, data storage, metadata, API, and data consumption, understand the expanded opportunities a data lakehouse opens up for generative AI, and how to maintain data quality throughout the pipeline with data observability.

Architecture 52

Architecture Data Lake Metadata Unstructured Data 52

Monte Carlo

JANUARY 5, 2024

In this article, we’ll peel back the 5 layers that make up data lakehouse architecture: data ingestion, data storage, metadata, API, and data consumption, understand the expanded opportunities a data lakehouse opens up for generative AI, and how to maintain data quality throughout the pipeline with data observability.

Architecture 52

Architecture Data Lake Metadata Unstructured Data 52

Big Data Tools

NOVEMBER 9, 2022

I’ll offer my impressions of recent developments in the data engineering space and highlight new ideas from the wider community. Apache Doris 1.1.3 – Here’s another interesting database for you. We aren’t aware of many MPP databases, and none of them are under the motley umbrella of the Apache Software Foundation.

Data Engineering 52

Data Engineering Data Engineer Engineering Big Data Tools 52