Data Engineering Weekly

MAY 5, 2024

The motivation for Machine Unlearning is critical from the privacy perspective and for model correction, fixing outdated knowledge, and access revocation of the training dataset. A key thought-provoking moment for me while reading the article is this quote.

Data Engineering 67

Data Engineering Data Engineer Engineering Google Cloud 67

Data Engineering Weekly

APRIL 7, 2024

dbt: 2024 State of Analytics Engineering The 2024 dbt’s state of analytical engineering report is out. What will the future of software engineers be? I strongly believe the concept of Data Product will play a bigger role in data engineering. Data Mesh continuously gaining popularity among the enterprises.

Data Engineering 70

Data Engineering Data Engineer Engineering Unstructured Data 70

Snowflake

APRIL 17, 2024

Yet while SQL applications have long served as the gateway to access and manage data, Python has become the language of choice for most data teams, creating a disconnect. By general availability, we aim to achieve feature parity, empowering you to complete 100% of your data engineering tasks entirely through Python.

Data Pipeline 107

Data Pipeline Python Data Engineering Data Engineer 107

Data Engineering Weekly

APRIL 14, 2024

link] Github: 4 ways GitHub engineers use GitHub Copilot The impact of LLM on software development is undeniable. Github shares some insights on how Github engineers use Github Copilot. It can drastically reduce the computational cost and energy requirement for training LLM, and it is an interesting development to watch.

Data Engineering 62

Data Engineering Data Engineer Engineering Business Intelligence 62

Knowledge Hut

MARCH 14, 2024

The process of disassembling a product, system, or program into its constituent parts, comprehending how they function, and then utilizing that knowledge to either reproduce or enhance the original is known as reverse engineering. I will be doing a detailed discussion of reverse engineering in software engineering in this article.

Software Engineer 52

Software Engineer Software Engineering Engineering Electronics 52

Netflix Tech

MARCH 10, 2021

ConsoleMe: A Central Control Plane for AWS Permissions and Access By Curtis Castrapel , Patrick Sanders , and Hee Won Kim At AWS re:Invent 2020, we open sourced two new tools for managing multi-account AWS permissions and access. Groups beyond software engineering teams are standing up their own systems and automation.

AWS 100

AWS Accessible Accessibility Cloud 100

Knowledge Hut

OCTOBER 27, 2023

In today's fast-paced technological environment, software engineers are continually seeking innovative projects to hone their skills and stay ahead of industry trends. Engaging in software engineering projects not only helps sharpen your programming abilities but also enhances your professional portfolio. cvtColor(image, cv2.COLOR_BGR2GRAY)

Software Engineer 98

Software Engineer Software Engineering Coding Project 98

Knowledge Hut

APRIL 24, 2024

In today's fast-paced technological environment, software engineers are continually seeking innovative projects to hone their skills and stay ahead of industry trends. Engaging in software engineering projects not only helps sharpen your programming abilities but also enhances your professional portfolio. cvtColor(image, cv2.COLOR_BGR2GRAY)

Software Engineer 52

Software Engineer Software Engineering Coding Project 52

Data Engineering Weekly

MARCH 3, 2024

There will be food, networking, and real-world talks around data engineering. This approach led to a successful expansion of Copilot access across the engineering team, resulting in a significant increase in productivity and adoption, demonstrating a commitment to enhancing developer experience while maintaining safety and security standards.

Data Engineering 67

Data Engineering Data Engineer Pipeline-centric Engineering 67

Data Engineering Weekly

MARCH 24, 2024

link] Meta: Logarithm - A logging engine for AI training workflows and services Logarithm indexes 100+GB/s of logs in real-time and thousands of queries a second!!! The logging engine to debug AI workflow logs is an excellent system design study if you’re interested in it.

Data Engineering 62

Data Engineering Data Engineer Engineering Metadata 62

Data Engineering Weekly

DECEMBER 3, 2023

link] Microsoft: Generative AI for Beginners Understanding Gen-AI becomes a mandatory skill for application developers and data engineers. link] Netflix: Incremental Processing using Netflix Maestro and Apache Iceberg Netflix writes about its incremental processing design with its orchestration engine Maestro on top of Iceberg.

Data Engineering 70

Data Engineering Data Engineer Engineering Bytes 70

Knowledge Hut

DECEMBER 27, 2023

What will be the key skills for a software engineer in 2024? The top software developers stand out from the crowd thanks to their extensive technical skills as software engineers. Thus, to become a highly sought-after software developer, you must increase and update the technical skills required for software engineering.

Software Engineer 98

Software Engineer Software Engineering Engineering Programming Language 98

RandomTrees

FEBRUARY 2, 2024

GPT-based data engineering accelerators make the working of data more accessible. They use intelligent language interfaces and make data accessible to more people. DataGPT OpenAI developed DataGpt for performing data engineering tasks. These accelerators combine information from different sources.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Pipeline 52

The Pragmatic Engineer

OCTOBER 10, 2022

Part of this article was originally published in The Scoop #27 , for subscribers of The Pragmatic Engineer Newsletter last week. The Business Insider article was not specific to software engineers but still spread heavily within tech circles. I talked with engineering managers at Meta to find out how much truth they see in this claim.

Engineering 130

Engineering Software Engineer Software Engineering Recruitment 130

Edureka

APRIL 19, 2024

Fundamentally, social engineering is not the same as a cyberattack. Rather, social engineering relies heavily on the psychology of convincing, attacking the mind like a classic with artists. Even though recovering from a social engineering attack is much more difficult, preventing one can have far less disastrous effects.

Engineering 52

Engineering Media Banking Education 52

databricks

SEPTEMBER 26, 2023

My summer internship on the PySpark team was a whirlwind of exciting events. The PySpark team develops the Python APIs of the open.

Accessible 75

Accessible Accessibility Python Engineering 75

Snowflake

FEBRUARY 16, 2023

This is the core problem Bright Data, a market leading public web data collection platform , is trying to solve: how can every organization in the world access publicly available web data at scale and speed? In the past we used to have to tie up engineering resources to consider if we could meet potentially demanding SLAs.

Accessible 52

Accessible Accessibility BI Data 52

Data Engineering Weekly

MARCH 10, 2024

Intuit advocate starting with existing models and enhancing them through prompt engineering or training on domain-specific knowledge before considering custom LLM development. link] Richard Crowley: SOC 2 compliance for startups and first-timers While SOC 2 might seem outside a data engineer's wheelhouse, understanding its principles is key.

Data Engineering 52

Data Engineering Data Engineer Engineering Datasets 52

Data Engineering Podcast

JANUARY 30, 2022

As a result it has become a standard tool for data engineers for a wide range of applications. Today’s episode is Sponsored by Prophecy.io – the low-code data engineering platform for the cloud. What are the main tasks that you have seen Pandas used for in a data engineering context?

Data Engineering 100

Data Engineering Data Engineer Engineering Data Pipeline 100

Cloudera

SEPTEMBER 15, 2020

introduces fine-grained authorization for access to Azure Data Lake Storage using Apache Ranger policies. Cloudera and Microsoft have been working together closely on this integration, which greatly simplifies the security administration of access to ADLS-Gen2 cloud storage. Use case #1: authorize users to access their home directory.

Accessible 75

Accessible Accessibility Cloud Cloud Storage 75

Knowledge Hut

MARCH 22, 2024

From my journey as a software engineer , one of the most intriguing concepts I've encountered is reverse engineering. When we are learning new concepts, reverse engineering projects can help us understand the inner workings of something. What is Reverse Engineering in Software Engineering?

Process 52

Process Engineering Electronics Software Engineer 52

Knowledge Hut

MARCH 20, 2024

Because of the surge in cyber-attacks, there is a massive demand for certified network security engineers. Thus, if you're interested in starting a career in cyber security or if you've already started one and want to know more about network security engineer salary and other pertinent information, read the blog through to the end.

Engineering 52

Engineering Banking Amazon Web Services Certification 52

Edureka

APRIL 19, 2024

In other words, you don’t need to be a programmer or have any coding background to dive into this creative process of building a chatbot using prompt engineering. With such platforms, the process becomes accessible to everyone, empowering users to bring to life digital avatars of well-known personalities, celebrities, or influencers.

Building 52

Building Engineering Coding Systems 52

Towards Data Science

DECEMBER 1, 2023

Pet Project for Data/Analytics Engineers: Explore Modern Data Stack Tools — dbt Core, Snowflake, Fivetran, GitHub Actions. Photo by Gaining Visuals on Unsplash Here is a simple and fast pet project for Data/Analytics Engineers, who want to kick the tires on Modern Data Stack tools including dbt Core, Snowflake, Fivetran, and GitHub Actions.

Data Engineering 73

Data Engineering Data Engineer Project Engineering 73

Knowledge Hut

DECEMBER 26, 2023

Becoming a software engineer is the dream of many and an aspiring career option for most students today. The path of becoming a software engineer is not an easy one. But what about the future of software engineering? In 2024, the demand for software engineers rose by nearly 17% across all industries, as per Opportunity Desk.

Software Engineer 98

Software Engineer Software Engineering Engineering Programming Language 98

Data Engineering Weekly

OCTOBER 1, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. Learn more and reach out for early access on the blog. See how it works today.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Pipeline 52

Data Engineering Weekly

SEPTEMBER 24, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. See how it works today. Search and Discrimination!

Data Engineering 67

Data Engineering Data Engineer Engineering Kafka 67

Cloudera

JULY 13, 2021

After the launch of CDP Data Engineering (CDE) on AWS a few months ago, we are thrilled to announce that CDE, the only cloud-native service purpose built for enterprise data engineers, is now available on Microsoft Azure. . Prerequisites for deploying CDP Data Engineering on Azure can be found here. Easy job deployment.

Data Engineering 125

Data Engineering Data Engineer Engineering Pipeline-centric 125

Knowledge Hut

MARCH 14, 2024

The team of software engineers in Apple plays a pivotal role in the development of innovative software solutions that work with different platforms like iOS, macOS, watchOS, and many more. In this blog, I will explain Apple software engineer salary as per different criteria. Who is a Software Engineer? Apple, Inc.,

Software Engineer 52

Software Engineer Software Engineering Engineering Healthcare 52

Ascend.io

FEBRUARY 28, 2024

The rise of generative AI is changing more than just technology; it’s reshaping our professional landscapes — and yes, data engineering is directly experiencing the impact. How can data engineers harness the power of AI? And crucially, what does the future hold for data engineering in an AI-driven world?

Data Engineering 52

Data Engineering Data Engineer Engineering Metadata 52

Data Engineering Weekly

AUGUST 13, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork, and SQL grunt work out of building complete customer profiles, so you can quickly ship actionable, enriched data to every downstream team. A best-in-class onboarding process is a reflection of inclusive and empathetic engineering culture.

Data Engineering 52

Data Engineering Data Engineer Engineering Food 52

Knowledge Hut

JUNE 26, 2023

Welcome to the world of data engineering, where the power of big data unfolds. If you're aspiring to be a data engineer and seeking to showcase your skills or gain hands-on experience, you've landed in the right spot. Companies are constantly seeking experienced data engineers who can create innovative data engineering initiatives.

Data Engineering 98

Data Engineering Data Engineer Coding Project 98

Ascend.io

SEPTEMBER 14, 2023

The rise of data-intensive operations has positioned data engineering at the core of today’s organizations. As the demand to efficiently collect, process, and store data increases, data engineers have started to rely on Python to meet this escalating demand. Why Python for Data Engineering?

Data Engineering 52

Data Engineering Data Engineer Python Engineering 52

Databand.ai

JUNE 28, 2023

Data Pipeline Observability: A Model For Data Engineers Eitan Chazbani June 29, 2023 Data pipeline observability is your ability to monitor and understand the state of a data pipeline at any time. But unfortunately, very little that happens in data engineering today is observable. Most data pipelines are built to move but not monitor.

Data Pipeline 98

Data Pipeline Data Engineering Data Engineer Engineering 98

Knowledge Hut

MARCH 13, 2024

Are you a software engineer seeking new horizons? The worldwide demand for tech talent has skyrocketed, and finding the best countries for software engineers in 2024 becomes imperative. Besides this, software engineers have an 87% employment rate, meaning that the profession has a lot of leverage to be explored.

Software Engineer 52

Software Engineer Software Engineering Engineering Education 52

Knowledge Hut

MARCH 13, 2024

The search engineer giant provides a platform for many exceptional software engineers beyond problem-solving. As an experienced software engineer with the skills to solve problems I have not encountered before, I have some important tips to share. Who is a Google Software Engineer? What Do Google Software Engineers Do?

Software Engineer 52

Software Engineer Software Engineering Process Engineering 52

Monte Carlo

JUNE 6, 2023

Generative AI is taking the world by storm – here’s what it means for data engineering and why data observability is critical for this groundbreaking technology to succeed. People that are not proficient in SQL and business intelligence will no longer need to ask an analyst or analytics engineer to create a dashboard for them.

Data Engineering 84

Data Engineering Data Engineer Engineering Business Intelligence 84