Pinterest Engineering

JULY 25, 2023

Soam Acharya | Data Engineering Oversight; Keith Regier | Data Privacy Engineering Manager Background Businesses collect many different types of data. Each dataset needs to be securely stored with minimal access granted to ensure they are used appropriately and can easily be located and disposed of when necessary.

Big Data 95

Big Data Accessible Accessibility Hadoop 95

Analytics Vidhya

APRIL 3, 2023

Introduction Companies can access a large pool of data in the modern business environment, and using this data in real-time may produce insightful results that can spur corporate success. Real-time dashboards such as GCP provide strong data visualization and actionable information for decision-makers.

Data Engineering 251

Data Engineering Data Engineer Engineering Data 251

Knowledge Hut

MAY 28, 2024

A software engineer studies, designs, develops, maintains, and retires Software. That’s why in almost every organization, there is a need for a software engineer. And this somehow raises the importance of software engineering today. If you are one to gain knowledge about software engineering, you are in the right place.

Software Engineer 96

Software Engineer Software Engineering Engineering Computer Science 96

Knowledge Hut

APRIL 23, 2024

In cybersecurity, Remote Access Trojans (RATs) have become a major concern for individuals and businesses alike. What is Remote Access Trojan (RAT)? A Remote Access Trojan (RAT) is a stealthy malware that grants unauthorized access and control of a victim's device. So, what exactly is the RAT?

Accessible 52

Accessible Accessibility Banking Programming 52

The Pragmatic Engineer

JANUARY 12, 2023

👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. Here are a few details from the data points, focusing on software engineering compensation. Source: The Pragmatic Engineer blog. Source: The Pragmatic Engineer blog. Software engineering: Software engineering compensation numbers.

Software Engineer 181

Software Engineer Software Engineering Engineering Raw Data 181

Towards Data Science

SEPTEMBER 27, 2023

Data Access API over Data Lake Tables Without the Complexity Build a robust GraphQL API service on top of your S3 data lake files with DuckDB and Go Photo by Joshua Sortino on Unsplash 1. The purpose of this post is to explore and demonstrate a different and simpler approach to tackle this requirement using lighter in-process query engines.

Data Lake 90

Data Lake Accessible Accessibility SQL 90

Snowflake

OCTOBER 19, 2023

I’m inspired by the opportunity to create a more accessible and fair financial system for underserved communities, alleviating the struggle and ultimately boosting economic growth. The majority of Americans struggle to access fair credit—especially those who don’t fit the profile of a typical borrower.

Accessible 86

Accessible Accessibility Banking Data Science 86

Monte Carlo

JANUARY 23, 2024

In the world of data engineering, Maxime Beauchemin is someone who needs no introduction. One of the first data engineers at Facebook and Airbnb, he wrote and open sourced the wildly popular orchestrator, Apache Airflow , followed shortly thereafter by Apache Superset , a data exploration tool that’s taking the data viz landscape by storm.

Data Engineering 98

Data Engineering Data Engineer Engineering Data 98

KDnuggets

JANUARY 26, 2024

Data Engineering ZoomCamp offers free access to reading materials, video tutorials, assignments, homeworks, projects, and workshops.

Data Engineering 146

Data Engineering Data Engineer Engineering Data 146

Data Engineering Podcast

FEBRUARY 5, 2023

In that time there have been a number of generational shifts in how data engineering is done. Sign up now for early access to Materialize and get started with the power of streaming data with the same simplicity and low implementation cost as batch cloud data warehouses.

Data Engineering 147

Data Engineering Data Engineer Engineering PostgreSQL 147

Data Engineering Podcast

NOVEMBER 12, 2023

Generative AI has accelerated the ability of developer tools to provide useful suggestions that speed up the work of engineers. Tabnine is one of the main platforms offering an AI powered assistant for software engineers. Summary Software development involves an interesting balance of creativity and repetition of patterns.

Software Engineer 147

Software Engineer Software Engineering Engineering Data Lake 147

Data Engineering Weekly

APRIL 7, 2024

dbt: 2024 State of Analytics Engineering The 2024 dbt’s state of analytical engineering report is out. What will the future of software engineers be? I strongly believe the concept of Data Product will play a bigger role in data engineering. Data Mesh continuously gaining popularity among the enterprises.

Data Engineering 70

Data Engineering Data Engineer Engineering Unstructured Data 70

Data Engineering Weekly

APRIL 28, 2024

Intuit: The Data Mesh Strategy Behind Intuit’s Global Financial Technology Platform The Data Product Builder platform is becoming increasingly important in enterprise data engineering. link] Teads Engineering: dbt unit-test framework Teads team writes about the dbt unit-test framework in a two-part series.

Data Engineering 52

Data Engineering Data Engineer Engineering Hospitality 52

Towards Data Science

NOVEMBER 4, 2023

In this article, I want to talk about crucial things that affect data engineers. I’d like to discuss some popular Data engineering questions: Modern data engineering (DE). The data engineering landscape is constantly changing but major trends seem to remain the same. What is it?

Data Engineering 82

Data Engineering Data Engineer Engineering BI 82

Snowflake

APRIL 17, 2024

Yet while SQL applications have long served as the gateway to access and manage data, Python has become the language of choice for most data teams, creating a disconnect. By general availability, we aim to achieve feature parity, empowering you to complete 100% of your data engineering tasks entirely through Python.

Data Pipeline 107

Data Pipeline Python Data Engineering Data Engineer 107

Scott Logic

SEPTEMBER 26, 2023

As a test engineer, it can be difficult to appreciate just how many considerations there are for creating software that everyone is able to access. You are able to confirm that all the links on your page are working, but are they as accessible on a smaller screen, or if a person is using their finger or a stylus pen?

Accessible 52

Accessible Accessibility Coding IT 52

Data Engineering Weekly

MAY 5, 2024

The motivation for Machine Unlearning is critical from the privacy perspective and for model correction, fixing outdated knowledge, and access revocation of the training dataset. A key thought-provoking moment for me while reading the article is this quote.

Data Engineering 67

Data Engineering Data Engineer Engineering Google Cloud 67

Data Engineering Weekly

APRIL 14, 2024

link] Github: 4 ways GitHub engineers use GitHub Copilot The impact of LLM on software development is undeniable. Github shares some insights on how Github engineers use Github Copilot. It can drastically reduce the computational cost and energy requirement for training LLM, and it is an interesting development to watch.

Data Engineering 62

Data Engineering Data Engineer Engineering Business Intelligence 62

Data Engineering Podcast

JANUARY 28, 2019

Summary Controlling access to a database is a solved problem… right? It can be straightforward for small teams and a small number of storage engines, but once either or both of those start to scale then things quickly become complex and difficult to manage.

Accessible 100

Accessible Accessibility Database Management 100

Knowledge Hut

MARCH 14, 2024

The process of disassembling a product, system, or program into its constituent parts, comprehending how they function, and then utilizing that knowledge to either reproduce or enhance the original is known as reverse engineering. I will be doing a detailed discussion of reverse engineering in software engineering in this article.

Software Engineer 52

Software Engineer Software Engineering Engineering Electronics 52

Netflix Tech

MARCH 10, 2021

ConsoleMe: A Central Control Plane for AWS Permissions and Access By Curtis Castrapel , Patrick Sanders , and Hee Won Kim At AWS re:Invent 2020, we open sourced two new tools for managing multi-account AWS permissions and access. Groups beyond software engineering teams are standing up their own systems and automation.

AWS 100

AWS Accessible Accessibility Cloud 100

The Pragmatic Engineer

NOVEMBER 24, 2022

👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. I was really hoping to not report anything more about Twitter, and that software engineers at the company would get space to heal after the traumatic events, and to focus on building the product. Most software engineers declined the offer.

Software Engineer 130

Software Engineer Software Engineering Engineering Coding 130

Data Engineering Podcast

MAY 18, 2024

Summary The purpose of business intelligence systems is to allow anyone in the business to access and decode data to help them make informed decisions. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management This episode is supported by Code Comments, an original podcast from Red Hat.

Building 278

Building Data Lake High Quality Data Business Intelligence 278

Knowledge Hut

DECEMBER 27, 2023

Software engineering is quite a vast course that demands you to grasp a lot of knowledge. For this, you must undertake a Complete Software Development course to succeed in any role you choose under software engineering. Also, you must go through certain software engineering books to make your knowledge and skills robust for the job.

Software Engineer 98

Software Engineer Software Engineering Engineering Computer Science 98

Knowledge Hut

OCTOBER 27, 2023

In today's fast-paced technological environment, software engineers are continually seeking innovative projects to hone their skills and stay ahead of industry trends. Engaging in software engineering projects not only helps sharpen your programming abilities but also enhances your professional portfolio. cvtColor(image, cv2.COLOR_BGR2GRAY)

Software Engineer 98

Software Engineer Software Engineering Coding Project 98

Data Engineering Weekly

MARCH 3, 2024

There will be food, networking, and real-world talks around data engineering. This approach led to a successful expansion of Copilot access across the engineering team, resulting in a significant increase in productivity and adoption, demonstrating a commitment to enhancing developer experience while maintaining safety and security standards.

Data Engineering 67

Data Engineering Data Engineer Pipeline-centric Engineering 67

Data Engineering Weekly

DECEMBER 3, 2023

link] Microsoft: Generative AI for Beginners Understanding Gen-AI becomes a mandatory skill for application developers and data engineers. link] Netflix: Incremental Processing using Netflix Maestro and Apache Iceberg Netflix writes about its incremental processing design with its orchestration engine Maestro on top of Iceberg.

Data Engineering 70

Data Engineering Data Engineer Engineering Bytes 70

Data Engineering Weekly

MARCH 24, 2024

link] Meta: Logarithm - A logging engine for AI training workflows and services Logarithm indexes 100+GB/s of logs in real-time and thousands of queries a second!!! The logging engine to debug AI workflow logs is an excellent system design study if you’re interested in it.

Data Engineering 62

Data Engineering Data Engineer Engineering Metadata 62

Knowledge Hut

APRIL 24, 2024

In today's fast-paced technological environment, software engineers are continually seeking innovative projects to hone their skills and stay ahead of industry trends. Engaging in software engineering projects not only helps sharpen your programming abilities but also enhances your professional portfolio. cvtColor(image, cv2.COLOR_BGR2GRAY)

Software Engineer 52

Software Engineer Software Engineering Coding Project 52

Knowledge Hut

DECEMBER 27, 2023

What will be the key skills for a software engineer in 2024? The top software developers stand out from the crowd thanks to their extensive technical skills as software engineers. Thus, to become a highly sought-after software developer, you must increase and update the technical skills required for software engineering.

Software Engineer 98

Software Engineer Software Engineering Engineering Programming Language 98

The Pragmatic Engineer

OCTOBER 10, 2022

Part of this article was originally published in The Scoop #27 , for subscribers of The Pragmatic Engineer Newsletter last week. The Business Insider article was not specific to software engineers but still spread heavily within tech circles. I talked with engineering managers at Meta to find out how much truth they see in this claim.

Engineering 130

Engineering Software Engineer Software Engineering Recruitment 130

RandomTrees

FEBRUARY 2, 2024

GPT-based data engineering accelerators make the working of data more accessible. They use intelligent language interfaces and make data accessible to more people. DataGPT OpenAI developed DataGpt for performing data engineering tasks. These accelerators combine information from different sources.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Pipeline 52

Towards Data Science

DECEMBER 4, 2023

A Glossary with Use Cases for First-Timers in Data Engineering An happy Data Engineer at work Are you a data engineering rookie interested in knowing more about modern data infrastructures? In this guide Data Engineering meets Formula 1. Data must be accessible and tailored to specific business units requirements.

Data Engineering 83

Data Engineering Data Engineer Engineering Data Lake 83

Cloudera

SEPTEMBER 10, 2021

Shared Data Experience ( SDX ) on Cloudera Data Platform ( CDP ) enables centralized data access control and audit for workloads in the Enterprise Data Cloud. This introduces new challenges around managing data access across teams and individual users. Customer 1 – Centralized data authorization management.

Cloud Storage 81

Cloud Storage Accessible Accessibility Cloud 81

Data Engineering Weekly

FEBRUARY 11, 2024

The structured approach the data products bring to create, access, manage, and delete data assets is certainly appealing. link] Alireza Sadeghi: Open Source Data Engineering Landscape 2024 Popular open-source tools like Airbyte and Snowplow have recently changed their licenses. What do you think of Data Products?

Data Engineering 62

Data Engineering Data Engineer Engineering SQL 62

Knowledge Hut

FEBRUARY 6, 2024

We generally came acr oss headlines "Customers report difficulty in accessing bank mobile and online", "Bank Website down, not working " , " Serv ice Unavailable " and such unpredictability is occurring on a regular frequency.  It is an engineer ’s responsibility to make the system as reliable as it can be.

Engineering 52

Engineering Amazon Web Services Banking Architecture 52

databricks

SEPTEMBER 26, 2023

My summer internship on the PySpark team was a whirlwind of exciting events. The PySpark team develops the Python APIs of the open.

Accessible 78

Accessible Accessibility Python Engineering 78