Marc Lamberti
OCTOBER 1, 2023
Airflow Sensors are one of the most common tasks in data pipelines. Why? Because a Sensor waits for a condition to be true to complete. Do you need to wait for a file? Check if an SQL entry exists? Delay the execution of a DAG? That’s the few possibilities of the Airflow Sensors. If you want to make complex and robust data pipelines, you have to understand how Sensors work genuinely.
The Pragmatic Engineer
SEPTEMBER 28, 2023
👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. We cover one out of four topics in today’s subscriber-only The Pulse issue. To get full newsletters twice a week, subscribe here. Willem Spruijt is a software engineer whom I worked on the same team with at Uber in Amsterdam, building payments systems.
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
The Pragmatic Engineer
SEPTEMBER 26, 2023
👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. In every issue, I cover topics related to Big Tech and startups through the lens of engineering managers and senior engineers. In this article, we cover a fresh industry trends: Cloud Developent Environments — which is analysis full subscribers have received 3 weeks ago.
Data Engineering Podcast
OCTOBER 1, 2023
Summary Artificial intelligence applications require substantial high quality data, which is provided through ETL pipelines. Now that AI has reached the level of sophistication seen in the various generative models it is being used to build new ETL workflows. In this episode Jay Mishra shares his experiences and insights building ETL pipelines with the help of generative AI.
Advertisement
Are you thinking of adding enhanced data matching and relationship detection to your product or service? Do you need to know more about what to look for when assessing your options? Our Entity Resolution Buyer’s Guide gives you step-by-step details about everything you should consider when evaluating entity resolution technologies. We discuss use cases, technology, and deployment options, top ten evaluation criteria and more.
Confessions of a Data Guy
SEPTEMBER 29, 2023
I always leave it to my dear readers and followers to give me pokes in the right direction. Nothing like the teaming masses to set you straight. Recently I was working on my Substack Newsletter, on the topic of Polars + Delta Lake, reading remove files from s3 … I left a question open on […] The post DuckDB + Delta Lake (the new lake house?
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
Marc Lamberti
OCTOBER 2, 2023
Airflow Variables are easy to use but easy to misuse as well. In this tutorial, you will learn everything you need about variables in Apache Airflow. What are they, how do they work, define one, get the value, and more. If you followed my course “Apache Airflow: The Hands-On Guide” variables shouldn’t sound unfamiliar. This time, I will give you all I know about variables so that, in the end, you will be ready to use Variables in your DAGs properly.
Waitingforcode
SEPTEMBER 27, 2023
It's always a huge pleasure to see the PySpark API covering more and more Scala API features. Starting from Apache Spark 3.4.0 you can even write arbitrary stateful processing jobs! But since the API is a little bit different than the one available on the Scala side, I wanted to take a deeper look.
Confessions of a Data Guy
OCTOBER 1, 2023
So perhaps you’re thinking it’s time to use Rust on your next project. You’ll find plenty of primers on how to get your feet wet in the language (and if you somehow made it this far without that much, The Book is that starting point), but maybe you’re feeling a bit lost amidst the seas […] The post Introduction to using Rust Libraries (cargo and crates) appeared first on Confessions of a Data Guy.
KDnuggets
SEPTEMBER 29, 2023
This tutorial provides an in-depth introduction to machine learning using PyTorch and its high-level wrapper, PyTorch Lightning. The article covers essential steps from installation to advanced topics, offering a hands-on approach to building and training neural networks, and emphasizing the benefits of using Lightning.
databricks
SEPTEMBER 28, 2023
We are excited to announce public preview of GPU and LLM optimization support for Databricks Model Serving! With this launch, you can deploy.
Marc Lamberti
SEPTEMBER 29, 2023
At the end of this introduction to Airflow, you will be all set for getting started with Airflow. You will start with the basics, such as what Airflow is and the essential concepts. Then you will set up and run your local development environment using the Astro CLI to create your first data pipeline. I hope you’re getting excited. Fasten your seatbelt, take a deep breath, and let’s go For a complete hands-on introduction to Apache Airflow, here is a 6-hour course at a discount.
Cloudera
SEPTEMBER 26, 2023
Organizations increasingly rely on streaming data sources not only to bring data into the enterprise but also to perform streaming analytics that accelerate the process of being able to get value from the data early in its lifecycle. As lakehouse architectures (including offerings from Cloudera and IBM) become the norm for data processing and building AI applications, a robust streaming service becomes a critical building block for modern data architectures.
Snowflake
SEPTEMBER 26, 2023
The Snowflake Native App Framework , which leverages Snowflake’s advanced architecture, allows for a new level of security for applications. This security spans not just the application consumer, but also the application providers. Controlling all software and infrastructure in the Snowflake Data Cloud, Snowflake can secure the application code to protect the intellectual property (IP) of builders.
KDnuggets
SEPTEMBER 29, 2023
Are you looking for the open source tools to help you in your data science journey? Look no further. Discover these game-changers that will elevate your data-driven decisions.
ArcGIS
SEPTEMBER 28, 2023
Esri Vector Basemaps are available for use in ArcGIS Pro, and that opens up some new possibilities for you.
databricks
SEPTEMBER 29, 2023
This article represents a collaborative effort between Plotly, Ballard Power Systems, and Databricks. Fleets of buses worldwide run on hydrogen fuel cells made.
Cloudera
SEPTEMBER 27, 2023
As data ecosystems evolve security becomes a paramount concern, especially within the realm of private cloud environments. Cloudera on Private Cloud with the Private Cloud Base (CDP PvC Base) stands as a beacon of innovation in the realm of data security, offering a holistic suite of features that work in concert to safeguard sensitive information. With the latest 7.1.9 release , the journey towards a more secure data ecosystem continues — one where businesses can unlock the full potential of th
Pinterest Engineering
SEPTEMBER 29, 2023
Sanchay Javeria | Software Engineer, Ads Data Infrastructure To support metrics reporting for ads from external advertisers and real-time ad budget calculations at Pinterest, we run streaming pipelines using Apache Flink. These jobs have guaranteed an overall 99th percentile availability to our users; however, every once in a while some tasks get hit with nasty direct out-of-memory (OOM) errors on multiple operators that look something like this: As is the case with most failures in a distribute
KDnuggets
OCTOBER 1, 2023
Explore the essentials of Google Cloud Platform for data science and ML, from account setup to model deployment, with hands-on project examples.
Robinhood
SEPTEMBER 29, 2023
The initiative supports Operation HOPE’s mission to expand economic opportunity Robinhood Markets, Inc. is deepening our partnership with Operation HOPE through the 1865 Project , a new initiative that supports the organization’s mission to expand economic opportunity in underserved communities through financial education and empowerment. With our support, the 1865 project will allow Operation HOPE to further grow and scale its work across America and fuel innovative programs and technologies.
databricks
SEPTEMBER 27, 2023
This blog is authored by Ben Dias, Director of Data Science and Analytics and Ioannis Mesionis, Lead Data Scientist at easyJet Introduction to.
Workfall
SEPTEMBER 26, 2023
Reading Time: 9 minutes In this blog, we will cover: What are Server-Sent Events? Why Stream Data Using Server-Sent Events (SSE)? What is FastAPI? Hands-On Conclusion What are Server-Sent Events? Server-Sent Events (SSE) is a simple and efficient technology for sending real-time updates from the server to the web browser over a single HTTP connection.
Pinterest Engineering
SEPTEMBER 26, 2023
Fan Jiang | Software Engineer, Closeup Candidate Retrieval; Liyao Lu | Software Engineer, Closeup Ranking & Blending; Laksh Bhasin | Software Engineer, Core ML Foundations; Chen Yang | Software Engineer, Core ML Foundations; Shivin Thukral | Software Engineer, Closeup Ranking & Blending; Travis Ebesu | Software Engineer, Closeup Ranking & Blending; Kent Jiang | Software Engineer, Core Serving Infra; Yan Sun | Engineering Manager, Closeup Ranking & Blending; Huizhong Duan | Engine
KDnuggets
SEPTEMBER 29, 2023
Cloud notebooks are game-changers for data science, providing free access to computing, pre-built environments, collaboration features, and third-party integrations - everything you need to enhance your workflow.
LinkedIn Engineering
OCTOBER 2, 2023
When Javier signed up for a programming course during the pandemic, he had no idea that his career was about to shift from the world of music to data science. As his interest in AI and computer science grew, Javier found a community at LinkedIn that supported his growth and provided more opportunities to learn and lead than he could have imagined. Making the leap from music to LinkedIn Engineering with REACH My journey to LinkedIn and passion for coding came from an entirely different background
Towards Data Science
SEPTEMBER 27, 2023
Data Access API over Data Lake Tables Without the Complexity Build a robust GraphQL API service on top of your S3 data lake files with DuckDB and Go Photo by Joshua Sortino on Unsplash 1. Intro Data lake tables are mostly utilized by data engineering teams using big data compute engines, such as Spark or Flink, as well as by data analysts and scientists creating models and reports with heavy SQL query engines, such as Trino or Redshift.
Snowflake
SEPTEMBER 27, 2023
From improving patient outcomes to increasing clinical efficiencies, better access to data is helping healthcare organizations deliver better patient care. Data from hospitals, pharmacies, clinics, insurers, community and public health organizations, telehealth visits and wellness apps can be combined to provide a comprehensive view of patient health.
ArcGIS
SEPTEMBER 29, 2023
Esri basemaps are designed to be used at multiple scales, but a static map needs everything in one view. How doe we get around that?
KDnuggets
OCTOBER 2, 2023
This article explores strategies for evaluating the reliability of labels generated by Large Language Models (LLMs). It discusses the effectiveness of different approaches and offers practical insights for various applications.
Pinterest Engineering
OCTOBER 2, 2023
Each summer, Pinterest welcomes Software Engineering Pinterns who spend 12 weeks with us creating impact within our product and teams. While Pinterns are fully immersed in their teams throughout the summer, they also get to attend exciting activities and events hosted by the University Recruiting team and within the company. Here’s a quick recap from this summer: Social events were a hit with boba tea making, creating your own vision board, chocolate making and a virtual escape room.
databricks
SEPTEMBER 28, 2023
We are excited to announce the public preview of the next generation of Databricks SQL dashboards, dubbed Lakeview dashboards. Available today, this new.
Snowflake
SEPTEMBER 28, 2023
Data is an invaluable asset in today’s marketing ecosystem. With its unique blend of cultures, economies, and regulatory environments, the EMEA market offers a nuanced picture of how marketers harness data technologies to understand their audiences, calibrate campaigns in real time, and adhere to complex government and industry regulations. In our second annual Snowflake Modern Marketing Data Stack 2023 report , we delve into actual usage and adoption of marketing technologies within the Snowfla
Confluent
SEPTEMBER 29, 2023
Use Confluent connectors, stream governance and stream processing and artificial intelligence (AI) to block spammers, reduce network load, and improve your customers' experiences in real time.
Expert insights. Personalized for you.
214 
Let's personalize your content