The Pragmatic Engineer
FEBRUARY 6, 2025
The past six months has been something of a Doomsday scenario-esque countdown for TikTok, as the start date of its ban in the US crept ever closer. In the event, TikTok did indeed go offline for a few hours on 19 January, before President Trump gave the social network a stay of execution lasting 75 days. How has this uncertainty affected software engineers at the Chinese-owned social network?
Snowflake
FEBRUARY 3, 2025
Generative AIs potential to drive innovation, improve efficiency and create competitive advantages is enormous. However, the ability to fully realize the benefits of generative AI hinges on one crucial factor: data strategy. Data Strategies for AI Leaders , a report co-written by MIT and Snowflake, underscores how organizations must invest in robust data foundations to succeed in the AI era.
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Engineering at Meta
FEBRUARY 4, 2025
Were sharing how Meta built support for data logs, which provide people with additional data about how they use our products. Here we explore initial system designs we considered, an overview of the current architecture, and some important principles Meta takes into account in making data accessible and easy to understand. Users have a variety of tools they can use to manage and access their information on Meta platforms.
Precisely
FEBRUARY 3, 2025
Key Takeaways A unified customer communication management (CCM) solution eliminates reliance on IT for communication updates, which empowers business users to create and deploy content quickly. Fast, personalized, and seamless customer communications help you build customer trust and drive loyalty. Save time and money with streamlined processes and automation that increase operational efficiency and improve the customer experience.
Speaker: Ben Epstein, Stealth Founder & CTO | Tony Karrer, Founder & CTO, Aggregage
When tasked with building a fundamentally new product line with deeper insights than previously achievable for a high-value client, Ben Epstein and his team faced a significant challenge: how to harness LLMs to produce consistent, high-accuracy outputs at scale. In this new session, Ben will share how he and his team engineered a system (based on proven software engineering approaches) that employs reproducible test variations (via temperature 0 and fixed seeds), and enables non-LLM evaluation m
Simon Späti
FEBRUARY 3, 2025
While everyone’s talking about AI revolutionizing business, there’s a quiet renaissance happening with one of the most influential business tools created: the pivot table. In 2025, we’re witnessing something remarkable - modern data tools are bringing pivot tables back to the forefront. But why would cutting-edge platforms invest in a decades-old spreadsheet feature?
Snowflake
FEBRUARY 4, 2025
AI is here to stay. While 2023 brought wonder and 2024 ushered in widespread experimentation, 2025 will mark the year that retailers get serious about AI's real-world applications. But its complicated: AI proofs of concept are graduating from the sandbox to production even as major AI innovators face competition from newer upstarts. At this point, the pace of AI evolution is outstripping the news cycle.
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
 |
Start Data Engineering
FEBRUARY 3, 2025
1. Introduction 2. Split your SQL into smaller parts 2.1. Start with a baseline validation to ensure that your changes do not change the output too much 2.2. Split your CTAs/Subquery into separate functions (or models if using dbt) 2.3. Unit test your functions for maintainability and evolution of logic 3. Conclusion 4. Required reading 1. Introduction If you’ve been in the data space long enough, you would have come across really long SQL scripts that someone had written years ago.
Precisely
FEBRUARY 6, 2025
As businesses evolve and delivery speeds increase, IT operations teams face environments where downtime isn’t an option. The traditional ways of operations management are over modernization and holistic approaches are now essential. For IT operations (ITOps) teams, 2025 means reassessing technology stacks, processes, and people. Success in tackling modernization of IT operations management starts with assessing where your team is.
KDnuggets
FEBRUARY 3, 2025
Check out this comparison of 5 AI frameworks to determine which you should choose.
Data Engineering Weekly
FEBRUARY 2, 2025
Automate Airflow deploys with built-in CI/CD. Streamline code deployment, enhance collaboration, and ensure DevOps best practices with Astro's robust CI/CD capabilities. Try Astro Free → Jay Alammar: The Illustrated DeepSeek-R1 DeepSeek triggered quite a conversation and had an economic impact last week. Many articles explain how DeepSeek works, and I found the illustrated example much simpler to understand.
Advertisement
Apache Airflow® is the open-source standard to manage workflows as code. It is a versatile tool used in companies across the world from agile startups to tech giants to flagship enterprises across all industries. Due to its widespread adoption, Airflow knowledge is paramount to success in the field of data engineering.
databricks
FEBRUARY 3, 2025
In the rapidly evolving landscape of data engineering and analytics, speed, scalability, and simplicity are invaluable. Serverless compute addresses these needs by eliminating.
Pinterest Engineering
FEBRUARY 3, 2025
Zhibo Fan | Machine Learning Engineer, Homefeed Candidate Generation; Bowen Deng | Machine Learning Engineer, Homefeed Candidate Generation; Hedi Xia | Machine Learning Engineer, Homefeed Candidate Generation; Yuke Yan | Machine Learning Engineer, Homefeed Candidate Generation; Hongtao Lin | Machine Learning Engineer, ATG Applied Science; Haoyu Chen | Machine Learning Engineer, ATG Applied Science; Dafang He | Machine Learning Engineer, Homefeed Relevance; Jay Adams | Principal Engineer, Pinner
KDnuggets
FEBRUARY 7, 2025
As tech layoffs increase, data scientists must adapt. Here's how to safeguard your data science job in 2025.
Picnic Engineering
FEBRUARY 6, 2025
In our previous blogs, we explored how Picnics Page Platform transformed the way we build new featuresenabling faster iteration, tighter collaboration and less feature-specific complexity. Advances we made there have been a major boost for productivity and allow us to focus on the things that actually deliver customer value. However, while focussing on our technical architecture, we havent yet discussed one major opportunity: bringing our commercial and customer teams into themix.
Speaker: Tamara Fingerlin, Developer Advocate
In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!
databricks
FEBRUARY 6, 2025
FOX Sports has a long history of driving the evolution of broadcast technology, from its high-definition coverage to experiments with virtual reality. Eventually.
Striim
FEBRUARY 5, 2025
AI thrives on real-time data. In a world where businesses generate massive volumes of data every second, success hinges on the ability to process, analyze, and act on that data instantly. Change Data Capture (CDC) and streaming technologies form the foundation for AI-driven analytics, ensuring data is always fresh, accurate, and actionable. Together, CDC and streaming empower businesses to: Supercharge AI models with real-time data: Provide AI with up-to-the-second insights to improve prediction
KDnuggets
FEBRUARY 6, 2025
Interact with the advanced multimodal Llama model locally through its intuitive user interface or powerful endpoint service.
Sync Computing
FEBRUARY 4, 2025
As data engineers, understanding the intricacies of your Databricks environment is important. You cant optimize performance, budget or ensure efficient resource allocations without it. Thankfully, Databricks gives you a behind-the-scenes look at how your workspace is running in system tables. Everything from query performance to job execution and cluster activity is in those tables.
Advertisement
Many software teams have migrated their testing and production workloads to the cloud, yet development environments often remain tied to outdated local setups, limiting efficiency and growth. This is where Coder comes in. In our 101 Coder webinar, you’ll explore how cloud-based development environments can unlock new levels of productivity. Discover how to transition from local setups to a secure, cloud-powered ecosystem with ease.
databricks
FEBRUARY 6, 2025
Introducing Serverless Support for AWS Instance Profiles: Uniform Data Access At Databricks, we continuously strive to simplify data access and drive innovation across.
Striim
FEBRUARY 4, 2025
Introduction At Striim, we use our Salesforce Reader to read from our Salesforce account and write into Google BigQuery where we join data from HubSpot to create Looker reports that multiple internal teams (Sales, Customer Success and Finance) use for reporting, analysis and drive action items for their departments. This recipe shows how you can build a data pipeline to read data from Salesforce and write to BigQuery.
KDnuggets
FEBRUARY 3, 2025
Fine-tune the DeepSeek model step by step. even if you're new to LLMs!
WeCloudData
FEBRUARY 6, 2025
We live in a highly data oriented world, thus it’s important to understand the key roles of the data ecosystem. Data scientists and engineers are two of the most important data professions and it is important to understand the difference between data engineering vs data science. At WeCloudData we specialize in preparing professionals for careers […] The post Data Engineering vs.
Advertisement
Large enterprises face unique challenges in optimizing their Business Intelligence (BI) output due to the sheer scale and complexity of their operations. Unlike smaller organizations, where basic BI features and simple dashboards might suffice, enterprises must manage vast amounts of data from diverse sources. What are the top modern BI use cases for enterprise businesses to help you get a leg up on the competition?
databricks
FEBRUARY 4, 2025
Databricks welcomes BladeBridge, a proven provider of AI-powered migration solutions for enterprise data warehouses. Together, Databricks and BladeBridge will help enterprises accelerate the.
ArcGIS
FEBRUARY 3, 2025
Explore how Dar used ArcGIS CityEngine for procedural urban design to transform a brownfield into a vibrant waterfront destination in Jeddah.
KDnuggets
FEBRUARY 7, 2025
Tools I use for coding, writing, grammar improvement, research, machine learning experiments, and organizing projects.
WeCloudData
FEBRUARY 3, 2025
Data is the new Gold. Everyday we use and generate data more than we often realize. Data is shaping our decisions, from scrolling through personalized social media feeds to checking weather forecasts before leaving home. Behind the scenes, Data Science powers banking apps to detect suspicious activity or when you get personalized recommendations on […] The post What is Data Science appeared first on WeCloudData.
Advertisement
With over 30 million monthly downloads, Apache Airflow is the tool of choice for programmatically authoring, scheduling, and monitoring data pipelines. Airflow enables you to define workflows as Python code, allowing for dynamic and scalable pipelines suitable to any use case from ETL/ELT to running ML/AI operations in production. This introductory tutorial provides a crash course for writing and deploying your first Airflow pipeline.
databricks
FEBRUARY 5, 2025
Were thrilled to announce the General Availability (GA) of Databricks Clean Rooms on AWS and Azure, a significant step forward in enabling secure.
Confluent
FEBRUARY 3, 2025
Confluent Cloud Freight clusters are now Generally Available on AWS.
KDnuggets
FEBRUARY 3, 2025
Learn to create a document-based chatbot with memory, powered by one of the world's top-performing LLMs.
Yelp Engineering
FEBRUARY 3, 2025
How we bring LLM intelligence to millions of daily searches at Yelp. From the moment a user enters a search query to when we present a list of results, understanding the users intent is crucial for meeting their needs. Were they looking for a general category of business for that evening, a particular dish or service, or one specific business nearby?
Speaker: Jay Allardyce, Deepak Vittal, Terrence Sheflin, and Mahyar Ghasemali
As we look ahead to 2025, business intelligence and data analytics are set to play pivotal roles in shaping success. Organizations are already starting to face a host of transformative trends as the year comes to a close, including the integration of AI in data analytics, an increased emphasis on real-time data insights, and the growing importance of user experience in BI solutions.
Expert insights. Personalized for you.
169 
Let's personalize your content