Start Data Engineering
MAY 18, 2022
1. Introduction 2. Set up 3. Reproducibility 3.1. Docker 3.2. Docker Compose 4. Developer ergonomics 4.1. Formatting and testing 4.2. Makefile 5. Conclusion 6. Further reading 7. References 1. Introduction Data systems usually involve multiple systems, which makes local development challenging.
Azure Data Engineering
MAY 15, 2022
When it comes to transforming structured data, (e.g., applying business logic, standardization etc.) stored in a database, SQL is the most convenient and fit-to-purpose option. Stored procedures provide a way to store the transformation logic as a set of SQL statements that can be re-executed as pre-compiled code. The Stored Procedure Activity in Data Factory provides and simple and convenient way to execute Stored Procedures.
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
KDnuggets
MAY 19, 2022
Read the best books on Programming, Statistics, Data Engineering, Web Scraping, Data Analytics, Business Intelligence, Data Applications, Data Management, Big Data, and Cloud Architecture.
Confluent
MAY 17, 2022
I’m proud to announce the release of Apache Kafka 3.2.0 on behalf of the Apache Kafka® community. The 3.2.0 release contains many new features and improvements. This blog will highlight […].
Speaker: Timothy Chan, PhD., Head of Data Science
Are you ready to move beyond the basics and take a deep dive into the cutting-edge techniques that are reshaping the landscape of experimentation? 🌐 From Sequential Testing to Multi-Armed Bandits, Switchback Experiments to Stratified Sampling, Timothy Chan, Data Science Lead, is here to unravel the mysteries of these powerful methodologies that are revolutionizing how we approach testing.
Data Engineering Podcast
MAY 15, 2022
Summary Designing a data platform is a complex and iterative undertaking which requires accounting for many conflicting needs. Designing a platform that relies on a data lake as its central architectural tenet adds additional layers of difficulty. Srivatsan Sridharan has had the opportunity to design, build, and run data lake platforms for both Yelp and Robinhood, with many valuable lessons learned from each experience.
Cloudera
MAY 18, 2022
Deciding to adopt an AI-first strategy is the easy part. Figuring out how to implement it takes a little more effort. It requires a clear-eyed vision built around well-defined goals and a realistic execution plan. Being AI-first means setting up your organization for the future. By leveraging data, analytics, and automation, a company can gain a better understanding of where it is and where it needs to go.
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
Confluent
MAY 18, 2022
There’s an increasing need for businesses to act intelligently and in real time to win in today’s digital-first world. To achieve this, forward-thinking companies are modernizing their data infrastructure with […].
Teradata
MAY 19, 2022
Faced with persistent supply chain disruption automotive companies need a new approach to planning. Find out more.
Cloudera
MAY 19, 2022
From fashion to data flow, in this #ClouderaLife Spotlight Margot talks about her career transition from fashion design to cloud computing and her co-founding of Cloudera’s Asian American and Pacific Islander community Employee Resource Group amid the racial tensions of 2021. . It started with feeling stuck and ended with a brand-new career (BTW, lots of hard work in the middle).
KDnuggets
MAY 16, 2022
This post provides a concise overview of 18 natural language processing terms, intended as an entry point for the beginner looking for some orientation on the topic.
Speaker: Anne Steiner and David Laribee
As a concept, Developer Experience (DX) has gained significant attention in the tech industry. It emphasizes engineers’ efficiency and satisfaction during the product development process. As product managers, we need to understand how a good DX can contribute not only to the well-being of our development teams but also to the broader objectives of product success and customer satisfaction.
Confluent
MAY 18, 2022
With more data being produced in real time by many systems and devices than ever before, it is critical to be able to process it in real time and get […].
Emeritus
MAY 19, 2022
With businesses increasingly relying on data for their day-to-day operations, the role of a data engineer has emerged as one of the most sought-after professions in the industry. But what does a data engineer do exactly? And why is it in demand? According to McKinsey, by 2025, smart workflows and seamless interactions between humans and… The post What Does a Data Engineer do?
Big Data Tools
MAY 19, 2022
Long time no see! Sorry about the silence, but luckily we’re back. Hi, I’m Pasha Finkelshteyn , and I’ll be your guide through this month’s news. I’ll offer my impressions of recent developments in the data engineering space and highlight new ideas from the wider community. If you think I missed something worthwhile, catch me on Twitter and suggest a topic, link, or anything else you want to see.
KDnuggets
MAY 18, 2022
Complete guide and blog post series on IT Operations Management with AIOps. Using AI and Machine Learning to manage IT complexity to deliver world class IT service while keeping the lights on.
Speaker: Aarushi Kansal, AI Leader & Author and Tony Karrer, Founder & CTO at Aggregage
Software leaders who are building applications based on Large Language Models (LLMs) often find it a challenge to achieve reliability. It’s no surprise given the non-deterministic nature of LLMs. To effectively create reliable LLM-based (often with RAG) applications, extensive testing and evaluation processes are crucial. This often ends up involving meticulous adjustments to prompts.
Rockset
MAY 17, 2022
This is the fourth post in a series by Rockset's CTO Dhruba Borthakur on Designing the Next Generation of Data Systems for Real-Time Analytics. We'll be publishing more posts in the series in the near future, so subscribe to our blog so you don't miss them! Posts published so far in the series: Why Mutability Is Essential for Real-Time Data Analytics Handling Out-of-Order Data in Real-Time Analytics Applications Handling Bursty Traffic in Real-Time Analytics Applications SQL and Complex Queries
Emeritus
MAY 19, 2022
Data is so ubiquitous and valuable that it is touted as the new currency. From data analytics to data engineering, everything is data-centric. As Carly Fiorina, the former Chief Executive Officer of Hewlett Packard, said, “The goal is to turn data into information, and information into insight.” Data allows leaders to make informed decisions that… The post What Does a Data Engineer Do and How Can You Become One?
Big Data Tools
MAY 19, 2022
Long time no see! Sorry about the silence, but luckily we’re back. Hi, I’m Pasha Finkelshteyn , and I’ll be your guide through this month’s news. I’ll offer my impressions of recent developments in the data engineering space and highlight new ideas from the wider community. If you think I missed something worthwhile, catch me on Twitter and suggest a topic, link, or anything else you want to see.
KDnuggets
MAY 20, 2022
Most companies haven’t seen ROI from machine learning since the benefit is only realized when the models are in production. Here’s how to make sure your ML project works.
Advertisement
Are you trying to decide which entity resolution capabilities you need? It can be confusing to determine which features are most important for your project. And sometimes key features are overlooked. Get the Entity Resolution Evaluation Checklist to make sure you’ve thought of everything to make your project a success! The list was created by Senzing’s team of leading entity resolution experts, based on their real-world experience.
Monte Carlo
MAY 16, 2022
Data lineage isn’t new, but automation has finally made it accessible and scalable—to a certain extent. In the old days (way back in the mid-2010s), lineage happened through a lot of manual work. This involved identifying data assets, tracking them to their ingestion sources, documenting those sources, mapping the path of data as it moved through various pipelines and stages of transformation, and pinpointing where the data was served up in dashboards and reports.
dbt Developer Hub
MAY 16, 2022
Analytics engineers (AEs) are constantly navigating through the names of the models in their project, so naming is important for maintainability in your project in the way you access it and work within it. By default, dbt will use your model file name as the view or table name in the database. But this means the name has a life outside of dbt and supports the many end users who will potentially never know about dbt and where this data came from, but still access the database objects in the datab
Data Engineering Podcast
MAY 15, 2022
Summary Industrial applications are one of the primary adopters of Internet of Things (IoT) technologies, with business critical operations being informed by data collected across a fleet of sensors. Vopak is a business that manages storage and distribution of a variety of liquids that are critical to the modern world, and they have recently launched a new platform to gain more utility from their industrial sensors.
KDnuggets
MAY 20, 2022
We give a taxonomy of the trustworthy GNNs in privacy, robustness, fairness, and explainability. For each aspect, we categorize existing works into various categories, give general frameworks in each category, and more.
Speaker: Margaret-Ann Seger, Head of Product, Statsig
Experimentation is often seen as an aspirational practice, especially at smaller, fast-moving companies who are strapped for time and resources. So, how can you get your team making decisions in a more data-driven way while continuing to remain lean and maintaining ship velocity? In this webinar, Margaret-Ann Seger, Head of Product at Statsig, will teach you how to build an experimentation culture from the ground-up, graduating from just getting started with data-driven development to operating
Monte Carlo
MAY 19, 2022
In his seminal 2017 blog post, The Downfall of the Data Engineer , Maxime Beauchemin wrote that the data engineer had the worst seat at the table. Data technology and teams have changed tremendously since that time, and now the Preset CEO and creator of Apache Airflow and Apache Superset has a brighter outlook on the future of the profession. I have also seen what was once a thankless position turn into a strategic driver of company value as data expanded beyond dashboards to machine learning mo
dbt Developer Hub
MAY 14, 2022
There are so many different date functions in SQL—you have DATEDIFF , DATEADD , DATE_PART, and DATE_TRUNC to name a few. They all have their different use cases and understanding how and when they should be used is a SQL fundamental to get down. Are any of those as easy to use as the EXTRACT function? Well, that debate is for another time… In this post, we’re going to give a deep dive into the EXTRACT function, how it works, and why we use it.
KDnuggets
MAY 16, 2022
A simple guide to reinforcement learning for a complete beginner. The blog includes definitions with examples, real-life applications, key concepts, and various types of learning resources.
KDnuggets
MAY 17, 2022
As a data scientist, you might have a great portfolio of technical skills, but if you can’t communicate effectively, you won’t be able to convey your ideas clearly during virtual meetings.
Speaker: David Bard, Principal at VP Product Coaching
In the fast-paced world of digital innovation, success is often accompanied by a multitude of challenges - like the pitfalls lurking at every turn, threatening to derail the most promising projects. But fret not, this webinar is your key to effective product development! Join us for an enlightening session to empower you to lead your team to greater heights.
KDnuggets
MAY 18, 2022
5 Free Hosting Platform For Machine Learning Applications; Data Mesh Architecture: Reimagining Data Management; Popular Machine Learning Algorithms; Reinforcement Learning for Newbies ; Deep Learning For Compliance Checks: What's New?
KDnuggets
MAY 20, 2022
Let's look at six must-have tools every data scientist should use.
KDnuggets
MAY 17, 2022
Hugging Face has released a free course on Deep RL. It is self-paced and shares a lot of pointers on theory, tutorials, and hands-on guides.
KDnuggets
MAY 19, 2022
What should you be looking for in an IDE? Find out here.
Advertisement
“Reimagined: Building Products with Generative AI” is an extensive guide for integrating generative AI into product strategy and careers featuring over 150 real-world examples, 30 case studies, and 20+ frameworks, and endorsed by over 20 leading AI and product executives, inventors, entrepreneurs, and researchers.
Expert insights. Personalized for you.
147 
Let's personalize your content