Start Data Engineering
MAY 11, 2022
1. Introduction 2. Objective 3. Design 4. Setup 4.1 Prerequisite 4.2 AWS Infrastructure costs 4.3 Data lake structure 5. Code walkthrough 5.1 Loading user purchase data into the data warehouse 5.2 Loading classified movie review data into the data warehouse 5.3 Generating user behavior metric 5.4. Checking results 6. Tear down infra 7. Design considerations 8.
Azure Data Engineering
MAY 28, 2022
ARM or Azure Resource Manager templates make it easy to manage deployments for Data Factory. When we connect Data Factory to a source control repository (e.g. GitHub or Azure DevOps Git), the data factory along with all its artefacts ( pipelines , datasets , linked services etc.) is saved in the repository in the form of ARM templates. We can then create DevOps pipelines to manage deployments by overriding the parameters to deploy to the production environments.
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Data Engineering Podcast
MAY 22, 2022
Summary Machine learning has become a meaningful target for data applications, bringing with it an increase in the complexity of orchestrating the entire data flow. Flyte is a project that was started at Lyft to address their internal needs for machine learning and integrated closely with Kubernetes as the execution manager. In this episode Ketan Umare and Haytham Abuelfutuh share the story of the Flyte project and how their work at Union is focused on supporting and scaling the code and communi
KDnuggets
MAY 26, 2022
Read the best books on Machine Learning, Deep Learning, Computer Vision, Natural Language Processing, MLOps, Robotics, IoT, AI Products Management, and Data Science for Executives.
Advertisement
Many organizations today are unlocking the power of their data by using graph databases to feed downstream analytics, enahance visualizations, and more. Yet, when different graph nodes represent the same entity, graphs get messy. Watch this essential video with Senzing CEO Jeff Jonas on how adding entity resolution to a graph database condenses network graphs to improve analytics and save your analysts time.
Confluent
MAY 17, 2022
I’m proud to announce the release of Apache Kafka 3.2.0 on behalf of the Apache Kafka® community. The 3.2.0 release contains many new features and improvements. This blog will highlight […].
Cloudera
MAY 4, 2022
Artificial intelligence (AI) has been a focus for research for decades, but has only recently become truly viable. The availability and maturity of automated data collection and analysis systems is making it possible for businesses to implement AI across their entire operations to boost efficiency and agility. AI has the potential to transform operations by improving three fundamental business requirements: process automation, decision-making based on data insights, and customer interaction.
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
Azure Data Engineering
MAY 15, 2022
When it comes to transforming structured data, (e.g., applying business logic, standardization etc.) stored in a database, SQL is the most convenient and fit-to-purpose option. Stored procedures provide a way to store the transformation logic as a set of SQL statements that can be re-executed as pre-compiled code. The Stored Procedure Activity in Data Factory provides and simple and convenient way to execute Stored Procedures.
Data Engineering Podcast
MAY 29, 2022
Summary The latest generation of data warehouse platforms have brought unprecedented operational simplicity and effectively infinite scale. Along with those benefits, they have also introduced a new consumption model that can lead to incredibly expensive bills at the end of the month. In order to ensure that you can explore and analyze your data without spending money on inefficient queries Mingsheng Hong and Zheng Shao created Bluesky Data.
KDnuggets
MAY 30, 2022
Also: Decision Tree Algorithm, Explained; Data Science Projects That Will Land You The Job in 2022; The 6 Python Machine Learning Tools Every Data Scientist Should Know About; Naïve Bayes Algorithm: Everything You Need to Know.
Confluent
MAY 12, 2022
Internet connectivity is something we sometimes take for granted. For many, most places we visit, work, or reside have some form of connectivity whether it be cellular, Wi-Fi, fiber, etc. […].
Speaker: Scott Sehlhorst
We know we want to create products which our customers find to be valuable. Whether we label it as customer-centric or product-led depends on how long we've been doing product management. There are three challenges we face when doing this. The obvious challenge is figuring out what our users need; the non-obvious challenges are in creating a shared understanding of those needs and in sensing if what we're doing is meeting those needs.
AltexSoft
MAY 12, 2022
We live in the world of sounds: Pleasant and annoying, low and high, quiet and loud, they impact our mood and our decisions. Our brains are constantly processing sounds to give us important information about our environment. But acoustic signals can tell us even more if analyze them using modern technologies. Today, we have AI and machine learning to extract insights, inaudible to human beings, from speech, voices, snoring, music, industrial and traffic noise, and other types of acoustic signals
Cloudera
MAY 9, 2022
Tuning Hive on Tez queries can never be done in a one-size-fits-all approach. The performance on queries depends on the size of the data, file types, query design, and query patterns. During performance testing, evaluate and validate configuration parameters and any SQL modifications. It is advisable to make one change at a time during performance testing of the workload, and would be best to assess the impact of tuning changes in your development and QA environments before using them in product
Netflix Tech
MAY 21, 2022
At Netflix, we want to entertain the world through creating engaging content and helping members discover the titles they will love. Key to that is understanding causal effects that connect changes we make in the product to indicators of member joy. To measure causal effects we rely heavily on AB testing , but we also leverage quasi-experimentation in cases where AB testing is limited.
Data Engineering Podcast
MAY 15, 2022
Summary Designing a data platform is a complex and iterative undertaking which requires accounting for many conflicting needs. Designing a platform that relies on a data lake as its central architectural tenet adds additional layers of difficulty. Srivatsan Sridharan has had the opportunity to design, build, and run data lake platforms for both Yelp and Robinhood, with many valuable lessons learned from each experience.
Speaker: Timothy Chan, PhD., Head of Data Science
Are you ready to move beyond the basics and take a deep dive into the cutting-edge techniques that are reshaping the landscape of experimentation? 🌐 From Sequential Testing to Multi-Armed Bandits, Switchback Experiments to Stratified Sampling, Timothy Chan, Data Science Lead, is here to unravel the mysteries of these powerful methodologies that are revolutionizing how we approach testing.
KDnuggets
MAY 30, 2022
A machine learning engineer is a programmer proficient in building and designing software to automate predictive models. They have a deeper focus on computer science, compared to data scientists.
Confluent
MAY 11, 2022
Walmart’s global presence, with its vast number of retail stores plus its robust and rapidly growing e-commerce business, make it one of the most challenging retail companies on the planet […].
AltexSoft
MAY 27, 2022
How many days will a particular person spend in a hospital? Healthcare facilities and insurance companies would give a lot to know the answer for each new admission. Today, we can employ AI technologies to predict the date of discharge. This article describes how data and machine learning help control the length of stay — for the benefit of patients and medical organizations.
Cloudera
MAY 5, 2022
Fraud, waste, and abuse (FWA) in government is a constant, multi-billion dollar issue that challenges agency leaders at all levels and across all sectors, from healthcare to education to taxation to Social Security. The scope and scale of public spending — federal outlays alone were approximately $6.6 trillion in fiscal year 2020 according to the Congressional Budget Office — make FWA an inherently difficult problem to solve.
Speaker: Aarushi Kansal, AI Leader & Author and Tony Karrer, Founder & CTO at Aggregage
Software leaders who are building applications based on Large Language Models (LLMs) often find it a challenge to achieve reliability. It’s no surprise given the non-deterministic nature of LLMs. To effectively create reliable LLM-based (often with RAG) applications, extensive testing and evaluation processes are crucial. This often ends up involving meticulous adjustments to prompts.
Teradata
MAY 19, 2022
Faced with persistent supply chain disruption automotive companies need a new approach to planning. Find out more.
Data Engineering Podcast
MAY 8, 2022
Summary Many of the events, ideas, and objects that we try to represent through data have a high degree of connectivity in the real world. These connections are best represented and analyzed as graphs to provide efficient and accurate analysis of their relationships. TigerGraph is a leading database that offers a highly scalable and performant native graph engine for powering graph analytics and machine learning.
KDnuggets
MAY 27, 2022
Colossal amounts of data need to be dealt with by specialists. It’s no wonder then that the job prospects in this industry are expected to rise much faster than in other occupations.
Confluent
MAY 30, 2022
Kafka is horizontally scalable, but it's not enough. So we made Confluent Cloud 10x more elastic - 10x faster to scale up to GB/s or down to zero, easier to use, and cost-effective.
Advertisement
Outdated or absent analytics won’t cut it in today’s data-driven applications – not for your end users, your development team, or your business. That’s what drove the five companies in this e-book to change their approach to analytics. Download this e-book to learn about the unique problems each company faced and how they achieved huge returns beyond expectation by embedding analytics into applications.
Monte Carlo
MAY 31, 2022
This article is sourced based on the interview between Lior Solomon, (now the former) VP of Engineering, Data, at Vimeo with the co-founders of Firebolt on their Data Engineering Show podcast which took place August 18, 2021. Watch the full episode. Vimeo is a leading video hosting, sharing, and services platform provider. The 1,000+ company helps small, medium and enterprise businesses scale with the impact of video.
Cloudera
MAY 18, 2022
Deciding to adopt an AI-first strategy is the easy part. Figuring out how to implement it takes a little more effort. It requires a clear-eyed vision built around well-defined goals and a realistic execution plan. Being AI-first means setting up your organization for the future. By leveraging data, analytics, and automation, a company can gain a better understanding of where it is and where it needs to go.
Teradata
MAY 12, 2022
The next generation of customers expects newer technologies & advanced self-service capabilities as the airline business becomes more competitive. How can airlines meet these expectations?
Data Engineering Podcast
MAY 1, 2022
Summary Building a data platform is an iterative and evolutionary process that requires collaboration with internal stakeholders to ensure that their needs are being met. Yotpo has been on a journey to evolve and scale their data platform to continue serving the needs of their organization as it increases the scale and sophistication of data usage. In this episode Doron Porat and Liran Yogev explain how they arrived at their current architecture, the capabilities that they are optimizing for, an
Speaker: Anne Steiner and David Laribee
As a concept, Developer Experience (DX) has gained significant attention in the tech industry. It emphasizes engineers’ efficiency and satisfaction during the product development process. As product managers, we need to understand how a good DX can contribute not only to the well-being of our development teams but also to the broader objectives of product success and customer satisfaction.
KDnuggets
MAY 19, 2022
Read the best books on Programming, Statistics, Data Engineering, Web Scraping, Data Analytics, Business Intelligence, Data Applications, Data Management, Big Data, and Cloud Architecture.
Confluent
MAY 30, 2022
What we’ve done to evolve from cloud Kafka to Confluent Cloud, a data streaming platform that’s 10X better than Kafka in elasticity, storage, resiliency, and more.
U-Next
MAY 26, 2022
Time is the only changing constant and with time, everything changes. Emotions, people and markets. Every now and then in our lives, there comes a time of disruption. Where routines are rattled and we are introduced to new things. . While this often sounds exciting, what these sudden changes put an end to are existing conventions and practices. . We are living at one such time. .
Cloudera
MAY 19, 2022
From fashion to data flow, in this #ClouderaLife Spotlight Margot talks about her career transition from fashion design to cloud computing and her co-founding of Cloudera’s Asian American and Pacific Islander community Employee Resource Group amid the racial tensions of 2021. . It started with feeling stuck and ended with a brand-new career (BTW, lots of hard work in the middle).
Speaker: David Bard, Principal at VP Product Coaching
In the fast-paced world of digital innovation, success is often accompanied by a multitude of challenges - like the pitfalls lurking at every turn, threatening to derail the most promising projects. But fret not, this webinar is your key to effective product development! Join us for an enlightening session to empower you to lead your team to greater heights.
Expert insights. Personalized for you.
246 
Let's personalize your content