Start Data Engineering
NOVEMBER 11, 2021
1. Introduction 2. Setting up services locally 3. Writing an end-to-end data pipeline test 4. Conclusion 5. Further reading 6. References 1. Introduction Data pipelines can have multiple software components. This makes testing all of them together difficult. If you are wondering What is the best way to end-to-end test data pipelines? Are end-to-end tests worth the effort?
Azure Data Engineering
NOVEMBER 10, 2021
In the previous post, we discussed the Switch Activity , which is useful for branching the control flow based on some condition. We will discuss about the Filter Activity in this post. The purpose of Filter Activity is to process array items based on some condition. Consider a scenario where we would like to set the value of a variable to the current array item that satisfies some business rule or condition.
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Uber Engineering
NOVEMBER 10, 2021
Introduction. Each day, Uber moves millions of people around the world and delivers tens of millions of food and grocery orders. This generates a large number of financial transactions that need to be stored with provable completeness, consistency, and compliance. … The post How Uber Migrated Financial Data from DynamoDB to Docstore appeared first on Uber Engineering Blog.
Confluent
NOVEMBER 8, 2021
How does Confluent provide fine-grained operational visibility to our customers throughout all of the multi-tenant services that we run in the cloud? At Confluent Cloud, we manage a large number […].
Advertisement
Many organizations today are unlocking the power of their data by using graph databases to feed downstream analytics, enahance visualizations, and more. Yet, when different graph nodes represent the same entity, graphs get messy. Watch this essential video with Senzing CEO Jeff Jonas on how adding entity resolution to a graph database condenses network graphs to improve analytics and save your analysts time.
Data Engineering Podcast
NOVEMBER 10, 2021
Summary A significant source of friction and wasted effort in building and integrating data management systems is the fragmentation of metadata across various tools. After experiencing the impacts of fragmented metadata and previous attempts at building a solution Suresh Srinivas and Sriharsha Chintalapani created the OpenMetadata project. In this episode they share the lessons that they have learned through their previous attempts and the positive impact that a unified metadata layer had during
KDnuggets
NOVEMBER 8, 2021
With a lot of excitement and research around NLP, there are growing opportunities to apply these technologies to real-world scenarios. It's not trivial to become familiar with NLP and these open-source data sets can help you increase your skills.
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
Cloudera
NOVEMBER 12, 2021
Arjun (my son) sat next to me at my desk. He was a bit nervous but we had practiced 3 times before he was ‘on stage’ in front of hundreds of people and the zoom meeting turned to him. My ten year old began to demonstrate how to deploy an Operational Database in AWS, showcasing how auto-scaling worked and how to set up replication. All of the sales team and my colleagues were quite impressed with him, and I am very proud of him.
DataKitchen
NOVEMBER 12, 2021
The post The Benefits and Drawbacks of DataOps in Practice first appeared on DataKitchen.
KDnuggets
NOVEMBER 12, 2021
The PyTorch Deep Learning framework has a C++ API for use on mobile platforms. This article shows an end-to-end demo of how to write a simple C++ application with Deep Learning capabilities using the PyTorch C++ API such that the same code can be built for use on mobile platforms (both Android and iOS).
Confluent
NOVEMBER 10, 2021
Schema Linking is a new feature that’s available in preview for both Confluent Cloud and Confluent Platform 7.0 and can be used to complement Cluster Linking, in order to keep […].
Speaker: Scott Sehlhorst
We know we want to create products which our customers find to be valuable. Whether we label it as customer-centric or product-led depends on how long we've been doing product management. There are three challenges we face when doing this. The obvious challenge is figuring out what our users need; the non-obvious challenges are in creating a shared understanding of those needs and in sensing if what we're doing is meeting those needs.
Cloudera
NOVEMBER 10, 2021
Cloudera Data Platform (CDP) brings many improvements to customers by merging technologies from the two legacy platforms, Cloudera Enterprise Data Hub (CDH) and Hortonworks Data Platform (HDP). CDP includes new functionalities as well as superior alternatives to some previously existing functionalities in security and governance. One such major change for CDH users is the replacement of Sentry with Ranger for authorization and access control. .
Pipeline Data Engineering
NOVEMBER 9, 2021
Ambitious data analysts, data scientists trying to take their careers to the next level, product owners aiming to build next-generation data products, software engineers dealing with legacy data stacks. they all are facing the same challenge: how do I get the data engineering skills that enable me to achieve my goals? This frustration is very real, and it is indeed very common!
KDnuggets
NOVEMBER 9, 2021
In this article we are discussing that HDF5 is one of the most popular and reliable formats for non-tabular, numerical data. But this format is not optimized for deep learning work. This article suggests what kind of ML native data format should be to truly serve the needs of modern data scientists.
Confluent
NOVEMBER 9, 2021
Companies are increasingly moving to the cloud, undergoing a transition that is often a multi-year journey across incremental stages. Along this journey, many companies embrace hybrid cloud architectures, either temporarily […].
Speaker: Timothy Chan, PhD., Head of Data Science
Are you ready to move beyond the basics and take a deep dive into the cutting-edge techniques that are reshaping the landscape of experimentation? 🌐 From Sequential Testing to Multi-Armed Bandits, Switchback Experiments to Stratified Sampling, Timothy Chan, Data Science Lead, is here to unravel the mysteries of these powerful methodologies that are revolutionizing how we approach testing.
Rockset
NOVEMBER 9, 2021
There are multiple change data capture methods available when using a MySQL or Postgres database. Some of these methods overlap and are very similar regardless of which database technology you are using, others are different. Ultimately, we require a way to specify and detect what has changed and a method of sending those changes to a target system.
Big Data Tools
NOVEMBER 8, 2021
The lockdowns are back again in Moscow, which means that conferences are again out of the question for me for some time. The good news is that I had time to put together this new installment of our Data Engineering Annotated! Hi, I’m Pasha Finkelshteyn , and I’ll be your guide through this month’s news. I’ll offer my impressions of recent developments in the data engineering space and highlight new ideas from the wider community.
KDnuggets
NOVEMBER 10, 2021
A machine can only understand numbers. As a result, converting text to numbers, called embedding text, is an actively researched topic. In this article, we review different word embedding techniques for converting text into vectors.
Teradata
NOVEMBER 10, 2021
When it comes to making progress on climate change, banks have a critical role in translating commitments into actions by influencing where, how & when money is spent.
Speaker: Aarushi Kansal, AI Leader & Author and Tony Karrer, Founder & CTO at Aggregage
Software leaders who are building applications based on Large Language Models (LLMs) often find it a challenge to achieve reliability. It’s no surprise given the non-deterministic nature of LLMs. To effectively create reliable LLM-based (often with RAG) applications, extensive testing and evaluation processes are crucial. This often ends up involving meticulous adjustments to prompts.
RudderStack
NOVEMBER 9, 2021
This post details the implementation of graceful shutdown on Rudder Server. You'll find a number of anti-patterns & learn how to make exiting graceful in Go.
Big Data Tools
NOVEMBER 8, 2021
The lockdowns are back again in Moscow, which means that conferences are again out of the question for me for some time. The good news is that I had time to put together this new installment of our Data Engineering Annotated! Hi, I’m Pasha Finkelshteyn , and I’ll be your guide through this month’s news. I’ll offer my impressions of recent developments in the data engineering space and highlight new ideas from the wider community.
KDnuggets
NOVEMBER 9, 2021
Beginners in the field can often have many misconceptions about machine learning that sometimes can be a make-it-or-break-it moment for the individual switching careers or starting fresh. This article clearly describes the ground truth realities about learning new ML skills and eventually working professionally as a machine learning engineer.
Data Engineering Podcast
NOVEMBER 6, 2021
Summary Business intelligence is often equated with a collection of dashboards that show various charts and graphs representing data for an organization. What is overlooked in that characterization is the level of complexity and effort that are required to collect and present that information, and the opportunities for providing those insights in other contexts.
Advertisement
Are you trying to decide which entity resolution capabilities you need? It can be confusing to determine which features are most important for your project. And sometimes key features are overlooked. Get the Entity Resolution Evaluation Checklist to make sure you’ve thought of everything to make your project a success! The list was created by Senzing’s team of leading entity resolution experts, based on their real-world experience.
ProjectPro
NOVEMBER 8, 2021
Computer vision is one of the most trending and compelling subfields of artificial intelligence. You must have encountered and used the applications of computer vision without even knowing it. Whether it is quality control of crops through image classification or image processing for electronic deposits, computer vision techniques are transforming industries across the globe.
Cloudera
NOVEMBER 8, 2021
On November 11 th we celebrate Veterans and Armistice Day honoring those who have served in the military. To commemorate this special occasion, we spotlighted two Clouderans who have served in the military both in the United States and the United Kingdom. In case you missed our first installment check out this Blog about William Daily. In this second installment, I sat down with Clouderan Paul Wooding who served in the British Army.
KDnuggets
NOVEMBER 11, 2021
The notion of self-service BI tools caught an expectation that they could provide a magic formula for easily helping everyone understand all the data. But, such an end-result isn't occurring in practice. To identify a better approach, we need to take a step back and determine what problem is actually trying to be solved.
AltexSoft
NOVEMBER 10, 2021
In 1901, a woman by the name of Julia Davis Chandler published the recipe that changed the world for good. People saw the very first recipe of a peanut butter and jelly sandwich. While both a peanut butter sandwich and a jelly sandwich are great individually, it’s hard to argue that together they make the most epic combo complementing each other’s best flavoring qualities.
Speaker: Anne Steiner and David Laribee
As a concept, Developer Experience (DX) has gained significant attention in the tech industry. It emphasizes engineers’ efficiency and satisfaction during the product development process. As product managers, we need to understand how a good DX can contribute not only to the well-being of our development teams but also to the broader objectives of product success and customer satisfaction.
ProjectPro
NOVEMBER 11, 2021
Are you willing to learn computer vision but cannot find the perfect guide to start your learning? If yes, then here’s a blog that will help you attain your goal in the easiest way possible! Read this blog until the end to learn how you can learn computer vision from scratch as a beginner. Before diving into the blog further, let us first look at why now is the best time to hone some computer vision skills.
Monte Carlo
NOVEMBER 10, 2021
Monte Carlo’s field-level lineage helps data teams track column-level dependencies from ingestion in the data warehouse or lake to dashboards and reports in the BI layer. It’s Friday evening, and you are wrapping up after a long week of work. Before logging off, you make a schema change to a table in your warehouse – and don’t think twice as you close your laptop and get started on your weekend.
KDnuggets
NOVEMBER 9, 2021
After a pause, we will be resuming KDnuggets Top Blog Rewards Program, starting with blogs published on KDnuggets in December. The program will be bigger, with $3,000 (USD) divided among top 8 most viewed guest blogs. Original blogs rewarded at the rate of 3X of reposts. Submit your original blog to KDnuggets first !
Cloudera
NOVEMBER 9, 2021
SANTA CLARA, Calif., Nov. 9, 2021 – Cloudera, the enterprise data cloud company, today announced Cloudera Data Platform capabilities available to help federal agencies meet requirements of the Biden Administration’s Executive Order on improving the Nation’s cybersecurity. Cloudera is committed to supporting the federal government in adhering to this executive order with the company’s technology and special government rates. .
Speaker: Margaret-Ann Seger, Head of Product, Statsig
Experimentation is often seen as an aspirational practice, especially at smaller, fast-moving companies who are strapped for time and resources. So, how can you get your team making decisions in a more data-driven way while continuing to remain lean and maintaining ship velocity? In this webinar, Margaret-Ann Seger, Head of Product at Statsig, will teach you how to build an experimentation culture from the ground-up, graduating from just getting started with data-driven development to operating
Expert insights. Personalized for you.
130 
Let's personalize your content