Start Data Engineering
SEPTEMBER 16, 2021
1. Introduction 2. What is scaling & why do we need it? 3. Types of scaling 4. Choose your scaling strategy 5. Conclusion 6. Further reading 7. References 1. Introduction Choosing tools/frameworks to scale your data pipelines can be confusing. If you have struggled with Data pipelines that randomly crash Finding guides on how to scale your data pipelines from the ground up Then this post is for you.
Confluent
SEPTEMBER 16, 2021
The full inventory of three online Kafka Summits in 2021 is now complete. Kafka Summit Americas wrapped just yesterday. Being a part of the event team and the Program Committee, […].
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Cloudera
SEPTEMBER 15, 2021
Leveraging the Internet of Things (IoT) allows you to improve processes and take your business in new directions. But it requires you to live on the edge. That’s where you find the ability to empower IoT devices to respond to events in real time by capturing and analyzing the relevant data. Edge computing relies on squeezing the power and functionality of a data center into a micro site as close to data sources as possible to enable real-time tasks.
Teradata
SEPTEMBER 12, 2021
Groupon is modernizing with Vantage on AWS to better match its data & analytics with demands of its global business. The Cloud allows Groupon to better leverage infrastructure dollars, support more technology projects and capture opportunity.
Advertisement
Many organizations today are unlocking the power of their data by using graph databases to feed downstream analytics, enahance visualizations, and more. Yet, when different graph nodes represent the same entity, graphs get messy. Watch this essential video with Senzing CEO Jeff Jonas on how adding entity resolution to a graph database condenses network graphs to improve analytics and save your analysts time.
DataKitchen
SEPTEMBER 14, 2021
The post What Should Enterprises Do to Offset Future Technology Disruption? first appeared on DataKitchen.
Confluent
SEPTEMBER 15, 2021
GraphQL and Apache Kafka® are sometimes troubled with misconceptions. One of the reasons for this is that people are often familiar with one but not the other. GraphQL is mostly […].
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
Rockset
SEPTEMBER 14, 2021
We’re introducing a new Rockset Integration for Apache Kafka that offers native support for Confluent Cloud and Apache Kafka, making it simpler and faster to ingest streaming data for real-time analytics. This new integration comes on the heels of several new product features that make Rockset more affordable and accessible for real-time analytics including SQL-based rollups and transformations.
dbt Developer Hub
SEPTEMBER 14, 2021
Hello there, Do you remember? The 21st day of September? ? Course you do it was two days ago. Well that's a win in your bucket and the day's barely begun! So let's get a win for someone else -- like Jeremy Cohen, the dbt Core product manager. I'm sure you know that half of the updates in this email are pushed automatically when we upgrade everyone to the latest version of dbt Cloud ?
Confluent
SEPTEMBER 14, 2021
Data governance initiatives aim to manage the availability, integrity, and security of data used across an organization. With the explosion in volume, variety, and velocity of data powering the modern […].
ProjectPro
SEPTEMBER 14, 2021
Cloud technology is widely used, with 94% of enterprises already using one or multiple cloud services. In a couple of years, the public cloud market will reach $623.3 billion. There are abundant options available in the cloud technology market, with AWS and Openstack as the two trendy choices. Table of Contents AWS vs. OpenStack - A Head to Head Comparison OpenStack vs.
Speaker: Scott Sehlhorst
We know we want to create products which our customers find to be valuable. Whether we label it as customer-centric or product-led depends on how long we've been doing product management. There are three challenges we face when doing this. The obvious challenge is figuring out what our users need; the non-obvious challenges are in creating a shared understanding of those needs and in sensing if what we're doing is meeting those needs.
Datakin
SEPTEMBER 13, 2021
Blog Datakin in 104 seconds Written by Ross Turk on Sep 13, 2021 Hi! I’m Ross from Datakin I’d like to show you a new approach to keeping your pipelines running smoothly. Datakin observes your jobs as they run, collecting metadata that helps you understand how data flows through your ecosystem. We believe that lineage provides the context required to keep troubleshooting and resolution times low.
Zalando Engineering
SEPTEMBER 12, 2021
2016 - First attempt at rolling out SRE Welcome to the first installment of our three part series following Zalando’s SRE journey. Be sure to come back for the other two, with the next one being published in a week. Site Reliability Engineering (SRE) is a recent discipline in the Software Engineering field that is growing in popularity, with many companies turning to this new way of working to solve their operational issues, or to support its growing scale.
Teradata
SEPTEMBER 14, 2021
Learn what Mitsui Sumitomo Insurance, one of Japan’s leading insurance and finance groups, has achieved through leveraging the power of data analytics.
RudderStack
SEPTEMBER 16, 2021
Learn how Touchless used a data-first design approach and leveraged RudderStack to help Wavedirect 4X Leads with dynamically generated SEO landing pages.
Speaker: Timothy Chan, PhD., Head of Data Science
Are you ready to move beyond the basics and take a deep dive into the cutting-edge techniques that are reshaping the landscape of experimentation? 🌐 From Sequential Testing to Multi-Armed Bandits, Switchback Experiments to Stratified Sampling, Timothy Chan, Data Science Lead, is here to unravel the mysteries of these powerful methodologies that are revolutionizing how we approach testing.
Preset
SEPTEMBER 13, 2021
Here we go step by step to build an open-source ingestion layer with Airbyte.
ProjectPro
SEPTEMBER 16, 2021
87% of Data Science Projects never make it to production - VentureBeat According to an analytics firm, Cognilytica, the MLOps market is anticipated to be worth $4 billion by end of 2025. Jobs over the next decade will be built on top of Data Science, but for production. Data Science has flourished over the decade on the promise that organizations will leverage analytics for profitable business decision-making.
Teradata
SEPTEMBER 12, 2021
The new Vantage BYOM feature allows data scientists and data engineers to finally operationalize all their predictive models. Find out more.
RudderStack
SEPTEMBER 14, 2021
This guide covers how to send data from your website to customer.io in less than five minutes.
Speaker: Aarushi Kansal, AI Leader & Author and Tony Karrer, Founder & CTO at Aggregage
Software leaders who are building applications based on Large Language Models (LLMs) often find it a challenge to achieve reliability. It’s no surprise given the non-deterministic nature of LLMs. To effectively create reliable LLM-based (often with RAG) applications, extensive testing and evaluation processes are crucial. This often ends up involving meticulous adjustments to prompts.
Data Engineering Podcast
SEPTEMBER 12, 2021
Summary The Cassandra database is one of the first open source options for globally scalable storage systems. Since its introduction in 2008 it has been powering systems at every scale. The community recently released a new major version that marks a milestone in its maturity and stability as a project and database. In this episode Ben Bromhead, CTO of Instaclustr, shares the challenges that the community has worked through, the work that went into the release, and how the stability and testing
ProjectPro
SEPTEMBER 16, 2021
Looking to land a job as a data analyst or a data scientist, SQL is a must-have skill on your resume. Everyone uses SQL to query data and perform analysis, from the biggest names in tech like Amazon, Netflix, and Google to fast-growing seed-stage startups in data. Before the world was taken over by the buzz of data science and analytics, data management still existed.
Cloudera
SEPTEMBER 13, 2021
About Cruise Control. There are two big gaps in the Apache Kafka project when we think of operating a cluster. The first is monitoring the cluster efficiently and the second is managing failures and changes in the cluster. There are no solutions for these inside the Kafka project but there are many good 3rd party tools for both problems. Cruise Control is one of the earliest open source tools to provide a solution for the failure management problem but lately for the monitoring problem as well.
Netflix Tech
SEPTEMBER 13, 2021
Written by Jose Fernandez , Arthur Gonigberg , Julia Knecht , and Patrick Thomas In 2017, Netflix Studios was hitting an inflection point from a period of merely rapid growth to the sort of explosive growth that throws “how do we scale?” into every conversation. The vision was to create a “Studio in the Cloud”, with applications supporting every part of the business from pitch to play.
Advertisement
Are you trying to decide which entity resolution capabilities you need? It can be confusing to determine which features are most important for your project. And sometimes key features are overlooked. Get the Entity Resolution Evaluation Checklist to make sure you’ve thought of everything to make your project a success! The list was created by Senzing’s team of leading entity resolution experts, based on their real-world experience.
DataKitchen
SEPTEMBER 17, 2021
Below is our final post (5 of 5) on combining data mesh with DataOps to foster innovation while addressing the challenges of a data mesh decentralized architecture. We see a DataOps process hub like the DataKitchen Platform playing a central supporting role in successfully implementing a data mesh. DataOps excels at the type of workflow automation that can coordinate interdependent domains, manage order-of-operations issues and handle inter-domain communication.
ProjectPro
SEPTEMBER 11, 2021
This blog contains a list of business analyst interview questions and answers. You will find it helpful if you are a hiring manager who is looking for business analyst questions to ask during an interview and also if you are a job seeker who is interested in business analyst jobs. Table of Contents Role of a Business Analyst: Skills and Opportunities 50 Business Analyst Interviews Questions and Answers Junior Business Analyst Interview Questions/Entry-level Business Analyst Interview Questions T
Cloudera
SEPTEMBER 14, 2021
In June, we announced the beginning of a new chapter for Cloudera, with a mission to make data and analytics easy and accessible, for everyone. With transformation comes change, and today I’m thrilled to announce the promotion of Sudhir “Suds” Menon, Ram Venkatesh and Paul Codding, three leaders driving our mission forward. The foundation of our mission is a move to a hybrid data cloud platform , an evolution of our Cloudera Data Platform , a hybrid and multi-cloud solution purpose-built with th
Netflix Tech
SEPTEMBER 15, 2021
By Ricky Gardiner , Alex Borysov Background In our previous post , we discussed how we utilize FieldMask as a solution when designing our APIs so that consumers can request the data they need when fetched via gRPC. In this blog post we will continue to cover how Netflix Studio Engineering uses FieldMask for mutation operations such as update and remove.
Speaker: Anne Steiner and David Laribee
As a concept, Developer Experience (DX) has gained significant attention in the tech industry. It emphasizes engineers’ efficiency and satisfaction during the product development process. As product managers, we need to understand how a good DX can contribute not only to the well-being of our development teams but also to the broader objectives of product success and customer satisfaction.
ProjectPro
SEPTEMBER 14, 2021
As a data science aspirant, you would have probably come across the following phrase more than once: “A data scientist is a person who is better at statistics than any programmer and better at programming than any statistician.” Before data science became a well-known career path, companies would hire statisticians to process their data and develop insights based on trends observed.
ProjectPro
SEPTEMBER 16, 2021
NoSQL databases are the new-age solutions to distributed unstructured data storage and processing. The speed, scalability, and fail-over safety offered by NoSQL databases are needed in the current times in the wake of Big Data Analytics and Data Science technologies. The edge that NoSql provides over their SQL counterparts is high scalability and faster read/write performances, highly appreciated features in Distributed Systems.
Expert insights. Personalized for you.
130 
Let's personalize your content