August, 2020

article thumbnail

Why You Need Data Engineers And Data Scientists To Be Successful!

Team Data Science

Data Science , Artificial Intelligence and Machine Learning. These topics are currently the hype in the field of Data Science. Everyone wants to become a Data Scientist. But isn't the work being done in the field of Data Engineereing the real MVP? Isn't it important to have Data Scientists AND Data Engineers on board to make a project successful? Yes, it is!

article thumbnail

Benchmarking Apache Kafka, Apache Pulsar, and RabbitMQ: Which is the fastest?

Confluent

Apache Kafka® is one of the most popular event streaming systems. There are many ways to compare systems in this space, but one thing everyone cares about is performance. Kafka […].

Kafka 145
Insiders

Sign Up for our Newsletter

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

article thumbnail

Designing Edge Gateway, Uber’s API Lifecycle Management Platform

Uber Engineering

The making of Edge Gateway, the highly-available and scalable self-serve gateway to configure, manage, and monitor APIs of every business domain at Uber. Evolution of Uber’s API gateway. In October 2014, Uber had started its journey of scale in what … The post Designing Edge Gateway, Uber’s API Lifecycle Management Platform appeared first on Uber Engineering Blog.

Designing 144
article thumbnail

Optimized shot-based encodes for 4K: Now streaming!

Netflix Tech

by Aditya Mavlankar , Liwei Guo , Anush Moorthy and Anne Aaron Netflix has an ever-expanding collection of titles which customers can enjoy in 4K resolution with a suitable device and subscription plan. Netflix creates premium bitstreams for those titles in addition to the catalog-wide 8-bit stream profiles¹. Premium features comprise a title-dependent combination of 10-bit bit-depth, 4K resolution, high frame rate (HFR) and high dynamic range (HDR) and pave the way for an extraordinary viewing

Media 131
article thumbnail

From Developer Experience to Product Experience: How a Shared Focus Fuels Product Success

Speaker: Anne Steiner and David Laribee

As a concept, Developer Experience (DX) has gained significant attention in the tech industry. It emphasizes engineers’ efficiency and satisfaction during the product development process. As product managers, we need to understand how a good DX can contribute not only to the well-being of our development teams but also to the broader objectives of product success and customer satisfaction.

article thumbnail

Building A Better Data Warehouse For The Cloud At Firebolt

Data Engineering Podcast

Summary Data warehouse technology has been around for decades and has gone through several generational shifts in that time. The current trends in data warehousing are oriented around cloud native architectures that take advantage of dynamic scaling and the separation of compute and storage. Firebolt is taking that a step further with a core focus on speed and interactivity.

article thumbnail

Data for Enterprise AI: at the very forefront of innovation

Cloudera

2020 may well go down as the year where what seems impossible today, did become possible tomorrow. It’s been a year filled with disruption and uncertainty. One day we were all going to the office, and the next we were working from home. Businesses had to literally switch operations, and enable better collaboration and access to data in an instant — while streamlining processes to accommodate a whole new way of doing things.

Banking 123

More Trending

article thumbnail

How Tencent PCG Uses Apache Kafka to Handle 10 Trillion+ Messages Per Day

Confluent

As one of the world’s biggest internet-based platform companies, Tencent uses technology to enrich the lives of users and assist the digital upgrade of enterprises. An example product is the […].

Kafka 139
article thumbnail

Teradata Vantage: Born for Cloud Before Cloud Was Born

Teradata

Teradata Workload Management enables Vantage to be fully optimized for cloud & hybrid deployments & to efficiently deliver the lowest cost for enterprise analytics.

Cloud 124
article thumbnail

Improving our video encodes for legacy devices

Netflix Tech

by Mariana Afonso , Anush Moorthy , Liwei Guo , Lishan Zhu , Anne Aaron Netflix has been one of the pioneers of streaming video-on-demand content?—?we announced our intention to stream video over 13 years ago, in January 2007?—?and have only increased both our device and content reach since then. Given the global nature of the service and Netflix’s commitment to creating a service that members enjoy, it is not surprising that we support a wide variety of streaming devices, from set-top-boxes and

article thumbnail

Metadata Management And Integration At LinkedIn With DataHub

Data Engineering Podcast

Summary In order to scale the use of data across an organization there are a number of challenges related to discovery, governance, and integration that need to be solved. The key to those solutions is a robust and flexible metadata management system. LinkedIn has gone through several iterations on the most maintainable and scalable approach to metadata, leading them to their current work on DataHub.

Metadata 100
article thumbnail

Peak Performance: Continuous Testing & Evaluation of LLM-Based Applications

Speaker: Aarushi Kansal, AI Leader & Author and Tony Karrer, Founder & CTO at Aggregage

Software leaders who are building applications based on Large Language Models (LLMs) often find it a challenge to achieve reliability. It’s no surprise given the non-deterministic nature of LLMs. To effectively create reliable LLM-based (often with RAG) applications, extensive testing and evaluation processes are crucial. This often ends up involving meticulous adjustments to prompts.

article thumbnail

The Future Of The Telco Industry And Impact Of 5G & IoT – Part II

Cloudera

In part 2 of the series focusing on the impact of evolving technology on the telecom industry, we sat down with Vijay Raja, Director of Industry & Solutions Marketing at Cloudera to get his views on how the sector is changing and where it goes next. Hi Vijay, thank you so much for joining us again. To continue where we left off, as industry players continue to shift toward a more 5G centric network, how is 5G impacting the industry from a data perspective?

article thumbnail

Most important tools for Data Engineers

Team Data Science

There are a huge number of tools and platforms for data engineers. It's this enormous selection that makes it difficult for newcomers to filter out the really important tools. In the course of the Data Engineer Coaching I was able to gain important experience in this regard and would like to tell you the most important tools on this basis today! During the coaching sessions I saw that a lot of tools keep coming up all the time: Kafka, Spark and AWS.

article thumbnail

What’s New in Apache Kafka 2.6

Confluent

On behalf of the Apache Kafka® community, it is my pleasure to announce the release of Apache Kafka 2.6.0. This another exciting release with many new features and improvements. We’ll […].

Kafka 136
article thumbnail

Use of Modeling and Simulation for Understanding COVID-19 Dynamics

Teradata

This post presents a simulation framework that leverages several mathematical models to simulate the spread of diseases such as COVID-19 in urban environments.

113
113
article thumbnail

How to Build an Experimentation Culture for Data-Driven Product Development

Speaker: Margaret-Ann Seger, Head of Product, Statsig

Experimentation is often seen as an aspirational practice, especially at smaller, fast-moving companies who are strapped for time and resources. So, how can you get your team making decisions in a more data-driven way while continuing to remain lean and maintaining ship velocity? In this webinar, Margaret-Ann Seger, Head of Product at Statsig, will teach you how to build an experimentation culture from the ground-up, graduating from just getting started with data-driven development to operating

article thumbnail

Power BI Template App for Stripe

FreshBI

So, what is a Power BI Template App? A Power BI Template App is a published Power BI solution that can be used by any company that has the data platform for which the Template App was created. Can you imagine picking your entire Power BI Solution off the shelf - one crafted for your specific business needs and your specific data structure. Power BI Template Apps are designed to be such an out-of-the-box solution and this blog post is an example of such for a Power BI Solution for Stripe.

BI 52
article thumbnail

Closing The Loop On Event Data Collection With Iteratively

Data Engineering Podcast

Summary Event based data is a rich source of information for analytics, unless none of the event structures are consistent. The team at Iteratively are building a platform to manage the end to end flow of collaboration around what events are needed, how to structure the attributes, and how they are captured. In this episode founders Patrick Thompson and Ondrej Hrebicek discuss the problems that they have experienced as a result of inconsistent event schemas, how the Iteratively platform integrat

article thumbnail

Connect the Data Lifecycle: The power of data

Cloudera

There’s no doubt that cloud has become ubiquitous, and thank goodness for that in 2020. We wouldn’t have survived the challenges of this year without cloud. It’s supported everything, from the sudden changes in the way we work to the way we access healthcare and even shop for vital goods. While cloud is the vehicle, it’s what sits on it that makes it so valuable — data.

article thumbnail

Analytics-on-the-fly: from batch to real-time user engagement

Rockset

It was the winter of 2007 when I logged into my newly created Facebook account for the very first time and I was amazed to see Facebook immediately show me three of my friends with whom I had lost touch since elementary school. One of them was working in London in a multinational bank, the other one was an engineer at Google in their Silicon Valley office office and the third one was running a restaurant in my town of Guwahati, a sleepy town on the India-Myanmar border.

Hadoop 52
article thumbnail

Entity Resolution Checklist: What to Consider When Evaluating Options

Are you trying to decide which entity resolution capabilities you need? It can be confusing to determine which features are most important for your project. And sometimes key features are overlooked. Get the Entity Resolution Evaluation Checklist to make sure you’ve thought of everything to make your project a success! The list was created by Senzing’s team of leading entity resolution experts, based on their real-world experience.

article thumbnail

Testing Kafka Streams – A Deep Dive

Confluent

Tools for automated testing of Kafka Streams applications have been available to developers ever since the technology’s genesis. Although these tools are very useful in practice, this blog post will […].

Kafka 127
article thumbnail

Architecting for Today’s Hybrid Analytic Ecosystem

Teradata

A modern analytic ecosystem embraces a hybrid approach and leverages the right technologies to meet the needs at the right cost/value ratio. Read more.

article thumbnail

Building a Sync Engine

Grouparoo

So you have data in your product database and you need to synchronize it with something else. Maybe you need to update a CRM or email system like Mailchimp , HubSpot , or Braze. Maybe it is more of an ETL thing and you need to move the data into Redshift or Snowflake. In all cases, what we have here is a need for a sync engine. A sync engine monitors a source (your product database) for changes in order to process them in some way (update an external system).

article thumbnail

A Practical Introduction To Graph Data Applications

Data Engineering Podcast

Summary Finding connections between data and the entities that they represent is a complex problem. Graph data models and the applications built on top of them are perfect for representing relationships and finding emergent structures in your information. In this episode Denise Gosnell and Matthias Broecheler discuss their recent book, the Practitioner’s Guide To Graph Data, including the fundamental principles that you need to know about graph structures, the current state of graph suppor

NoSQL 100
article thumbnail

The Big Payoff of Application Analytics

Outdated or absent analytics won’t cut it in today’s data-driven applications – not for your end users, your development team, or your business. That’s what drove the five companies in this e-book to change their approach to analytics. Download this e-book to learn about the unique problems each company faced and how they achieved huge returns beyond expectation by embedding analytics into applications.

article thumbnail

Streaming Analytics in the Real World

Cloudera

From leading banks, and insurance organizations to some of the largest telcos, manufacturers, retailers, healthcare and pharma, organizations across diverse verticals lead the way with real-time data and streaming analytics. These businesses use data-fueled insights to enhance the customer experience, reduce costs, and increase revenues. And Cloudera is at the heart of enabling these real-time data driven transformations. .

Insurance 103
article thumbnail

Case Study: Matter Uses Rockset to Bring AI-Powered Sustainable Insights to Investors

Rockset

The effects of climate change and inequality are threatening societies across the world, but there is still an annual funding gap of US$2.5 trillion to achieve the UN Sustainable Development Goals by 2030. A substantial amount of that money is expected to come from private sources like pension funds, but institutional investors often struggle to efficiently incorporate sustainability into their investment decisions.

NoSQL 40
article thumbnail

An Overview of Confluent Cloud Security Controls

Confluent

Whether you are a developer working on a cool new real-time application or an architect formulating the plan to reap the benefits of event streaming for the organisation, the subject […].

Cloud 104
article thumbnail

Move Fast – But Don’t Break Things

Teradata

Agile practices in the retail sector can deliver fast & compelling returns, but they can also lead to fragmentation, data silos, & unnecessary complexity. Learn more.

Retail 72
article thumbnail

The Path to Product Excellence: Avoiding Common Pitfalls and Enhancing Communication

Speaker: David Bard, Principal at VP Product Coaching

In the fast-paced world of digital innovation, success is often accompanied by a multitude of challenges - like the pitfalls lurking at every turn, threatening to derail the most promising projects. But fret not, this webinar is your key to effective product development! Join us for an enlightening session to empower you to lead your team to greater heights.

article thumbnail

How to Format Zendesk Tags

Grouparoo

In the process of integrating Grouparoo with Zendesk , I searched the documentation for the right way to format tags, but was unable to find it. I thought I'd write up a guide to help others on the same journey. In case you are "that person" and just want the answer, here it is: Tags needs to be lowercase and not have any spaces. You can have underscores.

article thumbnail

How Nielsen Scaled Access To Data Analytics Using Apache Superset

Preset

Learn why Nielsen migrated to Superset for visualization and dashboards.

article thumbnail

The Future Of The Telco Industry And Impact Of 5G & IoT – Part 1

Cloudera

Technology like IoT, edge computing and 5G are changing the face of CSPs. Communication Service Providers (CSPs) are in the middle of a data-driven transformation. The current scale and pace of change in the Telecommunications sector is being driven by the rapid evolution of new technologies like the Internet of Things (IoT), 5G, advanced data analytics and edge computing.

article thumbnail

Announcing the New Rockset Developer Tools

Rockset

We are excited to release a new ecosystem of developer tools intended to help advanced users edit, execute, and deploy Query Lambdas from a local development environment, while integrating seamlessly with Version Control and CI/CD systems. Right now, we are releasing three tools into an Open Beta: Rockset CLI Rockset VS Code Extension Rockset Developer UI In this blog, we will explore best practices for using these tools together.

SQL 40
article thumbnail

Driving Business Impact for PMs

Speaker: Jon Harmer, Product Manager for Google Cloud

Move from feature factory to customer outcomes and drive impact in your business! This session will provide you with a comprehensive set of tools to help you develop impactful products by shifting from output-based thinking to outcome-based thinking. You will deepen your understanding of your customers and their needs as well as identifying and de-risking the different kinds of hypotheses built into your roadmap.