Data Engineering Podcast

JULY 28, 2019

Summary The ETL pattern that has become commonplace for integrating data from multiple sources has proven useful, but complex to maintain. For a small number of sources it is a tractable problem, but as the overall complexity of the data ecosystem continues to expand it may be time to identify new ways to tame the deluge of information. In this episode Tim Ward, CEO of CluedIn, explains the idea of eventual connectivity as a new paradigm for data integration.

Data Integration 100

Data Integration Metadata Media Architecture 100

KDnuggets

JULY 26, 2019

Different neural network architectures excel in different tasks. This particular article focuses on crafting convolutional neural networks in Python using TensorFlow and Keras.

Python 123

Python Architecture 123

Cloudera

JULY 10, 2019

Open source has been core to the missions of both Hortonworks and Cloudera and central to our values and culture. With more than 700 engineers in the new Cloudera, our company writes a prodigious amount of open source code each year that’s contributed to more than 30 different open source projects. We’re also a very innovative open source company, having collectively launched more than a dozen new open source projects since the founding of the two companies. .

Consulting 106

Consulting Kafka Project Data Science 106

Teradata

JULY 9, 2019

Integrated data and analytics has a proven track record of helping organize operations, enhance customer experience and improve revenue and market growth.

Data 104

Data 104

Speaker: Timothy Chan, PhD., Head of Data Science

Are you ready to move beyond the basics and take a deep dive into the cutting-edge techniques that are reshaping the landscape of experimentation? 🌐 From Sequential Testing to Multi-Armed Bandits, Switchback Experiments to Stratified Sampling, Timothy Chan, Data Science Lead, is here to unravel the mysteries of these powerful methodologies that are revolutionizing how we approach testing.

Data Science

Confluent

JULY 11, 2019

I’ve been using KSQL from Confluent since its first developer preview in 2017. Reading, writing, and transforming data in Apache Kafka ® using KSQL is an effective way to rapidly deliver event streaming applications for clients (e.g., streaming insurance events ). Plus, I’ve also had the opportunity to deploy KSQL in some not-so-serious hobby projects (see Noise Mapping with KSQL, a Raspberry Pi and a Software-Defined Radio and ML and KSQL Let Me Know When I’ve Left the Heater Running ).

Kafka 83

Kafka Insurance SQL Architecture 83

Netflix Tech

JULY 30, 2019

v2.0 and beyond By Anoop Panicker and Kishore Banala Conductor is a workflow orchestration engine developed and open-sourced by Netflix. If you’re new to Conductor, this earlier blogpost and the documentation should help you get started and acclimatized to Conductor. Netflix Conductor: A microservices orchestrator In the last two years since inception, Conductor has seen wide adoption and is instrumental in running numerous core workflows at Netflix.

Metadata 75

Metadata Media AWS Transportation 75

KDnuggets

JULY 29, 2019

Check out our latest Top 10 Most Popular Data Science and Machine Learning podcasts available on iTunes. Stay up to date in the field with these recent episodes and join in with the current data conversations.

Machine Learning 123

Machine Learning Data Science Data 123

Pandora Engineering

JULY 29, 2019

Credit: Kanok Sulaiman Disclaimer: These are my experiences from being a Pandora software developer intern in the summer of 2019. All opinions expressed are my own, and represent no one except myself. I recently spent the last summer of my undergraduate program as an intern for Pandora Media in Oakland, CA. I gained a lot from my experience, and I’m writing this post to detail the application process, the lessons that I learned, and the company culture.

Java 52

Java Recruitment Algorithm Computer Science 52

Teradata

JULY 16, 2019

Teradata Vantage's NewSQL Engine's performance-enhancing options include column-row hybrid partitioning. Find out how to take advantage of this great feature.

Engineering 75

Engineering 75

Cloud Academy

JULY 16, 2019

Designing a streaming data pipeline presents many challenges, particularly around specific technology requirements. When designing a cloud-based solution, an architect is no longer faced with the question, “How do I get this job done with the technology we have?” but rather, “What is the right technology to support my use case?” In this blog post, we will walk through some initial scoping steps and walk through an example.

Manufacturing 52

Manufacturing MySQL Data Cloud 52

Speaker: Anne Steiner and David Laribee

As a concept, Developer Experience (DX) has gained significant attention in the tech industry. It emphasizes engineers’ efficiency and satisfaction during the product development process. As product managers, we need to understand how a good DX can contribute not only to the well-being of our development teams but also to the broader objectives of product success and customer satisfaction.

Engineering

Zalando Engineering

JULY 14, 2019

Project Highlights Kopf - Kubernetes Operator Pythonic Framework now supports built-in resources and can be used to write controllers of any kind (pods, namespaces, mixed), not only of custom resources. Check out the latest release for more details [link] Skipper publishes new releases weekly. Some of the important features were implemented such as support to proxy Kubernetes API server and support Kubernetes externalName services from ingress.

AWS 52

AWS SQL Python Data Engineering 52

Data Engineering Podcast

JULY 14, 2019

Summary Successful machine learning and artificial intelligence projects require large volumes of data that is properly labelled. The challenge is that most data is not clean and well annotated, requiring a scalable data labeling process. Ideally this process can be done using the tools and systems that already power your analytics, rather than sending data into a black box.

Machine Learning 100

Machine Learning Media Cloud Data 100

KDnuggets

JULY 24, 2019

Recently, researchers from the Google Brain team published a paper proposing a new method called Concept Activation Vectors (CAVs) that takes a new angle to the interpretability of deep learning models.

Deep Learning 122

Deep Learning 122

InData Labs

JULY 11, 2019

In the highly competitive business world of today, data reign supreme. Customer personal data, comprehensive operating statistics, sales figures, or inter-company information may play a core role in strategic decision making. It’s vital to keep an eye on the quantity and quality of data that can be captured and extracted from different web sources.

IT 52

IT Data Machine Learning Data Engineering 52

Speaker: Aarushi Kansal, AI Leader & Author and Tony Karrer, Founder & CTO at Aggregage

Software leaders who are building applications based on Large Language Models (LLMs) often find it a challenge to achieve reliability. It’s no surprise given the non-deterministic nature of LLMs. To effectively create reliable LLM-based (often with RAG) applications, extensive testing and evaluation processes are crucial. This often ends up involving meticulous adjustments to prompts.

Building

Teradata

JULY 28, 2019

In a cost-cutting culture, directly linking data projects to top business initiatives is a good way to keep them from getting clipped. Learn more.

Data 73

Data Project 73

Cloudera

JULY 15, 2019

At a time when machine learning, deep learning, and artificial intelligence capture an outsize share of media attention, jobs requiring SQL skills continue to vastly outnumber jobs requiring those more advanced skills. Influential data scientists often point to SQL as the most important yet underrated skill for anyone who works with data. SQL is today—and will remain for the foreseeable future—a vital foundational skill for a wide range of data professionals working in different roles across dif

Education 56

Education Data Analysis Big Data SQL 56

Rockset

JULY 30, 2019

Published on Forbes All businesses today are a series of real-time events. But what separates the good from the great is how they capture and operationalize that data. Companies like Uber have talked in-depth about how they use real-time analytics to create seamless trip experiences, from determining the most convenient rider pick-up points to predicting the fastest routes.

BI 40

BI Medical Manufacturing Retail 40

Data Engineering Podcast

JULY 1, 2019

Summary Anomaly detection is a capability that is useful in a variety of problem domains, including finance, internet of things, and systems monitoring. Scaling the volume of events that can be processed in real-time can be challenging, so Paul Brebner from Instaclustr set out to see how far he could push Kafka and Cassandra for this use case. In this interview he explains the system design that he tested, his findings for how these tools were able to work together, and how they behaved at diffe

Kafka 100

Kafka Finance Media Architecture 100

Advertisement

Are you trying to decide which entity resolution capabilities you need? It can be confusing to determine which features are most important for your project. And sometimes key features are overlooked. Get the Entity Resolution Evaluation Checklist to make sure you’ve thought of everything to make your project a success! The list was created by Senzing’s team of leading entity resolution experts, based on their real-world experience.

Project

KDnuggets

JULY 29, 2019

At my workplace, we produce a lot of functional prototypes for our clients. Because of this, I often need to make Small Data go a long way. In this article, I’ll share 7 tips to improve your results when prototyping with small datasets.

Datasets 121

Datasets Data 121

Confluent

JULY 17, 2019

Together, MongoDB and Apache Kafka ® make up the heart of many modern data architectures today. Integrating Kafka with external systems like MongoDB is best done though the use of Kafka Connect. This API enables users to leverage ready-to-use components that can stream data from external systems into Kafka topics, as well as stream data from Kafka topics into external systems.

MongoDB 21

MongoDB Kafka Medical Database 21

Teradata

JULY 14, 2019

Analytics can help enterprises answer the toughest business questions by leveraging all of the data across an organization.

Data 80

Data 80

Cloudera

JULY 9, 2019

Every business aims to deliver products and services quickly and efficiently based upon customer wants and needs. Today, much of that speed and efficiency relies on insights driven by big data. Yet big data management often serves as a stumbling block, because many businesses continue to struggle with how to best capture and analyze their data. Unorganized data presents another roadblock.

Big Data 78

Big Data Data Management Management Cloud 78

Advertisement

Outdated or absent analytics won’t cut it in today’s data-driven applications – not for your end users, your development team, or your business. That’s what drove the five companies in this e-book to change their approach to analytics. Download this e-book to learn about the unique problems each company faced and how they achieved huge returns beyond expectation by embedding analytics into applications.

Building

Rockset

JULY 18, 2019

Rockset is a schemaless SQL data platform. It is designed to support SQL on raw data. While most SQL databases are strongly and statically typed, data within Rockset is strongly but dynamically typed. Dynamic typing makes it difficult for us to adopt off-the-shelf SQL query optimizers since they are designed for statically typed data where the types of the columns are known ahead of time.

SQL 40

SQL Raw Data Designing Architecture 40

Data Engineering Podcast

JULY 8, 2019

Summary The market for data warehouse platforms is large and varied, with options for every use case. ClickHouse is an open source, column-oriented database engine built for interactive analytics with linear scalability. In this episode Robert Hodges and Alexander Zaitsev explain how it is architected to provide these features, the various unique capabilities that it provides, and how to run it in production.

Data Warehouse 100

Data Warehouse MySQL Data Lake Hadoop 100

KDnuggets

JULY 31, 2019

I had a feeling that R has developed as a language to such a degree that many of us are using it now in completely different ways. This means that there are likely to be numerous tricks, packages, functions, etc that each of us use, but that others are completely unaware of, and would find useful if they knew about them.

IT 120

IT Data Science Data 120

Confluent

JULY 10, 2019

Building off part 1 where we discussed an event streaming architecture that we implemented for a customer using Apache Kafka, KSQL, and Kafka Streams, and part 2 where we discussed how Gradle helped us address the challenges we faced developing, building, and deploying the KSQL portion of our application, here in part 3, we’ll explore using Gradle to build and deploy KSQL user-defined functions (UDFs) and Kafka Streams microservices.

Kafka 87

Kafka Java Bytes SQL 87

Speaker: David Bard, Principal at VP Product Coaching

In the fast-paced world of digital innovation, success is often accompanied by a multitude of challenges - like the pitfalls lurking at every turn, threatening to derail the most promising projects. But fret not, this webinar is your key to effective product development! Join us for an enlightening session to empower you to lead your team to greater heights.

Certification

Teradata

JULY 23, 2019

Large enterprises are investing heavily in cloud-based analytics technologies. What qualities should they be looking for in these cloud vendors? Find out more.

Cloud 69

Cloud IT Technology 69

Advancing Analytics: Data Engineering

JULY 2, 2019

It seems these days that every person I talk to is either a scientist, engineer or architect, we’re fairly obsessed with aligning our technical roles to respected professions that denote the amount of education & training that go into it – and that’s fair given how much time & effort goes into attaining these roles… but it really doesn’t help us define them.

Business Intelligence 52

Business Intelligence Data Engineering Data Engineer Engineering 52

Rockset

JULY 9, 2019

One of the main hindrances to getting value from our data is that we have to get data into a form that’s ready for analysis. It sounds simple, but it rarely is. Consider the hoops we have to jump through when working with semi-structured data, like JSON, in relational databases such as PostgreSQL and MySQL. JSON in Relational Databases In the past, when it came to working with JSON data, we’ve had to choose between tools and platforms that worked well with JSON or tools that provided good suppor

Relational Database 40

Relational Database PostgreSQL MySQL Database 40

Netflix Tech

JULY 1, 2019

Bringing Rich Experiences to Memory-Constrained TV Devices By Jason Munning, Archana Kumar, Kris Range Netflix has over 148M paid members streaming on more than half a billion devices spanning over 1,900 different types. In the TV space alone, there are hundreds of device types that run the Netflix app. We need to support the same rich Netflix experience on not only high-end devices like the PS4 but also memory and processor-constrained consumer electronic devices that run a similar chipset as w

Designing 66

Designing Bytes Electronics Project 66

Speaker: Jon Harmer, Product Manager for Google Cloud

Move from feature factory to customer outcomes and drive impact in your business! This session will provide you with a comprehensive set of tools to help you develop impactful products by shifting from output-based thinking to outcome-based thinking. You will deepen your understanding of your customers and their needs as well as identifying and de-risking the different kinds of hypotheses built into your roadmap.

Certification