Data Engineering Podcast

SEPTEMBER 3, 2023

Summary Cloud data warehouses and the introduction of the ELT paradigm has led to the creation of multiple options for flexible data integration, with a roughly equal distribution of commercial and open source options. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products.

Data Integration 162

Data Integration BI SQL Python 162

Data Engineering Podcast

FEBRUARY 26, 2023

Summary There has been a lot of discussion about the practical application of data mesh and how to implement it in an organization. Jean-Georges Perrin was tasked with designing a new data platform implementation at PayPal and wound up building a data mesh. Is a data mesh ever "done"? We feel your pain.

Building 147

Building Metadata Machine Learning Data Integration 147

Data Engineering Podcast

JULY 2, 2023

In this episode Razi Raziuddin shares how data engineering teams can support the machine learning workflow through the development and support of systems that empower data scientists and ML engineers to build and maintain their own features. How is this distinct from other forms of data pipeline development and delivery?

Machine Learning 130

Machine Learning Data Engineering Data Engineer Engineering 130

Data Engineering Podcast

JULY 16, 2023

Summary Data has been one of the most substantial drivers of business and economic value for the past few decades. In his recent book "Datapreneurs" he reflects on the people and businesses that he has known and worked with and how they relied on data to deliver valuable services and drive meaningful change.

SQL 130

SQL Machine Learning Python Data 130

Snowflake

MARCH 27, 2024

As organizations seek to drive more value from their data, observability plays a vital role in ensuring the performance, security and reliability of applications and pipelines while helping to reduce costs. One of our partners in this area is Observe , which offers a SaaS observability product that is built and operated on the Data Cloud.

Cloud 117

Cloud Data Governance Government Data 117

Precisely

NOVEMBER 14, 2023

When speaking to organizations about data integrity , and the key role that both data governance and location intelligence play in making more confident business decisions, I keep hearing the following statements: “For any organization, data governance is not just a nice-to-have! “ “Everyone knows that 80% of data contains location information.

Data Governance 119

Data Governance Government Metadata Retail 119

Data Engineering Podcast

JULY 9, 2023

Summary For business analytics the way that you model the data in your warehouse has a lasting impact on what types of questions can be answered quickly and easily. In this episode Maxime Beauchemin of Airflow and Superset fame shares his vision for the entity-centric data model and how you can incorporate it into your own warehouse design.

Database-centric 130

Database-centric Machine Learning SQL Data Engineering 130

Knowledge Hut

APRIL 25, 2024

The implementation of Data Science in agriculture is truly groundbreaking for farmers globally. Recent press claims that the DATOS Project used data from remote sensing along with artificial intelligence, machine learning, and other approaches to Data Science for agriculture. Are available online.

Data Science 98

Data Science Food Data Government 98

Data Engineering Podcast

NOVEMBER 13, 2022

Summary Building data products is an undertaking that has historically required substantial investments of time and talent. With the rise in cloud platforms and self-serve data technologies the barrier of entry is dropping. Atlan is the metadata hub for your data ecosystem.

Building 130

Building Metadata MongoDB MySQL 130

Snowflake

MARCH 5, 2024

An estimated 90% of the world’s data is unstructured, much of it in the form of visual content such as images and videos. Insights from analyzing this visual data can open up powerful new use cases that significantly boost productivity and efficiency, but enterprises need sophisticated computer vision technologies to achieve this.

Cloud 124

Cloud Medical Manufacturing Retail 124

Towards Data Science

OCTOBER 21, 2023

Efficient geospatial manipulations for OSM map data Photo by Tabea Schimpf on Unsplash If you’ve worked with OSM data before, you know it’s not the easiest to extract. OSM data can be huge, and finding performant solutions for what you want to analyze is often a challenge. Let’s get into what this library can do! ?

Datasets 78

Datasets Data Data Science Python 78

Data Engineering Podcast

MARCH 18, 2023

Summary As with all aspects of technology, security is a critical element of data applications, and the different controls can be at cross purposes with productivity. In this episode Yoav Cohen from Satori shares his experiences as a practitioner in the space of data security and how to align with the needs of engineers and business users.

Data Security 130

Data Security Machine Learning Data Science Metadata 130

Data Engineering Podcast

MARCH 5, 2023

Summary The ecosystem for data professionals has matured to the point that there are a large and growing number of distinct roles. With the scope and importance of data steadily increasing it is important for organizations to ensure that everyone is aligned and operating in a positive environment. We feel your pain.

Building 147

Building Database Design Metadata Machine Learning 147

Data Engineering Podcast

NOVEMBER 6, 2022

Summary A lot of the work that goes into data engineering is trying to make sense of the "data exhaust" from other applications and services. Atlan is the metadata hub for your data ecosystem. Data engineers don’t enjoy writing, maintaining, and modifying ETL pipelines all day, every day.

Building 130

Building IT Metadata MongoDB 130

Knowledge Hut

MARCH 7, 2024

The year 2024 saw some enthralling changes in volume and variety of data across businesses worldwide. The surge in data generation is only going to continue. Foresighted enterprises are the ones who will be able to leverage this data for maximum profitability through data processing and handling techniques.

Big Data 98

Big Data Bytes Data Governance Raw Data 98

Lyft Engineering

APRIL 24, 2024

At Lyft, data scientists tackle challenging technical problems every day. To support and empower our data scientists, Lyft’s Technical Learning Council (TLC) provides diverse and high-quality continuous learning opportunities to hone their technical skills. Written by Shumpei Goke and Jinshu Niu Why Technical Learning?

Data Science 66

Data Science Building Computer Science Software Engineer 66

Knowledge Hut

DECEMBER 29, 2023

The market for analytics is flourishing, as is the usage of the phrase Data Science. Professionals from a variety of disciplines use data in their day-to-day operations and feel the need to understand cutting-edge technology to get maximum insights from the data, therefore contributing to the growth of the organization.

Data Science 98

Data Science Data Mining Deep Learning Programming Language 98

Towards Data Science

FEBRUARY 6, 2024

This generalisation makes their data models complex and cryptic and require domain expertise. Even harder to manage, a common setup within large organisations is to have several instances of these systems with some underlaying processes in charge of transmitting data among them, which could lead to duplications, inconsistencies, and opacity.

Systems 83

Systems Raw Data Metadata Data Cleanse 83

Monte Carlo

MAY 3, 2024

It’s 2024, and the data estate has changed. Data systems are more diverse. But even though the data landscape is evolving, many enterprise data organizations are still managing data quality the “old” way: with simple data quality monitoring. For JetBlue’s data engineering team, that’s Microsoft Teams.

High Quality Data 52

High Quality Data Data Architecture Media 52

Knowledge Hut

JANUARY 18, 2024

Speaking of job vacancies, the two careers have high demands till date and in upcoming years are Data Scientist and a Software Engineer. Per the BLS, the expected growth rate of job vacancies for data scientists and software engineers is around 22% by 2030. What is Data Science? Get to know more about SQL for data science.

Software Engineer 97

Software Engineer Software Engineering Data Science Engineering 97

Knowledge Hut

APRIL 26, 2024

In the age of big data processing, how to store these terabytes of data surfed over the internet was the key concern of companies until 2010. Now that the issue of storage of big data has been solved successfully by Hadoop and various other frameworks, the concern has shifted to processing these data.

Data Science 52

Data Science Machine Learning Hadoop Programming Language 52

Knowledge Hut

JANUARY 29, 2024

Some techniques add to the development of technology in the business sectors, including Data Science and Cloud Computing, essential aspects of the technology industry. With the help of data science, one can gather all the critical analyses from vast chunks of data stored in clouds. In this model, the data is not 100% secure.

Cloud Computing 98

Cloud Computing Data Science Cloud Amazon Web Services 98

Knowledge Hut

MARCH 22, 2024

When I worked in the hospitality industry, the electricity abruptly went out while we were establishing the network and door locks. I have comprehensively analyzed the area of physical security, particularly the ongoing discussion surrounding fail safe vs fail-safe secure electric strike locking systems. What are Fail-Safe Door Locks?

Systems 105

Systems Electronics Hospitality Architecture 105

Snowflake

DECEMBER 21, 2023

In this edition, Patch.tech Co-Founder and CPO Whelan Boyd talks about how frustration with clogged data pipelines sparked the idea for Patch’s code packages, which allow engineers to distribute data sets with all the built-in elements that analysts and developers need to create apps. Whelan, can you tell us a bit about yourself?

Software Engineer 88

Software Engineer Software Engineering Database Data Pipeline 88

Knowledge Hut

DECEMBER 26, 2023

According to the World Economic Forum, the amount of data generated per day will reach 463 exabytes (1 exabyte = 10 9 gigabytes) globally by the year 2025. Thus, almost every organization has access to large volumes of rich data and needs “experts” who can generate insights from this rich data.

Data Science 97

Data Science BI Business Intelligence Data Mining 97

Knowledge Hut

APRIL 23, 2024

In the modern data-driven landscape, organizations continuously explore avenues to derive meaningful insights from the immense volume of information available. Two popular approaches that have emerged in recent years are data warehouse and big data. Data warehousing offers several advantages.

Data Warehouse 52

Data Warehouse Big Data Unstructured Data Hadoop 52

Knowledge Hut

APRIL 25, 2024

In the world of data science, Spatial data science is at the heart of contributing to solving planet-threatening problems such as deciding the location of solar park installation, building urban resilience, predicting crop yield, population density analysis for immunization or disease mapping, and more. What is Spatial Data Science?

Data Science 52

Data Science Telecommunication Transportation Big Data 52

Towards Data Science

AUGUST 2, 2023

A Step-by-Step Guide to Building an Effective Data Quality Strategy from Scratch How to build an interpretable data quality framework based on user expectations Photo by Rémi Müller on Unsplash As data engineers, we are (or should be) responsible for the quality of the data we provide. Where do I start?

Building 98

Building Data Consolidation Data Datasets 98

Knowledge Hut

JANUARY 18, 2024

Data Science is an amalgamation of several disciplines, including computer science, statistics, and machine learning. As the world on the internet is becoming our second home, Big Data has exploded. Data Science is the study of this big data to derive a meaningful pattern.

Data Science 98

Data Science Amazon Web Services Finance Big Data 98

Knowledge Hut

JANUARY 18, 2024

Most organizations are still in the initial stages of learning how to apply data science to gain business benefits and healthy returns. To maximize the benefits of data science, people need to have good technical knowledge of managing the data in a production environment, without which the awareness of full stack data science is insufficient.

Data Science 96

Data Science Computer Science Programming Language Machine Learning 96

Precisely

DECEMBER 6, 2023

The European Union’s General Data Protection Regulation (commonly known as GDPR) came into effect on the 25th May 2018. It is important to any user or processor of personally identifiable information (PII) as this new law marks a seismic change in the way that organizations must protect consumers’ data.

Retail 64

Retail Data Systems Data Integration 64

Knowledge Hut

JANUARY 18, 2024

Data science is a multidisciplinary field that requires a broad set of skills from mathematics and statistics to programming, machine learning, and data visualization. The world has been swept by the rise of data science and machine learning. Data scientists are in high demand, and the demand will only continue to rise.

Programming Language 98

Programming Language Data Science Programming Scala 98

Knowledge Hut

JANUARY 11, 2024

Data science has become popular in the last few years due to its successful application in making business decisions. Data scientists have been using data science techniques to solve challenging real-world issues in healthcare, agriculture, manufacturing, automotive, and many more.

Data Science 98

Data Science Transportation Hospitality Banking 98

Knowledge Hut

APRIL 29, 2024

Several countries around the world have announced nationwide lockdowns and large organizations have set a mandate for their teams to work remotely to combat the spread of COVID-19. Some are better equipped than others to deal with prolonged periods of disruption from their regular working routines.

Data Science 98

Data Science Media Technology Big Data 98

Towards Data Science

AUGUST 31, 2023

Optimizing queries, improving runtimes, and geospatial data science applications Photo by Tamas Tuzes-Katai on Unsplash Intro: why is a spatial index useful? In doing geospatial data science work, it is very important to think about optimizing the code you are writing. This is where concepts such as spatial indices come in.

Data Engineering 75

Data Engineering Data Engineer Engineering Data Science 75

U-Next

FEBRUARY 26, 2023

Introduction Data Science is revolutionizing the business world, and it has opened up unique opportunities for businesses to grow. Businesses are now looking for Data Scientists to help them make a difference in their company’s performance and reach even further. Average Salary per annum: INR 27 lakhs Number of Employees: 14.68

Data Science 98

Data Science Machine Learning Consulting Food 98