DataKitchen

MAY 10, 2024

Harnessing Data Observability Across Five Key Use Cases The ability to monitor, validate, and ensure data accuracy across its lifecycle is not just a luxury—it’s a necessity. Data Evaluation Before new data sets are introduced into production environments, they must be thoroughly evaluated and cleaned.

Data Ingestion 58

Data Ingestion Datasets Data Coding 58

ProjectPro

FEBRUARY 6, 2023

With over 10 million active subscriptions, 50 million active topics, and a trillion messages processed per day, Google Cloud Pub/Sub makes it easy to build and manage complex event-driven systems. Google Cloud Pub/Sub is a global, cloud-based messaging framework that has become increasingly popular among data engineers over recent years.

Google Cloud 52

Google Cloud Cloud Cloud Storage Data Ingestion 52

Christophe Blefari

JANUARY 20, 2024

Learn data engineering, all the references ( credits ) This is a special edition of the Data News. But right now I'm in holidays finishing a hiking week in Corsica 🥾 So I wrote this special edition about: how to learn data engineering in 2024. The idea is to create a living reference about Data Engineering.

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

DataKitchen

MAY 10, 2024

The Five Use Cases in Data Observability: Effective Data Anomaly Monitoring (#2) Introduction Ensuring the accuracy and timeliness of data ingestion is a cornerstone for maintaining the integrity of data systems. This process is critical as it ensures data quality from the onset.

Data Ingestion 40

Data Ingestion Transportation High Quality Data Data Schemas 40

Data Engineering Podcast

OCTOBER 23, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Database 100

Database MySQL Cloud MongoDB 100

DataKitchen

MAY 10, 2024

The Five Use Cases in Data Observability: Ensuring Data Quality in New Data Sources (#1) Introduction to Data Evaluation in Data Observability Ensuring their quality and integrity before incorporating new data sources into production is paramount. When looking at new data, does one patch the data?

Data Cleanse 51

Data Cleanse Data Ingestion Data Datasets 51

Cloudera

FEBRUARY 9, 2021

Today’s customers have a growing need for a faster end to end data ingestion to meet the expected speed of insights and overall business demand. This ‘need for speed’ drives a rethink on building a more modern data warehouse solution, one that balances speed with platform cost management, performance, and reliability.

Data Warehouse 70

Data Warehouse Cloud Kafka Cloud Storage 70

Data Engineering Weekly

APRIL 21, 2024

Meta: Introducing Meta Llama 3 - The most capable openly available LLM to date Meta is taking an interesting approach in the growing LLM market with the open source approach and distribution across all the leading cloud providers and data platforms. Counting is the hardest problem in data engineering.

Data Engineering 52

Data Engineering Data Engineer Engineering Medical 52

Booking.com Engineering

DECEMBER 2, 2022

In this article, we want to illustrate our extensive use of the public cloud, specifically Google Cloud Platform (GCP). From data ingestion, data science, to our ad bidding[2], GCP is an accelerant in our development cycle, sometimes reducing time-to-market from months to weeks. What are PPC’s Challenges?

Systems 52

Systems Cloud MySQL Relational Database 52

Snowflake

APRIL 9, 2024

Outreach data, available via Snowflake Marketplace , contains a huge amount of useful information including lead scoring, topics that are resonating with audiences, where sales reps spend the most time, which accounts are open to conversations and more. Each of these sources may store data differently. But that’s not all.

BI 76

BI Data Ingestion Data Aggregated Data 76

Precisely

MAY 13, 2024

Key Takeaways: Data integration is vital for real-time data delivery across diverse cloud models and applications, and for leveraging technologies like generative AI. The right data integration solution helps you streamline operations, enhance data quality, reduce costs, and make better data-driven decisions.

Data Integration 52

Data Integration Metadata Amazon Web Services Data Governance 52

Snowflake

MARCH 14, 2024

As a cohesive ERP solution, SAP is often one of the largest data resources in an organization, containing everything from financial and transactional data to master information about customers, vendors, materials, facilities, planning and even HR. What’s the challenge with unlocking SAP data?

IT 93

IT Data Ingestion Data AWS 93

Christophe Blefari

MARCH 4, 2023

Formula 1 is back (trying to jinx before it happens) (yes there is no link with the data news) ( credits ) Hello you, I hope this new Data News finds you well. But still, in order to make data works we still need to praise other data coworkers that have to do documentation and all the governance burden that no-one wants to do.

Machine Learning 130

Machine Learning AWS Data Data Lake 130

Data Engineering Podcast

JUNE 12, 2022

Summary Unstructured data takes many forms in an organization. From a data engineering perspective that often means things like JSON files, audio or video recordings, images, etc. Another category of unstructured data that every business deals with is PDFs, Word documents, workstation backups, and countless other types of information.

Unstructured Data 130

Unstructured Data MongoDB Scala MySQL 130

Snowflake

OCTOBER 3, 2023

We are excited to announce the availability of data pipelines replication, which is now in public preview. In the event of an outage, this powerful new capability lets you easily replicate and failover your entire data ingestion and transformations pipelines in Snowflake with minimal downtime.

Data Pipeline 85

Data Pipeline Management Data Ingestion Data 85

Snowflake

SEPTEMBER 14, 2023

More than 50% of data leaders recently surveyed by BCG said the complexity of their data architecture is a significant pain point in their enterprise. Your technology stack should accommodate growth—in data volumes as well as in your business. Snowflake was built from the ground up in the cloud.

Data Architecture 97

Data Architecture Architecture Hadoop Data Warehouse 97

Ascend.io

FEBRUARY 15, 2023

And now, I’m betting on the fact that there is a new data challenge on the horizon — that the classic scale challenge of storing and processing data will soon become table stakes, and a new problem of scaling engineering productivity will emerge. Big data would be a big deal.

Cloud 52

Cloud Amazon Web Services Hadoop Telecommunication 52

Ascend.io

FEBRUARY 15, 2023

And now, I’m betting on the fact that there is a new data challenge on the horizon — that the classic scale challenge of storing and processing data will soon become table stakes, and a new problem of scaling engineering productivity will emerge. Big data would be a big deal.

Cloud 52

Cloud Amazon Web Services Hadoop Telecommunication 52

Cloudera

SEPTEMBER 28, 2020

Cloudera delivers an enterprise data cloud that enables companies to build end-to-end data pipelines for hybrid cloud, spanning edge devices to public or private cloud, with integrated security and governance underpinning it to protect customers data. Data Science and machine learning workloads using CDSW.

Cloud 131

Cloud Kafka Professional Services Metadata 131

Data Engineering Podcast

NOVEMBER 6, 2022

Summary Despite the best efforts of data engineers, data is as messy as the real world. Entity resolution and fuzzy matching are powerful utilities for cleaning up data from disconnected sources, but it has typically required custom development and training machine learning models.

MongoDB 130

MongoDB Scala MySQL Data Lake 130

DataKitchen

MAY 10, 2024

The Five Use Cases in Data Observability: Mastering Data Production (#3) Introduction Managing the production phase of data analytics is a daunting challenge. Overseeing multi-tool, multi-dataset, and multi-hop data processes ensures high-quality outputs.

Raw Data 40

Raw Data Data Ingestion Datasets Data 40

Knowledge Hut

AUGUST 18, 2023

With the rise of cloud computing, there’s no better time to explore the top Google Cloud Certifications that can take your career to new heights. Having gone through the process myself, I can attest to the immense value & recognition that comes with earning a Google Cloud Certification. What is Google Cloud Certification?

Google Cloud 52

Google Cloud Certification Cloud Cloud Computing 52

Data Engineering Podcast

JULY 24, 2022

Summary The current stage of evolution in the data management ecosystem has resulted in domain and use case specific orchestration capabilities being incorporated into various tools. In this episode Nick Schrock discusses the importance of orchestration and a central location for managing data systems, the road to Dagster’s 1.0

MongoDB 100

MongoDB Scala MySQL Data Lake 100

Ascend.io

FEBRUARY 23, 2024

You are a data professional. Exciting, isn’t it Yet, embarking on the AI adoption journey introduces a series of challenges, with one of the most significant being the readiness of your data platform. In this article, we outline the essential prerequisites for an AI data platform.

Cloud Storage 52

Cloud Storage Data Ingestion Machine Learning Algorithm 52

Knowledge Hut

APRIL 23, 2024

Data handling became tedious only after huge amounts of it started to accumulate. For example, in 1880, the US Census Bureau needed to handle the 1880 Census data. They realized that compiling this data and converting it into information would take over 10 years without an efficient system.

Big Data 52

Big Data Amazon Web Services Media Cloud Computing 52

DataKitchen

MAY 10, 2024

The Five Use Cases in Data Observability: Fast, Safe Development and Deployment (#4) Introduction The integrity and functionality of new code, tools, and configurations during the development and deployment stages are crucial. This process is critical as it ensures data quality from the onset. Will I Create A Failed Deploy?

Data Ingestion 40

Data Ingestion Datasets Coding Data 40

Data Engineering Podcast

SEPTEMBER 11, 2022

Summary Any business that wants to understand their operations and customers through data requires some form of pipeline. Building reliable data pipelines is a complex and costly undertaking with many layered requirements. Data stacks are becoming more and more complex. Sifflet also offers a 2-week free trial.

Data Pipeline 130

Data Pipeline Building MongoDB Scala 130

Data Engineering Weekly

MARCH 17, 2024

Compliance is mandatory, with strict penalties for violations, emphasizing the importance of data scientists familiarizing themselves with the law to avoid prohibited AI uses and ensure ethical, safe AI development. It discusses the significance of data governance, sharing history, and generative AI's impact on data economy standards.

Data Engineering 70

Data Engineering Data Engineer Engineering Data Governance 70

Data Engineering Podcast

NOVEMBER 13, 2022

Summary CreditKarma builds data products that help consumers take advantage of their credit and financial capabilities. To make that possible they need a reliable data platform that empowers all of the organization’s stakeholders. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

MongoDB 100

MongoDB Scala Google Cloud MySQL 100

Rockset

JANUARY 30, 2024

In 2023, Rockset announced a new cloud architecture for search and analytics that separates compute-storage and compute-compute. With this architecture, users can separate ingestion compute from query compute, all while accessing the same real-time data. minutes to batch load the data.

Data Ingestion 64

Data Ingestion Utilities Architecture SQL 64

Towards Data Science

DECEMBER 1, 2023

Pet Project for Data/Analytics Engineers: Explore Modern Data Stack Tools — dbt Core, Snowflake, Fivetran, GitHub Actions. This hands-on experience will allow you to develop an end-to-end data lifecycle, from extracting data from your Google Calendar to presenting it in a Snowflake analytics dashboard. See Github repo.

Data Engineering 73

Data Engineering Data Engineer Project Engineering 73

Data Engineering Podcast

JULY 3, 2022

Summary The perennial challenge of data engineers is ensuring that information is integrated reliably. In order to quickly identify if and how two data systems are out of sync Gleb Mezhanskiy and Simon Eskildsen partnered to create the open source data-diff utility. Data teams are increasingly under pressure to deliver.

Data Integration 100

Data Integration MongoDB Scala MySQL 100

Cloudera

FEBRUARY 8, 2021

Within the vehicle, current electronics and wiring infrastructures were not designed for this complex data wrangling capability. In addition, moving outside the vehicle, existing fragmented approaches for data management associated with the machine learning lifecycle are limiting the ability to deploy new use cases at scale.

Manufacturing 107

Manufacturing Machine Learning Data Ingestion Electronics 107

Cloudera

AUGUST 11, 2022

But information broadly, and the management of data specifically, is still “the” critical factor for situational awareness, streamlined operations, and a host of other use cases across today’s tech-driven battlefields. . Universal Data Distribution Solves DoD Data Transport Challenges. hardware in the skies, sea, and land.

Transportation 92

Transportation Data Ingestion Architecture Data 92

Data Engineering Podcast

SEPTEMBER 25, 2022

Summary Data integration from source systems to their downstream destinations is the foundational step for any data product. With the increasing expecation for information to be instantly accessible, it drives the need for reliable change data capture. Data teams are increasingly under pressure to deliver.

Food 100

Food MongoDB Scala MySQL 100

Data Engineering Podcast

JULY 10, 2022

Summary Building and maintaining reliable data assets is the prime directive for data engineers. While it is easy to say, it is endlessly complex to implement, requiring data professionals to be experts in a wide range of disparate topics while designing and implementing complex topologies of information workflows.

Data Engineering 100

Data Engineering Data Engineer Engineering MongoDB 100

Cloudera

JULY 9, 2021

If you’ve followed Cloudera for a while, you know we’ve long been singing the praises—or harping on the importance, depending on perspective—of a solid, standalone enterprise data strategy. The ways data strategies are implemented, the resulting outcomes and the lessons learned along the way provide important guardrails.

Government 111

Government Building Cloud Data Ingestion 111