KDnuggets

NOVEMBER 28, 2019

Open source is becoming the standard for sharing and improving technology. Some of the largest organizations in the world namely: Google, Facebook and Uber are open sourcing their own technologies that they use in their workflow to the public.

Data Science 157

Data Science Project Technology Data 157

Data Engineering Podcast

FEBRUARY 18, 2024

Multiple open source projects and vendors have been working together to make this vision a reality. It is an open-source, cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability, a declarative programming model, and best-in-class testability.

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Data Engineering Podcast

MAY 5, 2024

It is an open-source, cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability, a declarative programming model, and best-in-class testability. The fundamental/foundational challenge of every AI use case is sourcing appropriate data. Want to see Starburst in action? Links Aigo.ai

Building 130

Building Data Lake High Quality Data Machine Learning 130

Data Engineering Podcast

MARCH 24, 2024

With their launch of Dagster+ as the redesigned commercial companion to the open source project they are investing in that capability with a suite of new features. What are the notable enhancements beyond the Dagster Core project that this updated platform provides? Want to see Starburst in action?

Data Lake 162

Data Lake High Quality Data Hadoop Data Pipeline 162

The Pragmatic Engineer

SEPTEMBER 22, 2023

It begins with a clean state, and can ship something that works for, say, 90% of existing Node projects, and break the remaining 10%. Bun also reveals the contrast between venture-funded open source and ‘community sweat’-funded open source. Bun has no such constraint.

Programming Language 239

Programming Language Coding Project Engineering 239

Data Engineering Podcast

APRIL 14, 2024

Datafold has recently launched data replication testing, providing ongoing validation for source-to-target replication. Validate consistency between source and target at any scale, and receive alerts about any discrepancies. How have you approached the evolution of RavenDB as you add new capabilities and mature the project?

Non-relational Database 147

Non-relational Database Relational Database Database Designing 147

Data Engineering Podcast

MARCH 31, 2024

To bring observability to dbt projects the team at Elementary embedded themselves into the workflow. And Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data. Want to see Starburst in action?

Project 130

Project Data Lake High Quality Data Data Workflow 130

Zalando Engineering

MARCH 16, 2019

Project Highlights Kube Metrics Adapter gained community attention as it was featured in a medium post 'Kubernetes autoscaling with Istio metrics'. Users provided very positive feedback on the project. Introscope is a newly released project. Kube Metrics Adapter is currently maintained by Developer Productivity team at Zalando.

Programming 40

Programming Coding PostgreSQL Project 40

Christophe Blefari

APRIL 19, 2024

Meta releases Llama 3 — After last week Mistral new models , this week ends with the new Meta open-source models. They did not released the code so I tried the closest solution open-source called SadTalker and tried it on me. OpenAI batches — OpenAI opened a new API endpoint to batches requests.

MySQL 130

MySQL Data Datasets SQL 130

Data Engineering Podcast

APRIL 21, 2024

It is an open-source, cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability, a declarative programming model, and best-in-class testability. If you've learned something or tried out a project from the show then tell us about it! Want to see Starburst in action?

Data Lake 182

Data Lake High Quality Data Data Pipeline Machine Learning 182

Data Engineering Podcast

APRIL 28, 2024

It is an open-source, cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability, a declarative programming model, and best-in-class testability. If you've learned something or tried out a project from the show then tell us about it! Want to see Starburst in action?

Building 147

Building Data Lake High Quality Data Machine Learning 147

Data Engineering Podcast

APRIL 7, 2024

Summary Maintaining a single source of truth for your data is the biggest challenge in data engineering. Datafold has recently launched data replication testing, providing ongoing validation for source-to-target replication. Validate consistency between source and target at any scale, and receive alerts about any discrepancies.

Data Lake 162

Data Lake High Quality Data BI Data Workflow 162

Christophe Blefari

OCTOBER 19, 2023

cross-project dependencies ( credits ) Over the last few years, dbt has become a de facto standard enabling companies to collaborate easily on data transformations. Whatever the number, there will be a critical point at which a single project no longer scale. Cross-project references is a key enabler to data team decentralisation.

Project 264

Project Finance SQL Government 264

Confessions of a Data Guy

MARCH 25, 2024

Ever wondered how to build and end-to-end project for an Open Source Python Package that gets published to PYPI? I built out lakescuman open-source package to help with Databricks Unity Catalog Delta Lake tables querying with Polars, DuckDB, or PyArrow.

Python 100

Python Building Project Data 100

Data Engineering Podcast

MARCH 10, 2024

The primary purpose of the catalog is to inform the query engine of what data exists and where, but the Nessie project aims to go beyond that simple utility. And Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data.

Data Lake 147

Data Lake High Quality Data Data Pipeline Architecture 147

Christophe Blefari

MARCH 2, 2024

Mistral ( credits ) Hello all, this is the Data News, this week edition might be smaller than usual in term of comments as I'm working on a Data News related project that takes me a bit of time, which will probably lead to a series of articles. Final comment, with these 2 announcement Mistral left the open side to go commercial / closed.

Data 162

Data Python Database BI 162

Knowledge Hut

MAY 2, 2024

TestComplete and Selenium are the two most important automation testing tools which provide an open platform for you to easily build continuous testing frameworks to test non-stop with a lightweight execution engine and distributed testing. There are fewer job openings on this tool currently. Open Source/Free tool.

Programming Language 88

Programming Language Java Python Programming 88

Christophe Blefari

APRIL 12, 2024

I've shared my journey with Apache Superset and why I consider Superset the best open-source alternative when it comes to building BI applications. OpenAI open-sourced a light library to evaluate language models  — you can use 7 different evals and check the results on OpenAI or Claude models.

BI 130

BI SQL Data Coding 130

KDnuggets

APRIL 10, 2024

Learn Python through tutorials, blogs, books, project work, and exercises. Access all of it on GitHub for free and join a supportive open-source community.

Python 133

Python Accessible Accessibility Project 133

Data Engineering Podcast

FEBRUARY 25, 2024

It is an open-source, cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability, a declarative programming model, and best-in-class testability. One of the major benefits of using open source components is the network effect of ecosystem integrations.

Database 162

Database Technology Data Lake High Quality Data 162

Christophe Blefari

JANUARY 26, 2024

Before jumping to AI projects you need first to start with words and to define metrics reflecting [your] values. see also open-sourcing the future of AI , which is a HuggingFace praising post at some point). PR comment template for dbt data projects — Great stuff. There is a Chinese company called 01.ai

Google Cloud 130

Google Cloud Algorithm Data Java 130

Data Engineering Podcast

FEBRUARY 4, 2024

And Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data. What are the core design philosophies that you rely on to prioritize the ongoing development of the project? Want to see Starburst in action?

SQL 173

SQL Data Lake High Quality Data Data Pipeline 173

Data Engineering Podcast

MARCH 3, 2024

Summary Artificial intelligence technologies promise to revolutionize business and produce new sources of value. It is an open-source, cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability, a declarative programming model, and best-in-class testability.

Programming 162

Programming Data Lake High Quality Data Data Pipeline 162

The Pragmatic Engineer

MAY 1, 2024

” It seems they’re aiming for something which performs better than human software engineers, today: Image source: Daniel Gross on X It’s a very bold claim whose success would mean a direct replacement of developers! Source: Cognition So far, all we have is video demos, and accounts of those with access to this tool.

Software Engineer 181

Software Engineer Software Engineering Programming Language Media 181

The Pragmatic Engineer

MARCH 19, 2024

” It seems they’re aiming for something which performs better than human software engineers, today: Image source: Daniel Gross on X It’s a very bold claim whose success would mean a direct replacement of developers! Source: Cognition So far, all we have is video demos, and accounts of those with access to this tool.

Software Engineer 258

Software Engineer Software Engineering Programming Language Media 258

Data Engineering Podcast

SEPTEMBER 3, 2023

Summary Cloud data warehouses and the introduction of the ELT paradigm has led to the creation of multiple options for flexible data integration, with a roughly equal distribution of commercial and open source options. What are the complexities or limitations of those tools that leave an opening for dlt? in the commercial space.

Data Integration 162

Data Integration BI SQL Python 162

Data Engineering Podcast

DECEMBER 24, 2023

Elad Eldor has experienced these challenges first-hand, leading to his work writing the book "Kafka: : Troubleshooting in Production" In this episode he highlights the sources of complexity that contribute to Kafka's operational difficulties, and some of the main ways to identify and mitigate potential sources of trouble.

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Christophe Blefari

APRIL 5, 2024

DBRX, a new state-of-the-art open LLM — Databricks has to be an AI company ( bragging vs. Snowflake ). This week they released a new open model that performs great. How we built Text-to-SQL at Pinterest — Pinterest open-sourced a tool called Querybook that they used to access Pinterest data every day.

SQL 130

SQL Data Python Datasets 130

Knowledge Hut

MAY 2, 2024

Finding the right automation testing tool for your project can be daunting. This blog post looks at two of the most common tools used in software development––UFT/QTP and Selenium––and discusses some of the key differences between them that you should consider when choosing an automation tool for your projects.

Programming Language 98

Programming Language Java Coding Skills Programming 98

Christophe Blefari

OCTOBER 20, 2023

They announced dbt Mesh a product enabling cross-project dependencies for teams with multiple dbt projects. In addition they also released an Explorer view that lets you navigate through all you project and see models, macros and more directly in one nice graph. No, you can activate multi-project collaboration with dbt Core.

Generalist 130

Generalist Entertainment NoSQL Datasets 130

Data Engineering Podcast

MARCH 17, 2024

Validating that the information is stored and processed correctly can be complex and time-consuming, especially when the source and destination speak different dialects of SQL. And Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data.

Database 147

Database Data Lake High Quality Data Data Workflow 147

Data Engineering Podcast

DECEMBER 10, 2023

Andrew Maguire got tired of solving that problem for each of the different roles he has ended up in, so he created the open source Anomstack project. Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack Data projects are notoriously complex. What are your goals for this project?

Data Lake 130

Data Lake High Quality Data SQL Architecture 130

Data Engineering Podcast

JANUARY 7, 2024

And Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data. What are the open questions today in technical scalability of data engines? If you've learned something or tried out a project from the show then tell us about it!

Data Process 147

Data Process Process Data Lake High Quality Data 147

Christophe Blefari

MARCH 1, 2023

dbt Core is an open-source framework that helps you organise data warehouse SQL transformation. dbt Labs also develop dbt Cloud which is a cloud product that hosts and runs dbt Core projects. a dbt project — a dbt project is a folder that contains all the dbt objects needed to work. dbt/ folder.

Data Warehouse 278

Data Warehouse SQL Metadata Raw Data 278

Data Engineering Podcast

FEBRUARY 11, 2024

And Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data. If you've learned something or tried out a project from the show then tell us about it! Want to see Starburst in action? Want to see Starburst in action?

Data Lake 147

Data Lake High Quality Data Government Data Pipeline 147

Christophe Blefari

MARCH 22, 2024

Exploration, Friday morning I read the last 7 days of 2 Twitter lists ( MDS , Data voices ) and I open interesting stuff in tabs. Then I use Feedly which is connected to ~500 websites, Reddit and Medium and opens interesting articles in tabs. Then I opens the elements saved from LinkedIn. But I could do it better I think.

Electronics 130

Electronics Media Data Python 130