Christophe Blefari
OCTOBER 19, 2023
cross-project dependencies ( credits ) Over the last few years, dbt has become a de facto standard enabling companies to collaborate easily on data transformations. With dbt, you can apply software engineering practices to SQL development. Managing your SQL patrimony has never been easier. So, yes, dbt is cool but there is a common pattern with it: you accumulate SQL queries.
Data Engineering Podcast
OCTOBER 15, 2023
Summary Building streaming applications has gotten substantially easier over the past several years. Despite this, it is still operationally challenging to deploy and maintain your own stream processing infrastructure. Decodable was built with a mission of eliminating all of the painful aspects of developing and deploying stream processing systems for engineering teams.
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Marc Lamberti
OCTOBER 20, 2023
Templates and Macros in Apache Airflow allow passing data to your DAGs at runtime. Imagine that you want to execute an SQL request with the execution date of your DAG. How can you do that? How can you use the DAG ID when you send notifications to know which DAG to look at? Or what if you need to know when the next DAG run will be? Well, macros and templates answer these questions.
Waitingforcode
OCTOBER 17, 2023
I've already written about watermarks in a few places in the blog but despite that, I still find things to refresh. One of them is the watermark used to filter out the late data, which will be the topic of this blog post.
Speaker: Dr. Greg Loughnane and Chris Alexiuk
Technology professionals developing generative AI applications are finding that there are big leaps from POCs and MVPs to production-ready applications. They're often developing using prompting, Retrieval Augmented Generation (RAG), and fine-tuning (up to and including Reinforcement Learning with Human Feedback (RLHF)), typically in that order. However, during development – and even more so once deployed to production – best practices for operating and improving generative AI applications are le
Christophe Blefari
OCTOBER 20, 2023
Writing about dbt like a sheep ( credits ) Hey, this week Coalesce—the dbt Labs annual conference—took place. During 3 days, people shared how they used dbt around the world. I'll, as usual, write a takeaway post after binge watching all keynotes, but this is for next week. Still dbt Labs announcements were mainly towards dbt Cloud with great features to drive adoption of the paid product.
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
KDnuggets
OCTOBER 18, 2023
Large Language Models (LLMs) have unlocked a new era in natural language processing. So why not learn more about them? Go from learning what large language models are to building and deploying LLM apps in 7 easy steps with this guide.
LinkedIn Engineering
OCTOBER 19, 2023
Authors: Bingfeng Xia and Xinyu Liu Background At LinkedIn, Apache Beam plays a pivotal role in stream processing infrastructures that process over 4 trillion events daily through more than 3,000 pipelines across multiple production data centers. This robust framework empowers near real-time data processing for critical services and platforms, ranging from machine learning and notifications to anti-abuse AI modeling.
Christophe Blefari
OCTOBER 14, 2023
( credits ) Hello, dear Data News reader, I hope you'll enjoy this new edition. It's amazing how quickly time flies and this summer I passed the 3-year mark since I started my freelance adventure. I'm so happy with what it's brought me. But I've got this internal alarm that goes off every 3 years asking me for new things. It's time for me to search for my future paths.
DoorDash Engineering
OCTOBER 19, 2023
As DoorDash’s microservices architecture has grown, so too has the volume of interservice traffic. Each team manages their own data and exposes access through gRPC services, an open-source remote procedure call framework used to build scalable APIs. Most business logic is I/O-bound because of calls to downstream services. Caching has long been a go-to strategy to improve performance and reduce costs.
Advertisement
Are you thinking of adding enhanced data matching and relationship detection to your product or service? Do you need to know more about what to look for when assessing your options? The Senzing Entity Resolution Buyer’s Guide gives you step-by-step details about everything you should consider when evaluating entity resolution technologies. You’ll learn about use cases, technology and deployment options, top ten evaluation criteria and more.
KDnuggets
OCTOBER 16, 2023
Want to break into data science? Check this list of free books for learning Python, statistics, linear algebra, machine learning and deep learning.
ArcGIS
OCTOBER 17, 2023
The new U.S. datums of 2022 will soon be released. This article covers what is coming and how you should prepare your data.
Engineering at Meta
OCTOBER 18, 2023
With the recent launches of MTIA v1 , Meta’s first-generation AI inference accelerator, and Llama 2 , the next generation of Meta’s publicly available large language model, it’s clear that Meta is focused on advancing AI for a more connected world. Fueling the success of these products are world-class infrastructure teams, including Meta’s custom AI silicon team, led by Olivia Wu, a leader in the silicon industry for 30 years.
InData Labs
OCTOBER 17, 2023
Big data is central to the efficient running of all modern organizations, but to be of use, raw data must be suitably organized. The way that businesses organize data assets is commonly known as data architecture, with the benefits of modern data architecture enabling teams to respond to changing demands with improved agility when compared. Запись The benefits of modern data architecture впервые появилась InData Labs.
KDnuggets
OCTOBER 18, 2023
Cloud databases have made it easier and cheaper to develop enterprise-level applications, offering flexibility, convenience, and standard database functionality. See what KDnuggets recommends.
databricks
OCTOBER 19, 2023
Machine learning (ML) is more than just developing models; it's about bringing them to life in real-world, production systems. But transitioning from prototype.
Engineering at Meta
OCTOBER 17, 2023
Systematic Code and Asset Removal Framework (SCARF) is Meta’s unused code and data deletion framework. SCARF guides engineers through deprecating a product safely and efficiently via an internal tool. SCARF combines this tooling with automation to reduce load on engineers. At Meta, we are constantly innovating and experimenting by building and shipping many different products, and those products comprise thousands of individual features.
ArcGIS
OCTOBER 17, 2023
The Living Atlas of the World just updated the tiled, hosted image service featuring transportation noise, from the USDOT.
KDnuggets
OCTOBER 17, 2023
Large language models (LLMs) are transforming the way we process and produce information. But, before considering either one of these models as a one-stop-solution, one must consider their key differences.
Medium Data Engineering
OCTOBER 19, 2023
In today’s data-driven world, organizations are increasingly relying on the expertise of data engineers to collect, process, and maintain… Continue reading on Medium »
Ripple Engineering
OCTOBER 19, 2023
RippleX has enabled its validator to vote in support of the XLS-30 amendment, introducing innovative AMM capabilities to the XRPL. We, at RippleX, place great emphasis on the strength that collaborative effort and shared responsibility bring to the enhancement and security of the XRPL. Today, we earnestly request the community's consideration of the XLS-30 amendment —a proposal poised to offer numerous advantages by bolstering liquidity, offering yield opportunities for liquidity pro
ArcGIS
OCTOBER 16, 2023
Streamline GIS workflows with ArcGIS Reality for ArcGIS Pro. Automate reality mapping, generate accurate geospatial products.
KDnuggets
OCTOBER 19, 2023
Looking to understand the semantic layer and how it can improve the AI-powered data experience? Read more to learn why a semantic layer can be the backbone of LLMs and reduce hallucinations.
Medium Data Engineering
OCTOBER 16, 2023
BigLake for AWS มี Region Support ยังไม่มาก และการเอา Data ออกมาจาก BigQuery ลง S3 ใช้ Glue ก็ดี Continue reading on Medium »
Snowflake
OCTOBER 19, 2023
Welcome to Snowflake’s Startup Spotlight, where we learn about amazing companies building their businesses on Snowflake. In this edition, Pave.dev President and Co-Founder Ema Rouf talks about breaking down barriers to accessible credit and financial lending, how running a startup is like climbing a mountain, and how building on Snowflake gives Pave the data sharing capabilities it needs to show financial institutions a better way to identify more creditworthy borrowers.
Cloudera
OCTOBER 20, 2023
There are some who believe that growing in your professional career and a desire to travel the world don’t mix well. I am not one of those people – in fact, I’m proof that these two ambitions can blend together to create a beautiful life. I’m Kinga Kamaras. My title at Cloudera is a Strategic Customer Success Manager. It’s a role I enjoy and growing in my career is a big ambition of mine.
KDnuggets
OCTOBER 19, 2023
Gradient descent is an optimization technique used to minimise errors in machine learning models. By iteratively adjusting parameters in the steepest direction of decrease, it seeks the lowest error value.
Medium Data Engineering
OCTOBER 18, 2023
Data modeling is a very important aspect of database design and plays a crucial role in designing the structure of data for various… Continue reading on Medium »
Snowflake
OCTOBER 18, 2023
Product-led growth (PLG) is a business model that emerged in the last decade with the enormous success of vendors like Slack and Datadog. Unlike traditional sales-led models, PLG models cut out the middlemen (sales reps, for example) and let customers just download and use the product without third-party onboarding. The relative novelty of the pricing model and its demonstrably successful application in growing these companies attracted a lot of attention.
databricks
OCTOBER 17, 2023
Introduction Large Language Models (LLMs) have given us a way to generate text, extract information, and identify patterns in industries from healthcare to.
KDnuggets
OCTOBER 16, 2023
This article has outlined how you can fine tune your GPT 3.5 Turbo models. You can do this by preparing your data, uploading your files, and then setting up a custom OpenAI session to handle the fine tuning.
Scott Logic
OCTOBER 18, 2023
Introduction In my previous blog post I discussed how migrating to the Cloud could help your organisation reach its Net Zero goals. I discussed how shifting your workloads away from on-premises data centres can reduce emissions by allowing you to leverage the expertise of cloud providers and their greater efficiency of scale. It should be noted this isn’t always clear cut - do consider how energy efficient your current hosting is and the embodied carbon of any hardware you’d be decommissioning.
Snowflake
OCTOBER 17, 2023
See how ctrl+s provides in-depth insights into supply chain sustainability, while protecting sensitive customer information—all through Snowflake’s powerful, scalable Data Cloud. Sustainability is an issue at the forefront of most companies’ agendas. But in complex, sprawling supply chains, identifying where carbon impacts come from can be extremely difficult.
Expert insights. Personalized for you.
264 
Let's personalize your content