The Pragmatic Engineer
OCTOBER 18, 2024
The below was originally published in The Pragmatic Engineer. To get timely analysis on the tech industry like this, on a weekly basis: sign up to The Pragmatic Engineer Newsletter. If you are into podcasts, check out The Pragmatic Engineer Podcast. Imagine Apple decided Spotify was a big enough business threat that it had to take unfair measures to limit Spotify’s growth on the App Store.
Simon Späti
JULY 28, 2024
A Vim-Inspired Approach to Efficient Note Management with Obsidian and Markdown
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
KDnuggets
OCTOBER 16, 2024
Where can you find projects dealing with advanced ML topics? GitHub is a perfect source with its many repositories. I’ve selected ten to talk about in this article.
Robinhood
NOVEMBER 13, 2024
Robinhood Crypto’s commitment to expanding access and maintaining a safe, easy-to-use platform deepens with the addition of 4 digital assets Today, Robinhood Crypto announced the addition of Solana (SOL), Pepe (PEPE), Cardano (ADA) & XRP (XRP) to its U.S. platform, bringing the total number of cryptocurrencies available for trading to 19. You can see a full list of crypto assets currently available in the U.S. here.
Advertisement
Whether you’re creating complex dashboards or fine-tuning large language models, your data must be extracted, transformed, and loaded. ETL and ELT pipelines form the foundation of any data product, and Airflow is the open-source data orchestrator specifically designed for moving and transforming data in ETL and ELT pipelines. This eBook covers: An overview of ETL vs.
Tweag
NOVEMBER 20, 2024
Two years ago I wrote a blog post to announce that the GHC wasm backend had been merged upstream. I’ve been too lazy to write another blog post about the project since then, but rest assured, the project hasn’t stagnated. A lot of improvements have happened after the initial merge, including but not limited to: Many, many bugfixes in the code generator and runtime, witnessed by the full GHC testsuite for the wasm backend in upstream GHC CI pipelines.
Rockset
JUNE 21, 2024
I’m excited to share that OpenAI has completed the acquisition of Rockset. We are thrilled to join the OpenAI team and bring our technology and expertise to building safe and beneficial AGI. From the start, our vision at Rockset was to fundamentally transform the way data-driven applications were built. We developed our search and analytics database, taking full advantage of the cloud, to eliminate the complexity inherent in the data infrastructure needed for these apps.
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
Engineering at Meta
DECEMBER 3, 2024
Meta releases a Request for Proposals (RFP) to identify nuclear energy developers to support AI innovation and clean and renewable energy goals.
Yelp Engineering
OCTOBER 6, 2024
The Yelp Reservations service (yelp_res) is the service that powers reservations on Yelp. It was acquired along with Seatme in 2013, and is a Django service and webapp. It powers the reservation backend and logic for Yelp Guest Manager, our iPad app for restaurants, and handles diner and partner flows that create reservations. Along with that, it serves a web UI and backend API for our Yelp Reservations app, which has been superseded by Yelp Guest Manager but is still used by many of our restaur
Confluent
SEPTEMBER 9, 2024
Confluent has acquired WarpStream, an innovative Kafka-compatible streaming solution. Read the full statement by Jay Kreps, co-founder and CEO of Confluent.
databricks
SEPTEMBER 19, 2024
Mosaic AI Model Training now supports fine-tuning up to 131K context length for Llama 3.1 models. More efficient training at long sequence lengths is made possible by several optimizations highlighted in this post.
Speaker: Tamara Fingerlin, Developer Advocate
In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!
The Pragmatic Engineer
APRIL 21, 2024
See a longer version of this article here: Scaling ChatGPT: Five Real-World Engineering Challenges. Sometimes the best explanations of how a technology solution works come from the software engineers who built it. To explain how ChatGPT (and other large language models) operate, I turned to the ChatGPT engineering team. "How does ChatGPT work, under the hood?
Rock the JVM
MAY 22, 2024
by Herbert Kateu 1. Introduction This article is a follow-up to the websocket article that was published previously. To recap, we created an in-memory chat application using WebSockets with the help of the Http4s library. The chat application had a variety of features implemented through commands directly in the chat window such as the ability to create users, create chat rooms, and switch between chat rooms.
KDnuggets
MARCH 27, 2024
Learn everything about data science by exploring our curated collection of free courses from top universities, covering essential topics from math and programming to machine learning, and mastering the nine steps to become a job-ready data scientist.
Robinhood
MAY 30, 2024
Robinhood Crypto customers in the United States can now use our API to view crypto market data, manage portfolios and account information, and place crypto orders programmatically Today, we are excited to announce the Robinhood Crypto trading API , ushering in a new era of convenience, efficiency, and strategy for our most seasoned crypto traders. Robinhood Crypto customers in the United States can use our new trading API to set up advanced and automated trading strategies that allow them to st
Advertisement
Many software teams have migrated their testing and production workloads to the cloud, yet development environments often remain tied to outdated local setups, limiting efficiency and growth. This is where Coder comes in. In our 101 Coder webinar, you’ll explore how cloud-based development environments can unlock new levels of productivity. Discover how to transition from local setups to a secure, cloud-powered ecosystem with ease.
ArcGIS
MAY 7, 2024
Discover the exciting new features of ArcGIS Pro 3.3. From water flow modeling to direct PDF support, this release has it all. Read our blog to learn more.
Tweag
JANUARY 17, 2024
I recently merged linear let- and where-bindings in GHC. Which means that we’ll have these in GHC 9.10, which is cause for celebration for me. Though they are much overdue, so maybe I should instead apologise to you. Anyway, I thought I’d take the opportunity to discuss some of GHC’s inner workings and how they explain some of the features of linear types in Haskell.
Data Engineering Weekly
AUGUST 6, 2024
TL;DR Aswin and I are thrilled to announce the release of the first version of our comprehensive guide for evaluating Change Data Capture. CDC Evaluation Guide Google Sheet Link: [link] CDC Evaluation Guide Github Link: [link] Change Data Capture (CDC) is a powerful technology in data engineering that allows for continuously capturing changes (inserts, updates, and deletes) made to source systems.
Engineering at Meta
MARCH 12, 2024
Marking a major investment in Meta’s AI future, we are announcing two 24k GPU clusters. We are sharing details on the hardware, network, storage, design, performance, and software that help us extract high throughput and reliability for various AI workloads. We use this cluster design for Llama 3 training. We are strongly committed to open compute and open source.
Advertisement
Large enterprises face unique challenges in optimizing their Business Intelligence (BI) output due to the sheer scale and complexity of their operations. Unlike smaller organizations, where basic BI features and simple dashboards might suffice, enterprises must manage vast amounts of data from diverse sources. What are the top modern BI use cases for enterprise businesses to help you get a leg up on the competition?
DoorDash Engineering
FEBRUARY 27, 2024
We reviewed the architecture of our global search at DoorDash in early 2022 and concluded that our rapid growth meant within three years we wouldn’t be able to scale the system efficiently, particularly as global search shifted from store-only to a hybrid item-and-store search experience. Our analysis identified Elasticsearch as our architecture’s primary bottleneck.
Confluent
MARCH 19, 2024
Seamlessly integrate Apache Kafka data into your lakehouse as Apache Iceberg tables, bridging the operational and analytical divide, with Tableflow. Read more in our blog post.
databricks
JUNE 12, 2024
“FactSet’s mission is to empower clients to make data-driven decisions and supercharge their workflows and productivity. To deliver AI-driven solutions across our entire.
The Pragmatic Engineer
OCTOBER 10, 2024
Automattic, creator of Wordpress, is being sued by one of the largest WordPress hosting providers. The conflict fits into a trend of billion-dollar companies struggling to effectively monetize open source, and are changing tactics to limit their competition and increase their revenue. This article was originally published a week ago, on 3 October 2024, in The Pragmatic Engineer.
Speaker: Jay Allardyce, Deepak Vittal, Terrence Sheflin, and Mahyar Ghasemali
As we look ahead to 2025, business intelligence and data analytics are set to play pivotal roles in shaping success. Organizations are already starting to face a host of transformative trends as the year comes to a close, including the integration of AI in data analytics, an increased emphasis on real-time data insights, and the growing importance of user experience in BI solutions.
Snowflake
APRIL 8, 2024
In 2020, Snowflake announced a new global competition to recognize the work of early-stage startups building their apps — and their businesses — on Snowflake, offering up to $250,000 in investment as the top prize. Four years later, the Snowflake Startup Challenge has grown into a premiere showcase for emerging startups, garnering interest from companies in over 100 countries and offering a prize package featuring a portion of up to $1 million in potential investment opportunities and exclusive
KDnuggets
NOVEMBER 6, 2024
Build an LLM application by easily picking and dropping components and connecting them, such as a vector store, web search, memory, and custom prompt.
Robinhood
OCTOBER 10, 2024
When Robinhood was founded, we set out to build a platform that gives everyone access to the financial markets. Over the last decade, we’ve disrupted and changed the industry for the better, becoming the first U.S. retail broker to offer commission-free trading, and saving investors billions in the process. In recent years, we’ve expanded our offering, ushering in a number of new cutting-edge products and services that help everyone – regardless of income – trade, invest, and earn.
ArcGIS
OCTOBER 21, 2024
By 2050, projected atmospheric carbon dioxide levels could nearly double, causing a 4.4°C temperature increase by the end of the century. Our study projected over 53% of Türkiye's Aegean olive-growing regions may become unsuitable for cultivation. Using GIS and ArcGIS Living Atlas Layers, we can identify vulnerable areas in future conditions and assess climate change impacts on Türkiye's significant olive production for conservation and land management.
Speaker: Nikhil Joshi, Founder & President of Snic Solutions
Is your manufacturing operation reaching its efficiency potential? A Manufacturing Execution System (MES) could be the game-changer, helping you reduce waste, cut costs, and lower your carbon footprint. Join Nikhil Joshi, Founder & President of Snic Solutions, in this value-packed webinar as he breaks down how MES can drive operational excellence and sustainability.
Tweag
MARCH 6, 2024
Three years ago, a blog post introduced destination-passing style (DPS) programming in Haskell, focusing on array processing, for which the API was made safe thanks to Linear Haskell. Today, I’ll present a slightly different API to manipulate arbitrary data types in a DPS fashion, and show why it can be useful for some parts of your programs. The present blog post is mostly based on my recent paper Destination-passing style programming: a Haskell implementation , published at JFLA 2024.
Precisely
SEPTEMBER 23, 2024
Are mainframes still relevant today? You bet! The following ten statistics paint a picture that shows mainframes are still going strong, with no signs of slowing. 1. The Mainframe Turns 60: A Milestone in Computing History. 60 years can really fly by! On April 7, 2024 , the Mainframe turned 60. At this milestone, we should all reflect on what the mainframe has done to the computing industry.
Engineering at Meta
JUNE 12, 2024
As we continue to focus our AI research and development on solving increasingly complex problems, one of the most significant and challenging shifts we’ve experienced is the sheer scale of computation required to train large language models (LLMs). Traditionally, our AI model training has involved a training massive number of models that required a comparatively smaller number of GPUs.
Data Engineering Weekly
JULY 10, 2024
The origin - a legend The origin of the modern data stack is a topic of intense debate, shrouded in uncertainty and mystery. Some attribute its incubation to Snowflake, Redshift, or Airflow, while others propose different theories. Rather than being the result of a single event, the term "modern data stack" emerged from a series of innovations and industry shifts, adding to the intrigue of its history.
Speaker: Anindo Banerjea, CTO at Civio & Tony Karrer, CTO at Aggregage
When developing a Gen AI application, one of the most significant challenges is improving accuracy. This can be especially difficult when working with a large data corpus, and as the complexity of the task increases. The number of use cases/corner cases that the system is expected to handle essentially explodes. 💥 Anindo Banerjea is here to showcase his significant experience building AI/ML SaaS applications as he walks us through the current problems his company, Civio, is solving.
Expert insights. Personalized for you.
322 
Let's personalize your content