Seattle Data Guy

APRIL 25, 2024

As data increased in volume, velocity, and variety, so, in turn, did the need for tools that could help process and manage those larger data sets coming at us at ever faster speeds. As a result, frameworks such as Apache Spark and Apache Flink became popular due to their abilities to handle big data processing… Read more The post Apache Spark Vs Apache Flink – How To Choose The Right Solution appeared first on Seattle Data Guy.

Big Data 130

Big Data Data Process Process Management 130

Waitingforcode

APRIL 24, 2024

As a data engineer you're certainly familiar with data skew. Yes, this bad phenomena where one task takes considerably more input than the others and often causes unexpected latency or failures. Turns out, stream processing also has its skew but more related to time.

Process 130

Process Data Engineering Data Engineer Engineering 130

The Pragmatic Engineer

APRIL 21, 2024

See a longer version of this article here: Scaling ChatGPT: Five Real-World Engineering Challenges. Sometimes the best explanations of how a technology solution works come from the software engineers who built it. To explain how ChatGPT (and other large language models) operate, I turned to the ChatGPT engineering team. "How does ChatGPT work, under the hood?

Engineering 328

Engineering Software Engineer Software Engineering Programming 328

Data Engineering Podcast

APRIL 21, 2024

Summary Generative AI has rapidly transformed everything in the technology sector. When Andrew Lee started work on Shortwave he was focused on making email more productive. When AI started gaining adoption he realized that he had even more potential for a transformative experience. In this episode he shares the technical challenges that he and his team have overcome in integrating AI into their product, as well as the benefits and features that it provides to their customers.

Data Lake 173

Data Lake High Quality Data Data Pipeline Machine Learning 173

Advertisement

Many organizations today are unlocking the power of their data by using graph databases to feed downstream analytics, enahance visualizations, and more. Yet, when different graph nodes represent the same entity, graphs get messy. Watch this essential video with Senzing CEO Jeff Jonas on how adding entity resolution to a graph database condenses network graphs to improve analytics and save your analysts time.

Database

Start Data Engineering

APRIL 22, 2024

1. Introduction 2. Ensure the code’s logic is working as expected with tests 2.1. Test types for data pipelines 2.2. pytest: A powerful Python library for testing 2.2.1. Set context, run code, check results & clean up 2.2.2. Tests are identified by their name 2.2.3. Use fixture to create fake data for testing 2.2.4. Define items to be shared among tests with conftest.

Coding 130

Coding Data Pipeline Python Data 130

Snowflake

APRIL 24, 2024

Building top-tier enterprise-grade intelligence using LLMs has traditionally been prohibitively expensive and resource-hungry, and often costs tens to hundreds of millions of dollars. As researchers, we have grappled with the constraints of efficiently training and inferencing LLMs for years. Members of the Snowflake AI Research team pioneered systems such as ZeRO and DeepSpeed , PagedAttention / vLLM , and LLM360 which significantly reduced the cost of LLM training and inference, and open sourc

Amazon Web Services 145

Amazon Web Services SQL AWS Architecture 145

databricks

APRIL 24, 2024

Unlock the power of Apache Spark™ with Unity Catalog Lakeguard on Databricks Data Intelligence Platform. Run SQL, Python & Scala workloads with full data governance & cost-efficient multi-user compute.

Data Governance 109

Data Governance Government Scala SQL 109

Knowledge Hut

APRIL 26, 2024

Machine Learning is a sub-branch of Artificial Intelligence, used for the analysis of data. It learns from the data that is input and predicts the output from the data rather than being explicitly programmed. Machine Learning is among the fastest evolving trends in the I T industry. It has found tremendous use in sectors across industries, with its ability to solve complex problems which humans are not able to solve using traditional techniques.

Machine Learning 91

Machine Learning Algorithm Deep Learning Programming Language 91

Start Data Engineering

APRIL 22, 2024

1. Introduction 2. Docker concepts 2.1. Define the OS and its configurations with an image 2.2. Use the image to run containers 2.2.1. Communicate between containers and local OS 2.2.2. Start containers with docker CLI or compose 3. Conclusion 1. Introduction Docker can be overwhelming to start with. Most data projects use Docker to set up the data infra locally (and often in production).

Data Engineering 130

Data Engineering Data Engineer Engineering Data 130

Scott Logic

APRIL 22, 2024

Generative AI has been grabbing headlines, but many businesses are starting to feel left-behind. Large-model AI is becoming more and more influential in the market, and with the well-known tech giants starting to introduce easy-access AI stacks, a lot of businesses are left feeling that although there may be a use for AI in their business, they’re unable to see what use cases it might help them with.

BI 97

BI Software Engineer Software Engineering Algorithm 97

Speaker: Scott Sehlhorst

We know we want to create products which our customers find to be valuable. Whether we label it as customer-centric or product-led depends on how long we've been doing product management. There are three challenges we face when doing this. The obvious challenge is figuring out what our users need; the non-obvious challenges are in creating a shared understanding of those needs and in sensing if what we're doing is meeting those needs.

Certification

KDnuggets

APRIL 23, 2024

This article introduces retrieval augmented generation, which combines text generation with informaton retrieval in order to improve language model output.

113

113

databricks

APRIL 23, 2024

We're thrilled to announce the General Availability (GA) of Databricks Asset Bundles (DABs). With DABs you can easily bundle resources like jobs.

123

123

Knowledge Hut

APRIL 26, 2024

You are here to read this article, so we assume you are already aware of the terms “hacking”, “hackers,” and other words associated with unauthorized access. Penetration testing or ethical hacking is the process of attempting to gain access to target resources and perform actual attacks to find loopholes in the system and measure the strength of security.

Cloud 86

Cloud Systems Database SQL 86

Cloudera

APRIL 26, 2024

This month is Allyship April—a time dedicated to deepening our understanding of allyship and its profound impact on fostering inclusive cultures. Allyship isn’t merely a buzzword; it’s a fundamental commitment to actively support and advocate for marginalized individuals and communities within our organization. This month, we’ve engaged in meaningful conversations, challenged our assumptions, and committed to tangible actions that drive positive change.

Education 77

Education Engineering IT Management 77

Speaker: Timothy Chan, PhD., Head of Data Science

Are you ready to move beyond the basics and take a deep dive into the cutting-edge techniques that are reshaping the landscape of experimentation? 🌐 From Sequential Testing to Multi-Armed Bandits, Switchback Experiments to Stratified Sampling, Timothy Chan, Data Science Lead, is here to unravel the mysteries of these powerful methodologies that are revolutionizing how we approach testing.

Data Science

Snowflake

APRIL 23, 2024

If you’re a Snowflake customer using ServiceNow’s popular SaaS application to manage your digital workloads, data integration is about to get a lot easier — and less costly. Snowflake has announced the general availability of the Snowflake Connector for ServiceNow, available on Snowflake Marketplace. The connector provides immediate access to up-to-date ServiceNow data without the need to manually integrate against API endpoints.

Data Warehouse 78

Data Warehouse Data Integration Consulting Metadata 78

KDnuggets

APRIL 22, 2024

Are you an aspiring data scientist? If so, these free data science courses from Stanford will help you move forward in your data science journey!

Data Science 120

Data Science Data 120

databricks

APRIL 23, 2024

For a limited time, we're offering 50% off training and certification at Data + AI Summit with the following code: TRAIN50FOTY. This offer.

Certification 99

Certification Coding Data 99

Knowledge Hut

APRIL 26, 2024

An agile framework is an iterative approach toward completing a project or a particular task under it. A framework helps in planning, managing, and executing tasks in a way that ensures successful project delivery. These frameworks are divided into two categories: frameworks that work within the teams and those that work at a larger scale for the entire organization.

Certification 82

Certification Project Programming Designing 82

Speaker: Aarushi Kansal, AI Leader & Author and Tony Karrer, Founder & CTO at Aggregage

Software leaders who are building applications based on Large Language Models (LLMs) often find it a challenge to achieve reliability. It’s no surprise given the non-deterministic nature of LLMs. To effectively create reliable LLM-based (often with RAG) applications, extensive testing and evaluation processes are crucial. This often ends up involving meticulous adjustments to prompts.

Building

Cloudera

APRIL 23, 2024

Back in October, we announced the first-ever Cloudera Climate and Sustainability Hackathon , powered by AMD. The Hackathon was intended to provide data science experts with access to Cloudera machine learning to develop their own Accelerated Machine Learning Project (AMP) focused on solving one of the many environmental challenges facing the world today.

Machine Learning 85

Machine Learning Data Science Retail Consulting 85

Zalando Engineering

APRIL 22, 2024

Introduction Our team is responsible for sending out communications to all our customers at Zalando - e.g. confirming a placed order, informing about new content from a favourite brand or announcing sales campaigns. During the preparation of those messages as well during sending those out via different service providers we have to deal with limited resources.

Algorithm 79

Algorithm Systems Kafka Designing 79

KDnuggets

APRIL 26, 2024

Find out all about Google Cloud's latest learning path, and learn how to use the Gemini language model in the Google Cloud.

Google Cloud 100

Google Cloud Cloud 100

databricks

APRIL 26, 2024

We’re excited to announce the Databricks Generative AI Hackathon winners. This hackathon garnered hundreds of data and AI practitioners spanning 60 invited companies.

Data 67

Data 67

Advertisement

Outdated or absent analytics won’t cut it in today’s data-driven applications – not for your end users, your development team, or your business. That’s what drove the five companies in this e-book to change their approach to analytics. Download this e-book to learn about the unique problems each company faced and how they achieved huge returns beyond expectation by embedding analytics into applications.

Building

Knowledge Hut

APRIL 26, 2024

What is Python 3? Python 3 is an interpreted language, which means that anyone can read and execute the code. Python is used to create websites, perform scientific research, data analysis etc. Python 3.9 is the latest version of Python. Why Learn Python 3? Python is one of the fastest growing and in-demand programming languages. It has a very easy learning curve, due in large part to its simple, user-friendly syntax.

Python 80

Python Programming Language Programming Data Analysis 80

ArcGIS

APRIL 22, 2024

A drawing alert notification system was added in ArcGIS Pro 3.2 as a method for resolving drawing issues in your ArcGIS Pro projects.

Project 86

Project Systems Data Management Database 86

Precisely

APRIL 25, 2024

Jean-Paul Otte recently joined Precisely as Head of Data Strategy Services for Europe. His specialty? Data! Jean-Paul sat down for an interview where we discussed his background as a former CDO, the challenges he faced, and how he developed his unique perspective and data governance expertise. Hello Jean-Paul, could you tell us a little about your background?

Data Governance 64

Data Governance Government Consulting BI 64

KDnuggets

APRIL 25, 2024

List of top MLOPs platforms that will help you with integration, training, tracking, deployment, monitoring, CI/CD, and optimizing the infrastructure.

90

90

Advertisement

Are you trying to decide which entity resolution capabilities you need? It can be confusing to determine which features are most important for your project. And sometimes key features are overlooked. Get the Entity Resolution Evaluation Checklist to make sure you’ve thought of everything to make your project a success! The list was created by Senzing’s team of leading entity resolution experts, based on their real-world experience.

Project

DoorDash Engineering

APRIL 23, 2024

DoorDash’s retail catalog is a centralized dataset of essential product information for all products sold by new verticals merchants – merchants operating a business other than a restaurant, such as a grocery, a convenience store, or a liquor store. Within the retail catalog, each SKU , or stock keeping unit, is represented by a list of product attributes.

Building 68

Building Retail Manufacturing Unstructured Data 68

Knowledge Hut

APRIL 26, 2024

The era of rapid change We are living in an era where change has become the norm rather than an exception. Emerging technologies and market unpredictability have further fueled change, impacting all industries globally. But the true test of an organization's capability is its ability to endure change and adapt to it. This is the philosophy of ‘Kaizen’ or changing for the better, that helps organizations stay competitive, relevant and in focus with the customer.

Certification 78

Certification Project Portfolio Consulting 78

ArcGIS

APRIL 23, 2024

Do you have questions about how to access, use, or nominate content within ArcGIS Living Atlas of the World? Check out this blog for answers.

Accessible 78

Accessible Accessibility Data Management Management 78

Snowflake

APRIL 22, 2024

Magnite stands as the largest independent sell-side advertising platform, providing an essential bridge between publishers and advertisers. At its core, Magnite streamlines the advertising process, facilitating the buying and selling of advertising space across various channels, including connected TV (CTV), mobile, and desktop environments. By leveraging advanced technology and data analytics, Magnite offers a comprehensive suite of tools and services designed to maximize ad revenue for publish

AWS 68

AWS Cloud Storage Cloud Technology 68

Speaker: Anne Steiner and David Laribee

As a concept, Developer Experience (DX) has gained significant attention in the tech industry. It emphasizes engineers’ efficiency and satisfaction during the product development process. As product managers, we need to understand how a good DX can contribute not only to the well-being of our development teams but also to the broader objectives of product success and customer satisfaction.

Engineering