Towards Data Science

DECEMBER 23, 2023

PySpark helps us clean, organize, and make sense of our catch, transforming raw data into valuable insights, much like how a skilled chef would prepare a variety of dishes from the day’s catch. We perform analytics to uncover insights and answers to questions, revealing the valuable pearls hidden within our sea of data.

SQL 74

SQL Data Analytics Hadoop Raw Data 74

Snowflake

SEPTEMBER 13, 2023

Data Governance Simplify masking policies on External Tables – public preview Previously administrators were required to configure a view on External Tables in order to mask sensitive columns. With this preview, administrators can simply associate masking policies to virtual columns in the External Table to mask sensitive columns.

Python 76

Python SQL Data Pipeline Data Ingestion 76

Towards Data Science

APRIL 6, 2023

For example, a unit test can verify that a column intended to be a boolean is indeed a boolean, but what if the percentage “true” in that column shifted a lot between one day and the next? Columns: Schema — are all the columns you expect present (and some columns you don’t?) Or even just a little bit.

Unstructured Data 81

Unstructured Data Metadata Data Coding 81

Data Engineering Podcast

SEPTEMBER 11, 2022

In order to reduce the amount of time and effort required to build pipelines that power critical insights Manish Jethani co-founded Hevo Data. All thanks to 50+ quality checks, extensive column-level lineage, and 20+ connectors across the Data Stack.

Data Pipeline 130

Data Pipeline Building MongoDB Scala 130

Snowflake

DECEMBER 12, 2023

At our recent Snowday event, we announced a wave of Snowflake product innovations for easier application development, new AI and LLM capabilities, better cost management and more. Snowflake allows adding new columns or dropping the NOT NULL constraint from columns missing in new data files. Learn more here. Learn more.

Metadata 83

Metadata Python Government AWS 83

Towards Data Science

OCTOBER 25, 2023

Contract tests scope: Checking column types. Checking expected constraints at the schema level like primary and foreign keys, not null columns. Checking accepted values for a given column. Checking valid ranges for a given column. Quality tests scope: Assessing completeness, e.g. percentage of not nulls in a column.

Data Pipeline 81

Data Pipeline SQL Data Architecture Data 81

Snowflake

NOVEMBER 1, 2023

That’s why we created Snowflake Cortex (in private preview), Snowflake’s new, intelligent, fully managed service that enables organizations to quickly analyze data and build AI applications — all within Snowflake. Users can also refine queries through conversation to filter down to the insights most relevant to the task.

Unstructured Data 97

Unstructured Data SQL Python Accessible 97

Workfall

JUNE 12, 2023

The possibilities become boundless when data-driven insights are at your fingertips. It finds applications in database replication, where it captures changes from the source database and updates target databases, ensuring data consistency without manual updates. Where Is CDC Used and Who Uses It?

Telecommunication 80

Telecommunication Metadata Healthcare Finance 80

Striim

NOVEMBER 17, 2023

In today’s data-driven world, the ability to leverage real-time data for machine learning applications is a game-changer. Inside this database, we have an iris_dataset table with the following column structure. Two key players in this field, Striim and Google BigQuery ML, offer a powerful combination to make this possible.

Machine Learning 52

Machine Learning Data Process PostgreSQL Process 52

Engineering at Meta

FEBRUARY 20, 2024

Arrow 15 not only enables efficient (zero-copy) in-memory communication across components using Velox and Arrow, but also increases Arrow’s applicability in modern execution engines, unlocking a variety of use cases across the industry. IF and SWITCH operations), the Velox Vectors layout has to allow developers to write columns out of order.

Data Management 95

Data Management Bytes Management Datasets 95

Snowflake

MARCH 21, 2023

Data Governance Track Masking & Row Access Policy References in Access History, Now in Public Preview For queries on a table or view protected by a row access policy and columns protected by a masking policy, Snowflake can now track the enforced masking and row access policies in the Account Usage ACCESS_HISTORY view. Commerce Placer.ai

Retail 64

Retail Healthcare Data Ingestion Consulting 64

Scott Logic

MARCH 25, 2024

Users will typically access the platform via the desktop application Power BI Desktop. Users will typically access the platform via the desktop application Tableau Desktop. Users accessing the server application from the internet are authenticated at the web tier. High-value data is protected by the third subnet: the data tier.

BI 52

BI Business Intelligence Cloud AWS 52

Towards Data Science

MARCH 2, 2024

For example, if the expectation is that the ID column should have no missing values, the presence of missing IDs would constitute a constraint violation. Typically, this column should only contain a single value. Given that this column should not have any missing entries, this issue is identified with 100% confidence for correction.

Datasets 62

Datasets Data Data Science Python 62

Snowflake

DECEMBER 20, 2023

Snowflake’s Data Cloud met ServiceNow’s foundational business needs for an EDP: Centralized data security: Snowflake’s Column-level Security, Row Access Policies, Object Tagging, and Dynamic Data Masking aligned with ServiceNow’s data security and governance objectives. We decided to go with Snowflake,” says Siva Prasad, Sr.

IT 63

IT Data Government Data Security 63

Towards Data Science

JANUARY 17, 2024

Think of Docker as a handy tool that simplifies running applications, and MinIO as a flexible storage solution perfect for handling lots of different types of data. Seamless Data Analytics Workflow: From Dockerized JupyterLab and MinIO to Insights with Spark SQL Once everything’s ready, we’ll start by preparing our sample data.

Big Data 72

Big Data Data Data Storage SQL 72

Snowflake

APRIL 20, 2023

Snowgrid: Cross-Cloud Data Governance Allow masked columns as inputs to row access policies and conditional masking policies – general availability (GA) Advanced use cases require a masked column in a mapping table to be used as input to a row access policy for row-level authorization. Learn more.

Medical 54

Medical Retail Python Pharmaceutical 54

Data Engineering Podcast

AUGUST 21, 2022

With their new managed database service you can launch a production ready MySQL, Postgres, or MongoDB cluster in minutes, with automated backups, 40 Gbps connections from your application hosts, and high throughput SSDs. All thanks to 50+ quality checks, extensive column-level lineage, and 20+ connectors across the Data Stack.

Lambda Architecture 100

Lambda Architecture MongoDB Scala MySQL 100

Knowledge Hut

JULY 21, 2023

Data visualization charts are an invaluable tool in simplifying data analysis and presenting insights in visually appealing ways. They allow stakeholders to quickly grasp key insights, make informed decisions, and communicate findings to others. It is simple to see how data evolves with column charts.

Raw Data 52

Raw Data BI Data Transparency Data 52

Databand.ai

JUNE 28, 2023

Specifically, observability provides insights into the pipeline’s internal states and how they interact with the system’s outputs. Yet reality is of course more closely governed by Murphy’s law, and on the output side of the black box, you will often see a host of strange values and cryptic missing columns. As do end users.

Data Pipeline 98

Data Pipeline Data Engineering Data Engineer Engineering 98

ThoughtSpot

MARCH 9, 2023

With this new integration, data teams will be able to exponentially increase their impact across an organization as business users self-serve personalized, actionable, and trustworthy insights like never before. Not without the right guardrails and application. Additional key features include: 1.

SQL 91

SQL Government Architecture Algorithm 91

DoorDash Engineering

JANUARY 30, 2024

High availability : With its replication strategy and decentralized nature, Cassandra guarantees high availability, making it a reliable choice for critical applications. Flexibility : Cassandra supports a flexible schema design, which is a boon for applications with evolving data structures.

NoSQL 84

NoSQL Database Systems Relational Database 84

Towards Data Science

JANUARY 24, 2024

While numerous articles shed light on various join strategies supported by Spark, the practical application of these strategies remains a prevalent concern for professionals in the field. David Vrba’s insightful article, “About Joins in Spark 3.0” , published on Towards Data Science, serves as a valuable resource.

SQL 78

SQL Data Science Datasets Data Engineering 78

Data Engineering Podcast

FEBRUARY 20, 2022

Their SDKs and plugins make event streaming easy, and their integrations with cloud applications like Salesforce and ZenDesk help you go beyond event streaming. With RudderStack you can use all of your customer data to answer more difficult questions and then send those insights to your whole customer data stack.

Systems 100

Systems Software Engineer Software Engineering Data Warehouse 100

Knowledge Hut

OCTOBER 11, 2023

Using Power BI, you can create data stories, reports and interactive dashboards which help organizations to identify patterns, forecast demand, and derive business insights. The book provides insights into Power BI's functional programming languages of DAX and M.

BI 52

BI Business Intelligence Data Analysis Data Preparation 52

Workfall

JULY 25, 2023

Sheety is a web application that allows you to turn Google Sheets into RESTful APIs without the need for writing backend code. It acts as an intermediary between your Google Spreadsheet and the applications you build, enabling you to interact with your Google Sheets data programmatically. Why Sheety? Why Sheety?

Python 76

Python Programming Language Coding Project 76

Knowledge Hut

NOVEMBER 27, 2023

Data Science is a field that uses scientific methods, algorithms, and processes to extract useful insights and knowledge from noisy data. Try to perform basic functions like changing column names, counting rows, checking counts of different values, transposing, grouping, adding new columns, deleting existing columns, etc.

Data Science 71

Data Science Healthcare Machine Learning Telecommunication 71

Knowledge Hut

FEBRUARY 28, 2024

In this blog, we will explore the future of big data in business, its applications, and the technologies that will drive its evolution. It involves extracting valuable insights from these massive datasets to gain a competitive advantage. After all, big data applications are immense and can convince all the naysayers.

Big Data 52

Big Data Cloud Storage Healthcare SQL 52

Snowflake

MARCH 23, 2023

Further, a user can simply attach a tag to a table column and Snowflake will automatically enforce the associated data access policy (that is, column-level masking). Further, these views provide a native way of tracking frequency of use for tables/views and columns as well as track data access (selects) and data manipulations (DMLs).

Government 69

Government Data Governance Certification Programming 69

Knowledge Hut

APRIL 23, 2024

In the modern data-driven landscape, organizations continuously explore avenues to derive meaningful insights from the immense volume of information available. By leveraging distributed computing and parallel processing, big data platforms enable organizations to extract meaningful insights and patterns from massive datasets.

Data Warehouse 52

Data Warehouse Big Data Unstructured Data Hadoop 52

U-Next

JUNE 27, 2023

Excel has a variety of new tools, like the Pivot tables, that help improve analytical applications. It enables us to convert columns to rows and rows to columns. It helps aggregate data by any field (column) and do complex computations on it. Choose a field in the row or column to sort.

Utilities 52

Utilities Data Analysis Raw Data Aggregated Data 52

Data Engineering Podcast

OCTOBER 23, 2022

Summary Agile methodologies have been adopted by a majority of teams for building software applications. With their new managed database service you can launch a production ready MySQL, Postgres, or MongoDB cluster in minutes, with automated backups, 40 Gbps connections from your application hosts, and high throughput SSDs.

Project 130

Project Metadata MongoDB MySQL 130

ProjectPro

DECEMBER 14, 2021

Data insights, improved quality, and correct data condensed in a single document have become more critical. Marketing Campaign Insights Analysis 4. Loan Application Analysis 10. Covid-19 Insights Analysis 13. The application can simply include Power BI dashboards and reports to create a smooth user experience.

BI 52

BI Project Business Intelligence Datasets 52

Knowledge Hut

DECEMBER 7, 2023

Row-Level Security, Column-Level Security, and Object-Level Security are some of the various layers of that in Power BI. In the past, developers did not place a high premium on application security. Security issues were typically taken into account once an application was developed.

BI 52

BI Business Intelligence Datasets Data Analysis 52

Data Engineering Podcast

MARCH 1, 2021

Datafold helps Data teams gain visibility and confidence in the quality of their analytical data through data profiling, column-level lineage and intelligent anomaly detection. Their SDKs and plugins make event streaming easy, and their integrations with cloud applications like Salesforce and ZenDesk help you go beyond event streaming.

Machine Learning 100

Machine Learning Data Warehouse Scala Hadoop 100

Monte Carlo

FEBRUARY 8, 2024

GigaOm GigaOm’s Data Observability Radar Report covers the problem data observability tools look to solve saying, “Data observability is critical for countering, if not eliminating, data downtime, in which the results of analytics or the performance of applications are compromised because of unhealthy, inaccurate data.”

BI 59

BI Data Pipeline Data Warehouse Data 59

Snowflake

FEBRUARY 13, 2023

Data Governance Column Lineage now generally available Security and compliance teams need to tag and protect column data as it flows from known sensitive columns to new columns, which they do by using data manipulation commands such as CTAS, MERGE, and INSERT. Visit our documentation page to learn more. Learn more.

Insurance 52

Insurance Banking Government Retail 52

Knowledge Hut

JULY 26, 2023

Database applications have become vital in current business environments because they enable effective data management, integration, privacy, collaboration, analysis, and reporting. Database applications also help in data-driven decision-making by providing data analysis and reporting tools. What are Database Applications?

Database 52

Database NoSQL Telecommunication MongoDB 52