Towards Data Science

MARCH 5, 2024

Like a dragon guarding its treasure, each byte stored and each query executed demands its share of gold coins. Join as we journey through the depths of cost optimization, where every byte is a precious coin. It is also possible to set a maximum for the bytes billed for your query. Photo by Konstantin Evdokimov on Unsplash ?

Bytes 71

Bytes Google Cloud Cloud Storage Utilities 71

ProjectPro

JANUARY 31, 2023

Big data operations require specialized tools and techniques since a relational database cannot manage such a large amount of data. Typically, data processing is done using frameworks such as Hadoop, Spark, MapReduce, Flink, and Pig, to mention a few. How is Hadoop related to Big Data? RDBMS stores structured data.

Big Data 52

Big Data Hadoop AWS Relational Database 52

Grouparoo

OCTOBER 6, 2021

This provided a nice overview of the breadth of topics that are relevant to data engineering including data warehouses/lakes, pipelines, metadata, security, compliance, quality, and working with other teams. For example, grouping the ones about metadata, discoverability, and column naming might have made a lot of sense.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

ProjectPro

NOVEMBER 22, 2021

StructType is a collection of StructField objects that determines column name, column data type, field nullability, and metadata. To define the columns, PySpark offers the pyspark.sql.types import StructField class, which has the column name (String), column type (DataType), nullable column (Boolean), and metadata (MetaData).

Hadoop 52

Hadoop Python Datasets Metadata 52