Documentation
¶
Overview ¶
Package bronze defines the strongly typed data models used for the bronze layer of the Data Master pipeline.
Each model represents a specific raw dataset (e.g., beer, brewery, profile, review) and is mapped to a corresponding Parquet schema via struct tags.
The package also provides a dynamic model registry that allows runtime loading and instantiation of models based on the fully qualified table name.
All models implement the Model interface, which supports schema-aware conversions from CSV and JSON inputs into typed Go structs.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Register ¶
func Register(m Model)
Register stores a model implementation in the global registry for later access.
It ensures that each model is registered only once, using the fully qualified table name (e.g., "schema.table") as the key. If a model is nil or has already been registered, the function panics to prevent silent misconfiguration.
Parameters:
- m: A model implementing the Model interface. Must not be nil and must have a unique TableName.
Usage:
Call this during application startup to make the model available for schema introspection, dynamic mapping, and downstream processing logic.
Types ¶
type Beer ¶
type Beer struct {
BeerID int64 `json:"beer_id" parquet:"beer_id"`
BreweryID int64 `json:"brewery_id" parquet:"brewery_id"`
BeerName string `json:"beer_name" parquet:"beer_name"`
BeerStyle string `json:"beer_style" parquet:"beer_style,dict"`
BeerAbv float64 `json:"beer_abv" parquet:"beer_abv,optional"`
Operation string `json:"operation" parquet:"operation,dict"`
}
Beer represents the schema of the beer dataset in the bronze layer.
This model is used for writing data in Parquet format and includes metadata required for downstream processing such as operation type.
Fields:
- BeerID: Unique identifier for the beer.
- BreweryID: Reference to the brewery that produced the beer.
- BeerName: Name of the beer.
- BeerStyle: Style/category of the beer (e.g., IPA, Stout).
- BeerAbv: Alcohol by volume (ABV) percentage. Optional field.
- Operation: Type of change (insert, update, delete) used for CDC processing.
type Brewery ¶
type Brewery struct {
BreweryId int64 `json:"brewery_id" parquet:"brewery_id"`
BreweryName string `json:"brewery_name" parquet:"brewery_name"`
Operation string `json:"operation" parquet:"operation,dict"`
}
Brewery represents the schema of the brewery dataset in the bronze layer.
This model is used for Parquet serialization and includes the minimum information necessary to identify and track breweries in the pipeline.
Fields:
- BreweryId: Unique identifier for the brewery.
- BreweryName: Name of the brewery.
- Operation: Type of change (insert, update, delete) for CDC tracking.
type Model ¶
func LoadModel ¶
LoadModel retrieves a registered model by its fully qualified table name.
It looks up the model from the global registry using the format "schema.table". If the model is not found, it returns an error indicating that the model has not been registered.
Parameters:
- fullyQualifiedTableName: The unique identifier for the model in the format "schema.table".
Returns:
- Model: The registered model instance.
- error: An error if the model is not found in the registry.
type Profile ¶
type Profile struct {
ProfileID int64 `json:"profile_id" parquet:"profile_id"`
ProfileName string `json:"profile_name" parquet:"profile_name"`
Email string `json:"email" parquet:"email"`
State string `json:"state" parquet:"state"`
Operation string `json:"operation" parquet:"operation,dict"`
}
Profile represents the schema of the profile dataset in the bronze layer.
This model is serialized in Parquet format and captures user profile details for downstream processing and enrichment.
Fields:
- ProfileID: Unique identifier for the profile.
- ProfileName: Display name of the user.
- Email: Email address associated with the profile.
- State: User's state or region.
- Operation: CDC operation type (insert, update, delete).
type Review ¶
type Review struct {
ReviewID int64 `json:"review_id" parquet:"review_id"`
BreweryID int64 `json:"brewery_id" parquet:"brewery_id"`
BeerID int64 `json:"beer_id" parquet:"beer_id"`
ProfileID int64 `json:"profile_id" parquet:"profile_id"`
ReviewOverall float64 `json:"review_overall" parquet:"review_overall,optional"`
ReviewAroma float64 `json:"review_aroma" parquet:"review_aroma,optional"`
ReviewAppearance float64 `json:"review_appearance" parquet:"review_appearance,optional"`
ReviewPalate float64 `json:"review_palate" parquet:"review_palate,optional"`
ReviewTaste float64 `json:"review_taste" parquet:"review_taste,optional"`
ReviewTime int64 `json:"review_time" parquet:"review_time,optional"`
Operation string `json:"operation" parquet:"operation,dict"`
}
Review represents the schema of the review dataset in the bronze layer.
This model is stored in Parquet format and captures detailed beer review data, including various rating aspects and relationships to beer, brewery, and user profile.
Fields:
- ReviewID: Unique identifier for the review.
- BreweryID: ID of the associated brewery.
- BeerID: ID of the reviewed beer.
- ProfileID: ID of the user who submitted the review.
- ReviewOverall: Overall rating given by the user.
- ReviewAroma: Aroma score given by the user.
- ReviewAppearance: Appearance score given by the user.
- ReviewPalate: Palate score given by the user.
- ReviewTaste: Taste score given by the user.
- ReviewTime: Timestamp (Unix) of when the review was submitted.
- Operation: CDC operation type (insert, update, delete).
Source Files