data_service

package
v0.9.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 4, 2025 License: MIT Imports: 10 Imported by: 0

README

NWPC Data Service

A service for data files produced by operation systems running by NWPC.

Build

Build from source code.

Use Makefile in project root directory to build nwpc_data_server in Linux.

Server

Start the data server on port 33483:

nwpc_data_server serve --address ":33483" --config-dir=some/config/dir

Client

Use nwpc_data_client service to connect to nwpc_data_server.

Use --action to specify remote action. There are three actions.

findDataPath

Get data path on server.

nwpc_data_client service --address=data-service-address \
    --action findDataPath \
    --data-type=some/data/type \
    start_time forecast_time

Output is same as nwpc_data_client hpc command.

getDataFileInfo

Get data file information on server.

nwpc_data_client service --address=data-service-address \
    --action getDataFileInfo \
    --data-type=some/data/type \
    --start-time=start_time \
    --forecast-time=forecast_time

When data file is found, two lines are printed: one for data file path and the other for data file size.

/g2/nwp/OPER_ARCH_TEST/nwp/GRAPES_GFS/GMF_GRAPES_GFS/Prod-grib/2019061621/ORIG/gmf.gra.2019061700000.grb2
303997137

When data file is not found or there is some error, error message is printed stderr. Such as

check file error: stat NOTFOUND: no such file or directory
downloadDataFile

Download data file from server.

nwpc_data_client service --address=data-service-address \
    --action downloadDataFile \
    --output-dir=outout/dir \
    --data-type=some/data/type \
    --start-time=start_time \
    --forecast-time=forecast_time

Data file is saved on output dir with original file name on remote server.

Develop

Prepare

Prepare build environment.

Download and build gPRC library.

Get gRPC plugin for golang.

The following commands can be used in Windows.

go get -u google.golang.org/grpc
go get -u github.com/golang/protobuf/protoc-gen-go
set PATH=%PATH%;%GOPATH%/bin
Upgrade gRPC codes

Run the following codes to re-generate gRPC codes after change data_service.proto.

cd data_service
protoc.exe ^
    -I data_service ^
    data_service/data_service.proto ^
    --go_out=plugins=grpc:.

License

Copyright © 2019 Perilla Roc at nwpc-oper.

nwpc-data-service is licensed under The MIT License.

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var StatusCode_name = map[int32]string{
	0: "Unknown",
	1: "Success",
	2: "Failed",
}
View Source 
var StatusCode_value = map[string]int32{
	"Unknown": 0,
	"Success": 1,
	"Failed":  2,
}

Functions

func RegisterNWPCDataServiceServer 

func RegisterNWPCDataServiceServer(s *grpc.Server, srv NWPCDataServiceServer)

Types

type DataFileResponse 

type DataFileResponse struct {
	Status               StatusCode `protobuf:"varint,1,opt,name=status,proto3,enum=StatusCode" json:"status,omitempty"`
	ErrorMessage         string     `protobuf:"bytes,2,opt,name=error_message,json=errorMessage,proto3" json:"error_message,omitempty"`
	FilePath             string     `protobuf:"bytes,3,opt,name=file_path,json=filePath,proto3" json:"file_path,omitempty"`
	FileSize             int64      `protobuf:"varint,4,opt,name=file_size,json=fileSize,proto3" json:"file_size,omitempty"`
	XXX_NoUnkeyedLiteral struct{}   `json:"-"`
	XXX_unrecognized     []byte     `json:"-"`
	XXX_sizecache        int32      `json:"-"`
}

func (*DataFileResponse) Descriptor 

func (*DataFileResponse) Descriptor() ([]byte, []int)

func (*DataFileResponse) GetErrorMessage 

func (m *DataFileResponse) GetErrorMessage() string

func (*DataFileResponse) GetFilePath 

func (m *DataFileResponse) GetFilePath() string

func (*DataFileResponse) GetFileSize 

func (m *DataFileResponse) GetFileSize() int64

func (*DataFileResponse) GetStatus 

func (m *DataFileResponse) GetStatus() StatusCode

func (*DataFileResponse) ProtoMessage 

func (*DataFileResponse) ProtoMessage()

func (*DataFileResponse) Reset 

func (m *DataFileResponse) Reset()

func (*DataFileResponse) String 

func (m *DataFileResponse) String() string

func (*DataFileResponse) XXX_DiscardUnknown 

func (m *DataFileResponse) XXX_DiscardUnknown()

func (*DataFileResponse) XXX_Marshal 

func (m *DataFileResponse) XXX_Marshal(b []byte, deterministic bool) ([]byte, error)

func (*DataFileResponse) XXX_Merge 

func (m *DataFileResponse) XXX_Merge(src proto.Message)

func (*DataFileResponse) XXX_Size 

func (m *DataFileResponse) XXX_Size() int

func (*DataFileResponse) XXX_Unmarshal 

func (m *DataFileResponse) XXX_Unmarshal(b []byte) error

type DataPathResponse 

type DataPathResponse struct {
	LocationType         string   `protobuf:"bytes,1,opt,name=location_type,json=locationType,proto3" json:"location_type,omitempty"`
	Location             string   `protobuf:"bytes,2,opt,name=location,proto3" json:"location,omitempty"`
	XXX_NoUnkeyedLiteral struct{} `json:"-"`
	XXX_unrecognized     []byte   `json:"-"`
	XXX_sizecache        int32    `json:"-"`
}

func (*DataPathResponse) Descriptor 

func (*DataPathResponse) Descriptor() ([]byte, []int)

func (*DataPathResponse) GetLocation 

func (m *DataPathResponse) GetLocation() string

func (*DataPathResponse) GetLocationType 

func (m *DataPathResponse) GetLocationType() string

func (*DataPathResponse) ProtoMessage 

func (*DataPathResponse) ProtoMessage()

func (*DataPathResponse) Reset 

func (m *DataPathResponse) Reset()

func (*DataPathResponse) String 

func (m *DataPathResponse) String() string

func (*DataPathResponse) XXX_DiscardUnknown 

func (m *DataPathResponse) XXX_DiscardUnknown()

func (*DataPathResponse) XXX_Marshal 

func (m *DataPathResponse) XXX_Marshal(b []byte, deterministic bool) ([]byte, error)

func (*DataPathResponse) XXX_Merge 

func (m *DataPathResponse) XXX_Merge(src proto.Message)

func (*DataPathResponse) XXX_Size 

func (m *DataPathResponse) XXX_Size() int

func (*DataPathResponse) XXX_Unmarshal 

func (m *DataPathResponse) XXX_Unmarshal(b []byte) error

type DataRequest 

type DataRequest struct {
	DataType             string   `protobuf:"bytes,1,opt,name=data_type,json=dataType,proto3" json:"data_type,omitempty"`
	LocationLevels       []string `protobuf:"bytes,2,rep,name=location_levels,json=locationLevels,proto3" json:"location_levels,omitempty"`
	StartTime            string   `protobuf:"bytes,3,opt,name=start_time,json=startTime,proto3" json:"start_time,omitempty"`
	ForecastTime         string   `protobuf:"bytes,4,opt,name=forecast_time,json=forecastTime,proto3" json:"forecast_time,omitempty"`
	XXX_NoUnkeyedLiteral struct{} `json:"-"`
	XXX_unrecognized     []byte   `json:"-"`
	XXX_sizecache        int32    `json:"-"`
}

func (*DataRequest) Descriptor 

func (*DataRequest) Descriptor() ([]byte, []int)

func (*DataRequest) GetDataType 

func (m *DataRequest) GetDataType() string

func (*DataRequest) GetForecastTime 

func (m *DataRequest) GetForecastTime() string

func (*DataRequest) GetLocationLevels 

func (m *DataRequest) GetLocationLevels() []string

func (*DataRequest) GetStartTime 

func (m *DataRequest) GetStartTime() string

func (*DataRequest) ProtoMessage 

func (*DataRequest) ProtoMessage()

func (*DataRequest) Reset 

func (m *DataRequest) Reset()

func (*DataRequest) String 

func (m *DataRequest) String() string

func (*DataRequest) XXX_DiscardUnknown 

func (m *DataRequest) XXX_DiscardUnknown()

func (*DataRequest) XXX_Marshal 

func (m *DataRequest) XXX_Marshal(b []byte, deterministic bool) ([]byte, error)

func (*DataRequest) XXX_Merge 

func (m *DataRequest) XXX_Merge(src proto.Message)

func (*DataRequest) XXX_Size 

func (m *DataRequest) XXX_Size() int

func (*DataRequest) XXX_Unmarshal 

func (m *DataRequest) XXX_Unmarshal(b []byte) error

type FileContentResponse 

type FileContentResponse struct {
	ChunkLength          int64    `protobuf:"varint,3,opt,name=chunk_length,json=chunkLength,proto3" json:"chunk_length,omitempty"`
	Chunk                []byte   `protobuf:"bytes,4,opt,name=chunk,proto3" json:"chunk,omitempty"`
	XXX_NoUnkeyedLiteral struct{} `json:"-"`
	XXX_unrecognized     []byte   `json:"-"`
	XXX_sizecache        int32    `json:"-"`
}

func (*FileContentResponse) Descriptor 

func (*FileContentResponse) Descriptor() ([]byte, []int)

func (*FileContentResponse) GetChunk 

func (m *FileContentResponse) GetChunk() []byte

func (*FileContentResponse) GetChunkLength 

func (m *FileContentResponse) GetChunkLength() int64

func (*FileContentResponse) ProtoMessage 

func (*FileContentResponse) ProtoMessage()

func (*FileContentResponse) Reset 

func (m *FileContentResponse) Reset()

func (*FileContentResponse) String 

func (m *FileContentResponse) String() string

func (*FileContentResponse) XXX_DiscardUnknown 

func (m *FileContentResponse) XXX_DiscardUnknown()

func (*FileContentResponse) XXX_Marshal 

func (m *FileContentResponse) XXX_Marshal(b []byte, deterministic bool) ([]byte, error)

func (*FileContentResponse) XXX_Merge 

func (m *FileContentResponse) XXX_Merge(src proto.Message)

func (*FileContentResponse) XXX_Size 

func (m *FileContentResponse) XXX_Size() int

func (*FileContentResponse) XXX_Unmarshal 

func (m *FileContentResponse) XXX_Unmarshal(b []byte) error

type NWPCDataServer 

type NWPCDataServer struct {
	ConfigDir string
}

func (*NWPCDataServer) DownloadDataFile 

func (s *NWPCDataServer) DownloadDataFile(req *DataRequest, stream NWPCDataService_DownloadDataFileServer) error

func (*NWPCDataServer) FindDataPath 

func (s *NWPCDataServer) FindDataPath(ctx context.Context, req *DataRequest) (*DataPathResponse, error)

func (*NWPCDataServer) GetDataFileInfo 

func (s *NWPCDataServer) GetDataFileInfo(ctx context.Context, req *DataRequest) (*DataFileResponse, error)

type NWPCDataServiceClient 

type NWPCDataServiceClient interface {
	FindDataPath(ctx context.Context, in *DataRequest, opts ...grpc.CallOption) (*DataPathResponse, error)
	GetDataFileInfo(ctx context.Context, in *DataRequest, opts ...grpc.CallOption) (*DataFileResponse, error)
	DownloadDataFile(ctx context.Context, in *DataRequest, opts ...grpc.CallOption) (NWPCDataService_DownloadDataFileClient, error)
}

NWPCDataServiceClient is the client API for NWPCDataService service.

For semantics around ctx use and closing/ending streaming RPCs, please refer to https://godoc.org/google.golang.org/grpc#ClientConn.NewStream.

func NewNWPCDataServiceClient 

func NewNWPCDataServiceClient(cc *grpc.ClientConn) NWPCDataServiceClient

type NWPCDataServiceServer 

type NWPCDataServiceServer interface {
	FindDataPath(context.Context, *DataRequest) (*DataPathResponse, error)
	GetDataFileInfo(context.Context, *DataRequest) (*DataFileResponse, error)
	DownloadDataFile(*DataRequest, NWPCDataService_DownloadDataFileServer) error
}

NWPCDataServiceServer is the server API for NWPCDataService service.

type NWPCDataService_DownloadDataFileClient 

type NWPCDataService_DownloadDataFileClient interface {
	Recv() (*FileContentResponse, error)
	grpc.ClientStream
}

type NWPCDataService_DownloadDataFileServer 

type NWPCDataService_DownloadDataFileServer interface {
	Send(*FileContentResponse) error
	grpc.ServerStream
}

type StatusCode 

type StatusCode int32
const (
	StatusCode_Unknown StatusCode = 0
	StatusCode_Success StatusCode = 1
	StatusCode_Failed  StatusCode = 2
)

func (StatusCode) EnumDescriptor 

func (StatusCode) EnumDescriptor() ([]byte, []int)

func (StatusCode) String 

func (x StatusCode) String() string

type UnimplementedNWPCDataServiceServer 

type UnimplementedNWPCDataServiceServer struct {
}

UnimplementedNWPCDataServiceServer can be embedded to have forward compatible implementations.

func (*UnimplementedNWPCDataServiceServer) DownloadDataFile 

func (*UnimplementedNWPCDataServiceServer) DownloadDataFile(req *DataRequest, srv NWPCDataService_DownloadDataFileServer) error

func (*UnimplementedNWPCDataServiceServer) FindDataPath 

func (*UnimplementedNWPCDataServiceServer) FindDataPath(ctx context.Context, req *DataRequest) (*DataPathResponse, error)

func (*UnimplementedNWPCDataServiceServer) GetDataFileInfo 

func (*UnimplementedNWPCDataServiceServer) GetDataFileInfo(ctx context.Context, req *DataRequest) (*DataFileResponse, error)

Source Files

View all Source files

Directories

Path Synopsis
cmd
test_client
download_file command
find_data_path command
get_data_file_info command

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.