testtools

package
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 13, 2018 License: Apache-2.0 Imports: 26 Imported by: 0

README

Testing Tools

WAL-G offers three prototyping programs to assist with testing and development:

compress

compress takes in a directory and minimum part size and creates a compressed tarball.

Example use cases:

Compress a local directory to disk.

./compress -out=$HOME/directory/to/be/compressed 1000000 directory/to/compress

Connect to Postgres and upload the compressed tarball to S3.

./compress -s 1000000 /dat/9.6/data
extract

extract takes in a target out directory and however many files/urls. It is often used in conjunction with generate to test the accuracy and speed of decompression and extraction.

Example use cases:

Extract local files.

./extract /target/out/directory file1 file2

Extract local files and randomly generated data.

./extract /out/directory file1                     \
https://localhost:8080/stride-10.bytes-100.tar.lzo \
https://localhost:8080/stride-100.bytes-1000.tar.lzo
generate

generate outputs randomly compressed tarballs hosted on localhost:8080. To configure the stride length N and size of the data M, use: https://localhost:8080/stride-N.bytes-M.tar.lzo

Randomly generated data can be downloaded using:

curl -k https://localhost:8080/stride-1.bytes-1.tar.lzo \
-o /path/to/new/file

The URLs can also be fed directly to extract. Currently, generate only supports LZOP compression.

NOTE: compress and extract support profiling options using the flags -p and -m. The first generates a CPU profile to cpu.prof while the latter generates a memory profile to mem.prof.

To access the profiles, use:

go tool pprof wal-g FILENAME
delta

delta runs concurrent testing of delta backups:

  • init pg_bench
  • make base backup
  • run bp_bench
  • run pg_bench and delta backup in parallel
  • make delta backup
  • restore delta chain
  • compare restoration results and highlight but difference
  • if critical differences found diff will exit with panic

delta must be invoked with env variables necessary for wal-g backup-push

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func AssertReaderIsEmpty 

func AssertReaderIsEmpty(t *testing.T, reader io.Reader)

func Contains 

func Contains(s *[]string, e string) bool

func CreateTar 

func CreateTar(w io.Writer, r *io.LimitedReader)

CreateTar creates a new tarball from the passed in reader and writes to a destination writer.

func Handler 

func Handler(w http.ResponseWriter, r *http.Request)

Handler allows for generation of random bytes by configuring the URL 'https://localhost:8080/stride-N.bytes-N.tar.lzo' where byte size and stride length are customizable.

Compressed tar files are automatically generated. Grab using curl ie. 'curl -sk ...'

func MakeDir 

func MakeDir(name string)

MakeDir creates a new directory with mode 0755.

func NewLz4CompressingPipeWriter 

func NewLz4CompressingPipeWriter(input io.Reader) *walg.CompressingPipeWriter

func NewMockS3Client 

func NewMockS3Client(err, notFound bool) *mockS3Client

func NewMockS3Folder 

func NewMockS3Folder(s3ClientErr, s3ClientNotFound bool) *walg.S3Folder

func NewMockS3Uploader 

func NewMockS3Uploader(multiErr, err bool, storage *MockStorage) *mockS3Uploader

func NewMockStoringS3Client 

func NewMockStoringS3Client(storage *MockStorage) *mockStoringS3Client

func NewMockUploader 

func NewMockUploader(apiMultiErr, apiErr bool) *walg.Uploader

func NewStoringCompressingMockUploader 

func NewStoringCompressingMockUploader(storage *MockStorage, deltaDataFolder walg.DataFolder) *walg.Uploader

func NewStoringMockS3Folder 

func NewStoringMockS3Folder(storage *MockStorage) *walg.S3Folder

func NewStoringMockUploader 

func NewStoringMockUploader(storage *MockStorage, deltaDataFolder walg.DataFolder) *walg.Uploader

func TimeTrack 

func TimeTrack(start time.Time, name string)

TimeTrack is used to time how long functions take.

Usage Example: defer timeTrack(time.Now(), "EXTRACT ALL")

Types

type BufferTarBall 

type BufferTarBall struct {
	// contains filtered or unexported fields
}

BufferTarBall represents a tarball that is written to buffer.

func (*BufferTarBall) AddSize 

func (tarBall *BufferTarBall) AddSize(add int64)

func (*BufferTarBall) AwaitUploads 

func (tarBall *BufferTarBall) AwaitUploads()

func (*BufferTarBall) CloseTar 

func (tarBall *BufferTarBall) CloseTar() error

func (*BufferTarBall) Finish 

func (tarBall *BufferTarBall) Finish(sentinelDto *walg.S3TarBallSentinelDto) error

func (*BufferTarBall) SetUp 

func (tarBall *BufferTarBall) SetUp(crypter walg.Crypter, args ...string)

func (*BufferTarBall) Size 

func (tarBall *BufferTarBall) Size() int64

func (*BufferTarBall) TarWriter 

func (tarBall *BufferTarBall) TarWriter() *tar.Writer

type BufferTarBallMaker 

type BufferTarBallMaker struct {
	BufferToWrite *bytes.Buffer
	// contains filtered or unexported fields
}

func (*BufferTarBallMaker) Make 

func (tarBallMaker *BufferTarBallMaker) Make(dedicatedUploader bool) walg.TarBall

type BufferTarInterpreter 

type BufferTarInterpreter struct {
	Out []byte
}

BufferTarInterpreter extracts data to a byte slice. Used for testing purposes.

func (*BufferTarInterpreter) Interpret 

func (tarInterpreter *BufferTarInterpreter) Interpret(reader io.Reader, header *tar.Header) error

Interpret handles in memory tar formats. Used for testing purposes.

type FileReaderMaker 

type FileReaderMaker struct {
	Key string
}

FileReaderMaker decompresses lzop tarballs from the passed in file.

func (*FileReaderMaker) Path 

func (f *FileReaderMaker) Path() string

func (*FileReaderMaker) Reader 

func (f *FileReaderMaker) Reader() (io.ReadCloser, error)

Reader creates a new reader from the passed in file.

type FileTarBall 

type FileTarBall struct {
	// contains filtered or unexported fields
}

FileTarBall represents a tarball that is written to disk.

func (*FileTarBall) AddSize 

func (tarBall *FileTarBall) AddSize(i int64)

func (*FileTarBall) AwaitUploads 

func (tarBall *FileTarBall) AwaitUploads()

func (*FileTarBall) CloseTar 

func (tarBall *FileTarBall) CloseTar() error

CloseTar closes the tar writer and file, flushing any unwritten data to the file before closing.

func (*FileTarBall) Finish 

func (tarBall *FileTarBall) Finish(sentinelDto *walg.S3TarBallSentinelDto) error

Finish alerts that compression is complete.

func (*FileTarBall) SetUp 

func (tarBall *FileTarBall) SetUp(crypter walg.Crypter, names ...string)

SetUp creates a new LZ4 writer, tar writer and file for writing bundled compressed bytes to.

func (*FileTarBall) Size 

func (tarBall *FileTarBall) Size() int64

func (*FileTarBall) TarWriter 

func (tarBall *FileTarBall) TarWriter() *tar.Writer

type FileTarBallMaker 

type FileTarBallMaker struct {
	Out string
	// contains filtered or unexported fields
}

FileTarBallMaker creates a new FileTarBall with the directory that files should be extracted to.

func (*FileTarBallMaker) Make 

func (tarBallMaker *FileTarBallMaker) Make(inheritState bool) walg.TarBall

Make creates a new FileTarBall.

type HTTPReaderMaker 

type HTTPReaderMaker struct {
	Client *http.Client
	Key    string
}

HTTPReaderMaker decompresses lzop tarballs from the passed in url.

func (*HTTPReaderMaker) Path 

func (h *HTTPReaderMaker) Path() string

func (*HTTPReaderMaker) Reader 

func (h *HTTPReaderMaker) Reader() (io.ReadCloser, error)

Reader creates a new request to grab the data generated by the random bytes generator.

type MockCompressingWriter 

type MockCompressingWriter struct {
	io.Writer
}

func (*MockCompressingWriter) Close 

func (writer *MockCompressingWriter) Close() error

func (*MockCompressingWriter) ReadFrom 

func (writer *MockCompressingWriter) ReadFrom(reader io.Reader) (n int64, err error)

type MockCompressor 

type MockCompressor struct{}

func (*MockCompressor) FileExtension 

func (compressor *MockCompressor) FileExtension() string

func (*MockCompressor) NewWriter 

func (compressor *MockCompressor) NewWriter(writer io.Writer) walg.ReaderFromWriteCloser

type MockDataFolder 

type MockDataFolder map[string]*bytes.Buffer

func NewMockDataFolder 

func NewMockDataFolder() *MockDataFolder

func (*MockDataFolder) CleanFolder 

func (folder *MockDataFolder) CleanFolder() error

func (*MockDataFolder) IsEmpty 

func (folder *MockDataFolder) IsEmpty() bool

func (*MockDataFolder) OpenReadonlyFile 

func (folder *MockDataFolder) OpenReadonlyFile(filename string) (io.ReadCloser, error)

func (*MockDataFolder) OpenWriteOnlyFile 

func (folder *MockDataFolder) OpenWriteOnlyFile(filename string) (io.WriteCloser, error)

type MockReadWriteCloser 

type MockReadWriteCloser struct{}

func (*MockReadWriteCloser) Close 

func (readWriterCloser *MockReadWriteCloser) Close() error

func (*MockReadWriteCloser) Read 

func (readWriterCloser *MockReadWriteCloser) Read(p []byte) (n int, err error)

func (*MockReadWriteCloser) Write 

func (readWriterCloser *MockReadWriteCloser) Write(p []byte) (n int, err error)

type MockStorage 

type MockStorage struct {
	// contains filtered or unexported fields
}

func NewMockStorage 

func NewMockStorage() *MockStorage

func (*MockStorage) IsEmpty 

func (storage *MockStorage) IsEmpty() bool

func (*MockStorage) Load 

func (storage *MockStorage) Load(key string) (value bytes.Buffer, exists bool)

func (*MockStorage) Store 

func (storage *MockStorage) Store(key string, value bytes.Buffer)

type NOPTarInterpreter 

type NOPTarInterpreter struct{}

NOPTarInterpreter mocks a tar extractor.

func (*NOPTarInterpreter) Interpret 

func (tarInterpreter *NOPTarInterpreter) Interpret(tr io.Reader, header *tar.Header) error

Interpret does not do anything except print the 'tar member' name.

type NopCloser 

type NopCloser struct{}

func (*NopCloser) Close 

func (closer *NopCloser) Close() error

type NopSeeker 

type NopSeeker struct{}

func (*NopSeeker) Seek 

func (seeker *NopSeeker) Seek(offset int64, whence int) (int64, error)

type ReadWriteNopCloser 

type ReadWriteNopCloser struct {
	io.ReadWriter
}

func (*ReadWriteNopCloser) Close 

func (readWriteNopCloser *ReadWriteNopCloser) Close() error

type StrideByteReader 

type StrideByteReader struct {
	// contains filtered or unexported fields
}

StrideByteReader allows for customizable "strides" of random bytes. Creates an infinite stream.

func NewStrideByteReader 

func NewStrideByteReader(s int) *StrideByteReader

NewStrideByteReader creates a new random byte stride generator with a seed of 0.

func (*StrideByteReader) Read 

func (sb *StrideByteReader) Read(p []byte) (int, error)

Read creates randomly generated bytes of 'stride' length.

Source Files

View all Source files

Directories

Path Synopsis
cmd
compress command
delta command
extract command
generate command

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.