README
¶
Base64 using SIMD
Translation of Alfred Klomp's base64 library to Go using the currently experimental simd/archsimd package.
This package currently only employs SIMD instructions on AMD64 architecture. It requires Go 1.26 (or above) toolchain with GOEXPERIMENT=simd set in the environment.
Performance
The benchmarks below use standard base64 alphabet with padding and employ the AVX2 instruction set.
This compares the same encode and decode operations using encoding/base64 package from the standard library with this package.
goos: linux
goarch: amd64
pkg: codeberg.org/jongray/base64simd
cpu: Intel(R) Core(TM) i7-6770HQ CPU @ 2.60GHz
│ base64 │ base64simd │
│ sec/op │ sec/op vs base │
Encode/size=10-8 16.97n ± 22% 19.71n ± 8% +16.15% (p=0.035 n=10)
Encode/size=100-8 125.95n ± 31% 45.27n ± 10% -64.06% (p=0.000 n=10)
Encode/size=1000-8 1129.0n ± 8% 176.1n ± 8% -84.40% (p=0.000 n=10)
Encode/size=10000-8 11.946µ ± 7% 1.230µ ± 8% -89.71% (p=0.000 n=10)
Encode/size=100000-8 113.56µ ± 8% 12.36µ ± 6% -89.12% (p=0.000 n=10)
Decode/size=16-8 29.81n ± 10% 22.66n ± 5% -23.97% (p=0.000 n=10)
Decode/size=136-8 129.95n ± 6% 87.42n ± 6% -32.72% (p=0.000 n=10)
Decode/size=1336-8 1154.0n ± 5% 214.0n ± 7% -81.46% (p=0.000 n=10)
Decode/size=13336-8 11.388µ ± 9% 1.715µ ± 5% -84.94% (p=0.000 n=10)
Decode/size=133336-8 114.01µ ± 7% 16.17µ ± 12% -85.82% (p=0.000 n=10)
geomean 1.342µ 350.6n -73.88%
Beware!
Compare the above results to the below using a different CPU. This package decode ns/op is significantly higher that of the standard encoding/base64 package. This suggests there can be differences in a CPU implementation of the SIMD instruction set. Be sure to benchmark in your environment.
goos: linux
goarch: amd64
pkg: codeberg.org/jongray/base64simd
cpu: AMD Ryzen 5 5600X 6-Core Processor
│ base64 │ base64simd │
│ sec/op │ sec/op vs base │
Encode/size=10-12 9.307n ± 3% 11.465n ± 5% +23.19% (p=0.000 n=10)
Encode/size=100-12 59.34n ± 2% 24.98n ± 7% -57.91% (p=0.000 n=10)
Encode/size=1000-12 573.65n ± 1% 80.87n ± 1% -85.90% (p=0.000 n=10)
Encode/size=10000-12 5639.0n ± 1% 567.6n ± 4% -89.93% (p=0.000 n=10)
Encode/size=100000-12 56.680µ ± 1% 5.505µ ± 2% -90.29% (p=0.000 n=10)
Decode/size=16-12 17.17n ± 6% 13.30n ± 9% -22.51% (p=0.001 n=10)
Decode/size=136-12 76.59n ± 6% 236.00n ± 2% +208.13% (p=0.000 n=10)
Decode/size=1336-12 666.5n ± 1% 2508.5n ± 1% +276.37% (p=0.000 n=10)
Decode/size=13336-12 6.504µ ± 1% 25.221µ ± 2% +287.77% (p=0.000 n=10)
Decode/size=133336-12 66.29µ ± 4% 253.13µ ± 1% +281.84% (p=0.000 n=10)
geomean 721.6n 570.3n -20.97%
By forcing the benchmark to use the AVX instruction set, as opposed to AVX2, the results look as expected.
goos: linux
goarch: amd64
pkg: codeberg.org/jongray/base64simd
cpu: AMD Ryzen 5 5600X 6-Core Processor
│ base64 │ base64simd │
│ sec/op │ sec/op vs base │
Encode/size=10-12 9.165n ± 4% 11.155n ± 1% +21.72% (p=0.000 n=10)
Encode/size=100-12 59.00n ± 4% 19.77n ± 2% -66.50% (p=0.000 n=10)
Encode/size=1000-12 577.10n ± 6% 97.69n ± 0% -83.07% (p=0.000 n=10)
Encode/size=10000-12 5657.0n ± 1% 838.6n ± 5% -85.18% (p=0.000 n=10)
Encode/size=100000-12 57.203µ ± 1% 8.213µ ± 4% -85.64% (p=0.000 n=10)
Decode/size=16-12 17.12n ± 5% 12.32n ± 2% -28.09% (p=0.000 n=10)
Decode/size=136-12 76.34n ± 6% 34.73n ± 0% -54.51% (p=0.000 n=10)
Decode/size=1336-12 669.3n ± 1% 136.2n ± 6% -79.64% (p=0.000 n=10)
Decode/size=13336-12 6.537µ ± 2% 1.178µ ± 5% -81.99% (p=0.000 n=10)
Decode/size=133336-12 65.85µ ± 4% 10.98µ ± 2% -83.32% (p=0.000 n=10)
geomean 721.1n 201.8n -72.02%
Usage
The package offers, similar to the standard library base64 package, an Encoding interface with implementations to cover the different combinations of base64 encoding. It does not duplicate functionality offered by the standard library package where appropriate, for example encoding/base64.Encoding.EncodedLen and encoding/base64.Encoding.DecodedLen.
Encode
Encode encodes data from sourceBytes using the chosen encoding, writing encoding/base64.Encoding.EncodedLen(len(source)) bytes to destinationBytes.
destinationBytes MUST have at least size encoding/base64.Encoding.EncodedLen(len(source)) else Encode will panic.
sourceBytes := []byte("foobar")
destinationBytes := make([]byte, base64.StdEncoding.EncodedLen(len(sourceBytes)))
base64simd.StdEncoding.Encode(destinationBytes, sourceBytes)
fmt.Println(string(destinationBytes))
// Output: Zm9vYmFy
Decode
Decode decodes data from sourceBytes using the chosen encoding, writing upto encoding/base64.Encoding.DecodedLen(len(source)) bytes to destinationBytes and returns the number of bytes written.
destinationBytes MUST have at least size encoding/base64.Encoding.DecodedLen(len(source)) else Decode will panic.
A encoding/base64.CorruptInputError error will be returned if the sourceBytes contains characters not supported by the base64 alphabet of the chosen encoding.
Decode does NOT support ignoring newlines (it is left to the caller to remove newlines prior to calling Decode) and will return a encoding/base64.CorruptInputError error if sourceBytes contains any.
sourceBytes := []byte("Zm9vYmFy")
destinationBytes := make([]byte, base64.StdEncoding.DecodedLen(len(sourceBytes)))
n, _ := base64simd.StdEncoding.Decode(destinationBytes, sourceBytes)
fmt.Println(string(destinationBytes[:n]))
// Output: foobar
Documentation
¶
Overview ¶
Package base64simd implements base64 encoding (as specified in RFC 4648) using SIMD instructions (currently experimental, enabled with GOEXPERIMENT=simd environment variable).
Index ¶
Examples ¶
Constants ¶
This section is empty.
Variables ¶
var RawStdEncoding = StdEncoding.withoutPadding()
RawStdEncoding performs standard base64 encoding using the standard 64 character alphabet, but without padding, as defined in RFC 4648.
var RawUrlEncoding = UrlEncoding.withoutPadding()
RawUrlEncoding performs alternate base64url encoding using the URL (and filename) safe alternate 64 character alphabet, but without padding, as defined in RFC 4648.
var StdEncoding = newEncoding("ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/")
StdEncoding performs standard base64 encoding using the standard 64 character alphabet as defined in RFC 4648.
var UrlEncoding = newEncoding("ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_").urlSafe()
UrlEncoding performs alternate base64url encoding using the URL (and filename) safe alternate 64 character alphabet as defined in RFC 4648.
Functions ¶
This section is empty.
Types ¶
type Encoding ¶
type Encoding struct {
// contains filtered or unexported fields
}
func (*Encoding) Decode ¶
Decode decodes data from source using the encoding e, writing upto encoding/base64.Encoding.DecodedLen(len(source)) bytes to destination and returns the actual number of bytes written.
destination MUST have at least size encoding/base64.Encoding.DecodedLen(len(source)) else Decode will panic.
A encoding/base64.CorruptInputError error will be returned if the source contains characters not supported by the base64 alphabet of the encoding.
Decode does NOT support ignoring newlines (it is left to the caller to remove newlines prior to calling Decode) and will return a encoding/base64.CorruptInputError error if source contains any newlines.
Example ¶
package main
import (
"encoding/base64"
"fmt"
"codeberg.org/jongray/base64simd"
)
func main() {
sourceBytes := []byte("Zm9vYmFy")
destinationBytes := make([]byte, base64.StdEncoding.DecodedLen(len(sourceBytes)))
n, _ := base64simd.StdEncoding.Decode(destinationBytes, sourceBytes)
fmt.Println(string(destinationBytes[:n]))
}
Output: foobar
func (*Encoding) Encode ¶
Encode encodes data from source using the encoding e, writing encoding/base64.Encoding.EncodedLen(len(source)) bytes to destination.
destination MUST have at least size encoding/base64.Encoding.EncodedLen(len(source)) else Encode will panic.
Example ¶
package main
import (
"encoding/base64"
"fmt"
"codeberg.org/jongray/base64simd"
)
func main() {
sourceBytes := []byte("foobar")
destinationBytes := make([]byte, base64.StdEncoding.EncodedLen(len(sourceBytes)))
base64simd.StdEncoding.Encode(destinationBytes, sourceBytes)
fmt.Println(string(destinationBytes))
}
Output: Zm9vYmFy
Source Files