README
¶
Kraken SQL Tag Datastore
This backend is designed to speed up Docker tag operations for the build-index. It stores tags in a SQL database rather than as files on the same backend storage as the blobs. While this adds an additional component to the Kraken, deployment it significantly enhances the performance of the build-index service, especially with registries that have a large number of Docker repositories or repositories with a large number of tags. This can eliminate the need for things like an nginx cache or result pagination that attempt to compensate for slow tag performance.
For example, Uber ATG's deployment of Kraken in had (as of August 2019) about 30,000 Docker repositories, with some repositories having over 16,000 tags.
- Listing tags the 16,353 tags in one particular repository took between 35 and 50 seconds, and often resulted in a
500 Server Errrorreturned to the user. - Listing 29,660 repositories in the Docker catalog took over 3 minutes and also often results in an error.
Using the SQL Tag Datastore:
- Listing 100,000 tags in a single repository takes 0.5 seconds.
- Listing 100,000 repositories in the Docker catalog takes 0.9 seconds.
Supported Databases
The backend uses GORM as the database interface, so it can support any database supported by GORM. By default it is only configured to use MySQL and SQLite (the latter only for testing), so see the GORM documentation on how to configure it to use other databases such as Postgres.
Benchmarks
Under the benchmark package there are a set of Golang benchmark tests that can be used to evaluate the performance of
the SQL Tag Datastore. These are kept separate from the standard set of unit tests because they create a much larger
test dataset, which can slow down the Bazel tests. Here is an example of the benchmarks output:
pkg: github.com/uber/kraken/lib/backend/sqlbackend/benchmark
BenchmarkStat-8 20000 64071 ns/op
BenchmarkDownload-8 30000 61641 ns/op
2019/09/03 13:57:25 Docker catalog took 617.229428ms for 100001 repos
2019/09/03 13:57:26 Docker catalog took 531.105741ms for 100001 repos
2019/09/03 13:57:26 Docker catalog took 534.448273ms for 100001 repos
BenchmarkListCatalog-8 2 532797142 ns/op
2019/09/03 13:57:27 Docker tag list took 460.731339ms for 100000 tags
2019/09/03 13:57:27 Docker tag list took 450.263591ms for 100000 tags
2019/09/03 13:57:28 Docker tag list took 447.621031ms for 100000 tags
2019/09/03 13:57:28 Docker tag list took 460.523944ms for 100000 tags
2019/09/03 13:57:29 Docker tag list took 447.264241ms for 100000 tags
2019/09/03 13:57:29 Docker tag list took 454.095311ms for 100000 tags
BenchmarkListTags-8 3 453979596 ns/op
PASS
Documentation
¶
Overview ¶
Copyright (c) 2016-2020 Uber Technologies, Inc.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Copyright (c) 2016-2020 Uber Technologies, Inc.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Copyright (c) 2016-2020 Uber Technologies, Inc.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type AuthConfig ¶
type AuthConfig struct {
SQL SQL `yaml:"sql"`
}
AuthConfig matches Langley format.
type Client ¶
type Client struct {
// contains filtered or unexported fields
}
Client implements a backend.Client for SQL.
func (*Client) Download ¶
Download gets the tag from the database and then writes the image ID to the output writer.
func (*Client) List ¶
func (c *Client) List(prefix string, _ ...backend.ListOption) (*backend.ListResult, error)
List lists names with start with prefix.
type Config ¶
type Config struct {
DebugLogging bool `yaml:"debug_logging"`
Dialect string `yaml:"dialect"`
ConnectionString string `yaml:"connection_string"`
Username string `yaml:"username"`
}
Config is used to initialize the SQL Backend Client
type Tag ¶
type Tag struct {
ID uint64 `gorm:"primary_key;auto_increment:true"`
Repository string `gorm:"not null;type:varchar(255);index:repository;unique_index:repository_tag"`
Tag string `gorm:"not null;type:varchar(128);unique_index:repository_tag"`
ImageID string `gorm:"not null;type:varchar(2056)"`
CreatedAt time.Time
UpdatedAt time.Time
}
Tag represents a Docker tag
type UserAuthConfig ¶
type UserAuthConfig map[string]AuthConfig
UserAuthConfig defines authentication configuration overlayed by Langley/Vault. Each key is the iam username of the credentials.
Source Files