openserverless

package module
v0.1.0-2408041450.dev Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 4, 2024 License: Apache-2.0 Imports: 35 Imported by: 0

README

WARNING: this is still work in progress

The code may not build, there can be errors, and it can even destroy your hard disk or send you in another dimension.

Documentation is also work in progress....

ops, the OpenServerless all-mighty CLI tool.

Ops is a task executor on steroids.

  • it embeds Task)
  • it embeds wsk
  • it embeds many other utility commands (check with ops -help)
  • automatically download other commad line tools he neeeds as prerequisites
  • automatically download a predefined set of tasks downloaded from github
  • taskfiles are organized in commands and subcommands, hiearachally
  • taskfiles have options powered by docopt
  • it supports plugins

The predefined set of tasks are all you need to install and manage an OpenServerless cluster.

You can also use for your own purposes, if you want.

Where to look for external documetation

Because ops is built on task, docopt and wsk you have to consult the following website for:

  • informations on the format of opsfile.yml: taskfiles
  • informations on the format of docopts.txt: docopts
  • informations on the OpenWhisk cli (the underlying engine for OpenServerless): wsk

Where ops looks for tasks

Ops is able either to run local tasks or download them from github.

Tasks also can contains prerequisite binaries, and task will download them automatically and put then in the PATH.

Se below for details

Local Taks

When you run ops [<args>...] it will first look for its ops root.

The olaris root is a folder with two files in it: opsfile.yml (a yaml taskfile) and opsroot.json (a json file with release information).

The first step is to locate the root folder. The algorithm to find the tools is the following.

If the environment variable OPS_ROOT is defined, it will look there first, and will check if there are the two files.

Then it will look in the current folder if there is a opsfile.yml. If there is, it will also look for opsroot.json. If it is not there, it will go up one level looking for a directory with opsfile.yml and opstools.json, and selects it as the ops root.

If there is not a opsfile.yml it will look for a folder called olaris with both a opsfile.yml and opstools.json in it and will select it as the ops root.

Then it will look in ~/.ops if there is an olaris folder with opsfile.yml and opsroot.json.

Download Tasks

If the local task resolution fails, it will download its tasks from github.

It is the same process that you can trigger manually with the command ops -update.

Where ops download tasks from GitHub

The repo to use is defined by the environment variable OPS_REPO, and defaults to https://github.com/apache/openserverless-task

The branch to use is defined at build time. It is noramlly named as the base version of the CLI. It can be overriden with the enviroment variable OPS_BRANCH.

When you run ops -update, if there is not a ~/.ops/<branch>/olaris it will clone the current branch, otherwise it will update it.

How ops execute tasks

Ops will look to the command line parameters ops <arg1> <arg2> <arg3> and will consider them as directory names. The list can be empty.

If there is a directory name <arg1> it will change to that directory. If there is then a subdirectory <arg2> it will change to that and so on until it finds a argument that is not a directory name.

If the last argument is a directory name, will look for a docopts.txt. If it is there, it will show this. If it's not there, it will execute a ops -task -t opsfile.yml -l showing tasks with description.

If it finds an argument not corresponding to a directory, it will consider it a task to execute,

If there is not a docopts.txt, it will execute as a task, passing the other arguments (equivalent to ops -task -t opsfile.yml <arg> -- <the-other-args>).

If there is a docopts.txt, it will interpret it as a docopt to parse the remaining arguments as parameters. The result of parsing is a sequence of <key>=<value> that will be fed to task. So it is equivalent to invoking task -t opsfile.yml <arg> <key>=<value> <key>=<value>...

Example

A command like ops setup kubernetes install --context=k3s will look in the folder setup/kubernetes in the ops root, if it is there, then select install as task to execute and parse the --context=k3s. It is equivalent to invoke cd setup/kubernetes ; task install -- context=k3s.

If there is a docopts.txt with a command <name> --flag --fl=<val> the passed parameters will be: _name_=<name> __flag=true __fl=true _val_=<val>.

Note that also this will also use the downloaded tools and the embedded commands of ops.

Embedded tools

Currently task embeds the following tools, and you can invoke them directly prefixing them with -: (ops -task, ops -basename etc). Use ops -help to list them.

This is the list of the tools (it could be outdated, check with ops -help):

Available tools:
-awk
-base64
-config
-datefmt
-die
-echoif
-echoifempty
-echoifexists
-empty
-envsubst
-extract
-filetype
-gron
-help
-info
-jj
-jq
-login
-needupdate
-plugin
-random
-remove
-rename
-replace
-retry
-serve
-sh
-task
-update
-urlenc
-validate
-version
-wsk

Environment variables for tasks

As a convenience, the system sets the following variables but you cannot override them:

  • OPS is the actual command you are using, so you can refer to ops itself in opsfiles as $OPS.
  • OPS_PWD is the folder where ops is executed (the current working directory). It is used to preserve the original working directory because ops changes to the acutal folder where the tasks are for execution.

The following environment variables are always set and you can ovverride them.

  • OPS_CMD is the actual command executed - defaults to the absolute path of the target of the symbolic link but it can be overriden. OPS will take this value.

  • OPS_VERSION can be defined to set ops's version value. It is useful to override version validations when updating tasks (and you know what you are doing). Current value is defined at build time and stored in sources in version.txt

  • OPS_HOME is the home dir, defaults to ~/.ops

  • OPS_REPO is the github repo where ops downloads its tasks. If not defined, it defaults to https://github.com/apache/openserverless-task.

  • OPS_BRANCH is the branch where ops looks for its tasks. The branch to use is defined at build time and it is ususally the base version (without the patch level). Check branch.txt for the current value

  • OPS_ROOT is the folder where ops looks for its tasks. If not defined, if will follow the algirithm described before to finding it locally. Otherwise download it from github, git clones or git updates the $OPS_REPO in the $OPS_BRANCH and store it is $OPS_HOME/$OPS_BRANCH/olaris

  • OPS_BIN is the folder where ops looks for binaries (external command line tools). If not defined, it defaults to ~/.ops/{{.OS}}-{{.ARCH}}/bin. All the prerequisites are downloaded in this directory. It is automatically added to the PATH at the beginning when executing opsfiles.

  • OPS_TMP is a temporary folder where you can store temp files - defaults to ~/.ops/tmp

  • OPS_APIHOST is the host for ops -login. It is used in place of the first argument of ops -login. If empty, the command will expect the first argument to be the apihost.

  • OPS_USER is set the username for ops -login. The default is nuvolaris. It can be overriden by passing the username as an argument to ops -login or by setting the environment variable.

  • OPS_PASSWORD: set the password for ops -login. If not set, ops -login will prompt for the password. It is useful for tests and non-interactive environments.

  • OPS_ROOT_PLUGIN is the folder where ops looks for plugins. If not defined, it defaults to the same directory where ops is located.

  • OPS_PORT is the port where ops will run embedded web server for the configurator. If not defined, it defaults to 9678.

  • OPS_OLARIS holds the head commit hash of the used olaris repo. If it is a local version its value is <local>. You can see the hash with ops -info.

Special purpose environment variables

The following variables have a special purpose and used for test and debug

  • DEBUG if set enable debugging messages
  • TRACE when set gives mode detailes tracing informations, also enable DEBUG=1
  • EXTRA appends extra arguments to the task invocation - useful when you need to set extra variables with a docopts active.
  • __OS overrides the value of the detected operating system - useful to test prereq scripts
  • __ARCH overrides the value of the detected architecture - useful to test prereq scripts
  • OPS_RUNTIMES_JSON is used for the values of the runtimes json if the system is unable to read in other ways the current runtimes json. It is normally compiled in when you buid from the current version of the runtimes.json. It can be overriden
  • OPS_COREUTILS is a string, a space separated list, which lists all the commands the coreutils binary provided. It should be kept updated with the values of the current version used. It can be overriden
  • OPS_TOOLS is a string, a space separated list, which lists all the commands provided as internal tool by the ops binary. It shold be kept updated with the current list of tools provided. It can be overriden defining externally
  • OPS_NO_DOCOPTS can be defined to disable docopts parsing. Useful to test hidden tasks. When this is enabled it also shows all the tasks instead of just those with a description.
  • OPS_NO_PREREQ disable downloading of prerequisites - you have to ensure at least coreutils is in the path to make things work

Documentation

Overview

Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Example (CheckUpdated_outdated) 
// clone olaris folder into a temp folder
tmpDir, err := os.MkdirTemp("", "ops-test")
if err != nil {
	pr("failed to create temp dir", err)
}
defer RemoveAll(tmpDir)

tmpDirBranch := joinpath(tmpDir, getOpsBranch())
olarisTmpPath := joinpath(tmpDirBranch, "olaris")

repo, _ := git.PlainClone(olarisTmpPath, false, &git.CloneOptions{
	URL: getOpsRepo(),
},
)

// run checkUpdated and check if it creates the latest_check file
createLatestCheckFile(tmpDirBranch)

if exists(tmpDirBranch, ".latestcheck") {
	pr("latest_check file created")
}

// change latest_check file mtime to 2 seconds ago
changeLatestCheckTime(tmpDirBranch, -2*time.Second)

// git reset olaris to a previous commit
resetOneCommit(repo)

// re-run checkUpdated and check output
checkUpdated(tmpDir, 1*time.Second)

Output:

latest_check file created
Checking for updates...
New tasks available! Use 'ops -update' to update.
Example (CheckUpdated_uptodate) 
// clone olaris folder into a temp folder
tmpDir, err := os.MkdirTemp("", "ops-test")
if err != nil {
	pr("failed to create temp dir", err)
}
defer RemoveAll(tmpDir)
tmpDirBranch := joinpath(tmpDir, getOpsBranch())
olarisTmpPath := joinpath(tmpDirBranch, "olaris")

_, _ = git.PlainClone(olarisTmpPath, false, &git.CloneOptions{
	URL: getOpsRepo(),
},
)

// run checkUpdated and check if it creates the latest_check file
createLatestCheckFile(tmpDirBranch)

if exists(tmpDirBranch, ".latestcheck") {
	pr("latest_check file created")
}

// change latest_check file mtime to 2 seconds ago
changeLatestCheckTime(tmpDirBranch, -2*time.Second)

// re-run checkUpdated and check output "Tasks up to date!"
checkUpdated(tmpDir, 1*time.Second)

Output:

latest_check file created
Checking for updates...
Tasks up to date!
Example (Download)

TODO: undestand why it fails when executed with others it works executed alone 

_ = os.Chdir(workDir)
OpsBranch = "0.1.0"
opsdir, _ := homedir.Expand("~/.ops")
_ = RemoveAll(opsdir)
_, _ = downloadTasksFromGitHub(true, true)
dir, err := locateOpsRoot(".")
pr(1, err, nhpath(dir))
_, _ = downloadTasksFromGitHub(true, true)
dir, err = locateOpsRoot(".")
pr(2, err, nhpath(dir))

Output:

Cloning tasks...
Tasks downloaded successfully
1 <nil> /home/.ops/0.1.0/olaris
Updating tasks...
Tasks are already up to date!
2 <nil> /home/.ops/0.1.0/olaris
Example (DownloadPrereq) 
bindir, err := EnsureBindir()
if err != nil {
	RemoveAll(bindir)
}

prqdir := joinpath(joinpath(workDir, "tests"), "prereq")
prq, _ := loadPrereq(prqdir)
fmt.Println("1", downloadPrereq("bun", prq.Tasks["bun"]))
fmt.Println("2", downloadPrereq("bun", prq.Tasks["bun"]))

prq, _ = loadPrereq(joinpath(prqdir, "sub"))
//fmt.Println(prq)
//fmt.Println(PrereqSeenMap)
fmt.Println("3", downloadPrereq("bun", prq.Tasks["bun"]))

Output:

downloading bun v1.11.20
1 <nil>
2 <nil>
3 WARNING: bun prerequisite found twice with different versions!
Previous version: v1.11.20, ignoring v1.11.21
Example (EnsureBindir) 
// cleanup
bindir, err := EnsureBindir()
if err == nil {
	RemoveAll(bindir)
} else {
	fmt.Println(err)
}
// ensure no dir
_, err = os.Stat(bindir)
if err != nil {
	fmt.Println(1, after(":", err.Error()))
}
bindir1, err := EnsureBindir()
if err != nil {
	fmt.Println(err)
}
_, err = os.Stat(bindir1)
fmt.Println(2, err)
fmt.Println(3, bindir == bindir1)

Output:

1  no such file or directory
2 <nil>
3 true
Example (EnsurePrereq) 
bindir, err := EnsureBindir()
if err == nil {
	RemoveAll(bindir)
}
dir := joinpath(joinpath(workDir, "tests"), "prereq")
fmt.Println(ensurePrereq(dir))
fmt.Println(ensurePrereq(joinpath(dir, "sub")))

Output:

downloading bun v1.11.20
downloading coreutils 0.0.27
<nil>
error in prereq bun: WARNING: bun prerequisite found twice with different versions!
Previous version: v1.11.20, ignoring v1.11.21
<nil>
Example (ExecPrereqTask) 
fmt.Println(execPrereqTask("bin", "bun"))

Output:

invoking prereq for bun
<nil>
Example (LoadArgs) 
_ = os.Chdir(workDir)
fmt.Println(1, loadSavedArgs())
_ = os.Chdir(joinpath("tests", "testdata"))
fmt.Println(2, loadSavedArgs())

Output:

1 []
2 [V1=hello V2=hello V2=world]
Example (LoadPrereq) 
//downloadPrereq("")
dir := joinpath(workDir, "tests")
_, err := loadPrereq(dir)
fmt.Println(npath(err.Error()))
prq, err := loadPrereq(joinpath(dir, "prereq"))
//fmt.Println(prq)
fmt.Println(err, *prq.Tasks["bun"].Description, prq.Tasks["bun"].Vars["VERSION"])

Output:

not found /work/tests
<nil> bun v1.11.20
Example (Locate) 
_ = os.Chdir(workDir)
dir, err := locateOpsRoot("tests")
pr(1, err, npath(dir))
dir, err = locateOpsRoot(joinpath("tests", "olaris"))
pr(2, err, npath(dir))
dir, err = locateOpsRoot(joinpath("tests", joinpath("olaris", "sub")))
pr(3, err, npath(dir))

Output:

1 <nil> /work/tests/olaris
2 <nil> /work/tests/olaris
3 <nil> /work/tests/olaris
Example (Locate_root) 
_ = os.Chdir(workDir)
OpsBranch = "0.1.0"
opsdir, _ := homedir.Expand("~/.ops")
_ = RemoveAll(opsdir)
_, err := locateOpsRoot(".")
pr(1, err)
dir, err := locateOpsRoot("tests")
pr(2, err, npath(dir))

Output:

1 cannot find opsfiles, download them with ops -update
2 <nil> /work/tests/olaris
Example (OpsArg) 
// test
_ = os.Chdir(workDir)
olaris, _ := filepath.Abs(joinpath("tests", "olaris"))
err := Ops(olaris, split("testcmd"))
pr(2, err)
err = Ops(olaris, split("testcmd arg"))
pr(3, err)
err = Ops(olaris, split("testcmd arg VAR=1"))
pr(4, err)
err = Ops(olaris, split("testcmd VAR=1 arg"))
pr(5, err)

Output:

(olaris) task [-t opsfile.yml testcmd --]
2 <nil>
(olaris) task [-t opsfile.yml testcmd -- arg]
3 <nil>
(olaris) task [-t opsfile.yml testcmd VAR=1 -- arg]
4 <nil>
(olaris) task [-t opsfile.yml testcmd VAR=1 -- arg]
5 <nil>
Example (PluginsPrint) 
p := plugins{
	local: make([]string, 0),
	ops:   make([]string, 0),
}
p.print()
// Output
// No plugins installed. Use 'ops -plugin' to add new ones.
Example (SetupTmp) 
_ = os.Chdir(workDir)
opsdir, _ := homedir.Expand("~/.ops")
RemoveAll(opsdir)
setupTmp()
fmt.Println(nhpath(os.Getenv("OPS_TMP")))
RemoveAll(opsdir)

Output:

/home/.ops/tmp
Example (TouchAndClean) 
bindir, err := EnsureBindir()
if err != nil {
	RemoveAll(bindir)
}

bindir, _ = EnsureBindir()
touch(bindir, "hello")
err = touchAndClean(bindir, "hello", "1.2.3")
fmt.Println(err, exists(bindir, "hello"), exists(bindir, "hello-1.2.3"), exists(bindir, "hello-1.2.4"))
err = touchAndClean(bindir, "hello", "1.2.4")
fmt.Println(err, exists(bindir, "hello"), exists(bindir, "hello-1.2.3"), exists(bindir, "hello-1.2.4"))

Output:

<nil> true true false
<nil> true false true

Index

Examples

Constants

View Source 
const CONFIGFILE = "config.json"
View Source 
const DOCOPTS = "docopts.txt"
View Source 
const DefaultOpsPort = 9768

default port for ops server

View Source 
const LATESTCHECK = ".latestcheck"
View Source 
const OPSFILE = "opsfile.yml"

default files

View Source 
const OPSREPO = "http://github.com/apache/openserverless-task"

repo where download tasks

View Source 
const OPSROOT = "opsroot.json"
View Source 
const PREREQ = "prereq.yml"

Variables

View Source 
var OpsBranch string
View Source 
var OpsVersion string
View Source 
var PrereqSeenMap = map[string]string{}
View Source 
var WSK_RUNTIMES_JSON string

Functions

func EnsureBindir 

func EnsureBindir() (string, error)

ensure there is a bindir for downloading prerequisites read it from OPS_BIN and create it otherwise setup one in ~/ops/<os>-<arch>/bin and sets OPS_BIN

func GetOpsRootPlugins 

func GetOpsRootPlugins() (map[string]string, error)

GetOpsRootPlugins returns the map with all the olaris-*/opsroot.json files in the local and ~/.ops folders, pointed by their plugin names. If the same plugin is found in both folders, the one in the local folder is used. Useful to build the config map including the plugin configs

func IsWskWrapperCommand 

func IsWskWrapperCommand(name string) ([]string, bool)

func Main 

func Main()

func Ops 

func Ops(base string, args []string) error

Ops parses args moving into the folder corresponding to args then parses them with docopts and invokes the task

Example 
// test
_ = os.Chdir(workDir)
os.Setenv("TEST_VAR", "evar")
olaris, _ := filepath.Abs(joinpath("tests", "olaris"))
err := Ops(olaris, split(""))
pr(1, err)
err = Ops(olaris, split("sub"))
pr(4, err)
err = Ops(olaris, split("sub opts"))
pr(5, err)
_ = Ops(olaris, split("sub opts ciao 1"))
// pr(6, err)

func Serve 

func Serve(olarisDir string, args []string) error

func Task 

func Task(args ...string) (int, error)

func Trace 

func Trace(args ...any)

Types

type OpsRootJSON 

type OpsRootJSON struct {
	Version string                 `json:"version"`
	Config  map[string]interface{} `json:"config"`
}

Represents opsroot.json. It is used to parse the file.

type Prereq 

type Prereq struct {
	Version int                   `yaml:"version"`
	Tasks   map[string]PrereqTask `yaml:"tasks"`
}

Define the Go structs

type PrereqTask 

type PrereqTask struct {
	Description *string           `yaml:"description,omitempty"` // Make description optional
	Vars        map[string]string `yaml:"vars,omitempty"`
}

type TaskNotFoundErr 

type TaskNotFoundErr struct {
	// contains filtered or unexported fields
}

func (*TaskNotFoundErr) Error 

func (e *TaskNotFoundErr) Error() string

Source Files

View all Source files

Directories

Path Synopsis
cmd
ops command
gosh is a proof of concept shell built on top of [interp].
 gosh is a proof of concept shell built on top of [interp].

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.