README
¶
dlib - Small independent-but-complementary Context-oriented Go libraries
dlib is a set of small independent-but-complementary Context-oriented Go libraries.
For the most part, each library within dlib is independent, and has a a different problem statement. If they are each independent, why are they lumped together in to dlib? Because they share common design principles; dlib is lumped together so that the user can trust that these principles have been reasonably consistently applied, and can spend less time evaluating the library before deciding whether to use it.
Design principles
The thing binding together the different packages within dlib are that they share common design principles. The tagline is that they are "small independent-but-complementary Context-oriented" packages, and each one of those words (except "but") has a lot of meaning packed in to it, and states one of the design principles:
-
Packages should be small.
The way we use "small" is a little overloaded, and means a few different things:
-
Packages should be small in their functionality and API; the functionality should easily fit in the user's head.
A user should be able to quickly look at the package and understand the nugget of functionality that it provides. A sprawling API is more things for the user to be distracted by and to try to fit in their head, taking space away from the actual problem they're trying to solve.
Don't make the user buy the whole enchilada if they just want the beans.
-
Packages should be small in their API.
The API of
dexecis minimal; it's just that of stdlibos/execthat people already know (with one function removed, at that). The size of the interface is mostly just typinggithub.com/datawire/dlib/dexecinstead ofos/exec.Explicitly not part of the way we use "small" is "small in implementation". Despite the small API of
dexec, it has by far the most lines of code of any package in dlib. This is because of the complexity in keeping the exact API ofos/exec; this complexity is hidden by the simplicity of the user not having to learn something new.Bigger APIs are more intimidating, harder to learn, harder to remember, and harder to discover.
-
Packages should be small in their opinions; the one opinion that they should cling to is "use Contexts!".
The core of
dlogdoesn't actually do much of anything; it delegates to a pluggable logging backend.dcontextdoesn't change the way you pass Contexts around, it doesn't force new opinions on code that interoperates with it; a specialdcontexthard/soft Context can be passed to a non-dcontext-aware function, and the right thing will happen; a plain Context can be passed to adcontext-aware function, and the right thing will happen.
The way we use "small" is related to the way that Rob Pike uses "simple" (slides).
-
-
Packages should be independent.
Similar to packages being small should be independent so as to not artificially increase their size. If the user has to use both packages to use one, then are they really separate? They're effectively one large package, but with worse discoverability. Packages being coupled means that you must now understand the functionality and API of both packages, and must accept the opinions of both packages.
One package is free to use another internally, just as long as that's an implementation detail and not something that the user needs to care about.
-
Packages should be complementary.
Despite being independent, the packages should complement each other. You don't have to use
dcontextif you're going to usedexec, but if you do, then you'll get graceful shutdown "for free". You don't have to usedlogif you're going to usedexec, but if you do, then you'll be able to configuredexec's output. -
Packages should be Context-oriented.
The one "opinion" that all of dlib clings to is to use Contexts. This allows us to reduce the other opinions that a package brings with it.
Different logging solutions in Go are usually incompatible; do you pass around a
*log.Logger, or alogrus.FieldLogger, or what; this opinion about logging affects all essentially all of your function signatures. The opinion of "use Contexts" means: You're passing around acontext.Contextanyway, so let's attach the logger implementation to that, so that opinions about which logger has the prettiest don't need to affect the code that is written, except for one-time setup in the final application'smain().When there is something that a package don't want to or can't take as an argument, rather than making it a global variable or other global state, it should be packed in to a Context. Then, rather than reading it out of a variable, the function making use of it can read it out of the Context. And that won't be a problem, because everything that might want some kind of ambient state will take a Context as an argument, right? Perhaps it helps to think of a Context as an explicit passing of the ambient environment that a function is executing it.
Contexts were added in Go 1.7, and turned out to be a paradigm shift. And it's often a long journey to actually agreeing that Contexts are a good idea (personally, it took @LukeShu years to come around). The Official position is that everything new should use Contexts, but because of years of historical pre-Context code, and because of people who still haven't come around to Contexts, a lot of things don't use Contexts. So when we say "Context-oriented", what we're saying is which side of history dlib is on.
And through the organic history of what is now dlib, we've seen how that one "Contexts are good" opinion has allowed our libraries to sidestep having other more ornery opinions and sidestep other tricky design decisions.
-
Defaults should be useful.
A zero
dgroup.GroupConfig{}is useful without filling in any settings; things that are on by default have aDisableXXXbool, and things that are off by default have anEnableXXXbool. The most-common configuration will be empty, and the second-most-common configuration will be the just the 1 itemEnableSignalHandling: true(which we can't make the default because it would be bad to set it up multiple signal handler in the same program).The above paragraph is a general statement of a cultural value in the Go community; "have meaningful zero values". However, being Context-oriented promotes that from a guideline to an imperative: if you are reading your data out of a Context (as
dlogdoes), then you can't rely on having packed the data in to the Context ahead-of-time; you must gracefully handle the case where you don't get a value out.The core of
dlogdoesn't actually do much of anything; it delegates to a pluggable logging backend, but it uses alogrus-based backend by default; few users will be upset by this default logging with colorized output and timestamps. Having useful defaults is a backing-assumption for being Context-oriented. Ifdlogdidn't have a useful default logger, then using it wouldn't be a no-brainer, usingdlogwould force the user of that package to care aboutdlogand whether or not they'd taken care to configure the logger ahead-of-time.
-
In all, dlib is lumped together so that the user can trust that these principles have been reasonably consistently applied. The user can pull in one package from dlib and trust that they won't have to worry about having to adjust their program to that package's opinions (except of course, for the opinion that you should use Contexts!).
Example
Everything is complementary, and so if you drink the Kool-Aid and want
to see how to use everything together, check out the example in
example_test.go
Documentation
¶
Overview ¶
Module dlib is a collection of small independent-but-complementary Context-oriented packages.
Example (Main) ¶
package main
import (
"context"
"net/http"
"os"
"time"
"github.com/sirupsen/logrus"
"github.com/datawire/dlib/dcontext"
"github.com/datawire/dlib/derror"
"github.com/datawire/dlib/dexec"
"github.com/datawire/dlib/dgroup"
"github.com/datawire/dlib/dhttp"
"github.com/datawire/dlib/dlog"
"github.com/datawire/dlib/dtime"
)
// This is an example main() program entry-point that shows how all the pieces of dlib can fit
// together and complement each other.
func main() {
// Start with the background Context as the root Context.
ctx := context.Background()
// The default backend for dlog is pretty good, but for the sake of example, let's customize
// it a bit.
ctx = dlog.WithLogger(ctx, func() dlog.Logger {
// Let's have the backend be logrus. The default backend is already logrus, but
// ours will be customized.
logrusLogger := logrus.New()
// The dlog default is InfoLevel; let's crank it up to DebugLevel.
logrusLogger.Level = logrus.DebugLevel
// Now turn that in to a dlog.Logger backend, so we can pass it to dlog.WithLogger.
return dlog.WrapLogrus(logrusLogger)
}())
// We're going to be doing several tasks in parallel, so we'll use "dgroup" to manage our
// group of goroutines.
grp := dgroup.NewGroup(ctx, dgroup.GroupConfig{
// Enable signal handling for graceful shutdown. The user can stop the program by
// sending it SIGINT with Ctrl-C, and that will start a graceful shutdown. If that
// graceful shutdown takes too long, and the user hits Ctrl-C again, then it will
// start a not-so-graceful shutdown.
//
// This shutdown will be signaled to the worker goroutines through the Context that
// gets passed to them. The mechanism by which the Context signals both graceful
// and not-so-graceful shutdown is what "dcontext" is for.
EnableSignalHandling: true,
})
// One of those tasks will be running an HTTP server.
grp.Go("http", func(ctx context.Context) error {
// We'll be using a *dhttp.ServerConfig instead of an *http.Server, but it works
// very similarly to *http.Server, everything else in the stdlib net/http package is
// still valid; we'll still be using plain-old http.ResponseWriter and *http.Request
// and http.HandlerFunc.
cfg := &dhttp.ServerConfig{
Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
dlog.Debugln(r.Context(), "handling HTTP request")
_, _ = w.Write([]byte("Hello, world!\n"))
}),
}
// ListenAndServe will gracefully shut down according to ctx; we don't need to worry
// about separately calling .Shutdown() or .Close() like we would for *http.Server
// (those methods don't even exist on dhttp.ServerConfig). During a graceful
// shutdown, it will stop listening and close idle connections, but will wait on any
// active connections; during a not-so-graceful shutdown it will forcefully close
// any active connections.
//
// If the server itself needs to log anything, it will use dlog according to ctx.
// The Request.Context() passed to the Handler function will inherit from ctx, and
// so the Handler will also log according to ctx.
//
// And, on the end-user-facing side of things, this supports HTTP/2, where
// *http.Server.ListenAndServe wouldn't.
return cfg.ListenAndServe(ctx, ":8080")
})
// Another task will be running external host operating system commands.
grp.Go("exec", func(ctx context.Context) error {
// dexec is *almost* a drop-in replacement for os/exec; the only breaking change is
// that .Command() doesn't exist anymore, you *must* use the .CommandContext()
// variant.
//
// There are two nice things using dexec instead of os/exec this gets us.
//
// Firstly: dexec logs everything read from or written to the command's stdin,
// stdout, and stderr; logging with dlog according to ctx. Logging of one of any of
// these can be opted-out of; see the dexec documentation.
//
// Secondly: When the Context signals a shutdown, os/exec responds by stopping the
// command by sending it the SIGKILL signal. Now, "sending the SIGKILL signal" is a
// bit of a misleading phrase, because SIGKILL is never actually sent to the
// process, it instead tells the operating system to just stop giving the process
// any CPU time; the process is just abruptly dead. dexec gives the process a
// chance to gracefully shutdown by sending it SIGINT for a graceful shutdown, and
// only sending it SIGKILL for a not-so-graceful shutdown.
cmd := dexec.CommandContext(ctx, "some", "long-running", "command", "--keep-going")
return cmd.Run()
})
// Another task will be running our own code.
grp.Go("code", func(ctx context.Context) error {
dataSourceCh := newDataSource(ctx)
theloop:
for {
select {
case <-ctx.Done():
// The channel <-ctx.Done() gets closed when either a graceful or
// not-so-graceful shutdown is triggerd.
//
// So, when the Context signals us to shut down, we'll break out of
// this `for` loop.
break theloop
// ... but until then, read from dataSourceCh, and process the data
// from it:
case dat := <-dataSourceCh:
// The doWorkOnData example function might be a little buggy and
// might panic(). We don't want that to cause us to stop processing
// more data early, so we'll use derror to catch any panics and turn
// them in to useful errors.
func() {
defer func() {
if err := derror.PanicToError(recover()); err != nil {
// Thanks to PanicToError, err has the panic
// stacktrace attached to it, which we can
// show using the "+" modifier to "%v".
dlog.Errorf(ctx, "doWorkOnData crashed: %+v", err)
}
}()
doWorkOnData(ctx, dat)
}()
// We want ensure we wait at least 10 seconds between each time we
// call DoWorkOnData, but we also don't want to stall a graceful
// shutdown by 10 seconds, so we'll use dtime.SleepWithContext
// instead of stdlib time.Sleep. (We also don't use stdlib
// time.After, because that would leak channels; we don't want
// memory leaks!)
dtime.SleepWithContext(ctx, 10*time.Second)
}
}
// OK, if we're here it's because <-ctx.Done() and we broke out of the `for` loop.
//
// We have some cleanup we'd like to do for a graceful shutdown, but we should bail
// early on that work if a not-so-graceful shutdown is triggered. If <-ctx.Done()
// is already closed because of a graceful shutdown, how do we detect when a
// not-so-graceful shutdown is triggered?
//
// Well, ctx is a "soft" Context; signaling a "soft" (graceful) shutdown. We'll use
// dcontext.HardContext to get the "hard" Context from it, which signals just a
// "hard" (not-so-graceful) shutdown.
ctx = dcontext.HardContext(ctx)
return gracefulCleanup(ctx)
})
if err := grp.Wait(); err != nil {
dlog.Errorf(ctx, "finished with error: %v", err)
os.Exit(1)
}
}
func newDataSource(_ context.Context) <-chan struct{} {
// Not actually implemented for this example.
return nil
}
func doWorkOnData(_ context.Context, _ struct{}) {
// Not actually implemented for this example.
}
func gracefulCleanup(_ context.Context) error {
// Not actually implemented for this example.
return nil
}
func main() {
// An "Example_XXX" function is needed to get this example to show up in the godoc.
main()
}
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Package dcontext provides tools for dealing with separate hard/soft cancellation of Contexts.
|
Package dcontext provides tools for dealing with separate hard/soft cancellation of Contexts. |
|
Package derrgroup is a low-level group abstraction; providing synchronization, error propagation, and cancellation callback for groups of goroutines working on subtasks of a common task.
|
Package derrgroup is a low-level group abstraction; providing synchronization, error propagation, and cancellation callback for groups of goroutines working on subtasks of a common task. |
|
Package dexec is a logging variant of os/exec.
|
Package dexec is a logging variant of os/exec. |
|
internal/cfg
Package cfg holds configuration shared by the Go command and internal/testenv.
|
Package cfg holds configuration shared by the Go command and internal/testenv. |
|
internal/testenv
Package testenv provides information about what functionality is available in different testing environments run by the Go team.
|
Package testenv provides information about what functionality is available in different testing environments run by the Go team. |
|
Package dgroup provides tools for managing groups of goroutines.
|
Package dgroup provides tools for managing groups of goroutines. |
|
Package dhttp is a simple production-ready HTTP server library for the 2020s.
|
Package dhttp is a simple production-ready HTTP server library for the 2020s. |
|
Package dlog implements a generic logger facade.
|
Package dlog implements a generic logger facade. |
|
Package dtime provides tools that FFS.
|
Package dtime provides tools that FFS. |
|
internal
|
|