Skip to content

Latest commit

 

History

History
442 lines (373 loc) · 20.1 KB

value.md

File metadata and controls

442 lines (373 loc) · 20.1 KB

Expand one value: value.Expand

Package github.com/ubgo/textdata/value expands a single string with the same engine and rules as dotenv and jsonc: $VAR, ${VAR}, the ${VAR:-default} family, nesting, $$, ${scheme:ref} through value plugins, and $(command) when you turn it on. Use it when the value does not live in a .env or JSONC file: a YAML field, a flag, a database row, a setting such as token: "$(gopass show -o work/api)" or token: "${gopass:work/api}" that should behave exactly as it would in a .env.

Before you start

go get github.com/ubgo/textdata@latest

value is in the base module (standard library only). The value plugins are separate modules, so a program that uses one adds it, for example go get github.com/ubgo/textdata/plugins/gopass@latest. The runnable programs are examples/value/env, examples/value/commands (needs a Unix shell), examples/value/secrets and examples/value/strict.

value.Expand(ctx, s, opts...) reads and runs nothing unless an option asks for it:

Option What it adds
WithEnv() variables from the process environment (os.LookupEnv)
WithLookup(fn) variables from your function: a map, a config section
WithCommands() $(command) through shell.Run, bounded by ctx
WithRunner(r) $(command) through your plugin.CommandRunner instead of the shell
WithPlugins(ps...) value plugins: Lookupers, Resolvers for ${scheme:ref}, a CommandRunner, ValueTransformers, ExpandObservers
WithStrict() every unresolved reference is an error instead of "" or the text as written (Step 7)

Step 1 — Expand from the environment

With WithEnv(), $NAME and ${NAME} read the process environment. Without it, the environment is never read and every name is unset (""), so a value expands the same on every machine until you choose otherwise.

fromEnv · examples/value/env/main.go:61-74

func fromEnv(w io.Writer) error {
	ctx := context.Background()
	in := "postgres://${DEMO_USER}@$DEMO_HOST/app"
	with, err := value.Expand(ctx, in, value.WithEnv())
	if err != nil {
		return err
	}
	without, err := value.Expand(ctx, in)
	if err != nil {
		return err
	}
	fmt.Fprintf(w, "with env:    %s\nwithout env: %s\n", with, without)
	return nil
}

Step 2 — Defaults and required values

The operators are the Docker Compose ones the formats use: ${VAR:-d} for a default when unset or empty, ${VAR-d} when unset only, ${VAR:+a} and ${VAR+a} for an alternate, ${VAR:?msg} and ${VAR?msg} to fail, and $$ for a literal dollar. A default may hold another reference. The full table is in expand § Step 2.

defaultsAndRequired · examples/value/env/main.go:80-95

func defaultsAndRequired(w io.Writer) error {
	for _, in := range []string{
		"${DEMO_PORT:-5432}",
		"${DEMO_REGION:-${DEMO_HOST}}",
		"${DEMO_USER:+user is set}",
		"cost: $$5",
		"${DEMO_USER:?DEMO_USER must be set}",
	} {
		out, err := value.Expand(context.Background(), in, value.WithEnv())
		if err != nil {
			return err
		}
		fmt.Fprintf(w, "%-36s => %s\n", in, out)
	}
	return nil
}

Sources are asked in the order you give them and the first one that has a name wins, so WithLookup(overrides), WithEnv() lets a map override the environment. A value a source returns is expanded in turn; a reference back to a name being expanded reads as unset, so a chain always ends.

layeredSources · examples/value/env/main.go:101-116

func layeredSources(w io.Writer) error {
	overrides := map[string]string{
		"DEMO_HOST": "localhost",
		"DSN":       "postgres://${DEMO_USER}@${DEMO_HOST}/app",
	}
	lookup := func(name string) (string, bool) {
		v, ok := overrides[name]
		return v, ok
	}
	out, err := value.Expand(context.Background(), "$DSN", value.WithLookup(lookup), value.WithEnv())
	if err != nil {
		return err
	}
	fmt.Fprintf(w, "layered:     %s\n", out)
	return nil
}

Step 3 — Turn on commands, deliberately

$(command) is kept as written until you pass WithCommands() or WithRunner(r). That is the no-surprise rule: expanding a value must never run a command because the value happened to contain one. Turn commands on only for values from a source you trust as much as your code, such as your own config file, never for input from a request or another user.

offByDefault · examples/value/commands/main.go:46-53

func offByDefault(w io.Writer) error {
	out, err := value.Expand(context.Background(), "token=$(echo never-runs)", value.WithEnv())
	if err != nil {
		return err
	}
	fmt.Fprintf(w, "off:      %s\n", out)
	return nil
}

With WithCommands(), each command runs through shell.Run with the context you passed to Expand: sh -c on Unix-likes, PowerShell on Windows (see shell § Step 1). The command's own references expand first ($(printf '%s' $WHO) sees WHO), and the trailing newline is trimmed. A failing command is an *expand.CommandError that unwraps to the shell's error.

commandsOn · examples/value/commands/main.go:59-75

func commandsOn(w io.Writer) error {
	who := func(name string) (string, bool) { return "ada", name == "WHO" }
	out, err := value.Expand(context.Background(), "hello $(printf '%s\\n' $WHO)!", value.WithLookup(who), value.WithCommands())
	if err != nil {
		return err
	}
	fmt.Fprintf(w, "on:       %s\n", out)

	_, err = value.Expand(context.Background(), "$(echo 'no such entry' >&2; exit 2)", value.WithCommands())
	var ce *expand.CommandError
	var ee *shell.ExitError
	if !errors.As(err, &ce) || !errors.As(err, &ee) {
		return fmt.Errorf("want *expand.CommandError wrapping *shell.ExitError, got %v", err)
	}
	fmt.Fprintf(w, "failed:   code=%d\n          %v\n", ee.Code, err)
	return nil
}

The context bounds every command. For a limit or another shell that holds no matter which context is passed, hand a shell.Runner to WithRunner; Expand still passes its context to it, so cancelling stops the command.

boundedCommands · examples/value/commands/main.go:81-91

func boundedCommands(w io.Writer) error {
	ctx, cancel := context.WithTimeout(context.Background(), shortTimeout)
	defer cancel()
	_, err := value.Expand(ctx, "$(sleep 30)", value.WithCommands())
	fmt.Fprintf(w, "deadline: ErrTimeout=%t\n", errors.Is(err, shell.ErrTimeout))

	r := shell.Runner{Timeout: shortTimeout}
	_, err = value.Expand(context.Background(), "$(sleep 30)", value.WithRunner(r))
	fmt.Fprintf(w, "runner:   ErrTimeout=%t\n", errors.Is(err, shell.ErrTimeout))
	return nil
}

Step 4 — Allow-list commands with WithRunner

Any plugin.CommandRunner (one method, RunCommand(cmd string) (string, error)) can stand in for the shell. Use it to allow only known commands, to run them in a sandbox, or to fake them in tests. Whatever the runner returns as an error becomes an *expand.CommandError naming the command.

allowListed · examples/value/commands/main.go:111-121

func allowListed(w io.Writer) error {
	runner := allowList{"gopass show -o work/api": "fake-api-token"}
	out, err := value.Expand(context.Background(), "$(gopass show -o work/api)", value.WithRunner(runner))
	if err != nil {
		return err
	}
	fmt.Fprintf(w, "allowed:  %s\n", out)
	_, err = value.Expand(context.Background(), "$(rm -rf ~)", value.WithRunner(runner))
	fmt.Fprintf(w, "refused:  %v\n", err)
	return nil
}

Step 5 — Read secrets with value plugins

WithPlugins installs the same plugins the formats use. Each one owns a scheme: ${gopass:path} (the password-store plugin), ${file:path}, ${vault:path#field}, ${aws:sm/…} and ${aws:ssm/…}. A scheme no installed plugin owns is read as an ordinary name, so installing a plugin never changes how other values read. In production only the construction changes; the plugin guides cover each backend's options.

newPlugins · examples/value/secrets/main.go:91-109

func newPlugins(dir, vaultAddr, awsEndpoint string) ([]plugin.Plugin, error) {
	gp := gopassplugin.New(gopassplugin.WithRunner(fakeGopass))
	fp := fileplugin.New(dir) // confined to dir by default
	vp, err := vault.New(vault.WithAddress(vaultAddr), vault.WithToken(fakeVaultToken))
	if err != nil {
		return nil, err
	}
	ap, err := aws.New(
		aws.WithRegion(fakeRegion),
		aws.WithCredentialsSource(func() (aws.Credentials, error) {
			return aws.Credentials{AccessKeyID: fakeKeyID, SecretAccessKey: fakeSecretKey}, nil
		}),
		aws.WithEndpoint(awsEndpoint),
	)
	if err != nil {
		return nil, err
	}
	return []plugin.Plugin{gp, fp, vp, ap}, nil
}

expandSecrets · examples/value/secrets/main.go:115-131

func expandSecrets(w io.Writer, plugins []plugin.Plugin) error {
	for _, in := range []string{
		"${gopass:work/api}",
		"${file:api.key}",
		"${vault:secret/data/app#password}",
		"${aws:ssm//app/prod/db_url}",
		"Bearer ${gopass:work/api}",
		"[${nobody:thing}]", // no plugin owns the scheme: an ordinary, unset name
	} {
		out, err := value.Expand(context.Background(), in, value.WithPlugins(plugins...))
		if err != nil {
			return err
		}
		fmt.Fprintf(w, "%-34s => %s\n", in, out)
	}
	return nil
}

Step 6 — Handle errors

Expand returns the error types the formats return, so one set of errors.As checks covers a .env, a JSONC file and a single value:

Error When
*expand.RequiredError ${VAR:?msg} or ${VAR?msg} was not satisfied
*expand.CommandError a $(command) failed; it unwraps to *shell.ExitError, shell.ErrTimeout, shell.ErrNoShell or your runner's error
*plugin.Error a plugin failed; it names the plugin and unwraps to its error, such as plugin.ErrNotFound or plugin.ErrUnavailable
value.ErrUnsupportedPlugin (wrapped) a plugin passed to WithPlugins has no ability a single value can use (a SaveGuard or Validator alone)
*expand.UnsetError, *expand.UnknownSchemeError, *expand.CommandOffError only with WithStrict: an unset name, an unclaimed ${scheme:ref}, a $(command) with commands off (Step 7); all match expand.ErrStrict

requiredError · examples/value/env/main.go:122-130

func requiredError(w io.Writer) error {
	_, err := value.Expand(context.Background(), "${DEMO_API_TOKEN:?set DEMO_API_TOKEN first}", value.WithEnv())
	var re *expand.RequiredError
	if !errors.As(err, &re) {
		return fmt.Errorf("want *expand.RequiredError, got %v", err)
	}
	fmt.Fprintf(w, "required:    key=%s\n             %v\n", re.Key, err)
	return nil
}

pluginErrors · examples/value/secrets/main.go:137-145

func pluginErrors(w io.Writer, plugins []plugin.Plugin) error {
	_, err := value.Expand(context.Background(), "${gopass:work/missing}", value.WithPlugins(plugins...))
	var pe *plugin.Error
	if !errors.As(err, &pe) {
		return fmt.Errorf("want *plugin.Error, got %v", err)
	}
	fmt.Fprintf(w, "missing: plugin=%s not found=%t\n", pe.Plugin, errors.Is(err, plugin.ErrNotFound))
	return nil
}

Step 7 — Make unresolved references errors with WithStrict

By default a reference nothing answers reads as "", as in Compose, and a $(command) with commands off stays as written. A caller that needs certainty, such as a tool reading an API token from its config, would then send an empty token or the literal text $(gopass show x). WithStrict() turns each of those into an error. Everything that resolves reads exactly as before: a defined name, a name set to "", ${VAR:-d} and ${VAR-d} (their default), and ${VAR:+a} ("" when unset is what it asks for).

strictResolves · examples/value/strict/main.go:52-65

func strictResolves(w io.Writer) error {
	for _, in := range []string{
		"Bearer ${API_TOKEN}",
		"[${EMPTY}]",
		"${REGION:-eu-west-1}",
	} {
		out, err := value.Expand(context.Background(), in, value.WithStrict(), value.WithLookup(lookup))
		if err != nil {
			return err
		}
		fmt.Fprintf(w, "%-24s => %s\n", in, out)
	}
	return nil
}

What it refuses, each as its own type that matches expand.ErrStrict:

Reference Error
a name no source defines ($TOKEN, ${TOKEN}, a typo) *expand.UnsetError with Name
a ${scheme:ref} no installed plugin claims (${vault:x} without the vault plugin, ${gopas:x}) *expand.UnknownSchemeError with Scheme and Ref
a $(command) without WithCommands or WithRunner *expand.CommandOffError with Cmd

${VAR:?msg} keeps its own *expand.RequiredError, which does not match ErrStrict. Each error carries Offset, the byte offset in s of the reference that failed, and Via, the variable whose value held it when the reference was reached through another one; never a value (pinned in the engine and through value). An unclaimed ${scheme:ref} is treated as one name in strict mode: ${vault:db-pass} is refused, never read as the name vault:db with the default pass as it is without WithStrict.

strictRefuses · examples/value/strict/main.go:72-100

func strictRefuses(w io.Writer) error {
	for _, in := range []string{
		"Bearer ${API_TOKN}",         // a typo: unset
		"${gopas:work/api}",          // a scheme no plugin claims
		"$(gopass show -o work/api)", // commands are off
		"${API_SECRET:?set it}",      // required: its own error
	} {
		_, err := value.Expand(context.Background(), in, value.WithStrict(), value.WithLookup(lookup))
		var (
			unset   *expand.UnsetError
			scheme  *expand.UnknownSchemeError
			cmdOff  *expand.CommandOffError
			require *expand.RequiredError
		)
		kind := "none"
		switch {
		case errors.As(err, &unset):
			kind = "unset " + unset.Name
		case errors.As(err, &scheme):
			kind = "unknown scheme " + scheme.Scheme
		case errors.As(err, &cmdOff):
			kind = "command off"
		case errors.As(err, &require):
			kind = "required " + require.Key
		}
		fmt.Fprintf(w, "%-28s strict=%-5t %s\n  %v\n", in, errors.Is(err, expand.ErrStrict), kind, err)
	}
	return nil
}

The formats have the same switch, named WithStrictExpand() there because jsonc's Strict() already means "parse as plain JSON": dotenv expansion, jsonc expansion. A format of your own sets expand.Engine.Strict (expand).

Try it

cd examples/value/env && go run .
with env:    postgres://ada@db.internal/app
without env: postgres://@/app
${DEMO_PORT:-5432}                   => 5432
${DEMO_REGION:-${DEMO_HOST}}         => db.internal
${DEMO_USER:+user is set}            => user is set
cost: $$5                            => cost: $5
${DEMO_USER:?DEMO_USER must be set}  => ada
layered:     postgres://ada@localhost/app
required:    key=DEMO_API_TOKEN
             expand: required variable DEMO_API_TOKEN is not set or is empty: set DEMO_API_TOKEN first
cd examples/value/commands && go run .
off:      token=$(echo never-runs)
on:       hello ada!
failed:   code=2
          expand: command "echo 'no such entry' >&2; exit 2" failed: shell: command "echo 'no such entry' >&2; exit 2" exited with code 2: no such entry
deadline: ErrTimeout=true
runner:   ErrTimeout=true
allowed:  fake-api-token
refused:  expand: command "rm -rf ~" failed: command not allowed
cd examples/value/secrets && go run .
${gopass:work/api}                 => fake-gopass-token
${file:api.key}                    => fake-file-key
${vault:secret/data/app#password}  => fake-vault-password
${aws:ssm//app/prod/db_url}        => postgres://db.internal/app
Bearer ${gopass:work/api}          => Bearer fake-gopass-token
[${nobody:thing}]                  => []
missing: plugin=gopass not found=true
cd examples/value/strict && go run .
Bearer ${API_TOKEN}      => Bearer fake-token
[${EMPTY}]               => []
${REGION:-eu-west-1}     => eu-west-1
Bearer ${API_TOKN}           strict=true  unset API_TOKN
  expand: variable API_TOKN is not set (offset 7)
${gopas:work/api}            strict=true  unknown scheme gopas
  expand: no resolver for scheme "gopas" in ${gopas:work/api} (offset 0)
$(gopass show -o work/api)   strict=true  command off
  expand: command substitution is off: $(gopass show -o work/api) (offset 0)
${API_SECRET:?set it}        strict=false required API_SECRET
  expand: required variable API_SECRET is not set or is empty: set it

Pitfalls

  • Unset is not an error. ${TOKEN} with nothing defining TOKEN is "". Write ${TOKEN:?set TOKEN} when one missing value must stop the program, or pass WithStrict() to refuse every unresolved reference.
  • Without a source, nothing has a value. Expand(ctx, "$HOME") is "": pass WithEnv() to read the environment.
  • A - after a colon is an operator, not part of a scheme. ${nobody:owns-this} is the name nobody:owns with the default this, exactly as in the formats. Scheme references such as ${vault:a/b} are not affected, because a scheme's colon is never followed by -, + or ?.
  • Command output is trimmed of trailing newlines only. A command printing " x \n" gives " x ".
  • The runner you give wins. WithRunner and a CommandRunner plugin replace the shell; the last one given is used. WithCommands after them does not bring the shell back.
  • A $(command) on Windows runs in PowerShell. A value written for sh ($(printf '%s' x)) needs WithRunner(shell.Runner{Shell: []string{"bash", "-c"}}) there; see shell § Pitfalls.
  • Transformers and observers see path "". A single value has no key or path; a ValueTransformer that picks values by path sees "".

Related