Layered configuration with command line arguments

[aschey]

Hey, thanks for making this library. I've been really enjoying it. Recently, I wanted to see if it would be easy to support the common three-layered configuration approach of CLI overrides, environment variables, and config files - in order of decreasing precedence. I think this should be doable with a few tweaks.

I tried using partials with clap in order to have a single source of truth for all config values:

#[derive(Config, Args, Debug)]
#[config(env_prefix = "APP_")]
pub struct AppConfig {
    #[arg(long)]
    #[setting(default = 3000)]
    port: usize,
    #[arg(long)]
    #[setting(default = true)]
    secure: bool,
    #[arg(long)]
    #[setting(
        default = vec!["localhost".to_string()],
        parse_env = schematic::env::split_comma, 
    )]
    allowed_hosts: Vec<String>,
}

#[derive(Parser, Debug)]
#[command(version, about)]
struct Cli {
    #[command(flatten)]
    config: PartialAppConfig,
    ...
}

This doesn't work because the extra annotations (#[derive(Args)] plus the #[arg] attributes) are not propagated to the generated PartialAppConfig struct. I think this can work if additional attributes are exposed to allow users to attach arbitrary attributes to the partial struct. Something like this maybe:

#[derive(Config, Args, Debug)]
#[config(partial_derive(Args), env_prefix = "APP_")]
pub struct AppConfig {
    #[setting(default = 3000, partial(arg(long)))]
    port: usize,
    ...
}

The second issue I ran into is with precedence. I'm combining the CLI config with the env/file sources like this:

let cli = Cli::parse();

let mut loader = ConfigLoader::<AppConfig>::new();
loader.file(config_path)?;
let mut partial_config = loader.load_partial(&())?;
partial_config.merge(&(), cli.config)?;
let final_config = config.finalize(&())?;

This almost works, but environment variables currently take precedence over the last partial. Some way to specify the priority might work here:

partial_config.merge_with_priority(&(), cli.config, Priority::Highest)?;

Curious what your thoughts are on this. I'd be happy to work on it if it's something you'd be okay with. Thanks!

[milesj]

Yeah schematic assumes env vars are the highest priority and there's a bit of logic that enforces that. This would be tough to change since this happens in finalize and not the layers/merging.

I'd say your best bet (for the interim) is to just manually mutate the final config outside of schematic.

let mut loader = ConfigLoader::<AppConfig>::new();
loader.file(config_path)?;

let mut config = loader.load()?.config;

if args.some_value {
  config.some_value = args.some_value;
}

[aschey]

Ah, your comment about finalize gave me an idea. It works if you merge in the final layer after calling finalize.

let mut loader = ConfigLoader::<AppConfig>::new();
loader.file(config_path)?;

let partial_config = loader.load_partial(&())?;
let mut final_partial_config = partial_config.finalize(&())?;
final_partial_config.merge(&(), cli.config)?;
let config = AppConfig::from_partial(final_partial_config);

Guessing this is a slight abuse of finalize, but it's something.

I also made a prototype for the first solution that I mentioned regarding passing additional attributes to the generated partial struct. I haven't tested it thoroughly yet, but I'm happy to clean it up and make a PR if you'd be interested.

[milesj]

Yah I don't think I'll change finalize, but feel free to submit a PR for the derive stuff.

[aschey]

I just realized that since clap also has environment variable integration, users who want CLI arguments to take precedence can use that instead of schematic's environment variable parsing and that will preserve the expected order of operations. I'm satisfied with that approach.