Config Loader

lean_automator.config.loader

Loads and manages application configuration from multiple sources.

This module provides functions to load configuration settings from YAML files (defaults), JSON files (e.g., costs), and environment variables (overrides, secrets). It automatically determines the paths for the default configuration files (config.yml, model_costs.json) by: 1. Checking specific environment variables (LEAN_AUTOMATOR_CONFIG_FILE, LEAN_AUTOMATOR_COSTS_FILE). 2. Searching upwards from this file's location for a project root marker (pyproject.toml) and looking for the files in that root directory. 3. As a fallback, looking in the current working directory (with a warning).

It exposes the loaded configuration via a singleton dictionary APP_CONFIG and provides helper functions to access specific sensitive values directly from the environment.

Functions

load_configuration(dotenv_path: Optional[str] = None, env_override_map: List[Tuple[str, List[str], Type]] = ENV_OVERRIDES) -> Dict[str, Any]

Loads configuration layers: YAML defaults, JSON costs, and env overrides.

Determines the paths for the default configuration file (config.yml) and costs file (model_costs.json) using the following priority: 1. Environment variables: LEAN_AUTOMATOR_CONFIG_FILE and LEAN_AUTOMATOR_COSTS_FILE. 2. Project Root Search: Looks for pyproject.toml upwards from this loader's directory and constructs paths relative to that root. 3. Current Working Directory (Fallback): If neither of the above yields a path, it tries the CWD (logging a warning).

After determining paths, it reads the base configuration from the YAML file, merges data from the JSON costs file into the 'costs' key, loads environment variables from a .env file (if specified or found), and finally applies overrides defined in env_override_map using values from environment variables.

Parameters:

Name	Type	Description	Default
dotenv_path	Optional[str]	Explicit path to the .env file. If None, python-dotenv searches standard locations.	None
env_override_map	List[Tuple[str, List[str], Type]]	The mapping defining which environment variables override which configuration keys and their types.	ENV_OVERRIDES

Returns:

Type	Description
Dict[str, Any]	A dictionary containing the fully merged configuration. Returns an
Dict[str, Any]	empty dictionary if the determined base config file cannot be found or parsed.
Dict[str, Any]	Cost data or overrides might be missing if respective files/variables are
Dict[str, Any]	absent or invalid.

Source code in lean_automator/config/loader.py

def load_configuration(
    dotenv_path: Optional[str] = None,
    env_override_map: List[Tuple[str, List[str], Type]] = ENV_OVERRIDES,
) -> Dict[str, Any]:
    """Loads configuration layers: YAML defaults, JSON costs, and env overrides.

    Determines the paths for the default configuration file (`config.yml`)
    and costs file (`model_costs.json`) using the following priority:
    1. Environment variables: `LEAN_AUTOMATOR_CONFIG_FILE` and
       `LEAN_AUTOMATOR_COSTS_FILE`.
    2. Project Root Search: Looks for `pyproject.toml` upwards from this loader's
       directory and constructs paths relative to that root.
    3. Current Working Directory (Fallback): If neither of the above yields a path,
       it tries the CWD (logging a warning).

    After determining paths, it reads the base configuration from the YAML file,
    merges data from the JSON costs file into the 'costs' key, loads environment
    variables from a .env file (if specified or found), and finally applies
    overrides defined in `env_override_map` using values from environment variables.

    Args:
        dotenv_path: Explicit path to the .env file. If None, `python-dotenv`
                     searches standard locations.
        env_override_map: The mapping defining which environment variables
                          override which configuration keys and their types.

    Returns:
        A dictionary containing the fully merged configuration. Returns an
        empty dictionary if the determined base config file cannot be found or parsed.
        Cost data or overrides might be missing if respective files/variables are
        absent or invalid.
    """
    config: Dict[str, Any] = {}

    # --- Determine Configuration File Paths ---
    effective_config_path: Optional[pathlib.Path] = None
    effective_costs_path: Optional[pathlib.Path] = None

    # 1. Check Environment Variables
    env_config_path_str = os.getenv(ENV_CONFIG_PATH)
    env_costs_path_str = os.getenv(ENV_COSTS_PATH)

    if env_config_path_str:
        effective_config_path = pathlib.Path(env_config_path_str)
        logger.info(
            f"Using config path from environment variable {ENV_CONFIG_PATH}: "
            f"'{effective_config_path}'"
        )
    if env_costs_path_str:
        effective_costs_path = pathlib.Path(env_costs_path_str)
        logger.info(
            f"Using costs path from environment variable {ENV_COSTS_PATH}: "
            f"'{effective_costs_path}'"
        )

    # 2. Find Project Root (if paths not set by env vars)
    if effective_config_path is None or effective_costs_path is None:
        project_root = _find_project_root(start_path=pathlib.Path(__file__).parent)

        if project_root:
            logger.info(f"Determined project root: '{project_root}'")
            if effective_config_path is None:
                effective_config_path = project_root / DEFAULT_CONFIG_FILENAME
                logger.info(
                    f"Derived config path from project root: '{effective_config_path}'"
                )
            if effective_costs_path is None:
                effective_costs_path = project_root / DEFAULT_COSTS_FILENAME
                logger.info(
                    f"Derived costs path from project root: '{effective_costs_path}'"
                )
        else:
            logger.warning(
                f"Could not find project root marker '{PROJECT_ROOT_MARKER}'. "
                "Falling back to current working directory for config/costs paths."
            )
            # 3. Fallback to CWD (only if project root not found AND env var not set)
            cwd = pathlib.Path.cwd()
            if effective_config_path is None:
                effective_config_path = cwd / DEFAULT_CONFIG_FILENAME
                logger.info(
                    f"Using fallback config path in CWD: '{effective_config_path}'"
                )
            if effective_costs_path is None:
                effective_costs_path = cwd / DEFAULT_COSTS_FILENAME
                logger.info(
                    f"Using fallback costs path in CWD: '{effective_costs_path}'"
                )

    # Ensure paths are resolved before use (makes error messages clearer)
    if effective_config_path:
        effective_config_path = effective_config_path.resolve()
    if effective_costs_path:
        effective_costs_path = effective_costs_path.resolve()

    # --- Load Base YAML Configuration ---
    if effective_config_path:
        try:
            with open(effective_config_path, encoding="utf-8") as f:
                loaded_yaml = yaml.safe_load(f)
                config = (
                    loaded_yaml if isinstance(loaded_yaml, dict) else {}
                )  # Ensure it's a dict
            logger.info(f"Loaded base config from '{effective_config_path}'.")
        except FileNotFoundError:
            logger.warning(f"Base config file '{effective_config_path}' not found.")
        except yaml.YAMLError as e:
            logger.error(
                f"Error parsing YAML '{effective_config_path}': {e}", exc_info=True
            )
            return {}  # Critical error parsing base config, return empty
        except Exception as e:
            logger.error(
                f"Unexpected error loading '{effective_config_path}': {e}",
                exc_info=True,
            )
            return {}
    else:
        logger.error(
            "Could not determine a valid path for the base configuration file."
        )
        return {}  # Cannot proceed without base config path

    # --- Load and Merge JSON Costs Data ---
    if effective_costs_path:
        try:
            with open(effective_costs_path, encoding="utf-8") as f:
                model_costs = json.load(f)
            if "costs" not in config or not isinstance(config.get("costs"), dict):
                config["costs"] = {}  # Ensure 'costs' key exists and is a dict
            # Safely update, preferring loaded costs over potential existing values
            config["costs"].update(model_costs if isinstance(model_costs, dict) else {})
            logger.info(f"Loaded and merged costs from '{effective_costs_path}'.")
        except FileNotFoundError:
            logger.warning(f"Costs file '{effective_costs_path}' not found.")
            config.setdefault(
                "costs", {}
            )  # Ensure 'costs' key exists even if file not found
        except json.JSONDecodeError as e:
            logger.error(
                f"Error parsing JSON costs '{effective_costs_path}': {e}", exc_info=True
            )
            config.setdefault("costs", {})  # Ensure 'costs' key exists after error
        except Exception as e:
            logger.error(
                f"Unexpected error loading '{effective_costs_path}': {e}", exc_info=True
            )
            config.setdefault("costs", {})
    else:
        logger.warning(
            "Could not determine a valid path for the costs file. "
            "'costs' section may be empty."
        )
        config.setdefault("costs", {})  # Ensure 'costs' key exists

    # --- Load .env file into environment variables ---
    try:
        # Pass override=True if you want .env to override existing env vars
        loaded_env = load_dotenv(dotenv_path=dotenv_path, verbose=True, override=True)
        if loaded_env:
            logger.info(
                ".env file loaded into environment variables (overriding existing)."
            )
        else:
            logger.debug(".env file not found or empty.")
    except Exception as e:
        # Catch potential errors during dotenv loading (e.g., permission issues)
        logger.error(f"Error loading .env file: {e}", exc_info=True)

    # --- Apply Environment Variable Overrides for config VALUES ---
    logger.info("Checking for environment variable overrides for config values...")
    override_count = 0
    for env_var, config_keys, target_type in env_override_map:
        env_value_str = os.getenv(env_var)
        if env_value_str is not None:
            try:
                # Attempt conversion
                typed_value = target_type(env_value_str)
                # Update the nested dictionary
                _update_nested_dict(config, config_keys, typed_value)
                logger.info(
                    f"Applied value override: '{'.'.join(config_keys)}' = "
                    f"'{typed_value}' (from env '{env_var}')"
                )
                override_count += 1
            except ValueError:
                logger.warning(
                    f"Value override failed: Cannot convert env var '{env_var}' "
                    f"value '{env_value_str}' to target type {target_type.__name__}."
                )
            except Exception as e:
                logger.error(
                    f"Value override error: Unexpected issue applying env var "
                    f"'{env_var}': {e}",
                    exc_info=True,
                )
    if override_count > 0:
        logger.info(f"Applied {override_count} environment variable value override(s).")
    else:
        logger.info("No environment variable value overrides applied.")

    return config

get_gemini_api_key() -> Optional[str]

Retrieves the Gemini API Key directly from environment variables.

Returns:

Type	Description
Optional[str]	The Gemini API key string if the 'GEMINI_API_KEY' environment
Optional[str]	variable is set, otherwise None.

Source code in lean_automator/config/loader.py

def get_gemini_api_key() -> Optional[str]:
    """Retrieves the Gemini API Key directly from environment variables.

    Returns:
        The Gemini API key string if the 'GEMINI_API_KEY' environment
        variable is set, otherwise None.
    """
    return os.getenv("GEMINI_API_KEY")

get_lean_automator_shared_lib_path() -> Optional[str]

Retrieves the Lean Automator Shared Lib Path from environment variables.

Returns:

Type	Description
Optional[str]	The path string if the 'LEAN_AUTOMATOR_SHARED_LIB_PATH' environment
Optional[str]	variable is set, otherwise None.

Source code in lean_automator/config/loader.py

def get_lean_automator_shared_lib_path() -> Optional[str]:
    """Retrieves the Lean Automator Shared Lib Path from environment variables.

    Returns:
        The path string if the 'LEAN_AUTOMATOR_SHARED_LIB_PATH' environment
        variable is set, otherwise None.
    """
    return os.getenv("LEAN_AUTOMATOR_SHARED_LIB_PATH")