ML Configurations

A small, highly opinionated python tool to handle configurations for machine learning pipelines. The library is designed to load configurations from both json and yaml files, as well as from standard python dictionaries.

Design rules

The configurations, once loaded are frozen. Each configuration file can contain only int, float, str, bool and None fields, as well as homogeneous lists of one of the same types. That’s all. No nested structures are allowed.

Installation

ML configurations can be installed directly from git by running

pip install ml-confs

Basic usage

A valid ml_confs configuration file configs.yml in YAML is:

int_field: 1
float_field: 1.0
str_field: 'string'
bool_field: true
none_field: null
list_field: [1, 2, 3]

To load it we just use:

import ml_confs

#Loading configs
configs = ml_confs.from_file('configs.yml')

#Accessing configs with dot notation
print(configs.int_field) # >>> 1

#Additionally, one can use the ** notation to unpack the configurations
def foo(**kwargs):
    # Do stuff...
foo(**configs)


#Saving configs to json format
configs.to_file('json_configs_copy.json') #Will create a .json file 

One can also pretty print a loaded configuration with configs.tabulate(), which in the previous example would output:

┏━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Key         ┃ Value     ┃ Type      ┃
┡━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━┩
│ int_field   │ 1         │ int       │
│ float_field │ 1.0       │ float     │
│ str_field   │ string    │ str       │
│ bool_field  │ True      │ bool      │
│ none_field  │ None      │ NoneType  │
│ list_field  │ [1, 2, 3] │ list[int] │
└─────────────┴───────────┴───────────┘

JAX Pytree registration

By default, ml_confs will try to register the configuration object as a JAX pytree, so that configs can be safely used with JAX transformations.

import ml_confs
import jax

configs = mlc.from_dict({'exp': 1.5})

@jax.jit 
def power_fn(x, cfg):
    return x**cfg.exp

assert f(2.0, configs) == 2.0**exp # This works!
assert jax.grad(power_fn)(3.0, configs) == 3.0**(exp - 1.0) * exp # This works too!

If JAX is not installed the following warning will be displayed:

Unable to import JAX. The argument register_jax_pytree will be ignored. To suppress this warning, load the configurations with register_jax_pytree=False.

If one is not interested in this feature, the warning can be silenced by explicitly setting register_jax_pytree to False upon configuration loading.

API Reference

function from_json

from_json(path: PathLike, register_jax_pytree: bool = False)

Load configurations from a JSON file.

Args:

• path (os.PathLike): Configuration file path.
• register_jax_pytree (bool, optional): Register the configuration as a JAX pytree. This allows the configurations to be safely used in JAX’s transformations. Defaults to False.

Returns:

• Configs: Instance of the loaded configurations.

---

function from_yaml

from_yaml(path: PathLike, register_jax_pytree: bool = False)

Load configurations from a YAML file.

Args:

• path (os.PathLike): Configuration file path.
• register_jax_pytree (bool, optional): Register the configuration as a JAX pytree. This allows the configurations to be safely used in JAX’s transformations. Defaults to False.

Returns:

• Configs: Instance of the loaded configurations.

---

function from_dict

from_dict(storage: dict, register_jax_pytree: bool = False)

Load configurations from a python dictionary.

Args:

• storage (dict): Configuration dictionary.
• register_jax_pytree (bool, optional): Register the configuration as a JAX pytree. This allows the configurations to be safely used in JAX’s transformations. Defaults to False.

Returns:

• Configs: Instance of the loaded configurations.

---

function from_file

from_file(path: PathLike, register_jax_pytree: bool = False)

Load configurations from a YAML/JSON file.

Args:

• path (os.PathLike): Configuration file path.
• register_jax_pytree (bool, optional): Register the configuration as a JAX pytree. This allows the configurations to be safely used in JAX’s transformations. Defaults to False.

Returns:

• Configs: Instance of the loaded configurations.

---

function to_json

to_json(path: PathLike, configs: Configs)

Save configurations to a JSON file.

Args:

• path (os.PathLike): File path to save the configurations.
• configs (Configs): Instance of the configurations.

---

function to_yaml

to_yaml(path: PathLike, configs: Configs)

Save configurations to a YAML file.

Args:

• path (os.PathLike): File path to save the configurations.
• configs (Configs): Instance of the configurations.

---

function to_file

to_file(path: PathLike, configs: Configs)

Save configurations to a YAML/JSON file.

Args:

• path (os.PathLike): File path to save the configurations.
• configs (Configs): Instance of the configurations.

---

function to_dict

to_dict(configs: Configs) → dict

Export configurations to a python dictionary.

Args:

• configs (Configs): Instance of the configurations.

Returns:

• dict: A standard python dictionary containing the configurations.

---

function pprint

pprint(configs: Configs)

Pretty print configurations.

Args:

• configs (Configs): An instance of the configurations.

---

class InvalidStructureError

---

class Configs

---

method tabulate

tabulate()

Print the configurations in a tabular format.

---

method to_dict

to_dict() → dict

Export configurations to a python dictionary.

Returns:

• dict: A standard python dictionary containing the configurations.

---

method to_file

to_file(path: PathLike)

Save configurations to a YAML/JSON file.

Args:

• path (os.PathLike): File path to save the configurations.

---

method to_json

to_json(path: PathLike)

Save configurations to a JSON file.

Args:

• path (os.PathLike): File path to save the configurations.

---

method to_yaml

to_yaml(path: PathLike)

Save configurations to a YAML file.

Args:

• path (os.PathLike): File path to save the configurations.

---

The API reference was automatically generated via lazydocs.

This site is open source. Improve this page.