`marsh.utils`¶

Collection of general utilities used throughout the framework.

class SequenceProtocol(*args, **kwargs)[source]¶: Protocol matching a sequence class.

class MappingProtocol(*args, **kwargs)[source]¶: Protocol matching a mapping class.

class NamedTupleProtocol(*args, **kwargs)[source]¶: Protocol matching a namedtuple class.

class SingletonMeta[source]¶

Converts a class into a singleton.

When the constructor of the class is called the first time, an instance is constructed. After this, the constructor will always return that instance unless it is garbage collected.

To allow for garbage collection of unused instances a weak reference is kept of the instance.

class CacheInfo(hits=0, misses=0, maxsize=None, currsize=0)[source]¶

Tuple with information for a cache.

Parameters:

hits (int) –
misses (int) –
maxsize (Optional[int]) –
currsize (int) –

hits: int¶: Alias for field number 0

misses: int¶: Alias for field number 1

maxsize: Optional[int]¶: Alias for field number 2

currsize: int¶: Alias for field number 3

classmethod from_info(info)[source]¶

Create an instance of this class from any cache info that follows the same attribute pattern.

Parameters:: info (_CacheInfoInputType) – The cache info to use as source
Return type:: CacheInfo
Returns:: An instance of this class containing the same information as the input.

class CacheType(*args, **kwargs)[source]¶

Represents the protocol for a cache in marsh.

cache_info()[source]¶

Get info from the cache.

Return type:: CacheInfo
Returns:: The cache info.

cache_clear()[source]¶

Clear the content of the cache.

Return type:: None

cache_disable()[source]¶

Disable the cache.

Return type:: None

cache_enable()[source]¶

Enable the cache.

Return type:: None

cache(*, maxsize: Optional[int] = None, typed: bool = False, safe: bool = True, binding: Optional[Literal['weak', 'ignore']] = None) → Callable[[_C], _C][source]¶

cache(fn: _C, /, *, maxsize: Optional[int] = None, typed: bool = False, safe: bool = True, binding: Optional[Literal['weak', 'ignore']] = None) → _C

Extended LRU cache decorator for functions.

Allows for safe usage of lru cache, where the TypeError produced by non-hashable types in the input arguments are caught before a new attempt is made to call the function, this time bypassing the cache compmletely.

Note

The bypass is triggered for TypeError. If the function itself can raise this error it will be called twice before the final error is propagated.

The wrapped function fulfills the CacheType protocol.

Parameters:

fn (Optional[TypeVar(_C, bound= Callable)]) – If None, the wrapper is returned, otherwise this argument is wrapped and returned.
maxsize (Optional[int]) – Maximum size of lru cache.
typed (bool) – Differentiate between different types of values that produce the same hash.
safe (bool) – Make a second attempt at calling the function and bypassing the cache if the first attempt fails with a TypeError.
binding (Optional[Literal[‘weak’, ‘ignore’]]) – Should only be used when a method to a class is decorated with this function. 'weak' will make sure that a weak reference to self/cls/metacls is stored in the cache instead of a strong reference. This allows for objects with lru_cache to be garbage collected when no longer used. 'ignore' allows the first argument to bypass the cache, only performing a lookup on the other arguments.

Return type:

Union[TypeVar(_C, bound= Callable), Callable[[TypeVar(_C, bound= Callable)], TypeVar(_C, bound= Callable)]]

Returns:

The wrapper, unless fn is given, which instead wraps the argument and returns it.

make_hash_key(*args, **kwargs)[source]¶

Make a single cached hashable key from arguments.

All arguments must be hashable.

Return type:: Any
Returns:: Hashable object that caches its hash, only caclulating it once.

make_typed_hash_key(*args, **kwargs)[source]¶

Make a single cached hashable key from arguments.

Includes the type of each value.

All arguments must be hashable.

Return type:: Any
Returns:: Hashable object that caches its hash, only caclulating it once.

class SafeDict[source]¶

Dictionary that works with unhashable keys.

Just like built-in dict the order is kept.

class ValueCache(typed=False)[source]¶

A dict-like cache that allows for non-hashable keys.

The hash for cachable keys are also hashed to improve lookup speed.

Parameters:: typed (bool) – The key type is also taken into consideration when matching keys.

cache_clear()[source]¶

Clear the cache, removing all stored items.

Return type:: None

cache_info()[source]¶

Get statistics for the cache.

Return type:: CacheInfo

cache_disable()[source]¶

Disable the cache.

Getting a value for a key raise KeyError. Setting a value for a key is a no-op (the value is not stored). Deleting a value in the cache is still permitted.

Return type:: None

cache_enable()[source]¶

Enable the cache.

Return type:: None

class CachePool[source]¶

A collection of caches.

add(name, cache)[source]¶

Add a cache to the collection.

Parameters:

name (str) – A name for the cache being added.
cached – The cache to add.
cache (CacheType) –

Return type:

None

new_callable_cache(*, name: str, maxsize: Optional[int] = None, typed: bool = False, safe: bool = True, binding: Optional[Literal['weak', 'ignore']] = None) → Callable[[_C], _C][source]¶

new_callable_cache(fn: _C, *, name: str, maxsize: Optional[int] = None, typed: bool = False, safe: bool = True, binding: Optional[Literal['weak', 'ignore']] = None) → _C

Wrap a function with cache() while simultaneously adding it to this pool.

See cache() for descriptions of arguments forwarded to the wrapper.

Parameters:: name (str) – The name associated with the new cache.
Return type:: Union[TypeVar(_C, bound= Callable), Callable[[TypeVar(_C, bound= Callable)], TypeVar(_C, bound= Callable)]]
Returns:: The cache wrapper.

new_value_cache(name)[source]¶

Create a new value cache and add it to this pool before returning it.

Parameters:: name (str) – The name associated with the new cache.
Return type:: ValueCache
Returns:: The value cache.

cache_clear()[source]¶

Clear every cache in this container.

This does not remove the caches from the container, it only calls the clear_cache() function on each cache.

Return type:: None

cache_info()[source]¶

Get the cache info for all caches in this container.

Return type:: Mapping[str, CacheInfo]
Returns:: A mapping of the name of each cache as key and its respective cache info as value.

cache_disable()[source]¶

Disable all caches in this container.

Return type:: None

cache_enable()[source]¶

Enable all caches in this container.

Return type:: None

class IterableFromIterator(iterator)[source]¶

Lazily stores results of an iterator.

Allows for multiple iterations over the wrapped iterator while only consuming it once.

Parameters:: iterator (Iterator[TypeVar(_T)]) – The iterator to cache.

class PriorityOrder[source]¶

A mutable sequence of values ordered by priority.

Higher priority items are moved forward while lower priority items are moved backward in the sequence.

RelativePriority¶

Relative priority input type.

alias of Union[None, _V, int, Iterable[_V], Iterable[int]]

reload()[source]¶

Reshuffle values in this container based on priority.

Return type:: None

class WeakTypeCache(skip_types=())[source]¶

A type cache that holds its values as weak references.

When iterated over the types are yielded in reverse order, the most recently added is yielded first.

Parameters:: skip_types (Iterable[Any]) – Any types to ignore. If attempting to store one of these types it becomes a no-op.

add(type_)[source]¶

Add a type to the cache.

Parameters:

type – The type to add.
type_ (Any) –

Return type:

None

clear()[source]¶

Clear the cache, removing all stored types.

Return type:: None

is_missing(value)[source]¶

Determine if a value is missing.

Supports marsh.MISSING, omegaconf.MISSING and dataclasses.MISSING.

Parameters:: value (Any) – Any value to test for missing.
Return type:: bool
Returns:: True if missing, else False.

is_sequence(value)[source]¶

Determine if a value is a sequence (excluding string, bytes).

Parameters:: value (Any) – Any value to evaluate.
Return type:: TypeGuard[Sequence]
Returns:: True if value is a sequence, else False.

is_sequence_type(value)[source]¶

Determine if a value is a sequence type (excluding string, bytes types).

Parameters:: value (Any) – Any value to evaluate.
Return type:: TypeGuard[Type[Sequence]]
Returns:: True if value is a sequence type, else False.

is_empty_tuple_type(value)[source]¶

Determine if a value is an empty tuple type (typing.Tuple[()], tuple[()]).

Parameters:: value (Any) – Any value to evaluate.
Return type:: TypeGuard[Type[Tuple]]
Returns:: True if value is a an empty tuple type, else False.

is_fixed_size_tuple_type(value)[source]¶

Determine if a value is fixed size tuple type.

Parameters:: value – Any value to evaluate.
Return type:: bool
Returns:: True if value is a a fixed size tuple type, else False.

is_mapping(value)[source]¶

Determine if a value is a mapping.

Parameters:: value (Any) – Any value to evaluate.
Return type:: TypeGuard[Mapping]
Returns:: True if value is a mapping, else False.

is_mapping_type(value)[source]¶

Determine if a value is a mapping type.

Parameters:: value (Any) – Any value to evaluate.
Return type:: TypeGuard[Type[Mapping]]
Returns:: True if value is a mapping, else False.

is_primitive(value)[source]¶

Determine if a value is primitive.

PrimitiveElement: int | float | bool | str

Parameters:: value (Any) – Any value to evaluate.
Return type:: TypeGuard[Union[int, float, bool, str]]
Returns:: True if value is primitive, else False.

is_primitive_type(value)[source]¶

Determine if a value is primitive type.

PrimitiveElement: int | float | bool | str

Parameters:: value (Any) – Any value to evaluate.
Return type:: TypeGuard[Union[Type[int], Type[float], Type[bool], Type[str]]]
Returns:: True if value is a primitive type, else False.

is_namedtuple(value)[source]¶

Determine if a value is a namedtuple.

Parameters:: value (Any) – Any value to evaluate.
Return type:: TypeGuard[NamedTupleProtocol]
Returns:: True if value is a namedtuple, else False.

is_namedtuple_type(value)[source]¶

Determine if a value is a namedtuple type.

Parameters:: value (Any) – Any value to evaluate.
Return type:: TypeGuard[Type[NamedTupleProtocol]]
Returns:: True if value is a namedtuple type, else False.

is_typed_namedtuple(value)[source]¶

Determine if a value is a typed namedtuple.

Parameters:: value (Any) – Any value to evaluate.
Return type:: TypeGuard[NamedTupleProtocol]
Returns:: True if value is a namedtuple, else False.

is_typed_namedtuple_type(value)[source]¶

Determine if a value is a typed namedtuple type.

Parameters:: value (Any) – Any value to evaluate.
Return type:: TypeGuard[Type[NamedTupleProtocol]]
Returns:: True if value is a typed namedtuple type, else False.

is_typeddict_type(value)[source]¶

Determine if a value is a typed dict type.

Parameters:: value (Any) – Any value to evaluate.
Return type:: bool
Returns:: True if value is a typed dict type, else False.

is_defaultdict_type(value)[source]¶

Determine if a value is a default dict type.

Parameters:: value (Any) – Any value to evaluate.
Return type:: bool
Returns:: True if value is a default dict type, else False.

is_callable(value)[source]¶

Determine if a value is callable.

Always returns False if the value is a protocol or typing alias.

Parameters:: value (Any) – Any value to evaluate.
Return type:: bool
Returns:: True if value is a callable, else False.

is_obj_instance(value)[source]¶

Determine if a value is an instantiated object.

Parameters:: value (Any) – Any value to evaluate.
Return type:: bool
Returns:: True if value is an iinstantiated object, else False.

is_annotated(value)[source]¶

Determine if a value originates from typing.Annotated.

Parameters:: value (Any) – Any value to evaluate.
Return type:: bool
Returns:: True if value originates from typing.Annotated, else False.

get_annotations(value)[source]¶

Extract the additional arguments of an instance of typing.Annotated

Parameters:: value (Any) – The typing.Annotated instance.
Return type:: Sequence
Returns:: The annotations.

is_protocol(value)[source]¶

Determine if a value is a protocol.

Any class that has typing.Protocol in its immediate bases is considered a protocol.

Parameters:: value (Any) – Any value to evaluate.
Return type:: bool
Returns:: True if value is a protocol, else False.

is_literal(value)[source]¶

Determine if a value is a typing.Literal.

Parameters:: value (Any) – Any value to evaluate.
Return type:: bool
Returns:: True if value is a typing.Literal, else False.

is_literal_string(value)[source]¶

Determine if a value is a typing.LiteralString.

Parameters:: value (Any) – Any value to evaluate.
Return type:: bool
Returns:: True if value is a typing.LiteralString, else False.

is_typing_alias(value)[source]¶

Determine if a value is a typing alias.

If the class name ends with '_GenericAlias', '_SpecialForm' or '_AnnotatedAlias' the value is considered a typing alias.

For python 3.9+ a check is made against types.GenericAlias.

Parameters:: value (Any) – Any value to evaluate.
Return type:: bool
Returns:: True if value is a typing alias, else False.

is_optional(type_)[source]¶

Determine if a type is a optional.

Parameters:

type – The type to evaluate.
type_ (Any) –

Return type:

bool

Returns:

True if value is typing.Union and None is in its arguments, else False.

get_type(value)[source]¶

Attempt to retrieve the type of an object.

If the value is a generic typing alias, the origin type is returned. I.e. get_type(typing.List[str]) returns the builtins list type.

If the value is an instance of a class, the class is returned.

If the value is a class, it is directly returned.

>>> assert get_type([1, 2, 3]) == get_type(list)

Parameters:: value (Any) – Value to retrieve type from.
Return type:: Any
Returns:: The type.

get_optional_type(type_)[source]¶

Get the non-optional version of an optional type.

Parameters:

type – The optional type.
type_ (Any) –

Return type:

Any

Returns:

The type as non-optional.

get_type_name(value)[source]¶

Get a string representation for the type of a value.

Parameters:: value (Any) – A type or instance of a type to get a string representation for.
Return type:: str
Returns:: The string representation of the value.

bytes_to_base64(value)[source]¶

Encode bytes to a base64 string.

Follows the format used by protobuf.

Parameters:: value (bytes) – The bytes to encode.
Return type:: str
Returns:: The base64 string representing the bytes argument.

base64_to_bytes(value)[source]¶

Decode a base64 string to bytes.

Follows the format used by protobuf.

Parameters:: value (str) – The base64 string to decode.
Return type:: bytes
Returns:: The bytes representation of the base64 string argument.

primitive_to_bool(value)[source]¶

Convert a primitive value to a bool.

Follows the YAML convention instead of using __bool__ function of the object, see str_to_bool() for more info.

Raises ValueError on failure.

Parameters:: value (Union[int, float, bool, str]) – The primitive value to convert into bool.
Return type:: bool
Returns:: The bool representation of the value argument.

str_to_bool(value)[source]¶

Cast a string to a boolean.

True: 1 | [Tt][Rr][Uu][Ee] | [Yy] | [Yy][Ee][Ss] | [Oo][Nn]

False: 0 | [Ff][Aa][Ll][Ss][Ee] | [Nn] | [Nn][Oo] | [Oo][Ff][Ff]

Inspired by https://yaml.org/type/bool.html

Raises ValueError if the string does not conform to this format.

Parameters:: value (str) – String to cast.
Return type:: bool
Returns:: The input string parsed as a bool.

str_to_int(value)[source]¶

Cast a string to an int.

Compared to using int to cast the value this function allows values such as "0.0" by first parsing the value as a float before casting it to an int (unless it contains decimals).

Raises ValueError if the input string can not be parsed as an int.

Parameters:: value (str) – String input to parse as int.
Return type:: int
Returns:: The parsed int.

float_to_int(value)[source]¶

Cast a float to an int.

Raises ValueError if the input float is not an integer (contains decimal values). This differs from using int to cast a float since decimal values produce an error instead of being dropped.

Raises ValueError on failure.

Parameters:: value (float) – Float input to cast as int.
Return type:: int
Returns:: Integer of the input value.

cast_primitive(type_, value)[source]¶

Cast a primitive value to a specified primitive type.

Uses primitive_to_bool(), str_to_int() and float_to_int() when applicable.

Raises ValueError if unsuccessful.

Parameters:

primitive – The primitive type (int, float, bool or str).
value (Union[int, float, bool, str]) – A primitive value to cast.
type_ (Type[_P]) –

Return type:

TypeVar(_P, int, float, bool, str)

Returns:

The casted value.

cast_literal(literal, value)[source]¶

Attempts to cast a primtive value to the same type as a primitive literal and match it.

Raises ValueError if not able to match.

Parameters:

literal (TypeVar(_P, int, float, bool, str)) – The primitive literal to cast to.
value (Union[int, float, bool, str]) – The value to cast.

Return type:

TypeVar(_P, int, float, bool, str)

Returns:

The literal.

match_literal(literals, value)[source]¶

Attempts to cast and match a value to a set of primitive literals.

Raises ValueError if not able to match.

Parameters:

literals (Iterable[Union[int, float, bool, str]]) – The primitive literals to match.
value (Union[int, float, bool, str]) – The value to cast and match.

Return type:

Union[int, float, bool, str]

Returns:

The literal that was matched.

cast_none(value)[source]¶

Attempt to cast a value to None.

Allowed values are case insensitive strings "null" and "none" or a missing value.

Raises ValueError on failure.

Parameters:: value (Any) – The value to cast to None
Return type:: None
Returns:: The casted value.

extract_description(doc)[source]¶

Omit all but the main text in a docstring.

Attempts to omit things such as arguments and return value from a docstring.

Parameters:: doc (str) – Docstring.
Return type:: str
Returns:: The description if found, else the entire docstring.

get_description(cls)[source]¶

Get the docstring for a class.

An attempt is made to omit all but the main text in the docstring. This means that the return value should be missing things such as arguments and return value descriptions.

Parameters:: cls – Class to retrieve description from.
Return type:: Optional[str]
Returns:: The description if found, else None.

get_attribute_description(cls, name)[source]¶

Get the docstring for an attribute of a class.

Parameters:

cls – Class for which an attribute description should be fetched.
name (str) – The name of the attribute.

Return type:

Optional[str]

Returns:

The attribute description if found, else None.

as_dataclass(cls)[source]¶

Safely convert a class into a dataclass.

This function can be called multiple times on the same class without it raising an error the second time.

Parameters:: cls (Type[TypeVar(_T)]) – Class to convert into dataclass.
Return type:: Type[TypeVar(_T)]
Returns:: Dataclass.

get_closest(value, candidates)[source]¶

Get the closest value to some input from a set of candidates.

If no close value is found, None is returned.

Parameters:

value (str) – The input value.
candidates (Iterable[str]) – The values to evaluate closeness to.

Return type:

Optional[str]

Returns:

The closest candidate. None if no candidate is close.

get_closest_error_message(value, candidates, key='key')[source]¶

Get an error message for a key.

The error message can take two forms, the first being 'unknown "{key}"' and the second being 'unknown "{key}", did you mean "{closest}"?' which is only returned when a close match is found among the candidates.

Parameters:

value (str) – The input value.
candidates (Iterable[str]) – The values to evaluate closeness to.
key (str) – A name for the key value.

Return type:

str

Returns:

The error message.

inspect_sequence_type(sequence_type)[source]¶

Get the type of the content of a sequence type.

Expects a single content type. If no content type is found or there are multiple content types typing.Any is returned.

Parameters:

type – The type variable of a sequence.
sequence_type (Any) –

Return type:

Any

Returns:

The type for the content of the input sequence type.

>>> get_sequence_type(typing.List[int])
<class 'int'>
>>> get_sequence_type(typing.Tuple[str, ...])
<class 'str'>
>>> get_sequence_type(list)
typing.Any
>>> get_sequence_type(typing.Tuple[str, int, float])
typing.Any

inspect_mapping_type(mapping_type)[source]¶

Get the key and value type from a mapping type.

Returns typing.Any for types that could not be inferred.

Parameters:

type – The type variable of a mapping.
mapping_type (Any) –

Return type:

_MappingKeyValueTypesInfo

Returns:

The key and value type.

>>> get_sequence_type(typing.Dict[str, int])
(<class 'str'>, <class 'int'>)
>>> get_sequence_type(typing.Mapping[str, typing.List[int]]])
(<class 'str'>, typing.List[int])
>>> get_sequence_type(dict)
(typing.Any, typing.Any)

is_testing()[source]¶

Discover if the current runtime is for testing with Pytest.

Return type:: bool

get_terminal_width()[source]¶

Attempt to get the width of the current terminal.

The terminal size is preprocessed in a similar way to argparse.

Return type:: int

marsh.utils¶

`marsh.utils`¶