dimcat package#

Subpackages#

Submodules#

dimcat.base module#

The dimcat.base module defines the three principal classes: DimcatSchema, DimcatObject, and DimcatConfig as well as their interactions.

dimcat.base.CONTROL_REGISTRY = False#: Raise an error if a subclass has the same name as another subclass. Set True for production.

class dimcat.base.DimcatConfig(options: Union[Dict, DimcatConfig, str] = (), dtype: Optional[str] = None, **kwargs)[source]#

Bases: MutableMapping, DimcatObject

Behaves like a dictionary but accepts only keys and values that are valid under the Schema of the DimcatObject specified under the key ‘dtype’. Every DimcatConfig needs to have a ‘dtype’ key that is the name of a DimcatObject and can specify zero or more additional key-value pairs that can be used to instantiate the described object.

When dealing with a DimcatConfig you need to be aware that the ‘dtype’ and ‘options’ can been different things when used as keys as opposed to attributes (DC represents a DimcatConfig):

DC['dtype'] is the name of the described DimcatObject (equivalent to DC.options_dtype)
DC.dtype returns the class name “DimcatConfig”, according to all DimcatObjects’ default behaviour
DC.options (equivalent to dict(DC) returns the key-value pairs wrapped by this config, which includes at least the ‘dtype’ key
DC['options'] is the value of the ‘options’ option, which exists only if it is part of the described object’s schema, for example if the described object is a DimcatConfig itself.

Examples

>>> from dimcat.base import DimcatConfig, DimcatObject
>>> DC = DimcatConfig(dtype="DimcatObject")
>>> DC.dtype
'DimcatConfig'
>>> DC['dtype']
'DimcatObject'
>>> DC.options
{'dtype': 'DimcatObject'}
>>> DC['options']
KeyError: 'options'
>>> config_config = DC.to_config()
>>> config_config.options
{'dtype': 'DimcatConfig', 'options': {'dtype': 'DimcatObject'}}
>>> config_config['options']
{'dtype': 'DimcatObject'}

class Schema(*, only: Optional[Union[Sequence[str], AbstractSet[str]]] = None, exclude: Union[Sequence[str], AbstractSet[str]] = (), many: Optional[bool] = None, load_only: Union[Sequence[str], AbstractSet[str]] = (), dump_only: Union[Sequence[str], AbstractSet[str]] = (), partial: Optional[Union[bool, Sequence[str], AbstractSet[str]]] = None, unknown: Optional[Literal['exclude', 'include', 'raise']] = None)[source]#

Bases: DimcatSchema

init_object(data, **kwargs) → DimcatConfig[source]#: Once the data has been loaded, create the corresponding object.

opts: Any = <marshmallow.schema.SchemaOpts object>#

serialize_if_necessary(data, **kwargs)[source]#

validate_dump(data, **kwargs)[source]#: Make sure to never return invalid serialization data.

complete() → Self[source]#: Returns a new Config with missing options filled in with the default values.

create() → DimcatObject[source]#: Creates the object that this DimcatConfig represents. The returned object corresponds to the type options_dtype and will be instantiated with options.

classmethod from_dict(options, **kwargs) → Self[source]#: Creates a new object from a config-like dict. Concretely, the received options will be updated with the **kwargs and enriched with a ‘dtype’ key corresponding to this object, before deserializing the dict using the corresponding marshmallow schema.

classmethod from_object(obj: DimcatObject)[source]#

property init_args: dict#

matches(config: DimcatConfig, covariant: bool = False, contravariant: bool = False) → bool[source]#

Returns True if both configs have the same options_dtype (optionally allowing subclasses) and the overlapping options are equal.

Parameters:

config – The other config to compare against.
covariant – If True, the other config’s dtype matches even if it is a superclass of this config’s dtype. In other words, I describe an object which is of type config.dtype or a subclass of it.
contravariant – If True, the other config’s dtype matches even if it is a subclass of this config’s dtype. In other words, I describe an object which is of type config.dtype or a superclass of it.

Returns:

Returns True if both configs have the same options_dtype (optionally allowing subclasses) and the overlapping options are equal.

property options: dict#: Returns the options dictionary wrapped and controlled by this DimcatConfig. Whenever a new value is set, it is validated against the Schema of the DimcatObject specified under the key ‘dtype’. Note that this property returns a copy of the dictionary including the ‘dtype’ key and modifying it will not affect the DimcatConfig. Also note that the returned value is different from DimcatConfig[“options”]

property options_class: Type#: The class of the described DimcatObject.

property options_dtype: str#: The dtype (i.e. class name) of the described DimcatObject.

property options_schema: DimcatSchema#: Returns the (instantiated) Dimcat singleton object for the class this Config describes.

summary_dict() → dict[source]#: Returns a summary of the object.

validate(partial=False) → Dict[str, List[str]][source]#: Validates the current status of the config in terms of ability to create an object. Empty dict == valid.

class dimcat.base.DimcatObject[source]#

Bases: ABC

All DiMCAT classes derive from DimcatObject, except for the nested Schema(DimcatSchema) class that they define or inherit.

class PickleSchema[source]#: Bases: object

Bases: DimcatSchema

init_object(data, **kwargs) → DimcatObject[source]#: Once the data has been loaded, create the corresponding object.

opts: Any = <marshmallow.schema.SchemaOpts object>#

class property dtype: str | enum.Enum#: Name of the class as enum member (if cls._enum_type is define, string otherwise).

filename_factory()[source]#

classmethod from_config(config: DimcatConfig, **kwargs) → Self[source]#: Creates a new object from a DimcatConfig. Concretely, the config’s options will be updated with the **kwargs and the ‘dtype’ key will be replaced according to this object, before deserializing the dict using the corresponding marshmallow schema.

classmethod from_dict(options, **kwargs) → Self[source]#: Creates a new object from a config-like dict. Concretely, the received options will be updated with the **kwargs and enriched with a ‘dtype’ key corresponding to this object, before deserializing the dict using the corresponding marshmallow schema.

classmethod from_json(config: str, **kwargs) → Self[source]#

classmethod from_json_file(filepath: str) → Self[source]#

info(return_str: Literal[False]) → None[source]#
info(return_str: Literal[True]) → str: Returns a summary of the dataset.

class property logger: Logger#: Instances of the Logger class represent a single logging channel. A “logging channel” indicates an area of an application. Exactly how an “area” is defined is up to the application developer. Since an application can have any number of areas, logging channels are identified by a unique string. Application areas can be nested (e.g. an area of “input processing” might include sub-areas “read CSV files”, “read XLS files” and “read Gnumeric files”). To cater for this natural nesting, channel names are organized into a namespace hierarchy where levels are separated by periods, much like the Java or Python package namespace. So in the instance given above, channel names might be “input” for the upper level, and “input.csv”, “input.xls” and “input.gnu” for the sub-levels. There is no arbitrary limit to the depth of nesting.

class property name: str#

str(object=’’) -> str str(bytes_or_buffer[, encoding[, errors]]) -> str

Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.__str__() (if defined) or repr(object). encoding defaults to sys.getdefaultencoding(). errors defaults to ‘strict’.

class property schema[source]#: Returns the (instantiated) DimcatSchema singleton object for this class.

summary_dict() → dict[source]#: Returns a summary of the object.

to_config() → DimcatConfig[source]#

to_dict() → dict[source]#

to_json() → str[source]#

to_json_file(filepath: str, indent: int = 2, **kwargs)[source]#

Serialize object to JSON file.

Parameters:

filepath – Path to the text file to (over)write.
indent – Prettify the JSON layout. Default indentation: 2 spaces
**kwargs – Keyword arguments passed to json.dumps().

to_options()[source]#

class dimcat.base.DimcatObjectField(*, load_default: ~typing.Any = <marshmallow.missing>, dump_default: ~typing.Any = <marshmallow.missing>, data_key: ~typing.Optional[str] = None, attribute: ~typing.Optional[str] = None, validate: ~typing.Optional[~typing.Union[~typing.Callable[[~typing.Any], ~typing.Any], ~typing.Iterable[~typing.Callable[[~typing.Any], ~typing.Any]]]] = None, required: bool = False, allow_none: ~typing.Optional[bool] = None, load_only: bool = False, dump_only: bool = False, error_messages: ~typing.Optional[dict[str, str]] = None, metadata: ~typing.Optional[~typing.Mapping[str, ~typing.Any]] = None)[source]#

Bases: Field

Used for (de)serializing attributes resolving to DimcatObjects.

class dimcat.base.DimcatSchema(*, only: Optional[Union[Sequence[str], AbstractSet[str]]] = None, exclude: Union[Sequence[str], AbstractSet[str]] = (), many: Optional[bool] = None, load_only: Union[Sequence[str], AbstractSet[str]] = (), dump_only: Union[Sequence[str], AbstractSet[str]] = (), partial: Optional[Union[bool, Sequence[str], AbstractSet[str]]] = None, unknown: Optional[Literal['exclude', 'include', 'raise']] = None)[source]#

Bases: Schema

The base class of all Schema() classes that are defined or inherited as nested classes for all DimcatObjects. This class holds the logic for serializing/deserializing DiMCAT objects. However, nested Schema() classes should generally not inherit directly from DimcatSchema but instead from DimcatObject.Schema because it defines the post_load hook init_object() for deserializing Dimcat objects. The parent class DimcatSchema does not define it, allowing DimcatConfig to define it differently. In marshmallow, hooks are additive and do not replace hooks of parent schemas.

Overall, this requires careful planning at what point in the object hierarchy the hooks are introduced in the corresponding nested Schema classes: Hooks of parent Schemas can be called via super() but they cannot be not called by omitting super(). For example, the post_dump hook validate_dump() is introduced in PipelineStep.Schema to automatically validate any serialized object right away (frictionless does that basically by trying if it can load the serialization data). For Data.Schema, however, this is not a safe default because most Data objects can be successfully validate only once their data has been stored to disk. Therefore, the post_dump hook is introduced in a second type of schema that all Data objects have, called PickleSchema.

The arbitrary metadata of the fields currently use the keys:

expose: Set False to mark fields that would normally not be exposed to the users in the context of a GUI.
Defaults to True if missing.
title: A human-readable title for the field.
description: A human-readable description for the field.

class Meta[source]#

Bases: object

ordered = True#

assert_type(obj, **kwargs)[source]#

If this fails, typically, a property of a DimcatObject does not have the type expected by the relevant field. To find out which one, you can use

for attr_name, field_obj in schema.dump_fields.items():
    value = field_obj.serialize(attr_name, obj)

(copied from marshmallow.schema._serialize()) where obj is the object to be serialized by schema.

dtype#: This field specifies the class of the serialized object. Every DimcatObject comes with the corresponding class property that returns its name as a string (or en Enum member that can function as a string). It is inherited by all objects’ schemas and enables their deserialization from a DimcatConfig.

get_attribute(obj: Any, attr: str, default: Any)[source]#: Defines how to pull values from an object to serialize.

Changed in version 3.0.0a1: Changed position of obj and attr.

class property name: str#: Qualified name of the schema, meaning it includes the name of the class that it is nested in.

opts: Any = <marshmallow.schema.SchemaOpts object>#

class dimcat.base.DimcatSettings(auto_make_dirs: bool = True, context_columns: ~typing.List[str] = <factory>, default_basepath: str = '~/dimcat_data', default_figure_path: str = '~/dimcat_data', default_figure_format: str = '.png', default_figure_width: int = 2880, default_figure_height: int = 1620, default_resource_name: str = 'unnamed', default_terminal_symbol: str = '⋉', never_store_unvalidated_data: bool = True, recognized_piece_columns: ~typing.List[str] = <factory>, package_descriptor_endings: ~typing.List[str] = <factory>, resource_descriptor_endings: ~typing.List[str] = <factory>)[source]#

Bases: DimcatObject

This is a dataclass that stores the default settings for the dimcat library. The current settings can be loaded anywhere in the code by calling get_setting(). Defining a new setting means adding it in three places:

as a class attribute with type annotation (=dataclass field) and default (factory where needed)
as a marshmallow field in the Schema with the same name and corresponding type, which gives access to marshmallow’s full serialization and validation functionality. By default, we add required=True to all settings.
to the file settings.ini, using Python’s config file syntax

Bases: Schema

dump_fields: dict[str, Field]#

exclude: set[Any] | MutableSet[Any]#

fields: dict[str, Field]#: Dictionary mapping field_names -> Field objects

load_fields: dict[str, Field]#

opts: Any = <marshmallow.schema.SchemaOpts object>#

unknown: types.UnknownOption#

auto_make_dirs: bool = True#

context_columns: List[str]#: the columns that are considered essential for locating elements horizontally and vertically and which are therefore always copied, if present, and moved to the left of the new dataframe in the order given here

default_basepath: str = '~/dimcat_data'#: where to serialize data if no other basepath is specified

default_figure_format: str = '.png'#: default format for all figures stored by DiMCAT.

default_figure_height: int = 1620#: default height in pixels for figures stored by DiMCAT

default_figure_path: str = '~/dimcat_data'#: where to store figures if no other path was specified

default_figure_width: int = 2880#: default width in pixels for figures stored by DiMCAT

default_resource_name: str = 'unnamed'#: default name for resources created from scratch

default_terminal_symbol: str = '⋉'#: default symbol to be used for the end of sequences

never_store_unvalidated_data: bool = True#: setting this to False allows for skipping mandatory validations; set to True for production

package_descriptor_endings: List[str]#

recognized_piece_columns: List[str]#: column names that are recognized as piece identifiers and automatically renamed to ‘piece’ when needed

resource_descriptor_endings: List[str]#: file endings that are recognized as frictionless resource descriptors

class dimcat.base.DtypeField(*, load_default: ~typing.Any = <marshmallow.missing>, dump_default: ~typing.Any = <marshmallow.missing>, data_key: ~typing.Optional[str] = None, attribute: ~typing.Optional[str] = None, validate: ~typing.Optional[~typing.Union[~typing.Callable[[~typing.Any], ~typing.Any], ~typing.Iterable[~typing.Callable[[~typing.Any], ~typing.Any]]]] = None, required: bool = False, allow_none: ~typing.Optional[bool] = None, load_only: bool = False, dump_only: bool = False, error_messages: ~typing.Optional[dict[str, str]] = None, metadata: ~typing.Optional[~typing.Mapping[str, ~typing.Any]] = None)[source]#: Bases: Field

class dimcat.base.FriendlyEnum(value)[source]#

Bases: LowercaseEnum

Like LowercaseEnum, Members of this Enum can be created from and compared to strings in a case-insensitive manner. In addition, this type of Enum is friendly enough to also allow for shortened values (i.e. having only the first few letters), as long as the abbreviation is unambiguous.

class dimcat.base.FriendlyEnumField(enum: type[enum.Enum], *, by_value: bool = True, **kwargs)[source]#

Bases: Enum

This fields is identical with the standard Marshmallow Enum field except for the fact that enum members are created based on enum values instead of enum names. This incorporates the benefits of the FriendlyEnum, i.e. case-insensitive creation from partial strings.

class dimcat.base.ListOfStringsField(cls_or_instance=<fields.String(dump_default=<marshmallow.missing>, attribute=None, validate=None, required=False, load_only=False, dump_only=False, load_default=<marshmallow.missing>, allow_none=False, error_messages={'required': 'Missing data for required field.', 'null': 'Field may not be null.', 'validator_failed': 'Invalid value.', 'invalid': 'Not a valid string.', 'invalid_utf8': 'Not a valid utf-8 string.'})>, **kwargs)[source]#: Bases: List

class dimcat.base.LowercaseEnum(value)[source]#

Bases: str, Enum

Members of this Enum can be created from and compared to strings in a case-insensitive manner.

class dimcat.base.ObjectEnum(value)[source]#

Bases: FriendlyEnum

An enumeration.

get_class() → Type[DimcatObject][source]#

dimcat.base.change_setting(key: str, value: Any) → None[source]#

dimcat.base.deserialize_config(config: DimcatConfig) → DimcatObject[source]#: Deserialize a config object into a DimcatObject.

dimcat.base.deserialize_dict(obj_data: dict) → DimcatObject[source]#: Deserialize a dict into a DimcatObject.

dimcat.base.deserialize_json_file(json_file: pathlib.Path | str) → DimcatObject[source]#: Deserialize a JSON file into a DimcatObject.

dimcat.base.deserialize_json_str(json_data: str) → DimcatObject[source]#: Deserialize a JSON string into a DimcatObject.

dimcat.base.get_class(name: str) → Type[DO][source]#: Resolve the given name to the class of the corresponding DimcatObject.

dimcat.base.get_pickle_schema(name, init=True)[source]#: Caches the intialized schema for each class. Pass init=False to retrieve the schema constructor.

dimcat.base.get_schema(name, init=True)[source]#: Caches the intialized schema for each class. Pass init=False to retrieve the schema constructor.

dimcat.base.get_setting(key: str) → Any[source]#

dimcat.base.is_default_descriptor_path(filepath: str) → bool[source]#

dimcat.base.is_instance_of(obj, class_or_tuple: Union[Type, str, Tuple[Union[Type, str], ...]])[source]#: Returns True if the given object is an instance of the given class or one of the given classes.

dimcat.base.is_name_of_dimcat_class(name: str) → bool[source]#: Returns True if the given name can be resolved to the name of a DimcatObject.

dimcat.base.is_subclass_of(name: str, parent: Union[str, Type[DimcatObject]]) → bool[source]#: Returns True if the DimcatObject with the given name is a subclass of the given parent.

dimcat.base.load_settings(config_filepath: Optional[str] = None, raise_exception: bool = False) → DimcatConfig[source]#: Get the DimcatSettings object.

dimcat.base.make_config_from_specs(specs: DO, instance_of: Optional[Union[Type[DO], str]] = None) → DimcatConfig[source]#

Returns a DimcatConfig corresponding to the given specs. If a DimcatConfig with dtype ‘DimcatConfig’ is received, the described (inner) DimcatConfig is returned.

Raises:: TypeError – If the feature cannot be converted to a dimcat configuration.

dimcat.base.make_default_settings() → DimcatConfig[source]#: Make a DimcatConfig object representing DimcatSettings with default values.

dimcat.base.make_object_from_specs(specs: Union[DO, Type[DO], DimcatConfig, MutableMapping, ObjectEnum, str], instance_of: Optional[Union[Type[DO], str]] = None) → DO[source]#: Returns the DimcatObject corresponding to the given specs.

dimcat.base.make_settings_from_config_file(config_filepath: str, fallback_to_default: bool = True) → DimcatConfig[source]#: Make a DimcatSettings object from a config file.

dimcat.base.make_settings_from_config_parser(config: ConfigParser) → DimcatConfig[source]#: Make a DimcatSettings object from a ConfigParser object.

dimcat.base.parse_config_file(config_filepath: str) → ConfigParser[source]#: Parse a config file and return a ConfigParser object.

dimcat.base.reset_settings(config_filepath: Optional[str] = None) → None[source]#: Reset the DiMCAT settings to the default or to those found in the settings.ini file at the given path.

dimcat.cli module#

dimcat.dc_exceptions module#

exception dimcat.dc_exceptions.BaseFilePathMismatchError(*args, message: Optional[str] = None, **kwargs)[source]#

dimcat package#

Subpackages#

Submodules#

dimcat.base module#

dimcat.cli module#

dimcat.dc_exceptions module#

dimcat.dc_warnings module#

dimcat.enums module#

dimcat.plotting module#

dimcat.utils module#

Module contents#