Core

A collection of dictionairy utilities.

Used to grow on the fly. Aims to provide general purpose abstact functionalities.

Note

All core functionalities are known to the toplevel dcttools model. Meaning they can be used like:

dcttools.depth(...)

`depth`	Find out a dictionairy's depth.
`kfltr`	Key Filter out any unwanted entries.
`kfrep`	Key Find and Replace dictionairy entries.
`kswap`	Key Swap, top and sublevel keys of a nested dict.
`flaggregate`	Flat Aggregate single level dicts and kwargs.
`naggregate`	Nested Aggregate, aggregate nested dicts of depth 1.
`maggregate`	Mixed Aggregate, aggregate non nested dicts, nested dicts and kwargs.
`to_dataframe`	Convert a mapping to a table controlling the number of columns.

dcttools.core.depth(dct)[source]

Find out a dictionairy’s depth.

Convention:

depth({}) = 0,
depth({1: 1}) = 1
depth({1: 1, 2: 1}) = 1
depth({1: {1: 1}}) = 2

Parameters: dct¶ (dict) – Dictionairy of which the depth is to be assessed.
Returns: Depth of depth.dct
Return type: int
Raises: TypeError – Raised when depth.dct is ot of instance dict.

Examples

>>> depth({})
0

>>> depth({1: 1})
1

>>> depth({1: 1, 2: 1})
1

>>> depth({1: {1: 1}})
2

dcttools.core.kfltr(dcts=(), fltr='', xcptns=(), **kwargs)[source]

Key Filter out any unwanted entries.

Return a new dictionairy containing only items with keys containing fltr or keys beeing listed in xcptns. (kfltr = abbrevation of key filter)

Parameters

dcts¶ (Container, default=()) – Container of Dictionaires of which the keys are to be filtered (dcts = dictionairies)
fltr¶ (str, default='') – String that is searched for in the dictionairy keys (fltr = abbrevation for filter)
xcptns¶ (Container, default=()) – Container of strings not to be frepped. Aka keys to be ignored. (xcptns = abbrevation for exceptions)
kwargs¶ – Key words . Of course you can always just add the kwargs to be frepped to the dcts iterable.

Returns

List of dicts and kwargs that have been frepped. If you only provide one dct and want it to be returned as only one use the tuple syntax:

returned_dict, = kfltr()

Return type

list

Examples

Generating a new dict containing only items with tweak_ in their keys:

>>> import pprint
>>> kwargs = {'t': {'cat1': 1, 'cat2': 2, 'cat3': 3},
...           'tweak_case': {'cat1': '1', 'cat2': '2', 'cat3': 1},
...           'tweak_txt': "See \'case\'"}
>>> pprint.pprint(*kfltr(dcts=[kwargs], fltr='tweak_'), width=70)
{'tweak_case': {'cat1': '1', 'cat2': '2', 'cat3': 1},
 'tweak_txt': "See 'case'"}

Add a kwarg and an exception for on the fly manipulation (using aggregate to gather them in 1 dict instead of 2):

>>> pprint.pprint(flaggregate(dcts=kfltr(
...     dcts=[kwargs], fltr='tweak', xcptns=['x'], x=10)))
{'tweak_case': {'cat1': '1', 'cat2': '2', 'cat3': 1},
 'tweak_txt': "See 'case'",
 'x': 10}

dcttools.core.kfrep(dcts=(), fnd='', rplc='', xcptns=(), **kwargs)[source]

Key Find and Replace dictionairy entries.

Find fnd in dictionairy keys and replace them with rplc. (kfrep = abbrevation of key find replace)

Parameters

dcts¶ (Container, default=()) – Iterable of dictionaires of which the keys are to be frepped. (dcts = abbrevation for dictionairies)
fnd¶ (str, default='') – String that is searched for in the dictionairy keys (fnd = abbrevation for find)
rplc¶ (str, default='') – String that the found string is replaced with. (Leaving it default just removes the found string) (rplc = abbrevation for replace)
xcptns¶ (Iterable, default=()) – Iterable of strings not to be frepped. Aka keys to be ignored. (xcptns = abbrevation for exceptions)
kwargs¶ – Key words to be frepped. Of course you can always just add the kwargs to be frepped to the dcts container.

Returns

List of dicts and kwargs that have been frepped. Original dicts are not altered!

Return type

list

Examples

Include kwargs into the dcts argument and remove the first_ prefices:

>>> dct = {'txt': 'hi', 's': 5, 'kwarg': 1}
>>> dflts = {'mst_hve': 'yes', 'first_kwarg': 0}
>>> kwargs = {'first_txt': 'hi there', 'first_mst_hve': 'no',
...           'first_kwarg': 2}
>>> dct, dflts, kwargs = kfrep(dcts=[dct, dflts, kwargs], fnd='first_')
>>> print(dct, dflts)
{'txt': 'hi', 's': 5, 'kwarg': 1} {'mst_hve': 'yes', 'kwarg': 0}
>>> print(kwargs)
{'txt': 'hi there', 'mst_hve': 'no', 'kwarg': 2}

Provide kwargs as kwargs and replace first_ prefices with second_:

>>> dct = {'txt': 'hi', 's': 5, 'kwarg': 1}
>>> dflts = {'mst_hve': 'yes', 'first_kwarg': 0}
>>> kwargs = {'first_txt': 'hi there', 'first_mst_hve': 'no',
...           'first_kwarg': 2}
>>> dct, dflts, kwargs = kfrep(
...     dcts=[dct, dflts], fnd='first_', rplc='second_',
...     xcptns='first_kwarg', **kwargs)
>>> print(dct, dflts)
{'txt': 'hi', 's': 5, 'kwarg': 1} {'mst_hve': 'yes', 'first_kwarg': 0}
>>> print(kwargs)
{'first_kwarg': 2, 'second_txt': 'hi there', 'second_mst_hve': 'no'}

Chaining filter and find and replace utility to only get kwargs previously containing a tweak_ prefix, deleting this prefix, sothe kwargs are ready to be passed to a third party API:

>>> kwargs = {'t': {'cat1': 1, 'cat2': 2, 'cat3': 3},
...           'tweak_case': {'cat1': '1', 'cat2': '2', 'cat3': 1},
...           'tweak_txt': "See \'case\'"}
>>> print(*kfrep(dcts=kfltr(dcts=[kwargs], fltr='tweak_'),
...              fnd='tweak_'))
{'case': {'cat1': '1', 'cat2': '2', 'cat3': 1}, 'txt': "See 'case'"}

dcttools.core.kswap(nstd_dct)[source]

Key Swap, top and sublevel keys of a nested dict.

Parameters: nstd_dct¶ (dict) – Nested dictionairy of depth 1 of which the top level keys and the nested keys are to be swapped. (nstd_dct= abbrevation for nested dictionairy)
Returns: Nested dictionairy with its top and sublevel keys swapped.
Return type: dict

Warning

Returns empty dictionairy on non-nested dictinairies!

Examples

>>> import pprint
>>> nd = {'tweak_case': {'cat1': '1', 'cat2': '2', 'cat3': 1}}
>>> pprint.pprint(kswap(nd))
{'cat1': {'tweak_case': '1'},
 'cat2': {'tweak_case': '2'},
 'cat3': {'tweak_case': 1}}

dcttools.core.flaggregate(dcts=(), **kwargs)[source]

Flat Aggregate single level dicts and kwargs.

Keys are overriden depending on the order the dicts are provided. Python’s last word policy is kept (i.e entries in kwargs will override the others)

Note

Yes, it does the same as unpacking dcts, and then unpacking everything in the order it was supplied. Though it brings the benefit of having a concise name, while creating a detailed log. Has the potential to declutter your code.

Parameters

dcts¶ (Container, default=()) – Container of dictionaires which are to be aggregated. The order in which they are provided is crucial. The dict coming last will potentially override every other dict. (dcts = dictionairies)
kwargs¶ – Key words to be aggregated. Of course you can always just add the kwargs to dcts. This is especially usefull if u want them to be potentially overriden.

Returns

Dictionairy containing the aggregated dicts and kwargs.

Return type

dict

Examples

>>> dct = {'txt': 'hi', 's': 5, 'kwarg': 1}
>>> dflts = {'mst_hve': 'yes', 'first_kwarg': 0}
>>> kwargs = {'first_txt': 'hi there', 'first_mst_hve': 'no',
...           'first_kwarg': 2}

Aggregate dct, dflts and kwargs, providing kwargs as kwargs:

>>> import pprint
>>> pprint.pprint(flaggregate([dct, dflts], **kwargs))
{'first_kwarg': 2,
 'first_mst_hve': 'no',
 'first_txt': 'hi there',
 'kwarg': 1,
 'mst_hve': 'yes',
 's': 5,
 'txt': 'hi'}

Aggregate dct, dflts, and kwargs, providing kwargs as part of dcts:

>>> pprint.pprint(flaggregate([dct, dflts, kwargs]))
{'first_kwarg': 2,
 'first_mst_hve': 'no',
 'first_txt': 'hi there',
 'kwarg': 1,
 'mst_hve': 'yes',
 's': 5,
 'txt': 'hi'}

Aggregate dct dflts and kwargs with prior filtering. Note how the dct provided last overrides the other keys:

>>> print(flaggregate(
...     dcts=kfrep(dcts=[dct, dflts], fnd='first_', **kwargs)))
{'txt': 'hi there', 's': 5, 'kwarg': 2, 'mst_hve': 'no'}

dcttools.core.naggregate(nstd_dcts=())[source]

Nested Aggregate, aggregate nested dicts of depth 1.

Keys are overriden depending on the order the dicts are provided. Python’s last word policy is kept (the last keyword will take precedence)

Parameters: nstd_dcts¶ (Container, default=()) – Container of nested dictionaires which are to be aggregated. The order in which they are provided is crucial. The dict coming last will potentially override every other. (nstd_dcts= abbrevation for nested dictionairies)
Returns: aggregated – Nested dictionairy containing all items present in nstd_dcts dicts.
Return type: dict

Examples

Aggregate two nested dicts. Note how the dct provided last overrides the other keys:

>>> import pprint
>>> nstd_dct = {'n1': {'txt': 'hi', 's': 5}, 'n3': {'s': 10}}
>>> nstd_dct_2 = {'n1': {'txt': 'hey', 'mst_hve': 'yes'}, 'n2': {'s': 2}}
>>> pprint.pprint(naggregate(nstd_dcts=[nstd_dct, nstd_dct_2]))
{'n1': {'mst_hve': 'yes', 's': 5, 'txt': 'hey'},
 'n2': {'s': 2},
 'n3': {'s': 10}}

Aggregate three nested dicts. Note how the dct provided last overrides the other keys:

>>> nstd_dct = {'n1': {'txt': 'hi', 's': 5}, 'n3': {'s': 10}}
>>> nstd_dct_2 = {'n1': {'txt': 'hey', 'mst_hve': 'yes'}, 'n2': {'s': 2}}
>>> nstd_dct_3 = {'n2': {'txt': 'there', }, 'n3': {'s': 3}, 'n4': {'s': 4}}
>>> pprint.pprint(naggregate(nstd_dcts=[nstd_dct, nstd_dct_2, nstd_dct_3]))
{'n1': {'mst_hve': 'yes', 's': 5, 'txt': 'hey'},
 'n2': {'s': 2, 'txt': 'there'},
 'n3': {'s': 3},
 'n4': {'s': 4}}

dcttools.core.maggregate(tlkys=(), dcts=(), nstd_dcts=(), **kwargs)[source]

Mixed Aggregate, aggregate non nested dicts, nested dicts and kwargs.

This function returns a nested dictionary having a key-value pair for every kwarg provided in dcts (and the kwargs already present in the respective top level entry) of each nstd_dct for each key listed in tlkys.

Hierarchy is as follows:dcts < nstd_dct < kwargs

Designed to be used when dynamically splitting kwargs into several kwargs stored in a nested dict to distribute them among sub function calls falling back on key-value pairs listed in the ordinary dicts.

A (potentially api provided) bunch of default kwargs listed in an ordinary dct will be used to populate (potentially api provided nested dictionairies) potentially tweaked by the enduser supplying custom kwargs, falling back on on the defaults if necessary.

(For a practical use case see the interfaces.visualie.nx module. Those functions are designed to be parameterized in an automated fashion to allow for complex behaviour utilizing a nested dict. They however enable the user to also tweak any number of parameters by providing nested or single layer kwargs.)

Parameters

tlkys¶ (Container, default=()) – Iterable of keys the algorithm should be applied to (tlkys = abbrevation of top level keys)
dcts¶ (Container, default=()) –
Container of (prefilled) dictionairies of default kwargs as in:
```
[{key1: value1, keyN: valueN, ...}, ...]
```
Will be overriden by nstd_dct. (dcts = abbrevation of dictionaires)
nstd_dcts¶ (Container, default=()) –
Container of (prefilled) nested dictionairies to be aggregated as in:
```
[{tlky1: {key1: value1}, tlky2: {key2: value2}, ...}, ...]
```
Key-value pairs will take precedence over respective pairs found in dcts while beeing overriden by respective pairs found in kwargs. (nstd_dicts = abbrevation of nested dictionairies)
kwargs¶ – Keyword arguments to aggregate. Key-value pairs will take precedence over respective pairs found in nstd_dcts.

Returns

A new nested dict containing the deep copies of all entries aggregated as in:

{tlkys: {nlkeys: nlvalues}}

(aggr_dict = abbrevation of aggregated dictionairy)

Return type

dict

Examples

Using a non nested dict as defaults to ensure each kwarg is present in a nested dict (supposedly) provided by an api:

>>> import pprint
>>> parameters = {'cat1': {'txt': 'hi', 's': 5},
...               'cat2': {'txt': 'hey', 's': 7}}
>>> dflts = {'type': 'default', 's': 3, 'first_step': 13,}
>>> pprint.pprint(maggregate(
...     tlkys=list(parameters.keys()), dcts=[dflts,],
...     nstd_dcts=[parameters, ]))
{'cat1': {'first_step': 13, 's': 5, 'txt': 'hi', 'type': 'default'},
 'cat2': {'first_step': 13, 's': 7, 'txt': 'hey', 'type': 'default'}}

Using tlkys for adding new nodes to nested dictionairy (supposedly) provided by an api while using kwargs to provide and update entries:

>>> parameters = {'cat1': {'txt': 'hi', 's': 5},
...               'cat2': {'txt': 'hey', 's': 7}}
>>> tlkys = ['cat1', 'cat2', 'cat3']
>>> kwargs = {'s': {'cat1': 1, 'cat2': 2, 'cat3': 3}}

>>> pprint.pprint(
...    maggregate(tlkys=tlkys, nstd_dcts=[parameters,], **kwargs))
{'cat1': {'s': 1, 'txt': 'hi'},
 'cat2': {'s': 2, 'txt': 'hey'},
 'cat3': {'s': 3}}

Not providing fall back defaults as flat dicts can lead to None parameters if kwargs do not provide entries for each top level key:

>>> parameters = {'cat1': {'txt': 'hi', 's': 5},
...               'cat2': {'txt': 'hey', 's': 7}}
>>> tlkys = ['cat1', 'cat2', 'cat3']
>>> kwargs = {'s': {'cat1': 1, 'cat2': 2, 'cat3': 3},
...           'txt': {'cat1': 'ovrrdn', 'cat2': 'overrdn'}}
>>> pprint.pprint(
...     maggregate(tlkys=tlkys, nstd_dcts=[parameters,], **kwargs))
{'cat1': {'s': 1, 'txt': 'ovrrdn'},
 'cat2': {'s': 2, 'txt': 'overrdn'},
 'cat3': {'s': 3, 'txt': None}}

Design Case:

Using dcts as defaults, nstd_dcts as (supposedly) api provided parameters tweaking them using kwargs prefixed by tweak_ provided by an upper layer function potentially containing hundreds of different kwargs:

Using a nested dict as (suppoedlyd) api provided parameters: (This wouild be outside the code you write and potentially be HUGE)

>>> api_returns = {'cat1': {'txt': 'hi', 's': 5},
...                'cat2': {'txt': 'hey', 's': 7}}

Using dcts as defaults (your own coding effort):

>>> dflts = {'s': 0}

Using a list of keys to expand the parameter set (your own coding effort):

>>> tlkys = ['cat1', 'cat2', 'cat3']

Manipulating certain top level keys and parameters using kwargs By using your own prefix, you can hard code a set of predefined behaviours unambiguously defined:

>>> kwargs = {'t': {'cat1': 1, 'cat2': 2, 'cat3': 3},
...           'tweak_case': {'cat1': '1', 'cat2': '2', 'cat3': 1},
...           'tweak_txt': "See \'case\'"}

Filter only the kwargs needed in this call and replace the prefix to match the api required keywords:

>>> kwargs, = kfrep(dcts=kfltr(dcts=[kwargs], fltr='tweak_'), fnd='tweak_')

Aggregate everything into a nested dict as the api expects, using your own defaults and alterations:

>>> pprint.pprint(maggregate(tlkys=tlkys, dcts=[dflts, ],
...                          nstd_dcts=[api_returns, ], **kwargs))
{'cat1': {'case': '1', 's': 5, 'txt': "See 'case'"},
 'cat2': {'case': '2', 's': 7, 'txt': "See 'case'"},
 'cat3': {'case': 1, 's': 0, 'txt': "See 'case'"}}

dcttools.core.to_dataframe(mapping, columns, fillvalue=None, index=None)[source]

Convert a mapping to a table controlling the number of columns.

Parameters

mapping¶ (Mapping) – Mapping to be converted into a pandas.DataFrame
columns¶ (int) – Number of key-value column pairs of the result table. (i.e. columns=2 results in a total of 4 columns, 2 representing the keys, the other 2 representing the values)
fillvalue¶ (str, None, default=None) –
If number of key-value pairs inside the mapping is not an integer multiple of columns the modulus amount of entries is filled using fillvalue.

(See also zip_longest())
index¶ (pandas.Index, default=None) –
Index used for createing the table.
Warning

Make sure number of index entries equals:
```
math.ceil(len(mapping.keys()) / columns))
```

Returns

DataFrame holding the data of mapping, using the number of columns stated.

Return type

pandas.DataFrame

Examples

Design Case (emulateing an empty string ('') with 'empty string' for verbosity):

>>> mapping = {
...     'flow_costs': 0,
...     'co2_emissions': 0,
...     'installed_capacity': 0,
...     'accumulated_min': None,
...     'accumulated_max': None}
>>> print(to_dataframe(mapping, columns=2, fillvalue='empty string'))
                  key value              key         value
0          flow_costs     0  accumulated_min          None
1       co2_emissions     0  accumulated_max          None
2  installed_capacity     0     empty string  empty string

Using default fill values and adding an Index:

>>> import pandas, math
>>> cols = 3
>>> print(to_dataframe(
...       mapping, columns=cols,
...       index=pandas.Index(
...           ['i' + str(i) for i in range(
...               math.ceil(len(mapping.keys())/cols))])))
              key value                 key value              key value
i0     flow_costs     0  installed_capacity   0.0  accumulated_max  None
i1  co2_emissions     0     accumulated_min   NaN             None  None