?iaFdZddlZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl Z ddl Z ddlZddlmZddlmZdde jj(fdZGddZGd d Zd Zdd Zd ZdZdZdZdZdZdZ dZ!dZ"ddZ#ddZ$dZ%dZ&ddZ'y)z~This library gathers utilities for hyperpyyaml loading Authors * Peter Plantinga 2020 * Aku Rouhe 2020 * Jianchen Li 2022 N)StringIO)RepresenterErrorTcjt|||}|jdttjd}|j d|d|j dt|j dt|j dt|j d td tjjj_d t jjjj_tj"|| }d tjjj_d t jjjj_|j%Dcgc]}|j'd s|}}|D]}||=|Scc}w)aThis function implements the HyperPyYAML syntax The purpose for this syntax is a compact, structured hyperparameter and function definition. This function implements a few extensions to the yaml syntax, listed below. **PyYAML complex tag shortcuts** Part of our clean structured hyperparameter interface is being able to specify python objects easily and cleanly. This is possible with native YAML using the following syntax: .. code-block:: yaml alignment_saver: !!python/object/new:speechbrain.data_io.TensorSaver kwargs: {save_dir: results/asr/ali} However, due to the extensive use within speechbrain yaml files, we have added a shortcut for this that has the following syntax: .. code-block:: yaml alignment_saver: !new:speechbrain.data_io.TensorSaver save_dir: results/asr/ali In this example, the alignment_saver will be an instance of the ``TensorSaver`` class, with ``'exp/asr/ali'`` passed to the ``__init__()`` method as a keyword argument. This is equivalent to: .. code-block:: python import speechbrain.data_io.data_io alignment_saver = speechbrain.data_io.TensorSaver( save_dir='exp/asr/ali' ) We have also implemented a few more shortcuts::: !!python/name: => !name: !!python/module: => !module: !!python/object/apply: => !apply: **References and copies** Allows internal references to any node in the file. Any node with tag ``!ref`` will create an object reference to the yaml object at the ```` location within the yaml itself, following reference chains. .. code-block:: yaml output_folder: results/asr alignment_saver: !new:speechbrain.data_io.TensorSaver save_dir: !ref Strings values are handled specially: references are substituted but the rest of the string is left in place, allowing filepaths to be easily extended: .. code-block:: yaml output_folder: results/asr alignment_saver: !new:speechbrain.data_io.TensorSaver save_dir: !ref /ali # results/asr/ali A more complex example for demonstration purposes: .. code-block:: yaml key1: {a: !new:object {arg1: 1}} key2: !ref Here, ``key2`` will contain a reference to the ``a`` object, so changing ``a.arg1`` will also change ``key2.arg1``. If you need a deep copy of the object instead of a shallow reference, you can use a similar syntax with the tag ``!copy``. For example: .. code-block:: yaml key1: {a: !new:object {arg1: 1}} key2: !copy These will also implement very basic arithmetic, so: .. code-block:: yaml key1: 1 key2: !ref + 3 # this is 4 **Tuples** One last minor enhancement is an implicit tuple resolver. Passing a string value of ``(3, 4)`` will be given a tag of ``!tuple`` which is then interpreted as a tuple. Arguments --------- yaml_stream : stream A file-like object or string from which to read. overrides : mapping or str A set of overrides for the values read from the stream. As yaml implements a nested structure, so can the overrides. See `speechbrain.utils.data_utils.recursive_update` overrides_must_match : bool Whether an error will be thrown when an override does not match a corresponding key in the yaml_stream. return_dict : bool Whether to return a dictionary rather than the default namespace. loader : Loader Use to parse the yaml_stream, could be `ruamel.yaml.Loader`, `yaml.Loader`, etc. Returns ------- hparams : dict Reflects the structure of ``yaml_stream``. Example ------- >>> yaml_string = """ ... a: 3 ... thing: !new:collections.Counter ... b: !ref ... """ >>> params = load_hyperpyyaml(yaml_string) >>> params["thing"] Counter({'b': 3}) z!tuple)tag constructorz^\(.*\)$()first!new:!name:!module:!apply:)TLoaderF__)resolve_referencesadd_constructor _make_tuplerecompileadd_implicit_resolveradd_multi_constructor_construct_object_construct_name_construct_module_apply_functionyamlrBaseConstructorconstruct_object __defaults__ruamelloadkeys startswith) yaml_stream overridesoverrides_must_matchloader tuple_patternhparamsk removal_keyskeys e/Volumes/fast/ai/experiments/voice-extract-mac/.venv/lib/python3.12/site-packages/hyperpyyaml/core.pyload_hyperpyyamlr/scF%[)=QRK x[AJJ{+M   = D   *;<   ?;   ->?   O<FD$$55BMFKK++<<Iii F3GFD$$55BMFKK++<<I '||~D~!d1CA~LD CL N Es F0F0c*eZdZdZdZdZedZy)RefTagzClass for dumping !ref tags to yaml Arguments --------- ref_str : str String including yaml keys in `` notation Example ------- See ``dump_hyperpyyaml`` !refc||_yN)ref_str)selfr5s r.__init__zRefTag.__init__s  cN|j|j|jSr4)represent_scalaryaml_tagr5cls representernodes r.to_yamlzRefTag.to_yamls++CLL$,,GGr8N)__name__ __module__ __qualname____doc__r;r7 classmethodr@r8r.r1r1s) HHHr8r1c$eZdZdZdZedZy) PlaceholderzfClass for dumping !PLACEHOLDER tags to yaml Example ------- See ``dump_hyperpyyaml`` !PLACEHOLDERc:|j|jdS)N)r:r;r<s r.r@zPlaceholder.to_yamls++CLL"==r8N)rArBrCrDr;rEr@rFr8r.rHrHs H>>r8rHc&tjj}|jj t t j |jj ttj |j||g|i|y)aDump yaml including placeholder and reference tags. Arguments --------- yaml_tree : dict An object to dump output_stream : stream A file stream for putting the yaml *args, **kwargs Arguments to forward to ruamel.yaml.YAML().dump() Example ------- >>> to_yaml = {'a': Placeholder(), 'b': RefTag('')} >>> stringio = StringIO() >>> dump_hyperpyyaml(to_yaml, stringio) >>> stringio.getvalue() 'a: !PLACEHOLDER\nb: !ref \n' N) r!rYAMLr>add_representerr1r@rHdump) yaml_tree output_streamargskwargs ruamel_yamls r.dump_hyperpyyamlrUsh(++""$K++FFNNC++K9L9LMKY ???r8cd}t|drFtjjtjj |j }t jj}|j|}|>t|dkDr0t|tr|j|} t|||t!d|||t#} |j%|||j'd|S#t$r td|wxYw#t($r8}|j*ddzf|_|j-|j.d}~wwxYw)aResolves inter-document references, a component of HyperPyYAML. Arguments --------- yaml_stream : stream A file-like object or string with the contents of a yaml file written with the HyperPyYAML syntax. overrides : mapping or str Replacement values, either in a yaml-formatted string or a dict. overrides_must_match : bool Whether an error will be thrown when an override does not match a corresponding key in the yaml_stream. This is the opposite default from ``load_hyperpyyaml`` because ``resolve_references`` doesn't need to be as strict by default. Returns ------- stream A yaml-formatted stream with all references and overrides resolved. Example ------- >>> yaml_string = """ ... constants: ... a: 3 ... b: !ref ... """ >>> overrides = {'constants': {'a': 4}} >>> resolve_references(yaml_string, overrides).getvalue() 'constants:\n a: 4\n b: 4\n' Nnamer) must_matchzLThe structure of the overrides doesn't match the structure of the document: rootz$. Please use the !apply tag instead.)hasattrospathdirnamerealpathrWr!rrMr"len isinstancestrrecursive_update TypeError ValueError_walk_tree_and_resolverrOseekrrRwith_traceback __traceback__)r%r&r' file_pathrTpreviewes r.rrsDBI{F#GGOOBGG$4$4[5E5E$FG ++""$K{+GY!!3 i %#((3I  Wi

>> deref('constants[a][b]', {'constants': {'a': {'b': 'c'}}}) 'c' N.r)maxsplitrmrnzThe reference "%s" is not validz!apply:getattr)suffix) splitstriprdcopydeepcopyr!rcomments CommentedSeq yaml_set_ctagTag)refrsrtattrbranchpartr?s r.derefr8s0 D czIIcAI. TF #zz# v >DE E }}V$$ {{##002   KKOO#3O 4  Mr8ctjd}t|tr|j |s|St |dkDr||ddvr t d||j|r0t|jd||}||gz }t||||S|j|}||Dcgc]}|d c}z }||fd}|j||} t| |||}t|Scc}w)aResolve a reference to a value, following chained references Arguments --------- reference : str a string containing '' in it where x[y] refers to a scalar node in the file. reference_list : list list of prior references in the chain, in order to catch circular references. full_tree : dict the dictionary in which to find all references and their values. copy_mode : bool Whether to perform a deep copy of the referenced node, rather than a shallow reference to the same object. Returns ------- scalar The dereferenced value, with possible string interpolation and arithmetic parsing. Example ------- >>> tree = {'a': 3, 'b': 'x', 'c': '', 'd': '/', 'e': '/'} >>> recursive_resolve('', [], tree) 1.0 >>> recursive_resolve('', [], tree) 'x/x' z<[^>]*>rNzCircular reference detected: <>rcTtt|djd||S)Nrr)rsrt)rarr)xrrts r. replace_fnz%recursive_resolve..replace_fns#51D)TYOPPr8)rrr`rasearchr_rd fullmatchrrr{findallsubparse_arithmetic) rqrrrsrtreference_finderrzmatchesmatchrrs r.r{r{lsBzz*- i %-=-D-DY-O >Q9qr0B#B8.II!!),iood+Y B9+%  9MM&&y1GW5WEuQxW55N% Q   z9 5C!#~y)LI I &&6s1 C3c ttj|djS#tt t f$r|cYSwxYw)aParses simple arithmetic operations in references Adapted from https://stackoverflow.com/a/9558001/1761970 Arguments --------- reference_string : str A string with references and possible arithmetic operations. Returns ------- str Result of parsing and applying the arithmetic. Example ------- >>> parse_arithmetic('2 * 6') 12 eval)mode) _ast_evalastparsebodyrc SyntaxErrorKeyError)reference_strings r.rrs@( #3&AFFGG {H -  s),AActjtjtjtj tj tjtjtjtjtjtjtjtjtji}t j"dk\r&t%|tj&r2|j(St%|tj*r |j,St%|tj.rE|t1|jt3|j4t3|j6St%|tj8r1|t1|jt3|j:St=|)N))rAddopaddSubrMultmulDivtruedivFloorDivfloordivPowpowModmodsys version_infor`ConstantrzNumnBinOptyperleftrightUnaryOpoperandrc)r?opss r.rrs   "&&  bkk   C 6! dCLL )::  dCGG $66M$ "!s4=!)DII"6 $**8MNN D#++ &!s4=!)DLL"9::or8c t|tjjst d|t|tjjst d||j D]\}}t|tjjr!||vrt |j|i|K|r1||vr-td|d|jDcgc]}|c}|||<ycc}w)aSimilar function to `dict.update`, but for a nested `dict`. From: https://stackoverflow.com/a/3233356 If you have to a nested mapping structure, for example: {"a": 1, "b": {"c": 2}} Say you want to update the above structure with: {"b": {"d": 3}} This function will produce: {"a": 1, "b": {"c": 2, "d": 3}} Instead of: {"a": 1, "b": {"d": 3}} Arguments --------- d : dict mapping to be updated u : dict mapping to update with must_match : bool Whether to throw an error if the key in `u` does not exist in `d`. Example ------- >>> d = {'a': 1, 'b': {'c': 2}} >>> recursive_update(d, {'b': {'d': 3}}) >>> d {'a': 1, 'b': {'c': 2, 'd': 3}} z'Expected to update a mapping, but got: z.Expected a mapping to use for update, but got z Override 'z' not found in: N) r` collectionsabcMappingrcryrbgetrr#)durXr+vr-s r.rbrbsN a00 1A!EFF a00 1HLMM 1 a00 1a1f QUU1b\1 - AQJZs*:1668;T8CC8;T:UVW WAaD rs 4  d6;;CUCUpfHH. > >@4CLOd  0&2 &*1h<'~ 4:4r8