)tizdZddlZddlZddlmZddlmZmZddlm Z m Z ddl m Z ddl Z ddlmZmZd d lmZd d lmZmZd d lmZmZmZd d lmZer ddlmcmZerddl m!Z!dZ"dZ#dZ$dZ%dZ&e"dddZ'dEdZ(dZ)dZ*dZ+dZ,dZ-dZ.dZ/dZ0Gd d!e1Z2d"Z3d#Z4e3d$Z5d%e fd&Z6d%e fd'Z7dFd(Z8dGd)Z9e jtd e jvde jxd*e jzd+e j|d,e j~d-e jd.e jd/e jd0e jd1i ZDeDjDcic]\}}|| c}}ZFd2ZGdHd3e jfd4ZIe3dFd5eJfd6ZKdFd5eJfd7ZLdId8ZMdFd9ZNGd:d;eOZPe4dJd<ZQdFd=ZRe3dKd>ZSd?ZTGd@dAZUdBZVdCZWedLdDZXycc}}w)MzB A set of basic tensor ops compatible with tpu, gpu, and multigpu N)Mapping)contextmanager nullcontext)update_wrapperwraps)Any)AcceleratorState PartialState)!TORCH_DISTRIBUTED_OPERATION_TYPES)DistributedTypeTensorInformation)is_npu_availableis_torch_distributed_availableis_torch_xla_available)is_torch_version)ReduceOpc6t|tjSN) isinstancetorchTensortensors j/Volumes/fast/ai/experiments/MLX_z-image/.venv/lib/python3.12/site-packages/accelerate/utils/operations.pyis_torch_tensorr-s fell ++c vt|tjjtjjtjj tjj tjjtjjtjjSr) rrxpu FloatTensor ByteTensor IntTensor LongTensor HalfTensor DoubleTensorBFloat16Tensorrs ris_torch_xpu_tensorr(1sm            rc"t|tSr)rr tensor_infos ris_tensor_informationr,>s k#4 55rcZt|txrt|dxr t|dS)z Checks if `data` is a `namedtuple` or not. Can have false positives, but only if a user is trying to mimic a `namedtuple` perfectly. _asdict_fields)rtuplehasattrdatas r is_namedtupler4Bs* dE " \wtY'? \GDR[D\\rcht|rt|t|St||S)zO Cast a generator to the same type as obj (list, tuple, or namedtuple) )r4typelist)obj generators r honor_typer:Js2 StCy$y/**tCy##rF test_typeerror_on_other_typec t|ttfrt|fd|DSt|trDt ||j Dcic]\}}|t|gdc}}S|r |giSr2tdt |djdjd|Scc}}w)ad Recursively apply a function on a data structure that is a nested list/tuple/dictionary of a given base type. Args: func (`callable`): The function to recursively apply. data (nested list/tuple/dictionary of `main_type`): The data on which to apply `func` *args: Positional arguments that will be passed to `func` when applied on the unpacked data. main_type (`type`, *optional*, defaults to `torch.Tensor`): The base type of the objects to which apply `func`. error_on_other_type (`bool`, *optional*, defaults to `False`): Whether to return an error or not if after unpacking `data`, we get on an object that is not of type `main_type`. If `False`, the function will leave objects of types different than `main_type` unchanged. **kwargs (additional keyword arguments, *optional*): Keyword arguments that will be passed to `func` when applied on the unpacked data. Returns: The same data structure as `data` with `func` applied to every object of type `main_type`. c3DK|]}t|gdyw)r;Nrecursively_apply).0oargsr=funckwargsr<s r z$recursively_apply..nsB A"!".7M`djs r;zUnsupported types (z ) passed to `z?`. Only nested list/tuple/dicts of objects that are valid for `z` should be passed.) rr0r7r:rr6itemsrA TypeError__name__)rEr3r<r=rDrFkvs` ```` rrArAUs,$ &       D' "tDz !JJL  )DAq$!".7M`dj)     4D*4*6** !$t*]4==/J++4+=+=*>>Q S   K s#C c t|s t|drdk(rd |jSt |ttfrt|fd|DSt |tr^t trgngt||jDcic]\}}||vr|nt|c}}S|S#t$r|jcYSt$r,}t rt trdn|Yd}~nd}~wwxYw |jS#t$r|jcYSwxYwcc}}w) a Recursively sends the elements in a nested list/tuple/dictionary of tensors to a given device. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to send to a given device. device (`torch.device`): The device to send the data to. Returns: The same data structure as `tensor` with all tensors sent to the proper device. tonpuznpu:0) non_blockingznpu:Nc3<K|]}t|yw)rP skip_keysN)send_to_device)rBtdevicerPrSs rrGz!send_to_device..s$ohncd^AvLT]^^hnsrR)rr1rNrIAssertionErrorrrintr0r7r:rstrr6rHrT)rrVrPrSerrorrKrUs ``` rrTrTslv'&$"7 U?F 99V,9? ? FUDM * ohno   FG $ i %" I  ItF|#LLN *DAqY1N1fS_kt,uu*    = %99V$ $  !fc*#F8_F   %99V,9? ? %99V$ $ % s5C) E D#/D#7"DD#'D::EEc d}t||S)aK Recursively gathers the information needed to rebuild a nested list/tuple/dictionary of tensors. Args: data (nested list/tuple/dictionary of `torch.Tensor`): The data to send to analyze. Returns: The same data structure as `data` with [`~utils.TensorInformation`] instead of tensors. cDt|j|jS)N)shapedtype)rr]r^rs r_get_data_structurez/get_data_structure.._get_data_structures v||6<<HHrr@)r3r_s rget_data_structurer`sI 0$ 77rc d}t||S)a: Recursively gathers the shape of a nested list/tuple/dictionary of tensors as a list. Args: data (nested list/tuple/dictionary of `torch.Tensor`): The data to send to analyze. Returns: The same data structure as `data` with lists of tensor shapes instead of tensors. c,t|jSr)r7r]rs r _get_shapezget_shape.._get_shapesFLL!!rr@)r3rcs r get_shaperds" Z ..rc,d}t||tS)z Recursively initializes tensors from a nested list/tuple/dictionary of [`~utils.TensorInformation`]. Returns: The same data structure as `data` with tensors instead of [`~utils.TensorInformation`]. cTtj|jd|jiSNr^)remptyr]r^r*s r_initialize_tensorz.initialize_tensors.._initialize_tensors"{{K--G[5F5FGGrr<)rAr,)data_structureris rinitialize_tensorsrlsH /K` aarct|tttfr&t |dk(rt dt |dt|ttfrt|dSt|tr%|jD]}t||cSn2t|tjstdt |d|jdS)a Recursively finds the batch size in a nested list/tuple/dictionary of lists of tensors. Args: data (nested list/tuple/dictionary of `torch.Tensor`): The data from which to find the batch size. Returns: `int`: The batch size. rz&Cannot find the batch size from empty .z0Can only find the batch size of tensors but got ) rr0r7rlen ValueErrorr6find_batch_sizekeysrrrIr])r3rKs rrqrqs$g./SY!^A$t*QOPP$ &tAw'' D' "A"47+ + ell +J4PT:,VWXYY ::a=rcD t|S#ttf$rYywxYw)a Same as [`utils.operations.find_batch_size`] except will ignore if `ValueError` and `TypeErrors` are raised Args: data (nested list/tuple/dictionary of `torch.Tensor`): The data from which to find the batch size. Returns: `int`: The batch size. N)rqrprIr2s rignorant_find_batch_sizerts, t$$  "   s c d}t||S)aS Recursively finds tensors in a nested list/tuple/dictionary and converts them to a list of numbers. Args: data (nested list/tuple/dictionary of `torch.Tensor`): The data from which to convert to regular numbers. Returns: The same data structure as `data` with lists of numbers instead of `torch.Tensor`. c|jj}|jtjk(r|j tj }|jSr)detachcpur^rbfloat16rNfloat32tolistrs r_convert_to_listz!listify.._convert_to_list!sF$$& <<5>> )YYu}}-F}}rr@)r3r|s rlistifyr}s -t 44rcPd}t||d}tj|S)Nc|jdk(r|jd}|js|j}t j |S)Nr)ndimclone is_contiguous contiguousxm all_gatherrs r_tpu_gather_onez$_tpu_gather.._tpu_gather_one.sI ;;! \\^D)F##%&&(F}}V$$rTr=)rAr mark_step)rrress r _tpu_gatherr-s%% OV NCLLN Jrcttjjjj dk(r*t ddrtjjfd}t||dS)Nr z<=z2.8c|jdk(r|jd}|js|j}jjdk7rtt j j|jz|jj}|||jdg|jddStjDcgc]}t j|}}t jj!||t j"|dScc}w)Nrgloor^rVr dim)rrrrbackendrrh num_processesnumelr^rVviewsizerange empty_like distributedrcat)routput_tensors_ gather_opstates r_gpu_gather_onez$_gpu_gather.._gpu_gather_oneDs ;;! \\^D)F##%&&(F == $&)@ #[[##flln4ll||N nf -&>&&r>FKKM!",=> > AFeFYFY@Z[@Z1e..v6@ZN[    ( ( @99^3 3\s*D?Tr) r rrall_gather_into_tensorrVr6rr synchronizerA)rrrrs @@r _gpu_gatherr<s_ NE!!88I ||E!&6tU&C 48 _f$ OOrceZdZdZy)DistributedOperationExceptionz An exception class for distributed operations. Raised if the operation cannot be performed due to the shape of the tensors. N)rJ __module__ __qualname____doc__rrrrcs   rrc.tfd}|S)zv Verifies that `tensor` is the same shape across all processes. Only ran if `PartialState().debug` is `True`. c 0tjtjk(stjs |i|S j d j }d|vr|d}n|d}tjjt|jk7rgtd|d|jjdtjjdtjjd|d t|}t|g}|df|j|dt|k(}|sDd jt!|Dcgc] \}}d |d |c}}} td |d |  |i|Scc}}w)Nrnrrz%One or more of the tensors passed to z were not on the z+ while the `Accelerator` is configured for z. Please move it to the z before calling z - zProcess z: znCannot apply desired operation due to shape mismatches. All shapes across devices must be valid. Operation: `z` Input shapes: - )r distributed_typerNOdebugrrJrVr6 find_devicerrd gather_objectcountrojoin enumerate) rDrF operationrshapesoutputare_sameir]process_shape_strfunctions rwrapperz!verify_operation..wrapperqs > * *o.@.@ @ H\H\T,V, ,**+1X->->,?@ v H%F!WF > % %V)<)A)A A/7 {BSTZTaTaTfTfSghST`TbTiTiTnTnSoo))5)>)>)C)C(DDTU^T__`b 6"x( !9 ||F1I.#f+=H$,MM[dek[l2m[lxqRWXaS5'3J[l2m$n!3''0k1GHYGZ\ ((( 3nsF rrrs` rverify_operationrls"  8_))4 Nrc.tfd}|S)z Checks that `verify_operation` failed and if so reports a more helpful error chaining the existing `DistributedOperationException`. c |i|S#t$r0}jdj}td|d|d}~wwxYw)NrnzError found while calling `z1`. Please see the earlier error for more details.)rrrJ)rDrFerrs rrz"chained_operation..wrappersc T,V, ,, #../q1B1B0CDI/-i[8ij  s A+?Arrs` rchained_operationrs"  8_ Nrctjtjk(r t |Stjt vr t |S|S)a4 Recursively gather tensor in a nested list/tuple/dictionary of tensors from all devices. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to gather. Returns: The same data structure as `tensor` with all tensors sent to the proper device. )r rrXLArr rrs rgatherrsF~&&/*=*==6""  ( (,M M6"" robjectcttjDcgc]}d}}tjj |||Dcgc] }|D]}| c}}Scc}wcc}}wr)rr rrrall_gather_object)rroutput_objectsyxs r_gpu_gather_objectrsc$),.*F*F$GH$Gqd$GNH ''?% 1~!q!AqA~ 11I 2s A$A)ctjtjk(r t dtjt vr t |S|S)a5 Recursively gather object in a nested list/tuple/dictionary of objects from all devices. Args: object (nested list/tuple/dictionary of picklable object): The data to gather. Returns: The same data structure as `object` with all the objects sent to every device. z&gather objects in TPU is not supported)r rrrNotImplementedErrorr r)rs rrrsG~&&/*=*==!"JKK  ( (,M M!&)) rc(dd}t||d|S)NcHtjj|||S)Nsrc)rr broadcast)rrs r_gpu_broadcast_onez*_gpu_broadcast.._gpu_broadcast_ones  ##F#4 rT)r=rrr@)r3rrs r_gpu_broadcastrs /4UX YYrc Tt|ttfrt|fdt |DSt|t rCt ||jDcic]\}}|t|d|c}}Stj|fdScc}}w)Nc3HK|]\}}t|d|yw)rnameN)_tpu_broadcast)rBrrUrs rrGz!_tpu_broadcast..s-"gUfTQPQ>!TF!A3-#H#HUfs"rrc|Srr)rrs rz _tpu_broadcast..s !C&r) rr7r0r:rrr6rHrr mesh_reduce)rrrrKrLs `` rrrs&4-(&"gU^_eUf"ghh FG $tF|RXR^R^R`aR`$!QQq$q} EER`abb >>$(8 99bs%B$  cd}t}tj|tj|j}|V|j }t |j}tjt||gzt|dt|dzt|d}||j}t|ddd }|dd}||fS) ze Grabs the shape of `tensor` only available on one process and returns a tensor of its shape irNr^r sum reductionrr) r rrhrXrVr]TENSOR_TYPE_TO_INTr^rr7roreducenonzero)rmax_tensor_dimensionr base_tensorr] tensor_dtyper^s rgather_tensor_shapers ! NE++2%))ELLYK  )&,,7 (- T%[L>5QY\(] $c%j1n%6Kk1134K  BC # $Ecr"K  rreturnct}t|\}}|7tj|t|j |j }t|dS)a Copies a tensor that only exists on a single device and broadcasts it to other devices. Differs from `broadcast` as each worker doesn't need to know its shape when used (and tensor can be `None`) Args: tensor (`torch.tensor`): The tensor that should be sent to all devices. Must only have it be defined on a single device, the rest should be `None`. rrr)r rrzerosTENSOR_INT_TO_DTYPErNrVr)rrr]r^s rcopy_tensor_to_devicesr sP NE&v.LE5 ~U*=e*DEHHV &E **r from_processctjtjk(rt ||dStjt vr t ||S|S)a Recursively broadcast tensor in a nested list/tuple/dictionary of tensors to all devices. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to gather. from_process (`int`, *optional*, defaults to 0): The process from which to send the data Returns: The same data structure as `tensor` with all tensors broadcasted to the proper device. zaccelerate.utils.broadcast)rrr)r rrrrr r)rrs rrrsM~&&/*=*==f,=YZZ  ( (,M Mf,77 rc,tjtjk(r2t |D]"\}}t j d|fd||<$|Stjtvr!tjj||S)a Broadcast a list of picklable objects from one process to the others. Args: object_list (list of picklable objects): The list of objects to broadcast. This list will be modified inplace. from_process (`int`, *optional*, defaults to 0): The process from which to send the data. Returns: The same list containing the objects from process 0. z&accelerate.utils.broadcast_object_listc|Srr)rrs rrz'broadcast_object_list..?s efgsetrr) r rrrrrrr rrbroadcast_object_list) object_listrrr8s ` rrr0s~&&/*=*== ,FAs^^,TVY[tuKN-   ( (,M M // /N rc"d}t|||S)aN Recursively takes a slice in a nested list/tuple/dictionary of tensors. Args: data (nested list/tuple/dictionary of `torch.Tensor`): The data to slice. tensor_slice (`slice`): The slice to take. Returns: The same data structure as `data` with all the tensors slices. c ||Srr)r tensor_slices r _slice_tensorz$slice_tensors.._slice_tensorSs l##rr@)r3r process_indexrrs r slice_tensorsrEs$ ]D, ??rcbtdttfr.tdfdt t dDStdt rRtddjDcic]!}|tDcgc]}|| c}#c}}StdtjrtjStttfrt dk(rdStdtdcc}wcc}}w)a Recursively concatenate the tensors in a nested list/tuple/dictionary of lists of tensors with the same shape. If there is only a single batch of data, it is returned as-is. Args: data (nested list/tuple/dictionary of lists of tensors `torch.Tensor`): The data to concatenate. dim (`int`, *optional*, defaults to 0): The dimension on which to concatenate. Returns: The same data structure as `data` with all the tensors concatenated. rc3dK|]"}tDcgc]}|| c}$ycc}ww)rN) concatenate)rBrdr3rs rrGzconcatenate..hs5#lXkSTKt0Dt!1t0D#$N$NXk0Ds 0 + 0rr z%Can only concatenate tensors but got )rr0r7r:rrorr6rrrrrrrI)r3rrKrs`` rrrYs $q'E4=)$q'#lX]^abfghbi^jXk#lmm DGW %tDG}UYZ[U\UaUaUcdUcPQaD-ADqadD-As!KKUcdee DGU\\ *yy3'' D5$- (SY!^Aw?T!W OPP .BdsD+  D& D+ &D+ c eZdZy)CannotPadNestedTensorWarningN)rJrrrrrrrssrrc,dd}t||d|||S)a3 Recursively pad the tensors in a nested list/tuple/dictionary of tensors from all devices to the same size so they can safely be gathered. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to gather. dim (`int`, *optional*, defaults to 0): The dimension on which to pad. pad_index (`int`, *optional*, defaults to 0): The value with which to pad. pad_first (`bool`, *optional*, defaults to `False`): Whether to pad at the beginning or the end. c t|ddrtjdt|St |j k\st |j  kr|Sdkrt |j z t j|j |jd}t|j}tfd|D |j k(r|S|j t } |<|jt||z}|r)t fdtt |D}n't fdtt |D}|||<|S) N is_nestedFzHCannot pad nested tensors without more information. Leaving unprocessed.r)rVc3(K|] }| ywrr)rBsrs rrGzFpad_across_processes.._pad_across_processes..s-u!qvusc3bK|]&}|k(rtz n td(ywrslice)rBrrmax_sizeold_sizes rrGzFpad_across_processes.._pad_across_processes..s7`t[\Q#Xh#.95QU;V`ts,/c3\K|]#}|k(rtdn td%ywrNrrBrrrs rrGzFpad_across_processes.._pad_across_processes..s/oZnUVqCxE!Xc]3U4[PZn),)getattrwarningswarnrror]rrrVrrxmaxr7 new_zerosr0r) rr pad_index pad_firstrsizesnew_size new_tensorindicesrrs ` @@r_pad_across_processesz3pad_across_processes.._pad_across_processessK 6; . MMZ, M #fll# #sc&,,.?-?'?M 7 3v||$ $C||FLL?Et   "-u-- v||C( (M<<>  %%eHo6B `efijrfs`tGoZ_`cdl`mZnooG$ 7rT)r=rrrrrFr@)rrrrr"s rpad_across_processesr$ws&" D v4ST]ir rc,dd}t||d|||S)z Takes a `tensor` of arbitrary size and pads it so that it can work given `num_processes` needed dimensions. New tensors are just the last input repeated. E.g.: Tensor: ([3,4,4]) Num processes: 4 Expected result shape: ([4,4,4]) cF ||z}|||zz }||zdk(r||z }n|||zz }|||zcxkDrdkrnn||z }|j t }||z|d<|jt|}t fdt t |D} ||| <|S)Nrr c3\K|]#}|k(rtdn td%ywrrrs rrGz@pad_input_tensors.._pad_input_tensors..s/kVjQR18a#/tLVjr)r]r7rr0rro) r batch_sizerr remainder last_inputsto_padrr r!rs ` @r_pad_input_tensorsz-pad_input_tensors.._pad_input_tensorss-/  I $=>  &! +"Z/F"jM&ABF & ,1 , 6)F<<> 6) %%eHo6 kV[\_`h\iVjkk$ 7rT)r=r(rrrr@)rr(rrr,s rpad_input_tensorsr-s(&  #   rc*dd}t||d||S)aW Recursively reduce the tensors in a nested list/tuple/dictionary of lists of tensors across all processes by the mean of a given operation. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to reduce. reduction (`str`, *optional*, defaults to `"mean"`): A reduction method. Can be of "mean", "sum", or "none" scale (`float`, *optional*): A default scaling value to be applied after the reduce, only valid on XLA. Returns: The same data structure as `data` with all the tensors reduced. c t}|j}|jtjk(r|S|jtj k(rOt jt jt j|g|t jnJ|jjtvr.tjj|tj|dk(r||j z}|S)Nmean)r rrrrrrr all_reduce REDUCE_SUMvaluer rrrSUMr)rrscaler cloned_tensors r_reduce_across_processesz(reduce.._reduce_across_processess   ! !_%7%7 7  ! !_%8%8 8 LLN MM"---% @ LLN  # # ) )-N N    ( ( E   U00 0MrT)r=rr5r0g?r@)rrr5r7s rrrs"$&  &di_d rc*d}d}t|||S)av Recursively converts the elements nested list/tuple/dictionary of tensors in FP16/BF16 precision to FP32. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to convert from FP16/BF16 to FP32. Returns: The same data structure as `tensor` with all tensors that were in FP16/BF16 precision converted to FP32. c"|jSr)floatrs r_convert_to_fp32z)convert_to_fp32.._convert_to_fp32 s||~rct|xs t|dxr,|jtjtj fvSrg)rr1r^rfloat16ryrs r_is_fp16_bf16_tensorz-convert_to_fp32.._is_fp16_bf16_tensors@'C767+C  MM NNZ J  rrjr@)rr<r?s rconvert_to_fp32r@s -vAU VVrc"eZdZdZdZdZdZy)ConvertOutputsToFp32ae Decorator to apply to a function outputting tensors (like a model forward pass) that ensures the outputs in FP16 precision will be convert back to FP32. Args: model_forward (`Callable`): The function which outputs we want to treat. Returns: The same function as `model_forward` but with converted outputs. c*||_t||yr) model_forwardr)selfrDs r__init__zConvertOutputsToFp32.__init__&s*t]+rc8t|j|i|Sr)r@rD)rErDrFs r__call__zConvertOutputsToFp32.__call__*s 1t114B6BCCrc,tjd)NzCannot pickle a prepared model with automatic mixed precision, please unwrap the model with `Accelerator.unwrap_model(model)` before pickling it.)pickle PicklingError)rEs r __getstate__z!ConvertOutputsToFp32.__getstate__-s"" `  rN)rJrrrrFrHrLrrrrBrBs ,D rrBc6tfd}|_|S)Nc|i|Srr)rDrFrDs rforwardz(convert_outputs_to_fp32..forward6sd-f--r)rB __wrapped__)rDrOs` rconvert_outputs_to_fp32rQ3s!(7M.(G Nrct|tr'|jD]}t|}||cSyt|tt fr|D]}t|}||cSyt|t jr |jSy)z Finds the device on which a nested dict/list/tuple of tensors lies (assuming they are all on the same device). Args: (nested list/tuple/dictionary of `torch.Tensor`): The data we want to know the device of. N) rrvaluesrr0r7rrrV)r3r8rVs rrr?s$ ;;=C %F! ! D5$- (C %F!  D%,, '{{ (rc#TKtjtjk7s6tj-tjj s t }n#ddl}|jj||||}|5ddddy#1swYyxYww)z Wrapper around `deepspeed.runtime.zero.GatheredParameters`, but if Zero-3 is not enabled, will be a no-op context manager. Nr) modifier_rank fwd_moduleenabled) r rr DEEPSPEEDdeepspeed_pluginis_zero3_init_enabledr deepspeedzeroGatheredParameters)paramsrUrVrWgather_param_contextr[s rr]r]Ts**o.G.GG++7 "33IIK*}(~~@@ -JPW A      sB B(B B(B%!B()FNr)rzbroadcast tensorr)NNr#r8)NNT)YrrJrcollections.abcr contextlibrr functoolsrrtypingrrrr r constantsr dataclassesrrimportsrrrversionsrtorch_xla.core.xla_modelcore xla_modelrtorch.distributedrrr(r,r4r:rArTr`rdrlrqrtr}rr Exceptionrrrrrrrrr;doublehalfryuint8int8int16int32int64boolrrHrrrrrXrrrr UserWarningrr$r-rr@rBrQrr])rKrLs00rrvs?#2+ 28; '))!#*, 6]$4CX]0f1h8$/$ b."5. $PN I  F&&2s2#&Z: KK LL! JJ NNA KK JJ KK KK KK JJ );(@(@(BC(B1q!t(BC2+5<<+"C*S*@(Q4 ; 44n%P&&RW0  4 *Q Ds G