+ QV-j]' ^RIHt^RIt^RIHt^RIHtHt^RIt^RI H t H t ^RIH t ^RI HtHtHtHt] P$.t]P(!]4t]!RR R 7t]!R R R 7t]!R R R 7t]!R R R 7t]!RR R 7t]!RR R 7t]!RR R 7t]!RR R 7t]!RR R 7t]P@PC4t"Rt#R RRllt$!RR] PJ4t&RRlt'RR/RRllt(RRlt)]!]4R4t*R#)!) annotationsN)Callable lru_cachewraps) storage_ptr storage_size)nn)is_torch_greater_or_equalis_torch_xla_availableis_torchdynamo_compilingloggingz2.8T) accept_devz2.6z2.4z2.3z2.2z2.1z2.0z1.13z1.12cJ^RIHpV!WVPVP4#)z A function that calls the internal `_softmax_backward_data` PyTorch method and that adjusts the arguments according to the torch version detected. )_softmax_backward_data)torchrdimdtype)parent grad_outputoutputrs&&& k/Volumes/fast/ai/experiments/ui-tars-smoke/.venv/lib/python3.14/site-packages/transformers/pytorch_utils.pysoftmax_backward_datar4s - !+vzz6<< PPc(V^8dQhRRRRRRRR/#)layerz nn.Linearindexztorch.LongTensorrintreturn)formats"r __annotate__r#?s*i0@sS\rcVPVPP4pVPPW!4P 4P 4pVP e`V^8Xd*VP P 4P 4pM/VP V,P 4P 4p\VPP44p\V4WR&\P!V^,V^,VP RJR7PVPP4pRVPn VPPVP44RVPn VP eLRVP n VP PXP44RVP n V#)a| Prune a linear layer to keep only entries in index. Used to remove heads. Args: layer (`torch.nn.Linear`): The layer to prune. index (`torch.LongTensor`): The indices to keep in the layer. dim (`int`, *optional*, defaults to 0): The dimension on which to keep the indices. Returns: `torch.nn.Linear`: The pruned layer as a new layer with `requires_grad=True`. N)biasFT)toweightdevice index_selectdetachcloner%listsizelenr Linear requires_gradcopy_ contiguous)rrrWbnew_size new_layers&&& rprune_linear_layerr7?sa HHU\\(( )E !!#-446<<>A zz !8 !!#))+A 5!((*002AELL%%'(HJHM (1+x{49OPSSTYT`T`TgTghI%*I" 1<<>*%)I" zz', $Q\\^,'+ $ rc>a]tRt^atRtV3RltRRltRtRtV;t #)Conv1Da 1D-convolutional layer as defined by Radford et al. for OpenAI GPT (and also used in GPT-2). Basically works like a linear layer but the weights are transposed. Args: nf (`int`): The number of output features. nx (`int`): The number of input features. c N<\SV`4WnW n\P !\ P!W!44Vn\P !\ P!V44Vn \PPVPRR7R#)g{Gz?)stdN) super__init__nfnxr Parameterremptyr'zerosr%initnormal_)selfr>r? __class__s&&&rr=Conv1D.__init__lsa ll5;;r#67 LLR1   .rcV^8dQhRR/#)rr strr!)r"s"rr#Conv1D.__annotate__tsBB#Brc :RP!R/VPB#)zConv1D(nf={nf}, nx={nx})r!)r"__dict__)rEs&r__repr__Conv1D.__repr__ts)00A4==AArc  VP4RRVP3,p\P!VPVP RVPR44VP 4pVP V4pV#)N)r-r>raddmmr%viewr')rExsize_outs&& rforwardConv1D.forwardws^668CR=DGG:- KK 166"affRj#94;; G FF8 r)r%r>r?r') __name__ __module__ __qualname____firstlineno____doc__r=rMrU__static_attributes__ __classcell__)rFs@rr9r9as/Brr9c(V^8dQhRRRRRRRR/#)r forward_fnzCallable[..., torch.Tensor] chunk_sizer chunk_dimr torch.Tensorr!)r"s"rr#r#~s6K&K&+K&K&K&  K&rcaaa \V4^8g QV R24h\\P!S4P4pV\V48wd\ RV R\V4 R24hV^8EdZV^,P S,pVF=pVP S,V8wgK\ RV RVP S, 24h V^,P S,V,^8wd*\ RV^,P S, RV 24hV^,P S,V,o \ ;QJd.VV 3R lV4FNK 5M!VV 3R lV44p\ ;QJd.V3R l\V!4FNK 5M!V3R l\V!44p\P!VSR 7#S!V!#) a This function chunks the `input_tensors` into smaller input tensor parts of size `chunk_size` over the dimension `chunk_dim`. It then applies a layer `forward_fn` to each chunk independently to save memory. If the `forward_fn` is independent across the `chunk_dim` this function will yield the same result as directly applying `forward_fn` to `input_tensors`. Args: forward_fn (`Callable[..., torch.Tensor]`): The forward function of the model. chunk_size (`int`): The chunk size of a chunked tensor: `num_chunks = len(input_tensors[0]) / chunk_size`. chunk_dim (`int`): The dimension over which the `input_tensors` should be chunked. input_tensors (`tuple[torch.Tensor]`): The input tensors of `forward_fn` which will be chunked Returns: `torch.Tensor`: A tensor with the same shape as the `forward_fn` would have given if applied`. Examples: ```python # rename the usual forward() fn to forward_chunk() def forward_chunk(self, hidden_states): hidden_states = self.decoder(hidden_states) return hidden_states # implement a chunked forward function def forward(self, hidden_states): return apply_chunking_to_forward(self.forward_chunk, self.chunk_size_lm_head, self.seq_len_dim, hidden_states) ```z" has to be a tuple/list of tensorszforward_chunk_fn expects z arguments, but only z input tensors are givenz/All input tenors have to be of the same shape: z, found shape zThe dimension to be chunked z( has to be a multiple of the chunk size c3H<"TFqPSSR7xK R#5i)rN)chunk).0 input_tensorra num_chunkss& r ,apply_chunking_to_forward..s%$ugtWc%7%7 %7%R%Rgts"c30<"TF pS!V!xK R#5iNr!)rginput_tensors_chunkr_s& rrjrksuZtCVj*=>Ztsre) r.inspect signature parameters ValueErrorshapetupleziprcat) r_r`ra input_tensorsnum_args_in_forward_chunk_fn tensor_shaperhinput_tensors_chunks output_chunksris f&f* @rapply_chunking_to_forwardr|~sR }  !Wm_4V#WW !$'w'8'8'D'O'O#P #s='99'(D'EEZ[^_l[mZno    A~$Q'--i8 )L!!), < El^T##/#5#5i#@"AC*   ! !) ,z 9Q >.}Q/?/E/Ei/P.QR"|%  #1%++I6*D  %u$ugt$uuu$ugt$uuuZ]_sZtuuZ]_sZtuu yyI66 } %%rindexingc$V^8dQhRRRRRR/#)rtensorsz!torch.Tensor | list[torch.Tensor]r}z str | Noner ztuple[torch.Tensor, ...]r!)r"s"rr#r#s#7787J7Zr7rc.\P!VRV/#)z Wrapper around torch.meshgrid to avoid warning messages about the introduced `indexing` argument. Reference: https://pytorch.org/docs/1.13/generated/torch.meshgrid.html r})rmeshgrid)r}rs$*rrrs >>7 6X 66rc V^8dQhRRRR/#)rtensorrbr ztuple[torch.device, int, int]r!)r"s"rr#r#s::l:/L:rc\'dn\R4'd]^RIHp\ W4'dFVP 4pVP VP4P4VP3#VP PR8Xd1\4'd!^RI pVPPV4pM \V4pVP V\!V43#)a Unique identifier to a tensor storage. Multiple different tensors can share the same underlying storage. For example, "meta" tensors all share the same storage, and thus their identifier will all be equal. This identifier is guaranteed to be unique and constant for this tensor's storage during its lifetime. Two tensor storages with non-overlapping lifetimes may have the same id. z2.5)DTensorxlaN)_torch_distributed_availabler torch.distributed.tensorr isinstanceto_localr(storagedata_ptrnbytestyper torch_xla_XLAC_xla_get_tensor_idrr )rr local_tensorr unique_ids& rid_tensor_storagers$#(A%(H(H4 f & &!??,L==,"6"6"8"A"A"CV]]R R }}U"'='?'? OO66v> ' ==)\&%9 99rcaaVV3RlpV#)z LRU cache decorator from standard functools library, but with a workaround to disable caching when torchdynamo is compiling. Expected to work with class methods. cX<aa\S/SB!S4o\S4VV3Rl4pV#)cD<\4'd S!V/VB#S!V/VB#rm)r )argskwargsfuncfunc_with_caches*,rwrapperGcompile_compatible_method_lru_cache..decorator..wrappers,'))T,V,,&777rr)rrrlru_args lru_kwargssf @r decorator6compile_compatible_method_lru_cache..decorators4#X<rs)#$& 7 ~   H %%>uQU%V"%>uQU%V"&?uQU%V"%>uQU%V"%>uQU%V"%>uQU%V"%>uQU%V"&?SW&X#&?SW&X# %00==?QDRYY:K&\7QU7:8yr