+ QV-ja4^RIt^RIt^RIHtHtHtHtHtHtH t ^RI H t ^RI H t ^RIHtHt!RR4t!RR ]P$P&4tR R lt!R R ]P$P&4t!RR]P$P&4tRRRllt!RR]P$P&4t!RR]P$P&4t!RR]P$P&4tR RRlltRtRRltRRltR#)!N) DynamicCache DynamicLayerDynamicSlidingWindowLayerEncoderDecoderCache StaticCache StaticLayerStaticSlidingWindowLayer)GenerationConfig)PreTrainedModel)is_torch_greater_or_equal"is_torch_greater_or_equal_than_2_6cba]tRt^"toRtR V3RlRlltRtRtRtRt Rt R R lt R t Vt R #)TorchExportableModuleForVLMad A wrapper class for exporting Vision-Language Models (VLMs) like SmolVLM2 for ExecuTorch. This class handles the export of three main components: 1. Vision encoder (processes images to visual features) 2. Connector/projector (maps visual features to text embedding space) 3. Text decoder (generates text from combined visual and text tokens) c&<V^8dQhRS[RS[/#)max_batch_size max_cache_len)int)format __classdict__s"u/Volumes/fast/ai/experiments/ui-tars-smoke/.venv/lib/python3.14/site-packages/transformers/integrations/executorch.py __annotate__(TorchExportableModuleForVLM.__annotate__,s**c*c*c WnW nW0nVPVnVPPVnVPP VnVPPVnRVn RVn RVn R#)z Initialize the exportable VLM module. Args: model: The VLM (e.g. SmolVLM) model instance max_batch_size: Maximum batch size. Always 1 for ExecuTorch max_cache_len: Maximum cache length for text generation N) modelrrconfig vision_modelvision_encoder connector text_model text_decoderexported_vision_encoderexported_connectorexported_text_decoder)selfrrrs&&&&r__init__$TorchExportableModuleForVLM.__init__,sq ,*ll $kk66..!KK22(,$"&%)"rc VPP4\P!^^RR\PR7pR^\P P P^\P P P//p\P P VPV3VRR7VnVP#)z$Export the vision encoder component.idtype pixel_valuesFargsdynamic_shapesstrict) revaltorchrandnfloat32exportDimAUTOr#)r&r,r/s& rexport_vision_encoder1TorchExportableModuleForVLM.export_vision_encoderDs   "{{1acG  5<<##((5<<##(( (-||':':   ) (;( $+++rc 2VPP4VPPPpVPPP pVPPP pW#,pWD,p\P!^WQ\PR7pR^\PPP//p\PPVPV3VRR7Vn VP#)zExport the connector component.r*image_hidden_statesFr-)r r1r vision_config hidden_size image_size patch_sizer2r3r4r5r6r7r$)r&vision_hidden_sizer>r?patches_per_dim num_patchesr;r/s& rexport_connector,TorchExportableModuleForVLM.export_connector\s "[[66BB[[..99 [[..99 $2%7 #kk![TYTaTab0!U\\5E5E5J5J1KL#(,,"5"5 NN%') #6# &&&rc \VPR7Vn^p\P!^V3\P R7p\P !V\P R7p\VPVPPP4p\PPRV^, R7pR^V/R^V//pVPPVVVRR7VnVP#) z"Export the text decoder component.rr*seq_length_dimmax input_idscache_positionF)rJrKr/r0)%TorchExportableModuleForDecoderOnlyLMr"exportable_text_decoderr2zeroslongarangeminrr text_configmax_position_embeddingsr5r6r%)r& seq_lengthrJrKmax_seq_length seq_len_dimr/s& rexport_text_decoder/TorchExportableModuleForVLM.export_text_decoderus(MSWSdSd'e$ KKJuzzB j CT//1H1H1`1`all&&'7^a=O&P  ![) q+.  &*%A%A%H%H)) &I& ")))rc VP!R/VBVP!R/VBVP!R/VBRVPRVPRVP /#)z'Export all components of the VLM model.rr r")r8rCrWr#r$r%)r&kwargss&,rr5"TorchExportableModuleForVLM.exports` "",V, ''   *6* d:: 00 D66  rc R#)ap Simplified forward pass for inference with guaranteed non-null input_ids and cache_position. Args: pixel_values: Input images [1, channels, height, width] (optional) input_ids: Text token IDs [1, seq_len] (required - won't be None) cache_position: Cache positions [seq_len] (required - won't be None) Returns: Output with logits for text generation NrZ)r&r,rJrKs&&&&rforward#TorchExportableModuleForVLM.forwardrNc R#)a Simplified generate method with guaranteed non-null input_ids. Args: pixel_values: Input images [1, channels, height, width] (optional) input_ids: Initial text tokens [1, seq_len] (required - won't be None) max_new_tokens: Maximum number of tokens to generate do_sample: Whether to use sampling or greedy decoding temperature: Temperature for sampling Returns: Generated sequences NrZ)r&r,rJmax_new_tokens do_sample temperaturer[s&&&&&&,rgenerate$TorchExportableModuleForVLM.generater`r) rr rMr$r%r#rrrr"r))NN2F?)__name__ __module__ __qualname____firstlineno____doc__r'r8rCrWr5r^re__static_attributes____classdictcell__)rs@rrr"s7**0,0'2*6      rrcaa]tRt^toRtR V3RlV3RllltR V3RlRlltR V3RlRllt]R V3RlR ll4t R t Vt V;t #)rLz A recipe module designed to make a `PreTrainedModel` exportable with `torch.export`, specifically for decoder-only LM with cache. This module ensures that the exported model is compatible with further lowering and execution in `ExecuTorch`. c t<V^8dQhRS[RS[R,RS[R,RS[PR,RR/#rr batch_sizeNrdevicereturnr rr2rv)rrs"rr2TorchExportableModuleForDecoderOnlyLM.__annotate__sUhhh$JhTz h  t# h  hrc h<\SV`4VPP4p\ VR4'dVP RJd \ R4h\ VR4'd#\VRR4e\WW44Vn R#\P!R4\WW44Vn R#)z Initializes the exportable module. Args: model (`PreTrainedModel`): The pretrained model to wrap. Raises: ValueError: If the model is configured with a unsupported cache implementation. use_cacheFz5The model must have caching enabled to be performant. layer_typessliding_windowNzmUsing `StaticCache` for export as `layer_types` is not specified or `sliding_window` is `null` in the config.) superr'rget_text_confighasattrr{ ValueErrorgetattr$TorchExportableModuleWithHybridCacherlogginginfo$TorchExportableModuleWithStaticCache)r&rrurrvr __class__s&&&&& rr'.TorchExportableModuleForDecoderOnlyLM.__init__s --/v{++v/?/?5/HTU U 6= ) )gf>NPT.U.a=eQ^gDJ LL >eQ^gDJrc<V^8dQhRS[PR,RS[PR,RS[PR,RS[P/#rrJN inputs_embedsrKrw)r2Tensor)rrs"rrrysXTT<<$&T||d*T t+ T  Trc :VPPWR7#)a Forward pass of the module, which is compatible with the ExecuTorch llm runner. Args: input_ids (`torch.Tensor`): Tensor representing current input token id to the module. inputs_embeds (`torch.Tensor`): Tensor representing current input embeddings to the module. cache_position (`torch.Tensor`): Tensor representing current input position in the cache. Returns: torch.Tensor: Logits output from the model. )rJr)rr^)r&rJrrKs&&&&rr^-TorchExportableModuleForDecoderOnlyLM.forwards"zz!!I!SSrc <V^8dQhRS[PR,RS[PR,RS[PR,RS[R,RS[R,RS[PP /#)rrJNrrKr/r0rw)r2rdictboolr5ExportedProgram)rrs"rrrys|Z Z <<$&Z ||d*Z  t+ Z t Z t Z   % %Z rc  "VRJVRJ, 'g \R4h\VPR4'dC\VPVPPVP4pVP pMU\VPR4'd"VPPP pMRp\ P!R4VeDRTRVeTM7\P!VPR ,\PVR 7/pMBR TRVeTM7\P!VP^,\PVR 7/p\PPVPRTTVeTMR R 7p V #)aG Export the wrapped module using `torch.export`. Args: input_ids (`Optional[torch.Tensor]`): Tensor representing current input token id to the module. Must specify either this or inputs_embeds. inputs_embeds (`Optional[torch.Tensor]`): Tensor representing current input embeddings to the module. Must specify either this or input_ids. cache_position (`Optional[torch.Tensor]`): Tensor representing current input position in the cache. If not provided, a default tensor will be used. dynamic_shapes (`Optional[dict]`): Dynamic shapes to use for export if specified. strict(`Optional[bool]`): Flag to instruct `torch.export` to use `torchdynamo`. Returns: torch.export.ExportedProgram: The exported program that can be used for inference. Examples: Export with input_ids: ```python # Prepare inputs input_ids = torch.tensor([[1, 2, 3]], dtype=torch.long, device=model.device) cache_position = torch.arange(input_ids.shape[-1], dtype=torch.long, device=model.device) # Export exported = exportable_module.export( input_ids=input_ids, cache_position=cache_position ) ``` Export with inputs_embeds: ```python # Prepare embeddings inputs_embeds = torch.randn(1, 3, 768, device=model.device) # batch_size=1, seq_len=3, hidden_size=768 cache_position = torch.arange(inputs_embeds.shape[1], dtype=torch.long, device=model.device) # Export exported = exportable_module.export( inputs_embeds=inputs_embeds, cache_position=cache_position ) ``` Nz2Need to specify either input_ids or inputs_embeds.base_model_prefixrcpuzfTorchExportableModuleForDecoderOnlyLM.export Can't infer device from the model. Set to CPU by default.rJrKr+rvrTr.r[r/r0rZ) rrrrrrvrwarningr2rPshaperOr5) r&rJrrKr/r0base model_device input_kwargsexported_programs &&&&&& rr5,TorchExportableModuleForDecoderOnlyLM.exportsUjT!mt&;<<QR R 4::2 3 34::tzz'C'CTZZPD;;L TZZ ) )::++22L L OOx   Y !-#1\\)//""5UZZP\] L  !-#1\\-"5"5a"8 S_` L!<<.. JJ)#/6T /  rcx<V^8dQhRS[PPRS[RS[RS[RS[RS[RS[RS[R S[/ #) rrpromptrbrcrdtop_ktop_prvrw)r2r5rstrrrfloat)rrs"rrryRsiLiL,,66iLiL iL  iL  iLiLiLiL iLrc  VP4p V!VRR7PPV4p V P4p ^p \ V P ^,4FPp V RW^,13,p\ P!V .\ PVR7pV !WR7pV ^, p KR \ V4EFpV RRR13,p\ P!V .\ PVR7pV !WR7pV'EdV^8d VV, pMTpV^8d7V\ P!VV4^,R,8p\R4VV&VR 8d\ P!VR R 7wpp\ P!\ P!VRR 7RR 7pVV8pVRRR13,P4VR&^VR&VPRVV4p\R4VV&\ P!VRR 7p\ P!V^R 7pMVP!RR R7pVP#4^8dVP%R4p\ P&!V V.RR 7p V ^, p VP)4VP*8XgEKM VP-V ^,R R7#)a Generate a sequence of tokens using an exported program. Args: exported_program (`torch.export.ExportedProgram`): The exported model being used for generate. tokenizer: The tokenizer to use. prompt (str): The input prompt. max_new_tokens (int): Maximum number of new tokens to generate. do_sample (bool): Whether to use sampling or greedy decoding. temperature (float): The temperature for sampling. top_k (int): The number of highest probability tokens to keep for top-k sampling. top_p (float): The cumulative probability for nucleus sampling. device (str): The device to use. Returns: str: The generated text. pt)return_tensorsNNNrrJrKN.z-infrjT) descendingdim) num_samples)rkeepdim)skip_special_tokensr).rN).:rgNN).r)modulerJtoclonerangerr2tensorrOtopkrsortcumsumsoftmaxscatter multinomialargmaxrsqueezecatitem eos_token_iddecode)r tokenizerrrbrcrdrrrvexported_modulerJ generated_ids curr_positionicurr_input_idscurr_cache_position_outputslogitsindices_to_remove sorted_logitssorted_indicescumulative_probssorted_indices_to_removeprobs next_token_ids&&&&&&&&& rre.TorchExportableModuleForDecoderOnlyLM.generateQs<+113fT:DDGGO ")  yq)*A&q!!e)|4N"',, ejjY_"`  .]A Q M+~&A*1bc62N"',, ejjY_"` &cGy?${2F$F19(.FE1J11Mm1\(\%05f F,-3;49JJvRV4W1M>',||EMM-UW4X^`'a$0@%/G,8PQTVYWYVYQY8Z8`8`8b,W578,V4)A(H(H^]u(v%05f F,- f"5 % 1 1%Q G !(2t D   "Q& - 5 5b 9 "II}m&D"MM Q M!!#y'='==k'p a 0dKKrrFNNN)NNNNN)Frjrirjr) rkrlrmrnror'r^r5 staticmethodrerprq __classcell__rrs@@rrLrLsV hhBTT&Z Z xiLiLiLiLrrLcV^8dQhR\\\\,,\\\,,3,/#)rrw)tuplerlist)rs"rrrs-uS49_cDIo%EFrc\VR4'dVPRVP)Uu.uF"pVR8Xd VPM VPNK$ ppVPRVP)Uu.uF4pVR8XdVP 'd VP M VPNK6 ppW23#\VRVPVP,4p\VRVP4pW23#uupiuupi)zqReturns a tuple `(num_heads, head_dim)` containing either 2 ints, or a list of int with the value for each layer.global_head_dimNfull_attentionhead_dimnum_key_value_heads) rr|num_kv_shared_layersrrattention_k_eq_vnum_global_key_value_headsrrr=num_attention_heads)rlayerr num_headss& rget_head_shapesrsv()) ++,Jv/J/J.JK K',/?&?F " "V__ TK   ++,Jv/J/J.JK  L((V-D-D-D  - -++ ,L    6:v/A/AVE_E_/_`F$96;U;UV    s(C65C;C;cxaa]tRtRtoRtR V3RlV3RllltR V3RlRllt]V3RlRl4tR t Vt V;t #) ria A recipe module designed to make a `PreTrainedModel` exportable with `torch.export`, specifically for decoder-only LM to `StaticCache`. This module ensures that the exported model is compatible with further lowering and execution in `ExecuTorch`. Note: This class is specifically designed to support export process using `torch.export` in a way that ensures the model can be further lowered and run efficiently in `ExecuTorch`. c t<V^8dQhRS[RS[R,RS[R,RS[PR,RR/#rtrx)rrs"rr1TorchExportableModuleWithStaticCache.__annotate__s\OfOfOf$JOfTz Of  t# Of  Ofrc <\S V`4VPP4pVPpVf \ R4hVP 'g \ R4hVPR8wd \ R4hVPf/M VPpVf"VPRR4pVf \R4hVf"VPRR4pVf \R 4hVfVPR VP4pWn \W5R 7Vn\VPP 4F?wr\#V \$4'gK\'V4VPP V&KA \)V4wrVPP*p VPP-W*WV4\VPP 4FhwrVP/R V 2V P0R R7VP/RV 2V P2R R7VP/RV 2V P4R R7Kj R#)a  Initializes the wrapper module with the pretrained model. Args: model (`PreTrainedModel`): The pretrained model to wrap. The model must have caching enabled and use a 'static' caching implementation. batch_size (`Optional[int]`): The batch size of the model. If not provided, we check if a value can be found in `generation_config.cache_config` and otherwise we raise a ValueError. max_cache_len (`Optional[int]`): The maximum cache length for generation. Same mechanism as `batch_size` if not provided. device (`Optional[torch.device]`): The device to use. If not provided, we check if a value can be found in `generation_config.cache_config` and otherwise we use `model.device` (no error is raised). Raises: AssertionError: If the pretrained model does not have caching enabled or if it does not use a 'static' caching implementation in `model.generation_config`. ValueError: If `batch_size` or `max_cache_len` is not provided, either as an argument or in `cache_config`. NvThe model must have a generation config to be exported with static caching. Please set `generation_config` in `model`.zvThe model must have caching enabled to be exported with static caching. Please set `generation_config.use_cache=True`.staticzThe model must use a 'static' caching implementation to be exported with static caching. Please set `generation_config.cache_implementation='static'`.ruFbatch_size must be provided, either as an argument or in cache_config.rImax_cache_len must be provided, either as an argument or in cache_config.rv)rr key_cache_F persistent value_cache_cumulative_length_)r~r'rrgeneration_configAssertionErrorr{cache_implementation cache_configgetrrvrr static_cache enumeratelayers isinstancer rrr+early_initializationregister_bufferkeysvaluescumulative_lengthr&rrurrvrrrrrrrr+rs&&&&& rr'-TorchExportableModuleWithStaticCache.__init__s.2 --/!33  $ = !*** A   1 1X = P  /;;CrIZIgIg   %)),=J! !ijj  (,,_dCM$ !lmm >!%%h =F 'mS"$"3"3":":;HA%!9::.9-.H!!((+<.f5     ..zhW]^"$"3"3":":;HA  :aS!15::% P  <s!3U\\e T  #5aS!95;R;R_d e.++zz'+   4 " ";; )) )rc<V^8dQhRS[PPRS[PRS[RS[P/#)rrprompt_token_idsrbrw)r2r5rrr)rrs"rrrbsI2P2P,,662P,,2P2P  2Prc  VPpVPR,pWB,pVP4F<wrgVPR4'gKVP^,p\ WX4pM .p \ \ WT44Fp VP 4PVRW^,13,\P!V .\PVR7R7p V PV^,V ,P44K \P!X R,RR7P4p V PV 4\V 4V8dVP 4P\P!V ..\PVR7\P!\V 4.\PVR7R7p \P!V R,RR7P4p V PV 4K\P!V .\PVR7#)ay Generate a sequence of tokens using an exported program. This util function is designed to test exported models by simulating the generation process. It processes the input prompt tokens sequentially (no parallel prefill). This generate function is not intended to replace the original `generate` method, and the support for leveraging the original `generate` is potentially planned! Args: exported_program (`torch.export.ExportedProgram`): The exported program generated via `torch.export`. prompt_token_ids (`torch.Tensor`): Tensor representing the input prompt token IDs. max_new_tokens (`int`): Maximum number of new tokens to generate. Note that the total generation length is limited by both `max_new_tokens` and the model's cache size. Returns: torch.Tensor: A tensor containing the generated sequence of token IDs, including the original prompt tokens. key_cacherrrrrrrr)rvr named_buffers startswithrQrrr^r2rrOappendrrlen) rrrbrvprompt_token_lenmax_generation_length buffer_namebufferrresponse_tokens input_posresult current_tokens &&& rre-TorchExportableModuleWithStaticCache.generateas."((+11"5 0 A#3#A#A#C K%%k22 & Q (+,A(Q% $D s#8KLI%,,.66*1ia-.G+GH$||YKuzzRXY7F  " "#3A#6y#A#F#F#H I M VH%52>CCE }-/"%::%,,.66,,'8 SYZ$||S-A,B%**]cd7F"LL)9rBGGIM  " "= 1||_-UZZOOr)rrr) rkrlrmrnror'r^rrerprqrrs@@rrrs@OfOfb/*/*b2P2P2Prrc\aa]tRtRtoRtRV3RlV3RllltRV3RlRlltRtVtV;t #) riz A recipe module designed to make a `PreTrainedModel` exportable with `torch.export`, specifically for decoder-only LM to hybrid `StaticCache`. This module ensures that the exported model is compatible with further lowering and execution in `ExecuTorch`. c t<V^8dQhRS[RS[R,RS[R,RS[PR,RR/#rtrx)rrs"rr1TorchExportableModuleWithHybridCache.__annotate__s\BfBfBf$JBfTz Bf  t# Bf  Bfrc \<\S V`4WnVPP 4pVP pVf \ R4hVP'g \ R4hVPf/M VPpVf"VPRR4pVf \R4hVf"VPRR4pVf \R4hVfVPRVP4p\WSR 7Vn \VPP4F?wr\!V \"4'gK\%V4VPPV&KA \'V4wrVPP(p VPP+W*WV4\VPP4FhwrVP-R V 2V P.R R 7VP-R V 2V P0R R 7VP-RV 2V P2R R 7Kj R#)a[ Initializes the exportable module. Args: model (`PreTrainedModel`): The pretrained model to wrap. batch_size (`Optional[int]`): The batch size of the model. If not provided, we check if a value can be found in `generation_config.cache_config` and otherwise we raise a ValueError. max_cache_len (`Optional[int]`): The maximum cache length for generation. Same mechanism as `batch_size` if not provided. device (`Optional[torch.device]`): The device to use. If not provided, we check if a value can be found in `generation_config.cache_config` and otherwise we use `model.device` (no error is raised). Raises: AssertionError: If the model doesn't have the expected configuration for hybrid StaticCache. ValueError: If `batch_size` or `max_cache_len` is not provided, either as an argument or in `cache_config`. Nrz Model must have caching enabled.rurrrrvrrrFrrr)r~r'rrrrrr{rrrrvrcacherrrr rrr+rrrrrrs&&&&& rr'-TorchExportableModuleWithHybridCache.__init__s,  --/!33  $ =  !CD D.;;CrIZIgIg  %)),=J! !ijj  (,,_dCM$ !lmm >!%%h =F!L "$**"3"34HA%!9::'2='A !!!$5.f5     '' xPVW"$**"3"34HA  :aS!15::% P  <s!3U\\e T  #5aS!95;R;R_d e5rc<V^8dQhRS[PR,RS[PR,RS[PR,RS[P/#rr)rrs"rrr%sT!!##d*!||d*! t+ !  !rc VPPF%pVPPVR,4K' VP VVRVPRR7pVP #)a Forward pass of the module, which is compatible with the ExecuTorch llm runner. Args: input_ids (`torch.Tensor`): Tensor representing current input token id to the module. inputs_embeds (`Optional[torch.Tensor]`): Tensor representing current input embeddings to the module. cache_position (`torch.Tensor`): Tensor representing current input position in the cache. Returns: torch.Tensor: Logits output from the model. r NTr )r(rrr rr)r&rJrrKrrs&&&& rr^,TorchExportableModuleWithHybridCache.forwardse(ZZ&&E  # # ) ).*= >'**' JJ  ~~r)r(rr rkrlrmrnror'r^rprqrrs@@rrrs( BfBfH!!!rrc V^8dQhR\R\PR,R\PR,R\R,R\R,/#)rrexample_input_idsNexample_cache_positionr/r0)r r2rrr)rs"rrrsT? ? ? ||d*? "LL4/? 4K ? 4K ? rc ^RIpVP!4;_uu_4VeTM*VP!^..VPVPR7pVeTM)VP!^.VPVPR7p\ R4'd5VP P \V4R RVRV/TVeTMRR7pMkVe\P!R4Ve\P!R 4VP PP\V4R RVRV/R RR 7pVuuRRR4# +'giR#;i) a Convert a `PreTrainedModel` into an exportable module and export it using `torch.export`, ensuring the exported model is compatible with `ExecuTorch`. Args: model (`PreTrainedModel`): The pretrained model to be exported. example_input_ids (`Optional[torch.Tensor]`): Example input token id used by `torch.export`. example_cache_position (`Optional[torch.Tensor]`): Example current cache position used by `torch.export`. dynamic_shapes(`Optional[dict]`): Dynamic shapes used by `torch.export`. strict(`Optional[bool]`): Flag to instruct `torch.export` to use `torchdynamo`. Returns: Exported program (`torch.export.ExportedProgram`): The exported program generated via `torch.export`. Nrz2.6.0rJrKTrzWDynamic shapes spec will be ignored by convert_and_export_with_cache for torch < 2.6.0.zSThe strict flag will be ignored by convert_and_export_with_cache for torch < 2.6.0.F)r.r[ pre_dispatchr0rZ) torch.export._traceno_gradrrOrvr r5rrr_trace_export)rr/r0r/r0r2rs&&&&& rconvert_and_export_with_cacher7s:, !, se5::ellK &1 #qcELLI  %W - -$||224U;#%68HJ`a-!'!3v 3  )m! uv %||22::4U;#%68HJ`a" ;   O s DD<< E c<aa]tRtRtoRtV3RltRtRtVtV;t #) Seq2SeqLMEncoderExportableModuleiHz A wrapper module designed to make a Seq2Seq LM encoder exportable with `torch.export`. This module ensures that the exported encoder model is compatible with ExecuTorch. c0<\SV`4WnR#N)r~r'encoder)r& encoder_modelrs&&rr')Seq2SeqLMEncoderExportableModule.__init__Ns $ rc:VPVR7P#))rJ)r<r)r&rJs&&rr^(Seq2SeqLMEncoderExportableModule.forwardRs||i|0BBBr)r<r-rs@@rr9r9Hs %CCrr9c<aa]tRtRtoRtV3RltRtRtVtV;t #)/Seq2SeqLMDecoderExportableModuleWithStaticCacheiVz A wrapper module designed to make a Seq2Seq LM decoder exportable with `torch.export`, specifically for use with static caching. This module ensures the exported decoder is compatible with ExecuTorch. c<\S V`4VP4VnVPVnVP Vn\ VP44Pp\VP VR7Vn \VPP4F?wrV\V\4'gK\V4VPPV&KA \!VP 4wrxVPP#W7V\$P&V4\)VP\+VP R74Vn\/4\VPP4FhwrVVP1RV 2VP2RR7VP1RV 2VP4RR7VP1RV 2VP6RR7Kj R#)r'rrFrrrN)r~r' get_decoderdecoderlm_headrnext parametersrvrrrrrr rrrr2r4rrr(%register_dynamic_cache_export_supportrrrr) r&rmax_static_cache_lengthrurrrrrrs &&&& rr'8Seq2SeqLMDecoderExportableModuleWithStaticCache.__init__]s ((* }} ll E,,./66 (t{{Jab"$"3"3":":;HA%!9::.9:Q.R!!((+<.dkk:  ..zhPUP]P]_kl():):LPTP[P[<\] -/"$"3"3":":;HA  :aS!15::% P  <s!3U\\e T  #5aS!95;R;R_d e.,,'"7 JJ  LL, r)r(rrFrGrr-rs@@rrBrBVs f>rrBcRaa]tRtRtoRV3RlltRtRtR RltRtRt Vt V;t #) Seq2SeqLMExportableModuleic <\SV`4WnVP4VnVP VnW0n\RVVRVRV/VPPR7VnRVn RVn R#)Trur)r{ max_lengthrrrN) r~r' full_model get_encoderr<rmax_hidden_seq_lengthr rrexported_encoderexported_decoder)r&rrurXrmax_cache_lengthrs&&&&&&rr'"Seq2SeqLMExportableModule.__init__s ((* ll %:"!1'!5j!100== " !% $rc \VP4PVPP4P 4p\ PPRVPR7p\ P!4;_uu_4\ PPW!3R^V//RR7pRRR4V# +'giX#;i)encoder_seq_lengthrHrJTr/r0N) r9r<rrVrvr1r2r5r6rXr4)r&encoder_input_idswrapped_encoderrVrYs&& r_export_encoder)Seq2SeqLMExportableModule._export_encoders:4<<HKKDOOLbLbchhjll&&';A[A[&\ ]]__$||22!5{UVXcTdFenr 3     _  s (C C c VPPp\VPVPPP R4VPPP R4R7P V4P4pVP V4pVP V4pVP V4p\PPRVPR7p\P!4;_uu_4\PPVWV3RRR^V/RR/R R 7pRRR4V# +'giX#;i) rru)rrKruencoder_hidden_seq_lengthrHrONrNrKTr_) rVrvrBrrrrr1r2r5r6rXr4)r&rOrNrK target_devicewrapped_decoderencoder_seq_len_dimrZs&&&& r_export_decoder)Seq2SeqLMExportableModule._export_decoders3.. ;oo(,(>(>(K(K(O(OP_(`11>>BB<P  R  TV .00? 5 8 8 G'**=9$ll../JPTPjPj.k]]__$||22">J'+a1D-E$d   3   _ s .E E cnVPPpVeTM&\P!R\PVR7pVeTM(\P !^..\PVR7pVeTM'\P !^.\PVR7pVeTM`\P !VPPPR4^ VPP3\PVR7p VPV4VnVPWyV4VnV#)Nrru)rg )rVrvr2onesrOrrNrrrrd_modelr4rbrYrirZ) r&r`rOrNrKrvexample_encoder_input_idsexample_decoder_input_idsr0example_encoder_hidden_statess &&&&& rr5 Seq2SeqLMExportableModule.exports''!, G5::fE "!, se5::fE " -8NellA3V[V`V`io>p  %0 "''4488FDKKL_L_`mm &!% 4 45N O $ 4 4 %F\!   rc L\P!4;_uu_4VPPpVPV8wdVP V4pVP P 4!V4p\P!^..\PVR7p^.p\V^, 4FpVPP 4!WT\P!V.\PVR74p\P!VR,RR7P4p VPV 4\P!V ..\PVR7pWPP8XgKM VuuRRR4# +'giR#;i)rrrNrr)r2r4rVrvrrYrrrOrrZrrrrr) r&rrbrencoder_outputrOrrr next_tokens &&& rre"Seq2SeqLMExportableModule.generates: ]]__??11L &&,6#3#6#6|#D "2299;A-...557%u||QCuzzbn7o #\\&*:CHHJ $$Z0%*LL:,uzzZf$g!!7!7!D!DD/"!A___sEFF F# )rr<rZrYrVrrX)rgirrhNNNN) rkrlrmrnr'rbrir5rerprqrrs@@rrSrSs%%,  ! F@!!!!rrScV^8dQhR\R\PR,R\PR,/#)rrr/Nexample_attention_mask)r r2r)rs"rrrs8   ||d* "LL4/ rc\4\P!4;_uu_4\PPVR RVRVR\ VP R7RR/RR7pVuuR R R 4# +'giR #;i) a Export a model with DynamicCache using `torch.export`, ensuring the exported model is compatible with `ExecuTorch`. Args: model (`PreTrainedModel`): The pretrained model to be exported. example_input_ids (`Optional[torch.Tensor]`): Example input token id used by `torch.export`. example_attention_mask (`Optional[torch.Tensor]`): Example attention mask used by `torch.export`. Returns: Exported program (`torch.export.ExportedProgram`): The exported program generated via `torch.export`. rJr r rDr{TF)r0NrZ)rJr2r4r5rr)rr/ryrs&&& rexport_with_dynamic_cacher{so"*+  <<..  . "8! torch.export support cf\PPP\ V44#r;)r2utils_pytree _dict_flatten_get_cache_dict dynamic_caches&r7register_dynamic_cache_export_support..Gs%++"5"5"C"COTaDb"cr.cf\PPP\ V44#r;)r2r~r_dict_flatten_with_keysrrs&rrrJs!u{{7J7J7b7b .8r)serialized_type_nameflatten_with_keys_fnch\PPP\ V4V4#r;)r2fxr_dict_flatten_specr)r(specs&&rrrQs" 0 0 C COTYDZ\` arz!already registered as pytree nodeN) r2r~rregister_pytree_noder_unflatten_dynamic_cacherlrkrregister_pytree_flatten_specrr)es rrJrJ?s  00  c $$0$;$;#S>S=T!U" 1  55  a  .c!f <  =sBB B3B..B3c$V^8dQhR\/#)rr()r)rs"rrrYs  < rc\;QJd&RVP4F 'gK RM RM!RVP44'd \R4h\'g\P !R4RVPUu.uFqP fKVP NK upRVPUu.uFqPfKVPNK up/#uupiuupi)z9Convert cache to dictionary format for pytree operations.c3X"TF p\V\\34'*xK" R#5ir;)rrr).0rs& r "_get_cache_dict..[s& fYePUz%,0I!JK K KYes(*TFzFThis pytree flattening function should only be applied to DynamicCachez[DynamicCache + torch.export is tested on torch 2.6.0+ and may not work on earlier versions.r value_cache)anyr RuntimeErrorr rrrr)r(rs& rrrYs s fY^YeYe fsss fY^YeYe fffcdd - -uv ellUlUjjjejjlU%,,[,,,  ,[ U[s C(C(C- C- c`V^8dQhR\PPP/#)rcontext)r2r~rContext)rs"rrrgs#  ekk.A.A.I.I rc\PPPW4p\ 4pVP R.4pVP R.4p\ \\V4\V444FIpV\V48d WF,MRpV\V48d WV,MRpVPWxV4KK V#)rrN) r2r~r_dict_unflattenrrrrIrupdate) rr dictionaryr(key_list value_listidxkeyvalues && rrrgs$$44VEJ NE~~k2.H r2JSXJ89"S]2hm#&Z#8 d S%: Lrrw)NN) rr2 cache_utilsrrrrrrr generation.configuration_utilsr modeling_utilsr pytorch_utilsr r rnnModulerLrrrr7r9rBrSr{rJrrrZrrrs >, U U pALEHHOOALH,@P588??@PFl588??l^? D Cuxx C8ehhoo8vI!I!X D4  r