+ QV-j.%^RIHt^RIHt^RIHtHt^RIHt^RI H t ^RI H t ]!4'd^RI t ]'d^RIHt]P !]4t!R R ] 4tR#) ) annotations) TYPE_CHECKING)is_torch_availablelogging) SinqConfig) HfQuantizer)get_module_from_nameN)PreTrainedModelca]tRt^!t$RtRtR]R&R]R&RV3RlltR R lt] R R l4t R t RRlt RRlt RRltRRltRtRtRRRlltRRltRtV;t#)SinqHfQuantizera| HF v5 quantizer for SINQ. Modes: - method="sinq" (default): * weight-only SINQ * param-level ConversionOps (`SinqQuantize`) during load for pure language models (each Linear.weight is turned into a SINQLinear module) * module-level quantization after load for multimodal models - method="asinq": * A-SINQ (activation-aware) SINQ quantization Tbool requires_parameters_quantizationrquantization_configcV^8dQhRR/#)rr)formats"w/Volumes/fast/ai/experiments/ui-tars-smoke/.venv/lib/python3.14/site-packages/transformers/quantizers/quantizer_sinq.py __annotate__SinqHfQuantizer.__annotate__2s00J0c F<\SV`!V3/VBRVnRVnR#)NF)super__init___normalized_device_str_do_param_level_sinq)selfrkwargs __class__s&&,rrSinqHfQuantizer.__init__2s& ,7726#*/!rcV^8dQhRR/#rreturnr r)rs"rrr8src R#Trrs&ris_serializableSinqHfQuantizer.is_serializable8srcV^8dQhRR/#r"r)rs"rrr<sdrc R#r%rr&s&r is_trainableSinqHfQuantizer.is_trainable;src Vfc\PP4'd"R\PP4/pMRR/p\P RV R24V#)Ncpuz:The device_map was not initialized. Setting device_map to zJ. If you want to use the model for inference, please set device_map='auto')torchcuda is_availablecurrent_deviceloggerinfo)r device_maps&&rupdate_device_map!SinqHfQuantizer.update_device_map?se  zz&&(( %**";";"=>  %[ KK))3 5[[  rc V^8dQhRRRR/#)rdtypez torch.dtyper#r)rs"rrrLs++rc :Vf\PpWnV#N)r0bfloat16r:)rr:s&&r update_dtypeSinqHfQuantizer.update_dtypeLs =NNE  rcV^8dQhRR/#)rr#Noner)rs"rrrRstrc ^RIHpV!4'g \R4h\PP 4'g\ PR4VPR4p\V\4'dB\VP44p\V4^8d\R\V4 R24hVP P"R8Xd VP$'g \'R4hR #R #) r)is_sinq_availablezMThe 'sinq' package is not installed. Please install it with: pip install sinqzNo CUDA device is available. Quantization and inference will run on the CPU. Please note that this will significantly slow down inference speed and increase quantization time.r6zkSinqHfQuantizer: multi-GPU device_map detected, but SINQ currently supports only a single CUDA device. Got z. Please use device_map=None.asinqzYou are using `method='asinq'` in the quantization config. Right now the calibrated version of SINQ is not supported in Hugging Face, please refer and use the official SINQ repository `to quantize a model with this method. N)utilsrC ImportErrorr0r1r2r4warningget isinstancedictsetvalueslen RuntimeErrorsortedrmethod pre_quantized ValueError)rargsrrCr6device_map_valuess&*, rvalidate_environment$SinqHfQuantizer.validate_environmentRs- ""mn nzz&&(( NNB ZZ - j$ ' ' #J$5$5$7 8 $%)"##)*;#<"==Z\  # # * *g 5d>P>P>P: ?Q 5rc V^8dQhRRRR/#)rcfgrr#rJr)rs"rrrns  *  rc  ^RIHpVPpT!\VP4VP e\VP 4MRRRR^\ VP4VR7#)z9 Build the dict that SINQLinear expects as quant_config. )sinq_base_quant_configNF)nbits group_size quant_zero quant_scale view_as_floataxis tiling_moderP)sinq.sinqlinear_hfrZrPintr[r\strra)rrXsinq_base_quant_config_fnrPs&& r_build_sinq_quant_dict&SinqHfQuantizer._build_sinq_quant_dictns[ [(cii..1nn.Hs3>>*dCOO,  rc$V^8dQhRRRRRR/#)rmodelr param_namerdr#r r)rs"rrrs"  o 3 ]a rc ^RIHpVP'dR#VPPR8XdR#VP 'gR#\ W4wrVVR8wdR#\WT4p\VRR4pT;'dV'*p V #)a Called per-parameter to decide whether to run `SinqQuantize` on it. - If `self.pre_quantized`, we do *not* quantize again (handled by SinqDeserialize instead). - For method="asinq": return False (ASINQ is not supported in Hugging Face). - For method="sinq": True only for SINQLinear.weight not in modules_to_not_convert. Note: After _process_model_before_weight_loading(), the modules are already SINQLinear, not nn.Linear. We check for SINQLinear modules that are not yet quantized (ready=False). ) SINQLinearFrDweightreadyT) rbrlrQrrPrr rIgetattr) rrirjrrlmodule tensor_nameis_sinqis_readyresults &&&, rparam_needs_quantization(SinqHfQuantizer.param_needs_quantizations 2      # # * *g 5(((25E ( "V067D1))\ rc ^RIHpV!V4#)z Return the ConversionOps used for param-level quantization (Sinq). The actual SINQLinear construction is in integrations/sinq.py. ) SinqQuantize)integrations.sinqrx)rrxs& rget_quantize_ops SinqHfQuantizer.get_quantize_opss 5D!!rc n^RIHpVP'd^RIHpV!.ROR.V!V4.R7.#.#)z If `pre_quantized=True`, interpret a checkpoint produced by SINQLinear.state_dict: .W_q .bias .meta via a WeightConverter + SinqDeserialize so that we reconstruct a SINQLinear module instead of a plain nn.Linear. )WeightConverter)SinqDeserializez.weight)source_patternstarget_patterns operations)z.W_qz.metaz.bias)core_model_loadingr}rQryr~)rr}r~s& rget_weight_conversions&SinqHfQuantizer.get_weight_conversionssH 9     ; % &/K / 56   rc V^8dQhRRRR/#)rrir keep_in_fp32_moduleszlist[str] | Noner)rs"rrrs) ) ) / ) rc ^RIHpTPYPP;'g.V4VnVPP R8H;'dVP '*VnVP 'dRMVPVP4p\V\4'dL\\VP44^4p\V\4'dRV 2pM4\V4pM(\ P"P%4'dRMRpV!VVPVVP&VVP R7pR#)z Called on meta-initialized model, before loading any weights. For SINQ, we replace nn.Linear modules with empty SINQLinear modules here. The actual quantization happens later in SinqQuantize.convert() when weights are loaded. )replace_with_sinq_linearsinqNzcuda:zcuda:0r/)modules_to_not_convert quant_config compute_dtypedevicerQ)ryrget_modules_to_not_convertrrrPrQrrfrIrJnextiterrLrcrdr0r1r2r:) rrir6rrrsinq_quant_dict first_device device_strs &&&&, r$_process_model_before_weight_loading4SinqHfQuantizer._process_model_before_weight_loadings A&*&E&E ,,CCIIrL`' # %)$<$<$C$Cv$M$h$hVZVhVhRh!"&"4"4"4$$:U:UVZVnVn:o j$ ' 'Z%6%6%8 91=L,,,$\N3  . %*ZZ%<%<%>%>EJ( #'#>#>(**,,  rcV^8dQhRR/#)rrir r)rs"rrrsrc ^RIHpV!4V#)a9 Called after *all* weights have been loaded. For SINQ: 1. Move non-SINQLinear modules to GPU (embeddings, norms, lm_head, etc.) - SINQLinear modules already have GemLite buffers on GPU - We skip moving SINQLinear's W_q/meta to avoid memory duplication 2. Patch HF save/load methods for SINQ serialization )patch_hf_pretrained_io) sinq.hf_ior)rrirrs&&, r#_process_model_after_weight_loading3SinqHfQuantizer._process_model_after_weight_loadings 6   r)rrr:rr<)__name__ __module__ __qualname____firstlineno____doc__r__annotations__rr'propertyr+r7r>rUrfrurzrrr__static_attributes__ __classcell__)rs@rr r !sr .2$d1##00   8 $ D":) Vrr ) __future__rtypingrrErrutils.quantization_configrbaserquantizers_utilsr r0modeling_utilsr get_loggerrr4r rrrrsK# /220   H %eker