)ti)ddlZddlZddlmZmZddlmZmZddlm Z ddl m Z ddl m Z mZmZmZmZmZmZe rddlZd Zd Zd Zy) N) SAGEMAKER_PARALLEL_EC2_INSTANCESTORCH_DYNAMO_MODES)ComputeEnvironmentSageMakerDistributedType)is_boto3_available)SageMakerConfig)DYNAMO_BACKENDS _ask_field _ask_options_convert_dynamo_backend_convert_mixed_precision#_convert_sagemaker_distributed_mode_convert_yes_no_to_boolcdtjd}ddddiddgd} |j|tj|d  ddgd d dgd}|j ||dtj|d y#|j j$rtd|dYywxYw)Niamz 2012-10-17AllowServicezsagemaker.amazonaws.comzsts:AssumeRole)Effect PrincipalAction)Version Statement)indent)RoleNameAssumeRolePolicyDocument)z sagemaker:*zecr:GetDownloadUrlForLayerzecr:BatchGetImagezecr:BatchCheckLayerAvailabilityzecr:GetAuthorizationTokenzcloudwatch:PutMetricDatazcloudwatch:GetMetricDatazcloudwatch:GetMetricStatisticszcloudwatch:ListMetricszlogs:CreateLogGroupzlogs:CreateLogStreamzlogs:DescribeLogStreamszlogs:PutLogEventszlogs:GetLogEventszs3:CreateBucketz s3:ListBucketzs3:GetBucketLocationz s3:GetObjectz s3:PutObject*)rrResource_policy_permission)r PolicyNamePolicyDocumentzrole z# already exists. Using existing one) boto3client create_rolejsondumpsput_role_policy exceptionsEntityAlreadyExistsExceptionprint) role_name iam_clientsagemaker_trust_policypolicy_documents s/Volumes/fast/ai/experiments/MLX_z-image/.venv/lib/python3.12/site-packages/accelerate/commands/config/sagemaker.py_create_iam_role_for_sagemakerr2&se$J i9R-S_o p  *FDZcd9e  $&*!$/ > ""#$67::oa@ #  = =F i[ CDEFsA B(B/.B/c\tjd}|j|ddS)Nr)rRoleArn)r$r%get_role)r-r.s r1_get_iam_role_arnr7\s.e$J     26 :5 AAc2tdddgt}d}|dk(r!tdd}|tjd<nGt d td }|tjd <td }|tjd <tdd}|tjd<tdddgt}|dk(r td}nd}t d|dt |tdtdd}d}|r tdd}tdtdd} d} | r tdd } td!tdd} d} | r td"d#} td$d%d&gt} i}td'tdd}|rd(}td)tDcgc]}|jc}td*||d+z<td,tdd}|retd-td.d||d/z<td0tdd||d1z<td2tdd||d3z<td4tdd||d5z<d6}| tjk7rt|td7}n|d8z }t|d9d:}d}| tjk7rtd;tdd}d<}| tj tj"fvrtd=td<}td>gd?t$}|r|d@k(r t dAt'|t(j*| d|||||||| | |BScc}w)CNzHow do you want to authorize?z AWS Profilez7Credentials (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY) rz'Enter your AWS Profile name: [default] default)r: AWS_PROFILEzNote you will need to provide AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY when you launch you training script with,`accelerate launch --aws_access_key_id XXX --aws_secret_access_key YYY`zAWS Access Key ID: AWS_ACCESS_KEY_IDzAWS Secret Access Key: AWS_SECRET_ACCESS_KEYz"Enter your AWS Region: [us-east-1]z us-east-1AWS_DEFAULT_REGIONzMDo you already have an IAM Role for executing Amazon SageMaker Training Jobs?zProvide IAM Role namez%Create new IAM role using credentialszEnter your IAM role name: #accelerate_sagemaker_execution_rolez$Accelerate will create an iam role "z " using the provided credentialsz2Do you want to use custom Docker image? [yes/NO]: FzPlease enter yes or no.)r: error_messagezEnter your Docker image: c4t|jSNstrlowerxs r1z%get_sagemaker_input..sQr8zODo you want to provide SageMaker input channels with data locations? [yes/NO]: z\Enter the path to the SageMaker inputs TSV file with columns (channel_name, data_location): c4t|jSrBrCrFs r1rHz%get_sagemaker_input..c!fllnr8z3Do you want to enable SageMaker metrics? [yes/NO]: z[Enter the path to the SageMaker metrics TSV file with columns (metric_name, metric_regex): c4t|jSrBrCrFs r1rHz%get_sagemaker_input..rJr8zWhat is the distributed mode?zNo distributed trainingzData parallelismz?Do you wish to optimize your script with torch dynamo?[yes/NO]:dynamo_z+Which dynamo backend would you like to use?rbackendzGDo you want to customize the defaults sent to torch.compile? [yes/NO]: zWhich mode do you want to use?c&tt|SrB)rintrFs r1rHz%get_sagemaker_input..s,SV4r8modez\Do you want the fullgraph mode or it is ok to break model into several subgraphs? [yes/NO]: use_fullgraphz7Do you want to enable dynamic shape tracing? [yes/NO]: use_dynamicz6Do you want to enable regional compilation? [yes/NO]: use_regional_compilationz:Which EC2 instance type you want to use for your training?c&tt|SrB)rrOrFs r1rHz%get_sagemaker_input..sLlmpqrmsLtr8z? [ml.p3.2xlarge]:c4t|jSrBrCrFs r1rHz%get_sagemaker_input..sSV\\^r8z ml.p3.2xlargezShould distributed operations be checked while running for errors? This can avoid timeout issues but will be slower. [yes/NO]: r z(How many machines do you want use? [1]: z2Do you wish to use FP16 or BF16 (mixed precision)?)nofp16bf16fp8rVzTorch dynamo used without mixed precision requires TF32 to be efficient. Accelerate will enable it by default when launching your scripts.) image_uricompute_environmentdistributed_typeuse_cpu dynamo_configec2_instance_typeprofileregion iam_role_namemixed_precision num_machinessagemaker_inputs_filesagemaker_metrics_filedebug)r rOr osenvironr,r2rrr rErrrNOr DATA_PARALLELMODEL_PARALLELrr rAMAZON_SAGEMAKER)credentials_configuration aws_profileaws_access_key_idaws_secret_access_key aws_regionrole_managementrbis_custom_docker_image docker_imageis_sagemaker_inputs_enabledreis_sagemaker_metrics_enabledrfr\r^ use_dynamoprefixrGuse_custom_optionsec2_instance_queryr_rgrdrcs r1get_sagemaker_inputr|as ,' QR ! K A% !JT]^ $/ =!  V ''<=*; &' *+D E.C *+@+VJ'1BJJ#$"W "IJ O !"#?@ =  4]OCcde&}5'</  L!"=?WX ",Y/ # !" * j $!  $.=/ $ "#!+ i $"  $' "$67+ MI/ J ,8 9 / 01QWWY 0 # - fy() ( U #3   -90"4! .M&6/ * 7An'7 7M&?2 3 5?I'7 5M&=0 1 BLH'7 BM&#== >V3666(  @Bt  22&'9;S]lm E3666 N #3  L4BBD\DkDkll! 6  #<% O o-  Y  .??)#+#'!35 U 1s L )r'rhutils.constantsrrutils.dataclassesrr utils.importsr config_argsr config_utilsr r r rrrrr$r2r7r|r8r1rsF  SM/(3FlB qr8