ipO dZddlZddlZddlmZddlmZddlm Z ddl m Z m Z m Z ddlmZddlmZdd lmZdd lmZmZdd lmZmZdd lmZdd lmZddlmZm Z ddl!m"Z"m#Z#edd\Z$Z%ee$e%d\Z$Z%ejMe$Z$gdZ'dhejPDchc] \}}|d c}}zZ)dRdZ*ejVjYdedZ-dZ.ejVjYdge#e"dZ/dZ0ejVjYde'ejVjYd ed!Z1d"Z2ejVjYd#d$d%Z3d&Z4d'Z5d(Z6d)Z7ejVjYd*d+d,gd-Z8ejVjYd.e#d/Z9ejVjYd0e'd1Z:d2Z;d3ZZCd?ZDejVjYd@dAdBgejVjYdCddDgdEZEdFZFejVjYdGdHdIgdJZGejVjYdKdLdMgdNZHejVjYdOd+d,gdPZIdQZJycc}}w)SzF Tests for HDBSCAN clustering algorithm Based on the DBSCAN test code N)stats)distance)HDBSCAN)CONDENSED_dtype_condense_tree _do_labelling)_OUTLIER_ENCODING) make_blobs)fowlkes_mallows_score)_VALID_METRICSeuclidean_distances)BallTreeKDTree)StandardScaler)shuffle)assert_allcloseassert_array_equal)CSC_CONTAINERSCSR_CONTAINERS ) n_samples random_state)r)kd_tree ball_treebruteautolabelcrtt|tz }|dk(sJt|t|kDsJy)N)lenset OUTLIER_SETr y)labels threshold n_clusterss w/Volumes/fast/ai/experiments/voice-extract-mac/.venv/lib/python3.12/site-packages/sklearn/cluster/tests/test_hdbscan.pycheck_label_qualityr+)s6S[;./J ??  +i 77 7 outlier_typectjtjd|}ddd|}t|d}t|d}tj }|dg|d<||g|d<t d j|}|j|k(j\}t|ddg||j|j\}t|ddgttddttd d z} t d j|| } t| j|j| y )O Tests if np.inf and np.nan data are each treated as special outliers. )infinitemissingc ||k(SNxr&s r*z#test_outlier_data..9sar,c,tj|Sr3)npisnanr5s r*r7z#test_outlier_data..:s  r,r probrFcopyrN)r9infnanr Xr?rfitlabels_nonzerorprobabilities_listrange) r-outlier prob_checkr r; X_outliermodelmissing_labels_idxmissing_probs_idx clean_indices clean_models r*test_outlier_datarR/sB FF66G (+J l +G 4E \ *6 2DIQ)Aq62&u';';TBKKM(1a&1q!%U1c](;;Mu%)))M*BCK{**EMM-,HIr,ctt}|j}tddj |}t ||t |d}tjt|5tddj tdddd}d|d <d |d <tjt|5tdd j |dddy#1swYVxYw#1swYyxYw) zy Tests that HDBSCAN works with precomputed distance matrices, and throws the appropriate errors when needed. precomputedTmetricr?z*The precomputed distance matrix.*has shapematchNz'The precomputed distance matrix.*valuesr)rr<r<)r<rF) r rCr?r fit_predictrr+pytestraises ValueError)D D_originalr'msgs r*test_hdbscan_distance_matrixr`Os AAJ M 5 A A! DFAz" 7C z -}40<N)rrYrCr+r's r*test_hdbscan_feature_arrayrnys# % , ,Q /Fr,algorVct|djt}t||dvrytt d}dt jtjdidt jtjdidd id t jtjdd d j|d}t|||d }|||jvr8tjt5|jtdddy|d k(r8tj t"5|jtdddy|jty#1swYyxYw#1swYyxYw)z Tests that HDBSCAN works with the expected combinations of algorithms and metrics, or raises the expected errors. F) algorithmr?)rrN)rrVr<p)rsw) mahalanobis seuclidean minkowski wminkowski)rqrV metric_paramsr?ry)rrYrCr+rrr9eyeshapeonesget valid_metricsrZr[r\rDwarns FutureWarning)rorVr' ALGOS_TREESrzhdbs r*test_hdbscan_algorithmsrs>t% 0 < F c~tdjt}|jd}t |dy)z Tests that HDBSCAN can generate a sufficiently accurate dbscan clustering. This test is more of a sanity check than a rigorous evaluation. Fr>333333?gq= ףp?)r(N)rrDrCdbscan_clusteringr+) clustererr's r*test_dbscan_clusteringrs5 U#''*I  ( ( -F$/r, cut_distance)皙??r<ctdd}tdd}tj}tjdg|d<dtj g|d<tjtj g|d<t d j|}|j| }tj||k(}t|ddgtj||k(}t|dgtttd t||zz }t d j||} | j| } t| ||y ) r/r1r r0r<rrtr=Fr>)rrN)r rCr?r9rArBrrDr flatnonzerorrHr$rI) r missing_labelinfinite_labelrLrMr'rNinfinite_labels_idx clean_idxrQ clean_labelss r*#test_dbscan_clustering_outlier_datars0 &i09M&z27;NIFFA;IaLrvv;IaLFFBFF#IaL   # #I .E  $ $, $ ?F-(?@)Aq62..>)AB*QC0Ss_s+=@S+S'TTUIu%)))I*>?K00l0KL|VI%67r,ctddtjtjdidj t}t |y)z4 Tests that HDBSCAN using `BallTree` works. rwrrr<F)rVrzr?N)rr9r}rCr|rYr+rms r*!test_hdbscan_best_balltree_metricrsAC1D+EEk!n r,ctttdz djt}t |j t sJy)z Tests that HDBSCAN correctly does not generate a valid cluster when the `min_cluster_size` is too large for the data. r<Fmin_cluster_sizer?N)rr#rCrYr$issubsetr%rms r*test_hdbscan_no_clustersrs; c!fqju = I I! LF v;   ,, ,r,c.tdttdD]s}t|dj t}|Dcgc] }|dk7s | }}t|dk7sGt j t j||k\rsJycc}w)zb Test that the smallest non-noise cluster has at least `min_cluster_size` many points rtr<FrrrN)rIr#rCrrYr9minbincount)rr'r true_labelss r*test_hdbscan_min_cluster_sizers "!SVQ/*:GSSTUV*0@&ERKu& @ { q 66"++k237GG GG 0@s  B Bcztj}t|djt}t |y)zA Tests that HDBSCAN works when passed a callable metric. FrUN)r euclideanrrYrCr+)rVr's r*test_hdbscan_callable_metricrs.  F F / ; ;A >Fr,treerrctd|d}d}tjt|5|j t dddy#1swYyxYw)z Tests that HDBSCAN correctly raises an error when passing precomputed data while requesting a tree-based algorithm. rTFrVrqr?z%precomputed is not a valid metric forrWN)rrZr[r\rDrC)rrr_s r*"test_hdbscan_precomputed_non_bruters< $U CC 1C z -   . - -s A  A csr_containercBtdjtj}t ||t}|j }tdj|j}t ||tjdftjdffD]\}}tj }||d<tdj|j}t ||dt|dk(sJ|j }||d<tdj|j}t ||d}tjt| 5td d d j|d d d y #1swYy xYw)z Tests that HDBSCAN works correctly when passing sparse feature data. Evaluates correctness by comparing against the same data passed as a dense array. Fr>r0r1rrrr z4Sparse data matrices only support algorithm `brute`.rWrrrN)rrDrCrEr+r?rr9rArBr rZr[r\) r dense_labels _X_sparseX_sparse sparse_labels outlier_valr-X_denser_s r*test_hdbscan_sparser sR&**1-55L %a I~~H'++H5==M|]3(*vvz&:RVVY>#$U+//9AA <7&Q AC z -{kFJJ8T . - -s .FFrqcddg}tdd|d\}}tdd j|}t||j|j D]$\}}}t ||d d t ||d d &t|dtjdd jt}|jjddk(sJ|j jddk(sJy)zj Tests that HDBSCAN centers are calculated and stored properly, and are accurate to the data. )rdrd)@rirr)rrcenters cluster_stdbothF) store_centersr?r<g?)rtolatol)rqrrr?N) r rrDzip centroids_medoids_rrCr|)rqrH_rcentercentroidmedoids r*test_hdbscan_centersr.s :&G 1gSV WDAq U 3 7 7 :C$'$N &qt<QT:%O      c!f  >>   "a '' ' <<  a A %% %r,ctjjd}|jdd}t ddddd j |}tj |d \}}t|dk(sJ||d k(d kDsJt dd ddddj |}tj |d \}}t|dk(sJ||d k(dk(sJy)zS Tests that HDBSCAN single-cluster selection with epsilon works correctly. rrtr=rdeomTF)rcluster_selection_epsiloncluster_selection_methodallow_single_clusterr?) return_countsrg ףp= ?r)rrrrrqr?N)r9random RandomStaterandrrYuniquer#)rng no_structurer' unique_labelscountss r*.test_hdbscan_allow_single_cluster_with_epsilonrGs ))   "C88C#L "%!&!   k,  IIfDAM6 }  "" " -2% & ++ +"&!&!  k, IIfDAM6 }  "" " -2% &! ++ +r,cddgddgddgddgg}td|gdd\}}td j|j}t t |t d |vz }|d k(sJt||d kDy)z Validate that HDBSCAN can properly cluster this difficult synthetic dataset. Note that DBSCAN fails on this (see HDBSCAN plotting example) g333333g333333?r"i)皙?gffffff?皙?rr)rrrrFr>rGz?N)r rrDrEr#r$intr )rrCr&r'r)s r*test_hdbscan_better_than_dbscanrjs u~t}q!fq"g>G +  DAq % $ $Q ' / /FS[!Cf $55J ??&!$t+r,z kwargs, XrTr<rtr"rc>tdddd|j|y)zo Tests that HDBSCAN works correctly for array-likes and precomputed inputs with non-finite points. r<F min_samplesr?Nr4)rrD)rCkwargss r*test_hdbscan_usable_inputsr~s! 00044Q7r,c|tjd}d}tjt|5t ddj |dddy#1swYyxYw)zd Tests that HDBSCAN raises the correct error when there are too few non-zero distances. )rrz#There exists points with fewer thanrWrTFrUN)r9zerosrZr[r\rrDrrCr_s r*-test_hdbscan_sparse_distances_too_few_nonzerorsK bhhx()A /C z -}5155a8 . - -s AA(c$tjd}d|ddddf<d|ddddf<||jz}||}d}tjt |5t dd j|dddy#1swYyxYw) zu Tests that HDBSCAN raises the correct error when the distance matrix has multiple connected components. )rr<Nr=z3HDBSCAN cannot be performed on a disconnected graphrWrTFrU)r9rTrZr[r\rrDrs r*0test_hdbscan_sparse_distances_disconnected_graphrs AAbqb"1"fIAab"#gJ ACCAaA ?C z -}5155a8 . - -s BBcd}d}tjt|5td|dj t dddtjt|5td|dj t dddt ttjttjz }t|d kDrItjt|5td|d dj t dddyy#1swYxYw#1swYxYw#1swYyxYw) zR Tests that HDBSCAN correctly raises an error for invalid metric choices. c|Sr3r4)r6s r*r7z2test_hdbscan_tree_invalid_metric..sr,zV.* is not a valid metric for a .*-based algorithm\. Please select a different metric\.rWrF)rqrVr?Nrr) rZr[r\rrDrCrHr$rrrr#)metric_callabler_metrics_not_kds r* test_hdbscan_tree_invalid_metricrs"O  z -)O%HLLQO . z -+oEJNNqQ . #h445FQ ]]:S 1 iq0A N R RST U2 1 . - - -2 1s#"D$&"D05%D<$D-0D9<Ectttdzd}d}tjt |5|j tdddy#1swYyxYw)zx Tests that HDBSCAN correctly raises an error when setting `min_samples` larger than the number of samples. r<Frz min_samples (.*) must be at mostrWN)rr#rCrZr[r\rD)rr_s r*!test_hdbscan_too_many_min_samplesrsB c!fqju 5C -C z -   . - -s AA#ctj}tj|d<d}t dd}t j t|5|j|dddy#1swYyxYw)zu Tests that HDBSCAN correctly raises an error when providing precomputed distances with `np.nan` values. rz(np.nan values found in precomputed-denserTFrUrWN) rCr?r9rBrrZr[r\rD)X_nanr_rs r*"test_hdbscan_precomputed_dense_nanrsR FFHE&&E$K 4C U 3C z -  . - -s A--A6rTFepsilonrcTd}t||ddgddgddgg\}}tdj|}t|j|j }|dz|d z|d zh}|dzd|d zd |d zdi} t ||| || } tt|D cic]!} | tj|| k(dd#} } tt|D cic] } | | | |  } } tj| j|}t| |y cc} wcc} w)zR Tests that the `_do_labelling` helper function correctly assigns labels. 0rr)rrFr>rrtr"rr<condensed_treeclusterscluster_label_maprrN)r rrDr_single_linkage_tree_rrrHr$r9where vectorizer~r)global_random_seedrrrrCr&estrrrr'_yfirst_with_label y_to_labelsaligned_targets r*test_labelling_distinctrsQ I 'F G G DAq u  ! !! $C# !!C4H4HNA y1}i!mB3q6lKl2v.r233lKK2R\\+//215Nv~.LKs &D D%cLd}d}tjdd|dfddd|dfddgt }t||h|d|dzdid d }|d dk}t |t |d k(k(sJt||h|d|dzdid d }|d |k}t |t |d k(k(sJy)z Tests that the `_do_labelling` helper function correctly thresholds the incoming lambda values given various `cluster_selection_epsilon` values. r=g?rtr<)r=r<rr<r)r=r"rr<)r=rrr<)dtypeTrvaluerN)r9arrayrrsum)r MAX_LAMBDArr' num_noises r*test_labelling_thresholdingr s IJXX :q !  :q !     N%$aQ:!"# Fw'!+I y>S2. .. . %$aQ:!"# Fw'*4I y>S2. .. .r,rrrctjjd}|jd}t|}d}t j t |5td|dj|dddy#1swYyxYw) zCheck that we raise an error if the centers are requested together with a precomputed input matrix. Non-regression test for: https://github.com/scikit-learn/scikit-learn/issues/27893 rdrtz>Cannot store centers when using a precomputed distance matrix.rWrTF)rVrr?N) r9rrr rZr[r\rrD)rrrCX_disterr_msgs r*0test_hdbscan_error_precomputed_and_store_centersr+sk ))   "C 8A  #FNG z 1 '  #f+ 2 1 1s BB  valid_algorrcFtd|djty)zTest that HDBSCAN works with the "cosine" metric when the algorithm is set to "brute" or "auto". Non-regression test for issue #28631 cosineFrN)rrYrC)rs r**test_hdbscan_cosine_metric_valid_algorithmr?s 8z>JJ1Mr, invalid_algoctd|d}tjtd5|j t dddy#1swYyxYw)zTest that HDBSCAN raises an informative error is raised when an unsupported algorithm is used with the "cosine" metric. rFrzcosine is not a valid metricrWN)rrZr[r\rYrC)rhdbscans r*,test_hdbscan_cosine_metric_invalid_algorithmrIs< XEJG z)G HA I H Hs A  Actjjdjd}d}tjt |5t d}|j|dddy#1swYyxYw)z\ Test that HDBSCAN raises a FutureWarning when the `copy` parameter is not set. rrzCThe default value of `copy` will change from False to True in 1.10.rWrrN)r9rrrZrrrrD)rCr_rs r*!test_hdbscan_default_copy_warningrTsW a ''1A PC m3 /r*   0 / /s A33A<)r)K__doc__numpyr9rZscipyr scipy.spatialrsklearn.clusterrsklearn.cluster._hdbscan._treerrr sklearn.cluster._hdbscan.hdbscanr sklearn.datasetsr sklearn.metricsr sklearn.metrics.pairwiser r sklearn.neighborsrrsklearn.preprocessingr sklearn.utilsrsklearn.utils._testingrrsklearn.utils.fixesrrrCr& fit_transform ALGORITHMSitemsr%r+mark parametrizerRr`rkrnrrrrrrrrrrrrr rArrrrrrrr rrrr)routs00r*r3s  "# ?'1H.0!F>Cb11q!!$1""1% d1H1B1H1H1JK1Jvq#c'l1JKK 8 ):;J<J>A0-/Q/Q./QR S "  ,>2%3-%P 078884 - H )[!9:;.9U:UDj1&2&0 ,F,( M "HBHHq"&&kBFFA;-G$HI M "aVaV$45 q!fq!f 88.9 9: 9.9 9: 9 V0 /$?QH-!/.@!/H&/R:x*@AB&'89N:N)[)ABC ]Ls0M!