SjE(dZddlmZddlZddlZddlZddlmZddlZejdddl m Z ddl Zeej!j"dZedz Zedz Zdd Zdd Zdd Zdd Zdd Zedk(reyy)zDCompare saved CAD-mapped weak-risk fits across AC28 and ME1a models.) annotationsN)PathAggz(results/ME1a_DT_comparison/model_rankingzresults/ME1a_DT_comparisoncLtj|jdS)Nutf-8encoding)jsonloads read_text)paths $전처리/compare_all_models_risk.pyloadrs ::dnngn6 77c0tfd|DS)Nc3fK|](}|ddk(s |ddk(s|dk(r |dk(r|*yw)budget.kind residual_geoeval_runsetupN).0rowrunrs r zresidual_row..sFFts8}'=#f+Q_B_ Os*s7|u/Dts 111)next)rowsrrs ``r residual_rowr!s FtF FFrcttdz }ttdz }ttdz }|ddk7s|ddk7r tdg}|dj D]\}}|d|}|j id |d d d d d|dd|dd|dd|dd|dd|dk(rdndd|dd|dd|dd|dd |dd!d"d#|d$d%d&|d$zd'D]\}}t |d(d)|d*k(rd+nd,}t |d(d-|d*k(rd.nd,}|j id |d d/d d0d|d1d|d2d|d3dd4d|d5d|d6d|d1d|d2d|d3dd4d |d5d!|d6d#|d$d%|d$|S)7Nz*results/models/cad_predictions/report.jsonz>experiments/ME1a_DT/results/models/cad_predictions/report.jsonzgnn_joint_hard/report.jsonr18h0114h06zUnexpected comparison runmodelsmodelfamilyzprior baselinegraph_nodes_requestedac28_apaverage_precisionac28_iou threshold_iou ac28_dicethreshold_dice ac28_coveragedisplay_coverage ac28_epochsepochsac28_training_runsRisk GNNzAC28 16h + 18h01 AC28 18h01me1a_apme1a_iou me1a_dice me1a_coverage me1a_epochsme1a_training_runs ME1a 14h06ac28_checkpoint checkpointme1a_checkpointzexperiments/ME1a_DT/))zResidual Geo GNN (individual)single)zResidual Geo GNN (joint)jointr AC28_18h01rAsingle_AC28_18h01rB ME1a_14h06single_ME1a_14h06zoptimized risk-only GNNr mapped_ap mapped_iou mapped_diceg?epochs_per_fit train_runs)rPROJECTBASE ValueErroritemsappendr!) acmehardr nameamtitlers rrecordsrXs. gDD EB gXX YB 33 4D %yGr%yG3455 Dh<%%'a xL  RWd RH.> R@WY[ R"5 6 R8BAoDV R !$4"5 R7FJ\H] R#AhK R*AS+=Ye R "5 6 R 9CAoDV R !!$4"5 R 8GJ\H] R#AhK R2F| R', R'(><(P R S(@ u f|EU]L]5Hcj k f|EU]L]5Hcj k ]We]X/H]Jach]+]0:AlO] !M"2]4CS]#D)9$:]=QRST`Ra]+ ]1;AlO ] !!M"2 ] 5DS ] #D)9$: ] =QRST`Ra ]',]:KAlO] ^ @ Krc :tjdddd\}}tdD]\}\}}tdD]\}\}}|d|t|fd } | D cgc]$} | d d k(rd n| d j drdnd&} } |||f} t j t| } | j| | D cgc]} |  c} | | j| | D cgc]} | d  c} d| jdtdtfd| Ddz| j|dk(r|jnd| j|d|| jddt| D]:\}} | j!| | j#ddzz|| d d!d"#<|j%d$d%|j't(d&z d'(tj*|ycc} wcc} wcc} w))N) T)figsizeconstrained_layout))ac28r6)me1ar=))apzAverage precision)iouz IoU at 0.5_c|SNr)rkeys rzfigure..Ds 3s8rrfr&r5z#cf6d28r' optimizedz#126c9az#8998aa)color)fontsizerg?c3(K|] }| ywrer)rrrfs rrzfigure..Ks'DGSCGsg= ףp=?raIoU: xg?)axisalphargMb?.3fcenter)varlzIWeak-label same-run fit; training setups and native output domains differr[zmodel_comparison.png)dpi)pltsubplots enumeratesorted startswithnparangelenbarh set_yticksset_xlimmax set_xlabelupper set_titlegridtextget_xlimsuptitlesavefigOUTclose)r figaxesipartlabeljmetricrWorderedrcolorsaxykrfs @rfigurer?s Q8MIC%&VW=D%"+,`"a AF!F8$CT';y$'M$<$<[$IyyYel nadB #g,'A GGAG4GSCG4FG C MM!g>gsc'lg>M K KK3sC'DG'D$Dt$KL M MMFdN&,,. F LLE7"UG, - GGBG '#G,3C2;;=#3d#::A#c(3U]hij-#bX LL\giLjKK,,#K6IIcNn5>s#)H H( Hc tjddt}tdz jddd5}t j |t |d }|j|j|dddd d d d ddd gd|d}tdz jtj|d dt|td|D}td|D}t|d}t|d}t|d}t|d} djdt!|ddD} djd|D} djgd t#j$|d!d"|d#d$d%t#j$|d!d"|d&d$d't#j$|d!d"|d(d$d)t#j$| d!d"| d*d$d+|d#d$d,|d&d$d-|d(d$d,|d*d$d.|d#|d#z d$d/| d0| d1} td2z j| dt'td2z y#1swYxYw)3NT)parentsexist_okz report.csvwr)r)newliner r) fieldnames)saved_same_run_weak_label_risk_comparisonFrZr)AC28ME1a_DTzUCAD-mapped 10 mm CSV-seeded weak risk mask on AC28 18h01 and ME1a 14h06 training runs)z7Unvalidated CAD-to-scan registration and defect extentszADifferent training epochs, source domains and CAD mapping supportzIRisk GNN was trained on two AC28 runs; other prior AC28 models used 18h01zMOptimized residual GNN predicts risk only; Risk GNN also predicts deformation)statusproduction_validpart_run_count evaluationheld_out_baselines_available limitationsr z report.json)indentr c32K|]}|ddk(s |yw)r&r5Nrrrs rrzwrite..g<4a1W:#;4 c32K|]}|ddk(s |yw)r&zMesh GNNNrrs rrzwrite..hrrc |dS)Nr*rrs rrgzwrite..iAiLrrhc |dS)Nr,rrs rrgzwrite..jQz]rc |dSNr7rrs rrgzwrite..krrc |dS)Nr8rrs rrgzwrite..lrrc3K|]V}dtj|dd|ddd|ddd|ddd|ddd|d dd|d dd|d d|d d Xyw)r& r*rs r,r7r8r0r:r2r; Nhtmlescapers rrzwrite..msIHA 4;;qz*+9Qy\#4FiPQR\P]^aObc|C !J-).qsAiLr)rfreversec 3K|]R}dtj|ddtj|ddtj|ddTyw)rr&rr4rr<rNrrs rrzwrite..rse\VZQR  AgJ 78 $++aPdNeBfAgh KK*>(?@AMVZsAAaAC28 and ME1a: all risk models compared

Is Risk GNN the best risk model?

These are fits to provisional masks on the training runs, with unvalidated scan registration. Output resolutions, CAD mapping coverage, loss functions and epoch counts differ. The table can compare saved diagnostics; it cannot identify the best production predictor. Other model families have no held-out run scores.

AC28 highest AP
r&ror*rsz&
AC28 highest IoU
r,z%
ME1a highest AP
r7z&
ME1a highest IoU
r8z2

Risk GNN: AC28 AP z, IoU z ; ME1a AP z$. Its AC28 AP lead over Mesh GNN is a. It does not lead AC28 IoU or either ME1a metric. The optimized residual GNN predicts risk only; Risk GNN also emits a deformation field.

All saved model results on common CAD nodes

a3
ModelAC28 APAC28 IoUME1a APME1a IoUAC28 coverageME1a coverageAC28 epochsME1a epochs
AP and IoU of all models on AC28 and ME1a

Training scope

The Risk GNN used both AC28 runs for its AC28 entry. Other original models used AC28 18h01. The residual GNN individual entry uses one run of each part; its joint entry uses both AC28 runs and ME1a. Every displayed score evaluates a run included in that model's training. Only the residual GNN has a separate leave-one-run-out experiment, and its held-out IoU was zero for all three runs. The five conditional diffusion variants generate outputs on different native domains; their diagnostic risk scores are in the main report and are not ranked here.

u
ModelAC28 trainingME1a training

Residual GNN held-out experiment · AC28 model gallery · ME1a model gallery

All metrics CSV · Machine-readable results · Main comparison report

z index.html)rmkdirrXopencsv DictWriterlist writeheader writerows write_textr dumpsrrrjoinr|rrprint) r fhwriterresultriskmeshac_apac_ioume_apme_iou table_rowsscopespages rwriterVsIIdTI* 9D  " "3W " EtDG}=f..t4 FDY^)*q 9s.3oF =$$TZZq%AG$T 4L <4< rsJ"  u x. " * *1 - ::--8F D.&R z57r