IOP Conference Series: Materials Science and Engineering PAPER • OPEN ACCESS Data-driven benchmarking methodology for evaluating PBF-LB/M machines with RMSD Analysis To cite this article: Usama Nadeem et al 2025 IOP Conf. Ser.: Mater. Sci. Eng. 1332 012040 View the article online for updates and enhancements. You may also like Improvement in the PBF-LB/M processing of the Al-Si-Cu-Mg composition through the use of pre-alloyed powder A Martucci, F Gobber, A Aversa et al. - Additive manufacturing of heat-resistant aluminum alloys: a review Chaoqun Wu, Jianyu Wen, Jinliang Zhang et al. - Surface analysis in additive manufacturing: a systematic literature review regarding powder bed fusion processes Tobias Grimm, Nick Hantke, Alsu Iusupova et al. - This content was downloaded from IP address 130.232.200.91 on 20/11/2025 at 08:04 Content from this work may be used under the terms of the Creative Commons Attribution 4.0 licence. Any further distribution of this work must maintain attribution to the author(s) and the title of the work, journal citation and DOI. Published under licence by IOP Publishing Ltd 20th Nordic Laser Materials Processing Conference IOP Conf. Series: Materials Science and Engineering 1332 (2025) 012040 IOP Publishing doi:10.1088/1757-899X/1332/1/012040 1 Data-driven benchmarking methodology for evaluating PBF-LB/M machines with RMSD Analysis Usama Nadeem1*, Nikhil Kamboj1, Chinmayee Nayak1, Heidi Piili1 1 Department of Mechanical Engineering, Faculty of Technology, University of Turku, Turku, 20520, Finland *E-mail: usnade@utu.�i Abstract. Laser based powder bed fusion for metals (PBF-LB/M) is an industrial additive manufacturing (AM) method offering high-precision manufacturing for complex geometries. However, comparing the performance of different PBF-LB/M machines remains dif�icult, especially when machines are from different manufacturers. This study introduces a new benchmark artifact with standard features for facilitating the evaluation and comparison of machine performance. Two industrial PBF-LB/M machines, EOS M290 and Aconity3D MIDI+, were used to fabricate the part under similar conditions. The additively manufactured (AMed) samples were then inspected using 3D scanning metrology tools, and the results were analyzed using a method called root mean square deviation (RMSD) to measure how far each feature deviates from the original design. The results showed apparent differences in how each machine handled certain features and provide useful information for choosing the right machine based on part geometry. 1. Introduction Many industries are adapting additive manufacturing (AM) technologies for their ability to produce complex, high-precision geometries without tooling restrictions [1]. Among them, laser based powder bed fusion for metals (PBF-LB/M) is a widely established process [2]. As more organizations adopt PBF-LB/M technology, selecting the right machine for a speci�ic application becomes a growing challenge. This is especially problematic in multi-vendor workshop settings where machine architecture, scanning strategy, system, recoater, and process parameters interplay to in�luence part quality and consistency [3]. While machine suppliers provide technical datasheets and optimized parameters, this often lacks meaningful information about actual geometric performance [4]. Benchmark artifacts have been used extensively in literature to assess dimensional accuracy, feature resolution, and other metrics of performance of AM systems [5], [6], [7], [8], [9]. However, existing designs focus on process optimization, material properties validation, or academic evaluation of printability limits. These artifacts are too simple for any meaningful assessment between machines or too complex to be measured ef�iciently with commonly available tools [10]. Furthermore, many lack a clear evaluation framework to support decision-making [11]. This study introduces a compact benchmark artifact for cross-machine evaluation, fabricated on two industrial PBF-LB/M systems. Deviations were measured via high-resolution optical and laser scanning, and root mean square deviation (RMSD) was used as a consistent accuracy metric. The method provides industry with an accessible tool for machine assessment and offers academia a validated artifact and data-driven benchmarking framework. 20th Nordic Laser Materials Processing Conference IOP Conf. Series: Materials Science and Engineering 1332 (2025) 012040 IOP Publishing doi:10.1088/1757-899X/1332/1/012040 2 2. Literature review In the early 1990s, Kruth [7] introduced the �irst benchmark artifact for the geometric evaluation of additive manufacturing (AM) focusing on test parts for dimensional accuracy. Castillo [12] expanded by measuring wall thickness variation linked to impact of scan strategy and thermal distortion. These earliest studies dealt predominantly with single machine process validation. Moylan NIST [4] benchmark was designed to help compare different machines by checking accuracy and repeatability using a standard test part. However, its widespread adoption is limited due to its complexity, large size, and high measurement effort. NASA and academic partners [13] developed benchmarking artifacts with complex geometries for multi-machine comparison. Using a standard test part, Moshiri [5] compared �ive PBF-LB/M systems and noted considerable differences in �ine hole and unsupported wall features. All benchmark artifacts in AM typically fall into three categories: geometrical, mechanical, or process-speci�ic [14]. While useful for process or material validation, many designs combine features, making them time-consuming to measure, particularly in small labs or workshops. Most artifacts were created to tune system processes rather for machine comparison and lack consistent evaluation metrics or procedures to quantify geometric deviation. Rebaioli and Fassi [6] highlight the need for benchmark designs that balance measurement complexity with feasibility, enabling meaningful, comparable data. Prior studies rely on metrics like maximum deviation, mean absolute error (MAE), or pointwise tolerance for evaluating the benchmark artifact, which often fail to capture cumulative geometric error. This study uses RMSD for geometry-speci�ic comparisons because of its statistical precision and greater sensitivity to larger deviations and captures local and global form errors simultaneously and returns scalar value suitable [15] for standard measure for benchmarking studies. 3. Material and method This study expands on the prior benchmark designs from the literature by adding geometry types used for dimensional measurements evaluation, like holes, cylinders, and inclined surfaces. The artifact is compact to reduce build time and material usage [4], includes features measurable by surface-based optical scanning [16], and repeats features in multiple zones to assess spatial consistency [4]. The design excludes internal channels or computed tomography (CT) dependent geometries, ensuring compatibility with accessible laser and optical scanners. Each feature was dimensioned to match standard geometric dimensioning and tolerancing (GD&T) tolerances. These features were grouped and labeled to enable localized measurement, as shown in Figure 1. While Table 1 summarizes the features in benchmark artifact in groups with number of instances, nominal dimensions, and tolerance focus groups for deviation and RMSD analysis. Figure 1. (a) CAD model of the benchmark artifact. (b,c) Color-coded feature groups are labeled. 20th Nordic Laser Materials Processing Conference IOP Conf. Series: Materials Science and Engineering 1332 (2025) 012040 IOP Publishing doi:10.1088/1757-899X/1332/1/012040 3 The artifact measured 100 × 100 × 15 mm, sized to minimize material use while �itting standard PBF-LB/M volumes. Each feature was dimensioned according to standard GD&T tolerances and sized for compatibility with surface-based optical scanning tools. Complex internal geometries and support-dependent structures were intentionally excluded to ensure easy repeatability across different machines. The benchmark artefact was produced on two commercial PBF-LB/M machines—the EOS M 290 (EOS GmbH, Germany) and the Aconity3D MIDI+ (Aconity3D GmbH, Germany), see Figure 2. Two identical artifacts were fabricated on each platform from 316L stainless-steel powder (15–45 µm) using the same STL �ile and the machines’ native slicers (EOSPRINT, Aconity Studio). Both builds used 40 µm layers, 0.11 mm hatch spacing, and ~1100 mm/s scan speed. Parts were oriented �lat and required no post-processing. Figure 3 illustrates the alignment process used in this study, where high-density 3D point clouds captured using a ZEISS T-Scan 20 laser triangulation scanner were aligned to the original CAD model. Figure 2. Benchmark artifact fabricated using (a) EOS M290 and (b) Aconity3D MIDI+ machines. Table 1. Summary of all the benchmark artifact features, grouped by category. ID Feature Type Features Dimensions / Angles Target Tolerance A Long thin wall A1 0.4 mm × 46 mm × 3 mm Residual stress B Circular holes B1–B9 Ø0.4 mm to Ø6 mm Circularity / Diameters C Notches C1–C6 0.5 mm to 4 mm widths Fine feature accuracy D Cylindrical boss D1–D2 Ø8 mm, Ø10 mm Diameters E Square bosses E1–E2 4×4×2 mm, 8×8×2 mm Dimensional accuracy XY F–G Staircases (±) F1–F5, G1–G5 1–5 mm step heights Z-direction resolution H Hemispheres H1 Ø7 mm Blend accuracy I Lateral fine features I1–I9 Ø0.4–2 mm, 1×1 mm squares Fine feature printability J Slots J1–J5 0.4 mm wide Thin wall test K Thin walls K1–K5 0.4 mm wide Wall thickness L Crosses (out/in) L1–L4 0.4 mm / 0.5 mm widths Sharp edge accuracy M Inclinations M1–M5 25° to 65° angles Overhang & distortion N Pins & Circular bosses N1–N6 Ø0.4 mm to Ø3 mm Diameters O Outer base dimensions O 50 × 50 × 10 mm Global accuracy Features were designed for GD&T-based evaluation and sized for compatibility with optical surface scanning. Figure 3. Work�low for deviation analysis using CAD and 3D scan models — (a) STL from CAD, (b) mesh from 3D scan, (c) superimposed models, (d) deviation map with measured distances. 20th Nordic Laser Materials Processing Conference IOP Conf. Series: Materials Science and Engineering 1332 (2025) 012040 IOP Publishing doi:10.1088/1757-899X/1332/1/012040 4 Figure 4. Surface roughness via Alicona G6: (a) roughness location on artifact; (b) scanned surface area; (c) 3D topographic pro�ile. A two-level metrology strategy was followed: one assessing dimensional accuracy and the other surface quality. The alignment was performed using initial registration and best-�it matching in Geomagic Control X, allowing precise measurement of how accurately each feature was built compared to the intended design. Surface roughness was measured using the Alicona In�initeFocus G6 system. The Alicona software reconstructed 3D topographies by identifying sharp focus points across multiple vertical scans, as illustrated in Figure 4. A �lat planar and overhanging surface were selected as optically accessible, support-free areas representative of AM quality and thermal behavior across both machines. Scanning was conducted at 20× magni�ication. The measurement process followed ISO 16610-71:2014, which applies a robust Gaussian regression �ilter to accurately capture and separate surface features of different scales. From these, areal roughness parameters such as Sa (arithmetic mean height) were calculated and used as the primary metric for comparing surface texture between machines. After measuring deviations for each feature group along with surface roughness, and build time, RMSD values were measured separately and normalized to assess the dimensional accuracy between EOS M290 and Aconity3D MIDI+ (e.g., holes, walls, overhangs) using equation (1). Where xᵢ is the measured point, xᵢʳᵉᶠ is the reference point on the CAD model, over 𝑛𝑛 data points. RMSD aggregates both local deviations and global form error into a single scalar value, making it effective for comparing part accuracy across different machines and feature types. Surface-roughness RMSD was obtained by taking the root-mean-square of the differences between Sa readings gathered at identical surface locations on the EOS and MIDI+ artefacts. Build-time RMSD was calculated analogously, using the differences between slicer-predicted and actual job durations. In both cases, a smaller RMSD signals closer agreement and thus higher accuracy. RMSD = �1 𝑛𝑛 ��𝑥𝑥𝑖𝑖 − 𝑥𝑥𝑖𝑖 ref�2𝑛𝑛 𝑖𝑖=1 (1) 4. Results and discussion To show how the measurement strategy works, Figure 5 displays the deviations of the circular holes (B1–B9) listed earlier in Table 1. 20th Nordic Laser Materials Processing Conference IOP Conf. Series: Materials Science and Engineering 1332 (2025) 012040 IOP Publishing doi:10.1088/1757-899X/1332/1/012040 5 These circular hole features are typically sensitive to process parameters such as laser spot size, scan path accuracy, and contour de�inition. As shown in Figure 5b, nominal CAD radii (red crosses) are compared with measured values (blue bars), and absolute deviation is indicated in orange. A ±0.05 mm tolerance band is plotted as a grey zone around the nominal radius. While many features remain within tolerance, several display substantial deviation. Speci�ically, feature #10 (nominal = 1.25 mm, measured = 1.22 mm) shows a minimal deviation of 0.025 mm, indicating high geometric accuracy. In contrast, feature #2 (nominal = 0.50 mm, measured = 0.68 mm) deviates by 0.18 mm, which exceeds the tolerance by a wide margin. This highlights reduced precision when reproducing �ine features, likely due to resolution limits or laser overexposure. These observations underline the value of localized feature analysis in benchmarking AM performance. RMSD values for each feature group were then calculated and are visualized in Figure 6 as a heatmap, where values closer to 0.00 indicate higher dimensional accuracy. EOS M290 yielded the lowest RMSD values for features most sensitive to thermal stability and scan-path transition: inclined surfaces (0.23 vs 1.08), residual-stress signs (0.94 vs 1.25), crosses (0.07 vs 0.08), Z-axis deviation (0.05 vs 0.06), and surface roughness (0.05 vs 2.00). These results re�lect its production-grade sealed chamber, closed-loop thermal management, and optimized scan strategy. Aconity3D MIDI+ excelled where �ine beam positioning was essential. It showed smaller RMSD values for hemispheres (0.17 vs 0.62), sub-0.5 mm cylinders (0.05 vs 0.26), thin walls/slots (0.07 vs 0.71), and features like notches (0.09 vs 0.19), nominal cylinders (0.08 vs 0.22), and squares (0.18 vs 0.40). This highlights its �iner laser spot and �lexible beam control, enabling precise energy delivery in narrow or curved areas. EOS, therefore, remains the safer Figure 6. RMSD heatmap comparing EOS M290 and Aconity3D MIDI+. Figure 5. Circular feature deviation: (a) Circular feature labels; (b) Nominal vs. measured radii with error bars and ±0.05 mm threshold plotted around each nominal radius. 20th Nordic Laser Materials Processing Conference IOP Conf. Series: Materials Science and Engineering 1332 (2025) 012040 IOP Publishing doi:10.1088/1757-899X/1332/1/012040 6 choice for components needing minimal distortion and premium surface �inish, while Aconity3D MIDI+ suits �ine-featured parts like pins, thin walls, and internal channels. Aconity's open-frame and modular recoater likely caused local powder spreading and thermal inconsistencies, contributing to its scatter in surface roughness. Further testing with modi�ied parameters could clarify these effects. 5. Conclusion This research proposed a compact benchmark artifact and an RMSD-based assessment method for enabling geometric accuracy evaluation of PBF-LB/M machines. The EOS M290 performed outstandingly on distortion-sensitive features. Aconity3D MIDI+ demonstrated superior geometric accuracy on highly complex �ine structural components. Each machine was evaluated and two artifacts from each machine yielded consistent replicate results, supporting comparison. The RMSD-based assessment method is practical and realistic, relying solely on accessible commercial scanners and software, making it especially useful for SMEs and multi-machine research. Notably, the entire benchmarking process from fabrication to testing occupies only one working day, favorable for useful industrial application. It enables feature level machine comparison based on actual print performance instead of just manufacturer-driven speci�ications and can be further enhanced by integrating multi criteria decision making (MCDA) to rank machines according to RMSD for each feature, tailoring system selection for production need. Acknowledgement This work was conducted under the “Kestävästi lisäävä!” project (Turku Innovation Centre of Additive Manufacturing, TICAM), funded by the European Regional Development Fund via the Helsinki-Uusimaa Regional Council (decision A80276). The project runs from May 1, 2023, to June 30, 2026, in collaboration with University of Turku, AÅ bo Akademi University, Turku University of Applied Sciences, and industry partners. The authors gratefully acknowledge the project and its partners for their support in advancing SME additive manufacturing capabilities in Turku region. References [1] Herzog D, Seyda V, Wycisk E and Emmelmann C 2016 Acta Mater 117 371–392. [2] Goetz D, Panzer H, Wolf D, Bayerlein F, Spachtholz J and Zaeh M F 2024 Modelling 5 841–860. [3] Tofail S A M, Koumoulos E P, Bandyopadhyay A, Bose S, O’Donoghue L and Charitidis C 2018 Mater Today 21 22–37. [4] Moylan S, Slotwinski J, Cooke A, Jurrens K and Donmez M A 2012 NIST Interagency/Internal Rep 7925. [5] Moshiri M, Tosello G and Mohanty S 2018 Proc 18th Int Conf Euspen 261–262. [6] Rebaioli L and Fassi I 2017 Int J Adv Manuf Technol 93 2571–2598. [7] Kruth J, Vandenbroucke B, Vaerenbergh J and Mercelis P 2005 Rapid Prototyp J 11 26–36. [8] Campanelli S L, Contuzzi N, Angelastro A and Ludovico A D 2010 New Trends Technol: Devices Comput Commun Ind Syst (ed Joo M) 233–252. [9] Kniepkamp M, Fischer J and Abele E 2016 Proc 27th Annu Int Solid Freeform Fabr Symp 1530–1537. [10] Rivas Santos V M, Thompson A, Sims-Waterhouse D, Maskery I, Woolliams P and Leach R 2020 Addit Manuf 32 100964. [11] Montero J, Weber S, Petroll C, Brenner S, Bleckmann M, Paetzold K and Nedeljkovic-Groha V 2021 Proc Des Soc 1 1657–1666. [12] Castillo L 2005 TNO Rep in collab with AIMME [Online]. [13] Gradl P R, Tinker D C, Ivester J, Skinner S W, Teasley T and Bili J L 2021 Addit Manuf 47 102305. [14] Mahesh M, Fuh J Y H, Wong Y S and Loh H T 2005 IEEE Int Conf Autom Sci Eng 19–24. [15] Cruz Sanchez F A, Boudaoud H, Muller L and Camargo M 2014 Virtual Phys Prototyp 9 151–167. [16] Chua C K, Wong C H and Yeong W Y 2017 3D Print Addit Manuf 1 181–212.