ARTIFICIAL INTELLIGENCE - FINITE ELEMENT METHOD - HYBRIDS FOR EFFICIENT

. Realistic structural analyses and optimisations using the non-linear finite element method are possible today yet suffer from being very time-consuming, particularly in case of reinforced concrete plates and shells. Hence such investigations are currently dismissed in the vast majority of cases in practice. The "Artificial Intelligence - Finite Element - Hybrids" project addresses the current unsatisfactory situation with an approach that combines non-linear finite element models for reinforced concrete shells with scientific machine learning algorithms to create hybrid AI-FEM models. The AI-based surrogate material model provides the material stiffness as well as the stress tensor for given concrete design parameters and the strain tensor. This paper reports on the current status of the project and findings of the calibration of the AI-based reinforced concrete material model. We successfully calibrated and evaluated k-nearest-neighbour, LGBM and ResNet algorithms and report their predictive capabilities. Finally, some light is shed on the future work of integrating the AI surrogate material models back into the finite element method in the course of the numerical analysis of reinforced concrete structures.


Introduction
Digital design and manufacturing methods, such as those to be developed at the new Immersive Design Lab (IDL) at ETH Zurich, offer great potential for significantly more efficient and sustainable construction. In order to ensure the structural safety, economic efficiency and sustainability of complex structures, reliable and powerful models for automatic analysis, optimisation and design are essential. However, such models are largely lacking to date. This is especially true for concrete structures, as their behaviour is highly non-linear. Realistic structural analyses and optimisations using the non-linear finite element method (FEM) are possible today, yet being very time-consuming even for the case of extraordinary computational capacities. For reasons of temporal and monetary efficiency, established traditional and in many cases excessively conservative design methods without structural optimisation are still used in the vast majority of projects in practice today. The increased public awareness of and demand for a sustainable built environment however urges civil engineers to make use of structural efficiency to the maximum level possible.
To that end, a two-phased research program [1,2] is proposed to address this unsatisfactory situation.
In the first phase, non-linear FEM for reinforced concrete plates and slabs developed at ETH Zurich [3][4][5][6][7][8] are combined with scientific machine learning (SciML) algorithms to create hybrid AI-FEM models, which are expected to be much more efficient compared to established analysis methods both in terms of the computing power required and the reliability of predicting the load-bearing behaviour. In the second phase of this project, the AI-FEM-Hybrids will be used within a novel Generative Design process for accelerated yet realistic conceptual design of bridges [1,2].
Within a FE analysis, a material model has to provide on the one hand the material stiffness matrix and on the other hand the current stress state. In the novel hybrid artificial intelligence finite element (AI-FEM) model implementation for material-nonlinear reinforced concrete slabs and plates elements, the mathematical description of the material model is performed using scientific Machine and Deep Learning algorithms, which act as functional approximators of the stress-strain relationship and the stiffness-strain relationship based on numerical simulations. The basic data set for training, validation and testing of the AI algorithms is generated by extensive simulations of strain states for different reinforced concrete configurations with the material model for reinforced concrete as implemented in the mentioned USERMAT in ANSYS MECHANICAL APDL on the basis of the cracked membrane model (CMM) in combination with a Reissner-Mindlin layer model. This paper describes the process of data generation and calibration of the AI-based reinforced concrete material model for different Machine and Deep Learning algorithms for stress and stiffness tensor prediction. This paper is organised as follows: we first provide background on the materials and methods from computational mechanics and scientific machine learning used in this paper in sec. 2. Sec. 3 reports and presents selected numerical results of the SciML-based substitute concrete material models w.r.t. their approximation accuracy and statistical qualities. Sec. 4 presents a discussion of current findings and sec. 5 sheds light on the future steps of incorporating the SciML-based substitute concrete material models into the FEM software ANSYS.

Materials and Methods
This section reports on materials and methods for database generation, development of machine and deep learning (ML/DL) models, and the evaluation criteria for the performance of the developed models as a surrogate for a FEM.

Concept
This research project explores the potential of using ML/DL surrogate models of mechanically consistent nonlinear finite element material models for reinforced concrete plates and shells, developed at ETH Zurich, as a data-driven yet physics-informed AI method within a FEM workflow. The project is divided into two parts.
The AI-FEM-Hybrids of phase one provide predictions of the material stiffness tensor as well as the stress tensor for a current strain state, where the SciML algorithms act as functional approximators of the stress-strain relationship and the stiffness-strain relationship, which was found to be applicable to a number of materials [9]. The basic data set for training, validation and testing of the AI algorithms is generated by an extensive simulation of strain-stressstiffness states for different reinforced concrete configurations with the material model for reinforced concrete as implemented in the mentioned USERMAT in ANSYS MECHANICAL APDL on the basis of the cracked membrane model (CMM) in combination with a layer model based on Reissner-Mindlin plate kinematics. The training, validation and testing of the SciML algorithms for stress and stiffness tensor prediction are presented and discussed in the following, where special consideration is given to model uncertainty quantification to allow for a future Eurocodecompliant use within the semi-probabilistic design philosophy. With the completion of the first phase, a validated software demonstrator of AI-FEM-Hybrids for the efficient non-linear analysis and design of reinforced concrete structures is available. In the second project phase, the AI-FEM-Hybrids approach will be extended to structural optimisation of concrete bridges. The pilot project in this phase focuses on concrete bridges with a geometry defined by a few parameters. While the case study is specific, the developed optimisation methodology is kept as general as possible to allow its application beyond parametric concrete bridge structures. At the end of the research project, a validated software demonstrator of the AI-FE-Hybrids will be available for the efficient non-linear analysis, design and optimisation of reinforced concrete structures. In the future, this will then allow for (i) realistic structural analyses taking into account non-linearities and (ii) structural optimisation to be carried out much more efficiently. The outcomes of this research are potentially of high interest to industry and engineering practice, as structural concrete is the most widely used construction material worldwide and incorporation of these ideas allows for more economic, yet sustainable and reliable design.
To achieve the objectives for phase one of the research project, the following sequence of steps was conducted:

FEM Material Model Data Generation
The dataset generation within the AI-FEM-Hybrid project was performed by nonlinear finite element analysis of a reinforced concrete structure utilising the CMM-USERMAT [3], a mechanically consistent material model for reinforced concrete implemented in ANSYS Mechanical APDL, cf. Figs. 1 and 2. Combined with a layer element (Shell181), the non-linear FEM analysis of reinforced concrete structures as girders or shells are possible. As presented in [3][4][5], an excellent agreement between experimental results and FEM analysis utilising the CMM-USERMAT is reported.
In order to provide the necessary dataset for the training of the machine learning algorithms, a dummy girder bridge (cf. Figs. 1 and 2) was arbitrary loaded incrementally until failure, which enables the export of  Figure 1. Dummy girder bridge used within ANSYS to generate the database. all necessary features (input parameters X: strain tensor, reinforcement properties, material constants, etc. acc. to Tab. 1) and results (stress and stiffness tensors acc. to Tab. 1) in every integration point (Gausspoints) for all converged load steps. This approach enables the collection of a considerable number of data points within one non-linear FEM computations, as in almost every integration point an individual data set exists. For simplicity, a dummy girder bridge was chosen to minimise the number of parameters, e.g. the diameter of the rebars or the material constants. As soon as the general framework of the AI-FEM-Hybrids is established (proof of concept), the results of any dummy experiment and the results of any FEM analysis using the CMM-USERMAT can be used to train the machine learning algorithms.

Data Pre-Processing
The data generated in the previous section is used to develop the ML resp. DL models. The dataset consists of over 11 million data points with 14 feature and 12 target variables, cf. Tab of the features as well as the targets are extremely different, hence we employ normalization of the data. For each numerical variable v i we deduce its mean m and divide it by its standard deviation s (yielding its z-score) using: For sake of brevity of this paper, we omit reporting further statistical properties or histograms of the dataset, which would usually be delivered in the exploratory data analysis step.

Machine Learning Model Development
In this study, three ML resp. DL algorithms for regression are investigated for predictive capabilities for stress and stiffness target tensors given the strain tensor together with concrete and reinforcement features: • k-nearest-neighbours (kNN) • LightGBM • Artificial Neural Networks: Residual Neural Networks (ResNet) While kNN is a strictly data-driven approach and hence needs to store some of the simulation data instances, LightGBM as well as ResNet are surrogate functions without the need to store the training data. Due to reasons of brevity of this paper, only selected results and implementation details can be reported. The ML models are developed using sklearn, Keras, and XGBoost (extreme gradient boosting) python libraries and run on the ETH Euler cluster 1 .

kNN
The k-nearest-neighbours algorithm is a decision method originally developed for classification tasks [10]. At inference time, it compares the previously unseen samples to all instances in a database and assigns classes by means of majority voting of the k nearest neighbours, where k and the distance metric are hyperparameters. The algorithm was later extended to support regression tasks [11] by interpolating between the values of the k nearest neighbours (e.g. through inverse distance weighted average) to obtain values for new, unseen samples.

LightGBM
LightGBM is a gradient boosting decision tree framework [12] that has found application in many different data mining, classification and regression tasks. The LightGBM algorithm variant used in this paper integrates a number of regression trees to approximate the dataset. It contains many of the advantages of other common gradient boosting decision tree algorithms, such as sparse optimization, parallel training, regularization, bagging, and early stopping. However, it grows trees leaf-wise by greedily choosing the leaf that will lead to the largest improvement. This tree-growth method, in combination with a histogram-based memory and computation optimization, make LightGBM considerably more computationally efficient than other frameworks.

Artificial Neural Networks
Fully-connected feed-forward neural networks (FFNN) consist of one or more layers, where each node is connected to every node in the following layer [13]. Despite being very straight forward architectures, FFNNs are universal function approximators [14] and have the advantage of being easy to implement and efficient to train. On the other hand, the lack of implicit bias limits their expressiveness and capability to generalise. Furthermore, they are prone to pathologies like vanishing gradients [15].
We therefore extend the FFNN by adding skip connections [15] as well as batch normalisation [16] and dropout [17] , cf. Fig. 3. These additions allow us to build deeper architectures, which were shown to generalise better in practice [18]. We will hereafter be referring to this model by ResNet.

Hyperparameter Tuning
We employ Bayesian Optimization in order to avoid running extensive grid-search over the entire hyperparameter space [19] [20]. At first, 25 random points vol. 36/2022   For the optimization procedure, we split the dataset into a training, validation and test set at ratios of (50; 25; 25)% of the whole dataset. The regressor is trained on the training set using the hyperparameters chosen by the Bayesian Optimization algorithm. Once the model loss converged, it is evaluated on the validation data set and the result serves as feedback for the Bayesian Optimization step to refine its posterior and select new hyperparameters. Finally, the test set is used to estimate the model's capability of generalizing.

Model Quantitative Performance
Metrics and AI Surrogate Model Selection Model selection refers to the process of seeking for a model in a set of candidate models, which delivers the best balance between model fit and complexity [22]. For regression modeling, the quantitative performance metrics used with this research are the mean absolute error (MAE), root mean squared error (RMSE), Rsquared (R2) value as well as the data variance V ar (which is the square of the standard deviation (SD)), cf. Tab. 5. The MAE shows the average difference between the actual values of the output variable in the original data vs. the predicted output values via the ML models. The lower the MAE, the more precise the performance of the model is in predicting future occurrences of the output. The RMSE is defined as the standard deviation of the response variable. Values of R2 range from 0 to 1, where 1 is a perfect fit, and 0 means there is no gain by using the model over using fixed background response rates. It estimates the proportion of the variation in the response around the mean that can be attributed to terms in the model rather than to random error. For comparing different models, we provide Taylor diagrams [23] and additionally report R2, RMSE and MAE values. Taylor diagrams are used to quantify the degree of correspondence between the modeled and observed data using the Pearson correlation coefficient, RMSE, and standard deviation. A model with the highest R2 and the lowest RMSE and MAE is preferred. The details of how MAE, RMSE, and R2 are calculated are shown in Tab. 5. where N is the sample size,ŷ i is the predicted and y i is the true target value, SSE is the sum of the square of error and SST is the total sum of squares.

Results
This section describes the results obtained with kNN, LightGBM, and ResNet and evaluates their performance as a surrogate for the FEM data generated and used within this study, cf. Sec. 2.3.
All three ML models were trained, validated and tested using the data from the total of 12 million data points. Comparing the R2 values, all models have high values of around 0.98. This indicates that the ML algorithms are able to describe over 98% of variations in the data, and are highly predictive of the output based on the feature variables used in the study. Regarding the MAE and RMSE, the kNN model performs one order of magnitude better than the other two ML models for the CF 0 configuration (without reinforcement) at a level of 0.3% resp. 0.05%, while for the CF 1 configuration (with reinforcement) the ResNets has the lowest RMSE value of 2.4% and the LGBM has the lowest MAE value of 1.8%.
The results suggest that all ML models developed for predicting the stress (all in unit: MPa) and stiffness (all in unit: MNm 2 ) tensor components are promising, with ResNet being the most predictive model amongst the three. The model performances, described by the mean absolute error MAE, root mean squared error RMSE and R-squared have converged to a stable minimum during training.

kNN Model Results
Fig. 4 provides selected results for the kNN algorithm in the two configurations "CF 0" (without reinforcement) and "CF 1" (with reinforcement).
It is important to note, that the size of the trained kNN model is of around 1.13 GB, as it needs to store all training samples in order to make new predictions.

ResNet Model Results
Fig. 6 provides selected results for the ResNet algorithm in the two configurations "CF 0" (without reinforcement) and "CF 1" (with reinforcement).

Overall Model Comparison Results
The performances of different AI models according to the mentioned criteria are reported in Tab. 6 for configuration "CF 0" (without reinforcement) and in Tab. 7 for configuration "CF 1" (with reinforcement). For model selection purposes, we provide selected results for the Taylor diagrams in the two configurations "CF 0" (without reinforcement) and "CF 1" (with reinforcement) in Fig. 7

Discussion
The choice for the three investigated AI algorithms was done under three aspects: modeling bias, computing efficiency and prediction precision. The kNN and LGBM belong to the family of non-parametric models, whereas ResNet is a parametric model. Nonparametric ML algorithms promise greater performance at the cost of higher data requirements and training times together with a risk of overfitting. Parametric ML/DL models on the other hand come with high modeling bias towards the functional relation, yet neural networks provide a great enough expressiveness for modeling the database. Concerning computational efficiency in the prediction stage (which is called a lot of times during FEM analysis in each iteration step) kNN requires to store the dataset and call it at prediction time, leading to impractical lengthy procedures.
LGBM and ResNet however are much more time efficient and hence to be preferred for an implementation into the FEM analysis.
The AI models perform better in CF 0 (without reinforcement) than in CF 1 accross the different quality measures. Looking at the R2 values, all models have high values of approx. 0.99 (CF 0) resp. 0.97 (CF 1). This means, that the ML algorithms are able to describe over 97% of variations in the data, and hence are highly predictive for the outputs. For CF 0 RMSE is around 0.35% and MAE is around 1.5%, while for CF 1 RMSE as well as MAE lie about 3% and are thus one magnitude bigger than for CF 0. It is noteworthy that on average, every target is predicted well, while the prediction's standard deviation is dependent on the respective target quantity. Given all model results, especially the stiffness tensor terms K 22 and K 23 show severe model predictive deviations.
For model selection, inspection of the Taylor diagrams in Fig. 7 suggests that all described AI models possess great approximation quality with slight differences, where kNN outperforms ResNet and LGBM (in the order of decreasing approximation quality). The results in summary suggest that the AI models developed for predicting the stress and stiffness responses of the CMM USER-MAT are promising and may be used within a FEM. Given the outlines in the beginning of this section towards computational efficiency, the ResNet and LGBM are chosen for further implementation into the FEM analysis process.
Another interesting thought is to use the calibration of AI algorithms on such numerical datasets for non-intrusive verification purposes of the implementa-

vol. 36/2022
AI-FEM-Hybrids for Nonlinear Concrete Materials tion of the ground truth data generating mechanism. Especially the data points in far distance to the diagonal line in Figs. 4, 5,6 give rise to outlier detection methods. It is currently under investigation whether the identified outliers are truly faulty data (i.e. the generating FEM material model is not correct) or the AI algorithm with its current state is not able to approximate the respective quantities well.

Conclusions and Outlook
This paper presented intermediate results of the first phase of a two-stage research project conducted currently at ETH Zürich. It reports on calibrating AIbased surrogates for a highly nonlinear reinforced concrete plate and shell model upon the CMM. All three AI algorithms have proven to furnish as valid candidates for surrogate models to predict the stress and stiffness tensors used within a FEM. The datadriven kNN algorithm performed slightly better that the LGBM and ResNet model for prediction standard deviation and RMSE, whereas on average all models show almost full correlation. Future research is concerned with further improving predictive capabilities by training an ensemble model using the kNN, ResNet and LGBM models. Finally, the non-intrusive verification analysis of the ANSYS USER-MAT by AI surrogate modelling will shed light on generalisation of this idea of physics informed ML [24].