Research progress in testing and evaluation technologies for autonomous driving

WANG De-cai, DONG Shi, HU Lei, HAO Pei-wen, ZHANG Qing, CHEN Yuan-zhao. Influencing factors and mechanism analysis for evaluation of fatigue characteristics of emulsified asphalt residues[J]. Journal of Traffic and Transportation Engineering, 2023, 23(2): 103-115. doi: 10.19818/j.cnki.1671-1637.2023.02.007

Citation:

ZHAO Xiang-mo's team supported by the National Key Research and Development Program of China (2021YFB2501200). Research progress in testing and evaluation technologies for autonomous driving[J]. Journal of Traffic and Transportation Engineering, 2023, 23(6): 10-77. doi: 10.19818/j.cnki.1671-1637.2023.06.002

Citation:

PDF( 19208 KB)

Research progress in testing and evaluation technologies for autonomous driving

doi: 10.19818/j.cnki.1671-1637.2023.06.002

ZHAO Xiang-mo's team supported by the National Key Research and Development Program of China (2021YFB2501200)

Funds:

National Key Research and Development Program of China 2021YFB2501200

Received Date: 2023-06-19
Publish Date: 2023-12-25

Abstract

Abstract

In view of the high cost, long cycle, low coverage, and lack of a perfect tool chain for autonomous driving vehicle tests in the actual complex traffic environment, the research status of seven major areas of testing and evaluation technologies for autonomous driving was analyzed, and the future development direction of testing and evaluation technologies for autonomous driving was predicted, including simulation and testing technology of autonomous driving vehicles, simulation and testing technology of traffic flow, hardware-in-the-loop testing technology, field testing technology, intelligence evaluation technology, testing and evaluation tool chain and its system's construction, certification and potential defect detection technology, etc. In terms of simulation and testing of autonomous driving vehicles, the research status of simulation and testing software for autonomous driving, vehicle dynamics models, background vehicle interaction behavior models for testing, simulation and testing of cloud control platform supervision, and standardization of vehicle simulation system was analyzed. The main problems currently existing in the simulation and testing of autonomous driving vehicles were summarized. In terms of simulation and testing of autonomous driving traffic flow, the research status of driving style models for test background vehicles, traffic flow modeling and simulation, traffic scenario generation methods, and acceleration testing methods was summarized, and the future development trend of simulation and testing of autonomous driving traffic flow was predicted. In terms of hardware-in-the-loop testing technology, the human-vehicle-road-loop multi-dimensional digital twin tests and construction methods of the system platform for autonomous driving vehicles were summarized. Typical sensor data from high-definition cameras, millimeter-wave radars, and ultrasonic radars, as well as simulation technology of vehicle-to-vehicle and vehicle-to-road communication signals, were reviewed. In terms of field testing technology, the development status of closed field testing, open road testing, and highway test-related testing fields, testing standards, and key technologies were summarized. In terms of intelligence evaluation technology, the research status of intelligence evaluation methods for autonomous driving was introduced from four aspects: the concept of autonomous driving intelligence, the quantification and evaluation of scene complexity, the intelligence evaluation systems of autonomous driving, and the social cooperation evaluation methods. In terms of testing and evaluation tool chain and system construction, the current situation of the testing and evaluation standard system for autonomous driving was introduced mainly from three aspects: the testing and evaluation tool chain technology, the autonomous driving testing evaluation system, and the current situation of autonomous driving testing standards. Finally, in terms of certification and potential defect detection technology, the current defect detection methods for autonomous driving were reviewed from the definition, cause, classification, and detection of autonomous driving defects. The challenges faced in the safety assurance of autonomous driving vehicles were summarized. Research results show that although the autonomous driving testing and evaluation technology has made great progress, the testing and evaluation standard system is still not perfect, and the existing testing tools and methods fail to meet the testing needs of autonomous driving vehicles at level three and above. The development and application level of virtual simulation and digital twin technology is low, and there are many deficiencies in the degree of simulation, testing efficiency, and vehicle testing ability. In the future, it is necessary to further strengthen the research and development of full-scene and high-fidelity modeling technology and real-time simulation software, establish an online accelerated twin testing system with virtual and real interaction, study the scenario generation and acceleration methods of autonomous driving full-stack hazardous testing, and integrate autonomous driving testing technologies and tools, so as to form a tool chain for autonomous driving testing and evaluation and improve standard specifications.
- intelligent transportation,
- autonomous driving,
- test and evaluation technology,
- traffic flow simulation test,
- hardware-in-the-loop test,
- field test,
- intelligence evaluation,
- test evaluation tool chain,
- defect detection

Author resume: ZHAO Xiang-mo(1966-), male, professor, PhD, xmzhao@chd.edu.cn

FullText(HTML)

FullText

Disclaimer: The English version of this article is automatically generated by Baidu Translation and only for reference. We therefore are not responsible for its reasonableness, correctness and completeness, and will not bear any commercial and legal responsibilities for the relevant consequences arising from the English translation.

0. 引言

Based on the above background, this article systematically reviews and analyzes the research progress of technologies such as autonomous driving simulation testing, hardware in the loop testing, field testing, evaluation methods, evaluation systems, certification, and defect recognition, addressing the core issues faced by autonomous driving testing and evaluation. It also looks forward to future development directions.

1. 自动驾驶测试评价面临的问题

In order to realize commercial application of autonomous vehicles at an early date, it is necessary to experience a lot of road tests, as well as the road license and efficiency evaluation of high-level auto drive system to adapt to the evolving traffic environment. The test of autonomous vehicles also needs to solve the basic problems of where the vehicle is, where the destination is, and how to test. To solve these problems, it involves multiple technical fields such as simulation tools, hardware platforms, software platforms, software algorithms, interaction, and security.

Figure 1. Three-pillars test method of autonomous driving

DownLoad: Full-Size Img PowerPoint

Table 1. Comparison of advantages and disadvantages of three autonomous driving test methods

方法	仿真测试	封闭场地测试	开放道路测试
对象	软件、硬件、虚拟环境	真实车辆、实际道路、模拟参与者	真实车辆、真实道路、真实参与者
优点	虚拟的测试环境；安全；丰富的测试场景；可重复且测试效率高	真实车辆和道路；封闭场景单一；可重复测试；安全可控	真实的车辆、道路和参与者；连续多变的测试场景；贴合实际应用
缺点	依赖车辆动力学模型；精度不高；无法验证车辆的执行能力	建设场地费用较大；测试场景数量有限；可升级性不高	路况不可控，安全风险大；需要大量的时间和成本投入；可重复性差

| Show Table

DownLoad: CSV

In the field of autonomous driving simulation testing, products represented by PreScan software from Siemens AG and Virtual Test Drive (VTD) software from VIRES AG dominate the commercial field of autonomous driving simulation testing. These types of software have long accumulated experience in vehicle dynamics modeling and are in a leading position in scene modeling and material rendering; In response to the demand for simulation testing of China's transportation environment and scale, Baidu and Tencent have developed Apollo and TAD Sim simulation testing software respectively, which have been deployed in multiple enterprises. In addition, Tongji University has also achieved significant research results in vehicle dynamics modeling. However, existing simulation testing techniques generally suffer from defects such as coarse-grained sensor models, distorted scene descriptions, and insufficient accuracy of vehicle dynamics models; At the same time, there are also problems with the relative independence of micro behavior simulation and macro traffic flow simulation functions, low integration and coupling of functions, single scene generation methods, and difficulty in achieving high coverage acceleration testing.

Figure 2. Integrated organizational structure

DownLoad: Full-Size Img PowerPoint

2. 自动驾驶车辆仿真测试技术

Figure 3. Organizational structure of simulation and testing technology of autonomous driving vehicles

DownLoad: Full-Size Img PowerPoint

2.1 自动驾驶仿真测试软件(朱西产、刘瑞)

Due to the approximately infinite scenarios faced by autonomous driving and the difficulty of scene generalization, the typical scenario testing methods commonly used in traditional automotive testing fields can no longer meet the testing needs of autonomous driving. Research shows that it takes over one billion kilometers of road testing to statistically prove that autonomous driving is safer than human driving. How to improve testing efficiency and reduce testing costs through simulation has become a research focus and hotspot. Europe has gradually improved a set of autonomous driving testing procedures based on simulation testing, in the loop testing, and road testing through projects such as AdaptIVe and PEGASUS. The United Nations World Forum for the Coordination of Vehicle Regulations Working Group on Autonomous Driving and Connected Vehicles has proposed a multi pillar testing method on this basis. Simulation has become one of the most important testing methods throughout the entire lifecycle of autonomous driving design, evaluation, licensing, and on road use. In order to improve the authenticity of simulation scenarios and the discrimination of simulation testing, high-level autonomous driving testing has put forward higher requirements for the accuracy and fidelity of simulation. Research is needed on key technologies such as high-precision vehicle dynamics models, anthropomorphic testing background vehicle models, high-value testing scenarios, high fidelity environments and sensor models, cloud control platforms, etc., to improve the effectiveness of autonomous driving simulation testing.

Table 2. Main simulation test tools and their characteristics

主要目的	软件	所属公司	车辆动力学模型	传感器与环境模型	交通参与者模型
车辆动力学仿真	CarSim	MSC	好	有	有
车辆动力学仿真	CarMaker	IPG	好	有	有
交通流仿真	VISSIM	PTV	无	无	良
交通流仿真	SUMO	DLR	无	无	良
ADAS仿真测试	PreScan	TASS	有	良	有
	VTD	VIRES	有	良	有
	SCANeR	OKTAL	有	良	有
	Panosim	天行健	良	良	有
	CARLA	Intel	无	有	有
自动驾驶仿真测试	Carcraft	Waymo	有	好	良
	TAD Sim	腾讯	有	好	良
	51Sim-One	51WORLD	有	好	良
	Apollo	百度	有	好	良

| Show Table

DownLoad: CSV

(4) Sensor models, environmental models, and traffic participant behavior models with higher fidelity. The simulation accuracy of perception systems, environment, and traffic participant behavior has a decisive impact on the effectiveness of simulation testing. High level autonomous driving simulation software should improve the modeling accuracy of sensor models and environmental models with the support of computing power, new modeling methods, and new computer technologies. At the same time, based on the widely used traffic flow models, driver behavior data collected from real roads should be used to improve the realism of the traffic participant model and make it have reasonable randomness and uncertainty.

2.2 车辆动力学模型(冷搏、刘铭)

In the control of autonomous vehicles, the establishment and analysis of vehicle models are the basis for controller design. Overall, vehicle models typically play a crucial role in two aspects. On the one hand, the control law in autonomous vehicle control sometimes originates from the mathematical representation of the vehicle system, namely the vehicle control model. On the other hand, there is vehicle system simulation, which refers to vehicle simulation models.

2.2.1 Vehicle Control Model

$\left[\begin{array}{c} v_x \\ v_y \\ r \end{array}\right]=\left[\begin{array}{cc} \cos (\mathit{\Psi}+\beta) & 0 \\ \sin (\mathit{\Psi}+\beta) & 0 \\ 0 & 1 \end{array}\right]\left[\begin{array}{l} v \\ \dot{\mathit{\Psi}} \end{array}\right]$

(1)

Figure 4. Kinematics model of bicycle

DownLoad: Full-Size Img PowerPoint

A dynamic model is a vehicle model that describes the motion of a vehicle's position, velocity, and acceleration by considering the internal forces, energy, or momentum of the system. In the longitudinal plane, the model typically includes lateral motion, longitudinal translational motion, and rotationzThe rotational motion of the shaft.Figure 5For the vehicle dynamics model, in the figure:CFor the center of mass of the vehicle;α_ijTo determine the lateral deviation angle of the vehicle;F_xijThe longitudinal force acting on the tire;F_yijTo apply lateral force on the tire;l_iThe distance from the center of mass to the center of the front or rear axle; subscriptiandjFor the position of the wheels,i=f, R represents the front and rear axles respectively,j=l, R represents left wheel and right wheel respectively;δFor the front wheel steering angle;lThe distance between the front and rear axes.

Figure 5. Dynamics model of vehicle

DownLoad: Full-Size Img PowerPoint

The vehicle dynamics model can be represented as^[7]

$\begin{array}{l} F_{x \mathrm{rr}}+F_{x \mathrm{rl}}+F_{x \mathrm{fl}} \cos (\delta)+F_{y \mathrm{fl}} \sin (\delta)+F_{x \mathrm{fr}} \cos (\delta)+ \\ \ \ \ \ \ \ \ F_{y \mathrm{fr}} \sin (\delta)=m_{\mathrm{b}} a_x \end{array}$

(2)

$\begin{array}{l} F_{y \mathrm{rr}}+F_{y \mathrm{rl}}-F_{x \mathrm{fl}} \sin (\delta)-F_{x \mathrm{fr}} \sin (\delta)+F_{y \mathrm{fl}} \cos (\delta)+ \\ \ \ \ \ \ \ \ F_{x \mathrm{fr}} \cos (\delta)+F_{y \mathrm{fr}} \cos (\delta)=m_{\mathrm{b}} a_y \end{array}$

(3)

$\begin{aligned} I_z \ddot{{\mathit{\Psi}}} & =\left(-F_{y \mathrm{rr}}-F_{y \mathrm{rl}}\right) l_{\mathrm{r}}+\left[F_{x \mathrm{f} \mathrm{l}} \sin (\delta)+F_{y \mathrm{fl}} \cos (\delta)+\right. \\ &\ \ \ \left.F_{x \mathrm{fr}} \sin (\delta)+F_{y \mathrm{fr}} \cos (\delta)\right] l_{\mathrm{f}}+\left[F_{x \mathrm{fr}} \cos (\delta)-\right. \\ &\ \ \ F_{x \mathrm{fl}} \sin (\delta)+F_{y \mathrm{fl}} \sin (\delta)-F_{y \mathrm{fr}} \sin (\delta)- \\ &\ \ \ \left.F_{x \mathrm{rl}}+F_{x \mathrm{rr}}\right] \frac{L}{2} \end{aligned}$

(4)

In the formula:m_bFor the overall spring-loaded mass of the vehicle;I_zTo wrap aroundzThe moment of inertia of the shaft;a_xAcceleration for longitudinal motion;a_yLateral motion acceleration;LFor the wheelbase.

Table 3. Advantages and disadvantages of different control models

模型类型	车辆模型	优点	缺点
几何模型	基于阿克曼转向的几何模型	简单，只需少量参数；有效描述车辆位置与路径的关系；控制器设计简单	未考虑车辆运动学与动力学特性
运动学模型	全车运动学模型	考虑到相对于局部坐标和全局坐标的车辆航向之间存在不同方向的可能性	未考虑车辆动力学特性；略微比几何模型复杂
	半车运动学模型(自行车模型)	模型简单，便于控制器的设计	未考虑车辆动力学特性；假设车辆相对于局部坐标的航向与相对于全局坐标的航向相同
	带侧偏角的运动学模型	模型中考虑车辆的侧偏特性	未考虑车辆动力学特性；增加了建模复杂度
动力学模型(线性)	全车动力学模型	考虑了所有车轮的力，尤其是在转向运动中	未考虑轮胎的非线性特性
动力学模型(线性)	半车动力学模型(自行车模型)	考虑了车辆的动力学特性，较为常用	未考虑轮胎的非线性特性；忽略左右车轮不同响应带来的影响
动力学模型(非线性)	全车动力学模型	考虑了所有车轮的力，尤其是在转向运动中；考虑轮胎响应相对于侧偏角的非线性	模型复杂，不能保证控制的实时性
动力学模型(非线性)	半车动力学模型(自行车模型)	考虑轮胎响应相对于侧偏角的非线性	未考虑转向时内外侧车轮的差异带来的影响

| Show Table

DownLoad: CSV

2.2.2 车辆仿真模型

Figure 6. Two-degree-of-freedom vehicle model

DownLoad: Full-Size Img PowerPoint

$m_{\mathrm{b}}\left(\dot{v}_y+v_x r\right)=C_{{\rm{ \mathsf{ α} }} \mathrm{f}} \delta-\frac{v_y\left(C_{{\rm{ \mathsf{ α} }} \mathrm{f}}+C_{{\rm{ \mathsf{ α} }} \mathrm{r}}\right)}{v_x}-\frac{r\left(l_{\mathrm{f}} C_{{\rm{ \mathsf{ α} }} \mathrm{f}}-l_{\mathrm{r}} C_{{\rm{ \mathsf{ α} }} \mathrm{r}}\right)}{v_x}$

(5)

$I_z \dot{r}=l_{\mathrm{f}} C_{{\rm{ \mathsf{ α} }} \mathrm{f}} \delta-\frac{v_y\left(l_{\mathrm{f}} C_{{\rm{ \mathsf{ α} }} \mathrm{f}}-l_{\mathrm{r}} C_{{\rm{ \mathsf{ α} }} \mathrm{r}}\right)}{v_x}-\frac{r\left(l_{\mathrm{f}}^2 C_{{\rm{ \mathsf{ α} }} \mathrm{f}}+l_{\mathrm{r}}^2 C_{{\rm{ \mathsf{ α} }} \mathrm{r}}\right)}{v_x}$

(6)

Figure 7. System coordinate system in Lagrangian approach

DownLoad: Full-Size Img PowerPoint

$\begin{array}{l} m_{\mathrm{b}} h \ddot{\theta}-m_{\mathrm{b}} h \ddot{\varphi}+\left(m_{\mathrm{f}} l_{\mathrm{f}}-m_{\mathrm{r}} l_{\mathrm{r}}\right) \ddot{\mathit{\Psi}}+\left(m_{\mathrm{b}}+m_{\mathrm{f}}+m_{\mathrm{r}}\right) \cdot \\ \ \ \ \left(\dot{v}_y+v_x r\right)=F_{y \mathrm{fl}}+F_{y \mathrm{fr}}+F_{y \mathrm{rl}}+F_{y \mathrm{rr}} \end{array}$

(7)

$\begin{array}{l} I_{z z} \ddot{\mathit{\Psi}}-I_{y z} \ddot{\theta}-I_{x z} \ddot{\varphi}+\left(m_{\mathrm{f}} l_{\mathrm{f}}-m_{\mathrm{r}} l_{\mathrm{r}}\right)\left(\dot{v}_y+v_x r\right)= \\ \ \ \ l_{\mathrm{f}}\left(F_{y \mathrm{fl}}+F_{y \mathrm{fr}}\right)-l_{\mathrm{r}}\left(F_{y \mathrm{rl}}+F_{y \mathrm{rr}}\right) \end{array}$

(8)

$\begin{aligned} & I_{x x} \ddot{\varphi}-I_{x z} \ddot{\mathit{\Psi}}-I_{y x} \ddot{\theta}+m_{\mathrm{b}} h\left(\dot{v}_y+v_x r\right)+\left(K_{\varphi \mathrm{f}}+K_{\varphi \mathrm{r}}-\right. \\ &\ \ \ \ \ \left.m_{\mathrm{b}} g h\right) \varphi+\left(C_{\varphi \mathrm{f}}+C_{\varphi \mathrm{r}}\right) \dot{\varphi}=d_{y \mathrm{f}}\left(F_{y \mathrm{fl}}+F_{y \mathrm{fr}}\right)- \\ & \ \ \ \ \ d_{y \mathrm{r}}\left(F_{y \mathrm{rl}}+F_{y \mathrm{rr}}\right) \end{aligned}$

(9)

$\begin{array}{l} -m_{\mathrm{b}} h\left(\dot{v}_y+v_x r\right)+I_{y y} \ddot{\theta}-I_{y x} \ddot{\varphi}-I_{y z} \ddot{\mathit{\Psi}}+\left(K_{\theta \mathrm{f}}+\right. \\ \ \ \ \ \ \left. K_{\theta \mathrm{r}}-m_{\mathrm{b}} g h\right) \theta+\left(C_{\theta \mathrm{f}}+C_{\theta \mathrm{r}}\right) \dot{\theta}=d_{z \mathrm{f}}\left(F_{z \mathrm{fl}}+F_{z \mathrm{fr}}\right)- \\ \ \ \ \ \ d_{z \mathrm{r}}\left(F_{z \mathrm{rl}}+F_{z \mathrm{rr}}\right) \end{array}$

(10)

Variational methods are an important part of classical mechanics. If in classical mechanics, the variational principle is only an abstract summary of mechanical phenomena, then in today's rapidly developing computational technology, variational methods have become an effective way to directly seek motion laws through numerical calculations without the need to establish dynamic equations. Yang Huaiguang^[20]Based on this, establish a universal dynamic model for Mars rovers.

2.3 测试背景车交互行为模型(刘瑞、袁田)

Many studies have also considered the personalized driving behavior characteristics of drivers. Butakov et al^[30]A two-layer model structure combining stochastic modeling and kinematic modeling is proposed to describe lane changing behavior. The lower kinematic model is a sine lane changing model that takes the vehicle's motion state and surrounding vehicle information as inputs. The upper model uses Gaussian mixture to establish kinematic model parameter values for specific drivers to adapt to the response behavior of individual drivers; Zhang et al^[31]Propose a generative hybrid model to simulate the following behavior of drivers. Firstly, calibrate an Intelligent Driver Model (IDM) with time-varying parameters to capture the heterogeneity of drivers. Then, apply a neural process (NP) - based model to generate the following behavior of any given observed driving style. By exploring the relationship between time-varying IDM parameters and NP intermediate variables, infer the following behavior of unobserved driving styles, and then use unsupervised clustering analysis to classify different driving styles and model different driving behaviors; Chen et al^[32]Cluster analysis was conducted using average speed and speed standard deviation as feature vectors, resulting in three driving styles: slow, fast, and stable. Based on road features, a human centered continuous speed model was established to develop a human like speed control strategy suitable for mountainous highways; Huang Jing and others^[33]useK-The means clustering method divides driving styles into aggressive, ordinary, and cautious, and improves the elliptical vehicle model to describe the driving safety zones of drivers with different styles. A minimum safety distance model is constructed under typical working conditions, achieving lane change trajectory planning that is suitable for driving styles; Ren et al^[34]Similarly, clustering driving styles into aggressive, stable, and cautious, and then using a multi-layer perceptron neural network to train driver models with different styles, significantly improved the prediction accuracy of left and right lane changes; Zhang et al^[35]Applying driving operation diagrams to describe the characteristics that affect driving style, studying the short-term driving style of vehicles based on historical trajectories, and using driving style and traffic information as inputs to a deep neural network, a lane change decision model with driving style awareness was established, which maximally imitates the decision-making actions of human drivers.

2.4 云控平台监管的仿真测试(毕欣、周小钧、仝盼盼)

2.4.1 Application and Research of Cloud Control Platform in Connected Vehicle Cloud Services

At present, research on cloud control platforms for intelligent connected vehicle cloud services is developing rapidly both domestically and internationally. The main research directions focus on safe driving, enhanced driving performance, and traffic and route planning^[54].

2.4.2 云控平台在监管方面的应用与研究

2.5 自动驾驶车辆仿真系统标准化(孙驰天、郭楚凡、张丰婕)

The Association for Standardization of Automation and Measuring Systems (ASAM) is a non-governmental standardization organization in the automotive industry, founded in 1998 by digital industry experts for data exchange in the development and testing of standardized electronic control units (ECUs). ASAM is committed to achieving free exchange of data information at all stages of the development process. As of 2019, a total of 295 host manufacturers, suppliers, and research institutions from Asia, Europe, and North America have joined as members. The standards launched by ASAM involve multiple automotive standard fields, including simulation, vehicle networking, measurement and calibration, diagnosis, automated testing, software development, ECU networking, and data management and analysis.

The OpenDRIVE standard defines a method for describing static scenes. High precision maps captured by visual sensors or LiDAR can be expressed in the OpenDRIVE format and are widely used by manufacturers such as Audi, Bosch, NavInfo, and Baidu. In terms of syntax, OpenDRIVE adopts the widely used XML format syntax; In terms of content, under the premise of conforming to the basic framework of XML format, the OpenDRIVE standard defines the content nodes of maps, roads, and buildings that need to be described, and users can expand these nodes; In road information description, it mainly includes major categories such as reference lines, lane markings, road horizontal and vertical slopes, side shapes, road network connections and intersection information, adjacent lane information, road surface information, and railway information, as well as more minor subcategories. Users need to fill in the corresponding data according to the classification. The OpenDRIVE standard unifies the content entries and formats in static scene data, facilitating compatibility between different simulation software for the same scene file.

The OpenCRG standard mainly focuses on the description of physical information related to road surfaces, mainly used for friction related issues in tire ground contact. OpenCRG will allow for realistic 3D rendering of road surfaces in simulation software, including information such as friction coefficient and grayscale of the road surface. Through curved grid like segmentation, OpenCRG will achieve high-precision description of roads and reduce memory usage and computation time. Meanwhile, OpenCRG supports interaction with OpenDRIVE format static scenes and OSI dynamic models, and is compatible with software such as MATLAB.

(5) OpenLABEL

In response to the problem of unclear definition of the design and operation scope in the autonomous driving industry, ASAM Operational Design Domain (ODD) first clarified the standardized definition and requirements, and developed a machine-readable ODD concept standardized description language. The standardized definition of ODD is particularly important for simulation based testing, as it allows users to share, compare, and reuse ODD concepts, avoiding industry waste of resources on repeatedly constructing standard components. In addition, the standardized definition of ODD will help the government standardize the assessment of the safety of autonomous vehicle deployment and eliminate communication barriers between autonomous vehicle manufacturers and the government.

Figure 8. Research framework and content expansion of C-ASAM working group

DownLoad: Full-Size Img PowerPoint

2.6 总结与展望

(1) The vehicle dynamics model has a significant impact on the authenticity of autonomous driving simulation, and in-depth research on vehicle dynamics is needed to improve the accuracy of the vehicle model.

(4) The standard format and interface for simulation testing are of great significance in promoting the development of autonomous driving simulation testing.

3. 自动驾驶交通流仿真测试技术

The purpose of autonomous driving traffic flow simulation testing is mainly to test the safety and traffic efficiency of autonomous driving in evolving traffic environments. Therefore, this section mainly introduces the development status of key technologies in autonomous driving testing, such as background vehicle driving style model, background traffic flow model, traffic scene generation method, and acceleration testing method, from the perspective of traffic flow simulationFigure 9As shown.

Figure 9. Organizational structure of simulationand testing technology of autonomous driving traffic flow

DownLoad: Full-Size Img PowerPoint

3.1 测试背景车驾驶风格模型(孙剑、岳李圣飒、田野、朱奕昕)

3.1.1 交通流仿真中的驾驶人模型

(2) Driver Model Based on Deep Learning

GAN uses the mutual confrontation between the generator and discriminator to make the simulated driving behavior distribution increasingly close to reality. GAN includes various structures, such as recursive GAN, parameter sharing GAN, conditional GAN, etc. Researchers use GAN to study driving behavior in lane changing, overtaking, merging, insertion, and unexpected scenarios^[79]We conducted simulation modeling and achieved good accuracy.

3.1.2 自动驾驶虚拟测试对驾驶人模型的要求

The micro traffic model improved based on human factors theory has the best interpretability and transferability among the three types of models, which can reveal the influence and cognitive process of human factors on driving behavior, and the relevant rules can be universally applied in different scenarios. At the same time, the complexity of this type of model is low. However, the authenticity and diversity of the model need to be improved.

The driver model based on reinforcement learning can reflect the driver's cognitive process through the reward function, and has certain interpretability and transferability, with higher accuracy than micro traffic models. However, current research mostly focuses on car following and lane changing scenarios, and the diversity ability is not as good as driver models based on deep learning.

3.1.3 虚拟测试双向交互仿真3类需求下驾驶人模型仿真方案

(3) Fragmented simulation

Segmental simulation is a simulation based on scene segments, with a shorter simulation time, and is often used for simulating high-value boundary scene libraries. The purpose of setting up fragmented simulation is to finely control high-value boundary scenes, simulate high-risk vehicle interactions and accident processes in real environments as realistically as possible, and accurately evaluate the safety of autonomous driving. Segmental simulation focuses on the authenticity of the model, especially on reproducing abnormal and erroneous driving behaviors related to risk and accident occurrence.

3.2 交通流建模与仿真技术(田野、孙剑)

The commonly used solution in the industry to address the problem of missing traffic flow environment is to integrate micro traffic flow simulation software. In the field of transportation engineering, traffic flow simulation software generates a traffic flow environment through virtualization to simulate and evaluate the expected effects of various traffic control measures. Microscopic traffic flow simulation software focuses on the real interaction process between vehicles, taking individual vehicles as objects, and constructing vehicle driving behavior models to simulate the motion state of vehicles under different road and traffic conditions. Therefore, the bicycles in the microscopic traffic simulation model can also be used as the background traffic flow for autonomous driving testing, and thus integrated into the autonomous driving testing platform.

Figure 10A complete micro traffic simulation prototype system was demonstrated, and the traffic flow model is the key to determining the simulation performance. According to the duration and spatial range of driving behavior, from large to small, driving behavior is usually divided into three levels: strategic level, tactical level, and operational level, corresponding to paths, sections, and lanes in the road network, respectively. Later, it also developed into a structured modeling paradigm.Figure 11The structure of the core model for micro traffic simulation was demonstrated. There are numerous sub models in the three levels, and their simulation focuses are also different. The strategic level behavior model mainly focuses on the problem of path selection from the starting point to the destination of travel, playing an important role in meso and macro simulations, while micro simulations tend to model driving behavior at the tactical and operational levels.

Figure 10. Prototype system of micro traffic simulation

DownLoad: Full-Size Img PowerPoint

Figure 11. Structure of micro traffic simulation key model

DownLoad: Full-Size Img PowerPoint

At present, virtual testing tools for ADAS, such as CarSim and PreScan, mostly use the method of integrating traffic flow simulation models to depict the traffic flow environment. However, due to the internal data transmission between ADAS testing tools and traffic flow simulation software, and the different road network formats used by the two software, road network matching is required. These interface related tasks need to be completed with the cooperation of the software vendor. Due to the openness of the interface, the operability of third-party users is very limited and cannot meet the requirements of open source simulation testing platforms.

Traditional traffic flow simulation motion modeling usually focuses on motion along the lane direction, that is, one-dimensional simulation based on lanes, such asFigure 12 (a)As shown, white represents pedestrian crossings, pink represents lane connecting lines, blue represents lane dividing lines, yellow represents vehicle paths, and deep red represents parking lines. Vehicles on the lane mainly update their positions according to the following model, and perform lane changing actions at limited positions to express the interaction between lanes. However, in reality, in a large number of unstructured roads, the behavior of vehicles is motion on the surface (both longitudinal and lateral). in compliance withFigure 12 (b)As shown, vehicles do not only focus on the direction of lane movement, but make motion planning decisions in a two-dimensional plane space. Therefore, traditional one-dimensional traffic flow simulation is suitable for evaluating the efficiency of transportation system operation, with typical indicators including travel time, delay, queue length, etc. It is a simulation methodology guided by efficiency evaluation. However, one-dimensional simulation is difficult to reflect the microscopic behavioral characteristics under highly interactive and complex interference, which is exactly what is needed for safety testing of autonomous vehicles. A two-dimensional simulation model based on regions can meet the requirements of safety oriented traffic flow modeling and simulation, providing better services for the safety testing of autonomous vehicles.

Figure 12. Lane-based modeling description of traditional simulation

DownLoad: Full-Size Img PowerPoint

Figure 13. Traffic flow based on area movement on unstructured roads in reality

DownLoad: Full-Size Img PowerPoint

However, current micro simulation methods for traffic flow mostly focus on normal traffic flow and belong to "rule-based" traffic flow simulation, with very little attention paid to the interference behavior between vehicles. Therefore, existing virtual testing methods urgently need to have the ability to reproduce typical interference behaviors in high demand testing scenarios, in order to simulate the potential interference of vehicles around autonomous vehicles, and better leverage the advantages of short virtual testing cycles and fast speeds to further promote the process of autonomous vehicle testing.

Due to the significant differences among traffic participants and the lack of hard isolation between different road types, urban roads experience various interferences from heterogeneous traffic participants. For example, as a traffic flow hub, intersections have a high accident rate, and a large part of the reason is due to the illegal passage of non motorized vehicles and pedestrians. In addition, there are also various non motorized vehicles crossing the solid line and entering the motor vehicle lane on the main and secondary roads connecting intersections. The above phenomena have raised higher requirements for the safe driving of autonomous vehicles, especially in response to violations, and have also posed new challenges to the modeling ability of traffic flow simulation.

In summary, the testing method based on road sampling scenarios involves "replay" testing of fixed scenarios, which can only achieve one-way interaction between the background vehicle and the main vehicle. Additionally, there are issues such as difficulty in collecting scene data, which further highlights its inherent limitations in autonomous driving testing. In recent years, the testing method of generating virtual traffic flow by simulating real traffic environments has received increasing attention. Currently, autonomous driving virtual testing platforms have certain traffic flow simulation capabilities, but the reliability and validity of their traffic flow simulations are generally low. Specifically, the existing one-dimensional and normal traffic flow simulation models mainly focus on evaluating traffic efficiency, and the various interactive behaviors between vehicles are not clearly described, which makes them unsuitable for vehicle safety testing. Therefore, it is necessary to further propose new requirements for traffic flow models for autonomous driving safety testing based on traditional traffic flow simulation, develop deep simulation capabilities such as two-dimensional simulation and interference simulation, integrate traffic flow simulation systems with autonomous driving virtual testing tools, and improve the reliability and validity of autonomous driving virtual safety testing.

3.3 交通场景生成方法(景首才、赵玉钰、杨扬)

3.3.1 加速搜索的关键场景生成方法

There are two main methods for accelerating scene search: sampling based methods and adaptive search based methods. The sampling methods mainly include importance sampling method and Monte Carlo sampling method. Zhao et al^[101]Propose an acceleration evaluation method based on importance sampling, and construct lane changing and following models for test background vehicles based on natural driving data. Analyze the efficiency of importance sampling and Monte Carlo sampling, propose an optimal importance sampling method based on iterative search, estimate the frequency of model conflicts and collisions, and validate the algorithm using following and lane changing scenarios; Feng et al^[102]To identify key scenarios, a unified framework for generating a test scenario library is proposed. To evaluate the importance of scenarios, a scenario criticality measurement index is constructed, and a scenario screening method based on importance sampling for high criticality measurement is proposed. The accelerated generation architecture is validated using three scenarios: lane change, following, and highway exit.

3.3.2 Data model driven method for generating hazardous scenarios

3.4 Accelerated testing methods (Zhang He, Sun Jian, Tian Ye)

3.4.1 Accelerated search method for dangerous scenarios

3.4.2 面向危险率的加速评估方法

3.4.3 面向场景空间的加速覆盖方法

3.4.4 总结

3.5 展望

4. 硬件在环测试技术

Figure 14. Organizational structure of hardware-in-the-loop testing technology

DownLoad: Full-Size Img PowerPoint

4.1 人-车-路-环多维数字孪生测试技术(彭铖、王振)

V2X technology can achieve real-time communication between traffic elements centered around vehicles, providing non visual perception data for vehicles, establishing communication links between vehicles and cloud data centers, and realizing communication between the real world and the virtual world^[140]In the environmental testing of autonomous vehicles, V2X technology can be used to send scene information to the testing vehicle and provide road virtual testing function, achieving the integration of data flow between existing physical space and digital space in two directions^[141]The China Academy of Information and Communication Technology has achieved sensor data upload and virtual scene information transmission through V2X communication in the digital twin autonomous driving testing system^[142-143]Ge and others^[144]A three-layer testing architecture was proposed, which establishes communication links between virtual and physical networks using 4G/5G networks, and collects vehicle and road information using LTE-V2X technology; Szalay et al^[145]A low latency networked testing based on digital twins in 5G environment was studied.

(2) Explore the large-scale architecture and efficient scenario generation of simulation testing systems based on augmented reality.

4.2 车路耦合的整车级系统平台构建技术(承靖钧、王振)

The extremely complex working mechanism makes the behavior of autonomous vehicles highly random and unexplainable. Therefore, testing the functions and performance of each subsystem alone is not sufficient to comprehensively explore the comprehensive capabilities of autonomous vehicles, nor can it guarantee the safety and reliability of the vehicles. On this basis, it is necessary to combine multi-dimensional digital twin testing platforms to conduct vehicle level testing (i.e. vehicle in the loop simulation testing) on autonomous vehicles^[147]Connect it as a test object to the digital twin testing platform, and evaluate its comprehensive ability by observing and analyzing its overall behavior.

4.2.1 基于封闭场地的整车在环测试

The closed field testing method provides real traffic scenarios and physical conditions for autonomous driving during actual vehicle testing. Compared with virtual simulation testing and open road testing methods, the closed field testing method balances the authenticity and efficiency of testing, and plays a key role in the development of autonomous driving. Therefore, it is more popular in the environmental simulation testing of autonomous driving vehicles. However, traditional closed field testing methods have extremely high costs in setting up testing scenarios, and the types and quantities of scenarios they can support are still limited by the scope of the site. Therefore, the whole vehicle in the loop testing method based on closed fields, combined with multidimensional digital twin testing platforms, provides rich and realistic testing scenarios for autonomous driving.

(3) Due to the time-varying physical conditions and limitations of the site, this method is not suitable for long-term, long-distance autonomous driving testing.

4.2.2 Vehicle in the loop testing based on test bench

In the traditional field of automobile inspection, it is usually necessary to use a test bench to test the overall capability of the vehicle, especially in the testing stage at the end of the production line, such as using a chassis dynamometer to verify the braking ability of the vehicle and the function of the anti lock braking system. Inspired by traditional automotive inspection, the Dutch Applied Science Research Organization first proposed the Vehicle Hardware in the Loop (VEHIL) testing method in 2000^[147]Placing autonomous driving on a chassis dynamometer indoors to simulate real road conditions, providing rich testing scenarios for autonomous driving through real-time interaction in a virtual traffic environment.

Based on VEHIL testing method, Gietelink et al^[158]Further utilizing the chassis dynamometer, a series of functional tests and fault diagnosis were conducted on ADV, such as Adaptive Cruise Control (ACC), Collaborative Adaptive Cruise Control, and FCW function; Albers et al^[159]A standardized testing process has been proposed for the vehicle level testing platform based on the drum test bench to adapt to the complex characteristics of ADV. This process configures and specifies the corresponding testing environment for ADV according to the relevant requirements of the testing task; Galko et al^[160]Use the chassis dynamometer manufactured by Horiba Corporation in Japan to test the ADAS of vehicles.

To solve this problem, Shijiazhuang Huayan Transportation Technology Co., Ltd. has developed a three degree of freedom steerable chassis dynamometer, which simulates three postures for ADV based on the road model in the virtual scene, namely pitch angle, roll angle, and heading angle; Zhao Xiangmo and others^[166]A multi-dimensional digital twin testing platform has been developed based on CAVTest at Chang'an University, which is designed for various road types (including long straight roads, sloping roads, S-shaped curves, etc.) and provides rich traffic scenarios. Combined with the three degree of freedom steerable roller test bench, the whole vehicle in the loop simulation test was carried out to test multiple functions of ADV, such as ACC, AEB, LKA, and lane changing, in a virtual real combination.

Schyr et al^[168]A whole vehicle in the loop testing framework, DrivingCube, was proposed using a powertrain test bench. The test results showed that the framework can simulate more real test cases for ADV, effectively improving testing efficiency; Li et al^[169]Based on KS company in Germany^[170]The powertrain test bench, SUMO, and CarMaker have built a joint simulation testing framework and verified the ACC and AEB functions of ADV. The test results show that the powertrain test bench can flexibly simulate different road conditions and truly reproduce the tire slip rate and nonlinear dynamics of the vehicle.

Regarding this issue, future research can focus on simulating the longitudinal/transverse acceleration of the vehicle's center of mass on the test bench based on the behavior and trajectory of virtual ADV in the digital twin testing platform, and then obtaining the real dynamic behavior of the vehicle through onboard inertial sensors.

4.3 自动驾驶典型传感器物理信息模拟技术(薛丁瑞、王振)

4.3.1 High definition camera sensor information simulation

High definition cameras, as the main in car perception devices for autonomous driving, are mainly used for identifying obstacles/other traffic participants and collecting elements such as traffic markings and signal lights. According to different types, it is mainly divided into monocular, binocular, infrared, and fisheye cameras, which are mainly composed of image acquisition, processing, and feature extraction modules^[171].

4.3.2 毫米波雷达信息模拟

The purpose of millimeter wave radar in the loop testing is to generate corresponding electromagnetic wave signals through radar perception models in the millimeter wave radar dark box based on expected physical information such as distance, speed, angle, and radar cross section, and send them to the millimeter wave radar under test, so that the millimeter wave radar can perceive different targets according to the designed scene.

4.3.3 GNSS模拟

Accurate positioning and high-definition maps can provide rich and precise prior information for autonomous vehicles, greatly improving the accuracy of autonomous driving perception. GPS satellite signal simulation research has been carried out earlier abroad, and mature products have been commercially available, including STEL-9200 from Stanford Telecom in the United States, GSS9000 and GS7000 series from SPIRENT in the United Kingdom, etc. After several generations of product updates, related products can now simulate differential signals and multiple output functions.

The main process of GNSS simulation is to simulate the positioning process between real satellites and related targets on Earth, establish a positioning model, and then simulate satellite signals. This device is capable of recording and replaying satellite historical signals with strong repeatability. GNSS simulation devices are widely used in military applications, so there are not many open technical documents available. In recent years, with the rapid development of autonomous driving technology, some institutions have attempted to use this technology for testing the positioning performance of autonomous vehicles, greatly reducing the cost of vehicle positioning testing and improving the efficiency of positioning testing.

4.3.4 超声波模拟

Compared to the above sensors, ultrasonic sensors have a smaller application range, but their principle is simple and the cost is lower. The main working principle of ultrasonic sensors is to detect by emitting 40 kHz ultrasonic waves, and then receive the reflected ultrasonic waves to calculate the distance to the target obstacle through time difference.

The simulation of ultrasound is similar to the dark box principle of millimeter wave radar, which is achieved by sending echoes through an ultrasound echo simulator. The time delay of the echo simulator's signal transmission can be manually adjusted to simulate the detection of target objects at different distances^[174].

4.3.5 V2X signal simulation

In recent years, in order to accelerate the commercialization process of autonomous driving, it has become urgent to conduct sufficient and comprehensive testing of various components of the autonomous driving perception system to ensure its safety on the road. For different sensors, researchers widely use various means such as dark box and virtual signal simulation to generate corresponding sensor signals. However, there are still some sensors (such as LiDAR) that are difficult to simulate signals, and some sensor signal simulations have low authenticity. Innovative regeneration is needed at the level of sensor perception principles to solve the above problems.

4.4 Summary and Outlook

Hardware in the loop simulation testing has high testing accuracy and efficiency, and has gradually become an indispensable part of autonomous vehicle testing and evaluation. Virtual simulation scenarios are the core of hardware in the loop testing, and interactivity, real-time performance, and authenticity are the starting points of hardware in the loop testing. This section extensively researched the latest research literature and technological development progress in the industry, and systematically summarized the digital twin testing technology, vehicle level system platform construction technology, and vehicle mounted sensor physical information generation technology platform in hardware in the loop testing based on the above three points. Although hardware in the loop simulation has made some progress in the field of autonomous vehicle testing, current research and technology are still unable to meet the testing needs of high-level (L3 or above) autonomous vehicles, especially in terms of real-time and authenticity. The development considerations for future research are as follows.

(1) Low latency communication technology for digital twin testing platform. Interaction is the foundation of the multidimensional digital twin testing platform, and communication is the technical support for achieving virtual real interaction. As a highly complex physical information system, in order to ensure safety and reliability, autonomous vehicles need to interact in real-time with surrounding elements in the virtual simulation scene during the testing process. Future research first needs to further refine the authenticity of the human vehicle road environment model in digital twin systems and expand the scale of virtual testing scenarios. On this basis, we will improve existing communication technologies, with a focus on V2X high-throughput and low latency communication research in large-scale testing scenarios, in order to enhance the reliability of autonomous vehicle testing. In addition, due to the wide variety of autonomous vehicles and the different focuses of virtual simulation scenarios used for testing, with the development of digital twin based autonomous driving testing technology, building a unified digital twin testing standardized communication protocol is also one of the future research directions.

(2) Vehicle center of mass acceleration simulation technology for test benches. The whole vehicle test bench can support the longitudinal/lateral movement of vehicles and simulate real road conditions, but in order to ensure the safety of testers, vehicles are usually fixed on the test bench, which cannot fully reflect the dynamic behavior of the vehicle. Moreover, there will be significant errors in estimating the vehicle's own motion parameters, which will affect the positioning effect of autonomous vehicles in digital twin scenarios. Therefore, in the future, it is necessary to improve the mechanical structure of the test bench and combine virtual autonomous driving behavior and motion trajectory in the digital twin testing platform to simulate the longitudinal/lateral acceleration of autonomous vehicles on the test bench.

(3) Lidar physical information generation technology. Lidar is an essential sensor in autonomous vehicles. Due to the complex working principle and large number of channels of LiDAR, it is difficult to generate corresponding reflection signals based on target information in virtual scenes. Therefore, there is still no target simulator for LiDAR on the market, which has led to the urgent need for hardware in the loop testing for LiDAR. In the future, we should delve deeper into the working principle of LiDAR and the basic principles of target detection, and study target simulators for LiDAR to generate physical information for detecting targets.

5. Automated driving field testing technology

Autonomous driving field testing mainly includes closed field testing and open road testing. Closed field testing can be combined with its own testing needs to build testing scenarios and detect the overall performance of the vehicle system; Open road testing can verify the performance of vehicles in real traffic environments, but in the actual testing process, the testing efficiency is low due to the uncertainty of the surrounding environment of open test roads. This chapter elaborates on the current development status of testing sites, testing standards, and key technologies related to closed site testing, open road testing, and highway testing. The organizational structure is as follows:Figure 15As shown.

Figure 15. Organizational structure of field testing technology of autonomous driving

DownLoad: Full-Size Img PowerPoint

5.1 Closed field testing (Huifei, Li Shanke)

Table 4. Construction categories of closed test fields for autonomous driving

场地名称	自动加速与制动测试场地、自动转向“S”型路线测试场地、弯道行驶测试场地、坡道停车和起步测试场地、道路入口测试场地、道路出口测试场地	人行横道减速测试场地、减速丘限速测试场地、道路限速测试场地、施工区测试场地、停车让行测试场地、减速让行测试场地、锥形交通路标测试场地	人行横道信号灯识别与响应测试场地、平面交叉口场景测试场地、进出环岛测试场地	隧道模拟测试场地、前方车辆静止测试场地、前方车辆紧急制动测试场地、前方车辆减速测试场地、主动换道测试场地、前方行人横穿测试场地
测试用途	基于不同的道路类型测试自动驾驶	基于不同的交通标志标线及其他道路交通设施测试自动驾驶	基于交叉口、环岛及信号灯等交通设施测试自动驾驶	测试其他交通参与者及定位信号遮蔽道路对自动驾驶的影响

| Show Table

DownLoad: CSV

At present, there is no complete and mature evaluation system or unified standard for testing self driving vehicles. Therefore, Liu Tianyang et al^[176]Comparative analysis of the current research and development status of intelligent connected vehicle test sites abroad, such asTable 5As shown, it provides reference for China's planning of autonomous vehicle testing sites.

Table 5. Comparative analysis of major foreign connected and automated vehicle test fields

名称	国家	占地面积/10⁴m²	建成时间	测试功能	特点分析
Mcity	美国	13.0	2015年	自动驾驶技术、V2X技术	强化试验，柔性化设计
Asta Zero	瑞典	200.0	2014年	车辆动力学、驾驶人行为、V2X技术	ADAS场景测试与模拟设备，具备完整的测试功能
Smart Road	美国		20世纪80年代	自动驾驶技术、智能交通系统、V2X技术	天气模拟系统、照明和能见度检测系统
City Circuit	英国	304.0		传统汽车、智能交通系统、智能网联汽车测试	网联汽车测试设备、跟踪定位与监控设备
Gomentum Station	美国	850.0	2014年	自动驾驶技术、V2X技术	2条真实的隧道，测试面积大
Castle Air Force Base	美国	24.3	2011年	自动驾驶技术
Willow Run	美国	136.0	2018年	自动驾驶技术、V2X技术	天然坑洞与3层立交桥

| Show Table

DownLoad: CSV

During the construction and operation of enclosed spaces, Wang Runmin and others^[181]This article elaborates on the research team's research on vehicle networking technology, autonomous driving testing and evaluation technology, autonomous driving closed testing scenarios and site construction methods, V2X based networked intelligent roadside system integration, and meteorological simulation of autonomous driving closed testing environment. Based on this, various testing systems have been constructed, including full coverage vehicle networking communication system, intelligent traffic signal control system, data acquisition system, video surveillance system, positioning system, digital map and recording and management system, etc. In addition, relevant institutions have jointly developed and integrated testing tools such as autonomous driving test data collection vehicles, pedestrian simulation testing platforms, and balloon vehicles, forming a technology chain and tool chain for autonomous driving closed field testing. At the same time, based on the research of testing methods for autonomous driving enclosed areas, the "Chang'an University Autonomous Driving Enclosed Area Testing Regulations" were formulated, which constructed a relatively complete autonomous driving enclosed area testing system from multiple aspects such as testing sites, testing equipment, and testing standards.

5.2 Open road testing (Huang Lei, Zhu Shihao, Chen Lingli)

Open road testing is the final round of autonomous driving testing, which combines fully realistic autonomous vehicles with scene information and is the final testing stage before the mass production of autonomous driving on the road. The United States, Europe, Japan and other countries have started to develop related directions earlier and have accumulated some experience. Although China started relatively late, the country's policy tilt has led to increasing investment by domestic enterprises in technological research, and related technologies have also developed rapidly.

5.2.1 自动驾驶开放道路测试相关标准、法规与政策(朱世豪、黄蕾)

With the development of intelligent connected technology, the demand for open testing roads by autonomous driving enterprises is becoming increasingly urgent. Beijing, Shanghai, Chongqing, Guangzhou, and other cities have successively issued implementation rules and management measures for open testing roads, which stipulate the basic principles, testing vehicle requirements, testing subject requirements, testing management, and accident liability determination of open testing roads. Moreover, local management measures have clearly stated that not all open road areas allow intelligent connected vehicles to be tested, and only some areas are available for testing. Before conducting open road testing, intelligent connected vehicles need to undergo certain functional tests in order to qualify for on road testing. In recent years, China's relevant standards, regulations, and policies on open road testing of intelligent connected vehicles have been as follows:Table 6As shown.

Table 6. Standards, regulations, and policies related to open road testing of connected and automated vehicles in China

时间	标准、法规与政策
2017.12	《北京市自动驾驶车辆道路测试管理实施细则(试行)》
2018.03	《上海市智能网联汽车道路测试管理办法(试行)》
2018.03	《重庆市自动驾驶道路测试管理实施细则(试行)》
2018.04	《智能网联汽车道路测试管理规范(试行)》
2018.08	《杭州市智能网联车辆道路测试管理实施细则(试行)》
2018.10	《深圳市智能网联汽车道路测试开放道路技术要求(试行)》
2018.12	广州市《关于智能网联汽车道路测试有关工作的指导意见》
2019.10	《自动驾驶车辆道路测试安全管理规范》
2021.07	《智能网联汽车道路测试与示范应用管理规范(试行)》
2022.10	《道路车辆自动驾驶系统测试场景词汇》
2023.11	《四部委关于开展智能网联汽车准入和上路通行试点工作的通知》

| Show Table

DownLoad: CSV

After years of development, there are currently 10 national level intelligent connected vehicle testing demonstration zones in China. Each demonstration zone has different focuses based on its own characteristics, but has made detailed determinations on the standards for autonomous driving on the road, accident identification, and takeover methods.

(2) Related foreign standards, regulations, and policies

The research and development of autonomous driving in foreign countries started earlier. In order to cooperate with the testing and driving of autonomous driving, countries such as the United States, Germany, and Japan have also introduced relevant road testing management measures to ensure the safety of test vehicles. In recent years, the relevant standards, regulations, and policies of open road testing for intelligent connected vehicles in European and American countries have been as follows:Table 7As shown.

Table 7. Standards, regulations, and policies related to open road testing of connected and automated vehicles in European and American countries

时间	国家	标准、法规与政策
2016.09	美国	《联邦自动驾驶汽车政策指南》
2017.05	德国	《自动驾驶法》
2016.05	日本	《自动驾驶汽车道路测试指南》
2015.07	英国	《自动驾驶汽车发展道路：道路测试指南》
2017.05	瑞典	《自动驾驶汽车公共道路测试规范》

| Show Table

DownLoad: CSV

5.2.2 Digital Twin Testing Technology for Urban Open Roads (Chen Lingli, Zhu Shihao)

In the testing of autonomous driving on urban open roads, digital twin technology collects real-world data through network communication technology, analyzes the data using big data processing technology, and feeds back the results to the physical space to solve real-world problems. The digital twin evaluation system mainly includes two parts: static twin environment construction and dynamic twin data overlay.

(1) Modeling Method for Static Digital Twin System of Urban Open Roads

① Road reconstruction based on high-precision maps and collected data

The road model is the part that requires the highest accuracy. High precision reconstruction of the road model can only be achieved based on accurate information from high-precision maps or by referencing and comparing actual collected data. in compliance withFigure 16、17As shown, using high-precision map road data, which mainly includes lane lines such as centerlines, contour lines, and markings, combined with actual collected road surrounding data, based on the idea of component assembly in building information modeling, road sections, markings, and facilities are reconstructed using a component first and then assembly approach^[182]The model built by this method has high detail, strong authenticity, and good visualization effect and operational value.

Figure 16. High-precision map

DownLoad: Full-Size Img PowerPoint

Figure 17. Completed road model

DownLoad: Full-Size Img PowerPoint

photocartography^[183]It is an efficient method of reconstructing models with rich detail information using camera shooting and visual technology. It is very effective for ancillary facilities, buildings, etc. around roads, and can achieve highly consistent results with real objects and very realistic and natural rendering effects.

For large-scale terrain and non major buildings, process modeling based on geographic information is used for automatic generation, which greatly improves efficiency while ensuring consistency with the real scene, and fully utilizes data information to further enhance the storage and rendering efficiency of scene data.

④ Integrate and construct the overall scene

Effectively and flexibly integrate all scene elements, and provide methods for editing, reconstructing, and combining static scenes to form new scenes and construct new data. Use the shader renderer and particle effects again to achieve scene rendering under different weather conditions.

(2) The Application of Digital Twins in Urban Transportation Systems

There is a strong coupling relationship between people, vehicles, roads, and environments in urban open roads. When modeling a single scene element, it is necessary to consider the impact of that element on other elements. Using this coupling relationship, a human vehicle road environment relationship architecture is established to create a vehicle road cloud integrated cloud control system. Build a digital twin city open road scene in virtual space, reflecting the dynamic changes of urban open roads in the real world in real time. In this way, digital twin technology can be used to synchronize the positioning, data communication, and interaction design of multiple elements and objects in the same road scene without affecting actual road traffic. This enables synchronous perception, real-time interaction, and decision-making behavior of different objects towards the environment or between different objects, thereby achieving digital twin simulation testing in urban systems.

Based on the construction results of the open road scene in the Jiading Intelligent Connected Vehicle Demonstration Zone in Shanghai, a roadside perception system was deployed for some scenes. To ensure the accuracy of the digital twin system, a multi-sensor fusion roadside perception system was constructed. The construction of the roadside system includes the following contents.

① Accurate perception technology based on multi-source sensor fusion on the roadside. Internal and external parameter calibration and unified timing processing are carried out by sensors such as LiDAR, cameras, and millimeter wave radar to achieve spatiotemporal synchronization and fusion of data. Then, artificial intelligence technologies such as deep learning are used to process and analyze the fused data. A perception model trained with massive prior data is input to output high-precision perception results, providing data support for digital twin evaluation.

② Multi sensor spatial calibration and relative pose stabilization technology. To ensure the accurate fusion of different sensor data, it is necessary to ensure the consistency of external parameters between sensors, and it will not be easily changed due to external factors in the operation life cycle. The relative position and attitude variables are recommended to be less than 0.1%, otherwise, joint calibration will be required again.

③ Multi sensor time synchronization technology. Time synchronization scheme is required to conduct unified time service for different sensors to ensure that the sensing time of each sensor data is consistent, so as to fuse different sensor data at the same time.

④ End to end low latency technology. The results of roadside fusion perception need to be incorporated into the urban digital twin system, and it is required that the time for the perception results to occur from the physical world to be transformed into semantic results and transmitted to vehicles for decision-making should not exceed the effective time of the data, otherwise the auxiliary role of the entire system will be ineffective.

5.3 高速公路测试系统构建技术(景峻、常玉涛、陈西广、纪艺)

5.3.1 Testing System Architecture Patterns and Development Methods (Jing Jun, Chang Yutao)

Currently, various testing systems are undergoing a comprehensive transformation towards networking, digitization, and intelligence. The spatiotemporal complexity of the systems is rapidly increasing, which puts higher demands on their scalability, robustness, and other aspects. Especially for scenarios such as highway autonomous driving sites and real highways that require higher real-time performance, the construction of an accelerated testing cloud platform requires a more systematic system architecture development method.

At present, the main platform systems in the industry generally adopt an end-to-end cloud architecture model, which brings advantages such as flexibility and scalability, but also further increases the complexity and difficulty of system construction. Coupled with the fast speed of highways and other characteristics, online testing systems are required to be high-speed, efficient, and high fidelity, which brings higher requirements to the system construction mode, development methods, etc. In addition, relevant research and practical work have also been carried out in the meso and micro level modes, such as the four-dimensional real-time control system mode for the time sensitive auto drive system in the NHTSA autopilot test framework^[184]To cope with the construction process of complex systems of the above types, a series of mature system architecture methods have been summarized in the field of information technology. The mainstream architecture method is the Zachman framework^[185]The mainstream architecture framework method developed based on the enterprise architecture plan^[186]Federal Enterprise Architecture Framework^[187]Wait; The second is the Technical Architecture Framework for Information Management (TAFIM) developed by the US Department of Defense based on ISO/IEC 14252, and the Open Group Architecture Framework (TOGAF) developed based on TAFIM^[188]The US Department of Defense Technical Reference Model developed based on TOGAF^[189]Architecture Framework with the US Department of Defense^[190].

5.3.2 测试用例构建关键技术(常玉涛、纪艺)

Considering that mileage based testing methods have drawbacks such as long testing cycles, low efficiency, and high costs, scenario based field testing has become an indispensable part of autonomous driving twin testing. Therefore, conducting research on key technologies for constructing test cases for highway scenarios is conducive to promoting the further development of autonomous driving testing.

Wang Xiaoyi and others^[194]Analyzed the impact of the 7-layer model architecture for expected functional safety on the perception system, selected appropriate experimental levels and factors for different systems based on the degree of impact, and designed an orthogonal design experimental scheme to improve testing efficiency while ensuring testing objectives; Xia et al^[195]Using Analytic Hierarchy Process to select important factors in the testing scenario and constructing virtual test cases through combination testing method; Zofka et al^[196]Propose a method for constructing high-risk test cases using real traffic data in a virtual environment by collecting data from multiple sensors; An Zeping and others^[197]Taking into account the dual factors of transportation demand and road environment attributes, a use case construction method for intelligent vehicle road collaboration system on highways is proposed. A fully functional, scenario rich, and element rich vehicle road collaboration application use case library suitable for different highway needs is constructed, providing reference for demonstration applications of highway vehicle road collaboration; Zhou Wenshuai and others^[112]Propose a test case design and description method based on actual traffic scenarios, verify by analyzing the entry of highway vehicles into the scenario, implement test case design using HighD dataset, and generate test cases using Monte Carlo method.

At present, there is no clear and unified definition of concepts such as test scenarios and test cases in the field of autonomous driving. Moreover, existing research mainly focuses on the analysis and construction of autonomous driving test scenarios, and there is relatively little research on the design and generation of test cases in highway scenarios.

(2) Test case structure elements and construction

Based on the research on the construction method of test cases, taking the highway testing system as the starting point, relevant concepts and relationships are proposed, such asFigure 18As shown.

Figure 18. Related concepts and relationships

DownLoad: Full-Size Img PowerPoint

The related concepts are mainly divided into two domains, namely the Dynamic Driving Task (DDT) functional domain and the DDT testing domain. The former is mainly divided into DDT tasks, DDT functions, use cases, and scenarios; The latter is mainly divided into testing tasks, testing cases, and testing scenarios. A test task is a description of the coupling relationship and execution sequence of multiple test functions, where one test task corresponds to one or more test functions; A use case refers to a description of how a system responds to external requests designed for testing functionality. Requirements are obtained through user usage scenarios, mainly including use case descriptions, test objects and purposes, and factors such as scenario and testing related indicators; Scenarios refer to specific cases that support test cases, mainly including factors such as scenario names, scenario descriptions, and scenario diagrams.

5.3.3 Testing Procedures (Chang Yutao, Chen Xiguang)

At present, the testing regulations for autonomous driving on highways are guided by the "Intelligent Connected Vehicle Road Testing Management Standards (Trial)" jointly issued by the Ministry of Industry and Information Technology, the Ministry of Public Security, and the Ministry of Transport in April 2018. The cumulative number of open intelligent connected vehicle testing roads exceeds 2000 kilometers, and the total road testing mileage exceeds 2 million kilometers; On August 3, 2018, the National Automotive Standardization Technical Committee and others released the "Intelligent Connected Vehicle Autonomous Driving Function Test Regulations (Trial)"^[198]Provided corresponding testing scenarios, testing procedures, and passing conditions for the automatic driving function testing of intelligent connected vehicles; On March 16, 2021, Shanghai Society of Automotive Engineering drafted the group standard Intelligent Transportation System - Expressway auto drive system - Performance Requirements and Road Test Procedures, which stipulates the road test requirements for expressway auto drive system for M1 and N1 vehicles with expressway automatic driving function; On July 27, 2021, the Ministry of Industry and Information Technology, the Ministry of Public Security, and the Ministry of Transport jointly issued the "Management Specification for Intelligent Connected Vehicle Road Testing and Demonstration Application (Trial)"^[199]Expand the scope of autonomous vehicle road testing and demonstration applications to highways; In December 2021, the group standard "Technical Requirements and Test Methods for Expressway auto drive system" proposed by the China Intelligent Connected Vehicle Industry Innovation Alliance has been officially included in the standard development plan of the Society of Automotive Engineers of China.

The testing regulations for autonomous driving in highway environments are still in the research and exploration stage, and there are no industry standards or national standards yet. The existing testing procedures do not specify the order and priority of testing items. In the subsequent specification development, priority should be given to conducting scenario testing related to braking, single road testing, and simple testing targets, in order to provide better technical guidance for autonomous driving enterprise testing.

5.4 Summary and Outlook

In terms of constructing a highway testing system, existing test case construction techniques can be comprehensively analyzed, combined with the conceptual structure and constituent elements of test cases. Based on the characteristics of high-speed vehicle operation and difficult construction of dangerous scenarios on highways, a virtual real combination of highway scene autonomous driving simulation test case construction method can be developed based on key technologies of digital twin and vehicle road coordination, providing more convenient and efficient technical support for highway autonomous driving testing.

6. Intelligent evaluation method for autonomous driving

Figure 19. Organizational structure of intelligence evaluation method of autonomous driving

DownLoad: Full-Size Img PowerPoint

6.1 Concept of Autonomous Driving Intelligence (Wu Xinzheng, Chen Junyi, Shen Yong)

Intelligent automotive technology is an effective way to reduce traffic accidents, alleviate traffic congestion, and reduce environmental pollution. Generally speaking, autonomous driving technology can be seen as a subfield of artificial intelligence^[200]Artificial intelligence refers to the intelligence exhibited by machines that is equivalent or similar to human intelligence when performing the same tasks as humans^[201]Furthermore, autonomous vehicles can also be regarded as intelligent agents such as Wooldridge^[202]In 1995, it was proposed that intelligent agents should possess the following attributes.

(2) Sociality: Intelligent agents can interact with other intelligent agents (or humans) through specific languages.

(3) Reactivity: Intelligent agents can perceive their surrounding environment and respond promptly to changes that occur within it.

(4) Initiative: Intelligent agents not only respond to the environment, but also actively exhibit goal-directed behavior.

As a type of intelligent agent possessing the above attributes, autonomous vehicles need to undergo sufficient testing and evaluation during the development and design process to verify that the vehicle is intelligent enough to replace humans in completing various driving tasks. Firstly, it is necessary to clarify the concept and connotation of vehicle intelligence. At present, based on the definition of artificial intelligence and the attributes of intelligent agents mentioned above, combined with the specific operating environment and driving tasks of autonomous vehicles, many scholars have elaborated on the concept of autonomous driving intelligence from two aspects: vehicle autonomous driving capability and performance.

6.1.1 Vehicle autonomous driving capability

Vehicle autonomous driving capability refers to the ability of a vehicle to replace the driver in completing driving tasks during the driving process, corresponding to the attributes of intelligent agents^[202]Autonomy, reactivity, and proactivity. Based on the division of specific driving tasks for drivers, Li et al^[200]It is believed that the intelligence of autonomous vehicles refers to their ability to autonomously complete perception, recognition, decision-making, and planning, and take action; Huang et al^[203]It is believed that intelligence refers to the comprehensive perception, cognition, analysis, communication, planning, decision-making, and execution abilities possessed by unmanned systems in order to achieve their assigned goals; Nagai et al^[204]It is believed that the intelligence of autonomous vehicles is reflected in their ability to adaptively determine the level of assisted driving based on environmental risks and achieve control performance that takes into account the vehicle's dynamic characteristics; Payalan et al^[205]This article summarizes the capabilities and challenges faced by ADAS systems, and proposes a definition of intelligence for next-generation vehicles. Vehicle intelligence refers to the ability of vehicles to be responsible for their own safety, comfort, entertainment, and energy consumption based on specific task modules and services such as sensors, vehicle ad hoc networks, and big data infrastructure.

Unlike the aforementioned studies, SAE J3016 in the United States and China's "Classification of Automotive Driving Automation" (GB/T 40429-2021) describe the intelligence of autonomous driving based on the vehicle's ability to perform complete dynamic driving tasks from the perspective of overall driving tasks, and divide the level of vehicle driving automation into 6 levels ranging from 0 to 5. The Chinese National Standardization Administration divides the level of vehicle driving automation into six elements: the completeness and continuity of dynamic driving tasks, the continuity of longitudinal or lateral motion control, the simultaneity of longitudinal/lateral control, the ability to automatically execute minimum risk strategies, and the existence of design operating ranges.

6.1.2 Vehicle autonomous driving performance

The autonomous driving performance of a vehicle refers to its performance in completing driving tasks, corresponding to the social, proactive, and autonomous attributes of intelligent agents. The various internal capabilities of autonomous vehicles, such as perception, cognition, decision planning, and control execution, will ultimately be unified at the vehicle level to support the completion of driving tasks. Therefore, the meaning of autonomous driving intelligence can be fully and comprehensively understood through the external performance of the vehicle. This article divides the performance of autonomous vehicles into basic performance that reflects intelligence and higher-order performance.

The basic performance that reflects intelligence is extended from traditional vehicle performance. For traditional vehicles, safety, comfort, and economy are three important dimensions for measuring vehicle performance. Among them, safety includes the passive safety performance, braking performance, and handling stability of the vehicle; Comfort includes the smoothness, vibration, and noise performance of the vehicle; Economy mainly refers to the fuel economy of vehicles. For autonomous vehicles, as the subject of driving tasks has changed from humans to intelligent systems, the assessment of the vehicle has shifted from the driver's perspective (i.e. whether the vehicle is driving well) to the passenger's perspective (i.e. whether the vehicle is driving well). Therefore, safety, comfort, and economy have been given new meanings to reflect the intelligence of the vehicle. In terms of safety, intelligence is mainly reflected in active safety performance such as vehicle collision avoidance and hazard mitigation capabilities^[206]In terms of comfort, due to the superior motion control and workshop coordination capabilities of autonomous vehicles, which differ greatly from natural driving behavior, they often cause discomfort to passengers^[207]Therefore, intelligence is also reflected in the consistency between vehicle driving behavior and passenger expectations, as well as the ability to prevent passengers from feeling out of control^[208]In addition, corresponding to the economic efficiency of traditional vehicles, in addition to energy efficiency, the time efficiency of vehicles efficiently completing predetermined driving tasks within a specified time is also an important manifestation of the intelligence of autonomous driving. Therefore, energy efficiency and time efficiency can be collectively referred to as the efficiency of autonomous driving vehicles. At present, many scholars have conducted research on autonomous vehicles, focusing on safety, comfort, and efficiency,Table 8Typical literature on the basic performance of autonomous vehicles has been summarized from two perspectives: development design and testing evaluation.

Table 8. Research on safety, comfort and efficiency of autonomous driving vehicles

角度	安全性	舒适性	高效性	相关文献
开发设计	$ \checkmark$			[211]、[212]
		$ \checkmark$		[213]
			$ \checkmark$	[214]
	$ \checkmark$	$ \checkmark$		[215]
	$ \checkmark$		$ \checkmark$	[216]、[217]
测试评价	$ \checkmark$			[218]
		$ \checkmark$		[219]
			$ \checkmark$	[220]
	$ \checkmark$	$ \checkmark$		[221]
	$ \checkmark$		$ \checkmark$	[222]
	$ \checkmark$	$ \checkmark$	$ \checkmark$	[200]

| Show Table

DownLoad: CSV

As intelligent agents, autonomous vehicles also have high-order performance. Combining Wooldridge and others^[202]The attributes of the proposed intelligent agents have been proposed by many scholars, and the intelligence of autonomous driving is also reflected in minimal human intervention as much as possible^[203]Excellent human-computer interaction and learning ability, as well as good interaction quality with other traffic participants^[209]Meng et al^[210]A comprehensive evaluation framework for autonomous driving intelligence is proposed, taking into account various dimensions of intelligence, including driving autonomy, social cooperation, and learning evolution. Among them, driving autonomy refers to the ability of a vehicle to autonomously and high-quality complete driving tasks without human intervention, which is a manifestation of the intelligence of the vehicle itself; Social cooperation refers to the ability of vehicles to predict interaction trends and implement interaction behaviors, used to characterize the social cooperation ability in vehicle intelligence^[223]Evolutionary learning refers to the ability of autonomous vehicles to repeat and generalize over time, reflecting the intelligence of autonomous vehicles in the temporal dimension.Figure 20Summarized the conceptual framework of autonomous driving intelligence considering both vehicle autonomous driving capability and performance.

Figure 20. Conceptual framework of autonomous driving intelligence

DownLoad: Full-Size Img PowerPoint

6.2 Quantification and Evaluation of Scene Complexity (Ma Yining, Chen Junyi, Xiong Lu)

Vehicle autonomous driving capability and performance are two important dimensions for discussing the intelligence of autonomous driving. However, both dimensions are based on the perspective of the vehicle itself, and in order to comprehensively evaluate the intelligence of autonomous driving, the complexity of the external environment in which the vehicle is located needs to be considered. In the research on the test and evaluation of automatic driving, the test scenario determines most of the driving tasks. Therefore, quantifying the complexity of the scenario can be one of the important indicators for the intelligence evaluation of the auto drive system. High complexity testing scenarios help expose defects in the performance of the tested system^[224]When a tested system performs exceptionally well in highly complex testing scenarios, it proves that the system has high intelligence^[225]LikeFigure 21As shown, when quantifying and evaluating the intelligence of autonomous driving, the evaluation of vehicle intelligence can be achieved through the assessment of input (scene complexity) and output (performance).

Figure 21. Vehicle intelligence evaluation system based on scenario complexity

DownLoad: Full-Size Img PowerPoint

This section provides an overview of the definition and quantification methods of scene complexity. The research on quantitative methods can be used to design controlled scenarios of different complexities to improve testing efficiency, as well as to evaluate uncontrolled testing scenarios to support the evaluation of the intelligence of the tested system.

6.2.1 Definition of Scene Complexity

In the field of scientific research, the definition of complexity varies depending on the disciplinary background. Through literature research and analysis, it was found that scholars have mainly defined or explained the concept of complexity from the following two perspectives: (1) describing complexity from the internal composition of the system. Like Xing et al^[226-227]After reviewing literature in several research fields (general concepts, information complexity, cognitive complexity, and display complexity), it is concluded that the commonalities in defining complexity in these fields mainly focus on three factors: the number and size of basic elements in the system, the types of elements, and the relationships between elements. (2) Observe complexity from outside the system. In the study of information theory, Kolmogorov complexity is considered the shortest length described in a certain language^[228]In the field of topology research, Crutchfield et al^[229]On the basis of Kolmogorov, the definition of complexity has been extended, stating that complexity can be defined as the ability of a system's model to reproduce the extreme values of observed data based on statistics within a specified tolerance range.

In the field of intelligent transportation research, Sussma^[230]Describing a complex transportation system from an internal perspective, it is composed of a set of related units (subsystems) whose degree and nature of relationships between these units (subsystems) are not fully understood, and their overall behavior is difficult to predict. In the field of autonomous driving, the "subsystems" mentioned in this study can be understood as different traffic elements in the scene, that is, the difficulty of predicting traffic elements can be used to describe the complexity of the transportation system; On this basis, Ma et al^[231]The complexity of autonomous vehicles in relevant environments and driving tasks is defined as the total uncertainty of scene information in the perception, cognition, and prediction processes; From an external observation perspective, Zhang et al^[232]Describe the complexity of a scene composed of dynamic traffic participants based on the observed drivable area of the traffic participants in the scene.

6.2.2 Quantitative method for scene complexity

In the study of quantifying complexity from the perspective of internal composition, some scholars have attempted to quantify the complexity of a scene by measuring the influence of internal elements. The most commonly used method is ontology combined with Analytic Hierarchy Process (AHP). This type of research first classifies different elements in the scene based on ontological methods, and then quantifies the impact coefficients of each element based on AHP. The greater the impact, the greater the complexity.

In the study of quantifying complexity from the external representation of observation scenes, Boelhouwer et al^[233]Obtain the complexity of infrastructure and traffic factors in the scene through online surveys, and calculate the overall complexity of the scene; Liu et al^[234]Firstly, the complexity of the scene is divided into three aspects: driving scene attributes, surrounding vehicle information, and prior environmental knowledge. Then, the random forest algorithm is used to classify the complexity of the observed scene information. Generally speaking, dynamic complexity is more complex than static complexity, Zhang et al^[232]A method for calculating the drivable area of dynamic traffic participants has been proposed, combined with information entropy^[235]The concept quantifies the dynamic complexity in the scene; Liu et al^[234]Classify the complexity of driving scenarios using environmental information; Mueller and others^[236]When evaluating the impact of driving scenarios on driver reactions, the complexity of driving scenarios is quantified as the difficulty of the driving tasks performed by the driver; Wang et al^[237]A traffic perception data classification paradigm based on quantifying the complexity of each road segment has been proposed, which includes the complexity of traffic elements and road semantics, obtained through subjective quantitative evaluation and prediction based on Support Vector Regression (SVR), respectively.

By comparing the above studies, it is found that most of the methods for quantifying scene complexity originate from subjective evaluation, especially those based on describing complexity through internal composition. All quantification methods contain subjective evaluation. In the study of defining complexity from outside the scene, Zhang et al^[232]We have adopted a purely objective approach to quantify complexity, therefore, it can be analyzed that the perspective of complexity definition is somewhat related to the subjective/objective nature of the quantification method. In addition, there are certain differences in the perspective of obtaining element data. The mainstream research method is to quantify scene complexity by obtaining data from a third-party perspective outside the vehicle, while research based on obtaining data from the perspective of the vehicle itself is relatively rare. The above results are summarized as followsTable 9As shown: ● is the weather environment; ▲ for road facilities; ★ For the transportation environment; For dynamic traffic participants; ◆ It is a self driving attribute.

Table 9. Quantitative methods of complexity in autonomous driving testing scenarios

复杂度定义视角	量化方法	主/客观	量化要素	数据获取视角	文献
基于场景内部构成描述并量化复杂度	Multi-Factor Analysis、Survey	主观	● ★ ◆	车外	[231]
	AHP	主观	■ ◆	车外	[195]
	Ontology、AHP	主客观结合	◆	车外、自车	[238]
	Ontology、AHP	主客观结合	▲★	车外、自车	[238]
基于场景外部观测定义并量化复杂度	SVR	主客观结合	● ■ ◆	车外	[235]
	Survey	主观	● ■ ◆	车外、自车	[234]
	Random Forest	主客观结合	● ◆	自车	[236]
	NASA-TLX Surveys	主客观结合	● ★ ■	自车	[237]
	Drivable Area、Entropy	客观	● ▲ ★ ◆	车外	[232]

| Show Table

DownLoad: CSV

6.3 Multi dimensional comprehensive evaluation system for autonomous driving intelligence (Meng Haolan, Xing Xingyu, Chen Junyi, Yu Zhuoping)

Traditional vehicle evaluation indicators mainly focus on the dynamic performance of the vehicle itself, which is still applicable to autonomous vehicles. A new systematic evaluation system needs to be established for the intelligence of autonomous vehicles to meet the evaluation requirements during the development and certification process. Both academia and industry have explored and researched this issue, attempting to elaborate on the intelligence of autonomous vehicles and establish corresponding evaluation index systems. At present, research on the intelligence evaluation of autonomous vehicles at home and abroad can be mainly divided into two categories: one is the intelligent car competition led by third-party organizations, which designs an evaluation system and scoring mechanism based on the competition objectives; The other type is theoretical research conducted by various research institutions, which analyzes the comprehensive performance of autonomous vehicles based on different evaluation theories and methods.

The early exploration of evaluating vehicle intelligence mainly focused on unmanned vehicle competitions, such as the unmanned vehicle challenge led by the US Defense Advanced Research Projects Agency and the China Intelligent Vehicle Future Challenge organized by the National Natural Science Foundation of China. Competitions are generally held on predetermined roads or areas, using artificially constructed simulated traffic environments and specifying the driving tasks that vehicles need to complete to assess the intelligence of unmanned vehicles. Methods for evaluating the intelligence of autonomous vehicles in various competitions, such asTable 10As shown.

Table 10. Comparison of intelligence evaluation methods of autonomous driving vehicles in different competitions

比赛/项目	国家	首届时间	考察功能或性能	指标
Percept OR	美国	2001年	自主行为能力	自主行驶里程、任务耗时、行驶速度、人工干预次数、任务失败次数
Grand Challenge	美国	2004年	自主行为能力	行驶里程、完成时间
Urban Challenge	美国	2007年	自主行为能力和部分交互能力	任务完成时间、完成质量、违反交通规则、危险行为
智能车未来挑战赛	中国	2009年	安全性、舒适性、敏捷性、智能性	完成任务总时间、任务完成质量、人工干预次数
自动驾驶车辆挑战赛	中国	2018年	智能性、安全性、人机交互、能效	任务完成时间、任务完成度、人工干预次数

| Show Table

DownLoad: CSV

The core of various autonomous vehicle competitions is the tasks set during the competition, and the evaluation of autonomous driving is based on the various indicators of the vehicles completing the tasks. Overall, the evaluation of various competitions revolves around the autonomous behavior ability, interaction ability, and learning ability of autonomous vehicles, reflecting the general requirements of the industry for autonomous vehicles.

In order to more accurately describe the performance of intelligence, some research institutions have conducted studies on evaluation theories and methods. Early testing and evaluation of autonomous vehicles were mainly based on a single function and scenario. With the advancement of technology, the evaluation of autonomous vehicles has gradually become more complex and integrated. In the process of evaluating autonomous vehicles, many studies have absorbed the evaluation experience of ALFUS unmanned system automatic level framework for generalized unmanned systems. Inspired by this, various evaluation methods have emerged, mainly divided into two categories: mileage based evaluation and scenario based evaluation. The overall evaluation index system includes vehicle control behavior, driving behavior, traffic behavior, etc., such asTable 11As shown.

Table 11. Comparison of intelligence evaluation methods of autonomous driving vehicles in different studies

来源	机构	评价目标/准则/指标	类型
Lowrie等^[239]	马里兰大学	总行驶里程、人工控制行为、人工干涉率等	基于里程
Pomerleau等^[240]	卡内基梅隆大学		基于里程
Broggi等^[241]	帕尔马大学		基于里程
Maurer等^[242]	慕尼黑国防大学		基于里程
	美国国家标准与技术研究院^[243]	人工干预度、环境复杂度、任务复杂度	基于里程
Wang等^[244]	中国科学院沈阳自动化研究所	感知技术、导航技术、人机交互、信息通讯技术、路径规划技术、多平台协同技术、运动控制技术、任务规划、学习适应	基于场景
Sun等^[245]	北京理工大学	车辆控制行为、基本和高级行车行为、基本和高级交通行为	基于场景
蒙昊蓝等	同济大学	行驶自治性、社会合作性、学习进化性	基于场景

| Show Table

DownLoad: CSV

The mileage based evaluation is similar to various unmanned vehicle competitions, still focusing on the overall completion of the evaluated object; Scenario based evaluation designs evaluation objectives based on the characteristics of specific scenarios, with stronger targeting. In order to overcome the shortcomings of strong scene dependence and weak extrapolation of evaluation methods based on a single scenario, the research team of Tongji University first proposed a subjective and objective evaluation framework for evaluating the driving intelligence of autonomous vehicles from the dimensions of the vehicle itself, space, and time, combining the concept of intelligent agents. This framework covers three characteristics that autonomous vehicles as intelligent agents should possess: driving autonomy, social cooperation, and learning evolution,Figure 22From the dimensions of individual, space, and time, it encompasses various evaluation objectives involved in existing evaluation research, making the evaluation system no longer dependent on specific scenarios, as shown on the coordinate axis in the figuret₁~t₅They represent the observation time points of the vehicle during the process of learning and evolution over time.

Figure 22. Driving autonomy, social cooperation and learning evolution of autonomous driving vehicles

DownLoad: Full-Size Img PowerPoint

Further selection of evaluation indicators is needed for different evaluation dimensions. For the autonomous driving of autonomous vehicles, the evaluation of driving quality can be composed of several indicators such as safety, comfort, and efficiency. According to the degree of quantification, evaluation indicators can be divided into quantitative indicators and non quantitative indicators (qualitative indicators), among which the numerical values of quantitative indicators are generally calculated and analyzed through test results^[246]The data of non quantitative indicators mainly comes from subjective evaluations of experts. The results of non quantitative indicators directly reflect human knowledge and experience, with higher credibility but lower quantification.

The evaluation method refers to the calculation of evaluation indicators through aggregation models to obtain comprehensive evaluation results. The aggregation model can be obtained through subjective, objective, or combined methods to reflect the degree of influence of each indicator on the final evaluation results or the degree of importance that the evaluation subject attaches to different indicators. The commonly used evaluation methods include fuzzy comprehensive evaluation method, grey correlation degree method, neural network method, weighted arithmetic mean method, TOPSIS method, etc.

6.4 Social Cooperation Evaluation Method (Meng Haolan, Chen Junyi, Yu Zhuoping)

With the continuous upgrading and popularization of technology and products, there will be more and more autonomous vehicles and human driven vehicles mixed on the road in the future, which will have a huge impact on the road traffic system. In a traffic environment entirely composed of humans, although different drivers have different driving styles, their perception mechanism of the traffic environment, understanding ability of the traffic scene, risk discrimination mechanism, collaborative and game decision-making mechanism, and vehicle control ability are homogeneous and similar. Therefore, under the premise of jointly complying with traffic regulations, the road traffic system composed of various traffic participants is in a relatively balanced state. However, there are significant differences between autonomous vehicles and drivers in the above aspects, which will make autonomous vehicles incompatible with current road traffic participants and create cooperation difficulties in local traffic systems, which can be referred to as social cooperation problems.

Social cooperation refers to the ability of autonomous vehicles to maximize the overall benefits of their own driving and surrounding traffic while satisfying traffic rules and avoiding collisions during the completion of driving tasks. This will be one of the important capabilities that autonomous vehicles should possess when driving on open roads, and the lack of this capability will make it difficult for autonomous vehicles to integrate into the natural transportation system. Especially in driving scenarios with weak traffic rule constraints, the level of this ability determines the performance of autonomous vehicles integrating into natural traffic. The establishment of a social cooperation evaluation method for autonomous vehicles has received increasing attention from the industry, and important components such as testing scenarios, evaluation systems, and background vehicle models in testing have been studied.

The key to researching methods for evaluating social cooperation is to analyze the attributes of social cooperation, in order to guide the selection of evaluation scenarios and the construction of evaluation systems. In the field of transportation, social cooperation mainly refers to designing benefit functions for traffic flow, such as efficiency, safety accident rate, traffic capacity, and the overall traffic status of large-scale transportation systems. In the automotive industry, social cooperation is mainly reflected in vehicles implementing profit maximizing driving strategies by predicting the behavior of other vehicles. Scholars start from scenarios such as intersections, roundabouts, and ramps, and study vehicle interaction decision-making by establishing interest functions, game models, cost functions, and other methods.

In terms of scenarios, there are common points of conflicting driving trajectories between vehicles in scenarios such as intersections, roundabouts, and ramps in various studies. In order to obtain scenarios suitable for social cooperation evaluation, some scholars take the ramp merging scenario as an example, based on natural driving data analysis, and define three indicators of vehicle interaction process, including initial state, driving action, and interaction performance（Figure 23）Cluster analysis of typical logical scenarios suggests that the less severe the trajectory conflicts and the larger the interactive space, the better the diversity of vehicle behavior^[247].

Figure 23. On-ramp scenario

DownLoad: Full-Size Img PowerPoint

In terms of evaluation system, various studies focus on the behavior of vehicles during interactive driving and its impact on a certain range of transportation systems. By designing process cost, comfort cost, safety cost, and fuel consumption cost functions, Wei et al^[248]Evaluated the social cooperative behavior of autonomous vehicles and selected the best autonomous driving strategies; Letter, etc^[249]Evaluate the driving strategy during lane merging from the perspective of efficiency through total travel time, average speed, and unit distance travel time; Ntousakis et al^[250]Compare the longitudinal trajectory planning results from the comfort dimension through speed, acceleration, and acceleration change rate; Parmar et al^[251]The control strategy for lane merging scenarios was evaluated from the perspectives of efficiency and energy consumption, utilizing fuel consumption and total travel time. Based on different evaluation dimensions, the research team of Tongji University summarized the evaluation of social cooperation into two evaluation perspectives: benefiting others and benefiting oneself. Through three evaluation perspectives of global, self owned, and other vehicles, expert evaluation was conducted, and an evaluation system covering efficiency, energy consumption, and comfort dimensions was designed on this basis. A subjective objective mapping evaluation model was constructed using neural networks^[252].

In addition to scenarios and evaluation systems, the background car model is an important testing tool for implementing evaluation methods into the evaluation process. Traditional testing targets move along predetermined trajectories, with high repeatability and weak autonomy; The car following model represented by the IDM model has a certain degree of longitudinal interaction ability, and vehicles in the adjacent lane do not affect it; A driving behavior model based on machine learning algorithms can achieve multiple driving styles or behavioral motivations through human like decision-making. The characteristics of the three types of models are as follows:Table 12As shown.

Table 12. Classification and advantage analysis of background vehicle models

控制方式	既定轨迹	跟驰模型	驾驶行为模型
可解释性	参数化、可解释	参数化、可解释	无解析表达
状态空间复杂度	极低	低、确定性	高、概率性
交互能力	无	纵向	纵向、侧向
优势	完全可重复性高	可解释性较强	类人决策实现多种驾驶风格或动机

| Show Table

DownLoad: CSV

Unlike traditional autonomous vehicle evaluation, in social cooperative evaluation, the opponent vehicle needs to have both vertical and lateral autonomous interaction capabilities, as well as interpretability and repeatability. The model that can be applied to standardized evaluation needs further research.

In terms of relevant standards, the industry has already released testing and evaluation group standards for the harmony of intelligent connected vehicle transportation. The Shanghai Transportation Industry Association has released a method for testing and evaluating the traffic harmony of intelligent connected vehicle ramp scenarios. The content takes the ramp merging scenario with acceleration lanes on highways as an example, and puts forward requirements for the testing and evaluation process, testing scenarios, and testing evaluation.

6.5 Summary and Outlook

In summary, with the development of autonomous driving technology, research on the intelligence evaluation of autonomous vehicles has made certain progress both domestically and internationally. Although there are differences in the evaluation methods due to different understandings of the connotation of intelligence in various studies, the commonality lies in explaining the intelligence of autonomous vehicles from external behavioral manifestations. By setting evaluation objectives, specifying evaluation indicators and their weights, and selecting appropriate aggregation models, the performance of autonomous driving vehicles is evaluated. The development trend of intelligent evaluation of future autonomous vehicles can be discussed from the following three aspects.

(1) In the research of scene complexity, scholars have defined and quantitatively evaluated the complexity based on different perspectives, and the quantitative results can be used as one of the indicators to evaluate the intelligence of the auto drive system. In terms of defining the complexity of autonomous driving testing scenarios, current research mainly focuses on defining the concept of complexity around the elements and attributes within the scenario. The definition of test scenario complexity needs to be further clarified in the future. It can not only derive more diverse definitions from external observation perspectives, but also adopt a comprehensive definition of scenario complexity by combining internal and external perspectives. In terms of quantification methods, the relationship between the complexity of multiple types of elements in a scene is often coupled and mutually influential. However, most current research simplifies the relationship between quantified elements as addition, which may lead to inaccurate quantification results. Therefore, further exploration and verification are needed in the calculation relationship of the complexity between elements.

(2) Through in-depth analysis, it is found that most of the evaluation research on intelligence mainly involves some manifestations of intelligence, such as autonomous behavior ability, which mainly reflects task completion; On the other hand, how to make scenario based evaluation methods independent of specific scenarios and enhance their extrapolation is a trend. In future research, following a certain theoretical basis to summarize the evaluation system, starting from higher dimensional intelligent behavior performance, covering all the characteristics that autonomous vehicles should have, needs to be further explored.

(3) Overall, social cooperation is a specific representation of the spatial dimension in the subjective and objective comprehensive evaluation theory of the intelligence of autonomous vehicles. Its advantages and disadvantages will directly affect the usability and social acceptance of autonomous vehicles, determining whether they can transition from the human-machine hybrid traffic stage to large-scale autonomous driving traffic. The proposal of this attribute provides a clear design goal for the integration of autonomous vehicles into the existing traffic environment, and its evaluation method still needs further practice.

7. Automated Driving Test Evaluation Tool Chain and System Method

This section mainly summarizes the tool chain for autonomous driving testing and evaluation, introduces the current status of the autonomous driving testing and evaluation standard system, mainly including the testing and evaluation tool chain, autonomous driving testing and evaluation system, and autonomous driving testing standards.

7.1 Testing and Evaluation Tool Chain (Yang Lan, Li Shijie)

The existing tool chain for autonomous driving testing and evaluation generally focuses on virtual testing tools, mainly simulation platforms provided by OEM, Tier and other enterprises, which can achieve system integration of tools such as VIL/HIL/MIL, such asTable 13As shown.

Table 13. Tool chain integration schemes for autonomous driving test in various countries

序号	名称	国家	合作单位	特征
1	Test Bed Lower Saxony for Automated and Connected Mobility	德国	德国航天中心、德国交通部、德国西门子、大众、沃尔沃	覆盖城乡、高速公路等多种道路，支持硬件、软件、驾驶人与车辆在环测试工具的自动驾驶测试系统
2	VTD	美国	德国VIRES	提供复杂交通场景与物理驱动模拟，包含软硬件、驾驶人与车辆在环的测试环境
3	ASAM OpenX	德国	ASAM	提供驾驶和交通模拟器界面，包含用于验证自动驾驶功能的静态和动态内容
4	Drive Sim/Drive Constellation	美国	英伟达	围绕车端、桌面端、云端构建了GPU硬件统一架构和CUDA软件架构，为自动驾驶应用领域提供支持
5	Apollo	中国	百度	包括完整的软硬件和服务系统，包括车辆平台、硬件平台、云端数据服务等；场景库包括了法规标准场景、危险工况场景和能力评估场景共200种
6	华为八爪鱼(HUAWEI Octopus)	中国	华为	在数据采集、数据挖掘、数据标注、算法训练、仿真平台等方面提供完整解决方案，包含大量数据集和场景库
7	TAD Sim	中国	腾讯	通过工业级车辆动力学模型、虚实结合等技术打造虚实结合、线上线下一体的自动驾驶仿真平台

| Show Table

DownLoad: CSV

The Test Bed Lower Saxony for Automated and Connected Mobility project is jointly built by the German Aerospace Center, the German Ministry of Transport, Siemens, Volkswagen, and Volvo for automated and networked testing of autonomous vehicles during development and validation processes^[253]It includes multiple types of roads, forming an autonomous driving testing tool chain that includes hardware, software, driver and vehicle in the loop testing tools. The various tools in the toolchain are called in an integrated manner, while combining virtual and real testing environments in the best way to conduct digital twin testing, achieving efficient, flexible, and open integration testing.

VTD can be applied to real-time complex traffic scene generation (including radar, infrared, camera and other sensor imaging) for active safety of automobiles and semi physical testing of unmanned vehicles, as well as traffic scene display in the development of automobile driving simulators. At the same time, the system is also applied to real-time simulation of industrial and other visual systems, including road network modeling, weather and environment simulation, traffic scene modeling, traffic sound effect simulation, scene simulation management, and high-precision real-time image rendering.

ASAM OpenX is used to solve the problems caused by inconsistent data and interface formats in various stages of simulation testing^[73]Describe the testing scenario, driving actions, and judgment conditions. OpenDRIVE corresponds to static map scenarios, while OpenSCENARIO corresponds to dynamic behavior scenarios. Input this information into the simulation environment to build the simulation scenario.

NVIDIA has built a unified GPU hardware architecture and CUDA software architecture around the car, desktop, and cloud, and released the autonomous driving platform NVIDIA DRIVE^[254]The platform is equipped with self-developed software architectures Drive AV and Drive IX. In the field of simulation, Nvidia has launched the Drive Constellation simulation system and Drive Sim, which are built on two different servers for generating virtual driving vehicles, generating sensor signals, and processing simulated sensor data.

Baidu Apollo can quickly build a set of auto drive system through four major parts: vehicle platform, hardware platform, software platform and cloud data service, providing a complete software and hardware solution and service system. Apollo collaborated with Unity to develop a virtual simulation environment based on the Unity engine, which enhances real-world images by simulating traffic flow to create realistic simulation scenes.

In terms of cloud services, Huawei and Tencent are also working on researching cloud ecosystem development systems. Huawei has launched its autonomous driving cloud service, HUAWEI Octopus. The system integrates cloud training and cloud parallel simulation, with rich simulation scenarios and high concurrency instance processing capabilities. It provides over 200000 simulation scenario instances, with daily virtual testing mileage exceeding 10 million kilometers, and supports concurrent testing of 3000 instances. The system provides a complete solution for autonomous driving testing from aspects such as data collection, data mining, data annotation, algorithm training, simulation platform, dataset, and scene library.

Tencent has released the TAD Sim simulation platform, which combines professional game engines, industrial grade vehicle dynamics models, and integrated virtual and real traffic flow technologies to create an autonomous driving simulation platform that combines virtual and real elements and integrates online and offline processes. It can achieve geometric, logical, and physical reconstruction of scenes. TAD Sim supports cloud operation, including two modes of scenario based cloud simulation and virtual city based cloud simulation, which can achieve accelerated simulation and high concurrency simulation, meet various scenarios and driving possibilities in the real world, and accelerate the process of autonomous driving testing.

Testing tools are the foundation of the autonomous driving testing system. Currently, there are problems such as unclear definitions, inconsistent interfaces, and incomplete content among different testing tools, resulting in poor coordination between tools and difficulty in connecting them, which is not conducive to the construction of the testing system. Therefore, in order to accelerate the standardization testing cycle of autonomous driving, it is urgent to further clarify the testing content of each simulation testing tool, determine the testing boundaries of each tool, unify the input and output interfaces of each testing platform, form data sharing among various testing tools, increase the coordination between testing tools, achieve standardized testing processes and standards for autonomous driving, and accelerate the construction of a complete autonomous driving testing tool chain and testing system.

7.2 Autonomous driving testing and evaluation system (Yang Lan, He Zeyu)

The evaluation system for autonomous driving testing is a crucial link in the autonomous driving toolchain. In the industrial sector, the traditional development and testing process for autonomous driving adopts a V-shaped development process, which includes two stages: design and development process and testing and verification process. The design and development process is mainly based on Model Based Design (MBD), mainly built on MATLAB and Simulink. The testing and verification process is implemented using different testing methods at different stages. With the increasing complexity of auto drive system and traffic environment, the traditional MBD cannot meet the test requirements. It is urgent to establish a systematic, scientific and effective test and evaluation method for automatic driving vehicles. By analyzing the key technologies, safety features, and new requirements for testing and evaluation methods of autonomous driving, the International Organization for the Automobile Industry has proposed a "three pillar" testing and evaluation method for autonomous vehicles, which includes actual road testing, closed field testing, and audit evaluation with simulation testing^[255]The United Nations Informal Working Group on the Evaluation and Management of Autonomous Driving has proposed a testing and evaluation method that includes technical means such as scenario directory, simulation testing, closed field testing, actual road testing, audit evaluation, and in use monitoring reports. The Ministry of Industry and Information Technology is gradually clarifying the requirements for product process assurance and testing evaluation of intelligent connected vehicles, and promoting research on the admission management of intelligent connected vehicles.

Since 2013, governments, car companies, and parts suppliers around the world have conducted relevant research, such asTable 14As shown, Waymo, Cruise, and PEGASUS focus on using virtual simulation testing to simulate various driving environments and accelerate the testing process. The AdaptIVe project and BMW conducted testing and verification of specific levels of autonomous vehicles through a combination of real vehicle testing, virtual simulation testing, and real vehicle testing. The Toyota Research Institute (TRI) in Japan has verified the safety of autonomous vehicles under different levels of driving modes through open road testing. After 2018, China's autonomous vehicle testing and evaluation projects have gradually been launched. The Vehicle Road Collaboration Joint Laboratory jointly established by Alibaba Damo Institute and the Ministry of Transport is committed to the intelligence of autonomous driving and road traffic through open road testing. The Institute of Automation of the Chinese Academy of Sciences, Tencent and Baidu have improved testing efficiency, reduced testing costs and accelerated the landing of autonomous vehicles through virtual simulation testing combined with real vehicle testing. Chang'an University has taken the lead in hosting the first key research and development plan for autonomous driving testing under China's 14th Five Year Plan, the "Autonomous Driving Simulation and Digital Twin Testing Evaluation Tool Chain".

Table 14. Projects related to autonomous driving test and evaluation in various countries

序号	项目/机构	国家	时间	方法	核心	测试目标	测试工具
1	Waymo	美国	2013年	虚拟仿真测试	模拟各种复杂路况	提升测试里程数与加速测试	基于Carcraft的虚拟仿真测试
2	AdaptIVe	欧洲	2014年	面向L2级以上实车测试	侧重于用户体验相关指标的研究	研究车载传感器与车对车互联、V2X	开放道路测试
3	Cruise	美国	2016年	虚拟仿真测试	现实场景转化到可以编辑的虚拟场景	推进L4级自动驾驶车辆测试	基于Morpheus的虚拟仿真测试
4	PEGASUS	德国	2016年	虚拟仿真测试	通过虚拟仿真环境对自动驾驶技术进行测试和验证	提高自动驾驶车辆安全性	基于研发的虚拟仿真软件
5	BMW	德国	2017年	面向L4级虚拟仿真测试与实车测试	通过数据驱动进行严格的测试	提高自动驾驶车辆安全性	基于Unity的虚拟仿真测试与开放道路测试
6	TRI	日本	2017年	实车测试	高级驾驶辅助模式和完全自动驾驶模式的综合测试	防止车辆直接撞向前方物体，探测驾驶人行为	开放道路测试
7	车路协同联合实验室	中国	2018年	实车测试	以车辆的需求为主体，传送不同的信息给不同的车辆	研究CAV与路侧基站的信息感知及交互能力，CAV开启与否对AEB功能的影响	开放道路测试
8	中国科学院自动化研究所	中国	2019年	虚拟仿真测试与实车测试	提出一种新的图灵测试方法测试无人车对复杂场景的理解和决策能力	研究无人车对复杂交通场景的理解与决策的能力	基于无人驾驶测试与验证系统的虚拟仿真测试与封闭道路测试
9	长安大学	中国	2021年	自动驾驶仿真与数字孪生测试评价工具链	面向演进交通环境下的自动驾驶上路许可与效能评价	加速突破适应演进交通环境的高等级自动驾驶系统上路许可与效能评价中的关键技术，开发面向智能评级、缺陷识别及安全认证的完整工具链	实时仿真软件、加速仿真云平台、场地孪生测试系统、整车交通在环测试平台与综合效能评价体系
10	百度Apollo	中国	2021年	面向L3/L4级虚拟仿真测试与实车测试	侧重于实现量产园区自动驾驶、限定区域城市自动驾驶、城市全网自动驾驶	研究使用纯路侧感知能力，真正实现开放道路连续路网L4级别自动驾驶闭环的车路协同技术	基于Apollo Studio虚拟仿真测试与开放道路测试

| Show Table

DownLoad: CSV

At the same time, some universities and research institutes have conducted relevant research on the testing and evaluation system of autonomous vehicles. The Blue Book on Autonomous Driving Simulation Technology in China (2019), jointly released by Tsinghua University Suzhou Automotive Research Institute and 51VR, proposes a simulation testing and evaluation system for autonomous driving vehicles from four aspects: driving safety, driving comfort, traffic coordination, and standard matching; Zhao Xiangmo and others^{[166, 256]}Propose an indoor rapid testing platform for autonomous driving vehicles based on vehicle in the loop simulation, which can quickly test performance indicators such as intelligent perception and behavioral decision-making of autonomous driving vehicles. In subsequent research, a virtual simulation testing platform for autonomous driving vehicles based on a multi degree of freedom drum platform is also proposed, and the effectiveness of the platform is verified through adaptive car following and collision scenario testing of autonomous driving vehicles; Son and others^[257]A method was proposed to evaluate the performance of autonomous vehicles by selecting different key indicators for different testing conditions; Liu Fawang and others^[258]After comparing the advantages and disadvantages of three methods, namely simulation testing, closed field testing, and actual road testing, a scenario based "three pillar" fusion testing and evaluation method is proposed to provide a guarantee for the comprehensive evaluation of the safety of intelligent connected vehicles with autonomous driving functions; Kim and others^[259]A measurement method based on monocular or binocular cameras was proposed to evaluate the safety of autonomous vehicles, and the reliability of the proposed method was verified by comparing the results of actual vehicle tests; Zhang et al^[260]A multi logic scenario safety evaluation method for functional safety of autonomous vehicles based on natural driving trajectories is proposed to evaluate the comprehensive performance of the tested autonomous vehicles in various scenarios.

7.3 Autonomous driving testing standards (Yang Lan, Yuan Jiaqi)

7.3.1 Standard specifications for autonomous driving testing

Various countries around the world have gradually carried out the construction of relevant policies, standards, and norms for autonomous driving testing and evaluation, in order to accelerate the implementation process of autonomous driving. Since 2001, autonomous driving standards have attracted widespread attention from standardization organizations around the world. Organizations such as the International Organization for Standardization (ISO) and the Society of American Engineers (SAE) have proposed standards related to autonomous driving testing to lead autonomous vehicles to undergo more comprehensive and thorough testing. Currently, hundreds of autonomous driving standards have been released one after another^[261], such asFigure 24As shown.Figure 25We have compiled statistics on the global development and release of autonomous driving standards, with CEN being the European Institute for Standardization, ETSI being the European Telecommunications Standards Institute, and 3GPP being the International Telecommunication Standards Organization.

Figure 24. Trend of issued number of autonomous driving standards in the past decade

DownLoad: Full-Size Img PowerPoint

Figure 25. Global release of autonomous driving standards

DownLoad: Full-Size Img PowerPoint

At the same time, countries are committed to issuing regulations for autonomous driving testing and evaluation, promoting the sustainable development of autonomous vehicles.Table 15Summarized the relevant regulations issued by China, the United States, Japan, Germany, and the United Kingdom in the past 10 years.

Table 15. Regulations on autonomous driving issued by various countries in past decade

国家	法规	时间	机构/地点	主要内容
中国	《北京市自动驾驶车辆道路测试管理实施细则(试行)》	2017年	北京市交通委员会、北京市公安交通管理局、北京市经济和信息化局	规定了自动驾驶的研发和测试中对测试单位申请条件、车辆要求与驾驶要求等
	《智能网联汽车道路测试管理规范(试行)》	2018年	工信部、公安部、交通运输部	发布智能网联汽车发展技术路线图，制定了发展目标和战略规划
	《智能网联汽车自动驾驶功能测试规程(试行)》	2018年	智能网联汽车产业创新联盟、全国汽车标准化技术委员会智能网联汽车分技术委员会	规定了14个自动驾驶功能与34个场景
	《智能网联汽车道路测试与示范应用管理规范(试行)》	2021年	交通运输部、工信部、公安部	主要明确了道路测试、示范应用及测试区(场)的定义，将道路测试和示范应用的范围扩展到包括高速公路在内的公路、城市道路和区域，并对省、市级相关主管部门的主要职责与工作机制进行了说明
	《自动驾驶汽车运输安全服务指南(试行)》	2022年	交通运输部	要求从事运输经营的自动驾驶车辆应当具备车辆运行状态记录、存储和传输功能，向运输经营者和属地交通运输主管部门及时传输相关信息
	《道路车辆动驾驶系统测试场景词汇》	2022年	工信部	规范了自动驾驶系统、动态驾驶任务、设计运行范围及条件等概念，明确了场景、动静态环境与实体要素之间的关系，与功能安全、预期功能安全等国际标准建立了配套关系
美国	《自动驾驶法案(H.R. 3388)》	2017年	众议院	首次对自动驾驶车辆的生产、测试和发布进行管理
	《自动驾驶车辆法》	2021年	亚利桑那州	只要车内有驾驶人，L3和L4级就可以在亚利桑那州的公共道路上行驶
	《自动驾驶客运服务试验和部署计划书》	2021年	加利福尼亚州	将自动驾驶所有活动分为4种，并形成4种不同计划的监管规则
德国	《道路交通法第八修正案》	2017年	联邦议院	允许L3级自动驾驶上路行驶，规定记录装置，明确交通事故责任
德国	《道路交通法修正案(自动驾驶法)》	2021年	联邦政府、联邦议院、参议院	允许自动驾驶L4级车辆在特定场景下和特定区域内行驶
日本	《自动驾驶汽车道路测试指南》	2016年	警察厅	要求驾驶人须坐在驾驶位置上进行测试并遵守相应法律法规，且先进行封闭试验场地测试
日本	《道路交通法》	2019年	众议院、政府内阁	允许L3级自动驾驶上路行驶
英国	《自动驾驶车辆发展道路：无人驾驶技术规则综述》	2016年	商务部、交通运输部	支持Google、捷豹、沃尔沃、日产等自动驾驶车辆制造厂商在英国进行公共道路测试
	《汽车技术和航空法案》	2017年	交通运输部	规定了自动驾驶车辆发生事故时的责任分配问题以及影响责任判定的因素
	《自动与电动汽车法案》	2018年	交通运输部	规定了自动驾驶车辆引起的交通事故保险赔付方法
	《自动驾驶联合报道》	2022年	英格兰和威尔士法律委员会、苏格兰法律委员会	提出75条自动驾驶法规方面的建议

| Show Table

DownLoad: CSV

7.3.2 Content of Autonomous Driving Test Standards

While countries are committed to public road testing permits and safety responsibility allocation for autonomous vehicles, various organizations and institutions have also provided detailed explanations of the testing functions and evaluation indicators covered by autonomous vehicles at all levels.Table 16Summarized some standard functions of autonomous driving, among which the standards related to automatic lane keeping system and automatic emergency braking system have the highest proportion in autonomous driving function testing.

Table 16. Test functions included in autonomous driving test standards

标准	功能
标准	ACC	FCW	变道决策	盲区检测	LKA	车道偏离预警	AEB	低速跟踪
ISO 17387：2008			√
ISO 22178：2009								√
ISO 15623：2013		√
ISO 17361：2017						√
ISO 15622：2018	√
SAE J2400—2014		√
SAE J2399—2021	√
《智能运输系统自适应巡航控制系统性能要求与检测方法》(GB/T 20608—2006)	√
《智能运输系统车道偏离报警系统性能要求与检测方法》 (GB/T 26773—2011)						√
《乘用车自动紧急制动系统(AEBS)性能要求及试验方法》(GB/T 39901—2021)							√
NHTSA^[262]		√		√	√	√	√
Euro-NCAP^[263]		√			√	√	√
A-NCAP^[264]	√	√			√	√	√
C-NCAP^[265]							√

| Show Table

DownLoad: CSV

The standard specifications proposed by most organizations provide a functional safety system framework for automobiles, including guidance for vehicle lifecycle modification activities, specific risk assessment methods, necessary safety requirements for acceptable residual risks, effectiveness and deterministic measures to ensure sufficient and acceptable safety levels.Table 17Statistically analyzed the performance evaluation indicators included in some testing standards.

Table 17. Evaluation indicators included in autonomous driving test standards

标准	评价指标
标准	伤害的严重性	暴露于危险中的可能性	危险的可控性	风险可接受性	警报准确性	系统报警重复性	警报可接受性	碰撞警报及时性	安全等级独立性	目标识别准确性	防撞击性	探测区域有效性	车道辅助系统的避让性	抗翻滚性
ISO 15623:2013					√			√		√	√	√
ISO 17361:2017					√	√	√
ISO 26262:2018	√	√	√						√
ISO 21448:2022	√	√	√	√
SAE J2400—2003^[266]						√	√	√		√		√
SAE J2980—2018	√	√	√
SAE J2399—2021										√
GB/T 26773—2011					√	√	√
GB/T 34590—2017	√	√	√						√
NHTSA 26555^[262]	√	√			√					√	√
NHTSA NCAP^[267]	√				√	√	√	√			√			√
C-NCAP^[267]	√										√		√
Euro-NCAP^[268]	√										√		√
J-NCAP^[269]	√										√

| Show Table

DownLoad: CSV

7.4 Summary and Outlook

At present, the evaluation tool chain for autonomous driving testing is generally dominated by virtual testing tool chains, which are used for the integration of SIL, MIL, HIL and other tools. The technological development of full process testing is still immature; The non standardization of testing tools and the phenomenon of information silos are quite serious, standards have not yet been promoted, and tools are independent of each other. The testing and evaluation tools and processes for a single function are no longer sufficient to meet the testing and evaluation requirements of high-level autonomous vehicles. Further research is needed on cross platform system logic structure, testing business, data and function system integration methods, testing data sharing and synchronization methods, and the construction of a comprehensive and unified testing and evaluation system for autonomous vehicles.

8. Autonomous vehicle certification and potential defect detection technology (Zhang Yutan, Min Haigen)

This section analyzes the current admission certification standards and management mechanisms for autonomous vehicles. It summarizes the current defect detection technologies for autonomous driving from the aspects of definition, cause analysis, classification, and detection of autonomous driving defects. It also summarizes the challenges faced in ensuring the safety of autonomous driving vehicles. The organizational structure is as follows:Figure 26As shown.

Figure 26. Organizational structure of certification and potential defect detection technology of autonomous driving vehicles

DownLoad: Full-Size Img PowerPoint

8.1 Autonomous driving access certification standards and sandbox supervision system (Zhang Yutan)

In recent years, with the rapid development of autonomous driving technology, many mass production projects of automobile manufacturers have entered the public eye. Technical analysis shows that although some autonomous vehicles have reached the L3 level, considering regulations and certifications, L2 level assisted driving is still promoted and promoted. At present, although ISO/SAE industry standards provide a strict framework for vehicle information security, expected functional safety of vehicles, and detailed guidance at the operational level, this series of standards has no geographical restrictions or legal constraints. The United Nations World Forum for the Coordination of Vehicle Regulations recently released three important regulations on intelligent connected vehicles, which stipulate that new vehicle models must meet the regulations before they can be launched in the European Union from July 2022, mainly including certification standards related to information security (UN R155), software upgrades (UN R156), and automatic lane keeping systems (UN R157). Information security certification mainly includes information security management process system certification and vehicle type certification. The information security management process system is mainly used to identify, analyze, and defend against potential risks such as information security threats, vulnerabilities, and attacks. Vehicle type certification mainly covers the relevant performance requirements and test methods of vehicle components, systems, and vehicle products, and is mainly based on the vehicle framework directive for vehicle product certification. Software upgrade certification includes the host factory verifying the configuration and quality control process, security execution upgrade process, software upgrade network security process, and approving the registered software upgrade, which requires the authenticity and integrity of the software upgrade, and the host factory can reliably perform the upgrade. The certification of the automatic lane keeping system mainly specifies system safety and fault safety response, human-machine interface and driver operation information, network safety and software updates, as well as operational safety requirements and corresponding testing standards applicable to this function. It is the first international regulation in the field of L3 level autonomous driving. The above three regulations mark that the auto drive system has entered the era of compliance from compliance with standards to compliance with regulations.

To ensure that the existing regulatory certification system does not hinder the investment of new products and technologies, foreign countries have established pilot zones to provide various driving scenarios for enterprises, and enterprises can conduct testing in restricted scenarios; The government can add clauses to exempt companies from testing for parts that cannot be tested under relevant regulations. China draws on relevant foreign experience to introduce a sandbox regulatory system, which can effectively guide enterprises to identify problems^[270]Improve design and reduce risks. Enterprises need to submit a written application for sandbox supervision to the State Administration for Market Regulation. Based on the enterprise's application, the State Administration for Market Regulation will conduct a preliminary evaluation. The enterprise will develop a vehicle deep testing plan for testing technology and conduct testing. During the mid-term and final stages of testing, relevant evaluation reports will be submitted to the regulatory authority. After completing the deep testing and achieving the expected goals, the enterprise will automatically exit the sandbox supervision. Sandbox regulation encourages enterprises to engage in technological innovation and provides a safe testing zone for them, allowing them to conduct reasonable testing of new technologies beyond regulatory requirements, which is conducive to the development of cutting-edge technologies such as intelligent driving.

8.2 Definition of Defects in Autonomous Driving (Zhao Ning)

The "Regulations on the Management of Recall of Defective Automotive Products" clearly define defects as situations that generally exist in the same batch, model, or category of automotive products due to design, manufacturing, labeling, or other reasons that do not comply with national or industry standards for safeguarding personal and property safety, or other unreasonable dangers that endanger personal and property safety. This definition specifies two elements, one is non-compliance with standards, and the other is unreasonable danger.

According to the statistics on automobile recalls released by the Defective Product Management Center of the State Administration for Market Regulation from 2004 to 2018, the recall situation of automobile parts and subsystems is as follows: airbag and seat belt defects account for 33% of the total recall amount, engine defects account for 25%, electronic and electrical components account for 12%, braking/wheels account for 8%, steering/suspension account for 8%, power transmission system accounts for 6%, vehicle body accounts for 5%, and other parts account for 1%. In recent years, with the continuous improvement of automotive intelligence, there have been frequent recalls related to autonomous vehicles. For example, recall cases caused by partial non illumination of parking lights due to coding errors in the SCN software for parking lights; Due to improper programming of the electronic parking brake control program, there is a possibility of continuous overheating and damage to the electronic parking brake motor, which could endanger personal and property safety and lead to a recall; The recall case was triggered due to the risk of rear end collision caused by the AEB system mistakenly recognizing the metal seam on the road surface as an obstacle and accidentally braking; Due to improper programming of the vehicle controller program, there is a possibility of abnormal engine shutdown during driving when using the remote function of the mobile phone, which may lead to recall cases.

From the above cases, it can be seen that the defects of autonomous vehicles include not only the common non-compliance with design specifications (or standards) and random hardware failures of traditional automotive components and electronic appliances, but also software errors caused by the large amount of software code contained in their complex functions, sensor perception errors, and loss of control due to network security vulnerability attacks. These defects are resolved through functional safety software and hardware development processes and testing methods.

However, when autonomous vehicles are on the road, there may still be some performance deficiencies within their capability boundaries, leading to accidents that endanger personal and property safety. For example, when an autonomous vehicle passes through an elevated speed measuring camera at night, the moment the camera is exposed and photographed causes the perception camera to be unable to recognize the target object ahead, resulting in the loss of target tracking, leading to control failure and causing accidents. At this point, the system did not perceive any errors or hardware failures, but rather because the camera's dynamic range did not cover such a scene.

Auto drive system is a highly complex system, and the technology itself has natural limitations. For example, as mentioned earlier, cameras have limited perception performance for strong light, millimeter wave radar and lidar have limited detection performance due to electromagnetic wave scattering in rainy, snowy, and hazy weather, deep neural network algorithms have limitations in generating parameter differences for different training samples, and particle filtering methods in synchronous localization and mapping algorithms have limited time characteristics. There is also insufficient consideration in the design of functional scenarios, unexpected possibilities for various scenarios, and possible misuse caused by users' lack of understanding or misunderstanding of system functions. These are the contents of the research on the expected functional safety of the auto drive system, which should be summarized as the defects of the auto drive system.

As for whether the hazard event caused by the defect of the auto drive system belongs to "unreasonable risk", the acceptance criteria often referred to in the industry are: the residual risk of any new system should not be higher than those of existing systems with similar functions or hazards, that is, the risk tolerance principle; Weighing the cost of reducing risk (for situations where there is no existing system as a reference); The risk level caused by defects in the auto drive system is not higher than that generated by skilled and prudent drivers in the same scenario.

For unreasonable risks caused by defects of the auto drive system, if at the product development stage, the original commissioned production of the whole vehicle should reduce the risk probability through reasonable testing and improvement strategies. If it occurs during the sandbox testing phase, complete test data and defect solutions should be provided, and it should be demonstrated in subsequent sandbox testing phases that the defect issue has been resolved. If it occurs during the user's usage phase, there may be a situation where the product is recalled.

To sum up, this paper defines the defects of the auto drive system in the way of enumeration as follows: the product design does not conform to the national and industrial standards to ensure personal and property safety; Random hardware failure; Software error; The attack of information security vulnerability database leads to the failure of auto drive system; Risk events where the risk level of the auto drive system is higher than that of a skilled and prudent driver due to sensor perception and recognition limitations; In complex traffic scenarios, autonomous driving trajectory planning and control algorithms are unable to cope with traffic scenarios that skilled and cautious drivers can handle; Risk events that the risk level of the auto drive system is higher than that of the skilled and prudent driver due to insufficient control of vehicle stability; Reasonable and foreseeable personnel misuse.

8.3 Root cause analysis of defects in autonomous driving (Pei Shuanghong)

The defects of the auto drive system summarized above, especially the expected functional safety defects of insufficient functional design and insufficient performance of sensing components, need to be analyzed and studied for specific functional scenarios. Therefore, the analysis of automatic driving scenarios is crucial to the exploration of the cause mechanism of auto drive system defects. However, as mentioned above, the dangerous scenes encountered by the auto drive system in real roads are inexhaustible. Therefore, a method needs to be found to map the inexhaustible dangerous scenes into the limited parameter space.

ISO 34502： 2022 The scenario based auto drive system safety assessment framework standard proposes such a method, which decomposes the dynamic driving task of the auto drive system into three sub tasks of perception, decision-making and control, and each sub task is associated with one or several specific physical principles. If the process of implementing this physical principle is triggered by scenario risk factors, it may lead to the inability to achieve normal physical performance, which is manifested as a system defect. If the physical principles of perception, decision-making and control sub tasks and the corresponding scenario risk factors are fully analyzed, all potential defects of the auto drive system can be covered.Figure 27It indicates that the dangerous scene of the auto drive system can be divided into three sub tasks: perception, decision-making and control. In the figured_xandd_yThe longitudinal and transverse relative distances between the two vehicles respectively;V_yFor the lateral speed of the vehicle;G_xReduce the longitudinal speed of the vehicle.

Figure 27. Key scenarios of autonomous driving system

DownLoad: Full-Size Img PowerPoint

ISO 34502： 2022 The scenario based auto drive system safety assessment framework standard also proposes a hypothesis, that is, if the automated driving scenario and scenario risk factors can be deconstructed, then for a specific dangerous scenario, the deconstructed risk factors will be generalized to a reasonable value space according to the analysis of physical principles, and then the generalized scenario risk factors will be logically reconstructed, thus forming a logical scenario. Assuming that this logical scenario contains three risk factors: decision-making, perception, and control, represented by A, B, and C respectively,N_A、N_B、N_CIf we represent the number of elements in the generalization vectors of these three risk factors separately, then this logical scenario can be reconstructed asN_AN_BN_CThe possibility of parameter combinations, which can generateN_AN_BN_CA specific testing scenario. Due to the uniqueness of physical principles, the values of risk factors corresponding to physical principles theoretically have boundaries. In this way, the dangerous scenes that cannot be enumerated in the real world will be transferred to the limited logical scene space, that is, to study the defect cause mechanism of the auto drive system is to study the scene risk factors and their reasonable combinations.

ISO 34502： The advantage of the 2022 scenario based auto drive system safety assessment framework standard is that it does not rely on the auto drive system itself and has strong versatility. In other words, if the ODDs of two different auto drive system are the same, for scenario based risk factor analysis of one of them, the logical scenario library and safety assessment methods can also be applied to the other auto drive system, regardless of whether the two auto drive system have the same sensor selection, software and hardware architecture, etc.

The three terms in the ISO 34502:2022 scenario based auto drive system safety assessment framework standard: perception interference, traffic interference and vehicle interference are introduced to represent the impact of scenario risk factors on the physical principles of perception, decision-making and control.

8.3.1 Traffic Disturbance

The physical principles related to the sub tasks of autonomous driving decision-making are as follows:Figure 28As shown, the positions and motion behaviors of surrounding traffic vehicles that pose a danger to the self driving motion decision are listed in 12 grids. Traffic vehicles in different locations have different motion threats, including entry, exit, acceleration, and deceleration.Figure 29The method of cross analyzing scene risk factors and traffic physics principles to obtain traffic interference scenarios was demonstrated. The left column shows the classification of scene risk factors, including main roads, merging areas, diverging areas, and the concentration of main vehicle lane keeping and lane changing under these three road structures. Upward refers to the category of traffic vehicle motion that implies the position relationship of traffic vehicles. pass throughFigure 29Cross analysis has identified 24 potential hazardous scenarios that may be encountered in real traffic.

Figure 28. Several situations posing risks to movement decision-making of autonomous driving vehicles

DownLoad: Full-Size Img PowerPoint

Figure 29. Traffic interference scenarios obtained from cross analysis of scenario risk factors and traffic physics principles

DownLoad: Full-Size Img PowerPoint

8.3.2 Perceived interference

Perceived interference can be divided into three categories: sensor perception performance limitations, occlusion blind spots, and V2X communication performance limitations. The limitations of perceptual performance are related to the specific types of sensors and the physical principles of their perceptual recognition. Taking millimeter wave radar as an example, a two-dimensional matrix is used to cross analyze the scene risk factors and the physical principles of millimeter wave radar detection, and the causal mechanism of millimeter wave radar perceived interference is obtained, such asTable 18As shown.

Table 18. Analysis framework of sensing interferences from millimeter-wave radars

毫米波雷达						物理原理(信号感知/传感器安装方位角)
						被测目标物的信号								其他探测物的信号
						频率	相位			能量				噪音	非期望的信号
							探测方向		传播延迟变化	无信号(部分信号)	高频		信号差异大	噪音	非期望的信号
							反射	折射	传播延迟变化	无信号(部分信号)	混叠	谐波	信号差异大	低信噪比	有效信号比例低	无效信号增长
感知干扰的因果因子	自车/ 传感器		自车		车辆姿态改变									○	○	○
			传感器		装配松动			○	○					○	○	○
			传感器		传感器失效	○		○	○		○	○		○		○
			前表面		黏附物			○	○					○
			前表面		物理特性改变			○	○					○
	环境		结构化目标物	道路表面	形状									○	○	○
					道路情况										○	○
					路面材质										○	○
				路边物体	反射		○				○				○	○
					遮挡					○				○
					背景
				头顶物体	反射										○	○
					遮挡					○
					背景
			空间		空间物体					○				○	○	○
			空间		空间电磁波和光线										○	○
			移动目标物		反射		○				○	○			○	○
					遮挡
					背景
	感知目标物	车道	线条		颜色/材质
					形状
					脏污/模糊
					相对位置
			具有高度的结构化物体		颜色/材质									○
					形状-反射强度大						○	○	○
					形状-反射强度小									○
					脏污									○
					相对位置									○
			道路边缘	道路边缘(平坦)	颜色/材质									○
					形状									○
					脏污/模糊									○
					相对位置									○
				道路边缘(不均匀)	颜色/材质									○
					形状									○
					脏污/模糊									○
					相对位置									○
		道路上的障碍物	掉落物		颜色/材质									○
					形状									○
					相对位置/运动									○
			动物		颜色/材质
					形状									○
					相对位置/运动									○
			临时建筑		颜色/材质									○
					形状-反射强度大						○	○	○
					形状-反射强度小									○
					脏污									○
					相对位置									○
		移动目标物	其余车辆		颜色/材质
					形状-反射强度大						○	○
					形状-反射强度小									○
					脏污									○
					相对位置									○
			摩托车		颜色/材质									○
					形状/大小									○
					黏附物									○
					相对位置									○
			自行车		颜色/材质									○
					形状/大小									○
					黏附物									○
					相对位置									○
			行人		颜色/材质									○
					形状/大小									○
					相对位置									○

| Show Table

DownLoad: CSV

Blind spot is a type of perceptual interference that autonomous vehicles often encounter during operation. Blind spot obstruction can be divided into obstruction by traffic vehicles, obstruction by road structures (such as barriers), and obstruction caused by different road surface elevations (such as downhill slopes, ramps, etc.). The situation of vehicle obstruction can be divided into front/rear obstruction, side obstruction, oblique obstruction, and the situations where these three types of obstruction occur in straight roads, left bends, and right bends, respectively.

8.3.3 Vehicle interference

The driving of a vehicle is essentially a relationship between force and motion. The forces exerted on the vehicle are mainly the forces acting on the body of the vehicle and the forces acting on the wheels. The forces acting on the vehicle body include the force of wind, the gravitational force generated by road surface inclination, and the centrifugal force when the car turns. The forces acting on the wheels include road friction (braking and driving forces), the impact force of road bumps, and the impact force during tire blowouts.

8.4 Classification of Defects in Autonomous Driving (Min Haigen, Fang Yukun)

The emergence of autonomous driving technology has shifted a significant portion of driving errors from drivers to producers who design and manufacture autonomous vehicles, and defects in the production process will affect driving safety. in compliance withTable 19As shown, this article classifies defects in autonomous driving based on differences in production processes and sources of safety risks.

Table 19. Classification of autonomous driving defects

分类依据	名称	含义	示例
生产环节	制造缺陷	制造过程中产品偏离预期设计	传感器功能障碍
	设计缺陷	未采纳替代设计规避可预计的风险	自动驾驶软件设计忽略了某些对人身财产安全具有重要影响的因素
	警示缺陷	说明或警示不充分带来的不合理危险	某些特定工况下，自动驾驶系统未能向用户发出警示和接管请求
安全风险来源	功能安全缺陷	导致自动驾驶电子电气系统功能异常的安全风险因素	软硬件故障
	预期功能安全缺陷	由于自动驾驶功能实现不足而导致安全风险的因素	传感器感知性能不足
	信息安全缺陷	可能使自动驾驶系统处于网络攻击威胁的因素	通信协议漏洞

| Show Table

DownLoad: CSV

According to the differences in production processes, Wang Lebing^[271]Classify the defects of autonomous vehicles into manufacturing defects, design defects, and warning defects. Manufacturing defects refer to situations where a product deviates from its intended design during the manufacturing process, such as sensor dysfunction or installation errors, resulting in the failure of the autonomous driving perception module and posing safety risks; Design defect refers to the foreseeable risk of damage to a product that could have been reduced or avoided by the producer or seller through the adoption of a reasonable alternative design, but the product has unreasonable danger due to the failure to adopt that alternative design. For example, the designer of autonomous driving software ignored certain factors that have a significant impact on personal and property safety, resulting in potential safety risks to the autonomous driving function; Warning defect means that the foreseeable risk of damage caused by the product could have been reduced or avoided by providing reasonable guidance, education or warning by the manufacturer or seller, but the product has unreasonable risks due to insufficient instructions or warnings. For example, under certain specific conditions, the auto drive system does not send a warning and takeover request to the user, making the vehicle in an unsafe driving state.

From a safety perspective, autonomous driving safety mainly involves three aspects: functional safety, expected functional safety, and information safety. Road vehicle functional safety refers to the absence of hazards caused by abnormal performance of electronic and electrical systems, resulting in unreasonable risks; Expected functional safety refers to the absence of unreasonable risks caused by the inadequacy of expected functions or their implementation; Information security refers to the absence of unreasonable risks caused by internal or external network attacks. The software and hardware of the auto drive system are highly integrated to support the realization of all dynamic driving tasks. Its defects may bring the above three safety risks. In this context, firstly, sensors, actuators, computing platforms, software, communication, etc. may experience systematic or random hardware failures, causing harm to the entire vehicle and resulting in functional safety issues; Secondly, due to the insufficient and uncertain implementation of scene perception, decision-making algorithms, artificial intelligence, etc., the functional limitations of autonomous driving are caused, resulting in non failure risks and affecting the safety of the entire vehicle's driving, that is, expected functional safety issues; The third is the security problem caused by the abnormal function of the auto drive system caused by network attacks, that is, the information security problem. Therefore, the defects of the auto drive system are summarized as "factors that make the auto drive system appear safety risks", and the automatic driving defects are classified as functional safety defects, expected functional safety defects and information safety defects from the perspective of safety risk sources. Functional safety defects guide the safety risk factors that cause functional abnormalities in the electronic and electrical systems of autonomous driving, such as software and hardware failures, product design and manufacturing that do not comply with national and industry standards, etc; Expected functional safety defects refer to factors that lead to safety risks due to insufficient implementation of autonomous driving functions, such as inadequate sensor sensing performance, inadequate implementation of autonomous driving perception, planning, decision-making, and control algorithms, foreseeable personnel misuse, etc; Information security defects refer to factors that may make the auto drive system vulnerable to network attacks, such as communication protocol vulnerabilities and missing data verification.

8.5 Autonomous driving defect detection (Min Haigen, Fang Yukun)

The defect of auto drive system brings huge safety risks, and detecting the defect of automatic driving products is of great significance for ensuring driving safety. The defect detection methods discussed in this section focus on autonomous driving products and do not involve defects related to human misuse.

The existence of defects or abnormal autonomous driving functions can be reflected by the observation data of specific observation objects. From the perspective of data analysis, anomalies refer to individuals in the data that are significantly different from most samples. Analyzing the collected system operation data and capturing abnormal data patterns is an effective means of detecting defects. Therefore, defect detection can be achieved to some extent by detecting data anomalies during the operation of various subsystems in autonomous driving. Common methods include model-based methods, signal based methods, data-driven methods, and knowledge-based methods, which will be elaborated one by one in this section.

8.5.1 Model based defect detection method for autonomous driving vehicles

Model based defect detection methods require a system model that can clearly describe the relationships between system variables. Using this system model, defect detection algorithms can be developed to monitor the consistency between model predictions and system measurements^[272]Model based methods can be further divided into methods based on verification relationships^[273]Parameter estimation method^[274]And observer based methods^[275]This type of method requires a system model as prior knowledge, and the diagnostic process largely depends on the accuracy of the model. The model characterizes the prior knowledge of the system. For autonomous vehicles, the established model is usually used to describe the prior knowledge of the following objects: (1) components/subsystems of autonomous vehicles; (2) Autonomous driving vehicles for bicycles; (3) A group of intelligent connected vehicles with collaborative relationships. At the component or subsystem level, the model describes the dynamic characteristics of a specific component or subsystem, and defect detection is achieved by checking the consistency between the predicted values of the model and the measured values of the system. From the perspective of autonomous vehicles, the model used for defect detection should reflect at least one aspect of the vehicle's characteristics, such as the vehicle's kinematics or dynamics model reflecting the vehicle's motion characteristics. For a group of intelligent connected vehicles with collaborative relationships, each agent has the function of communicating with some or all of the agents within the group. The model is used to describe how the agents interact with each other, and defects are detected by comparing the actual state or behavior of a specific agent with the expected state or behavior given by the interaction model.

8.5.2 Signal based defect detection method for autonomous driving vehicles

The signal based approach assumes that defects in a system or process can be reflected in certain specific measurement signals. It extracts features from the original signal and detects defects by comparing the differences between the extracted features and the features in the system's healthy state^[272]For autonomous vehicles, signal processing technology is typically used to diagnose specific components or subsystems, and signals used for symptom (or pattern) analysis can be extracted in the time domain, frequency domain, and time-frequency domain. Signal processing technology is often combined with other methods, especially data-driven methods, in defect detection of autonomous vehicles. Signal processing techniques such as wavelet transform and empirical mode decomposition are usually used for data preprocessing or feature extraction.

8.5.3 Data driven defect detection method for autonomous vehicles

The precise model or prior knowledge of system health status is not always available, especially for complex systems such as autonomous vehicles, so data-driven methods are an important aspect of defect detection. Data driven methods require a large amount of historical data to extract hidden features. Based on feature extraction, defect detection is achieved by observing the real-time consistency between the system's features and the learned features^[276]Data driven defect detection methods can be divided into statistical analysis methods and non statistical analysis methods. Using statistical analysis methods to compare online data with known defect symptoms or historically observed normal behaviors for diagnosis; Non statistical analysis based methods learn system features from historical data and establish defect detection models. Under the statistical framework, commonly used methods such as principal component analysis^[277]Independent Component Analysis^[278]And support vector machine^[279]These methods require a certain amount of training data to capture key features of the system or process through statistical analysis. In non statistical data-driven frameworks, artificial neural networks are widely used in defect detection due to their powerful nonlinear approximation and adaptive learning abilities, including convolutional neural networks, Bayesian networks, autoencoders, long short-term memory networks, etc. Convolutional neural networks are known for capturing relationships between multiple variables and extracting features from spatial data^[280]Bayesian networks combine the characteristics of probabilistic models and can effectively represent the causal relationship between defects and symptoms^[281]Autoencoder is a typical reconstruction based anomaly detection method, and a trained autoencoder will output higher reconstruction errors when encountering abnormal instances^[282]Long short-term memory networks are a typical prediction based method, which assumes that normal instances are more predictive in the time dimension than abnormal instances, and trained models will return higher prediction errors when encountering anomalies^[283].

8.5.4 Knowledge based defect detection method for autonomous driving vehicles

Knowledge based methods require a knowledge base about the defect characteristics of the research object, as well as an inference engine that applies inference methods to relevant knowledge to detect defects, suitable for complex or multi-component systems without detailed mathematical models^[284]Knowledge based defect detection methods can be divided into two categories: causal analysis and expert systems^[285]Causal analysis methods typically rely on causal models of defect symptom relationships. Failure Mode, Effects, and Hazard Analysis (FMEA) and Fault Tree Analysis (FTA) are two typical defect detection methods based on causal analysis. An expert system is a system that includes and simulates the behavior of human experts in a certain professional field. It is capable of processing, memorizing, and recombining information based on new data, learning and reasoning unknown knowledge, making appropriate decisions, and explaining why such decisions are made^[286]Usually, if then rules are used to represent the knowledge of expert systems, and the use of Boolean logic in expert systems may make the system more sensitive to various uncertainties. Fuzzy logic provides the possibility to solve this problem. This method divides the feature space into fuzzy sets, uses fuzzy rules for reasoning, and provides reasoning that approximates human reasoning^[276].

8.6 Defect testing method for autonomous driving safety (Min Haigen, Fang Yukun)

It is of great significance for developing safe automatic driving products to find and eliminate the factors that lead to safety risks of the auto drive system through defect testing. This section will elaborate defect testing methods from the aspects of functional safety, expected functional safety and information safety.

8.6.1 Functional safety defect testing method

Functional safety defects guide the safety risk factors that cause functional abnormalities in the electronic and electrical systems of autonomous driving, such as software and hardware failures. For the testing of functional safety defects, fault injection can be carried out by selecting fault samples reasonably, observing and analyzing the dynamic behavior of the system through experiments, and evaluating the fault tolerance performance of the system when faults exist. Reasonably selecting samples from a known set of fault modes and allocating them to testing nodes is a key step in functional safety defect testing. Proportional stratified sampling and proportional random sampling are commonly used for fault sample allocation^[287]The proportional stratified sampling method first determines the number of fault samples to be injected into the constituent units, and then selects samples from the constituent unit fault mode library; The random sampling method directly randomly selects patterns from the fault mode library to form a sample set, without focusing on key fault modes for sampling, resulting in poor sample coverage and low system testing compliance in actual environments. The probabilistic sample allocation method based on failure rate considers the probability of failure occurrence. When the estimation of failure rate information is inaccurate, it will affect the credibility of the final test results. In addition, this method is easy to ignore the small probability failure samples, which in the actual auto drive system is easy to cause serious safety impact. The fault sample allocation methods of stratified sampling and random sampling have optimized the sample allocation scheme under certain conditions, but for the testing and verification of complex systems, the factors affecting sample allocation considered are not comprehensive and reasonable enough. The fault sample allocation scheme based on multiple factors has been valued and studied^[287]Like Deng Lu and others^[288]Considered failure rate, hazard level, and severity; Yu Siqi and others^[289]Considered factors such as failure rate, hazard level, detection and isolation time, maintainability, and testing and development costs; Li Tianmei^[290]A sample selection method based on fault diffusion intensity was proposed, and the traditional sample allocation method based on fault rate was improved; Shi Junyou and others^[291]We constructed a fault mode information model and evaluated the adequacy of the sample set by combining sufficiency measures and criteria. Fault injection is the core of testing verification, which can be mainly divided into hardware based, software based, and simulation based fault injection methods. Hardware based fault injection is closer to the real faults that occur in the system operation site, but for highly integrated systems, it is impossible to open and decompose fault testing, resulting in many hardware fault injections being unable to be carried out; Software based fault injection is relatively easy to implement, but the injected faults are limited to the software related parts; Simulation based fault injection covers a relatively wide range of fault types, with high controllability and observability. However, model development is time-consuming and the reliability of faults depends on the accuracy of the model. The use of simulation for fault injection has the characteristics of high safety, high efficiency, low cost, and strong fault repeatability. It can easily construct small probability fault samples that are not easy to occur in reality. However, the reliability of simulation affects the confidence of test results.

8.6.2 Expected functional safety defect testing method

Expected functional safety defects refer to factors that lead to safety risks due to insufficient implementation of autonomous driving functions, such as inadequate sensor sensing performance and inadequate implementation of autonomous driving perception, planning, decision-making, and control algorithms. The essence of expected functional safety defects lies in the insufficient performance of software and hardware functions, that is, in the vehicle level hazards caused by expected functional safety issues, the system behavior that leads to such hazards is due to performance limitations of software and hardware, while the system behavior that causes such hazards is triggered by triggering conditions in the scenario. The triggering condition is a specific condition in the scene that can trigger subsequent system reactions that lead to harmful behavior, or directly trigger harmful behavior caused by insufficient system functionality, and is the direct cause of vehicle hazard events. Performance limitations refer to the technical capability limitations of the component itself, which, when combined with one or more triggering conditions, can lead to hazardous behavior and are the fundamental cause of vehicle level hazardous events^[292]Therefore, the construction of an autonomous driving scenario library is crucial for testing expected functional safety defects. Considering that the auto drive system cannot exhaust the dangerous scenes encountered in the real road, it is necessary to find a method to map the endless dangerous scenes to the limited parameter space. ISO 34502： The 2022 standard decomposes the dynamic driving task of the auto drive system into three sub tasks: perception, decision-making and control. Each sub task is associated with one or several specific physical principles. If the process of implementing physical principles is triggered by scenario risk factors, it may lead to the inability to achieve normal physical performance, which is manifested as expected functional safety defects. ISO 34502： In the 2022 standard, the scenario risk factors will be further deconstructed, that is, for a specific hazardous scenario, the deconstructed risk factors will be generalized to a reasonable value space based on physical principles analysis, and then the generalized scenario risk factors will be logically reconstructed to form a logical scenario. So, the problem of constructing hazardous testing scenarios has been transformed into the problem of constructing limited logical scenarios, and the construction of logical scenarios can be further transformed into the problem of decomposing scenario risk factors and determining the range of risk factor values. Once the test scenario containing specific trigger conditions is built, the expected functional safety defects of the auto drive system can be tested, and the system performance and corresponding trigger conditions in case of danger can be recorded. Feng et al^[100]When constructing autonomous driving test scenarios, sparse but adversarial adjustments were made. While ensuring unbiased evaluation, background vehicles were trained to assist autonomous driving vehicles in learning when to execute adversarial strategies, significantly reducing the required testing mileage; Priisalu et al^[293]Transforming the problem of generating key test scenarios for autonomous driving into a training algorithm learning how to place pedestrians in the scene, inducing the generation of collision prone test scenarios to accelerate the testing process.

8.6.3 Information Security Defect Testing Method

Information security defects refer to factors that may make the auto drive system vulnerable to network attacks, such as communication protocol vulnerabilities. For the auto drive system, networking is an important trend in its future development. The vehicle diagnosis port, controller area network (CAN) bus, electronic control unit, sensor system, etc. may be attacked by network. The common attack scenarios for information security in intelligent connected vehicles can be roughly divided into remote attacks, short-range attacks, and contact attacks^[294]Remote attackers maliciously modify vehicle firmware and send control commands through remote network connections; In short-range attack scenarios, attackers control vehicles or affect their normal communication through communication methods such as Bluetooth and WiFi; In contact attack scenarios, attackers can directly obtain vehicle data and attack hardware. Kim and others^[295]The information security attacks of ICV are divided into attacks against vehicle control system, auto drive system components, V2X communication, risk assessment system, etc. To address information security vulnerabilities, network attack models targeting specific components can be constructed for testing, such as Pham^[296]Summarized common network attack models for information security in intelligent connected vehicles, such asTable 20As shown in the table, OBD refers to On Board Diagnostics system.

Table 20. Common network attack models for information security of connected and automated vehicles

网络攻击模型	目标组件	访问方式
针对OBD设备的攻击	OBD	物理端口
通过OBD对CAN总线的攻击	CAN	物理端口
远程访问ECU对CAN总线的攻击	CAN	远程访问
通过CAN总线对ECU进行攻击	ECU	物理端口
远程攻击ECU	ECU	远程访问
激光雷达欺骗	激光雷达	远程访问
激光雷达干扰	激光雷达	远程访问
雷达欺骗	雷达	远程访问
雷达干扰	雷达	远程访问
GNSS欺骗	GNSS	远程访问
GNSS干扰	GNSS	远程访问
对抗训练的图片干扰	相机	远程访问
网络信息伪造	连接机制	远程访问
网络拒绝服务攻击	连接机制	远程访问

| Show Table

DownLoad: CSV

The information security defects of the auto drive system can be tested by using the current commonly used network security testing methods. Zhou Yuanyuan^[297]By utilizing testing tools such as CAN Pick, AUTO-X, Nmap, BurpSuite+exus5, Wireshark, HackRF+Gqrx, and combining methods such as penetration testing, DoS attacks, protocol cracking, application programming interface attacks, brute force cracking, fuzz testing, code reverse analysis, port scanning and attacks, cloud hijacking, IVI communication, SQL injection, and man in the middle deception, a comprehensive and systematic vehicle networking information security testing framework has been developed.

8.7 Summary and Prospect (Min Haigen, Fang Yukun)

8.7.1 Functional safety still needs to be strengthened

In terms of functional safety, in addition to the common non conformance to design specifications (or standards) and random hardware failures of traditional automobile components and electronic appliances, the auto drive system also includes a large number of software failures contained in its complex functions, which makes it face more redundant, heterogeneous and monitoring requirements in the system hardware and software design. How to design fault diagnosis and fault-tolerant control mechanism of auto drive system so that the vehicle can still maintain the required performance when facing functional safety risks and ensure that the vehicle meets functional safety requirements remains to be studied.

8.7.2 Insufficient construction of expected functional safety testing scenarios

The expected functional safety risks are caused by a combination of insufficient system performance and certain triggering conditions in driving scenarios. In the auto drive system, the non failure risk caused by the performance limitation increases (for example, the sensor itself does not fail, but the performance may decrease, and the input may be biased, leading to problems in automatic driving decision-making and execution, which will lead to harmful behavior of the vehicle). The driving scene becomes an important factor affecting safety. The unknown unsafe scene poses a challenge to the auto drive system. There is uncertainty in machine learning, which may also lead to insufficient design, leading to non failure risk, and endangering the safety of automatic driving^[298]In addition to continuously improving system performance (such as enhancing the perception performance of sensors in hardware and improving the performance of autonomous driving perception, decision planning, and control execution algorithms in algorithms), building test scenarios, identifying triggering conditions corresponding to specific expected functional safety risks, and designing corresponding strategies to avoid possible risks are important aspects of ensuring the safety of expected autonomous driving functions. The dangerous scenarios encountered by the auto drive system in the real road are inexhaustible, and how to build test scenarios efficiently and with high fidelity is still a huge challenge.

8.7.3 Insufficient information security protection

Networking has become one of the trends in the development of autonomous vehicles. While it brings more information to assist in autonomous driving planning and decision-making, it also introduces information security issues, making cars vulnerable to malicious remote control, threats, and attacks. From the perspective of the actual network security of connected car enterprises, industry chain related enterprises, especially traditional car companies, have prominent problems such as weak awareness of network security, insufficient protection capabilities, and insufficient investment in information security^[299]In addition, the network security testing and evaluation mechanism for the Internet of Vehicles is not yet perfect, and the software and hardware information security testing and verification capabilities from components, subsystems to the entire vehicle are insufficient.

8.7.4 The uncertainty of machine learning methods

Machine learning methods are widely used in autonomous driving related applications due to their excellent performance, but the uncertainty of machine learning methods themselves is one of the sources of safety related issues in autonomous driving. Due to the complexity and diversity of the actual environment, the assumptions of the machine learning model itself may not be able to accurately describe it, resulting in inconsistent outputs of the system for given inputs, that is, outputs with uncertainty, which in turn leads to uncertainty in the system's behavioral decisions and may cause safety related accidents in autonomous driving.

8.7.5 Influence of uncertainty of road traffic participants' behavior on auto drive system

In the development process of autonomous driving, it will coexist with human driving for a long time, and the behavior of road traffic participants is highly uncertain. This uncertainty may lead to errors in decision-making algorithms and trigger traffic safety risks, posing a huge challenge to autonomous driving decision-making algorithms.

8.7.6 The formulation of safety standards for autonomous driving and related legislation need to be continuously followed up

The rapid development of autonomous driving technology has led to various entities in the industry chain launching their own safety solutions. There is an urgent need for a unified set of autonomous driving safety standards to regulate the safety mechanism design of each entity. In addition, relevant legislation also needs to be followed up in a timely manner, clarifying the responsible parties for safety accidents related to autonomous driving, and promoting all entities in the autonomous driving industry chain to fulfill their safety obligations towards products.

9. conclusion

Although significant progress has been made in the testing and evaluation technology of autonomous driving, the standard system for testing and evaluation is still incomplete. Existing testing tools and methods are difficult to meet the testing needs of L3 level and above autonomous driving vehicles. The development and application level of virtual simulation and digital twin technology is relatively low, and there are many shortcomings in realism, testing efficiency, and vehicle level testing capabilities.

(1) This article first analyzes the simulation testing technology of autonomous driving from two aspects: vehicle simulation and traffic flow simulation. With the aim of improving the authenticity of testing, the simulation testing technology of autonomous driving vehicles is analyzed from the aspects of vehicle dynamics, personification testing, computational requirements, and standard formats; Starting from key technologies such as background vehicle driving style model, background traffic flow model, traffic scene generation method, acceleration testing method, and virtual simulation cloud platform for autonomous driving testing, this paper analyzes the simulation testing technology of autonomous driving traffic flow. In the future, further breakthroughs are needed in high-precision and high fidelity modeling technologies for dynamics, sensors, and other elements. It is necessary to further study the theory of dangerous scene generation and accelerated testing, and form a complete high fidelity simulation software and high concurrency testing cloud platform.

(2) Hardware in the loop simulation testing has high testing accuracy and efficiency, and is a necessary part of autonomous driving simulation testing and evaluation. This article systematically reviews and summarizes three core technologies in hardware in the loop testing: digital twin testing technology, vehicle level system platform construction technology, and vehicle mounted sensor physical information generation technology. It analyzes the importance of interactivity, real-time performance, and authenticity in hardware in the loop testing.

(3) A comprehensive analysis was conducted on autonomous driving field testing from the aspects of closed field testing, open road testing, and highway testing system construction. In terms of closed site construction, most of the domestic test sites have been preliminarily built, and the testing capabilities are constantly improving. In the future, further research is needed on site twin online acceleration testing methods for vehicle cloud field collaboration, and the construction of virtual real fusion scenarios that meet the testing needs of autonomous vehicles.

(4) A review was conducted on the research status of intelligent evaluation methods for autonomous driving, and the integration methods of toolchains at home and abroad were sorted out. The current status of the evaluation standard system for autonomous driving testing was analyzed in detail. Introduced the definition, causal analysis, and defect classification of autonomous driving defects, and studied the current defect detection methods for autonomous driving.

Writing team

plan:Zhao Xiangmo¹Huifei²Xu Zhigang¹

Submission:Zhao Xiangmo¹Zhu Xichan³Huifei²Huang Lei⁴Sun Zhengliang⁵

Unified draft:Zhao Xiangmo¹Huifei²Xu Zhigang¹Jing Shoucai¹Mu Kenan²

compose:(In alphabetical order of surname strokes)

Ma Yining³Wang Zhen¹Fang Yukun¹The field⁶Tong Panpan³Xing Xingyu³Bi Xin³Zhu Shihao⁴Zhu Xichan³Zhu Yixin⁶，

Liu Rui⁷Liu Ming³Sun Jian⁶Sun Chitian⁸Ji Yi⁹Li Shijie¹Li Shanke¹Yang Yang¹⁰Yang Lan¹Wu Xinzheng³，

He Zeyu¹Yu Zhuoping³Cold stroke³Min Haigen¹Shen Yong³Zhang He⁶Zhang Fengjie⁸Zhang Yutan¹¹Chen Junyi³Chen Lingli⁴，

Yue Li Shengsa⁶Zhou Xiaojun³Cheng Jingjun¹Zhao Ning¹¹Zhao Yuyu¹Yuan Tian⁷Yuan Jiaqi¹Guo Chufan⁸Huang Lei⁴Chang Yutao⁹，

Peng Cheng¹Huifei²Jing Jun⁹Jing Shoucai¹Meng Haolan³Pei Shuanghong¹¹Xiong Lu³Xue Dingrui¹Mu Kenan²

Company:

1. School of Information Engineering, Chang'an University, Xi'an, Shaanxi 710064, China

2. School of Electronic and Control Engineering, Chang'an University, Xi'an, Shaanxi 710064, China

3. School of Automotive Engineering, Tongji University, Shanghai 201804

4. Shanghai International Automobile City (Group) Co., Ltd., Shanghai 201800

5. Institute of Traffic Management Science, Ministry of Public Security, Wuxi, Jiangsu 214151, China

6. Key Laboratory of Road and Traffic Engineering, Ministry of Education, Tongji University, Shanghai 201804

7. School of Automotive Engineering, Chang'an University, Xi'an, Shaanxi 710018, China

8. Tencent Cloud Computing (Beijing) Co., Ltd., Beijing 100084

9. Shandong Expressway Information Group Co., Ltd., Jinan, Shandong 250102

10. Beijing Baidu Netcom Technology Co., Ltd., Beijing 100084

11. Dongfeng Motor Group Co., Ltd. Technical Center, Wuhan 430056, Hubei

References(299)

References

[1]	GAO Feng, DUAN Jian-li, HAN Zai-dao, et al. Automatic virtual test technology for intelligent driving systems considering both coverage and efficiency[J]. IEEE Transactions on Vehicular Technology, 2020, 69(12): 14365-14376. doi: 10.1109/TVT.2020.3033565
[2]	DENG Xiao-feng, WANG Run-min, XU Zhi-gang, et al. Development status of intelligent networked automobile testing and demonstration base in China[J]. Auto Industry Research, 2019(1): 6-13. (in Chinese) doi: 10.3969/j.issn.1009-847X.2019.01.001
[3]	SUN Hang, LI Zhi-jun, ZHANG Lin-lin, et al. Research on the test and evaluation technique of real roads for automated driving vehicles based on OEDR and ODC[J]. Automotive Engineering, 2022, 44(6): 842-850. (in Chinese)
[4]	WANG Jun-min, STEIBER J, SURAMPUDI B. Autonomous ground vehicle control system for high-speed and safe operation[J]. International Journal of Vehicle Autonomous Systems, 2009, 7(1/2): 18-35. doi: 10.1504/IJVAS.2009.027965
[5]	PING E P, HUDHA K, JAMALUDDIN H. Hardware-in-the-loop simulation of automatic steering control for lanekeeping manoeuvre: outer-loop and inner-loop control design[J]. International Journal of Vehicle Safety, 2010, 5(1): 35-59. doi: 10.1504/IJVS.2010.035318
[6]	ZAKARIA M A, ZAMZURI H, MAMAT R, et al. A path tracking algorithm using future prediction control with spike detection for an autonomous vehicle robot[J]. International Journal of Advanced Robotic Systems, 2013, DOI: 10.5772/56658.
[7]	ELBANHAWI M, SIMIC M, JAZAR R. The role of path continuity in lateral vehicle control[J]. Procedia Computer Science, 2015, 60: 1289-1298. doi: 10.1016/j.procs.2015.08.194
[8]	RAFFO G V, GOMES G K, NORMEY-RICO J E, et al. A predictive controller for autonomous vehicle path tracking[J]. IEEE Transactions on Intelligent Transportation Systems, 2009, 10(1): 92-102. doi: 10.1109/TITS.2008.2011697
[9]	GUO Lie, GE Ping-shu, YANG Xiao-li, et al. Intelligent vehicle trajectory tracking based on neural networks sliding mode control[C]//IEEE. Proceedings 2014 International Conference on Informative and Cybernetics for Computational Social Systems (ICCSS). New York: IEEE, 2014: 57-62.
[10]	SCHIEHLEN W O. Computer generation of equations of motion[C]//Springer. Computer Aided Analysis and Optimization of Mechanical System Dynamics. Berlin: Springer, 1984: 183-215.
[11]	SCHIEHLEN W O, KREUZER E J. Symbolic computerized derivation of equations of motion[C]//Springer. Dynamics of Multibody Systems: Symposium Munich. Berlin: Springer, 1977: 290-305.
[12]	ORLANDEA N, CALAHAN D A, CHACE M A. A sparsity-oriented approach to the dynamic analysis and design of mechanical systems—part 2[J]. Journal of Engineering for Industry, 1977, 99(3): 780-784. doi: 10.1115/1.3439313
[13]	CHACE M A. Methods and experience in computer aided design of large-displacement mechanical systems[C]//Springer. Computer Aided Analysis and Optimization of Mechanical System Dynamics. Berlin: Springer, 1984: 233-259.
[14]	WANG Jing, YANG Zhi-gang, WANG Yong. A four-degree freedom dynamics model of automotive four-wheel-steering[J]. Journal of Shandong Jiaotong University, 2008, 16(1): 11-15. (in Chinese) doi: 10.3969/j.issn.1672-0032.2008.01.003
[15]	LIU Z, PAYRE G, BOURASSA P. Nonlinear oscillations and chaotic motions in a road vehicle system with driver steering control[J]. Nonlinear Dynamics, 1996, 9: 281-304. doi: 10.1007/BF01833746
[16]	BLAJER W. An effective solver for absolute variable formulation of multibody dynamics[J]. Computational Mechanics, 1995, 15(5): 460-472. doi: 10.1007/BF00350358
[17]	WANG Yang-yang, JIN Xiao-xiong, ZUO Shu-guang, et al. A research on vehicle dynamics simulation platform based on R-W method of multi-body dynamics[J]. Automotive Engineering, 2008, 30(1): 22-25. (in Chinese) doi: 10.3321/j.issn:1000-680X.2008.01.005
[18]	JIN Ting, GONG Yun-qiu, WEI Chun-yu. Vehicle dynamics modeling based on vortex[J]. Applied Mechanics and Materials, 2014, 624: 289-292. doi: 10.4028/www.scientific.net/AMM.624.289
[19]	HU Jun-qing. Study of the ride dynamics characteristics of the off-road vehicle with two degree of freedom articulated body[D]. Changchun: Jilin University, 2011. (in Chinese)
[20]	YANG Huai-guang. Research on modeling and simulation of mars rovers with active suspension system for variable moving conditions[D]. Harbin: Harbin Institute of Technology, 2020. (in Chinese)
[21]	WANG Jian-qiang, ZHANG Lei, ZHANG De-zhao, et al. An adaptive longitudinal driving assistance system based on driver characteristics[J]. IEEE Transactions on Intelligent Transportation Systems, 2013, 14(1): 1-12. doi: 10.1109/TITS.2012.2205143
[22]	ZHAO Xiao-cong, HE Ren, WANG Jian-qiang. How do drivers respond to driving risk during car-following? Risk-response driver model and its application in human-like longitudinal control[J]. Accident Analysis and Prevention, 2020, 148: 105783. doi: 10.1016/j.aap.2020.105783
[23]	WADA T, DOIS, TSURU N, et al. Characterization of expert drivers' last-second braking and its application to a collision avoidance system[J]. IEEE Transactions on Intelligent Transportation Systems, 2010, 11(2): 413-422. doi: 10.1109/TITS.2010.2043672
[24]	PAPATHANASOPOULOU V, ANTONIOU C. Towards data-driven car-following models[J]. Transportation Research Part C: Emerging Technologies, 2015, 55: 496-509. doi: 10.1016/j.trc.2015.02.016
[25]	YU Rong-jie, ZHANG Rui-ci, AI Hao-an, et al. Personalized driving assistance algorithms: case study of federated learning based forward collision warning[J]. Accident Analysis and Prevention, 2022, 168: 106609. doi: 10.1016/j.aap.2022.106609
[26]	KHODAYARI A, GHAFFARI A, KAZEMI R, et al. A modified car-following model based on a neural network model of the human driver effects[J]. IEEE Transactions on Systems, Man, and Cybernetics—Part A: Systems and Humans, 2012, 42(6): 1440-1449. doi: 10.1109/TSMCA.2012.2192262
[27]	LEFÈVRE S, CARVALHO A, BORRELLI F. A learning-based framework for velocity control in autonomous driving[J]. IEEE Transactions on Automation Science and Engineering, 2016, 13(1): 32-42. doi: 10.1109/TASE.2015.2498192
[28]	WANG Wen-shuo, ZHAO Ding, HAN Wei, et al. A learning-based approach for lane departure warning systems with a personalized driver model[J]. IEEE Transactions on Vehicular Technology, 2018, 67(10): 9145-9157. doi: 10.1109/TVT.2018.2854406
[29]	ALI Y, HUSSAIN F, BLIEMER M C J, et al. Predicting and explaining lane-changing behaviour using machine learning: a comparative study[J]. Transportation Research Part C: Emerging Technologies, 2022, 145: 103931. doi: 10.1016/j.trc.2022.103931
[30]	BUTAKOV V A, IOANNOU P. Personalized driver/vehicle lane change models for ADAS[J]. IEEE Transactions on Vehicular Technology, 2014, 64(10): 4422-4431.
[31]	ZHANG Yi-fan, CHEN Xin-hong, WANG Jian-ping, et al. A generative car-following model conditioned on driving styles[J]. Transportation Research Part C: Emerging Technologies, 2022, 145: 103926. doi: 10.1016/j.trc.2022.103926
[32]	CHEN Zhi-gui, WANG Xue-song, GUO Qi-ming, et al. Towards human-like speed control in autonomous vehicles: a mountainous freeway case[J]. Accident Analysis and Prevention, 2022, 166: 106566. doi: 10.1016/j.aap.2022.106566
[33]	HUANG Jing, JI Zhong-xun, PENG Xiao-yan, et al. Driving style adaptive lane-changing trajectory planning and control[J]. China Journal of Highway and Transport, 2019, 32(6): 226-239, 247. (in Chinese)
[34]	REN Guo-qing, ZHANG Yong, LIU Hao, et al. A new lane-changing model with consideration of driving style[J]. International Journal of Intelligent Transportation Systems Research, 2019, 17(3): 181-189. doi: 10.1007/s13177-019-00180-7
[35]	ZHANG Yi-fan, XU Qian, WANG Jian-ping, et al. A learning-based discretionary lane-change decision-making model with driving style awareness[J]. IEEE Transactions on Intelligent Transportation Systems, 2023, 24(1): 68-78. doi: 10.1109/TITS.2022.3217673
[36]	DESJARDINS C, CHAIB-DRAA B. Cooperative adaptive cruise control: a reinforcement learning approach[J]. IEEE Transactions on Intelligent Transportation Systems, 2011, 12(4): 1248-1260. doi: 10.1109/TITS.2011.2157145
[37]	ZHANG Yi, SUN Ping, YIN Yu-han, et al. Human-like autonomous vehicle speed control by deep reinforcement learning with double Q-learning[C]//IEEE. 2018 IEEE Intelligent Vehicles Symposium (Ⅳ). New York: IEEE, 2018: 1251-1256.
[38]	ZHU Bing, JIANG Yuan-de, ZHAO Jian, et al. A car-following control algorithm based on deep reinforcement learning[J]. China Journal of Highway and Transport, 2019, 32(6): 53-60. (in Chinese)
[39]	LI Guo-fan, YANG Liang, LI Shen, et al. Human-like decision making of artificial drivers in intelligent transportation systems: an end-to-end driving behavior prediction approach[J]. IEEE Intelligent Transportation Systems Magazine, 2022, 14(6): 188-205. doi: 10.1109/MITS.2021.3085986
[40]	HAN Lei, WU Lei, LIANG Fu-jian, et al. A novel end-to-end model for steering behavior prediction of autonomous ego-vehicles using spatial and temporal attention mechanism[J]. Neurocomputing, 2022, 490: 295-311. doi: 10.1016/j.neucom.2021.11.093
[41]	WANG Xin-peng, CHEN Zhi-jun, WU Chao-zhong, et al. A method of automatic driving decision for smart car considering driving style[J]. Journal of Transport Information and Safety, 2020, 38(2): 37-46. (in Chinese)
[42]	ZHU Mei-xin, WANG Xue-song, WANG Yin-hai. Human-like autonomous car-following model with deep reinforcement learning[J]. Transportation Research Part C: Emerging Technologies, 2018, 97: 348-368. doi: 10.1016/j.trc.2018.10.024
[43]	LI Dian-zhao, OKHRIN O. Modified DDPG car-following model with a real-world human driving experience with CARLA simulator[J]. Transportation Research Part C: Emerging Technologies, 2023, 147: 103987. doi: 10.1016/j.trc.2022.103987
[44]	JIAO Xin-yu, YANG Dian-ge, JIANG Kun, et al. Driving trajectory curvature prediction of vehicle on highway based on end-to-end learning mechanism[J]. Automotive Engineering, 2018, 40(12): 1494-1499. (in Chinese)
[45]	CHEN Long, HU Xue-min, TIAN Wei, et al. Parallel planning: a new motion planning framework for autonomous driving[J]. IEEE/CAA Journal of Automatica Sinica, 2019, 6(1): 236-246. doi: 10.1109/JAS.2018.7511186
[46]	ARADI S. Survey of deep reinforcement learning for motion planning of autonomous vehicles[J]. IEEE Transactions on Intelligent Transportation Systems, 2020, 23(2): 740-759.
[47]	KIRAN B R, SOBH I, TALPAERT V, et al. Deep reinforcement learning for autonomous driving: a survey[J]. IEEE Transactions on Intelligent Transportation Systems, 2022, 23(6): 4909-4926. doi: 10.1109/TITS.2021.3054625
[48]	WU Yuan-qing, LIAO Si-qin, LIU Xiang, et al. Deep reinforcement learning on autonomous driving policy with auxiliary critic network[J]. IEEE Transactions on Neural Networks and Learning Systems, 2023, 34(7): 3680-3690. doi: 10.1109/TNNLS.2021.3116063
[49]	GUO Hong-yu, KEYVAN-EKBATANI M, XIE Kun. Lane change detection and prediction using real-world connected vehicle data[J]. Transportation Research Part C: Emerging Technologies, 2022, 142: 103785. doi: 10.1016/j.trc.2022.103785
[50]	LI Guo-fa, QIU Yi-fan, YANG Yi-fan, et al. Lane change strategies for autonomous vehicles: a deep reinforcement learning approach based on transformer[J]. IEEE Transactions on Intelligent Vehicles, 2023, 8(3): 2197-2211. doi: 10.1109/TIV.2022.3227921
[51]	CHEN Qi, TANG Si-hai, YANG Qing, et al. Cooper: cooperative perception for connected autonomous vehicles based on 3D point clouds[C]//IEEE. 2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS). New York: IEEE, 2019: 514-524.
[52]	WANG Chao-jie, GONG Si-yuan, ZHOU An-ye, et al. Cooperative adaptive cruise control for connected autonomous vehicles by factoring communication-related constraints[J]. Transportation Research Procedia, 2019, 38: 242-262. doi: 10.1016/j.trpro.2019.05.014
[53]	ARTHURS P, GILLAM L, KRAUSE P, et al. A taxonomy and survey of edge cloud computing for intelligent transportation systems and connected vehicles[J]. IEEE Transactions on Intelligent Transportation Systems, 2022, 23(7): 6206-6221. doi: 10.1109/TITS.2021.3084396
[54]	WANG Hao-xin, LIU Ting-ting, KIM B, et al. Architectural design alternatives based on cloud/edge/fog computing for connected vehicles[J]. IEEE Communications Surveys and Tutorials, 2020, 22(4): 2349-2377. doi: 10.1109/COMST.2020.3020854
[55]	LIANG Si-yuan, WU Hao, ZHEN Li, et al. Edge YOLO: real-time intelligent object detection system based on edge-cloud cooperation in autonomous vehicles[J]. IEEE Transactions on Intelligent Transportation Systems, 2022, 23(12): 25345-25360. doi: 10.1109/TITS.2022.3158253
[56]	WHAIDUZZAMAN M, SOOKHAK M, GANI A, et al. A survey on vehicular cloud computing[J]. Journal of Network and Computer Applications, 2014, 40: 325-344. doi: 10.1016/j.jnca.2013.08.004
[57]	HOU Yun-fei, SELIMAN S M S, WANG En-shu, et al. Cooperative and integrated vehicle and intersection control for energy efficiency (CIVIC-E²)[J]. IEEE Transactions on Intelligent Transportation Systems, 2018, 19(7): 2325-2337. doi: 10.1109/TITS.2017.2785288
[58]	KUMAR N, RAHMAN S S, DHAKAD N. Fuzzy inference enabled deep reinforcement learning-based traffic light control for intelligent transportation system[J]. IEEE Transactions on Intelligent Transportation Systems, 2021, 22(8): 4919-4928. doi: 10.1109/TITS.2020.2984033
[59]	YUAN Jing, ZHENG Yu, XIE Xing, et al. Driving with knowledge from the physical world[C]//ACM. Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. New York: ACM, 2011: 316-324.
[60]	XU Wen-chao, ZHOU Hai-bo, CHENG Nan, et al. Internet of vehicles in big data era[J]. IEEE/CAA Journal of Automatica Sinica, 2017, 5(1): 19-35.
[61]	KAIWARTYA O, ABDULLAH A H, CAO Yue, et al. Internet of vehicles: motivation, layered architecture, network model, challenges, and future aspects[J]. IEEE Access, 2016, 4: 5356-5373. doi: 10.1109/ACCESS.2016.2603219
[62]	SASAKI K, SUZUKI N, MAKIDO S, et al. Vehicle control system coordinated between cloud and mobile edge computing[C]// IEEE. 2016 55th Annual Conference of the Society of Instrument and Control Engineers of Japan (SICE). New York: IEEE, 2016: 1122-1127.
[63]	WAN Jia-fu, ZHANG Da-qiang, ZHAO Sheng-jie, et al. Context-aware vehicular cyber-physical systems with cloud support: architecture, challenges, and solutions[J]. IEEE Communications Magazine, 2014, 52(8): 106-113. doi: 10.1109/MCOM.2014.6871677
[64]	ZHANG Yang, PENG Guo-xiong, YANG Xiao-guang. Present situation and trend of ITS development in Japan[J]. Journal of China and Foreign Highway, 2003, 23(3): 8-10. (in Chinese) doi: 10.3969/j.issn.1671-2579.2003.03.003
[65]	YUAN Jun. Design of driving behavior scoring model based on data mining technology[J]. Information Recording Materials, 2020, 21(10): 153-155. (in Chinese)
[66]	ZHANG Zheng. The analysis and design of driving safety guards based on the score of driving behavior in vehicle networking[D]. Nanjing: Nanjing University of Posts and Telecommunications, 2017. (in Chinese)
[67]	ZHU Shuang. Research on the automobile insurance ratemaking mode and method based on UBI in the internet of vehicle[D]. Beijing: Beijing Jiaotong University, 2015. (in Chinese)
[68]	JIANG Xian-tao, YU F R, SONG Tian, et al. Intelligent resource allocation for video analytics in blockchain-enabled internet of autonomous vehicles with edge computing[J]. IEEE Internet of Things Journal, 2022, 9(16): 14260-14272. doi: 10.1109/JIOT.2020.3026354
[69]	ZHU Feng-hua, LYU Yi-sheng, CHEN Yuan-yuan, et al. Parallel transportation systems: toward IoT-enabled smart urban traffic control and management[J]. IEEE Transactions on Intelligent Transportation Systems, 2020, 21(10): 4063-4071. doi: 10.1109/TITS.2019.2934991
[70]	WANG Zi-ran, LIAO Xi-shun, ZHAO Xuan-peng, et al. A digital twin paradigm: vehicle-to-cloud based advanced driver assistance systems[C]//IEEE. 2020 IEEE 91st Vehicular Technology Conference (VTC2020-Spring). New York: IEEE, 2020: 1-6.
[71]	ASAM. Project types: learn which project types are available at ASAM[EB/OL]. (2023-03-21)[2023-04-01]. https://www.asam.net/active-projects/project-types.
[72]	ASAM. Development process: learn how ASAM standards are developed[EB/OL]. (2023-03-21)[2023-04-01]. https://www.asam.net/active-projects/development-process.
[73]	ZHOU Bo-lin, ZHANG Zong-shi, CHEN Chen. ASAM OpenX and scenario based simulation test for autonomous driving[J]. Standard Science, 2021(S1): 110-122. (in Chinese)
[74]	C-ASAM. The type of ASAM standard project[EB/OL]. (2022-03-29)[2022-11-03]. http://www.c-asam.net/?s=news-show-id-2.html. (in Chinese)
[75]	LU Guang-quan, CHENG Bo, WANG Yun-peng, et al. A car-following model based on quantified homeostatic risk perception[J]. Mathematical Problems in Engineering, 2013, 2013: 408756.
[76]	YANG H H, PENG H. Development of an errorable car-following driver model[J]. Vehicle System Dynamics, 2010, 48(6): 751-773. doi: 10.1080/00423110903128524
[77]	LINDORFER M, MECKLENBRÄUKER C F, OSTERMAYER G. Modeling the imperfect driver: incorporating human factors in a microscopic traffic model[J]. IEEE Transactions on Intelligent Transportation Systems, 2018, 19(9): 2856-2870. doi: 10.1109/TITS.2017.2765694
[78]	CAI Jun-yu, JIANG Hao-bin, WANG Jun-yan. Implementation of the human-like lane changing driver model based on Bi-LSTM[J]. Discrete Dynamics in Nature and Society, 2022, 2022: 9934292.
[79]	JULKA S, SOWRIRAJAN V, SCHLOETTERER J, et al. Conditional generative adversarial networks for speed control in trajectory simulation[C]//Springer. 7th International Conference on Machine Learning, Optimization, and Data Science. Berlin: Springer, 2021: 436-450.
[80]	LIU Jian-bang, MAO Xin-yu, FANG Yu-qi, et al. A survey on deep-learning approaches for vehicle trajectory prediction in autonomous driving[C]//IEEE. 2021 IEEE International Conference on Robotics and Biomimetics (ROBIO). New York: IEEE, 2021: 978-985.
[81]	LIANG Ming, YANG Bin, HU Rui, et al. Learning lane graph representations for motion forecasting[C]//Springer. 16th European Conference on Computer Vision. Berlin: Springer, 2020: 541-556.
[82]	VARADARAJAN B, HEFNY A, SRIVASTAVA A, et al. Multipath + + : efficient information fusion and trajectory aggregation for behavior prediction[C]//ICRA. 2022 International Conference on Robotics and Automation (ICRA). New York: ICRA, 2022: 7814-7821.
[83]	ALBABA B M, YILDIZ Y. Driver modeling through deep reinforcement learning and behavioral game theory[J]. IEEE Transactions on Control Systems Technology, 2022, 30(2): 885-892. doi: 10.1109/TCST.2021.3075557
[84]	SUN Ruo-yu, HU Shao-chi, ZHAO Hui-jing, et al. Human-like highway trajectory modeling based on inverse reinforcement learning[C]//IEEE. 2019 IEEE Intelligent Transportation Systems Conference (ITSC). New York: IEEE, 2019: 1482-1489.
[85]	HUANG Zhi-yu, WU Jing-da, LYU Chen. Driving behavior modeling using naturalistic human driving data with inverse reinforcement learning[J]. IEEE Transactions on Intelligent Transportation Systems, 2022, 23(8): 10239-10251. doi: 10.1109/TITS.2021.3088935
[86]	SUN Jian, ZHANG He, ZHOU Hua-jun, et al. Scenario-based test automation for highly automated vehicles: a review and paving the way for systematic safety assurance[J]. IEEE Transactions on Intelligent Transportation Systems, 2022, 23(9): 14088-14103. doi: 10.1109/TITS.2021.3136353
[87]	ALTHOFF M, KOSCHI M, MANZINGER S. Common road: composable benchmarks for motion planning on roads[C]// IEEE. 2017 IEEE Intelligent Vehicles Symposium (Ⅳ). New York: IEEE, 2017: 719-726.
[88]	SHAH S, DEY D, LOVETT C, et al. Airsim: high-fidelity visual and physical simulation for autonomous vehicles[C]//Springer. Field and Service Robotics: Results of the 11th International Conference. Berlin: Springer, 2018: 621-635.
[89]	CARLA. Carla simulator[EB/OL]. (2022-09-12)[2022-11-03]. https://carla.readthedocs.io/en/latest.
[90]	FAN Hao-yang, ZHU Fan, LIU Chang-chun, et al. Baidu apollo em motion planner[J]. arXiv, 2018, DOI: 1807.08048.
[91]	BENEKOHAL R F, TREITERER J. CARSIM. Car-following model for simulation of traffic in normal and stop-and-go conditions[J]. Transportation Research Record, 1988, 1194: 99-111.
[92]	Siemens. Simcenter prescan software[EB/OL]. (2022-09-21) [2022-11-03]. https://plm.sw.siemens.com/en-US/simcenter/autonomous-vehicle-solutions/prescan/.
[93]	QI Xiao, NI Ying, XU Yi-ming, et al. Autonomous vehicles' car-following drivability evaluation based on driving behavior spectrum reference model[J]. Transportation Research Record: Journal of the Transportation Research Board, 2021, 2675(7): 129-141. doi: 10.1177/0361198121994857
[94]	SONG Zhi-jin, WANG Hui-zi, SUN Jian, et al. Experimental findings with VISSIM and TransModeler for evaluating environmental and safety impacts using micro-simulations[J]. Transportation Research Record: Journal of the Transportation Research Board, 2020, 2674(8): 566-580. doi: 10.1177/0361198120925077
[95]	MA Zi-an, SUN Jian, WANG Yun-peng. A two-dimensional simulation model for modelling turning vehicles at mixed-flow intersections[J]. Transportation Research Part C: Emerging Technologies, 2017, 75: 103-119. doi: 10.1016/j.trc.2016.12.005
[96]	MA Zi-an, XIE Jian-bo, QI Xiao, et al. Two-dimensional simulation of turning behavior in potential conflict area of mixed-flow intersections[J]. Computer-Aided Civil and Infrastructure Engineering, 2017, 32(5): 412-428. doi: 10.1111/mice.12266
[97]	XU Yi-ming, MA Zi-an, SUN Jian. Simulation of turning vehicles' behaviors at mixed-flow intersections based on potential field theory[J]. Transportmetrica B: Transport Dynamics, 2019, 7(1): 498-518. doi: 10.1080/21680566.2018.1447407
[98]	SUN Jian, LIU Han, MA Zi-an. Modelling and simulation of highly mixed traffic flow on two-lane two-way urban streets[J]. Simulation Modelling Practice and Theory, 2019, 95: 16-35. doi: 10.1016/j.simpat.2019.04.005
[99]	BERGAMINI L, YE Ya-wei, SCHEEL O, et al. SimNet: learning reactive self-driving simulations from real-world observations[C]//IEEE. 2021 IEEE International Conference on Robotics and Automation (ICRA). New York: IEEE, 2021: 5119-5125.
[100]	YAN Xin-tao, ZOU Zheng-xia, FENG Shuo, et al. Learning naturalistic driving environment with statistical realism[J]. Nature Communications, 2023, 14: 2037. doi: 10.1038/s41467-023-37677-5
[101]	ZHAO Ding, HUANG Xia-nan, PENG Hu-ei, et al. Accelerated evaluation of automated vehicles in car-following maneuvers[J]. IEEE Transactions on Intelligent Transportation Systems, 2018, 19(3): 733-744. doi: 10.1109/TITS.2017.2701846
[102]	FENG Shuo, FENG Yi-heng, YU Chun-hui, et al. Testing scenario library generation for connected and automated vehicles, part Ⅰ: methodology[J]. IEEE Transactions on Intelligent Transportation Systems, 2021, 22(3): 1573-1582. doi: 10.1109/TITS.2020.2972211
[103]	SUN Jian, ZHOU Hua-jun, XI Hao-chen, et al. Adaptive design of experiments for safety evaluation of automated vehicles[J]. IEEE Transactions on Intelligent Transportation Systems, 2022, 23(9): 14497-14508. doi: 10.1109/TITS.2021.3130040
[104]	DING Wen-hao, CHEN Bai-ming, LI Bo, et al. Multimodal safety-critical scenarios generation for decision-making algorithms evaluation[J]. IEEE Robotics and Automation Letters, 2021, 6(2): 1551-1558. doi: 10.1109/LRA.2021.3058873
[105]	DENG Wei-wen, LI Jiang-kun, REN Bing-tao, et al. A survey on automatic simulation scenario generation methods for autonomous driving[J]. China Journal of Highway and Transport, 2022, 35(1): 316-333. (in Chinese) doi: 10.3969/j.issn.1001-7372.2022.01.027
[106]	CHEN Ji-qing, SHU Xiao-xiong, LAN Feng-chong, et al. Construction of autonomous vehicles test scenarios with typical dangerous accident characteristics[J]. Journal of South China University of Technology (Natural Science Edition), 2021, 49(5): 1-8. (in Chinese)
[107]	WANG Run-min, ZHU Yu, ZHAO Xiang-mo, et al. Research progress on test scenario of autonomous driving[J]. Journal of Traffic and Transportation Engineering, 2021, 21(2): 21-37. (in Chinese) doi: 10.19818/j.cnki.1671-1637.2021.02.003
[108]	ZHU Bing, ZHANG Pei-xing, ZHAO Jian. Clustering evaluation method of autonomous driving safety for multi-dimensional logical scenario[J]. Automotive Engineering, 2020, 42(11): 1458-1463, 1505. (in Chinese)
[109]	MENZEL T, BAGSCHIK G, MAURER M. Scenarios for development, test and validation of automated vehicles[C]//IEEE. 2018 IEEE Intelligent Vehicles Symposium (Ⅳ). New York: IEEE, 2018: 1821-1827.
[110]	DING Wen-hao, LI Bo, EUN K J, et al. Semantically controllable scene generation with guidance of explicit knowledge[J]. arXiv, 2021, DOI: 2106.04066.
[111]	JESENSKI S, STELLET J E, SCHIEGG F, et al. Generation of scenes in intersections for the validation of highly automated driving functions[C]//IEEE. 2019 IEEE Intelligent Vehicles Symposium (Ⅳ). New York: IEEE, 2019: 502-509.
[112]	ZHOU Wen-shuai, ZHU Yu, ZHAO Xiang-mo, et al. Vehicle cut-in test case generation methods for testing of autonomous driving on highway[J]. Automobile Technology, 2021(1): 11-18. (in Chinese)
[113]	ZHU Yu, ZHAO Xiang-mo, XU Zhi-gang, et al. Automatic generation algorithm of lane-change virtual test scenario on highways for automated vehicles using Monte Carlo simulation[J]. China Journal of Highway and Transport, 2022, 35(3): 89-100. (in Chinese)
[114]	DING Wen-hao, CHEN Bai-ming, XU Min-jun, et al. Learning to collide: an adaptive safety-critical scenarios generating method[C]//IEEE. 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). New York: IEEE, 2020: 2243-2250.
[115]	CHOI S, KIM J, YEO H. TrajGAIL: generating urban vehicle trajectories using generative adversarial imitation learning[J]. Transportation Research Part C: Emerging Technologies, 2021, 128: 103091. doi: 10.1016/j.trc.2021.103091
[116]	ZHAO Xiang-mo, ZHAO Yu-yu, JING Shou-cai, et al. Generalization generation of hazardous lane-changing scenarios for automated vehicle testing[J]. Acta Automatica Sinica, 2023, 49(10): 2211-2223. (in Chinese)
[117]	YU Rong-jie, TIAN Ye, SUN Jian. Highly automated vehicle virtual testing: a review of recent developments and research frontiers[J]. China Journal of Highway and Transport, 2020, 33(11): 125-138. (in Chinese) doi: 10.3969/j.issn.1001-7372.2020.11.012
[118]	ZHU Bing, FAN Tian-xin, ZHAO Jian, et al. Accelerate test method of automated driving system based on hazardous boundary search[J]. Journal of Jilin University (Engineering and Technology Edition), 2023, 53(3): 704-712. (in Chinese)
[119]	SAEED A, AB HAMID S H, MUSTAFA M B. The experimental applications of search-based techniques for model-based testing: taxonomy and systematic literature review[J]. Applied Soft Computing, 2016, 49: 1094-1117. doi: 10.1016/j.asoc.2016.08.030
[120]	MULLINS G E, STANKIEWICZ P G, HAWTHORNE R C, et al. Adaptive generation of challenging scenarios for testing and evaluation of autonomous vehicles[J]. Journal of Systems and Software, 2018, 137: 197-215. doi: 10.1016/j.jss.2017.10.031
[121]	XU Yi-ming, Zou Ya-jie, SUN Jian. Accelerated testing for automated vehicles safety evaluation in cut-in scenarios based on importance sampling, genetic algorithm and simulation applications[J]. Journal of Intelligent and Connected Vehicles, 2018, 1(1): 28-38. doi: 10.1108/JICV-01-2018-0002
[122]	ZHAO Ding, LAM H, PENG Hu-ei, et al. Accelerated evaluation of automated vehicles safety in lane-change scenarios based on importance sampling techniques[J]. IEEE Transactions on Intelligent Transportation Systems, 2017, 18(3): 595-607. doi: 10.1109/TITS.2016.2582208
[123]	HUANG Zhi-yuan, LAM H, LEBLANC D J, et al. Accelerated evaluation of automated vehicles using piecewise mixture models[J]. IEEE Transactions on Intelligent Transportation Systems, 2018, 19(9): 2845-2855. doi: 10.1109/TITS.2017.2766172
[124]	HUANG Zhi-yuan, LAM H, ZHAO Ding. An accelerated testing approach for automated vehicles with background traffic described by joint distributions[C]//IEEE. 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC). New York: IEEE, 2017: 933-938.
[125]	ZHANG He, SUN Jian, TIAN Ye. Accelerated testing for highly automated vehicles: a combined method based on importance sampling and normalizing flows[C]//IEEE. 2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC). New York: IEEE, 2022: 574-579.
[126]	WANG Xin-peng, PENG Hu-ei, ZHANG Song-an, et al. An interaction-aware evaluation method for highly automated vehicles[C]//IEEE. 2021 IEEE International Intelligent Transportation Systems Conference (ITSC). New York: IEEE, 2021: 394-401.
[127]	TUNCALI C E, FAINEKOS G. Rapidly-exploring random trees for testing automated vehicles[C]//IEEE. 2019 IEEE Intelligent Transportation Systems Conference (ITSC). NewYork: IEEE, 2019: 661-666.
[128]	KIM Y, TAY S, GUANETTI J, et al. Hardware-in-the-loop for connected automated vehicles testing in real traffic[J]. arXiv, 2019, DOI: 1907.09052.
[129]	FENG Shuo, YAN Xin-tao, SUN Hao-wei, et al. Intelligent driving intelligence test for autonomous vehicles with naturalistic and adversarial environment[J]. Nature Communications, 2021, 12: 748. doi: 10.1038/s41467-021-21007-8
[130]	ROCKLAGE E, KRAFT H, KARATAS A, et al. Automated scenario generation for regression testing of autonomous vehicles[C]//IEEE. 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC). New York: IEEE, 2017: 476-483.
[131]	AMERSBACH C, WINNER H. Defining required and feasible test coverage for scenario-based validation of highly automated vehicles[C]//IEEE. 2019 IEEE Intelligent Transportation Systems Conference (ITSC). New York: IEEE, 2019: 425-430.
[132]	SHAFTO M, CONROY M, DOYLE R, et al. Modeling, simulation, information technology and processing roadmap[J]. National Aeronautics and Space Administration, 2010, 11: 1-32.
[133]	LIU Da-tong, GUO Kai, WANG Ben-kuan, et al. Summary and perspective survey on digital twin technology[J]. Chinese Journal of Scientific Instrument, 2018, 39(11): 1-10. (in Chinese)
[134]	QI Qing-lin, TAO Fei. Digital twin and big data towards smart manufacturing and industry 4.0: 360 degree comparison[J]. IEEE Access, 2018, 6: 3585-3593. doi: 10.1109/ACCESS.2018.2793265
[135]	LIAO Xi-shun, WANG Zi-ran, ZHAO Xuan-peng, et al. Cooperative ramp merging design and field implementation: a digital twin approach based on vehicle-to-cloud communication[J]. IEEE Transactions on Intelligent Transportation Systems, 2022, 23(5): 4490-4500. doi: 10.1109/TITS.2020.3045123
[136]	SCHROEDER G, STEINMETZ C, PEREIRA C E, et al. Visualising the digital twin using web services and augmented reality[C]//IEEE. 2016 IEEE 14th International Conference on Industrial Informatics (INDIN). New York: IEEE, 2016: 522-527.
[137]	WANG Zi-ran, WU Guo-yuan, BORIBOONSOMSIN K, et al. Cooperative ramp merging system: agent-based modeling and simulation using game engine[J]. SAE International Journal of Connected and Automated Vehicles, 2019, 2(2): 115-128.
[138]	CHEN Qi, WANG Xu-gang, YANG Jing, et al. Trajectory-following guidance based on a virtual target and an angle constraint[J]. Aerospace Science and Technology, 2019, 87: 448-458. doi: 10.1016/j.ast.2019.02.034
[139]	YANG Chun-ying, DONG Jiang-hong, XU Qing, et al. Multi-vehicle experiment platform: a digital twin realization method[C]//IEEE. 2022 IEEE/SICE International Symposium on System Integration (SⅡ). New York: IEEE, 2022: 705-711.
[140]	NIAZ A, SHOUKAT M U, JIA Yan-bing, et al. Autonomous driving test method based on digital twin: a survey[C]// IEEE. 2021 International Conference on Computing, Electronic and Electrical Engineering (ICE Cube). New York: IEEE, 2021: 1-7.
[141]	HETZER D, MUEHLEISEN M, KOUSARIDAS A, et al. 5G connected and automated driving: use cases and technologies in cross-border environments[C]//IEEE. 2019 European Conference on Networks and Communications (EuCNC). New York: IEEE, 2019: 78-82.
[142]	ZHAO Ding, PENG Hu-ei. From the lab to the street: Solving the challenge of accelerating automated vehicle testing[J]. arXiv, 2017, DOI: 1707.04792.
[143]	HUANG Wu-ling, WANG Kun-feng, LYU Yi-sheng, et al. Autonomous vehicles testing methods review[C]//IEEE. 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC). New York: IEEE, 2016: 163-168.
[144]	GE Yu-ming, WANG Yang, YU Run-dong, et al. Demo: research on test method of autonomous driving based on digital twin[C]//IEEE. 2019 IEEE Vehicular Networking Conference (VNC). New York: IEEE, 2019: 1-2.
[145]	SZALAY Z, FICZERE D, TIHANYI V, et al. 5G-enabled autonomous driving demonstration with a V2X scenario-in-the-loop approach[J]. Sensors (Basel), 2020, 20(24): 7344. doi: 10.3390/s20247344
[146]	LI Li, WANG Xiao, WANG Kun-feng, et al. Parallel testing of vehicle intelligence via virtual-real interaction[J]. Science Robotics, 2019, 4(28): eaaw4106. doi: 10.1126/scirobotics.aaw4106
[147]	VERHOEFF L, VERBURG D J, LUPKER H A, et al. VEHIL: a full-scale test methodology for intelligent transport systems, vehicles and subsystems[C]//IEEE. Proceedings of the IEEE Intelligent Vehicles Symposium 2000. New York: IEEE, 2002: 369-375.
[148]	BOCK T, SIEDERSBERGER K H, MAURER M. Vehicle in the loop-augmented reality application for collision mitigation systems[C]//IEEE. Fourth IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR'05). New York: IEEE, 2005: 1-9.
[149]	MIQUET C. New test method for reproducible real-time tests of ADAS ECUs: "Vehicle-in-the-Loop" connects real-world vehicles with the virtual world[C]//Springer. 5th International Munich Chassis Symposium 2014. Berlin: Springer, 2014: 575-589.
[150]	PFEFFER R, LEICHSENRING T. Continuous development of highly automated driving functions with vehicle-in-the-loop using the example of Euro NCAP scenarios[C]//Springer. Simulation and Testing for Vehicle Technology: 7th Conference. Berlin: Springer, 2016: 33-42.
[151]	PARK C, CHUNG S, LEE H. Vehicle-in-the-loop in global coordinates for advanced driver assistance system[J]. Applied Sciences, 2020, 10(8): 2645. doi: 10.3390/app10082645
[152]	FAYAZI S A, VAHIDI A, LUCKOW A. A Vehicle-in-the-Loop (VIL) verification of an all-autonomous intersection control scheme[J]. Transportation Research Part C: Emerging Technologies, 2019, 107: 193-210. doi: 10.1016/j.trc.2019.07.027
[153]	ITIC. International Transportation Innovation Center[EB/OL]. (2013-09-20)[2022-11-03]. http://www.itic-sc.com.
[154]	TETTAMANTI T, SZALAI M, VASS S, et al. Vehicle-in-the-loop test environment for autonomous driving with microscopic traffic simulation[C]//IEEE. 2018 IEEE International Conference on Vehicular Electronics and Safety (ICVES). New York: IEEE, 2018: 1-6.
[155]	GRIGGS W, ORDÓNEZ-HURTADO R, RUSSO G, et al. Control Strategies for Advanced Driver Assistance Systems and Autonomous Driving Functions: Development, Testing and Verification[M]. Berlin: Springer, 2018.
[156]	SOLMAZ S, HOLZINGER F, MISCHINGER M, et al. Towards Connected and Autonomous Vehicle Highways: Technical, Security and Social Challenges[M]. Berlin: Springer, 2020.
[157]	ARD T, GUO Long-xiang, DOLLAR R A, et al. Energy and flow effects of optimal automated driving in mixed traffic: vehicle-in-the-loop experimental results[J]. Transportation Research Part C: Emerging Technologies, 2021, 130: 103168. doi: 10.1016/j.trc.2021.103168
[158]	GIETELINK O, PLOEG J, DE SCHUTTER B, et al. VEHIL: a test facility for validation of fault management systems for advanced driver assistance systems[J]. IFAC Proceedings Volumes, 2004, 37(22): 397-402. doi: 10.1016/S1474-6670(17)30376-2
[159]	ALBERS A, DVSER T. A new process for configuration and application of complex validation environments using the example of vehicle-in-the-loop at the roller test bench[C]//ASME. Proceedings of the ASME 2010 International Mechanical Engineering Congress and Exposition. New York: ASME, 2010: 807-816.
[160]	GALKO C, ROSSI R, SAVATIER X. Vehicle-hardware-in-the-loop system for ADAS prototyping and validation[C]//IEEE. 2014 International Conference on Embedded Computer Systems: Architectures, Modeling, and Simulation (SAMOS XIV). New York: IEEE, 2014: 329-334.
[161]	VERBURG D J, VAN DER KNAAP A C M, PLOEG J. VEHIL: developing and testing intelligent vehicles[C]//IEEE. 2002 Intelligent Vehicle Symposium. New York: IEEE, 2002: 537-544.
[162]	MITSCHKE M, WALLENTOWITZ H. Dynamik der Kraftfahrzeuge[M]. Berlin: Springer, 1972.
[163]	PACEJKA H B, BESSELINK I J M. Magic formula tyre model with transient properties[J]. Vehicle System Dynamics, 1997, 27(S1): 234-249.
[164]	Dürr Group. X-road-curve dynamically tests autonomous cars[EB/OL]. (2018-04-11)[2022-11-03]. https://www.durr-group.com/en/duerrmore/autonomous-driving.
[165]	SOLMAZ S, RUDIGIER M, MISCHINGER M. A vehicle-in-the-loop methodology for evaluating automated driving functions in virtual traffic[C]//IEEE. 2020 IEEE Intelligent Vehicles Symposium (Ⅳ). New York: IEEE, 2020: 1465-1471.
[166]	ZHAO Xiang-mo, CHENG Jing-jun, XU Zhi-gang, et al. An indoor rapid-testing platform for autonomous vehicle based on vehicle-in-the-loop simulation[J]. China Journal of Highway and Transport, 2019, 32(6): 124-136. (in Chinese)
[167]	DIEWALD A, KURZ C, KANNAN P V, et al. Radar target simulation for vehicle-in-the-loop testing[J]. Vehicles, 2021, 3(2): 257-271. doi: 10.3390/vehicles3020016
[168]	SCHYR C, BRISSARD A. Driving cube-a novel concept for validation of powertrain and steering systems with automated driving[C]//CRC Press. In Proceedings of the 13th International Symposium on Advanced Vehicle Control (AVEC'16). Boca Raton: CRC Press, 2016: 79-84.
[169]	LI He-xuan, NALIC D, MAKKAPATI V, et al. A real-time co-simulation framework for virtual test and validation on a high dynamic vehicle test bed[C]//IEEE. 2021 IEEE Intelligent Vehicles Symposium (Ⅳ). New York: IEEE, 2021: 1132-1137.
[170]	VAN BRUMMELEN J, O'BRIEN M, GRUYER D, et al. Autonomous vehicle perception: the technology of today and tomorrow[J]. Transportation Research Part C: Emerging Technologies, 2018, 89: 384-406. doi: 10.1016/j.trc.2018.02.012
[171]	WANG Xiao-gang. Intelligent multi-camera video surveillance: a review[J]. Pattern Recognition Letters, 2013, 34(1): 3-19. doi: 10.1016/j.patrec.2012.07.005
[172]	HASCH J, TOPAK E, SCHNABEL R, et al. Millimeter-wave technology for automotive radar sensors in the 77 GHz frequency band[J]. IEEE Transactions on Microwave Theory and Techniques, 2012, 60(3): 845-860. doi: 10.1109/TMTT.2011.2178427
[173]	WANG Wen-wei. Research and implementation of drum-based intelligent vehicle-in-the-loop accelerated evaluation[D]. Xi'an: Chang'an University, 2020. (in Chinese)
[174]	YU Hong-feng, WANG Yu-peng, HUA Dian, et al. Camera ADAS HIL simulation and study[J]. Automobile Electric Parts, 2019(12): 4-7. (in Chinese)
[175]	ZHOU Hai-bo, XU Wen-chao, CHEN Jia-cheng, et al. Evolutionary V2X technologies toward the Internet of vehicles: challenges and opportunities[J]. Proceedings of the IEEE, 2020, 108(2): 308-323. doi: 10.1109/JPROC.2019.2961937
[176]	LIU Tian-yang, YU Zhuo-ping, XIONG Lu, et al. Current development status and construction advice for proving ground of intelligent and connected vehicles[J]. Automobile Technology, 2017(1): 7-11, 32. (in Chinese) doi: 10.3969/j.issn.1000-3703.2017.01.002
[177]	LI Xiao-chi, ZHAO Xiang-mo, XU Zhi-gang, et al. Modular flexible test bed for intelligent and connected transportation system[J]. China Journal of Highway and Transport, 2019, 32(6): 137-146. (in Chinese)
[178]	DUAN Xu-ting, YANG Yue, TIAN Da-xin, et al. A V2X communication system and its performance evaluation test bed[C]// IEEE. 2014 IEEE 6th International Symposium on Wireless Vehicular Communications (WiVeC 2014). New York: IEEE, 2014: 1-2.
[179]	SHI Meng-kai, LU Chang, ZHANG Yi, et al. DSRC and LTE-V communication performance evaluation and improvement based on typical V2X application at intersection[C]//IEEE. 2017 Chinese Automation Congress (CAC). New York: IEEE, 2017: 556-561.
[180]	CHEN Zi-xuan, MA Wan-jing, HAO Ruo-chen, et al. Hardware-in-the-loop simulation testbed for intelligent and connected vehicle[C]//Technical Committee on Annual Conference of ITS China. The 13th Annual Conference of ITS China. Beijing: Technical Committee on Annual Conference of ITS China, 2018: 54-63. (in Chinese)
[181]	WANG Run-min, ZHANG Xin-rui, WANG You-dao, et al. Research and practice on construction technology of closed test field autonomous driving[J]. Automobile Applied Technology, 2020(4): 33-36. (in Chinese)
[182]	QIAO Guan-dong. Research on advantages and application of BIM technology in urban road design[J]. Construction and Design for Engineering, 2023(4): 91-93. (in Chinese)
[183]	BAO Xiu-zhi, CHEN Xin-wang, LU Ming-yue. Photogrammetric methods for real three-dimensional graphics of buildings[J]. Railway Investigation and Surveying, 1996(3): 1-4. (in Chinese)
[184]	THORN E, KIMMEL S C, CHAKA M, et al. A framework for automated driving system testable cases and scenarios[R]. Washington DC: National Highway Traffic Safety Administration, 2018.
[185]	ZACHMAN J A. A framework for information systems architecture[J]. IBM Systems Journal, 1987, 26(3): 276-292. doi: 10.1147/sj.263.0276
[186]	SPEWAK S H, HILL S C. Enterprise Architecture Planning: Developing a Blueprint for Data, Applications and Technology[M]. Boston: QED Pub. Group, 1993.
[187]	CHETAL A. Federal enterprise architecture[J]. Alphascript Publishing, 2002, 46(4): 258-274.
[188]	LAGERSTRÖM R, SOMMESTAD T, BUSCHLE M, et al. Enterprise architecture management's impact on information technology success[C]// IEEE. 2011 44th Hawaii International Conference on System Sciences. New York: IEEE, 2011: 1-10.
[189]	SIMON G A. DoD (department of defense) protocol reference model[R]. Santa Monica: System Development Corporation, 1982.
[190]	WOOD W G, BARBACCI M, CLEMENTS P, et al. DoD architecture framework and software architecture workshop report[R]. Pittsburgh: Software Engineering Institute of Carnegie Mellon University, 2003.
[191]	SUN Yu-sheng, LIU Lin, XUE Tong. Comparative research on IT planning reference model[J]. Computer and Digital Engineering, 2019, 47(11): 2771-2779, 2823. (in Chinese) doi: 10.3969/j.issn.1672-9722.2019.11.026
[192]	LENG Bing-bo. The automobile manufacturing industry information system integration application research based on EA[D]. Shenyang: Shenyang Normal University, 2013. (in Chinese)
[193]	ZHAO Xiang-mo, GAO Ying, XU Zhi-gang, et al. IntelliWay: an integrated architecture and testing methodology for intelligent highway using varied coupling modularization[J]. China Journal of Highway and Transport, 2023, 36(1): 176-201. (in Chinese) doi: 10.3969/j.issn.1001-7372.2023.01.015
[194]	WANG Xiao-yi, LI Jun-cheng, MA Xue-han. Research on test case construction method of autonomous vehicle perception system[J]. Quality and Standardization, 2022(1): 51-54. (in Chinese)
[195]	XIA Qin, DUAN Jian-li, GAO Feng, et al. Test scenario design for intelligent driving system ensuring coverage and effectiveness[J]. International Journal of Automotive Technology, 2018, 19(4): 751-758. doi: 10.1007/s12239-018-0072-6
[196]	ZOFKA M R, KUHNT F, KOHLHAAS R, et al. Data-driven simulation and parametrization of traffic scenarios for the development of advanced driver assistance systems[C]//IEEE. 2015 18th International Conference on Information Fusion (Fusion). New York: IEEE, 2015: 1422-1428.
[197]	AN Ze-ping, HE Jing, YAO Xiang-lin. Research on the construction of expressway intelligent vehicle infrastructure cooperative systems application scenario library[J]. Highway, 2021, 66(12): 270-274. (in Chinese)
[198]	MA Ya-fang, WANG Wen-jie, GU Ke, et al. Test and analysis of autonomous driving function of intelligent and connected vehicles[J]. Auto Time, 2021(23): 38-39, 46. (in Chinese) doi: 10.3969/j.issn.1672-9668.2021.23.017
[199]	LIU Xin. Ministry of industry and information technology: standardizing intelligent networked vehicle road test and demonstration application[J]. China Plant Engineering, 2021(16): 1. (in Chinese)
[200]	LI Li, HUANG Wu-ling, LIU Yue-hu, et al. Intelligence testing for autonomous vehicles: a new approach[J]. IEEE Transactions on Intelligent Vehicles, 2016, 1(2): 158-166. doi: 10.1109/TIV.2016.2608003
[201]	LI Li, LIN Yi-lun, ZHENG Nan-ning, et al. Artificial intelligence test: a case study of intelligent vehicles[J]. Artificial Intelligence Review, 2018, 50(3): 441-465. doi: 10.1007/s10462-018-9631-5
[202]	WOOLDRIDGE M, JENNINGS N R. Intelligent agents: theory and practice[J]. The Knowledge Engineering Review, 1995, 10(2): 115-152. doi: 10.1017/S0269888900008122
[203]	HUANG Hui-ming, PAVEK K, NOVAK B, et al. A framework for autonomy levels for unmanned systems (ALFUS)[C]//AUVSI. Proceedings of the AUVSI's unmanned systems North America. New York: AUVSI, 2005: 849-863.
[204]	NAGAI M S, INOUE H. Research into ADAS with autonomous driving intelligence for future innovation[C]//Springer. 5th International Munich Chassis Symposium 2014. Berlin: Springer, 2014: 779-793.
[205]	PAYALAN Y F, GUVENSAN M A. Towards next-generation vehicles featuring the vehicle intelligence[J]. IEEE Transactions on Intelligent Transportation Systems, 2020, 21(1): 30-47. doi: 10.1109/TITS.2019.2917866
[206]	WU Xin-zheng, XING Xing-yu, CHEN Jun-yi, et al. Risk assessment method for driving scenarios of autonomous vehicles based on drivable area[C]//IEEE. 2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC). New York: IEEE, 2022: 2206-2213.
[207]	MENG Hao-lan, CHEN Jun-yi, WANG Bin, et al. Effect of negative stimulus caused by vehicle maneuver on passenger's expectation[J]. Journal of Tongji University (Natural Science), 2022, 50(5): 759-766. (in Chinese)
[208]	BELLEM H, THIEL B, SCHRAUF M, et al. Comfort in automated driving: an analysis of preferences for different automated driving styles and their dependence on personality traits[J]. Transportation Research Part F: Traffic Psychology and Behaviour, 2018, 55: 90-100. doi: 10.1016/j.trf.2018.02.036
[209]	MA Yi-ning, SUN Chen, CHEN Jun-yi, et al. Verification and validation methods for decision-making and planning of automated vehicles: a review[J]. IEEE Transactions on Intelligent Vehicles, 2022, 7(3): 480-498. doi: 10.1109/TIV.2022.3196396
[210]	MENG Hao-lan, CHEN Jun-yi, FENG Tian-yue, et al. An interactive car-following model (ICFM) for the harmony-with-traffic evaluation of autonomous vehicles[C]//SAE International. 2023 WCX SAE World Congress Experience. Warrendale: SAE International, 2023: 01881765.
[211]	ZHU Xi-chan, WEI Hao-zhou, MA Zhi-xiong. Assessment of the potential risk in car-following scenario based on naturalistic driving data[J]. China Journal of Highway and Transport, 2020, 33(4): 169-181. (in Chinese) doi: 10.3969/j.issn.1001-7372.2020.04.017
[212]	SHALEV-SHWARTZ S, SHAMMAH S, SHASHUA A. On a formal model of safe and scalable self-driving cars[J]. arxiv, 2017, DOI: 1708.06374.
[213]	BAE I, MOON J, SEO J. Toward a comfortable driving experience for a self-driving shuttle bus[J]. Electronics, 2019, 8(9): 943. doi: 10.3390/electronics8090943
[214]	TASHIRO M, MOTOYAMA H, ICHIOKA Y, et al. Simulation analysis on optimal merging control of connected vehicles for minimizing travel time[J]. International Journal of Intelligent Transportation Systems Research, 2020, 18(1): 65-76. doi: 10.1007/s13177-018-0172-8
[215]	CHU K, LEE M, SUNWOO M. Local path planning for off-road autonomous driving with avoidance of static obstacles[J]. IEEE Transactions on Intelligent Transportation Systems, 2012, 13(4): 1599-1616. doi: 10.1109/TITS.2012.2198214
[216]	CHAI Chen, ZENG Xian-ming, WU Xiang-bin, et al. Evaluation and optimization of responsibility-sensitive safety models on autonomous car-following maneuvers[J]. Transportation Research Record, 2020, 2674(11): 662-673. doi: 10.1177/0361198120948507
[217]	CAO Peng, XU Zhan-dong, FAN Qiao-chu, et al. Analysing driving efficiency of mandatory lane change decision for autonomous vehicles[J]. IET Intelligent Transport Systems, 2019, 13(3): 506-514. doi: 10.1049/iet-its.2018.5253
[218]	CHEN Bai-ming, CHEN Xiang, WU Qiong, et al. Adversarial evaluation of autonomous vehicles in lane-change scenarios[J]. IEEE Transactions on Intelligent Transportation Systems, 2022, 23(8): 10333-10342. doi: 10.1109/TITS.2021.3091477
[219]	MENG Hao-lan, CHEN Jun-yi, XING Xing-yu, et al. Study on objective representation of passenger discomfort[J]. Journal of Tongji University (Natural Science), 2019, 47(S1): 54-63. (in Chinese)
[220]	SUN Yang, XIONG Guang-ming, CHEN Hui-yan, et al. A cost function-oriented quantitative evaluation method for unmanned ground vehicles[J]. Advanced Materials Research, 2011, 301/302/303: 701-706.
[221]	SUN Yang, YANG He, MENG Fei. Research on an intelligent behavior evaluation system for unmanned ground vehicles[J]. Energies, 2018, 11(7): 1764. doi: 10.3390/en11071764
[222]	PAN Ang, ZHANG Xin, IRYO-ASANO M, et al. Efficiency and safety evaluation of left-turn vehicles and crossing pedestrians in signalized intersections under the autonomous vehicle mixed flow condition[J]. International Journal of Intelligent Transportation Systems Research, 2022, 20(1): 103-116. doi: 10.1007/s13177-021-00276-z
[223]	CHEN Jun-yi, CHEN Lei, MENG Hao-lan, et al. Evaluation method of the harmony with traffic in two-vehicle interaction scenarios[J]. Automobile Technology, 2020(11): 1-7. (in Chinese)
[224]	XIA Qin, DUAN Jian-li, GAO Feng, et al. Automatic generation method of test scenario for ADAS based on complexity[C]//SAE International. SAE Intelligent and Connected Vehicles Symposium. Warrendale: SAE International, 2017: 1-9.
[225]	MA Yi-ning, JIANG Wei, WU Jing-yu, et al. Self-evolution scenarios for simulation tests of autonomous vehicles based on different models of driving styles[J]. China Journal of Highway and Transport, 2023, 36(2): 216-228. (in Chinese)
[226]	XING Jing, MANNING C A. Complexity and automation displays of air traffic control: literature review and analysis[R]. Washington DC: US Department of Transportation, 2005.
[227]	XING Jing. Measures of information complexity and the implications for automation design[R]. Washington DC: US Department of Transportation, 2004.
[228]	CASTI J L. Connectivity, Complexity and Catastrophe in Large-Scale Systems[M]. New York: John Wiley and Sons, 1979.
[229]	CRUTCHFIELD J P, YOUNG K. Inferring statistical complexity[J]. Physical Review Letters, 1989, 63(2): 105-108. doi: 10.1103/PhysRevLett.63.105
[230]	SUSSMAN J M. The new transportation faculty: the evolution to engineering systems[J]. Transportation Quarterly, 1999, 53(3): 15-26.
[231]	MA Yi-ning, PAN Xin-fu, XIONG Lu, et al. Definition and quantification of the complexity experienced by autonomous vehicles in the environment and driving task[C]// CICTP. 2020 COTA International Conference of Transportation Professionals. Reston: CICTP, 2020: 1030-1042.
[232]	ZHANG Ling-tong, MA Yi-ning, XING Xing-yu, et al. Research on the complexity quantification method of driving scenarios based on information entropy[C]//IEEE. 2021 IEEE International Intelligent Transportation Systems Conference (ITSC). New York: IEEE, 2021: 3476-3481.
[233]	BOELHOUWER A, BEUKEL A P V D, VOORT M C, et al. Determining infrastructure-and traffic factors that increase the perceived complexity of driving situations[C]//Springer. Advances in Human Aspects of Transportation: Proceedings of the AHFE 2020 Virtual Conference on Human Aspects of Transportation. Berlin: Springer, 2020: 3-10.
[234]	LIU Yong-kang, HANSEN J H L. Towards complexity level classification of driving scenarios using environmental information[C]//IEEE. 2019 IEEE Intelligent Transportation Systems Conference (ITSC). New York: IEEE, 2019: 810-815.
[235]	COVER T M, THOMAS J A. Elements of Information Theory[M]. New York: John Wiley and Sons, 2001.
[236]	MUELLER J, STANLEY L, MARTIN T, et al. Driving simulator and scenario effects on driver response[C]//ⅡSE. Proceedings of the 2014 Industrial and Systems Engineering Research Conference. Montreal: ⅡSE, 2014: 507-513.
[237]	WANG Jia-jie, ZHANG Chi, LIU Yue-hu, et al. Traffic sensory data classification by quantifying scenario complexity[C]//IEEE. 2018 IEEE Intelligent Vehicles Symposium (Ⅳ). New York: IEEE, 2018: 1543-1548.
[238]	GAO Feng, DUAN Jian-li, HE Ying-dong, et al. A test scenario automatic generation strategy for intelligent driving systems[J]. Mathematical Problems in Engineering, 2019, 2019: 1-10.
[239]	LOWRIE J W, THOMAS M, GREMBAN K, et al. The autonomous land vehicle (ALV) preliminary road-following demonstration[C]//SPIE. Intelligent Robots and Computer Vision Ⅳ. Washington DC: SPIE, 1985: 336-350.
[240]	POMERLEAU D. RALPH: rapidly adapting lateral position handler[C]//IEEE. Proceedings of the Intelligent Vehicles' 95. Symposium. New York: IEEE, 2002: 506-511.
[241]	BROGGI A, BERTOZZI M, FASCIOLI A. ARGO and the Mille Miglia in automatico tour[J]. IEEE Intelligent Systems and Their Applications, 1999, 14(1): 55-64. doi: 10.1109/5254.747906
[242]	MAURER M, BEHRINGER R, FURST S, et al. A compact vision system for road vehicle guidance[C]//IEEE. Proceedings of 13th International Conference on Pattern Recognition. New York: IEEE, 1996: 313-317.
[243]	MCWILLIAMS G T, BROWN M A, LAMM R D, et al. Evaluation of autonomy in recent ground vehicles using the autonomy levels for unmanned systems (ALFUS) framework[C]// IEEE. Proceedings of the 2007 Workshop on Performance Metrics for Intelligent Systems. New York: ACM, 2007: 54-61.
[244]	WANG Yue-chao, LIU Jin-guo. Evaluation methods for the autonomy of unmanned systems[J]. Chinese science bulletin, 2012, 57(26): 3409-3418. doi: 10.1007/s11434-012-5183-2
[245]	SUN Yang, CHEN Hui-yan. Research on test and evaluation of unmanned ground vehicles[J]. Acta Armamentarii, 2015, 36(6): 978-986. doi: 10.3969/j.issn.1000-1093.2015.06.003
[246]	CEHN Jun-yi, LI Ru-bing, XING Xing-yu, et al. Survey on intelligence evaluation of autonomous vehicles[J]. Journal of Tongji University (Natural Science), 2019, 47(12): 1785-1790, 1824. (in Chinese) doi: 10.11908/j.issn.0253-374x.2019.12.014
[247]	MENG Hao-lan, CHEN Jun-yi, CHEN Lei, et al. Clustering analysis of typical ramp scenarios based on naturalistic driving data[J]. Journal of Tongji University (Natural Science), 2021, 49(S1): 123-131. (in Chinese)
[248]	WEI Jun-qing, DOLAN J M, LITKOUHI B. Autonomous vehicle social behavior for highway entrance ramp management[C]//IEEE. 2013 IEEE Intelligent Vehicles Symposium (Ⅳ). New York: IEEE, 2013: 201-207.
[249]	LETTER C, ELEFTERIADOU L. Efficient control of fully automated connected vehicles at freeway merge segments[J]. Transportation Research Part C: Emerging Technologies, 2017, 80: 190-205. doi: 10.1016/j.trc.2017.04.015
[250]	NTOUSAKIS I A, NIKOLOS I K, PAPAGEORGIOU M. Optimal vehicle trajectory planning in the context of cooperative merging on highways[J]. Transportation Research Part C: Emerging Technologies, 2016, 71: 464-488. doi: 10.1016/j.trc.2016.08.007
[251]	PARMAR H, CHAKROBORTY P, KUNDU D. Modelling automobile drivers' toll-lane choice behaviour at a toll plaza using mixed logit model[J]. Procedia-Social and Behavioral Sciences, 2013, 104: 593-600. doi: 10.1016/j.sbspro.2013.11.153
[252]	CHEN Jun-yi, CHEN Lei, MENG Hao-lan, et al. Evaluation model of harmony with traffic based on neural network[J]. Journal of Tongji University (Natural Science), 2021, 49(1): 135-141. (in Chinese)
[253]	VIRES. Virtual test drive[EB/OL]. (2022-11-11)[2022-11-03]. https://hexagon.com/products/virtual-test-drive.
[254]	NVIDIA. NVIDIA drive end-to-end solutions for autonomous vehicles[EB/OL]. (2022-01-13)[2022-11-03]. https://developer.nvidia.com/drive.
[255]	OICA. Future certification of automated/autonomous driving Systems[EB/OL]. (2019-01-28)[2022-11-03]. https://unece.org/fileadmin/DAM/trans/doc/2019/wp29grva/GRVA-02-09e.pdf.
[256]	WANG Run-min, ZHAO Xiang-mo, XU Zhi-gang, et al. Design of virtual simulation test platform based on vehicle-in-the-loop for automatic driving[J]. Automobile Technology, 2022(4): 1-7. (in Chinese)
[257]	SON T D, BHAVE A, VAN DER AUWERAER H. Simulation-based testing framework for autonomous driving development[C]// IEEE. 2019 IEEE International Conference on Mechatronics (ICM), New York: IEEE, 2019: 576-583.
[258]	LIU Fa-wang, CAO Jian-yong, ZHANG Zhi-qiang, et al. A scenario-based "three-pillar" safety testing and assessment method for intelligent and connected vehicles[J]. Chinese Journal of Automotive Engineering, 2023, 13(1): 1-7. (in Chinese)
[259]	KIM B J, LEE S B. Safety evaluation of autonomous vehicles for a comparative study of camera image distance information and dynamic characteristics measuring equipment[J]. IEEE Access, 2022, 10: 18486-18506. doi: 10.1109/ACCESS.2022.3151075
[260]	ZHANG Pei-xing, ZHU Bing, ZHAO Jian, et al. Safety evaluation method in multi-logical scenarios for automated vehicles based on naturalistic driving trajectory[J]. Accident Analysis and Prevention, 2023, 180: 106926. doi: 10.1016/j.aap.2022.106926
[261]	WU Sheng-hao, ZHENG Su-li, YANG Lu-qi. Development trend of standardization of overseas autonomous vehicles and its enlightenment to China[J]. Standard Science, 2021(11): 16-24. (in Chinese) doi: 10.3969/j.issn.1674-5698.2021.11.003
[262]	HE Jin-peng, MA Fang-wu, LIU Wei-guo, et al. Comparative test between ISO 15623 on forward vehicle collision warning system and NHTSA 26555 assessment program[J]. Automobile Technology, 2014(7): 28-33. (in Chinese)
[263]	CAO Jian-yong, CAO Yin, ZHANG Jian-wen. Analysis of vehicle active safety testing technology based on Euro-NCAP[J]. Quality and Standardization, 2016(3): 55-58. (in Chinese)
[264]	TANG Bo. Advanced crash test technology of ANCAP[J]. Automobile and Parts, 2013(9): 21-23. (in Chinese)
[265]	GUO Fei, ZHAO Lei, DANG Guo-feng. Research on automobile detection organization and C-NCAP system[J]. Science and Technology Industry Parks, 2019(7): 36-37. (in Chinese)
[266]	PEREZ M A, KIEFER R J, HASKINS A, et al. Evaluation of forward collision warning system visual alert candidates and SAE J2400[J]. SAE International Journal of Passenger Cars-Mechanical Systems, 2009, 2(1): 750-764. doi: 10.4271/2009-01-0547
[267]	LI Hai-yan, YANG Zhen, HE Li-juan, et al. Comparative study and prospect of pedestrian protection assessment in global NCAP[J]. Automotive Engineering, 2021, 43(5): 730-738. (in Chinese)
[268]	WANG Yu-mei, GUO Qing-xiang. Effect on updated Euro-NCAP on safety evaluation for a passenger car[J]. Automotive Engineer, 2014(4): 18-22. (in Chinese) doi: 10.3969/j.issn.1674-6546.2014.04.003
[269]	LIN Gao-ze, YANG Hai-yan, ZHOU Da-yong, et al. Pedestrian protection test method comparison between E-NCAP V8.0 and J-NCAP[J]. Automotive Engineer, 2015(12): 13-17. (in Chinese)
[270]	WANG Yun-peng. Insisting on safety base line and enhancing regulation innovation—build an inclusive and prudent vehicle safety sandbox regulatory system[J]. Research on China Market Supervision, 2022(4): 5-6, 54. (in Chinese)
[271]	WANG Le-bing. Defects in autonomous vehicles and their product liability[J]. Tsinghua Law Review, 2020, 14(2): 93-112. (in Chinese)
[272]	GAO Zhi-wei, CECATI C, DING S X. A survey of fault diagnosis and fault-tolerant techniques—part Ⅰ: fault diagnosis with model-based and signal-based approaches[J]. IEEE Transactions on Industrial Electronics, 2015, 62(6): 3757-3767. doi: 10.1109/TIE.2015.2417501
[273]	JIANG Yu-chen, YIN Shen, KAYNAK O. Optimized design of parity relation-based residual generator for fault detection: data-driven approaches[J]. IEEE Transactions on Industrial Informatics, 2021, 17(2): 1449-1458. doi: 10.1109/TII.2020.2987840
[274]	ZHANG Zi-long, ZHAO Zhi-bin, LI Xiao-long, et al. Faster multiscale dictionary learning method with adaptive parameter estimation for fault diagnosis of traction motor bearings[J]. IEEE Transactions on Instrumentation and Measurement, 2020, 70: 3504113.
[275]	CHOI K, KIM Y, KIM S K, et al. Current and position sensor fault diagnosis algorithm for PMSM drives based on robust state observer[J]. IEEE Transactions on Industrial Electronics, 2021, 68(6): 5227-5236. doi: 10.1109/TIE.2020.2992977
[276]	GAO Zhi-wei, CECATI C, DING S X. A survey of fault diagnosis and fault-tolerant techniques—part Ⅱ: fault diagnosis with knowledge-based and hybrid/active approaches[J]. IEEE Transactions on Industrial Electronics, 2015, 62(6): 3768-3774.
[277]	BEN LAKHAL N M, ADOUANE L, NASRI O, et al. Interval-based solutions for reliable and safe navigation of intelligent autonomous vehicles[C]//IEEE. 2019 12th International Workshop on Robot Motion and Control (RoMoCo). New York: IEEE, 2019: 124-130.
[278]	BANDO M, ONO Y, HIEIDA Y, et al. GNSS fault detection with unmodeled error[J]. Advanced Robotics, 2017, 31(15): 763-779.
[279]	LIU Jing, WEI Yan-hui, HAO Sheng-gong. A fault diagnosis method for INS/DVL/USBL integrated navigation system based on support vector regression[C]//IEEE. OCEANS 2019-Marseille. New York: IEEE, 2019: 1-8. .
[280]	VAN WYK F, WANG Yi-yang, KHOJANDI A, et al. Real-time sensor anomaly detection and identification in automated vehicles[J]. IEEE Transactions on Intelligent Transportation Systems, 2020, 21(3): 1264-1276.
[281]	KANAPRAM D T, MARIN-PLAZA P, MARCENARO L, et al. Self-awareness in intelligent vehicles: feature based dynamic Bayesian models for abnormality detection[J]. Robotics and Autonomous Systems, 2020, 134: 103652.
[282]	AN J, CHO S. Variational autoencoder based anomaly detection using reconstruction probability[J]. Special Lecture on IE, 2015, 2(1): 1-18.
[283]	PANG Guan-song, SHEN Chun-hua, CAO Long-bing, et al. Deep learning for anomaly detection: a review[J]. ACM Computing Surveys, 2021, 54(2): 1-38.
[284]	CHI Yuan-fang, DONG Yan-jie, WANG Z J, et al. Knowledge-based fault diagnosis in industrial internet of things: a survey[J]. IEEE Internet of Things Journal, 2022, 9(15): 12886-12900.
[285]	KHALASTCHI E, KALECH M. On fault detection and diagnosis in robotic systems[J]. ACM Computing Surveys, 2018, 51(1): 1-24.
[286]	SANDOVAL-PILLAJO L, TARUPI A, BASANTES A, et al. Expert system for diagnosis of motor failures in electronic injection vehicles[C]//IEEE. 2019 International Conference on Information Systems and Computer Science (INCISCOS). New York: IEEE, 2019: 259-266.
[287]	LIU Lei. Research on intelligent fault diagnosis and testability verification and evaluation methods of equipment[D]. Zhengzhou: Zhengzhou University, 2017. (in Chinese)
[288]	DENG Lu, XU Ai-qiang, ZHAO Xiu-li. Allocation plan of samples based on failure attribute in testability demonstration test[J]. Journal of Test and Measurement Technology, 2014, 28(2): 103-107. (in Chinese)
[289]	YU Si-qi, JING Bo, HUANG Yi-feng. Study on allocation scheme of failure sample in testability validation tests based on contribution[J]. China Measurement and Test, 2015, 41(2): 91-95. (in Chinese)
[290]	LI Tian-mei. Research on optimization design and integrated evaluation of testability verification test for equipments[D]. Changsha: National University of Defense Technology, 2010. (in Chinese)
[291]	SHI Jun-you, KANG Rui, TIAN Zhong. Study on sufficiency of sample set in testability demonstration based on information model[J]. Journal of Beijing University of Aeronautics and Astronautics, 2005, 31(8): 874-878. (in Chinese)
[292]	FENG Hao. Research on safety of the intended functionality of perception module for highway pilot system[D]. Changchun: Jilin University, 2022. (in Chinese)
[293]	PRⅡSALU M, PIRINEN A, PADURARU C, et al. Generating scenarios with diverse pedestrian behaviors for autonomous vehicle testing[C]//PMLR. Proceedings of the 5th Conference on Robot Learning. New York: PMLR, 2021: 1247-1258.
[294]	360 Intelligent and Connected Vehicle Security Laboratory. Analysis on the development trend of intelligent network automobile information security[J]. Intelligent Connected Vehicles, 2020(3): 54-63. (in Chinese)
[295]	KIM K, KIM J S, JEONG S, et al. Cybersecurity for autonomous vehicles: review of attacks and defense[J]. Computers and Security, 2021, 103: 102150.
[296]	PHAM M, XIONG Kai-qi. A survey on security attacks and defense techniques for connected and autonomous vehicles[J]. Computers and Security, 2021, 109: 102269.
[297]	ZHOU Yuan-yuan. Analysis and application of Internet of vehicles information security testing technology[J]. Beijing Automotive Engineering, 2020(2): 23-27. (in Chinese)
[298]	LIU Fa-wang, LI Yan-wen. Research on functional safety and safety of the intended functionality for automated driving system[J]. Industrial Technology Innovation, 2021, 8(3): 62-68. (in Chinese)
[299]	ZOU Bo-song, ZHU Ke-yi, WANG Hui-jie. Test and analysis of intelligent network automobile information security[J]. Intelligent Connected Vehicle, 2020(6): 56-58. (in Chinese)

Relative Articles

[1]	CUI Jian-xun, YAO Jia, ZHAO Bo-yuan. Review on short-term traffic flow prediction methods based on deep learning[J]. Journal of Traffic and Transportation Engineering, 2024, 24(2): 50-64. doi: 10.19818/j.cnki.1671-1637.2024.02.003
[2]	GUO Yan-yong, LIU Pei, YUAN Quan, LIU Pan, XU Jin, ZHANG Hui. Review on research of road traffic safety of connected and automated vehicles[J]. Journal of Traffic and Transportation Engineering, 2023, 23(5): 19-38. doi: 10.19818/j.cnki.1671-1637.2023.05.002
[3]	WANG Zhi-zhong, TIAN Ye, SUN Jian, HUANG Yan, YU Rong-jie. Construction and rethinking on public service platform for autonomous driving test[J]. Journal of Traffic and Transportation Engineering, 2023, 23(6): 1-9. doi: 10.19818/j.cnki.1671-1637.2023.06.001
[4]	QU Xu, GAN Rui, AN Bo-cheng, LI Lin-heng, CHEN Zhi-jun, RAN Bin. Prediction of traffic swarm movement situation based on generalized spatio-temporal graph convolution network[J]. Journal of Traffic and Transportation Engineering, 2022, 22(3): 79-88. doi: 10.19818/j.cnki.1671-1637.2022.03.006
[5]	CHEN Ting, YAO Da-chun, GAO Tao, QIU Hui-hui, GUO Chang-xin, LIU Zhan-wen, LI Yong-hui, BIAN Hao-yi. A fused network based on PReNet and YOLOv4 for traffic object detection in rainy environment[J]. Journal of Traffic and Transportation Engineering, 2022, 22(3): 225-237. doi: 10.19818/j.cnki.1671-1637.2022.03.018
[6]	QIU Wei-zhi, SHANGGUAN Wei, CHAI Lin-guo, CHU Duan-feng. Multi-scale filtering synchronization method for vehicle-infrastructure cooperative twin-simulation testing[J]. Journal of Traffic and Transportation Engineering, 2022, 22(3): 199-209. doi: 10.19818/j.cnki.1671-1637.2022.03.016
[7]	LI Rui-min, DAI Jing-chen. Review on impact of autonomous driving on travel behaviors[J]. Journal of Traffic and Transportation Engineering, 2022, 22(3): 41-54. doi: 10.19818/j.cnki.1671-1637.2022.03.003
[8]	ZHAO Tong, SHANGGUAN Wei, CHAI Lin-guo, GUO Peng. Scenario factor analysis and test case generation for vehicle-infrastructure cooperative mixed traffic[J]. Journal of Traffic and Transportation Engineering, 2022, 22(3): 263-276. doi: 10.19818/j.cnki.1671-1637.2022.03.021
[9]	SHANGGUAN Wei, LI Xin, CHAI Lin-guo, CAO Yue, CHEN Jing-jing, PANG Hao-jie, RUI Tao. Research review on simulation and test of mixed traffic swarm in vehicle-infrastructure cooperative environment[J]. Journal of Traffic and Transportation Engineering, 2022, 22(3): 19-40. doi: 10.19818/j.cnki.1671-1637.2022.03.002
[10]	YAO Rong-han, QI Wen-yan, GUO Wei-wei. Structural equation model of drivers' takeover behaviors in autonomous driving environment[J]. Journal of Traffic and Transportation Engineering, 2021, 21(2): 209-221. doi: 10.19818/j.cnki.1671-1637.2021.02.018
[11]	MA Yong-jie, CHENG Shi-sheng, MA Yun-ting, MA Yi-de. Review of convolutional neural network and its application in intelligent transportation system[J]. Journal of Traffic and Transportation Engineering, 2021, 21(4): 48-71. doi: 10.19818/j.cnki.1671-1637.2021.04.003
[12]	WANG Run-min, ZHU Yu, ZHAO Xiang-mo, XU Zhi-gang, ZHOU Wen-shuai, LIU Tong. Research progress on test scenario of autonomous driving[J]. Journal of Traffic and Transportation Engineering, 2021, 21(2): 21-37. doi: 10.19818/j.cnki.1671-1637.2021.02.003
[13]	TANG Li, QING San-dong, XU Zhi-gang, ZHOU Hou-qing. Research review on public acceptance of autonomous driving[J]. Journal of Traffic and Transportation Engineering, 2020, 20(2): 131-146. doi: 10.19818/j.cnki.1671-1637.2020.02.011
[14]	QIN Yan-yan, ZHANG Jian, CHEN Ling-zhi, LI Shu-qing, HE Zhao-yi, RAN Bin. Cell transmission model of mixed traffic flow of manual-automated driving[J]. Journal of Traffic and Transportation Engineering, 2020, 20(2): 229-238. doi: 10.19818/j.cnki.1671-1637.2020.02.019
[15]	CHEN Xiao-bo, CHEN Cheng, CHEN Lei, WEI Zhong-jie, CAI Ying-feng, ZHOU Jun-jie. Interpolation method of traffic volume missing data based on improved low-rank matrix completion[J]. Journal of Traffic and Transportation Engineering, 2019, 19(5): 180-190. doi: 10.19818/j.cnki.1671-1637.2019.05.018
[16]	YANG Lin-jian, ZHAO Xiang-mo, HE Bing-hua, WEI Qiu-yue, AN Yi-sheng. An ant colony optimization algorithm of stochastic user equilibrium traffic assignment problem[J]. Journal of Traffic and Transportation Engineering, 2018, 18(3): 189-198. doi: 10.19818/j.cnki.1671-1637.2018.03.019
[17]	WU Chao-zhong, WU Hao-ran, LYU Neng-chao. Review of control switch and safety of human-computer driving intelligent vehicle[J]. Journal of Traffic and Transportation Engineering, 2018, 18(6): 131-141. doi: 10.19818/j.cnki.1671-1637.2018.06.014
[18]	Wang Xiao-yuan, Yang Xin-yue. Nonparametric microscopic simulation model of driving behavior[J]. Journal of Traffic and Transportation Engineering, 2007, 7(1): 76-80.
[19]	Zhang Fu-ren. Intelligent selection system of adaptive optimal route based on GIS and grey-fuzzy decision theory[J]. Journal of Traffic and Transportation Engineering, 2007, 7(5): 106-111.
[20]	PAN Ming-yang, YAN Fei, XIE Hai-yan. Intelligent traffic simulation based on agent and cellular automata[J]. Journal of Traffic and Transportation Engineering, 2006, 6(2): 70-74.

Supplements(0)

Cited By

Cited by

Periodical cited type(2)

1.	宋亮，高冲，范鹏，仰建岗，高杰. 蒸发方式对乳化沥青残留物高温流变性能与疲劳性能影响分析. 市政技术. 2024(11): 23-28+37 .
2.	潘惠雄，陈豫慧，龙兴灵，陈军，向一鸣，赵西杰. 直投式SBS胶乳改性乳化沥青的制备与性能研究. 江西建材. 2024(08): 87-91 .

Other cited types(3)

Figures(29) / Tables(20)

Get Citation

PDF

XML

Article Metrics

Article views (2855) PDF downloads(443)