A vehicle trajectory optimization method was proposed to address the challenges of frequent driving mode transitions and time-varying sensor noise in mixed traffic environments with connected and autonomous vehicles (CAVs) at intersections. It integrated fuzzy logic (FL) with improved iterative adaptive cubature Kalman filter (IACKF). First, nonlinear vehicle motion state transition models were established for four conditions (free-flow, car-following, acceleration/deceleration, and stop-and-queue). This effectively overcame the noise error accumulation caused by constant speed/acceleration assumptions in traditional filtering methods. Further, a two-level driving state recognition controller based on fuzzy logic theory was developed. It processed input information with error noise, and identified driving modes characterized by real-time vehicle speed, the position difference between the front and rear vehicles, and speed difference. Finally, the compromise factor of the IACKF algorithm was dynamically optimized based on the Bayesian information criterion and residual maximum likelihood estimation. By adaptive adjustment of both measurement noise covariance and system noise covariance, a vehicle trajectory optimization method was developed incorporating an iterative updating mechanism. Research results demonstrate that the FL + improved IACKF method exhibits the best performance in comparison with extended Kalman filter (EKF), cubature Kalman filter (CKF), and IACKF in terms of trajectory optimization accuracy and robustness, particularly in acceleration/deceleration and stop-and-queue modes. During peak hours, the root mean square error (RMSE) of vehicle position is reduced by 71.1%-74.2%, 60.6%-65.2%, and 45.9%-52.2%, and reductions in off-peak hours reach 63.8%-64.5%, 47.2%-49.5%, and 35.6%-36.0%. The performance of the optimization method is optimal in the case of process noise convariance and measurement noise convariance taking specific empirical values (control groups 3 and 5), the initial distance of 50-60 m, the initial speed of 17-21 m·s-1, and low traffic flow and proportion of heterogeneous vehicles. The proposed method can provide crucial technical support for CAV cooperative control, traffic flow optimization, etc.