A Method for Indoor Passenger Identity Recognition on Large Cruise Ships Based on Vision and Inertial Sensors

The internal structure and scenes on cruise ships are complex and the surveillance camera offers limited depth information, which makes it difficult to identify the location, heading, changes in heading, and the identity of the passengers by the traditional passenger identity recognition method (PIRM) based on a single surveillance camera. To fill the gap, a novel method for indoor PIRM based on vision and inertial sensors is proposed. The YOLOv5 algorithm is used to extract the bounding box of each passenger and assign the pixel coordinate for each box; the pixel coordinate is further converted into the world coordinate system fixing on the camera according to the 2D-3D coordinate transformation formula; an improved neural network model then is used to estimate the true heading angle of passengers in the camera coordinate system. The inertial sensor data from passengers' smartphones are collected to detect the acceleration of the passengers and their walking states; the true heading angle of passengers in the world geodetic system is calculated by integrating magnetic field intensity; then, the extracted visual and inertial sensor data are fused, and limited features of passengers and their relationships are encoded, including walking state, step length, relative heading angle, relative distance, so as to solve the error accumulation problem of sensor signals. A similarity calculation formula between the features is proposed based on the two multi-correlation graphs, and the Vision and Inertial Sensors Graph Matching (VIGM) algorithm is employed to solve the maximum similarity matrix, which could identify the same passenger in both graphs. Lastly, to validate the proposed method, four scenes on the"Yangtze River Golden 3"cruise ship are employed (including the lobby, chess room, multi-function hall, and corridor), and it is found that: the average matching accuracy (AMA) of the proposed VIGM algorithm reaches 83.9% with the 1—3 s time window; the AMA of the proposed algorithm is only 4.5% lower than the ViTag algorithm using high-cost depth cameras. The results of experiments show that the proposed PIRM and VIGM algorithm have low implementation costs but equivalent performance compared to the method using high-cost depth cameras on large cruise ships.