AI technology generates human skeletal points using the output of 3D TOF sensor Human Pose Estimation Software

Summary

It has been said that falls and tumbles account for more than half of all accidents at nursing care facilities. If accidents can be reduced, the burden on these facilities can be significantly reduced. For the device to work in the facilities, it must work in sunlight and the dark at night. In order not to impose a burden on care receivers, sensing technology that takes privacy into account is necessary to achieve this.
Since OMRON has the technology to develop algorithms using AI technology with OKAO Vision*, we can build the necessary algorithms for pose estimation based on this technology. The pose estimation technology uses AI technology to estimate human skeletal points from distance data output from the 3D TOF sensor module B5L. With this technology, it has become possible to sense a person's condition that is less affected by sunlight, operates in the dark, and is privacy-conscious. Therefore, we believe that developing algorithms from skeletal point information can quantitatively recognize a person's posture and that analyzing skeletal points can be used to estimate a person's falls and tumbles.

What is Pose Estimation?

What is Pose Estimation: Skeleton Detection

Pose estimation is a technology that uses distance measurement data output from 3D TOF sensor B5L to estimate human skeletal points and output their coordinates using OMRON's proprietary algorithm that utilizes AI technology. From these skeletal points, it is possible to quantitatively recognize what kind of posture a person is in and to understand a person's posture, which is generally called pose estimation or skeletal estimation. As an application of pose estimation, we believe that customers can use the algorithm to analyze these skeletal points to estimate the possibility of a person falling and tumbling.

Background of Development

A growing number of people being cared for has led to a chronic labor shortage in the nursing field due to the aging population. It is expected that the labor shortage will worsen in the future.
It is said that falls and tumbles account for more than half of all accidents at nursing care facilities. Accident response and post-accident handling place a heavy burden on the nursing care field. To reduce the burden, when it is presumed that an accident has occurred, the system first remotely checks the condition of the person being cared for, and then only if necessary, a caregiver rushes to the person's room, thus economizing on manpower. However, installing cameras in the room of a person being cared for is often avoided for privacy reasons.
We have devised a solution to grasp a person's condition while considering privacy by developing a pose estimation technology algorithm that estimates human skeletal points from distance data output from the 3D TOF sensor B5L.
Since OMRON has the technology to develop algorithms using AI technology with OKAO Vision*, we can build the necessary algorithms for pose estimation based on this technology through deep learning.

* "OKAO" and "OKAO Vision" are registered trademarks of OMRON Corporation in Japan and other countries.

AI technology generates human skeletal points using the output of privacy-conscious 3D TOF sensor

Features of Pose Estimation

Optimizing understanding of human condition with algorithms based on AI technology

Based on the "OKAO Vision"* technology, we have developed a highly accurate algorithm to optimize the human skeletal points through AI technology from the output of a distance to a person measured by B5L.
Deep learning has reduced the risk of the skeletal points of one person being cared for becoming transferred to another person or the legs of a chair or table.

The video accurately displays skeletal points based on pose estimation following one to three persons' movements, such as sitting and bending.

Human Pose Estimation:Optimizing understanding of human condition with algorithms based on AI technology

OMRON Pose Estimation Software - Privacy-Conscious Software --

Technology that enables GPU-less edge processing

By applying the compact and high-speed technology developed over many years in the development of OKAO Vision*, an image-sensing product that can be embedded in PCs and mobile devices, to this product, it can be operated on a PC without the need for the host to provide a device such as a GPU.

Stable operation under sunlight and at night without lighting

Due to the living environment of the person being cared for, sunlight may enter the room through windows. However, B5L's light resistance allows it to output distance data to the person cared for, even in sunlight or at night without lighting. From this distance data, the skeletal points are estimated using the algorithms developed by OMRON.

Note: B5L is equipped with an LED for distance measurement and can measure distance even at night when there is no light.

The video shows that skeletal point estimation for pose estimation is affected little, even in an environment with sunlight through opened curtains.

Stable operation under sunlight and at night without lighting

OMRON Pose Estimation Software - Output in sunlight or at night without lighting –

General Specifications

General Specifications:skeleton point

ItemSpecifications
B5L installation angle *1Roll direction: Horizontal to ground
Pitch direction: Horizontal or oblique to the human body
(0 to -60°)
B5L installation height *11.8 to 2.5 m
Sensing distance *10.5 to 3.0 m
Detection condition *1Human body with at least 7 of the 13 skeletal points visible
Max. number of detections *23
Output points *3Position coordinates (x, y) and confidence level for the 13 skeletal points in the figure above
Human body detectionIncluded
Target deviceLaptop *No GPU required
Detectable postureSitting, standing, walking, bending, lying down
* Skeletal points can be detected in the above postures
Processing speed *4ca. 5FPS

*1 Depending on the installation height, camera angle, and distance between the B5L and the human body, a human body that does not show 7 or more skeletal points within the angle of view due to being too close or other reasons may not be detected or the detected coordinates may not be accurate.
*2 If four or more people are in the image, the confidence level of the human skeletal points is comprehensively judged internally, and the top three people with the highest results are output.
*3 Skeletal points that are hidden by something and not in the image tend to be output with a low confidence level.
*4 Values for typical images in OMRON's DB for Intel Core i7 CPU @ 3GHz.

System Configuration Example

The 3D TOF sensor B5L and the pose estimation software must be used as a set, and a host PC is required to install the pose estimation software.
Shown below is an example of a system in which the output data from the 3D TOF sensor B5L is used for estimating skeletal points using pose estimation software installed in the host PC, outputting XY coordinate data, and judging whether the person being cared for has fallen or not, as well as sending the judgment results to the caregiver.

System Configuration Example:Care Application

Example Application

Extract human skeletal point information

Example Application:Care and monitor

Care and monitor

Application

From skeletal point data for pose estimation, falls can be estimated.

Example ApplicationEstimates a person's falls or tumbles.

Software for Trial

If you would like to try and evaluate our pose estimation software, please contact us via our contact form, a distributor who handles OMRON products, or an OMRON salesperson.

Contact Us