One of the problems with ” life recorders”, wearable video cameras that record your every movement, is making sense of the huge datasets they produce. Given a day’s worth of inane footage–get up, wander into bathroom, brush teeth etc–what can you usefully do with it?
Various groups have proposed that such cameras could make excellent aide memoires, immediately locating lost car keys and remembering old faces. They’d also allow you to relive memories with otherwise forgotten detail–what was she wearing the day we met?
Advertisement
That’s all well and good but there is a serious practical problem. How can a computer make sense of the endless stream of footage?
This story is only available to subscribers.
Don’t settle for half the story.
Get paywall-free access to technology news for the here and now.
Scene change detection is relatively straight forward in many videos such as TV programs and movies because a scene change usually coincides with a change in the camera’s perspective. All you have to do is look for theses changes, a task that is made easier if the images are well lit and have relatively little blur, as is the case in most professional recorded films.
Life stream videos are different. Here the camera’s perspective is always the same while the individual frames are often blurred by movement, washed out in scenes with too much light or blacked out in scenes with too little. All of this makes the task of scene change recognition that much harder.
Today, Svebor Karaman et amis at the University of Bordeaux in France say they have taken a step towards solving this problem with a new way of categorising daily activities in footage taken from a shoulder-mounted camera.
They define a scene as a sequence of frames in which the camera is relatively still, which they can easily determine by measuring trajectory of the corners of the image. They then categorise each sequence according to the colours present in the frames, which remain relatively constant even when the individual frames are blurred or dim. Finally, they manually label these scense with titles such as “moving in the kitchen” or “moving in home office” .
A computer can then use this information to detect similar patterns elsewhere in the footage. The result is a reasonably accurate picture of the activities that an ordinary person caries out on a daily basis.
That has at least one important application. The motivating factor for Karaman and co is to find an objective way of studying the patterns of behaviour of people with dementia.
An objective measure would be hugely useful. Doctors usually rely on the accounts given by relatives or carers whose perception of whether a patient is better or worse can be coloured by all kinds of other factors. The data from a lifestreaming camera, on the other hand, can tell you exactly how many times a patient visited the kitchen on Wednesday, for example, and how that compared to the same period six months ago.
Advertisement
The hope, say Karaman and co, is that this kind of data can be an important tool in evaluating the onset of dementia and the way it is advancing.
But they will have their work cut out to make this technique useful. At the moment, their technique gives information about behaviour only on a relatively coarse scale and requires significant input from a human to help “train” the program to recognise places such as kitchens and home offices. All this is site specific and will have to be repeated in other home environments.
Unless they can find a way to automate this process or at least make it much easier and faster, this will limit the appeal of this software.
On the other hand, the software may be a hugely useful research tool. The detailed study of the behaviour of dementia patients could throw up other indicators that could be used in understanding the progression of the disease. And that could ultimately be its greatest value.
Ref: arxiv.org/abs/1007.4134: Human Daily Activities Indexing in Videos from Wearable Cameras for Monitoring of Patients with Dementia Diseases
