Brain strain detector | Innovations

Brain strain detector

Brain strain detector

Updated 12 July 2012, 16:16 AEST

Computers can now monitor a worker's brain for cognitive overload

DESLEY BLANCH:   The day has arrived when your computer can monitor your levels of stress and identify when you are over-stretched, concentration gone and you're not thinking clearly when at your desk.

And, if you're an air traffic controller or a traffic incident operator where small mistakes can very quickly escalate; Australian scientists have the device that ensures you don't reach that point.

Developed at NICTA -- the National Information and Communications Technology Research Centre of Excellence in Sydney, the technology is being marketed as BrainGauge and it works by monitoring the imperceptible changes in voices that humans can't hear.

BrainGauge is being used by the NSW centre that manages traffic jams, in Canada by Ottawa's ambulance service and in Australian defence research to assess the mental strain on people during flight operations. And it's being marketed to call centres as a recruitment tool and as a monitor.

Bruce Whitby is the managing director of BrainGauge.

DESLEY BLANCH : Bruce, computers are pushing us to even higher levels of work achievement and productivity, and now that same computer can monitor how we're handling or not handling the stress of it while at this desk that we've got in front of us. So I see irony there, do you?

BRUCE WHITBY : Yes I do, I actually agree with you that modern technology today is driving society to increased levels of load. But we're in a position now where we can actually start using that technology to help us measure what load we're at quantitatively, and then we can get our management to hopefully take action and intervene.

DESLEY BLANCH : Well we as employees may be able to hide any signs of stress, but your computer system won't be fooled. So what's it looking for, what are we giving away without knowing it?

BRUCE WHITBY : I wouldn't underestimate the ability of the human to actually detect cognitive load, especially trained people like psychologists. At a simple and a human detectable level we can detect ahs and ums and delayed speech as we think, but the computer can be trained to pick up these and it can be trained to pick up some other factors, and once trained it can pick up very fine changes really early, long before a human can actually detect these changes.

DESLEY BLANCH : So the device detects what's called cognitive overload, so what is cognitive overload and how does it happen?

BRUCE WHITBY : Cognitive overload reflects a person's ability to think, reason, operate and remember clearly and accurately. For example when we're performing a task we're doing many things. We're accepting inputs like listening, watching or touching, we're thinking and reasoning to try and make sense of what we're doing, and we're probably controlling something; driving a car, moving a mouse around, and talking. So we only have a finite amount of working memory and processing power to perform this juggling act of everything that we're doing. So if the pace or the complexity of what we're doing increases we start running out of working memory and become cognitively over-loaded.

Now physiologically a number of things start happening; firstly we stop accepting more information or we start processing information exclusively, so we stop listening and only watching, that sort of stuff. We start feeling stressed and emotional and from a business perspective we start making mistakes, we start getting things wrong.

But from our point of view and BrainGauge's perspective it affects our muscle control. And muscle control affects speech. So what we are effectively doing is picking up those changes to your speech as a result of a change in your muscle control which has been caused by a high cognitive overload. And then BrainGauge is the world's first cognitive load measurement technology based on speech analysis.

DESLEY BLANCH : So tell us how the system works as all that's required is a microphone and an internet connection for sending speech data to the BrainGauge servers for processing. So what happens there?

BRUCE WHITBY : What we have is effectively a statistical and mathematical model which we've taken and we've programmed and we've written programs to run on different hardware platforms. And effectively our program accepts speech as input, it analyses the speech for the cognitive overload signals, and then produces an output telling you how you rate against other speakers.

DESLEY BLANCH : So the science behind it because it's more than speech recognition, so what does BrainGauge use in addition to voice analysis?

BRUCE WHITBY : I need to draw a slight distinction here. Speech recognition allows you to detect the meanings of words, so if somebody says no or yes you know they've answered positively or negatively to a question. We're not doing that, speech recognition is a science in its own right.

What we're doing is we're listening to the nuances of how people say things, so it's the changes in your way of saying things as you get more cognitively loaded. Now practically in a business application we more than likely use both applications or both technologies to get a business application.

DESLEY BLANCH : Any environment where people need to speak in order to complete their tasks is suited to your technology, and we have mentioned some. But tell us how it's being used in training and learning with athletes at the Australian Institute of Sport?

BRUCE WHITBY : Ok that was one of our earlier trials. Around 2009 Dr Fang Chen who's the chief research officer on this undertook trial studies and at that point we were trying to confirm the correlation between speech predicting cognitive load. And that was one of our experiments.

We effectively had candidates sitting in front of a screen watching a game in progress, so this was a lab study, so they were watching a game in progress and then we would stop the game at a certain point and ask them to describe what they'd just seen and what the next moves would be doing, and we made those scenarios more and more complex and that way we could work out how loaded the athletes actually were.

DESLEY BLANCH : So you won't be claiming kudos when our athletes do well in London? Have you got any representatives there?

BRUCE WHITBY : I'm afraid not, no, the only study we did was that netball and basketball studies that we did at AIS in about 2009, that's all we've done I'm afraid.

DESLEY BLANCH : Well you're also marketing the product to those who sell over the phone, and you believe it gives an employer an opportunity to assess a job applicant. So how does that work?

BRUCE WHITBY : From a practical perspective if you're a candidate and you're applying for a contact centre role, you get an email asking you to conduct an online test. You sit in front of your computer, you put on your headset with your microphone and you go to our website and log in and you go through the questionnaire, giving your responses talking into your mic. We then analyse that information and then compare your profile to our profile of successful candidates in a contact centre and rank you appropriately.

DESLEY BLANCH : So in fact you go through and assess people and have some kind of base that's already in your system?

BRUCE WHITBY : Yes we do, yes.

DESLEY BLANCH : Well you're guiding this new start-up company as it spins out of NICTA, and that's where Dr Fang Chen led the project, and she's now on board with you as head of your research group. So tell us how the project started, was it based on something that she was researching or had a special interest in?

BRUCE WHITBY : Dr Fang has a very keen interest in human behaviour and understanding how technology can actually improve human performance, that's what she's interested in. So this goes back quite a while. Our first patent was around about 2005, so there was work done beforehand to try and draw the correlation between how you measure cognitive load and some observable human behaviour characteristics. So it wasn't only speech, we looked at eye movement, looked at pen movement, a whole load of other sort of modes as they call it. So between 2005 and 2010 a whole lot of trials were done trying to work out which was the best mode and how we could predict cognitive load. And speech turned out to be the easiest, least obtrusive one and one of the most accurate ones to use. So that's why we're now launching speech as a product.

DESLEY BLANCH : So where does the project stand now and where do you go from here?

BRUCE WHITBY : It's really exciting and great to be part of a new technology in a new application area, but that's double the problem; you've got a new technology and a new application area and finding a new market. But fortunately Dr Fang and her research teams have done a wonderful job academically, getting patents registered and a whole lot of papers published. It's now my role to go ahead and actually commercialise it and find specific business uses for it out in the industry.

But BrainGauge is a world-first in terms of actually measuring cognitive load by listening to speech. And speech is great because it's a large part of what people do in business. So it's low-cost and unobtrusive.

DESLEY BLANCH : What interest is coming to you in the product so far?

BRUCE WHITBY : We're getting a lot of interest, we're getting a lot of people asking us to apply to a whole lot of different areas; distance learning, all the emergency service type areas you're talking about, contact centres, trading rooms. So any business application area where there's risk if people start making mistakes because they're cognitively over-loaded, bad customer service, all those things affect business and we can determine when people are reaching their cognitive load levels.

DESLEY BLANCH : So put it into a scenario, someone is in front of a computer and is going through this process and the machine is feeding back that they have an issue, what happens then?

BRUCE WHITBY : You've just described a real-time environment there, so if you're sitting in a contact centre and that's happening, the supervisor could intervene and assist you in the rest of that call. A large number of times the reason you're being cognitively over-loaded in a trained environment like a contact centre is you're presented with a new problem or customers are asking new questions that you actually aren't trained for, or you're not 100 per cent sure of the ideas, and having a supervisor with more experience just assisting you could get you over the mark. Also taken in the contact centre they just listen to your calls and then they can work out what training they need by understanding when people are cognitively loaded they can say, you know, most of our contact centre staff are cognitively loaded on these calls and if it's a common thread in terms of the subject matter, then you can get over that by providing correct training.

DESLEY BLANCH : So at this stage you're just getting it out there into these kinds of environments?

BRUCE WHITBY : Yes we're actively looking at that, we've also got research areas in training of pilots. So as I said we've got quite a few different application areas we're looking at, but from the business perspective generally in the contact centre, and contact centre type environments, emergency services are very similar environments to a contact centre.

DESLEY BLANCH:  Managing Director of BrainGauge is Bruce Whitby whose technology is giving voice to the strain on a worker’s brain. Don’t you think it rather ironic that as computers push us to even higher levels of work achievement and productivity; that those same computers can monitor how we’re handling or not handling the stress of it while at the desk.


Bruce Whitby


Managing Director of BrainGauge


Contact the studio

Got something to say about what you're hearing on the radio right now?

Send your texts to +61 427 72 72 72

Add the hashtag #raonair to add your tweets to the conversation.

Email us your thoughts on an issue. Messages may be used on air.