Working NS2 Voice control

Caboose

I hope this is the appropriate place to post this.

So, I've been experimenting with voice control for various applications using dragonfly, which is best described on their website.

Dragonfly is a speech recognition framework. It is a Python package which offers a high-level object model and allows its users to easily write scripts, macros, and programs which use speech recognition.

It currently supports the following speech recognition engines:

Dragon NaturallySpeaking (DNS), a product of Nuance.
Window Speech Recognition (WSR), as included in Microsoft Windows Vista.

Basically, if you are using Windows Vista or higher this is pretty easy.

Here's a video demonstrating what I'm talking about.



It's kind of buggy at the moment, and it's not developed using NS2's lua functionality, but it kind of works. if there's interest, I can provide more detailed information on how to do this yourself. Right now it's kind of a hack job and it's held together with duct tape.

:edit:
I cleaned it up a bit, broke it down into multiple files.
If you have DNS these are the files you'll need.

http://sourceforge.net/projects/natlink/files/pythonfornatlink/python2.6/pythonneededfornatlink2.6.exe/download
below may need to be run as Administrator
http://sourceforge.net/projects/natlink/files/natlink/natlinktest4.1/setup-natlink-4.1beta.exe/download
https://code.google.com/p/dragonfly/downloads/detail?name=dragonfly-0.6.5.win32.exe&can=2&q=

This is what I used, it has all the python libraries needed, I haven't tested on any other version of Python. If you have a recent version of DNS, those files, along with my files is all you should need to make this work.

http://dl.dropbox.com/u/40823411/ns2_dragonfly_b0.1.zip

Extract that to C:\NatLink\NatLink\MacroSystem or the appropriate location for NatLink.
NatLink is only for DNS, you don't need to install if if you're using WSR.

If you use WSR, I couldn't get dragonfly to work with it. People say it's possible, if you can get it working, maybe you could share how you went about it.

https://code.google.com/p/dragonfly/
http://pythonhosted.org/dragonfly/installation.html#installation

DerAkademiker

Awesome. Both the idea and the implementation.
The buggy behavior seems to come from the voice recognition, not your code...does Dragonfly allow you to train the voice recognition for more accurate prediction?

Feels like watching StarTrek though and seems to make sense to use (there are not much applications where you can say that).

Nice!

Rellik_pt

looks like a nice ideia but during a match i cant see how could work, if i want to warn someone about a location i dont want to my camera to change because i said terminal but just warning the players

DerAkademiker

Rellik_pt wrote: »

looks like a nice ideia but during a match i cant see how could work, if i want to warn someone about a location i dont want to my camera to change because i said terminal but just warning the players

Make it stop listening when VOIP button is pressed?

Caboose

Voice recognition can easily be toggled with a hotkey, at least using DNS. a lot of the bugs that were in that video were because I was talking for the video, and just recording my microphone in general.

I want to try to make it a press and hold hotkey to recognize commands, that way it doesn't interfere with communications. My voice profile also hasn't been used a lot for dictation, I could do more training, that would be a good thing.

This could definitely be better integrated than it is.

@DerAkademiker Dragonfly is just a layer on top of the speech engine (WSR or DNS) any training is done by the engine. You could modify the Grammar and change or add alternate commands if you want.

I'll try to gather the necessary files and upload them with instructions sometime later tonight. WSR may require some experimentation because I don't use it, but I've seen it work for other people.

:EDIT: Uploaded, check first post.

FunkyFungus

could also be interesting for spectator mode (streaming matches)

very nice project

Caboose

I got it to work Using WSR, it's actually very, very easy.

Just copy 'C:\Python26\Lib\site-packages\dragonfly\examples\dfly-loader-wsr.py (or dragonfly-main.py if it's there instead) into a directory wherever you want, then copy my NS2 macros into that ditrectory and run that file. It will look in the directory and run whatever scripts are also there.

Running dfly-loader-wsr.py (double click it) launches WSR, no need to have it already running, in fact, it might not work if it is.

You still need to have that version of python installed and dragonfly (no natlink though)

Then say 'Enable Natural Selection 2' to enable general commands and 'Enable (mapname) commander mode' to get map location changer to work. To switch maps, just say switch map before you enable another maps commander mode.

You may have to set up WSR if you haven't already.

Solaris

This is a very impressive proof of concept. I doubt it will be more "fluent" than using mouse and hotkeys, but it actually might provide disabled/injured players a way to play this game as commander in an actually useful way!

On the fun side, I hereby propose a mod that rebinds the "attack" with the voice-command "pew" if you're a Marine, and "chomp" if you're an Alien.

Hypergrip

Solaris wrote: »

I hereby propose a mod that rebinds the "attack" with the voice-command "pew" if you're a Marine, and "chomp" if you're an Alien.

I would pay money to watch a stream of some devs and/or top players sitting in a room yelling "chomp chomp chomp" and "pew pew pew" at their screens!
(actually fighting in game, of course, not JUST yelling. I have that at my day job too often already)
And have Reddog, Wasabi and/or Blind commentate it.

aeroripper

Pretty interesting, but I don't see myself using it. I'd get it confused with issuing orders to marines over voicecomm. Time is better spent getting quicker at using the hotkeys and clicking around the map IMO.

Calego

That is pretty awesome. Idk how you could do this but being able to tell it to research such might actually make it pretty handy. I love seeing stuff like this being developed. Now we just need a Kinect port so we can exercise while playing!

Caboose

I suppose I could make a function that remembers the cursor position when I place certain structures, then recall the map location and mouse coords when you ask it to research something. It would only work if you place the structure in a predefined location without moving the view though.

If I can find a way to store the current view location, I could do all kinds of cool stuff, including update new maps much easier, which I haven't done for a few builds.