Posted by RKlenka
My Senior design project was something called the “Interactive Mirror System” or IMS.
In short the idea was you stand in front of a mirror, like your bathroom mirror. The system will recognize you via face recognition and log you in. The system will then display summaries of your data for the day, such as Calendar, Facebook, Twitter, News, Stocks, Weather, Music, Health and all of that. You would then be able to use gestures and voice through the Kinect to interact with the system. You could also send emails, update social media, and so on.
The first obstacle was face recognition with Kinect. First idea was to use the Kinect’s depth and skeletal tracking and create our own recognizer. Instead we took the easy way out and just converted the Kinect video feed to work with OpenCV. What we did was then tie that into the Kinect’s skeletal tracker so when the Kinect saw there was a person it would then run face recognition then connect the result to that skeleton. This allowed for tracking and labeling of multiple people while connecting and tracking them to their skeleton. We did it using the color feed, but now that i’m thinking about it i wonder how that would have worked merged with the IR feed, that might have solved some of the other problems we where having… Anyhow getting face recognition to work with the Kinect was pretty simple. So then we had a user matched to a skeleton so this allowed for hand and gesture tracking.
So then the next step, which we thought would be small, was the GUI and APIs to connect and deal with all of the services we wanted. The good thing was they all mostly used the same type of authentication, but pretty much everything else was different. Facebook was the easiest but liked to kick you out and re-authenticate often. Twitter and Fitbit made you enter extra data. Getting it all to work nicely and consistently together was a real pain in the ass. The great majority of time was spent dealing with all this.
This is what all was in it:
- A 32-inch LCD Monitor with 1080p resolution
- Microsoft Kinect for Windows
- A Intel i3 Processor
- Windows Embedded 8
- 8GBs of DDR2 Ram
- 32GB SSD Hard drive
- 6 USB 2.0 Ports 4 outside + 2 Inside
- 2 USB 3.0 Ports (Inside)
- 802.11 Wireless
- Bluetooth 4.0
- Bluetooth keyboard and mouse combo
- One sided mirror
- Stained Wooden Enclosure
- FitBit Aria Wireless Scale
- FitBit One Health Sensor
Outside of the FitBit Scale and Health Sensor i want to say this was about 500 bucks worth of stuff. The great majority of it was just to support the Kinect because it was a system hog. It would work just fine without Kinect due to the voice commands and wireless keyboard/mouse. So you remove the Kinect and you can cut the specs down quite a bit. It would also run just fine on a standard webcam with very little modifications to the back end.
It was running Windows Embedded 7 with the Kinect add-ons. This made it very easy to develop and work on using Visual Studio, Kinect SDK and OpenCV. The only real issue we where having was on Embedded 7 the voice recognition was a bit tricky to get running. It would run no problem on full Windows 7 or 8 but kept getting is driver issues and missing components on the embedded system.
Video or it did not happen
We have a video of it working somewhere. We made it as a backup in case it fails on presentation day. I will post it if i ever find it.
Did it work?
Yea it worked pretty well. It did everything it was designed to do. It would track people, gestures, voice and face. It would display Facebook, email and everything else we it wanted it to. It even had a GLADoS Easter egg where it would make fun of you and say quotes. The GLADoS sections were a hit with the other students.
The only main issue we had was when it was displayed in a public area with a lot of traffic and people. It would only track one person (on purpose, it could actually do many) and it would attempt to isolate that persons voice using the location and face data but with everyone yelling at it to post random things to Facebook and twitter it would get confused.
So why did you do it?
Well originally I wanted to use the extra people and resources to make a heads up navigation for a vehicle based off of OBDMobile but everyone wanted to do something new. Our second choice was a interesting fitness app that involved running and music. But one of our TAs said “I dont like running so your not doing that” and kicked it away. So our third choice was this. The idea was we could use Visual Studio, WPF and Kinect SDK to make something cool and interesting (and easy!). So in actuality this was a pretty easy project that was designed to look and sound harder than it was 😛 .
Where is it now?
We gutted it.
We pretty much split what was splittable with next semesters teams. The kinetic itself didn’t even last the day before i new team stole it. Another group needed the Wifi and CPU and last i heard is they used the TV to play some Smash Brothers.
You can view our “User Manual” here: http://www.rklenka.com/projects/IMS/IMS_Setup&UserManual.pdf
Tech Documents: http://www.rklenka.com/projects/IMS/MirrorsInc_SRS.pdf http://www.rklenka.com/projects/IMS/MirrorsInc_STP.pdf
Due to a issue the night before our 8am final presentation I ended up staying at school all night. In our audience for the SD presentation are a lot of rather important people from the Engineering department. Part of the presentation involved a Fitbit fitness tracker, which is this little clip on item. I click that on my pants pocket, right above where i clip on my 5 and a half inch Voyager knife. So being half asleep during my presentation i reach to show the Fitbit tracker, but instead i grab the hilt of my pretty large knife. Luckily i caught myself just before i pull out a huge knife in front of a bunch of important people.