Category Archives: School

Senior Design Project – Interactive Mirror System

Interactive Mirror System

My Senior design project was something called the “Interactive Mirror System” or IMS.

In short the idea was you stand in front of a mirror, like your bathroom mirror. The system will recognize you via face recognition and log you in. The system will then display summaries of your data for the day, such as Calendar, Facebook, Twitter, News, Stocks, Weather, Music, Health and all of that. You would then be able to use gestures and voice through the Kinect to interact with the system. You could also send emails, update social media, and so on.

mirror

The first obstacle was face recognition with Kinect. First idea was to use the Kinect’s depth and skeletal tracking and create our own recognizer.  Instead we took the easy way out and just converted the Kinect video feed to work with OpenCV. What we did was then tie that into the Kinect’s skeletal tracker so when the Kinect saw there was a person it would then run face recognition then connect the result to that skeleton. This allowed for tracking and labeling of multiple people while connecting and tracking them to their skeleton. We did it using the color feed, but now that i’m thinking about it i wonder how that would have worked merged with the IR feed, that might have solved some of the other problems we where having… Anyhow getting face recognition to work with the Kinect was pretty simple. So then we had a user matched to a skeleton so this allowed for hand and gesture tracking.

Inside IMS

 

 

mirrorCutout

So then the next step, which we thought would be small, was the GUI and APIs to connect and deal with all of the services we wanted. The good thing was they all mostly used the same type of authentication, but pretty much everything else was different. Facebook was the easiest but liked to kick you out and re-authenticate often. Twitter and Fitbit made you enter extra data. Getting it all to work nicely and consistently together was a real pain in the ass. The great majority of time was spent dealing with all this.

gui2

Hardware

This is what all was in it:

  • A 32-inch LCD Monitor with 1080p resolution
  • Microsoft Kinect for Windows
  • A Intel i3 Processor
  • Windows Embedded 8
  • 8GBs of DDR2 Ram
  • 32GB SSD Hard drive
  • 6 USB 2.0 Ports 4 outside + 2 Inside
  • 2 USB 3.0 Ports (Inside)
  • Ethernet
  • 802.11 Wireless
  • Bluetooth 4.0
  • Bluetooth keyboard and mouse combo
  • One sided mirror
  • Stained Wooden Enclosure
  • FitBit Aria Wireless Scale
  • FitBit One Health Sensor

Outside of the FitBit Scale and Health Sensor i want to say this was about 500 bucks worth of stuff. The great majority of it was just to support the Kinect because it was a system hog. It would work just fine without Kinect due to the voice commands and wireless keyboard/mouse. So you remove the Kinect and you can cut the specs down quite a bit. It would also run just fine on a standard webcam with very little modifications to the back end.

Software

It was running Windows Embedded 7 with the Kinect add-ons. This made it very easy to develop and work on using Visual Studio, Kinect SDK and OpenCV. The only real issue we where having was on Embedded 7 the voice recognition was a bit tricky to get running. It would run no problem on full Windows 7 or 8 but kept getting is driver issues and missing components on the embedded system.

Video or it did not happen

We have a video of it working somewhere. We made it as a backup in case it fails on presentation day. I will post it if i ever find it.

Did it work?

Yea it worked pretty well. It did everything it was designed to do. It would track people, gestures, voice and face. It would display Facebook, email and everything else we it wanted it to. It even had a GLADoS Easter egg where it would make fun of you and say quotes. The GLADoS sections were a hit with the other students.

The only main issue we had was when it was displayed in a public area with a lot of traffic and people. It would only track one person (on purpose, it could actually do many) and it would attempt to isolate that persons voice using the location and face data but with everyone yelling at it to post random things to Facebook and twitter it would get confused.

So why did you do it?

Well originally I wanted to use the extra people and resources to make a heads up navigation for a vehicle based off of OBDMobile but everyone wanted to do something new. Our second choice was a interesting fitness app that involved running and music. But one of our TAs said “I dont like running so your not doing that” and kicked it away. So our third choice was this. The idea was we could use Visual Studio, WPF and Kinect SDK to make something cool and interesting (and easy!). So in actuality this was a pretty easy project that was designed to look and sound harder than it was 😛 .

Where is it now?

We gutted it.

We pretty much split what was splittable with next semesters teams. The kinetic itself didn’t even last the day before i new team stole it. Another group needed the Wifi and CPU and last i heard is they used the TV to play some Smash Brothers.

You can view our “User Manual” here: http://www.rklenka.com/projects/IMS/IMS_Setup&UserManual.pdf
Tech Documents: http://www.rklenka.com/projects/IMS/MirrorsInc_SRS.pdf      http://www.rklenka.com/projects/IMS/MirrorsInc_STP.pdf

Omake

Due to a issue the night before our 8am final presentation I ended up staying at school all night. In our audience for the SD presentation are a lot of rather important people from the Engineering department. Part of the presentation involved a Fitbit fitness tracker, which is this little clip on item. I click that on my pants pocket, right above where i clip on my 5 and a half inch Voyager knife. So being half asleep during my presentation i reach to show the Fitbit tracker, but instead i grab the hilt of my pretty large knife. Luckily i caught myself just before i pull out a huge knife in front of a bunch of important people.

gui1

Advertisements

Schools Over

Well i am finally done with college, I get to walk the stage on Sunday and pick up my Software Engineering Degree. I have already excepted a full time position working on what should be some cool projects.

So then what now? Well activity here will probably stay about the same as it is (relatively nonexistent). Im still waiting on the Sprint Windows Phone 8, though i am playing with the idea of just going and picking up a cheap prepaid Windows Phone 8 like the one Walmart is selling: http://www.walmart.com/ip/Nokia-Lumia-521-4G-Smartphone/24099994 to work on it. I also have a list of small projects i would also like to work on.

My senior design project, which took up most of my time this last 6 months or so, was a”Interactive Mirror”. It was a mirror that you stood in-front of. It would know someone was in front of it then do a face recognition scan to see who you are. If recognized it would log you into and display a GUI for a whole bunch of stuff such as Facebook, Twitter, Fitbit, News, Weather, Calendar. and all kinds of other cool stuff.  It had voice control, voice dictation, hand gestures, mouse control, mouse and keyboard support and so on. It played music, it displayed images. I of course was dumb and did not take that many pictures of it, nor did i take any video of it running:

Interactive Mirror SystemInside IMS

Learned a lot about the Microsoft Kinect and there things so don’t be to surprise if you see some write ups about how it was done.

I have a few weeks until Work starts so ill try and get some cool stuff done by then.

Winter Break

Its winter break now, so i try to come up with a todo list for the next month or at least try and attempt to be constructive.

I had a fire sale on my BT connectors, wanted to clear out stock and i did, even threw in free copies of OBDMobile to hurry it along.

Once that was done, and now i know that WP7/8/RT/Surface does not openly allow ad-hoc or blue tooth serial port profile i told my self i would make OBDMobile free so i did.

The one on Intel’s AppUp though i still need to “free”, but i think i sold like 2 copies on there so i guess there is no hurry.

A “jail broken” ROM for my phone came out in the last month or so, and the guys of XDA has figured out, from what i know, how to access the serial port among other things so my main goal for this break is to get a home-brewed version of OBDMobile running on Windows Phone 7. Stuff like this is what i will always miss from Windows Mobile 6 is the ability to do almost anything on it, but i guess that’s the trade off for having a enjoyable UI.  From what i can tell Microsoft has been trying to get away from BT SPP for years now, but with all the parts already there i don’t see why they cant keep supporting it. My guess it must be a security thing, in class in the Senior Design lab i was easily able to connect and mess with another teams blue tooth thing that allowed serial port access. Which i guess was not to nice since they started freaking out and they had a big presentation the next day for it lol.

Speaking of Senior Design, my group is wanting to work on that all break and being the unofficial “tech lead” of the group might mean ill have to help 😛

As for SideAmp, which i guess its a project only i seem to love, it is now pretty much unneeded anymore. MS killed off SideShow, which i have been toying with porting to Windows Phone 7/8 but there is not really a big reason to. You can still install SideShow on Windows 8 and even SideBar gadgets which i find more useful in a desktop setting than Live Tiles are but they are unsupported. After updating to Windows 8 x64 my sideshow drivers on my devices no longer work, and though i asked for updates, they are not interested. I played with updating the drivers myself but W8 has some parts that make it not particularly fun. Though i can still access it on non-sideshow ways and i might port a smaller copy of SideAmp for just that purpose for Zune and WinAmp but im pretty sure some one has already done something similar a long time ago.  Thought about making a WinRT SideAmp app but if  Zune, i mean “X-Box Music” is still a mess then i dont see the point in it.

I am still a bit surprised that they abandoned SideShow, at least without replacing it with something better. I see plenty of apps on the marketplace that involve some sort of “driver” or a method of getting data between the phone and computer, something SideShow was pretty decent at. I know MS wants people to interact and use Azure which lead to some pretty stupid discussions with MS when WP7 first came out.  WP8 and its NFC communications stuff, and with Nokia actually trying things with it like in its overly expensive charger speaker dock system. Maybe someone should develop a general NFC usb connector thingy, though i know some laptops have something built in now.

Oh well, in the end i will probably just sit around drinking hot chocolate and blowing all my time and money on Steam’s Winter sale.

Spring Semester

Best laid plans right?

Well as i was hoping wouldn’t happen, XMAS got me into holiday mode and with a mix of good Steam Sales and a back log of Netflix and Anime i pretty much did nothing constructive over break.

I spent a whole week just marathoning games, such as Crysis (and warhead), SW: Empire at War(surprisingly fun!), FEAR 2, Bullet Storm, Homefront (was lame), Chantelise, Battlefront 2, Red Faction G, and others. To then spent the next week marathoning anime: Air, Air Movie(better than the show), Kanon, Clannad, Clannad AS, Hen Zemi, Hurahi-chan, and a few missed specials from Elfen Lied, Clannad, K-On and caught up with Wafku, good times but none productive.

So now school is starting back up, had planned to have a OBDMobile for Android and SideAmp for WP7 running by now but nope. Now i have about 16 hours of classes set for this semester. Good thing is that most of them are related to projects i am working on or interested in so that should be very helpful. In fact some can be used for extra credit.

Of course school comes first but other than that i should be able to become productive again. Im sure some of these i said last time but my first goal now is to get a decent “demo” disk to ship out to replace the crappy factory one with the OBD Connectors. Then i want to revise the debugging and licensing stuff for OBDMobile and try and make it officially “Windows 7” certified. Then have a Android version running around (one school project is to basically convert a C# program to Java how convenient) then with a Windows/Android/Whatever else base is to then advertise it everywhere to get feed back and reviews. Then use that data/feedback to do OBDMobile 2 over the summer. Along that time i will have some smaller WP7 and .net projects every now and then. If i keep on it right then i can do the projects along the school projects and get two birds with one stone.

On another good note is that i have noticed more and more Windows Phone 7’s on campus. I can only think of one out in the public, but i have seen quite a lot on campus.

So should be a good semester, with a slight chance of graduating in the Fall.

Code Signing

Yay yesterday i got approved for my code signature, so now everything has my name on it. I went and signed i think all the software downloads here. For those who are interested a good guide is: http://www.wintellect.com/cs/blogs/jrobbins/archive/2007/12/21/code-signing-it-s-cheaper-and-easier-than-you-thought.aspx he walk you through the process. Now the calling aspect is out sourced and took about 2-3 weeks. So when they ask for a good time to call to verify they dont mean today or tomorrow its more like the next few weeks. Also there is a simple GUI Wizards for code signing that makes it very easy. Also know that Microsoft doesnt except Comodo signature for Certification, but the word is they want you to use verisign for personal verification, and doesnt care what signage you use.

Also means i can, and did, submit OBDMobile to Intel’s AppUp once that goes through im hoping to have at least enough of a footprint to get a hold of some hardware to work on a Windows Phone 7 version. Im also going to then start rounding up updates for OBDMobile depending on how it works with Intel. I am wanting to make it more touch and tablet like for Windows 8 which should work smoothly with WP7/8 development.

Intel showed some interest in SideAmp but i dont think they fully understand it, its a plug in for WinAmp and Zune, not a stand alone Media Player. Now right now it is dedicated to just SideBar and SideShow, which are not big successes for Microsoft and Sideshow for example has been pretty much killed off. I might roll SideAmp into say a more touch based, tablet media base but i bet MS already has that covered for Windows 8, but as long as my PicoLCD device works ill keep using it (drivers doesnt work on x64, though by then ill get desperate enough to roll my own) I even played with the idea of making some WP7 port of SideShow and still might (wheres my BT API Microsoft!)

Ether way it should be a pretty interesting semester with all of this and school plus other projects.

SideAmp 1.2

SideAmp 1.2 is out, it adds Zune support and also can display and save album covers from the mp3tags.

Right now i am trying to finish up my major projects, SideAmp is done for now, OBDMobile is up and running, all that is really left is finishing up RingTimer for WP7.
Oh and code signing everything when Comodo finally approves it (going on week three now of waiting). Which is funny because Intel would have given it to me for free, and its not even the one Microsoft wants for its Certifications.

Im trying to get all that done, not only because i need to, but mostly to get “ready” when school projects start adding up. I have a bad tendency of getting fixated on a project, and only one project at a time. This last month has been OBDMobile, today was SideAmp, tomorrow will probably be RingTimer. Then hopefully it will move over to school work, but probably back to new projects, like Adding Windows Phone 7 support to SideAmp, adding Windows 7 Sensor API and other such things to OBDMobile, finishing the GUI for OBDMobile for WP7 for when BT or AdHoc is ready. Plus a list of others. Its funny you can tell where i am by my weight. When im fat then you know i am working on a bunch of projects either for school or for me. When im skinny then i caught up and have nothing new so i work out all the time, your talking i can go +/- 30 pounds in a year easily. Take a guess what side i am on now (poor Wife)

Summer Time

Well it is now summer time, finals are over and i think i did OK, official grades will be announced next week.

Summer classes isnt for a few weeks so im hoping to get a lot of development work done.

Most of the next update for OBDMobile is done, mostly final touches and GUI optimizations because its a bit of a mess.

The netbook version of OBDMobile should come out quickly after words, i dont know if i want to wait until they are both done and have some way of integrating the two since they will run off like %90 of the same code or what. I will then start working on the ground work on the Windows Phone 7 version, the problem with that one is that they have not released a blue tooth API for it yet, meaning it might only work with the WIFI version if at all so that’s kind of a downer. I can think of a few ways of getting it to work but none of them are all to pretty. I also want to update Ringtimer, FlickPick and the others and get them out on WP7. People keep asking me for the P90X app so i guess i should release that too. I need to figure out what to do with SideAmp since Microsoft abandoned SideShow. I know a guy who worked on a android/WP& plug in but i have not heard anything from him in a wail. Intel though who have contacting me at least once a month since like forever to get SideAmp on their AppUp marketplace i keep saying that SideAmp is nothing to special and that OBDMobile is better for netbooks but he doesn’t seam to agree. When i gave him a little test demo which i knew i should not have done on both SideAmp and OBDMobile for Netbooks (not connected to a car or gps or anything) it was a huge fail for both programs, lessons learned i guess. Oh well.

Its summer, i should have passed everything and i have some cool projects to work on so im excited, at least until summer classes start.

Finals and Adsense

Well one more day of Finals, two tests left to take.

When i should have been studing i keep adding little things to the site, set up this blog here, comment system there and i then noticed that my Google Adsense is no longer showing up.

Well it looks like it stopped working some time in November of 2010 and i am just now noticing it in May. Well i guess that what happens when i use Ad-Blocker on my own site.
I guess they change the code sometime and i never received a email about it, according to the records they owe me about 12 dollars in past revenue. Oh well ill fix it tomorrow.

Adsense site works fine in IE but not at all in Firefox how amusing. Microsoft is running some contest for student developers through June im hoping to get in on some of that action. Im having a bit of trouble it seams with Microsoft’s Marketplace, its been saying that i have 0 sales ever which i know is a mistake, but i guess its a known cause since like April 5th. They say they are working on it, i wonder if it will just start from zero or will we regain ‘lost’ numbers. Its always something i guess.