Digital Dispatches: Prototyping an Interactive Time Line for Harvard GSD 075

The landmark exhibition at Harvard’s GSD culls from 75 years of teaching and practice, showing seminal artifacts in plexi covered cases with vertical annotations done in journalistic style. The challenge put to the design team at INVIVIA was to develop an interactive timeline placed at the entrance to the exhibit that would allow visitors to gain a quick temporal context or an in-depth view of a specific period or set of artifacts. The interactive table had to be intuitive to use, show graphic and textual detail in high resolution, support multiple users and be quite robust. Oh, and it had to be finished and installed in 2 months!

We looked at several options. Using a large touch sensitive table was attractive in that it would allow many users and provide a natural interface that would make it easy to browse subjects in depth.  But the cost and complexity of the large interactive surface seemed prohibitive.  A simpler approach, that quickly became the favorite, used top projection onto a matte white surface (plywood covered with adhesive backed matte vinyl) and multi-blob detection to notice where the users hands were. This approach doesn’t allow you to notice whether the user it touching the surface or above the surface, so selecting/deselecting an object becomes difficult.  So we came up with an interaction scheme using different horizontal zones that had different interaction meanings.  The computer vision challenge is to notice only the user’s hands even though there is a continuously changing projected image on top of them.  This gets solved by flooding the table with IR light and filtering out the white light to the webcam with thick Neutral Density filters.

IR image

The IR image with multi-Blob detection

White-light view

White-light view of table with projection

The multi-Blob tracker was a repurposing of Python code used for many touch table applications, based on Markus Gritsch’s VideoCapture python module (http://videocapture.sourceforge.net/). The vision algorithm starts with capturing a base frame and then subtracting each subsequent frame from it to find what moved on the table. In this difference frame we define a Region of Interest and look for candidate blobs.  These candidate blobs are flooded and culled to remove too small blobs generated by noise in the camera signal. Each real blob is measured and these measures are written to a file. Then this process is repeated for the next frame.

We decided on full HD 1920 by 1080PDLPprojectors because of the high visual quality of theDLPimage, the fact that there are many competing projectors in this market segment which keeps the price down, and the relatively high resolution.

We built a crude but effective prototyping area in INVIVIA’s recently renovated top floor space by supporting a 16ft, 2by12” board between the top of the AC control room and a 10ft ladder mounted on tables. This allowed us to quickly build projector housings out of MDF with a bandsaw and drywall screws and attach them to the board with drywall screws and eventually with sliding supports for fine position adjustments.

These projectors must be operated nearly horizontally or they will overheat so each projector housing had a 45 deg. mirror attached to angle the image toward the table below.

Initially we hoped to use 3 projectors arranged horizontally giving us a 5760 by 1080 total image size, but realized after initial testing this that stretching the 1080pixel image across the 32in dimension of the table would not handle the small type we hoped to annotate the images.   We wound up buying an additional projector, turning each projector 90deg. so that the 1920 pixels stretched across the 32in table and shrunk the table length to 4320 pixels for a ratio of 2.25:1.

Installation view

Installation view of projector/computer tray and table

Projector sled

Projector sled with mirror

The final aspect of the projection setup that needed to be worked out was the IR emitters.  We chose rectangular array emitters normally used outdoors to illuminate parking areas for nighttime surveillance.

The final installation used a well reinforced plywood projector/computer tray hung below a false ceiling.  Openings were cut for the projector light, the IR emitters, and a webcam. Each projector was mounted on a slide that held the 45deg mirror and allowed easy adjustment of the critical space between projectors that almost gave a sense of one continuous image.

Advertisements

Z-Depth Touch Table Proposal

Recent INVIVIA brainstorms and discussions have pointed to the desirability of adding Z-Depth to the Touch Table idea. The least obtrusive way to do this would be to illuminate the hand from below the screen and use stereo cameras situated there as well. I’ve made numerous tests of this approach using different screen materials and illuminator positions.

What I’ve learned is that the ideal screen material would be a great diffuser of the white projector light and yet be transparent to the IR light illuminating the hand and provide a clear view for the IR camera. I have not found such a material. What I find instead is that a terrific screen material is almost opaque to the IR source and IR camera, and a material that functions well for the IR camera is a terrible projector screen.

To illustrate the problem I used frosted mylar, the material that is the closest I could find to fulfilling both criteria: allowing the cameras to see through somewhat and still provide an adequate projector screen.

So here’s what the camera saw, with the hand touching, looking at the underside of the frosted mylar illuminated by a couple of rows of IR LEDs (MS Surface works something like this, I assume)

zdepthhandilluminationcameraviewlarge.jpg

When the hand is slowly pulled away from the mylar (here at 1inch increments) the diffusion of the image (very good for the projection) blurs the hand so that at 4inches away it is no longer recognizable, not by human eyes and certainly not by a video camera.

zdepthcameraviewhandilluminationseq.jpg

 

If all we needed was 2inches of hand travel we could probably make a frosted mylar screen work.

Looking back at the ZDepth stereo demo from last year is instructive. It shows that you need at least 8-10 inches of depth of movement in order not to severely constrain the user. (the cameras have the blue lights and are located above the screen)

zdepthdemoclosefarexamples.jpg

 

Here’s what the stereo cameras see:

pythonhandfollowerstereoviewnearfar.jpg

 

The Proposal:

 

Until we find the magical screen material which diffuses in white light and is transparent in IR, I propose that we adapt the approach I used successfully in the Hand Follower demo above to the Touch Table environment.

My idea is that we build (or at least think hard about building) a Z-Depth module that is simply a small wide angle USB video camera, a stacked and angled fan of IR LEDs, and a housing to protect them (and in this case hold the connector to get the signals into the table.)

zdepthtopflatfantogether.jpg

Here are other housing variations:

If we can’t find a suitable small USB camera, this housing is big enough to cover the existing Logitech version

zdepthsideflatfantogether.jpg

And, of course, there is the obligatory Egg variation:

zdeptheggfantogether.jpg

…and this final (not very well crafted) variation where the parts are partially hidden in the 1inch table depth and clunky plastic parts are traded away for lots more veneer…

zdepthtableinsetledfanscameratogether.jpg

….obviously, ideas for other variations (or changes to these variations) are welcome….

 

Solving the stray window light problem:

When you look again at the vertical ZDepth demo pictures you realize that they were made in a pretty controlled environment where stray light was easily controlled. The INVIVIA conference room is another matter entirely.

My living room has windows big enough to offer a similar experimental environment for testing. The problem that has to be dealt with is that at least one of the cameras on the table corner will likely be looking across the table to a well lit window. Blacking out the window entirely is not an option. What is needed is a way to modulate the IR light coming through the window without drastically changing the white light amount or color.

What I found was a family of heat absorbing commercial window films, intended to keep heat out during the summer and in during the winter that do the job pretty well.

windowfilmtestsmall.jpg

 

Here you see the video camera (blue light to the right of the screen) looking at my hand and the window, to the left of the screen is the IR Illuminator. In the video capture window on the screen the only things visible are my hand and the source, even though the bright window in the background is in view…

Apparently there are non reflective versions of this film that would decrease the surface crinkly effect on the film.. That version was not available in Home Depot…

 

 

 

 

 

Flush Touch Screen, version 5


The version 3 Touch Screen’s geometry was greatly determined by the thickness of UV coated polycarbonate sheet I had found at HomeDepot that seemed to do the best job of aiding the FTIR effect as well as providing smooth finger glide across the screen. Since the sheet was .095″ thick and the LEDs are .2″ in diameter LED guide had to have a substantial bevel in order to align the LED and the screen.

I have since found thicker UV coated plastic after a short web search and determined after a couple of quick prototypes using existing simple LED guides that the thicker sheet did in fact exhibit satisfactory FTIR qualities. It seemed right to design a flush touch table LED guide that would allow us to inset a touch screen in an existing table, mount the projector and camera underneath and begin in-house touch table experiments.

Here is the AutoCAD side view of the LED guide:

flushtouchtableautocadsideview2shrunk.jpg

Since this is just a (quick and dirty) proof of concept prototype, I only machined enough of an LED guide to hold 18 LEDs or about 5.1″ worth of LED, (twice, one for each side)…

flushtouchtableprotoshrunk.jpg

A closer view of the flush edge LED guide:

flushtouchtableframecloseupshrunk.jpg

The side view, seen if screen extends to edge of table…

flushtouchtabletestimage.jpg

All this assuming that we might end up with a table that looks something like this:

flushtouchtableblackedgesideshrunk.jpg

Anyway, I will bring both the beveled and flush prototypes in to evaluate…

Inside the LED Guide

Just to make whats happening with the LED guide a little clearer, here’s the underside of the guide with the LED PCB board in the background. Note that the LEDs have to turn the corner in this version.

flushtouchtableledguideundershrunk.jpg

Here the LED PCB has been put in place (with some difficulty, ’cause there’s not much room in there)

flushtouchtableledguideunderpcbshrunk.jpg

Finally, a shot looking from the side the touch screen and video screen see. I discovered after this version 4 proto was made that the holes need to be deeper in order to mitigate the strong side illumination lobes from the LEDs. (Sadly I didn’t get a picture of the version 5 deep holes until the whole thing was taped together)..

flushtouchtableledguideholesshrunk.jpg

Version 3 of TouchScreen…

Oct 16, 2007

I thought you’d all like to see the first integrated screen and LED Guide working…

Whats different from previous versions, you might ask?
– LED Guide, touch surface, video screen and backing sheet are now in one strong, reasonably nice looking frame
– LED orientation gives even light distribution across screen using only side arrays
– Heat dissipation into frame is reduced and now manageable.

what needs to be done next?
– Narrow beam LED circuits need to be made and tested. As it stands I am 2 circuits short of filling the screen with light and
found some narrow beam, very bright LEDs that should be tried… they will reduce spillover and may improve performance.

– decide on housing shape (table, kiosk…convertible?) and build it (or whatever variations seem to make sense) so you guys can have a new toy to play with and we can begin to get some applications developed and running on it…

here are some development process pix:

the AutoCAD cross section

Milling the slot to hold the touch surface…

just finished cutting the shelf that makes connection to next piece

the four pieces before cutting off clamp ends

partially assembled showing LED circuits..

……more as it happens…..

cheers, ronmac

Draft of Touch Screen Instruments

We’ve decided that an instrument is the most interesting way to go re: the touchscreen interface.

Here are a few xylophones. Their shape is designed to look like a xylophone but they aren’t too specific or refined. Any feedback would be appreciated, e.g. if you think they need to comply to certain specifications, etc. that would be helpful. After I hear back from you, I’ll draw up another prototype and we’ll go from there. Thanks!

instrument-prototype-touchscreen.jpg

Invivia’s Research Blog Launches!

Welcome to Invivia’s informal blog for their ongoing research projects!

Edith’s Notes from 7/12

INVIVIA multi-touch displays: what could they be? beyond surface! We imagined different shapes and textures of displays, at different scales and orientations. We though of possible activities—or experiences—that, more than others, scream for X-touch displays (ex: things people like to do using 2 hands, ten fingers, or working together, coordinating touch-points and traces among themselves; transactions that become “magical” when mediated through touch; objects that unveil themselves in unusual ways when placed on touch-screen… In this note, I started to list a few projects / ideas under different categories,

Categories:

  1. form – scale up / scale down — horizontal/vertical/slanted
  2. form – bend / wrap / fold – from flexible to rect-verso to3D
  3. Experience – playing music, data visualization/manipulation
  4. other areas to explore – art installations, science,

___________________________________________

Short video [13MB] (pretty old)
http://cs.nyu.edu/~jhan/ftirsense/
http://www.perceptivepixel.com/
http://www.ted.com/index.php/talks/view/id/65
Han, J. Y. 2005. Low-Cost Multi-Touch Sensing through Frustrated Total Internal
Reflection. In Proceedings of the 18th Annual ACM Symposium on User Interface
Software and Technology

Scale up / scale down – Horizontal / vertical / slanted
UP- multi-touch walls / boards (vertical), tables / floors (horizontal)
http://www.engadget.com/2007/06/24/prometheans-interactive-whiteboard-gets-multiinput-upgrade
http://technabob.com/blog/2007/06/09/worlds-largest-multi-touch-screen/
when horizontal, boards and walls become tables, pools, and floors. You can use feet (stepping), wheels (rolling), or skates (gliding) to navigate. You can dip fingers in ewater and see circles on the surface. You can put objects on surface, play around with them as if they were real (photos, cameras, albums, maps). You can manipulate traces.

You can transfer contents between objects put on table: ex. Digital camera, download pics (cf. surface). The bigger the floor-display, the more options you have to immerse yourself, with your props, like an actor on a stage.

DOWN – multi-touch-tiles / mini-screens (portable devices)
EX: Nintendo http://www.engadget.com/2007/07/11/nintendo-wii-fit-gets-you-stepping/
Ex: Wii Fit. Wii Balance Board with dual sensors detects your weight and balance on each side as you exercise. The board is wireless, and holds the potential of full-body games involving the Wiimote and the Wii Balance Board simultaneously. If you’ve had enough gaming, you can also have the Balance Board diagnose your body mass index. imagine multi-player scenario if attached on ceiling, tile could be engineered to sense pulls instead of pushes

Bend – Wrap – Fold (1)
Flexible large-scale: From walls to drapes and curtains
Khronos – Alvaro Cassinelli. NextFest 2006
http://www.technovelgy.com/ct/Science-Fiction-News.asp?NewsNum=912
Khronos Projector uses a flexible sheet screen; the projector itself stores a video of a single scene over time. When you press on the flexible screen, the projected scene varies in time. (cf. video on website)

[related] the energy curtain – Touched by the sun… The energy curtain captures energy from sunlight, returns light at night
http://www.technovelgy.com/ct/Science-Fiction-News.asp?NewsNum=798
It is the product of a research project funded by the Swedish Energy Agency. It was demonstrated at the Wired NextFest 2006 festival of technology at the Javits Center in NYC about a month ago Flexible small scale: From tiles to patches – from portables to wearables I imagine multi-touch / multi-sensor patches, bandaids, jewelery, and gloves…But then, again, I may be way out there given the technology we work with.
http://www.technovelgy.com/ct/Science-Fiction-News.asp?NewsNum=1093

——————————————————————————-

Cool website; technovelgy – where science meets fiction
http://www.technovelgy.com/ct/Science-Fiction-News.asp?NewsNum=797

Bend – Wrap – Fold (2)
Two-sided multi-touch screen — Ex: Under the table interaction (UIST 2006) Two-sided touch-table allows input from both bottom and top
http://www.youtube.com/watch?v=fDDhbHrLSnk
It would be great to have a large-scale two-sided multi-touch screen Screen could be placed vertically or horizontally (front/back or over/under) It could be “see-through” or not. All kinds of activities can be imagined. 3D multi-touch screens (DOME, column, cylinder) this is a huge area to explore. Could be worth further exploring

Activities
Playing music
Digital DJing –
http://module-records.blogspot.com/2006/01/hawtin-wants-multi-touch-screen.html
Jazz editor
http://www.jazzmutant.com/lemur_overview.php
drawing music…
here goes a cool pre-multi-touch-sensor version……
http://www.technovelgy.com/ct/science-fiction-news.asp?NewsNum=754
creation of Japanese musician and artist Miyako Dub Squall The musician/artist draws with different colors on a standard LP record-sized blank piece of paper. The sensors in the arm of the turntable can distinguish between five different colors (black, red, yellow, blue and green). As the turntable spins the ‘platter’ the music plays

Information visualization and manipulation
Multi-touch sensing on LED matrix displays
http://cs.nyu.edu/~jhan/ledtouch/index.html

Art Installations
Media Mirror
An interactive video installation, in which 200 channels of live cable television are continuously rearranged in real-time to form a mosaic representation of the person that stands in front of it. (in progress)
http://cs.nyu.edu/~jhan/mediamirror/index.html

TraceEncounters
A minimalist social network tracking pin one thousand pins distributed at Ars Electronica 2004 (with Brad Paley, Peter Kennard)
http://www.traceencounters.org