I'm starting a blog to talk about a hobby project I'm undertaking for my house. The concept and basic implementation is best explained by showing a mock-up in action. Check out the video link below!
Why? Ok, now that you've seen the concept video - you may wonder about the why & how. To answer the why question, whenever I go somewhere beautiful I wish I could take that view home with me. I can take a photo, but when I get back home and look at - I just don't get the same feeling. To make something appear real, I need:
Motion - the more frames per second that can be shown, the more realistic movement looks
Immersion - the image needs to surround you. At any point in time, we see ~110 degrees around us which a small picture or screen fails to recreate
High resolution - The human eye can decern close to 0.3 pixels per arc minute. Low resolution loses realism.
Sense of depth - We take depth ques through many means, one important que for distant objects is motion parallax which occurs when objects change position more slowly the further away they are from you.
Sound - Audio plays in important role in recreating a scene as well as proving some depth ques.
Some of my favorite views:
How?Ultra-high resolution isn't a new idea, today's newest movie video projectors use resolutions up to 4096x2160. While IMAX film can supposedly resolve vertical lines at nearly twice this resolution. Red digital cinema sells digital cameras that can film at 4096x2160 (8 megapixels at 24fps), and has announced plans to sell a 28000 X 4000 camera later this year (112 megapixels) at 30 fps. My goal is to achieve somewhere around 100 megapixels at 60 fps which more than double announced cameras from red.com. To accomplish this resolution there are several challenges for recording, editing, and display.
Filming. At this time, I'm thinking the least expensive option is to buy a bunch of low-end HD video cameras and configure them such that each one films a part of the scene, and then stitch the video frames from each camera together to form a larger frame. 100 megapixels could be accomplished for about $30,000 which is considerably cheaper than an IMAX camera (up to $500k new), Red camera ($30k for 1/5th the resolution). As well, prices at the low end of cameras come down significantly faster than the high-end. I'm still looking at cameras but my leading choices right now are Aiptek Action HD GVS and Sanyo VPC-FH1 HD.
Aug. 30, 2009
Updating with some related links
A True Virtual Window, A thesis by ADRIJAN SILVESTER RADIKOVIC
A PHD student who describes how he implemented a similar idea. The main difference between his implementation and what I'm targeting are:
- He used a static image, I'm planning to do video at 60fps
- I'm planning higher reolutions than he used (100 megapixels versus 67 megapixels
(100 megapixels at 60fps may be a pipe dream, but I'd like to try!)
- He used only one camera for head tracking sampling at a fairly low framerate, I'm planning to use 6+ sampling at 100fps
- He performed head tracking using the normal visible color specture, I'm planning to use infared lights and sample images in infared color specture.
- He used a pan and zoom camera for head tracking, I'm planning on using static cameras.
- He used only one video display and computer for rendering, I'm planning on using multiple displays and computers
Panoramic Video Textures
A method to create high-resolution looping video is presented using a single panned video camera.br<>
This technique is very interesting, but is limited in the types of scenes that can be recorded and replayed.
Sept. 7, 2009
Results of some initial filming test
Over memorial day weekend, I went to a few spots around San Francisco and filmed some scenery using Sony's HD Handycam (1920x1080). Since I only had one camera, I filmed different parts of a large scene at different time intervals - this will create artifacts at the seams for moving objects but should give a rough feel for how the final results will look. I learned a few things using this borrowed camera:
30 fps looks good most of the time, but definitely noticeable for faster moving objects. 60 would look a bit nicer
Interlaced video is unacceptable. You really need progressive frame captures to ensure you don't get ugly
tearing artifacts
The more motion in the scene the more interesting it is too watch. I found getting close to the ocean and filming waves looks great compared to far away shots, however too close means depth perception won't match reality when projected onto a 2d surface.
My quad-core Xeon doesn't has enough horse power to decode 4 1920x1080x30 Mpeg2 streams
simultaneously, I need to investigate where the bottlenecks are coming from
Anyway, I updated the video above to include some of my test footage, check it out (jump to 1:30 inside the video to see what I filmed today).
A month ago, I got the book
Practical Multi-Projector
Display Design and read it cover-to-cover with great interest. I
highly recommend the book as it was pretty easy to follow compared to the usual
thesis papers you get from PhD students. The basic idea is to
calculate geometry and luminance mappings from screen space to projector space
by projecting various patterns, taking a photo of the screen with a high
resolution camera, and then determining how the original patterns actually got
mapped. This mapping information can be expressed as triangle strips when
projecting onto non-linear surface such as curved walls. During rendering,
transforming triangle strips can be accomplished with very little overhead.
Luminance and color correction can also be specified at points in the triangle
mesh while the alpha component can be used to blend the edges of two displays
together to make it look seamless.
The company Scalable Display has
implemented many of the techniques outlined in Practical Multi-Project Display
Design and using their software would be a great way to save time on the
project.
Coincidently, I was contacted by a company called Scalable Display and
recently meet with the CEO and Founder of
Scalable Display (Andrew and Raj).
They expressed interest in doing some joint development on this project. I
got a demo of their system with using 2 projectors similar to
this YouTube demo.
They said they are working on one 100 megapixel project with the Navy which was
pretty exiting to hear. I'd love to see that system in action.
Scalable Display's largest commercial market is military simulators.
I have 2 projectors now and going to play around with edge blending.
For simple 2 projector edge blending, there are some easier solutions including
nVidia's PowerWall, Matrox
PJ-4OLP, and others. Many people with 2 projectors just want to
watch movies, and VLC has a plugin called "Panoramix" that will also do
projector blending.
To do some initially test & software, I need some high resolution footage.
My eventual plan is to create super high resolution video by stitching a lot of
individual video sequences together (just like how you create panoramic images).
However, there is no such video stitching application out there, which means
I'll have to write it myself. In order to skip that for the short term, I hired clai.tv
to come up to San Francisco and film some test 4k footage (4096x2048) using
their Red Cinema digital camera. Without their help, I'd have a hard
time working this camera. As you can see in the video below, the camera is pretty complex.
Below is a thumbnail from one frame captured by the camera,
click it to see the full resolution image capture.
Nov 28, 2009:
"Hiring" Open Source Developer for Video
Stitching project
If you have experience working on image stitching and
interesting in apply/extending this to process video, I'm willing to fund your
work. The project should take a set of movie files and perform the
following:
- Calculate feature descriptors for individual video streams and
automatically find overlaps in both space and time
- Produce a single high resolution output video (jpeg stills are sufficient)
- Support various projections and mappings
- Support image, gamma, and color blending
- Experiment with synthesizing "tween" frames in order to blend videos that are
not synchronized at the millisecond level
Ideally this work would extend an existing open source project
like Panotools.
If you are interested, please contact me at
jc@thisdomain.
Dec 13, 2009:
First attempt at head tracking using TrackIR
I was experimenting with head tracking using TrackIR from
Natural Point. This near infrared camera does a pretty good job for
the price. The main limitation is has is a pretty limited capture
volume with one camera, though it appears you can use Natural Point's SDK to get
data from multiple cameras, so this could be used to increase the capture volume
you are working with. Natural Point's other software tools which do
multi-camera calibration and multi-camera triangulation are not available for
TrackIR, it appears they try hard to encourage non-gaming consumer to move to
the OptiTrack camera systems. As well, I don't think TrackIR support a
sync-signal so capturing fast moving objects will likely have higher errors (but
ok for my purposes).
The results of my test turned out pretty well, here is a video:
More to come.... Got ideas / suggestions? Email me: jc@thisdomain.com
