TemporalDitheringofIllumination

forFastActiveVisionSrinivasaG.NarasimhanSponsors:NSF,ONRShuntaroYamazakiSanjeevJ.KoppalComputationalilluminationforVisionStructured-lightscanningSeparationoflighttransportDualPhotography[Zickleretal,Raskaretal,Zhangetal,Levoyetal,Debevecetal,Nayaretal]OfficeoftheFutureProgrammableIlluminationprojectorcameracomputerimage(60Hz)sceneactivevisionIlluminationfromDLPprojectorsVideoacquiredat60HzVideoacquiredat1000HzFastwayofencodingsceneilluminationHigh-speedIlluminationDitheringcameracomputerimage(60Hz)fastactivevisionhigherfpssensorsceneDMDcontroller(9600Hz)micro-mirrorstimemirrorstatetimecameraintensityPulse-WidthModulation(PWM)CodingDLPprojectorMicro-mirrorarraysforImagingandDisplayImperceptibledisplay[Raskaretal98]Scanning[Cottingetal,2004]Programmableimaging[Nayaretal04]Stereoscopicdisplay[Jonesetal,2007]Colorwheeleffect

+

Numberofbrightnesses[TexasInstruments,DLP.com]ReverseengineeringaDLPprojectorIntensitymosaicinputtoprojectorVideocapturedat3000HzVideocapturedat500KHzVideocapturedat1000HzMeasuredintensityFrames2000210001100Frames1100Projectedvalue0255Projectedvalue187215255165ReverseengineeringaDLPprojectorApplicationsFaststructured-lightbasedreconstructionSceneCameraDLPProjectorStructuredPatternCapturedvideo(30Hz)Capturedvideo(3000Hz)Reconstruction(30Hz)Reconstruction(120Hz)Reconstruction–differentview(120Hz)Reconstruction(30Hz)Reconstruction(120Hz)DifferentViewCapturedvideo(30Hz)Capturedvideo(3000Hz)Reconstruction(30Hz)Reconstruction(120Hz)Reconstruction–differentview(120Hz)Reconstruction(30Hz)Reconstruction(120Hz)DifferentViewCapturedvideo(30Hz)Capturedvideo(3000Hz)Reconstruction(30Hz)Reconstruction(120Hz)Reconstruction–differentview(120Hz)Reconstruction(30Hz)Reconstruction(120Hz)DifferentViewCapturedvideo(30Hz)Capturedvideo(3000Hz)Reconstruction(30Hz)Reconstruction(120Hz)Reconstruction–differentview(120Hz)Reconstruction(30Hz)Reconstruction(120Hz)DifferentViewReconstruction(jigglingeffect)CapturedVideo(30Hz)Patternsditheringat3000HzMovingclothshownat30HzReconstruction(120Hz)camerafpsobjectspeed120fps500fps1000fps3000fpsMultiplexingIlluminationDitheringMulti-spectralIlluminationMultiplexingMultiplexedintensity:Multiplexedgrayscalevideomulti-spectralsourceintensities(R,G,B)singlesourceimage[Schechner05,Grossberg,07]SceneilluminatedbythreeprojectorsMulti-spectralIlluminationMultiplexingMultiplexedintensity:De-multiplexedcolorvideosinglesourceimagemulti-spectralsourceintensities(R,G,B)[Schechner05,Grossberg,07]IlluminationMultiplexingMultiplexedintensity:Multiplexedgrayscalevideosinglesourceimagesourceintensities[Schechneretal03]SceneilluminatedbythreeprojectorsComputedShape(300Hz)High-speedPhotometricStereoDe-multiplexedintensity:singlesourceimagecomputedsourcedirection[Horn86]Frame1Frame5Frame10High-speedPhotometricStereoSceneilluminatedbythreeprojectorsSurfacenormals(300Hz)DLPPhotographyDLPprojectorstillcamerasceneDLPPhotographyBallet:Rightlegkick(1sexposure)DLPPhotographyBallet:Handandheadmotion(1sexposure)DLPPhotographyBallet:Bodymotion(1sexposure)Blurringundernormallighting:MotionBlurringunderDLPilluminationBlurringunderDLPillumination:tItInoticethesmearnoticethehigh-frequenciesobjectobjectmotionobjectprojectordithering+colorwheel[similartoFluttershutter,Raskaretal,06]ditheringPSFrectangularPSFFrequenciesofMotionPSFmagnitudefrequencyConclusionsHighspeedilluminationcontrolusingmicro-mirrors.Reverseengineeringanoff-the-shelfDLPprojector.Speedupofvirtuallyanyactivevisiontechnique.SkylightDLPFrequenciesofBlurPSF:DitheringPSFSkylightPSFMotionblur:MotionBlurringunderDLPilluminationDeblurredresult:SkylightDLPFrequenciesofBlurPSF:DLPditheringPSF‘Box’functionMeasuredPSFforskylightMeasuredPSFforfluorescentlightMotionblur:MotionBlurringunderDLPilluminationDeblurredresult:(Direct+Global)Frame2727Frame2730DirectcomponentGlobalcomponentDirectandGlobalseparationfordynamicscenesFlippedCheckerPatternsSeparationResultDLPprojector

BeamsplitterHighspeedcameraFrame2719[Nayaretal,06]Co-locatingprojectorandcameraDLPprojector

BeamsplitterHighspeedcameraSide-viewwithshadow

Co-locatedviewwithoutshadow

Side-viewshowsdistortedpattern

Co-locatedviewshowsnodistortionSetupReverseengineeringaDLPprojector10000hz3000hz1000hz500hzHigh-speedIlluminationditheringcameraLensLenscomputerspatio-temporalmodulation(60Hz)DLPprojectorhighspeeddithering(10kHz)fastactivevisionhigherfpssceneFastprojector-camerapixelcorrespondencesPatternsditheringat3000HzStripecorrespondencesMovingstatueshownat30HzRecovered3Dstructure(holesduetoocclusionsanddarkcastshadows)FastreconstructionofrigidobjectsPatternsditherat3000HzFastreconstructionofnon-rigidobjectsMovingclothat30HzReconstructionshownat120HzReconstructionshownat30HzDeblurringat125HzBlurredimageunderDLPilluminationDeblurredimageHigh-speedphotometricstereoMovingStatueInstantaneousshapeDepthmapofmovingstatueFastphotometricstereo:StatueSurfacenormalsNeedlemap3DshapebyintegratingnormalsIlluminationde-multiplexingfordynamicscenesIlluminatedbythreeprojectorsmultipleshadowsmultiplespecularitiesIlluminationmultiplexedimages:Illuminationde-multiplexedimages:sourceintensitiessinglesourceimageIlluminatedbyprojector1Illuminatedbyprojector2Illuminatedbyprojector3[Schechneretal,03]Patternsditherat3000HzFastreconstructionofrigidobjectsPing-pongballsdropped,shownat30HzReconstructionshownat120HzMotionblurMotionblurDeblurre