Method and apparatus for real-time group interactive augmented-reality area monitoring, suitable

38 Pages·2013·3.1 MB·English

US007796155B1
(12) United States Patent (10) Patent N0.: US 7,796,155 B1
Neely, III et al. (45) Date of Patent: Sep. 14, 2010
(54) METHOD AND APPARATUS FOR 5,659,691 A * 8/1997 Durward et al. ........... .. 715/757
REAL-TIME GROUP INTERACTIVE 5,802,284 A 9/1998 Karlton et al.
AUGMENTED-REALITY AREA 5,912,700 A * 6/1999 Honey et al. .............. .. 348/157
MONITORING, SUITABLE FOR ENHANCING 5,923,365 A * 7/1999 Tamir et al. ............... .. 348/169
THE ENJOYMENT OF ENTERTAINMENT 5,953,076 A 9/1999 Astle et al.
EVENTS
6,154,250 A ll/2000 Honey et al.
(75) Inventors: Howard Neely, III, Manhattan Beach,
CA (US); Ronald T. Azuma, Santa
Monica, CA (US); Jerry Isdale,
(Continued)
Thousand Oaks, CA (US); Mike Daily,
Thousand Oaks, CA (US) OTHER PUBLICATIONS
(73) Assignee: HRL Laboratories, LLC, Malibu, CA Berman, D.K., “Local Bells Look to Fiber to Stem Some Losses,”
(Us) The Wall Street Journal, Jun. 19, 2003.
(Continued)
(*) Notice: Subject to any disclaimer, the term of this
patent is extended or adjusted under 35 Primary ExamineriMarsha D Banks Harold
U.S.C. 154(b) by 1663 days. Assistant Examiner4Chikaodili E Anyikire
(74) Attorney, Agent, or FirmiTope-McKay & Assoc.
(21) Appl. No.: 11/019,991
(57) ABSTRACT
(22) Filed: Dec. 20, 2004
Related US. Application Data
A real-time, group interactive, augmented-reality system for
(60) Provisional application No. 60/531,456, ?led on Dec. monitoring an area, suitable for enhancing the enjoyment of
19, 2003. entertainment events, is presented. The system comprises a
sensor data acquisition system for receiving sensor data, in
(51) Int. Cl. real-time, from a plurality of data acquisition sensors dis
G06F 17/00 (2006.01) posed about an area to be monitored, and a plurality of display
G06T 1/00 (2006.01) centers communicatively coupled With the data acquisition
G06T 15/00 (2006.01) system for receiving sensor data therefrom. A display center
G06T 17/00 (2006.01) includes an user input, a data acquisition sensor selector, an
(52) US. Cl. ..................... .. 348/157; 348/143; 345/418; interaction interface for communicating in real-time With
345/419; 345/420; 345/421; 345/422 users of other display centers, an interface for connecting a
(58) Field of Classi?cation Search ............... .. 348/ 143, display device, and a local display device augmenting system,
348/157; 345/4l8i422 Which are con?gured to alloW a user to interactively augment
See application ?le for complete search history. in real-time his/her local display of events With user-inputted
doodles, user requested information and statistics, and user
(56) References Cited selected vieWing angles. The system alloWs users to interac
tively share the locally augmented-reality display With users
U.S. PATENT DOCUMENTS
of other display centers.
4,970,666 A 11/1990 Welsh et al.
5,397,133 A * 3/1995 Penzias ..................... .. 463/22
5,543,856 A 8/1996 Rosser et al. 14 Claims, 16 Drawing Sheets
Senior Data
Acquisition System
Swami/Y
Regishsred
Geomem: Model
Database
Recognition
lnlmma?on
Database
Evan!
Dalahas: Central Dav/a
Augmenting
System
Relatsd
lntormi?on
Database
Display Center
(wire be)
406
US 7,796,155 B1
Page 2
US. PATENT DOCUMENTS Connexion by Boeing, “Frequently Asked Questions,” 2002, http://
active.boeing.corn/connexion/Bocy.cfm?left:2A.
6,266,100 B1 7/2001 Gloudemans et al. Cavallaro, R., “The FoxTrak Hockey Puck Tracking System,” IEEE
6,317,127 B1 11/2001 Daily et al. Computer Graphics and Applications, Mar.-Apr. 1997, pp. 6-12.
6,380,933 B1 4/2002 Sharir et al. SEC Form 10-K for Clear Channel Communications, Inc., Dec. 31,
6,449,010 B1* 9/2002 Tucker ..................... .. 348/157 2001.
Glaskowsky, P.N., “Microsoft Weighs in with X-Box,” Microproces
222000000223///000110871862904401300 AAA111*** 11121///222000000322 NKFaraevteaamybaa enmt aea lt. al.. ............. .. 382/103 sor Report, Cahners Publications, Apr. 2000.
GueZiec, A., “Tracking Pitches for Broadcast Television,” Computer,
Mar. 2002, pp. 38-43.
OTHER PUBLICATIONS HeinZl, M., “Telecoms Hunt for Broadband ‘Killer Apps’ to use
Nearly empty Data Delivery Pipes,” The Wall Street Journal, Jun. 14,
Dockstader, S. L., A.M. Tekalp, “Tracking Multiple Objects in the 2001.
Presence of Articulated and Occluded Motion,” Proceedings of the Hindus, L.A, “CMOS: The Image of the Future,” Advanced Imagin
Workshop on Human Motion, Austin, TX Dec. 7-8, 2000, pp. 88-95. ing Magazine, May 2001, pp. 33-35.
Dockstader, S.L., A.M. Tekalp, “Multiple camera tracking of inter Data sheet, “Kodak Digital Science KAC-1310 CMOS Image
acting and occluded human motion,” Proceedings of the IEEE, Oct. Senson,” Eastman Kodak Company, Feb. 8, 2001, http://www.kodak.
2001, pp. 1441-1455. corn/US/plugins/acrobat/en/digital/ccd/kac1310Long.pdf.
Web site, “EA Sports NBA Live 2003,” Electronic Arts Inc., http:// Kanade, T., et al, “VirtualiZed Reality: Digitizing a 3D Time-Varying
www.easports.com/platforms/games/nbalive2003/homej sp. Event As Is and in Real Time,” Proceedings of the International
Gavrila, D.M., L.S. Davis, “3D model based tracking of humans in symposium on Mixed Reality (ISMR 99), Spriner-Verlag, Secaucus,
action: a multi-view approach,” Proceedings CVPR ’96: 1996 IEEE NJ, 1999, pp. 41-57.
Computer Society Conference on Computer Vision and Pattern Rec Longman, J., “Sports Use Technology and TLC. to Hold on to Fans,”
ognition, Jun. 18-20, 1996, San Francisco, 1996. The NewYork Times, Jul. 31, 2001.
Li, Z., H. Wang, “Real-Time 3D Motion Tracking with Known Geo Web page, Xbox Video Game System, Microsoft Corporation, 2001,
metric Models,” Real Time Imaging 5, Academic Press 1999, pp. http://www.xbox.com/hardware/consoles/xbox.htrn.
167-187. Web page, high de?nition av pack, Microsoft Corporation, 2001,
Web page, “Nokia N-Gage,” Nokia, http://www.n-gage.com/n-gage/ http://www.xbox.com/hardware/adapters/highde?nitionavpack.
home.html. htrn.
Rehg, J .M., T. Kanade, “Model-based tracking of self-occluding Pollefeys, M., “Self-Calibration and Metric 3D Reconstruction from
articulated objects,” Proceedings, Fifth International Conference on Uncalibrated Image Sequences,” Ph.D. dissertation, Katholieke
Computer Vision, IEEE Computer Society Press, Jun. 20-23, 1995, Universiteit Leuven Haverlee, Belgium, May 1999.
Cambridge, MA, 1995, pp. 612-617. Tran, K.T.L., J. Angwin, “Sony, AOL Announce Agreement to Link
Mills, D.L., Network Time Protocol (Version 3), Network Working PlayStation 2 to Internet,” The Wall Street Journal, May 15, 2001.
Group, Internet Engineering Task Group, Mar. 1992 (http://wwwntp. Hartley, R. and Zisserman, A., Multiple View Geometry in Computer
org ). Vision, Cambridge University Press, Cambridge UK, 2000.
Press release, “Yao Ming Basketball by Sorrent to be Distributed Faugeras, O. and Luong, Q-T, “The Geometry of Multiple Images,”
Exclusively via QUALCOMM’s BREW Solution,” QUALCOMM the MIT Press, Cambridge, MA, 2001.
Incorporated, Apr. 29, 2003. “Phone Giants Set Standards for Broadband Construction,” Associ
Press Release, “Sony Ericsson unveils the T606, a CDMA phone with ated Press reported in The Wall Street Journal, May 29, 2003.
state-of-the-art mobile entertainment and imaging features,” Sony de Bonvoisin, S., “EU Clears General Motors Takeover of Daewoo
Ericsson, Mar. 4, 2003. Motor,” The Wall Street Journal, Dow Jones & Company, Jul. 23,
Web page, “Sony Ericsson-Speci?cations for the T310,” Sony Erics 2002.
son, Jun. 20, 2003. Grenier, M.P., “Broadband User Growth Slows as Higher Costs Prove
Yang, S-j, “SKT Launches New Technology in Incheon,” The Korea Obstacle,” The Wall Street Journal, Dow Jones & Company, Oct. 18,
Herald, Jan. 28, 2002. 2001.
Sweet, D., “Where Vince, and Even Luc, Become Human Highlight
American National Standard for Information TechnologyiFibre
ChanneliPhysical and Signaling Interface (FC-PH), American Reels,” The Wall Street Journal, Dec. 6, 2000.
National Standards Institute, Inc., X3230, Nov. 14, 1994. * cited by examiner
US. Patent Sep. 14, 2010 Sheet 1 0f 16 US 7,796,155 B1
I sensor data acquisition system" V ilk
} receiving data in real-time from plurality of 102
104 data acquisition sensors 1
i \ disposed about an area to be monitored 5
I “plurality of display centers”
I
I
I
I
I
“input” for receiving input from a user \ 108
I
l
I
I
I 1
I
I “sensor selector” for allowing the user to
I
I 1 10 _/ select at least one data acquisition
l
l sensor from which to display data
i
i
I
i
i \
I ‘y
I 1 14 “interaction interface for communicating," 1 12
I
I \ in real-time, with other display centers to
I
I allow a user of a display center to interact
I with users of other display centers
I
I
I
I
|
I “local data augmenting system" for locally
I
| augmenting the display of data with user input \
|
I 116
I
I
| 1 18
|
\ "interface for connecting" the display center
|
I
| with a display device
|
I
US. Patent Sep. 14, 2010 Sheet 2 of 16 US 7,796,155 B1
3D Model Output
200 K)
I
Data Processing output I 204
202 \ System k
Image Stream 206
Input f input > processor J
A
otherf’ v
208
Input memory J
FIG. 2
US. Patent Sep. 14, 2010 Sheet 3 0f 16 US 7,796,155 B1
300
FIG. 3
US. Patent Sep. 14, 2010 Sheet 4 0f 16 US 7,796,155 B1
400
l/ 420
Sensor Data Object Recognition
Acquisition System L ' System
"angle- 414 422
controller”
Registered
Geometric Model
404 416 Database
j 402 ‘ /
Select and Route 424
“sensor Data
selector” 418 J
Recognition
Information
Database
Display Center urrent' 426
(about the aa) DaEtzggtse entra i
iaizi'stfrg —
///////////////////
////////////
Display Center
A
(networked device)
%
\ _ t Game Mode
r?'l‘iLZZ'JEZ; : System
&— Graphical \ 434
Doodle
Model
406 Generator -\ 432 436
J
FIG. 4
US. Patent Sep. 14, 2010 Sheet 5 0f 16 US 7,796,155 B1
i
I
502 504 j 506 I
I
1
i d Tr. t > 812mm Kelly QB
i
3wQ‘ ~;n g{aern a IOS . @ I:I‘ ,, Pass c omp= le. haI na_
. 15v 70H1 53 g
I
l
l
l
i
l
l
l
NFL Game with Augmentation
FIG. 5
US. Patent
Sep. 14, 2010 Sheet 6 0f 16 US 7,796,155 B1
I
6 0 2
I
l
I
l
l
l
l
l
l
l
I
l
I
l
l
l
I
I
I
l
l
l
I
l
I
I
l
l
I
l
l
l
l
I
I
I
I
I
I
l
I
I
I
I
I
I
l
l
l
l
FIG. 6
US. Patent Sep. 14, 2010 Sheet 7 0f 16 US 7,796,155 B1
702a
706 708
/ /
710
6
D O: SWITCH _Z
COURT
702n
\NSTRUMENTED
DiGITAL CAMERA
FIG. 7
US. Patent Sep. 14, 2010 Sheet 8 0f 16 US 7,796,155 B1
800
________________
I 802 808 :l
: / 810 / :
i / .
: DIGITAL 812 g
l: CAMERA / I
: 804 :
: / 814 :
: 3-HT"; / CAMERA 818 :
' RA PROCESSOR '
5l SENSOR UNIT _L— .
: 806 :
i / l
I Z-AXIS 816 :
i ANGULAR / :
: POSI'HON .
: SENSOR :
I 1
FIG. 8