(12) Patent Application Publication (10) Pub. No.: US 2015/0121416 ...

(19) United States (12) Patent Application Publication (10) Pub. No.: US 2015/0121416 A1 MOON et al.

US 2015O121416A1

(43) Pub. Date: Apr. 30, 2015

(54) VIDEO DISPLAY DEVICE, TERMINAL DEVICE, AND METHOD THEREOF

Publication Classification

(51) Int. Cl. (71) Applicant: LG ELECTRONICS INC., Seoul (KR) H4N2L/8 (2006.01)

H4N2L/24 (2006.01) (72) Inventors: Kyoungsoo MOON, Seoul (KR); Jaehee H4N2L/2478 (2006.01)

CHUNG, Seoul (KR): Hyeonjae LEE, H4N2L/643 (2006.01) Seoul (KR); Jinpil KIM, Seoul (KR): H4N2L/2475 (2006.01) Aettie J, Seoul (KR); Kyungho KIM, H4N2L/2126 (2006.01) Seoul (KR); Sanghyun KIM, Seoul H4N2L/214 (2006.01) (KR) (52) U.S. Cl.

CPC ....... H04N 2 1/8173 (2013.01); H04N 2 1/4263 (21) Appl. No.: 14/553,547

(22) Filed: Nov. 25, 2014

Related U.S. Application Data (63) Continuation of application No. 13/567,689, filed on

Aug. 6, 2012. (60) Provisional application No. 61/549,236, filed on Oct.

20, 2011, provisional application No. 61/543.288, A video display apparatus pairs with a first terminal device. filed on Oct. 4, 2011, provisional application No. The video display apparatus receives an audio-visual content, 61/525,835, filed on Aug. 21, 2011.

(2013.01); H04N 2 1/4104 (2013.01); H04N 2 I/441 (2013.01); H04N 2 1/64322 (2013.01);

H04N 2 1/4753 (2013.01); H04N 2 1/478 (2013.01)

(57) ABSTRACT

and displays the audio-visual content. The video display apparatus acquires enhanced service information on an avail

(30) Foreign Application Priority Data able enhanced service for the audio-visual content, and noti fies an occurrence of the available enhanced service to the first

Jun. 14, 2012 (KR) ........................ 10-2012-0063969 terminal device.

10

CONTENT PROYONG SERVER 20

50 -

50

CONTENT RECOGNIZING

ENHANCED SERVICE PROVIDING SERVER

MULTCHANNEL WIDEO DISTRIBUTING SERVER

SERVICE PROVIDING SERVER

ENHANCED SERVICE INFOR MATION PROVIDING SERVER


70 400

TERMINAL DEVICE

WIDEO DISPLAY DEVICE

BROADCAST RECEIVING DEVICE

Patent Application Publication

FIG.1 10

CONTENT PROVONG SERVER

CONTENT RECOGNIZING SERVICE PROVIDING SERVER




50-N-

50

MULTCHANNEL WIDEO DSTRIBUTING SERVER

Apr. 30, 2015 Sheet 1 of 96 US 2015/O121416 A1

70 4. OO

TERYNAL DEVICE

100

WICEO DISPLAY DEVICE

BROADCAST RECE WING DEVICE

60

Patent Application Publication Apr. 30, 2015 Sheet 2 of 96 US 2015/O121416 A1

FIG.2 10

CONTENT PROVIDING SERVER

WATERMARK 70 SERVER

400

TERMINAL DEVICE

1OO

MAGE DISPLAY DEVICE




50

50



US 2015/O121416 A1 Patent Application Publication

US 2015/O121416 A1 Apr. 30, 2015 Sheet 4 of 96 Patent Application Publication


FIG.5

10


FNGERPRINT SERVER 400

TERMINAL DEVICE

VIDEO DISPLAY DEVICE

ENHANCED SERVICE INFOR MATON PROVIDING SERVER


50

ENHANCED SERVICE 50 PROVIDING SERVER



60


BOMBS (BINH B'GOES |-?---------------

US 2015/O121416 A1

| | |

r– – – – – – – – – – –|

0

Apr. 30, 2015 Sheet 7 of 96

L'OIH



FIG.8

10


WATERMARK SERVER

FNGERPRINT SERVER



40

R

O

NA CE

TERMINAL DEY

WOEO DISPLAY DEVICE

22

40

ENHANCED SERVICE 50 PROVIDING SERVER


BROADCAST RECEWING DEVICE

60


00700||090907 22| 20||

US 2015/0121416 A1 Apr. 30, 2015 Sheet 10 of 96 Patent Application Publication

Å80|||||||

|

0[ "OIH


FIG.11

era Go App store situences s Broadband NetWOrk

w 1.

ACR M/W (e.g. Browser) ACRApp ||F

ACR Native

136a

ACR plat for Orm API

Fl

133a


FIG.12

OBTAN CONTENT INFORMATION OF MAN AW CONTENT

OBTAIN ENHANCED SERVICE FILE

EXECUTE OBTANED ENHANCED SERVICE FILE IN BACKGROUND

DISPLAY COLOR BUTTON S4O7

PLAY ENHANCED SERVICE EXECUTED IN BACKGROUND

ON PLAY DEVICE

TERMINATE ENHANCED SERVICE S411

S403

S405

S409


FIG.13

1390

(e.g. BrOWSer) XML 1380

ACR Engine 136b it

ACR plat for Orm API 135b

East Operating System E. 131b.

133b

Patent Application Publication Apr. 30, 2015 Sheet 14 of 96

FIG.14

OBTAIN INFORMATION ON ACR ENGINE THAT ACR APPLICATION TSELF USES

START ACR ENGINE THAT ACR APPLICATION TSELF USES

OBTAN CONTENT INFORMATION BY ACR APPLICATION THROUGH ACR ENGINE

OBTANAVAILABLE ENHANCED SERVICE BY ACRAPPLICATION

DISPLAY COLOR BUTTON BY ACRAPPLICATION

PLAY ENHANCED SERVICE BY ACRAPPLICATION

TERMINATE ACRAPPLICATION BY ACR MODLEWARE

S501

S503

S505

S507

S509

S511

S513

US 2015/O121416 A1


FIG.15

Brave 9 GApp store Clernet w -- Broadband NetWOrk

ACR M/W ACR Application IF Script (e.g. Browser)

ACR Pat form AP

... . NetWork Or

Operating System H

133C


FIG.16

TRANSMIT QUERY INCLUDING SPECIFIC INFORMATION TO ACR ADJUSTING S601

SERVER BY ACR ENGINE

RECEIVE FINGERPRINT SERVER ADDRESS BY ACR ENGINE S603

BTAIN FNGERPRINT SERVER ADORES OBTANF NT SER S S605 BY ACRAPPLICATION

OBTAIN FEATURE INFORMATION BY ACRAPPLICATION S607

TRANSMT QUERY INCLUDING FEATURE INFORMATION BY ACR APPLICATION S609

RECEIVE CONTENT INFORMATION IN RESPONSE TO S611 QUERY BY ACRAPPLICATION

RECEIVE AVAILABLE ENHANCED SERVICE S613 BY ACRAPPLICATION

DISPLAY COLOR BUTTON BY ACR APPLICATION S615

PLAY ENHANCED SERVICE BY ACR APPLICATION S617

TERMINATE ACRAPPLICATION BY ACR S619 YDOLEWARE


FIG.17

GOOgle Nave?

s Broadband NetWOrk

ACP Native App

ACR Native

1360

ACR Pat form AP

Frt. irrel

133d


FIG.18

CONFIRN ACR ENGINE MATCHING TO MAN AW w CONTENT BY ACR ENGINE MANAGEMENT UNIT

SLEEP REMAINING ACR ENGINES EXCEPT FOR ACR ENGINE RECOGNIZING CONTENT SUCCESSFULLY BY ACR ENGINE MANAGEMENT UNIT

OBTAN CONTENT INFORMATION OF MAN AW CONTENT BY ACR ENGINE MANAGEMENT UNIT

CONFIRMANAIABLE APPLICATION BY ACR ENGINE MANAGEMENT UNIT

EXECUTE CONFIRMED APPLICATION IN BACKGROUND BY ACR ENGINE MANAGEMENT UNIT

DISPLAY COLOR BUTTON BY APPLICATION S707

PLAY ENHANCED SERVICE S709

CHANNELS CHANGED OR MAN AN CONTENT

S CHANGED?

YES CONFIRMACR ENGINE MATCHING TO APPLICATION

BY ACR ENGINE MANAGEMENT UNIT

CONNECT CONFIRMED ACR ENGINE WITH APPLICATION BY ACR ENGINE MANAGEMENT UNIT

USER INPUT FOR TERMINATING APPLICATION

S RECEIVED?

YES

TERMINATE APPLICATION BY APPLICATION MANAGEMENT UNIT

S701

S702

S703

S704

S705

S710

S711

S712

S713

S714


EXTRACT ACERTAIN SECTION OF S801 MAN AV CONTENT

OBTAN CONTENT INFORMATION OF MAN AW S803 CONTENT ON THE BASIS OF EXTRACTEO SECTION

PERFORMSYNCHRONIZATION S805

OBTAIN ENHANCED SERVICE START TIME OF ENHANCED SERVICE ON THE BASIS OF OBTANED

CONTENT INFORMATION

S807

START OBTANED ENHANCED SERVICE WHEN SYNCHRONIZED PLAY TIME OF MAN AW CONTENT

|S|DENTICAL TO START TIME OF ENHANCED SERVICE

S809

Patent Application Publication Apr. 30, 2015 Sheet 20 of 96 US 2015/0121416A1

FIG20

CONTENT FNGERPRINT SERVER

(SY AY SAPLE QUERY/REPLY FP

Network ||F

QUERY RESULT

{{MetadataX Content ID . Secret Garden Play time : 68 min BrOadcaster : SBS Start time : 11 OOOMS End time : 1450OMS

Time line


FIG21

210 200 220

SUBSCREEN ON WHCH ENHANCED SERVICE S

DISPLAYEO


FIG.22

230 200

MY APPLICATIONS

Broadc Broadc BrOadC BrOadc COMMON aster A aster B aster C aster D App

Mage

SUBSCREEN ON WHICH ENHANCED SERVICES

DSPLAYED

SUBSCREEN ON WHICH MENUS OSPLAYED

240 (B)


330 Voting will start in N Minutes. GO to oria e 370 32O PROGRAMINFORSOCIAL RECOY CATALOG


FIG.24

310 300

PARTICIPATION ADDITIONAL INFORMATION AREA AOVERTISEENT DSCOVERY AREA

Voting will Start in N Minutes. GO to

330


FIG.25

1N USER INPUT CORRESPONDING

< PICTION SETIO D-S901 MENU DISPLAY S RECEIVED?

DISPLAY APPLICATION SELECTION MENU S903

USER INPUT CORRESPONDING CON SELECTION OF WIDEO

< DISPLAY DEVICE D-so MANUFACTURER APPLICATION IS RECEIVED?

EXECUTE WIDEO DISPLAY DEVICE MANUFACTURER APPLICATION S907


FIG.26

310 300

4.

BroadC BroadC BrOadC COMMOn r 370


Do you think .... ?

CHARACER INFORMATION

330 Voting will Start in N Minutes, GO to 320 ADVERT SEVENT AREA


310 300 342 34

Do yOu think ... ? 360

Do you think ... ? YES O80% NO 20%

330 Voting will Start in N minutes. GO to 32O

SFMFNT O ADVERTISEMENT E. 37


370 330 WOting will Start in N Minutes. G0 to 320 PROGRAMINFORSOCIAL RECOMCATALOG


FIG.30

OBTAIN CONTENT INFORMATION S1001

SEARCH SUB MENUS SELECTED? S1002

YES

OBTAIN SEARCH IST ON THE BASIS OF CONTENT INFORMATION S1003

DISPLAY SEARCH LIST S1007

ONE OF SEARCH TEMS |S SELECTED?

S1009

YES

SEARCH INFORMATION ON SELECTED SEARCH TEM S1011

DISPLAY SEARCHED INFORMATION S 1013

Patent Application Publication Apr. 30, 2015 Sheet 31 of 96 US 2015/O121416A1

310 300 342 341

330 Voting will Start in N Minutes, G0 to RTSEMENT AREA 320--PROGRAN SOCIAL RECOM CATALOG orse


31 O 300 342 341

BRTHDAY: ... FAYORATE FOOO. . .

330

320 Voting Will Start in N minutes. G0 to


FIG.33

S1 101

TWITTER SUB VENUS SELECTED? S1 102

YES

OBTAIN WITTER CHATTING LIST ON THE BASIS OF S1 103 CONTENT INFORMATION

DISPLAY CHATTING DSPLAY S1107


31 O 300 343 342 341

ENTRE Topchel

Guy A. : What do you like most? . . . Miss A:... do love to have the . juicy. Stake,

Rich: Ren is the start of next TV show of this? ... . . . . Miss B: Not know-AOY, But you...'... Cantook Up Ytaia, Iovii, COM... Guy A. : Waitt'll at you know ...

Erise it 330

320


FIG.35

OBTAN CONTENT INFORMATION

NEWS SUB MENUS SELECTED? S1202

OBTAIN NEWS ON THE BASIS OF CONTENT INFORMATION

DISPLAY NEWS

S1203

S1207


310 300 343 342 341

Strong. Earthquake Hits Indonesia's Papua:

S0 of east for Washington?

330

320 Voting will start in N minutes. Go to

ADVERTISEVENT AREA 370


FIG.37

S1301

WEWER LIST SUB MENUS

SELECTED?

YES

OBTAIN INFORMATION ON FRIEND STAND S1303 WATCHING ON THE BASIS OF CONTENT INFORMATION

DSPLAY FRIEND LIST S1307

S1302

FRIEND IS SELECTED? S1309

YES S1311

SELECTED FRIEND NO WEWS MAN AW CONTENT

S1313 S1315 YES

OSPLAY CHATTING WINDOW FOR SELECTED TRANSMIT MESSAGE RECOMMENDING FRIEND ON ADDITIONAL INFORMATION AREA WATCHING OF MAN AW CONTENT


FIG.38

310 300 343 342 341

PICTURE NAME: B

PICTURE NAME: A (IN WATCHING)

PICTURE NAME: C

PICTURE NAME: ) (N WATCHING)

330 CTTACT 320 ACVERSEMENT AREA


SOCIAL NETWORK SERVICE PROVIDING SERVER

st S1403 CONTENT A D / /ONTENTA ID CONTENT A D

USer D FRIEND A D se FRIENDB ID CONTENT A D

Aisi SE. IMAGE DISPLAY DEVICE IMAGE DISPLAY DEVICE

(FRIENDA) (FRIENDB) WDEO DISPLAY DEVICE

100


FIG.40

S1511

75 77 FRIEND A D

SOCIAL NETWORK SERVICE PROVIDING SERVER ACR SERVER FRIEND BID

S1501

S1505 \ FRIEND A D

S1509 FRIEND BO CONTENT A D

USer D

VIDEO DISPLAY DEVICE S1507 S 1515

(FRIENDA) ) ot A D IDEO DISPLAY DEVICE CONTENT A D FRIEND A D

(FRIENDB) CONTENT A D

S1513 N CONTENT A D FRIEND BID


100


FIG.41

S1601 OBTAN CONTENT INFORMATION

PREFERENCE DISPLAY SUB MENUS SELECTED? S1602

YES

OBTAN PREFERENCE DISPLAY LIST ON THE BASIS OF CONTENT INFORMATION

DSPLAY PREFERENCE OSPLAY LIST

S1603

S1607

REFERENCE DISPLAY S SELECTED S1609

S1611 POST PREFERENCE DISPLAY CONTENT ON SNS


310 300 342 341

SCENE 3: LKE

SCENE #2 KE. T.

KE.

IKE

320 INFOR ADVERT SEVENT AREA 330 Voting will Start in N minutes. GO to


FIG.43

S1701 OBTAN CONTENT INFORMATION

SUB MENUS SELECTED? S1702

OBTAN DIGITAL MEDALIST ON THE BASIS OF CONTENT INFORMATION

DISPLAY DIGITAL MEDALIST

S1703

S1707

DIGITAL MEDIA IS SELECTED? S1709

S 1711 DOWNLOADD GTAL MEDIA


31 O 300

EPISODE #2

EPISODE #3

PREVIEWEPISODE 4 Voting Will Start in N Minutes, GO to

Hora woul AVERTISEET AE 330

320 370


310 300 343 342 341

WOOH

WOOH2

WOOH3

WOOH4.

330 Voting Will Start in N Minutes. GO to 320


310 300 342 34

CON APPLICATION It

CON APP. CATION F2

CON APPLICATION A3

APPLICATION #4

330 ETHACT WE 320 As TiSEENT AREA


FIG.47

S1801

SUB MENUS SELECTED? S1802

YES

OBTAN CATALOG LIST ON THE BASIS OF CONTENT INFORMATION

DISPLAY CATALOG LIST

S1803

S1807

S1809

YES

DISPLAY PURCHASE PAGE S1811


310 3OO 342 341

K. Motipi PURCHASE REVIEWBOOKMARK

MAKER: . . . IMAGE | PRICE: .

DESCRIPTION: . PURCHASE REVIEWBOOKNARK

330 Voting Wil Start in N MinuteS. G0 to 320 ADVERTSEMENT AREA


310 300 3

SINGER: . . . |MAGE PRICE . . .

DESCRIPTION: ... ACASE REVIEWBOOKARK SINGER: :

iMAGE PRCE: , , , DESCRIPTION: , ,

ERCHASE REVIEEOKAK 330 ADVERTISEMENT AREA


FIG.50

310 300 342 341

PRCE: , , IMAGE ESEPTION:

PURCHASE REVIEWSOKNAR PRCE: ...

IMAGE ESRIPTION: PURCHASE REVIEWEKNARK


FIG.51

SET ENVIRONMENT SELECT DES RED SETING MODE

YOU MAY SE ACR (AUTOMATIC CONTENT RECOGNITION) FUNCTION

ACR MODE k common App. D


FIG.52

S1901 SC YES

EXTRACT A CERTAIN SECTION OF MAN AN CONTENT S1902

OBTAN CONTENT INFORMATION OF MAN AN CONTENT ON THE BASIS OF EXTRACTED SECTION S1903

PERFORMSYNCHRONIZATION S1905

OBTAIN ENHANCED SERVICE ON THE BASIS OF OBTANED CONTENT INFORMATION S1907

CONFIRMACR MODE S1909

PLAY ENHANCED SERVICE ACCORONG TO ACR MODE S1911


500

USER

S2005

Apr. 30, 2015 Sheet 53 of 96

FIG.53 ACO

TERYNAL DEVICE

100

EXECUTE INTERWORKING APP S2OOO

SEARCH FOR DEVICE S2001

\ DISPLAY SEARCHED DEVICE S2003 INPUT SELECTION FOR YOEO DISPLAY DEVICE

DISPLAY AUTHEN CATION NUMBER INPUT WINDOW INPUT AUTHENTCATION NUMBER

INPUT SELECTION FOR USER TYPE INPUT TV PASSWORD

YES REQUEST AUTHENTICATION NUMBER

S2011 AUTHENTCATION NUMBER GENERATION RESULT

S2019

S2O2

S2O23

S2O31- DISPLAY AUTHENTICATION RESULT CAPABILITY INFORMATION S2O33

REQUEST AUTHENTCATION

AUTHENT CATION RESULT

US 2015/0121416 A1

WIDEO DISPLAY DEWICE

S2009

GENERATE AUTHENTCATION NUMBER

DISPLAY AUTHENTICATION NUMBER

S2015

PERFORM AUTHENTCATION

ISSUE SESSION VALUE

S2029

PARING PROCESS


FIG.55

INPUT AUTHENT CATION KEY

Enter pairing key

USER TYPE FAMILY v

Enter TV paSSWOrd

NPUT AUTHENT CATION KEY

Enter pairing key

USER TYPE GUEST V


FIG.56

TERNAL WDEO DISPLAY DEVICE DEVICE

PARING(S2302)

EXECUTE INTERWORKING APP

ACR CAPABILITY REQUEST(S23O3) ACR CAPABILITY RESPOND(S2305)

S2307

ACR ACR SUPPORTED? CONTROL

PROCESS YES ACR SETTING STATE REQUEST (S2309) ) ACR SETTING STATE RESPONO(S2311

S233

<d ACR ON REQUEST (S2315) NO YES

RECOGNIZE CONTENT

INTERWORKING BETWEEN DEVICES(S2318)


FIG.57

400 1OO


ERMINAL DEVICE

EXECUTE ACR INTERWORKING APP

PARING(S2503) ACR CONTROL (S2505)

S2507 RECOGNIZE CONTENT

REQUEST EVENT REGISTRATION(S2509)

REGISTER EVENT

CONFIRM EVENT REGISTRATION (S251)

S2512

NOTIFY EVENT (S2513) INTERWORKING BETEEN DEVICES

S2514 STORE EVENT

PERFORM OPERATION ACCORONG TO EVENT S2515

INQUIRY(S2516)

|NQUERY RESPONSE(S2517)

EVENT REGISTRATION RELEASE REQUEST (S2518)

EVENT REGISTRATION RELEASE

CONFIRM EVENT REGISTRATION RELEASE (S2521)


500 4OO 1OO

ENHANCED TERMINAL VIDEO DISPLAY SERVICE PROVIDING DEVICE

SERVER DEVICE

EXECUTE ACR INTERWORKING APP-S270

S2703 PARING -F

S2705 ACR CONTROL

RECOGNIZE CONTENT

CONTENT RECOGNITION STATE REQUEST S279

CONTENT RECOGNITION STATE RESPONSE--S272 ENHANCED SERVICE occur.N: EVENT REGISTRATION REQUEST S2723

ENHANCED SERVICE OCCURRENCE EVENT REGISTRATION CONFIRMATION S2725

ENHANCED SERVICE OCCURRENCE EVENT S2726

DISPLAY THAT THERE IS AVAILABLE INTER ENHANCEO SERVICE WORKING

BEINEEN RECEIVE ENHANCEO SERVICE EXECUTION INPUT DEVICES

RECEIVE ENHANCED SERVICE TYPE SELECTION INPUT

ENHANCED SERVICE TRANSFER REQUEST S2733

CONTROL ENHANCEO S2737 SERVICE

ENHANCED SERVICE REQUEST ENHANCED SERVICE TRANSFER RESPONSE S2735.

ENHANCEO SERVICE S2739 APP

INTER- EXECUTE ENHANCED SERVICE-S274 WORKING ENHANCED SERVICE EVENT S2743

ENHANCED SERVICE EVENT RESPONSE S2745


FIG.59


FIG.60

THERE IS INFORMATION REGARDING PROGRAM N WATCHING

WANT TO WIEW2


FIG.61

THERE IS INFORMATION REGARDING PROGRAM N WATCHING

WHICH DEVICE YOU WANT TO SELECT?


FIG.62

s SEARCH RESULT (Oi OjdKf j| / Interesting She's beautiful

(Wpeop / GOOd

(Smeil / Very nice


Giojdkfj| / Interesting. She's beautiful

GWOeop / GOOd

(Smeil / Very nice

HMMMMMMMM. . . . . . . . . . . . . . .

ow ER TOY Cui o P) As D F GH J K Cl

Shifz (xc) v B NMD de D C. 123 C return


FIG.64

JE / ) 2

Aft


FIG.65

SONG 1

SONG 2

SONG 3

SONG 4


FIG.66

400 1OO

TERMINAL DEVICE

EXECUTE ENHANCED SERVICE

RECEIVE USER INPUT FOR ENHANCED SERVICE TERMINATION

CONTROLLER

CONTROL FUNCTION TRANSFER RELEASE REQUEST (S2751)

RELEASE CONTROL FUNCT ON TRANSFER

CONTROL FUNCTION TRANSFER RELEASE RESPONSE(S2755)

MAGE DISPLAY DEVICE

ENHANCED SERVICE TRANSFER RELEASE REQUEST (S2757)

RELEASE ENHANCED SERVICE TRANSFER

ENHANCED SERVICE TRANSFER RELEASE RESPONSE(S2761)

TERMINATE ENHANCED SERVICE


FIG.67 400 1OO

WCEO DISPLAY DEVICE

PARNG

STORE WIDEO DISPLAY STORE TERMINAL DEVICE INFORMATION DEVICE INFORMATION

PARNG RELEASE

TERMINAL DEVICE

S2805

S2807

DISPLAY AWALABLE ENHANCED SERVICE

RECEIVE INPUT FOR ENHANCED SERVICE

RECEIVE INPUT FOR ENHANCED SERVICE TYPE SELECTION

DISPLAY AWALABLE TERMINAL DEVICE LIST

RECEIVE INPUT FOR TERMINAL DEVICE SELECTION

ENHANCED SERVICE EXECUTION REQUEST (S2819)


PARING

ENHANCED SERVICE EXECUTION RESPONSE(S2825) S2823

CONTROL ENHANCED SERVICE


410 400 1OO

TERNINAL ERMINAL WDEO DISPLAY DEVICE DEVICE DEVICE

ENHANCED SERVICE REQUEST (CONTROLLER) S290

S2903 CONTROL ENHANCED SERVICE

ENHANCED SERVICE RESPONSE (OK) S2905 S2911 CONTROL FUNCTION TRANSFER CONFLCT EVENT REGISTRATION REQUEST S290/

h CONTROL FUNCTION TRANSFER CONFLC EVENT REGISTRATION CONFIRMATION S2909 ENHANCED SERVICE REQUEST (CONTROLLER) ENHANCED SERVICE RESPONSE (CONFLCT)

CONTROL FUNCTION TRANSFER CONFLCT EVENT -S2915

RECEIVE USER INPUT FOR ENHANCED SERVICE TERMINATION

CONTROL FUNCTION TRANSFER RELEASE REQUEST--S2019

CONTROL FUNCT ON TRANSFER REGISTRATION RELEASE

s CONTROL FUNCTION TRANSFER RELEASE RESPONSE-S2923 CONTROL FUNCTION TRANSFER AWALABILITY NOTIFICATION

ENHANCED SERVICE REQUEST (CONTROLLER)

| 292 S2927 S2929

ENHANCED SERVICE RESPONSE (OK) --S2931

CONTROL ENHANCED SERVICE


FIG.69 400 100

TERMINAL VIDEO DISPLAY DEVICE DEVICE

RECEIVE USER

ENHANCED SERVICE DISPLAY CONTROL REQUEST (S3102)

INPUT

CONTROL ENHANCED SERVICE DISPLAY

ENHANCED SERVICE DISPLAY CONTROL RESPONSE(S3105)


FIG.70

400 100


RECEIVE USER INPUT

MEDIA PLAYBACK CONTROL REQUEST (S3302)

CONTROL MEDIA PLAYBACK

MEDIA PLAYBACK CONTROL RESPONSE(S3305)


FIG.71 400 100


RECEIVE USER INPUT

MEDIA PLAYBACK CONTROL REQUEST (S3502)

CONTROL MEDIA PLAYBACK

MEDIA PLAYBACK CONTROL RESPONSE(S3505)


FIG.72 400 100

WDEO DISPLAY DEVICE

TERMINAL DEVICE

EXECUTE ACR INTERWORKING APP

PARNG

ACR CONTROL

COMMON APPEXECUTION REQUEST

EXECUTE COMMON APP

S3711

S3703

S3705

S3707

COMMON APPEXECUTION RESPONSE INTERWORKING

COMMON APP TRANSFER REQUEST S3713 BETWEEN DEVICES

CONTROL COMMON APP

COMMON APP TRANSFER RESPONSE S3717

EXECUTE COMMON APP-S3719 COMMON APP TRANSFER RELEASE REQUEST S3721

S3723 CONTROL COMNON APP

COMMON APP TRANSFER RELEASE RESPONSE S3725

TERMINATE COMMON APP-S3727 COMMON APP TERMINATION REQUEST

S3731- TERMINATE COMMON APP COMMON APP TERMINATION RESPONSE S3733


VOL/CH - I Touchpad Button I ACR


FIG.74

ACR ADO Cat iOn ACR History

ACRONO Common App v. Ge (c) (6) (G) (e) (IVE AA


FIG.75

ACR AOO iCat iOn ACR Histor ACR))) Top Chef 2000:08:21 http:llac?.app. CO.K. http:liac?. Seccondapp, CO.Kf MACR Status MACR Event

Debug MOce

N WO/CH

CK (1) EB 11:09


FIG.76

ACR Application ACR HiSt Or

G (e) Debug Wode

Top Chef

Matchlist Matchlist v Check in

VOL/CH CK (1N BBE 11:09


FIG.77

ACR ADO Cat iOn ACR HiSt Or G (6) O Debug Mode

JE / ) -2 \

F. VOL/CH

KK (4) EEB 11:09


FIG.78

ACR AOO iCat iOn a ACR HiSt Or G) (6) (C) TV O Debug Mode Guest001 : Nice Me : Interesting, KKK. Interesting, KKK.


FIG.79

ACR ADO Cat iOn ACR HiStor G) (6) Debug Mode

DOWNLOAD-SORTED BY DATE

(QTODAY O TEST DOWNLOAD

(2) LAST 7 DAYS (OLAST MONTH

SORTED BY SZE

CK 1) EB 11:09


FIG.80

ACR ADO iCat iOn . . . . . . ACR H Stor ACR Application HiStory ()

v. Application Title Content Title Launched time http://appur I. Com(C)(6) O Application Title Content Title Launched time http://appur .COM (D)(6) OApplication Title Content Title Launched time http://appur.com(C)(6)


FIG.81

ACR AOD. iCat iOn ACR HiSt Or

WOL 11:09


FIG.82

ACR ADO Cat iOn ACR HiSt Or AOR NOT FOUND TV APPLICATION 2ND DEVICE APPLICATION MACR Status M Common App MACR Event http:// G) (e) TV M Debug Mode


FIG.83

RECEIVE SERVICE PROVIDER DSCOVERY INFORMATION THROUGH P BASED NETWORK

ACCESS SERVICE PROVIDER

RECE WE APPLICATION INFORMATION TABLE FROM PROVIDER

S3901

S3903

S3905

S3907


={{!-}-ÉT?R?ST?T?S

98 °50IJI


8


Application

O

ApplicationLOCat iOn


FIG.87

RECEIVE PROGRAM MAP TABLE

CONFIRM PACKET DENT FER OF TRANSPORT STREAM PACKET CARRYING APPLICATION

NFORMATION TABLE

S4101

S4103

RECEIVE TRANSPORT STREAM PACKET CARRYING APPLICATION INFORMATION TABLE ON THE BASS

OF CONFIRMED PACKET DENT FER

EXTRACT APPLICATION INFORMATION TABLE FROM RECEIVED TRANSPORT STREAM PACKET

S405

S4107


FIG.88 400 100

WIDEO DISPLA DEVICE

TERMINAL DEVICE

EXECUTE INTERWORKING APP

PARNG

ACR CAPABILITY REQUEST

ACR CAPABILITY RESPONSE

S4303

S4305

S4307

S4309 ACR SUPPORTED?

NO

TURN ON ACR S4310

OBTAN PART OF MAN AW CONTENT S4311

CONTENT RECOGNITION


INTERWORKING OF ENHANCED SERVICE

INTERWORKING BETEEN DEVICES S4319


FIG.89

DO DO


FIG.90

CURRENTLY WATCHING PROGRAM S "INTERESTING DRAMA 01"

CONFIRM


FIG.91 400 100

WDEO DISPLAY DEVICE

TERMINAL DEVICE

DISPLAY U TO INQUIRE OF USER WHETHER TO PROVIDE ENHANCED SERVICE TO WIDEO DISPLAY DEVICE

ENHANCED SERVICE S4321


ENHANCED SERVICE EXECUTION RESPONSE S4325

ENHANCED SERVICE INTERWORKING

CONTROL ENHANCED SERVICE S4329 S4327


FIG.92

WAN TO WATCH WIDEO RELATED TO CURRENTLY WATCHING

"INTERESTING DRAMA 01" ON TV?

CONFIRM


DISPLAY ENHANCED INFORMATIO RELATED TO CURRENTLY

WATCHING CONTENT


FIG.96 400

490 POWER SUPPLY UNIT

WRELESS COMMUNICATION UNIT

A5 DISPLAY UNT

SOUND OUTPUTTING MODULE

HAPTIC UNIT

452

453

454 414 CONTROL UNIT OCA AREA

COMMUNICATIONMODULE

POSITION INFORMATION MODULE

415

420 48O

421 CAMERA

422 MIKE

USER INPUT UNIT

SENSING UNIT

48

MEMORY

470

430 MULTIMEDIA MODULE

440 460

INTERFACE UNIT

US 2015/O121416 A1

VIDEO DISPLAY DEVICE,TERMINAL DEVICE, AND METHOD THEREOF

CROSS-REFERENCE TO RELATED APPLICATIONS

0001. The present application is a continuation applica tion of U.S. application Ser. No. 13/567,689 filed Aug. 6, 2012 and claims benefit and priority under 35 U.S.C. 119 and 35 U.S.C. 365 to U.S. Patent Applications No. 61/525,835 (filed on Aug. 21, 2011), No. 61/543.288 (filed on Oct. 4, 2011), No. 61/549,236 (filed on Oct. 20, 2011), and Korean Patent Application No. 10-2012-0063969 (filed on Jun. 14, 2012), which are hereby incorporated by reference in its entirety.

BACKGROUND

0002 The present invention relates to a video display device, a terminal device, and method thereof. 0003. As digital broadcasting is paving the way for its extensive spread, a broadcasting station transmits both main audio-visual (AV) contents and enhanced service data, which may be linked to the main AV contents to provide information and services or may be separately provided for other pur poses. 0004. However, a video display device in each home may unlikely receive broadcast signals directly through air under a current broadcasting environment. Rather, most of cases, a Video display device in each home is connected to a broadcast receiving device to display uncompressed audio-visual con tents that the broadcast receiving device provides. 0005 Additionally, the broadcast receiving device receives contents from a server (called a Multichannel Video Programming Distributor (MVPD)). The MVPD receives a broadcast signal from a broadcasting station, extracts con tents from the received broadcast signal, converts the extracted content into signals having a proper format for transmission, and provides the converted signals to a broad cast receiving device. During these processes, the MVPD may exclude extracted enhanced service data or may add another enhanced service data, so that the broadcasting sta tion may serve to provide AV contents but may not provide enhanced services led by the broadcasting station. 0006 Since the broadcast receiving device extracts main AV data from signal received from the MVPD and provides only uncompressed audio-visual data to a video display device, only enhanced services provided from the broadcast receiving device not the video display device are available. Furthermore, the broadcasting station may not provide enhanced services led by itself. 0007 Additionally, even when the broadcasting station provides enhanced services through wireless channels, the enhanced services may not be provided enough due to band width limitation. 0008. Additionally, when an enhanced service led by a broadcasting station is provided, it is displayed on a portion of a display unit of a video display device. Therefore, it may cause inconvenience for a user to watch contents.

SUMMARY

0009 Embodiments provide an image display, a terminal device, and an operating method thereof, which provide an available enhanced service led by abroadcasting station with out causing inconvenience for a user to watch contents.

Apr. 30, 2015

0010. In one embodiment, an operating method of a video display apparatus comprises pairing with a first terminal device; receiving an audio-visual content; displaying the audio-visual content; acquiring enhanced service informa tion on an available enhanced service for the audio-visual content; and notifying an occurrence of the available enhanced service to the first terminal device.

0011. In further embodiment, a video display apparatus, comprises a display unit; a receiver for receiving an audio visual content; an enhanced service manager for acquiring enhanced service information on an available enhanced ser Vice for the audio-visual content; and a reproducing control ler for reproducing the audio-visual content through the dis play unit, wherein the enhanced service manager pairs with a first terminal device, and notifies an occurrence of the avail able enhanced service to the first terminal device.

0012. In further another embodiment, an operating method of a terminal device comprises pairing with a video display apparatus; receiving, from the video display appara tus, an occurrence of the available enhanced service for an audio-visual content reproduced by the video display appa ratus; and executing the available enhanced service. 0013. In further another embodiment, a terminal device comprises a display unit; a communication unit for commu nicating with a video display apparatus; controller for pairing with a video display apparatus through the communication unit, for receiving, from the video display apparatus, an occurrence of the available enhanced service for an audio visual content reproduced by the video display apparatus, and for displaying the available enhanced service on the display unit

0014. In further another embodiment, an operating method of a video display apparatus comprises receiving an audio-visual content; displaying the audio-visual content; acquiring enhanced service information on an available enhanced service for the audio-visual content; and requesting an execution of the available enhanced service to a selected terminal device, if a terminal device is selected and executing of the available enhanced service is selected.

0015 The details of one or more embodiments are set forth in the accompanying drawings and the description below. Other features will be apparent from the description and drawings, and from the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

0016 FIG. 1 is a block diagram illustrating the network topology according to the embodiment. 0017 FIG. 2 is a block diagram illustrating a watermark based network topology according to an embodiment. 0018 FIG. 3 is a ladder diagram illustrating a data flow in a watermark based network topology according to an embodiment.

0019 FIG. 4 is view illustrating a watermark based con tent recognition timing according to an embodiment. 0020 FIG. 5 is a block diagram illustrating a fingerprint based network topology according to an embodiment. 0021 FIG. 6 is a ladder diagram illustrating a data flow in a fingerprint based network topology according to an embodi ment.

0022 FIG. 7 is a view illustrating an XML schema dia gram of ACR-Result type containing a query result according to an embodiment.

US 2015/O121416 A1

0023 FIG. 8 is a block diagram illustrating a watermark and fingerprint based network topology according to an embodiment. 0024 FIG. 9 is a ladder diagram illustrating a data flow in a watermark and fingerprint based network topology accord ing to an embodiment. 0025 FIG. 10 is a block diagram illustrating the video display device according to the embodiment. 0026 FIG. 11 is a hierarchy diagram of an enhanced ser Vice management unit according to an embodiment. 0027 FIG. 12 is a flowchart illustrating an operation of an enhanced service management unit according to an embodi ment.

0028 FIG. 13 is a hierarchy diagram of an enhanced ser Vice management unit according to an embodiment. 0029 FIG. 14 is a flowchart illustrating an operation of an enhanced service management unit according to an embodi ment.



0034 FIG. 19 is a flowchart illustrating a method of syn chronizing a playback time of a main AV content with a playback time of an enhanced service according to an embodiment. 0035 FIG. 20 is a conceptual diagram illustrating a method of synchronizing a playback time of a main AV con tent with a playback time of an enhanced service according to an embodiment. 0036 FIG.21 is a screen when an auto playing application

is executed according to an embodiment. 0037 FIG. 22 is a screen when a content provider appli cation is executed according to an embodiment. 0038 FIG. 23 is a screen when a video display device manufacturer application having a resizing user interface is executed according to an embodiment. 0039 FIG. 24 is a screen when a video display device manufacturer application having an overlay user interface is executed according to an embodiment. 0040 FIG. 25 is a flowchart illustrating a method of executing a video display device manufacturer application according to an embodiment. 0041 FIG. 26 is a screen when an application selection menu is displayed according to an embodiment. 0.042 FIG. 27 is a screen when an all sub menu of a program main menu in a video display device manufacturer application is selected according to an embodiment. 0043 FIG.28 is a screen when a participation sub menu of a program main menu in a video display device manufacturer application is selected according to an embodiment. 0044 FIG. 29 is a screen when a discovery sub menu of a program main menu in a video display device manufacturer application is selected according to an embodiment. 0045 FIG. 30 is flowchart illustrating a searching method according to an embodiment.

Apr. 30, 2015

0046 FIG. 31 is a screen when a search sub menu of an information main menu in a video display device manufac turer application is selected according to an embodiment. 0047 FIG.32 is another screen when a search sub menu of an information main menu in a video display device manu facturer application is selected according to an embodiment. 0048 FIG. 33 is flowchart illustrating a chatting window displaying method according to an embodiment. 0049 FIG. 34 is a screen when a Twitter sub menu of an information main menu in a video display device manufac turer application is selected according to an embodiment. 0050 FIG. 35 is a flowchart illustrating a news obtaining method according to an embodiment. 0051 FIG. 36 is a screen when a news sub menu of an information main menu in a video display device manufac turer application is selected according to an embodiment. 0052 FIG. 37 is flowchart illustrating a viewer list dis playing method according to an embodiment. 0053 FIG.38 is a screen when a viewer list sub menu of a Social main menu in a video display device manufacturer application is selected according to an embodiment. 0054 FIG. 39 is a signal flow when a video display device obtains a friends list and watching information according to an embodiment. 0055 FIG. 40 is a signal flow when a video display device obtains a friends list and watching information according to another embodiment. 0056 FIG. 41 is flowchart illustrating a preference dis playing method according to an embodiment. 0057 FIG. 42 is a screen when a preference display sub menu of a social main menu in a video display device manu facturer application is selected according to an embodiment. 0.058 FIG. 43 is flowchart illustrating a digital media downloading method according to an embodiment. 0059 FIG. 44 is a screen when a previous episode sub menu of a recommendation main menu in a video display device manufacturer application is selected according to an embodiment.

0060 FIG. 45 is a screen when a VOD sub menu of a recommendation main menu in a video display device manu facturer application is selected according to an embodiment. 0061 FIG. 46 is a screen when an application sub menu of a recommendation main menu in a video display device manufacturer application is selected according to an embodi ment.

0062 FIG. 47 is flowchart illustrating a catalog displaying method according to an embodiment. 0063 FIG. 48 is a screen when a style sub menu of a recommendation main menu in a video display device manu facturer application is selected according to an embodiment. 0064 FIG. 49 is a screen when a music sub menu of a recommendation main menu in a video display device manu facturer application is selected according to an embodiment. 0065 FIG. 50 is a screen when another sub menu of a recommendation main menu in a video display device manu facturer application is selected according to an embodiment. 0.066 FIG. 51 is a view of a user interface for controlling an ACR function according to an embodiment. 0067 FIG. 52 is a flowchart illustrating a method of con trolling a video display device according to the embodiment. 0068 FIG. 53 is a ladder diagram illustrating a pairing procedure according to an embodiment.

US 2015/O121416 A1

0069 FIG. 54 is a front view of a terminal having an interworking application executed according to an embodi ment.

0070 FIG.55 is an authentication number input window according to an embodiment. 0071 FIG. 56 is a ladder diagram illustrating an ACR control method according to an embodiment. 0072 FIG. 57 is a ladder diagram illustrating an inter working process between devices according to an embodi ment.

0073 FIG. 58 is a ladder diagram illustrating an inter working process between devices according to another embodiment. 0074 FIG. 59 is a front view of a terminal device display ing an enhanced service notification UI according to an embodiment. 0075 FIG. 60 is a front view of a terminal device display ing an enhanced service notification UI according to another embodiment. 0076 FIG. 61 is a front view of a terminal device display ing an enhanced service notification UI according to another embodiment. 0077 FIG. 62 is a front view of a terminal device where an enhanced service is executed according to an embodiment. 0078 FIG. 63 is a view when an enhanced service is executed in a video display device and a terminal device according to another embodiment. 007.9 FIG. 64 is a front view of a terminal device where an enhanced service is executed according to another embodi ment.

0080 FIG.65 is a front view of a terminal device where an enhanced service is executed according to another embodi ment.

0081 FIG. 66 is a ladder diagram illustrating an inter working process between devices according to another embodiment. 0082 FIG. 67 is a ladder diagram illustrating an inter working process between devices according to another embodiment. 0083 FIG. 68 is a ladder diagram illustrating an inter working process between devices according to another embodiment. 0084 FIG. 69 is a ladder diagram illustrating an inter working process between devices according to another embodiment. 0085 FIG. 70 is a ladder diagram illustrating an inter working process between devices according to another embodiment. I0086 FIG. 71 is a ladder diagram illustrating an inter working process between devices according to another embodiment. 0087 FIG. 72 is a ladder diagram illustrating an inter working process between devices according to another embodiment.

0088 FIG.73 is a front view of a terminal device where an interworking application is executed according to another embodiment.



Apr. 30, 2015

0091 FIG. 76 is a front view of a terminal device where an interworking application is executed according to another embodiment.






(0097 FIG. 82 is a top front view of a display unit in a terminal device where an interworking application is executed according to another embodiment. (0098 FIG. 83 is a flowchart illustrating a method of obtaining an AIT through an IP based network according to an embodiment.

(0099 FIG. 84 is view illustrating a structure of SD&S information according to an embodiment. 0100 FIG. 85 is view illustrating a structure of SD&S information according to another embodiment. 0101 FIG. 86 is view illustrating a structure of SD&S information according to an embodiment. 0102 FIG. 87 is a flowchart illustrating a method of obtaining information on an available enhanced service through a broadcast network according to an embodiment. 0103 FIG. 88 is a ladder diagram illustrating a method of providing an enhanced service according to another embodi ment.


0105 FIG. 90 is a front view of a terminal device execut ing the service for showing content information according to another embodiment.

0106 FIG. 91 is a ladder diagram illustrating an inter working method between devices according to another embodiment.

0107 FIG. 92 is a front view of a terminal device display ing UI that inquires of a user to determine whether to provide an enhanced service to a video display device according to an embodiment.

(0.108 FIG. 93 is a front view of a video display device receiving an enhanced service from a terminal device and executing it. 0109 FIG.94 is a block diagram illustrating a structure of a fingerprint based video display device according to another embodiment.

0110 FIG.95 is a block diagram illustrating a structure of a watermark based video display device according to another embodiment.

0111 FIG. 96 is a block diagram of a terminal device according to an embodiment.

(12) Patent Application Publication (10) Pub. No.: US 2015/0121416 ...

Documents

Transcript of (12) Patent Application Publication (10) Pub. No.: US 2015/0121416 ...