Note: Within nine months of the publication of the mention of the grant of the European patent in the European PatentBulletin, any person may give notice to the European Patent Office of opposition to that patent, in accordance with theImplementing Regulations. Notice of opposition shall not be deemed to have been filed until the opposition fee has beenpaid. (Art. 99(1) European Patent Convention).

Printed by Jouve, 75001 PARIS (FR)

(19)E

P0

996

292

B1

(Cont. next page)

TEPZZZ996 9 B_T(11) EP 0 996 292 B1

(12) EUROPEAN PATENT SPECIFICATION

(45) Date of publication and mention of the grant of the patent: 17.05.2017 Bulletin 2017/20

(21) Application number: 99308040.7

(22) Date of filing: 12.10.1999

(51) Int Cl.:H04N 19/66 (2014.01) H03M 13/35 (2006.01)

(54) Method and apparatus for receiving MPEG video over the internet and decoding thereof

Dekoder für MPEG-kodierte, über das Internet übertragene Bildsignale

Décodeur d’images codées par MPEG transmisées sur l’Internet

(84) Designated Contracting States: DE FR GB

(30) Priority: 22.10.1998 US 176982

(43) Date of publication of application: 26.04.2000 Bulletin 2000/17

(73) Proprietor: ALCATEL LUCENT92100 Boulogne-Billancourt (FR)

(72) Inventor: Macdonald Boyce, JillManalapan,New Jersey 07726 (US)

(74) Representative: Wetzel, Emmanuelle et alAlcatel-Lucent Intellectual Property Business Group70430 Stuttgart (DE)

(56) References cited: EP-A- 0 676 875 US-A- 5 231 486US-A- 5 481 312 US-A- 5 771 081US-A- 5 825 430

• BOYCE J M: "Packet loss resilient transmission of MPEG video over the Internet" SIGNAL PROCESSING: IMAGE COMMUNICATION, SEPT. 1999, ELSEVIER, NETHERLANDS, vol. 15, no. 1-2, 1 September 1999 (1999-09-01), pages 7-24, XP000867165 ISSN: 0923-5965

• RICHARDSON I E G ET AL: "MPEG coding for error-resilient transmission" FIFTH INTERNATIONAL CONFERENCE ON IMAGE PROCESSING AND ITS APPLICATIONS (CONF. PUBL. NO.410), FIFTH INTERNATIONAL CONFERENCE ON IMAGE PROCESSING AND ITS APPLICATIONS (CONF. PUBL. NO.410), EDINBURGH, UK, 4-6 JULY 1995, pages 559-563, XP000869826 1995, London, UK, IEE, UK ISBN: 0-85296-642-3

• CARLE G ET AL: "RTMC: an error control protocol for IP-based audio-visual multicast applications" PROCEEDINGS 7TH INTERNATIONAL CONFERENCE ON COMPUTER COMMUNICATIONS AND NETWORKS (CAT. NO.98EX226), PROCEEDINGS 7TH INTERNATIONAL CONFERENCE ON COMPUTER COMMUNICATIONS AND NETWORKS, LAFAYETTE, LA, USA, 12-15 OCT. 1998, pages 566-573, XP002128250 1998, Los Alamitos, CA, USA, IEEE Comput. Soc, USA ISBN: 0-8186-9014-3

• Archived Request for Comment Perkins, C., Hodson, O.: "Options for Repair of Streaming Media" June 1998, Internet Engineering Task Force IETF, Request for Comments RFC 2354. Retrieved from Internet: <URL:http://ring.exp.fujixerox.co.jp/pub/d oc/RFC/rfc2354.txt> 20 January 2000 XP002128256

• ARAVIND R ET AL: "Packet loss resilience of MPEG-2 scalable video coding algorithms" IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, OCT. 1996, IEEE, USA, vol. 6, no. 5, pages 426-435, XP000627031 ISSN: 1051-8215

2

EP 0 996 292 B1

• PODOLSKY M ET AL: "Simulation of FEC-based error control for packet audio on the Internet" PROCEEDINGS. IEEE INFOCOM ’98, THE CONFERENCE ON COMPUTER COMMUNICATIONS. SEVENTEENTH ANNUAL JOINT CONFERENCE OF THE IEEE COMPUTER AND COMMUNICATIONS SOCIETIES. GATEWAY TO THE 21ST CENTURY (CAT. NO.98CH36169), PROCEEDINGS IEEE INFOCOM’98 CONFERENCE O, pages 505-515 vol.2, XP002128251 1998, New York, NY, USA, IEEE, USA ISBN: 0-7803-4383-2

• CARLE G ET AL: "Survey of error recovery techniques for IP-based audio-visual multicast applications" IEEE NETWORK, NOV.-DEC. 1997, IEEE, USA, vol. 11, no. 6, pages 24-36, XP000737463 ISSN: 0890-8044

• ANDRONICO M ET AL: "Performance analysis of priority encoding transmission of MPEG video streams" IEEE GLOBECOM 1996. COMMUNICATIONS: THE KEY TO GLOBAL PROSPERITY. CONFERENCE RECORD (CAT. NO.96CH35942), PROCEEDINGS OF GLOBECOM’96. 1996 IEEE GLOBAL TELECOMMUNICATIONS CONFERENCE, LONDON, UK, 18-22 NOV. 1996, pages 267-271 vol.1, XP000742163 1996, New York, NY, USA, IEEE, USA ISBN: 0-7803-3336-5

EP 0 996 292 B1

3

5

10

15

20

25

30

35

40

45

50

55

Description

Field of the Invention

[0001] This invention relates to the transmission andreception of coded video signals over the Internet, andmore specifically, to the transmission and reception ofvideo signals that have been coded using compressionefficient inter-frame coding techniques such as thoseused in MPEG, MPEG-2, H.261, and H.263 standards.

Background of the Invention

[0002] With the exploding popularity of the public In-ternet in the past several years for transporting all typesof data, there has been much recent interest in transmit-ting digitally encoded real-time audio and video over theInternet using the Universal Datagram Protocol (UDP).Because UDP is an unreliable protocol, network packetlosses will likely occur and, as a result, will adverselyaffect the quality of the received audio and video. Recov-ery from packet losses may be performed solely by thereceiver, or better quality can be achieved by involvingboth the sender and the receiver in the error recoveryprocess. In networks that support prioritization, such asATM, video quality can be improved in the presence ofpacket loss by using scalable video coding (see, e.g., R.Aravind, M. Civanlar, A. Reibman, "Packet Loss Resil-ience of MPEG-2 Scalable Video CodingAlgorithms, "IEEE Transactions on Circuits and Systemsfor Video Technology, Vol. 6, No. 5, October 1996). Thereis currently, however, no widespread support for prioriti-zation on the public Internet. Overviews of proposedmethods for error recovery for streaming of audio andvideo over the Internet, which involve both the senderand the receiver are disclosed by C. Perkins and O. Hod-son in "Options for Repair of Streaming Media", InternetEngineering Task Force Internet RFC 2354, June 1987,and G. Carle and E. Biersack in "Survey of Error Recov-ery Techniques for IP-Based Audio-Visual Multicast Ap-plications," IEEE Network, November/December 1997.While the general methods described in these overviewsmay be applicable to IP transmission of both audio andvideo, most of the studies published where specifiedtechniques have been implemented involve audio only.Because of its higher data rates, and propagation of er-rors through inter-frame coding, it is more difficult to main-tain video quality than audio, and audio techniques,therefore, cannot be directly applied to video signals.In US5231486 (A), a high definition video system proc-esses a bitstream including high and low priority variablelength coded Data words. The coded Data is separatedinto packed High Priority Data and packed Low PriorityData by means of respective data packing units. The cod-ed Data is continuously applied to both packing units.High Priority and Low Priority Length words indicatingthe bit lengths of high priority and low priority componentsof the coded Data are applied to the high and low priority

data packers, respectively. The Low Priority Length wordis zeroed when High Priority Data is to be packed fortransport via a first output path, and the High PriorityLength word is zeroed when Low Priority Data is to bepacked for transport via a second output path.Moreover, in US5481312 (A), an apparatus and methodof transmitting a video bitstream from a transmitter overa facility to a receiver, the video bitstream including aplurality of high priority segments and associated low pri-ority segments, is disclosed. The transmitter first trans-mits high priority information, representative of the highpriority segments, of the video bitstream over the facilityusing a first packet delivery mechanism having a firstprobability of success and subsequently transmits a lowpriority partition, including the low priority segments, ofthe video bitstream over the facility using a second packetdeliver mechanism having a second probability of suc-cess which is substantially lower than that of the first de-livery mechanism. At the receiver, the high priority parti-tion is received and used to generate the high prioritysegments. When each low priority segment of the lowpriority partition is received, the associated high prioritysegment is interleaved in real time with the received lowpriority segment to create an interleaved video bitstream.The high priority information may be sent as a high prioritypartition (including the high priority segments), sent as ahigh priority format (used by the receiver to generate highpriority segments), or sent as a high priority identifier(used to identify previously stored high priority segmentsat the receiver).[0003] Many of the currently popular schemes fortransmitting digital video over the Internet, such as Mo-tion-JPEG and wavelet-based schemes, us intra-framecoding. Inter-frame coding techniques, such as thoseused in MPEG-1, MPEG-2; H.261, and H.263 standards,are generally more compression-efficient than intra-frame techniques. However, the inter-frame standardssuffer more from Internet packet loss because errors inone frame may propagate for many frames. An MPEGvideo sequence includes intra-frame coded (I) frames,and inter-frame predicted coded (P), and bi-directionalinter-frame coded (B) frames. I and P frames are usedin the prediction of subsequent frames while B framesare not used in the prediction of subsequent frames. Forexample, consider anMPEG video sequence with I frames occurring every 15frames. In MPEG coding, because of inter-frame predic-tion, all predictive P and B frames rely upon the previousI frame. Thus, if an error occurs while transmitting the Iframe, the effect persists for 15 frames, or 500 ms, whichis quite noticeable to a viewer. The received video qualitycan be improved both through error concealment tech-niques that are applied at the decoder, and by error re-silience techniques that are applied at the sender.[0004] Error resilience techniques using Forward Er-ror/Erasure Correction (FEC) add redundant data to amedia stream prior to transmission, so that packet lossescan be repaired at the receiver without requiring contact

1 2

EP 0 996 292 B1

4

5

10

15

20

25

30

35

40

45

50

55

with or retransmissions from the sender. Forward Er-ror/Erasure Correction techniques are well suited to mul-ticast applications, because they avoid the use of retrans-missions. The same redundant data can be used to repairthe loss of different packets at separate receivers in amulticast group. If re-transmission were used instead,multiple re-transmission requests would have to be sent.Forward Error/Erasure Correction techniques for multi-media generally fall into one of two categories, media-independent FEC and media-specific FEC (see, e.g., C.Perkins and O. Hodson, "Options for Repair of StreamingMedia," Internet Engineering Task Force Internet RFC2354, June 1998).[0005] In media-independent FEC, well-known infor-mation theory techniques for protecting any type of dataare used. In, "Media-independent Error Correction usingRTP," Internet Engineering Task Force Internet Draft,May 1997 by D. Budge, R. McKenzie, W. Mills, and P.Long, several variations of exclusive-OR (XOR) opera-tions are used to create parity packets from two or moredata packets. More complex techniques such as ReedSolomon (RS) coding can also be used (see, e.g., G.Carle and E. Biersack, "Survey of Error Recovery Tech-niques for IP-Based Audio-Visual Multicast Applica-tions," IEEE Nefwork, November/December 1997).Reed-Solomon encoding is an example of a systematicforward error/erasure correction code. A systematic for-ward error/erasure correction code is one in which theinformation bytes are transmitted in the codeword withoutmodification. Thus, in the absence of channel errors, noReed-Solomon decoding is necessary to recover the in-formation bytes. When an RS(n,k) codeword is construct-ed from byte data, h parity bytes are created from k in-formation bytes, and all n = k + h bytes are transmitted.Such a Reed Solomon decoder can correct up to any h/2byte errors, or any h byte erasures, where an erasure isdefined as an error in a known position. When RS codingis applied to protect packetized data against packet loss,k information packets of length j bytes are coded using jRS codewords. For each RS codeword, k informationbytes are taken from k different packets (one from eachpacket), and the h constructed parity bytes are placedinto h separate parity packets, and all n = k + h packetsare transmitted. Because the transmitted packets arenumbered, and packets are assumed to be received per-fectly or not at all, the receiver can determine which pack-ets are missing, and thus a packet loss can be consideredto be an erasure. Hence, if any h (or fewer) of the n trans-mitted packets are lost, the original k information packetscan be recovered perfectly.[0006] A key advantage of RS coding is its ability toprotect against several consecutive errors, depending onthe parameter choices. The overhead rate for RS codingis h/k, and it is most efficient for protection against bursterrors for large values of k. For example, an RS(6,4) codeand an RS(4,2) code both can protect against a burstlength of 2 errors. But the RS(4,2) code has 100% over-head, while the RS(6,4) code has only 50% overhead.

Reducing the overhead percentage by increasing theblock length, however, leads to delay because largeblock lengths require buffering of large amounts of dataprior to transmission.[0007] In media-specific FEC coding unlike in media-independent FEC coding where the multimedia streamis just treated as data, knowledge of the specific type ofmultimedia stream to be transmitted is used. In "Simula-tion of FEC-Based Error Control for Packet Audio on theInternet," INFOCOM, March 1998, San Francisco, CA byM. Podolsky, C. Romer, and S. McCanne, and in "Reli-able Audio for Use over the Internet," Proc. INET ’95,Honolulu, HI, pp. 171-178, June 1995, by V. Hardman,M.A. Sasse, M. Handley, and A. Watson. a redundantlow-bit rate audio stream is transmitted along with thestandard audio stream, but delayed by one packet. If astandard audio packet is lost, the receiver uses the low-bit rate version of that audio instead, received in the nextpacket. This method protects against single packet loss-es.[0008] In the aforenoted article by Perkins and Hodson,a suggestion is made to combine media-specific and me-dia-independent techniques by applying the media-inde-pendent FEC techniques to the most significant bytes ofa coder’s output, rather than applying FEC over the entiremultimedia bitstream. No specific information about howthis can be accomplished is given however. A methodfor adding resilient information to inter-frame coded vid-eo, such as MPEG video, in order to protect video qualityagainst packet loss, but which has low overhead and lowdelay is desirable.[0009] In the art, Richardson, et al "MPEG Coding ForError-Resilient Transmission," Fifth International Confer-ence On Image Processing And Its Applications, 1995,pp. 559-563 describes a layered coding scheme forMPEG encoded video and investigates the tolerance ofdifferent layers to bit error effects.

Summary of the Invention

[0010] A method and apparatus according to thepresent invention are set out in the independent claims,to which the reader is now referred. Preferred featuresare laid out in the dependent claims.[0011] In accordance with the present invention, an in-ter-frame coded video signal, such as an MPEG videosignal, employs a data splitting function to split such avideo stream into a high priority and a low priority parti-tion. Systematic Forward Error/Erasure Correction cod-ing is then performed on only the data in the high prioritypartition. The Forward Error/Erasure Corrected high pri-ority partition data and the non-Forward Error/ErasureCorrected low priority partition data are then combinedinto packets and transmitted over the same network to areceiver, where they are decoded. Depending on the de-gree of protection against errors or erasures offered bythe particular FEC code that is used, the loss of one ormore packets containing high priority data can be cor-

3 4

EP 0 996 292 B1

5

5

10

15

20

25

30

35

40

45

50

55

rected with no loss of data in the high priority partition.The effect of the loss of the low partition data in the lostpacket or packets, which low partition is not protected,has much less of a deleterious effect on the quality of thedecoded video signal than would the loss of data fromthe high priority partition data. Advantageously, by limit-ing the application of the Forward Error/Erasure Correc-tion to only the higher priority partition data, and thusprotecting against loss only that "more important data",the overhead requirement is reduced for protectionagainst a given packet loss.[0012] In the preferred embodiment, a Reed Solomonencoder is applied to the high priority data for an entireframe. For each RS(n,k) codeword, one information byteis taken from each of k packets and the constructed paritybytes are placed in h different packets, where n = k + h.Each individual frame’s data is arranged in the n equallength packets that contain a combination of: packetheaders; high priority data comprising one of either in-formation bytes or parity bytes; and low priority databytes, the latter comprising only information bytes sinceno error-correction coding is performed on the low prioritydata. The same number of bytes of high priority data (in-formation or parity in any one packet) are placed in eachof the n equal length packets, and the same number ofbytes of low priority data (information only) are placed inthese same n packets, which together represent the vid-eo frame. Amongst these n equal length packets, k pack-ets only contain high priority partition information bytesand h packets only contain the high priority parity bytes.The parity byte in each high priority byte position in eachof theses h packets is formed from the RS(n,k) code asit is applied to the k high priority partition information bytesin a corresponding byte position in the k other high prioritypartition information-containing packets associated withthe frame. Advantageously, arranging the packets in thismanner minimizes the amount of overhead and delay fora given packet loss protection rate.[0013] A receiving decoder, upon receiving the pack-ets associated with a frame separates the high prioritypartition bytes and low priority partition bytes in eachpacket according to the numbers of such bytes or eachtype included within each packets, which numbers aretransmitted in the packet headers. RS(n,k) decoding isapplied byte position-by-byte position across the high pri-ority partition portion within the received packets. If up toh of the n frame packets are lost, the RS decoding proc-ess recovers each high priority byte in the lost packet orpackets. Full reconstruction of the high priority partitioninformation bytes that were transmitted in the k packetsof the n packets that contained high priority partition datais thus effected. Although the low priority partition datain the lost packets is unrecoverable, the fully recoveredhigh priority partition data enables the video picture to bedecoded, albeit in what might be at a reduced qualitylevel for that frame or that portion of a frame in whichonly the high priority partition information is available.

Brief Description of the Drawing

[0014]

FIG. 1 is a block diagram of a first example of a videoencoder functioning with the decoder of the presentinvention which uses a data partitioning MPEG en-coder to code and split the coded video signal intoHP and LP partitions;FIG.-2 the arrangement of HP data and parity infor-mation, and LP data within the packets associatedwith a frame for an example of RS(4,3) encoding;FIG. 3 is a flow chart that details the steps for deter-mining the n and k parameters associated with aframe as a function of the number of LP and HP bytesin the frame;FIG. 4 is a block diagram of a second example of avideo encoder in accordance with the decoder of thepresent invention in which a standard MPEG encod-er is used to encode the video signal and a datasplitter divides the coded signal into HP and LP par-titions;FIG. 5 is a block diagram of an embodiment of avideo decoder in accordance with the present inven-tion in which lost packets are reconstructed, HP andLP partitions reformed, and a data partitioning MPEGdecoder is used to decode the video signal from theHP and LP partitions; andFIG. 6 is a block diagram of a second embodimentof a video decoder in accordance with the presentinvention in which a data merger combines the HPand LP partitions, which are then inputted to a stand-ard MPEG decoder to decode the video signal.

Detailed Description

[0015] The MPEG-2 standard (ITU-T Recommenda-tion H.262, "GENERIC CODING OF MOVING PIC-TURES AND ASSOCIATED AUDIO INFORMATION",July 1995) includes several means for performing scal-able video coding, including spatial scalability, SNR scal-ability, and data partitioning. In scalable video coding,receivers with different available bandwidths can receiveand decode appropriate representations of coded video,by receiving a base layer, and if bandwidth is available,receiving one or more enhancement layers. The morelayers received, the higher the quality of the decodedvideo. In the aforenoted-paper by Aravind, Civanlar andReibman, data partitioning was applied to provide pro-tection against packet loss for transmitting an MPEG-coded video signal in a network that supports prioritiza-tion such as ATM. Specifically, protection against packetloss was shown to be achievable by transmitting the baselayer at a higher priority than the enhancement layer(s).Since the public Internet does not currently support pri-oritization of packets this technique cannot be applied tothe transmission of coded video over the Internet.[0016] The inventor has recognized, however, that an

5 6

EP 0 996 292 B1

6

5

10

15

20

25

30

35

40

45

50

55

advantage can be achieved by using, in a non-prioritizednetwork, such as the public Internet, a data splitting func-tionality that is used in the prior art for packet protectionover a network that does support prioritization. By incor-porating a data splitting functionality together with a For-ward Error/Erasure Correction functionality for transmit-ting an inter-frame compression-coded video signal overa non-prioritized public Internet, the amount of overheadneeded for packet protection is reduced, while achievingthe improvement in the subjective video quality that pack-et protection affords. Further, by combining high priorityand low priority data in the same packets, the delay forequal overhead and protection is advantageously re-duced.[0017] FIG. 1 is a first embodiment of an encoder inaccordance with the present invention. In this embodi-ment of the present invention, an encoder which is com-pliant with MPEG-standardized data partitioning is usedto split an incoming video data stream into a high prioritypartition and a low priority partition. With reference toFIG. 1, an incoming video data stream on 101 is inputtedto a such a compliant data partitioning MPEG-standard-ized encoder 102 which, in accordance with the standard,such as the MPEG-2 data partitioning standard, com-pression-codes the input video bitstream and splits thecompression-coded bitstream into two output layers. Thefirst layer is the base layer, referred to herein as the highpriority (HP) partition. The second layer is the enhance-ment layer, referred to herein as the low priority (LP) par-tition.[0018] As is well known to one skilled in the art of MPEGcoding, an MPEG-coded video bitstream includes head-ers at the sequence level, at the group-of-picture (GOP)level, at the picture (frame) level, and at the slice level.As is well known, a slice consists of a group of adjacentmacroblocks, where each macroblock in itself consistsof a group of four adjacent luminance blocks of data andtwo chrominance blocks. At the picture level, a frame isclassified as being as an intra-frame coded (I) frame, aninter-frame coded predictive (P) frame, or a bi-directionalinter-frame predictive (B) frame. At the macroblock level,for the predictive-type P and B type frames, informationis included which indicates whether the macroblock isinter or intra coded with respect to another frame, as wellas motion vector information. The information at the blocklevel includes low frequency and high frequency discretecosine transformation (DCT) coefficients derived fromactual pixel information.[0019] In accordance with MPEG standards, data par-titioning can be effected at a plurality of different prioritybreakpoints, such as above the macroblock layer, at themacroblock layer, or within the macroblock layer. Gen-erally, the more critical parts of the coded bitstream, suchas the headers, the motion vectors and the low frequencyDCT coefficients, are allocated to the HP partition, andthe less critical data, such as the higher frequency DCTcoefficients, are allocated to the LP partition. In accord-ance with the standard, a priority breakpoint can be cho-

sen for each slice, which determines which codewordtypes are arranged into which partition.[0020] In the embodiment of the invention in FIG. 1,which uses the standardized data-partitioning MPEG en-coder 102, priority breakpoints are determined in accord-ance with the type of frame (I, P, or B) in the data streamthat is being partitioned. Specifically, in this embodiment,for each I frame, all the data is placed in the HP partition,with no data being placed in the LP partition. Other em-bodiments may divide the data from each I frame be-tween the HP and LP partitions. For each B frame, asmuch frame data as the MPEG standards relating to datapartitioning allows is placed in the LP partition, with theremaining data placed in the HP partition. For each Pframe, frame data is divided between the HP and LP par-titions so that data elements through the first two DCTcoefficients of each block are placed in the HP partition,with the higher order DCT coefficients placed in the LPpartition. Different priority breakpoints in the three frametypes between the LP and HP partitions other than de-scribed above for this embodiment could equally be used.It would be expected, however, that a higher breakpointwould be used for I frames than for P frames, which, inturn, would be higher than the breakpoint for B frames.In this standards-compliant embodiment, sequence,GOP, and picture headers are copied into both partitionsfor error resilience.[0021] In accordance with the invention, the HP parti-tion is encoded with a systematic Forward Error/ErasureCorrection code. Specifically, in this preferred embodi-ment, Reed Solomon coding is applied to the HP datafor a single frame across the entire frame. For this em-bodiment, only data from a single frame is operated uponby a Forward Error/Erasure Correction encoder/pack-etizer 103 at a time to minimize delay that would be in-curred if data from multiple frames needed to be accu-mulated before being coded and transmitted. In otherembodiments, B frames as well as their immediately fol-lowing anchor frame (I or P) may be operated upon to-gether by the Forward Error/Erasure Correction encod-er/packetizer 103.[0022] Reed Solomon encoding is applied to the HPdata output of the MPEG encoder 102 for the frame. Aswill be described in detail hereinafter, the number of bytesin the HP partition and the number of bytes in the LPpartition are used together and separately, together witha maximum allowable packet length and a desired de-gree of packet protection, to determine the number ofequal-length packets, n, into which, for that frame, theHP information bytes, the LP information bytes, and theprotective HP parity bytes will fit. In accordance with theinvention, each of the n equal-length packets contains acombination of both LP information bytes, and one of HPinformation and parity bytes. The parity bytes are deter-mined by forming an RS(n, k) codeword by taking oneHP information byte from each of k packets that containthe HP information bytes and placing the h parity bytesconstructed from these k bytes, one byte at a time, into

7 8

EP 0 996 292 B1

7

5

10

15

20

25

30

35

40

45

50

55

h different packets, where n = k + h. Thus, the parity bytesin those h packets in byte position m (where m variesbetween 1 and the number of HP information bytes in apacket) are derived from the HP information bytes in thatsame byte position m in the k packets. Since the paritybytes in each byte position in those h packets are calcu-lated from the HP information bytes in the same bytepositions in the other k packets, each equal-length packetcontains an equal number of HP data bytes, the latterbeing either all information bytes or all parity bytes. Itfollows, therefore, that each equal-length packet thencontains an equal number of LP information bytes. Be-cause the number of HP information bytes in the frameis not necessarily divisible by the number of k packets,padding bytes are added, as necessary, at the end of thekth packet. Similarly, padding bytes may be applied tothe nth packet for LP information bytes.[0023] FIG. 2 shows an example of an arrangement ofa frame group of packets for an RS (4, 3) code. As canbe noted, three packets (k = 3) contain HP informationbytes information bytes and LP information bytes. Onepacket (h = 1) contains the HP parity bytes and the LPinformation bytes. Arranging the packets in the mannerillustrated by this figure minimizes the amount of over-head for a given packet loss protection rate without add-ing to delay. The packet header in each packet containsthe packet number, the number of the packets in theframe group of packets (n), the number of packets withparity data in the frame group of packets (h), the temporalreference value for the frame, the frame type (I, P, or B),the HP/LP priority breakpoints, and the number of HPand LP bytes in each packet.[0024] With reference back to FIG. 1, the output of thedata partitioning MPEG encoder 102 consists, for an in-put stream of video data associated with a frame, of twodata streams, an HP partition data stream on output 104and a LP partition data stream on output 105, both ofwhich are inputted to the FEC Coder/Packetizer 103. Thenumber of bytes in the HP partition of the input frame,HP#, and the number of bytes in the LP partition of theinput frame, LP#, are inputted to a processor 106 withincoder/packetizer 103. As will be described in detail here-inafter, processor 106, using the values HP# and LP#,together with a predetermined maximum number of bytesper frame, determines the number of packets, n, neededto transmit the entire frame, including packet headers,the HP partition information bytes, the HP partition paritybytes, and the LP partition information bytes. Further,processor 106 determines the number of packets, h(equal to n-k), which are needed to protect the HP parti-tion information bytes in the other k packets for a desiredlevel of protection. Processor 106 also determines thebyte-length of each of the n packets. Processor 106 thendetermines the numbers of HP data or parity bytes andLP data bytes to be apportioned into each of these npackets, with LP information bytes allocated to each ofthe n packets, HP information bytes being allocated tothe k packets, and HP parity bytes being allocated to the

other h packets, as per the example shown in FIG. 2.[0025] Once processor 106 has determined all the pa-rameters that determine the configuration of each packetand the number of such packets, a Reed SolomonRS(n,k) encoding is effected, byte by byte on the byteswithin the HP partition. Specifically, the HP partition videostream is inputted to a byte interleaver 107, which in re-sponse to the parameters determined by processor 106,forms sequential codewords that are inputted to a ReedSolomon encoder 108. For example, if the number of HPbytes per packet are calculated to be 650, the 1st, 651st,1301st, et seq., bytes in the HP stream are interleavedby interleaver 107 to form an input word to RS encoder108 in order to determine the bytes for the first parity byteposition for the h packets which are to contain HP paritybytes. The 2nd, 652nd, 1302nd, et seq., bytes are theninterleaved to form the next input word to RS encoder108 to determine the parity bytes for the second paritybyte position for the h HP parity packets. In such a man-ner, the RS encoder 108 is sequentially presented witha series of k-byte input words, each byte in each k-byteinput word comprising one information byte from whatwill be, when re-interleaved, k different packets. For eachk-byte input word, h parity bytes are determined. Thus,for each k-byte input word, RS encoder 108 outputs ann-byte word containing k information bytes and h paritybytes.[0026] In order to reassemble these vertically orientedcodewords, each containing bytes properly belonging inn different packets, back into a row-oriented packet for-mat, a de-interleaver 109 sequentially places each bytefrom each n-byte output word into a separate one of ndifferent rows. The first k of such n rows thus contain allHP information bytes and the last h of such rows containall HP parity bytes. When the n rows are reconstructed,each such row is combined by a packetizer 110 with apacket header and the calculated fixed number of LPpartition information bytes apportioned to each packetfrom the LP output 105 of the data partitioning MPEGencoder 102. Each packet, then in a format as shown inFIG. 2 for the illustrative example for an RS(4,3) code,is sequentially outputted on output 111 for transmissiononto a UDP/IP network.[0027] For the example of FIG. 2, if none of the packetscontaining the frame information are lost in transmission,the decoded video will be perfectly decoded (i.e., identi-cal to that encoded). If one of the packets (or up to hpackets in the general case for R(n,k) coding) is lost, thenall of the HP information in that packet is recoverableusing a Reed Solomon decoder. The LP information datain the lost packet could not, however, be recovered sinceit is not protected. The portions of the picture that corre-spond to the macroblocks whose LP data was receivedwill, however, be perfectly decoded, and those that cor-respond to macroblocks whose LP data is lost will decodeonly the HP data for those macroblocks. Those macrob-locks which are decoded using only HP data may be no-ticeable to a viewer, but are unlikely to be visually offen-

9 10

EP 0 996 292 B1

8

5

10

15

20

25

30

35

40

45

50

55

sive as long as the HP/LP priority breakpoint is properlychosen. The exact visual quality of macroblocks decodedusing only HP data depends on this HP/LP priority break-point, as well as the characteristics of the video sourcematerial. In the embodiment of FIG. 1 in which a stand-ardized data partitioning MPEG encoder 102 is used tocompression-code the input video stream and form theHP and LP partitions, the lowest level header copied intoboth partitions is the picture header. Thus, once anypacket is lost, and with it the LP partition information con-tained therein, which cannot be recovered, the LP parti-tion data that is received in the next received packetscannot be properly incorporated into the decoding proc-ess since there is no identifiable spatial entry point withwhich that received data can be associated until a nextpicture header is received. Thus, with the embodimentof FIG. 1, the perceptual effect of lost LP partition infor-mation will extend until a packet is received that containsa next picture header within the received LP partition in-formation. Once that picture header is received, the LPpartition data that follows can be properly incorporatedwith the received HP partition data.[0028] As previously noted, the exact arrangement ofthe frame data into packets is a function of the numberof bytes of HP partition information and LP partition in-formation in the frame, the parameters of the RS(n,k)code, and the maximum packet size. For Internet Proto-col (IP) transmission, the maximum packet size is set,for this embodiment, to the Ethernet Maximum TransportUnit (MTU) size of 1500 bytes. The choice of the RS(n,k)code parameters depends on the number of packetsneeded to transmit the frame, on the network loss con-ditions, and the overhead percentage considered accept-able. Because Internet packet loss has a tendency to besomewhat bursty, the k/n ratio can be chosen to be lowerfor lower values of n and higher for higher values of n.Table 1 is an example of pairs of n and k values in which,based on the bursty nature of packet loss, the ratio k/nis set lower for lower values of n and higher for highervalues of n.

Table 1

n k

2 1

3 2

4 3

5 4

6 4

7 5

8 6

9 6

10 7

11 8

[0029] Given a list of (n,k) values such as that given inTable 1, the particular (n,k) pair to use for each framecan be determined using an iterative process. FIG. 3 isa flowchart that details the steps that processor 106 fol-lows for determining the these parameters. After the datapartitioning MPEG Encoder 102 splits the bitstream intothe HP partition and the LP partition, there are HP# bytesin the HP partition and LP# bytes in the LP partition. Atstep 301, an initial estimate is formed for the number ofpackets, n, needed to transmit this information, assuminga maximum packet size PSmax, which in this embodimentis equal to the Ethernet MTU size minus the number ofpacket header bytes used. This initial estimate of n isformed from n = ceil (HB# + LB#)/PSmax, where ceil () isthe ceiling function. This initial estimate of n is calculatedwithout consideration of the HP parity bytes, which mustbe incorporated into the frame. The value of k that cor-responds to the current estimate of n is retrieved, at step302, using Table 1. At step 303, the necessary packetsize for the HB# information bytes is calculated from Cur-PacketSize = ceil (HB#/k) + ceil (LB#ln), because theHB# information bytes are split among k packets (withparity bytes in the remaining h packets) and the LB# in-formation bytes are split among n packets. At step 304,CurPacketSize is compared with PSmax. If CurPackef-Size is smaller than PSmax, the current (n,k) parametersmay be used. If not, the packet size is too large, and nis incremented at step 305. Table 1 may yield a new kfor this incremented value of n. CurPacketSize is thenrecalculated using these new parameters. The processends at step 306 when the resultant parameters yields aCurPacketSize of less than PSmax. The actual number,HBP, of HP bytes (information or parity) in each of the nframes is then equal to ceil (HB#/k) and the actualnumber, LPB, of LP information bytes in each of the nframes is equal to ceil (LP#/n), the total number of bytes,less the length of the packet header, in each frame, TPB,thus being equal to HBP+LPB.[0030] As an illustrative example, a P frame will be as-sumed to have a size of 5632 bytes, in which HB# = 2129HP bytes and LB# = 3503 LP bytes. PSmax is assumedto be equal to 1475 bytes, derived from a 1500 byte MTUsize minus 25 bytes for a packet header. The initial esti-mate of the number of packets is n=ceil((2129+3503)/1475) = ceil (3.818) = 4. From Table 1, forn=4, it is seen that k=3. Using these parameters, HPB,LPB and TPB are calculated: HPB=ceil (2129/3)=710;LPB=ceil (3503/4)=876; and TPB = 710+876=1586,

(continued)

n k

12 9

13 9

14 10

15 11

11 12

EP 0 996 292 B1

9

5

10

15

20

25

30

35

40

45

50

55

which is greater than PSmax of 1475. Thus, that set ofparameters is not valid and another iteration is neces-sary. For this next iteration, n is incremented to 5. FromTable 1, for n=5, it is seen that k=4. Using these param-eters, HPB, LPB and TPB are calculated:HPB=ceil(2129/4)=533; LPB=ceil(3503/5)=701; andTPB=533+701=1234, which is less than PSmax of 1475.[0031] Once the parameters of n, k, HPB and LBP aredetermined for the frame, the aforedescribed Reed Solo-mon coding and arrangement of the HP information andparity bytes, and LP information bytes into packets canbe effected. Specifically, once processor 106 determinesthese parameters from the partitioned HP and LP datafor each frame, that frame’s data is interleaved, RS en-coded, de-interleaved and packetized for transmissionover a UDP/IP network, as previously described.[0032] In the embodiment of FIG. 1, a data partitioningMPEG encoder 102 was used to split the bitstream intothe HP and the LP partitions. FIG. 4 is a block diagramof an alternative embodiment which although not main-taining strict MPEG compliance, in order to reduce over-head and improve performance, can still be used withstandards compliant MPEG encoders and decoders.This embodiment may applied to MPEG-1 or MPEG-2video, as well as other similar video coding standardssuch as H.261 and H.263. In this embodiment a standardMPEG encoder (or other standard encoder) 401 com-pression codes the input video stream. A data splitter402 then divides the compression-coded output of en-coder 401 into HP and LP partitions. For purposes of thepresent invention, the combination of the standard MPEGencoder 401 with a data splitter 402 enables the splittingfunction to be performed more efficiently and with im-proved performance as compared to the data partitioningMPEG encoder 102 in FIG. 1. Specifically, only sliceheaders are duplicated in both partitions and all otherdata is placed in one partition or the other, but not both.Because of the frame alignment of packets and the con-tainment of frame information in the packet header them-selves, it is not necessary to duplicate the picture headersand above in each of the partitions, thereby minimizingoverhead. By providing slice headers in both the LP andHP partitions, however, a point of entry is provided forthe LP data upon a packet loss. Thus, in a received packetthat follows a lost packet, the LP data can be incorporatedinto the decoded picture after the decoder receives, withthe LP partition data, the next slice header. Advanta-geously, as compared the embodiment of FIG. 1, in whichthe point of insertion for the received LP data followinga packet loss was the next frame, the embodiment ofFIG. 4 provides a point of insertion for the LP data at thenext slice, thereby improving the picture quality of thedecoded video signal.[0033] As previously described, the data partitioningMPEG encoder 102 partitions I frames so that all framedata is placed in the HP partition to minimize the effectof an error in an I frame from propagating to other frames.Since B frames are not used for prediction, as much data

as is possible via the data partitioning standard is placedin the LP partition. In the embodiment of FIG. 4, all datafor all B frames is placed in the LP partition as opposedto the embodiment of FIG. 1 in which some data, in ac-cordance with the standards, is required in the HP parti-tion. In both the embodiment of FIG. 1 and FIG. 4, the Pframe data is divided between the HP and LP partitionsin the manner described above. Within each P framemacroblocks can be either inter-frame or intra-frame cod-ed. In the embodiment of FIG. 4, different priority break-points are chosen for inter and intra coded macroblocksin each P frame, which the data partitioning MPEG en-coder 102 in FIG. 1 could not do. Inter-coded macrob-locks decoded using only the HP partition (and not theLP partition) may retain high frequency information fromthe corresponding motion-compensated macroblocks inthe previous frame, while intra-coded macroblocks maynot. Hence, it is desirable to set the priority breakpointfor intra-coded macroblocks to include more DCT coef-ficients in the HP partition than are included in inter-codedmacroblocks. Advantageously, this reduces the over-head rate for given level of quality or improves the qualityfor the same overhead rate.[0034] The HP and LP outputs of data splitter 402 areapplied to a FEC coder /packetizer 403, which functionsin the same manner as FEC coder/packetizer does inFIG. 1, which functions have been described above. Thepackets outputed by FEC coder/packetizer 403 are thentransmitted over a UDP/IP network.[0035] Although the embodiment in FIG. 4 includes aseparate standard MPEG encoder 401 and data splitter402, it should be recognized that a data coder that sup-ports the described data splitting operation could equallyperform the same functions that encoder 401 and splitter402 together do.[0036] The encoder networks in FIG. 1 and FIG. 4transmit output packets over a UDP/IP network such asthe public Internet, to corresponding decoders connectedto that network. In the case of a multicasting encoder,the transmission may be simultaneously broadcast to aplurality of end-users who will each receive the transmit-ted information. Since different packets may be lost alongthe routes to different of such end-users, the FEC tech-niques employed in the present invention to protectagainst packet loss enable each individual decoder torecover the particular packets that may have been loston the path to that end-user up to, of course, the level ofpacket protection provided by the specific RS code usedby the encoder.[0037] A decoder network 500 that is associated withthe encoder network of FIG. 1 is illustrated in FIG. 5. Asa data partitioning MPEG encoder 102 is incorporatedwithin the encoder network of FIG. 1, the decoder networkof FIG. 5 incorporates a complimentary data partitioningMPEG decoder 510. In FIG. 5, the serial packets trans-mitted by the encoder network over the UDP/IP network501 are inputted to a de-packetizer/decoder 502. De-packetizer/decoder 502 includes a de-packetizer 503

13 14

EP 0 996 292 B1

10

5

10

15

20

25

30

35

40

45

50

55

which receives the serial packets and strips the headerinformation in each packet and passes it to a processor504. The packet header information includes: a packetnumber; a frame number; the type of frame (I, B, P); the(n,k) parameters which define the packet structure of theframe; and the number of HP partition bytes, HPB, andLP partition bytes, LPB, in each packet. Processor 504,from the header information, determines the start of theframe, and from the parameter n, "knows" that that manypackets are used to define the frame. Further, from thepacket numbers received, processor 504 determineswhich particular of those n packets, if any, are missingand their position within the sequence of these n packets.In response to receiving all such information from proc-essor 504, de-packetizer 503 strips off the packet headerof each packet within the frame and divides the data ineach packet into its HP and LP portions. For those pack-ets which are determined to be missing, de-packetizer503 inserts "0" bytes or an error code in the HP and LPdata streams. The HP serial byte stream output of de-packetizer 503 consists of n sub-packets-worth of HPdata, each sub-packet containing HPB bytes. This HPstream is inputted to an interleaver 505 to decode theRS(n,k) encoded words that exist across the sub-packetsand to replace any missing data from up to h lost packets.Thus, for each byte position across the sub-packet, abyte is selected from each such sub-packet at that byteposition to form an input word to an RS decoder 506.[0038] As an example, using the frame structure shownin FIG. 2 where n is equal to 4 and k is equal to 3, andwith HPB being equal 650 bytes, interleaver 505 selectsthe 1st, 651st, 1301st and 1951st bytes in the HP bytestream to form a 4-byte input word to RS decoder 506 todetermine the 3-byte corresponding output word, eachbyte in the three-byte output word being an HP informa-tion byte in the first byte position in each of the threeinformation sub-packets. Interleaver 505 then presentsthe 2nd, 652nd, 1302nd and 1952nd bytes in the HP bytestream to RS decoder 506 to determine the bytes in thesecond byte position in each of the three information sub-packets. Interleaver 505 similarly processes the 3rd

through the 650th bytes. The three information bytes foreach byte position are thus sequentially outputted by RSdecoder 506. Since the RS(4,3) code is capable of cor-recting up to one erasure, if one of the four packets islost and thus not received by the decoder network 500,RS decoder 506 will determine the missing byte in eachof the byte positions in the lost HP sub-packet. Thus, ifprocessor 504 determines that the third packet is lost,"0" bytes are inserted in each byte position from the1301st through the 1950th bytes in the HP byte stream.Processor 504 passes the location of these missing bytesto RS decoder 506, which when sequentially presentedwith the bytes from the three received packets, recoverseach missing byte in each byte location in the third pack-et. Thus, RS decoder 504 is able to recover the entiresub-packet of HP data in the single lost third packet. Ifmore than one packet is lost in this example, or more

than h packets in the general case, then RS decoder isunable to recreate the missing data and a sequence errorcode that can be recognized by the data partitioningMPEG decoder 510 is inserted in the HP byte stream inplace of the recovered data.[0039] Since RS decoder 506 outputs a k-byte wordfor each n-byte input word, in which each of the k bytesis associated with a different sub-packet, de-interleaver507 re-sequences each k-byte output word, one byte ata time, into k separate HP sub-packets to reformulatethe HP information in each transmitted packet. The HPinformation in these k sub-packets that is re-created and,if necessary, recovered, is outputted by de-interleaver507 and inputted on a first input of the data partitioningMPEG decoder 510. Where lost packets are unrecover-able, the data stream contains sequence error codes.[0040] The LP information in the n LP sub-packets de-rived by de-packetizer 503 is simultaneously inputted ona second input of data partitioning MPEG decoder 510.For those packets that are lost, the LP data cannot berecovered since forward error/erasure correction was notapplied to the LP partition. For those packets, therefore,the LP output of de-packetizer 503 includes a codewordthat will be recognized by the data partitioning MPEGdecoder 510 as being indicative of missing data.[0041] The LP and HP data stream inputs to data par-titioning MPEG decoder 510 are equivalent, in the ab-sence of any lost packets, to the HP and LP data streamoutputs of the data partitioning MPEG encoder 102 inFIG. 1. If up to and including h packets are lost, the HPdata stream inputted to decoder 510 is equivalent to theHP data stream outputted by encoder 102 and the LPdata stream inputted to decoder 510 includes codewordsmarking lost data. When more than h packets are lost sothat RS decoder 506 cannot recover the lost HP data,both the HP and LP data stream inputted to data parti-tioning MPEG decoder 510 have missing data, the HPdata stream including sequence error codes indicatingthe absence of actual data and the LP data stream in-cluding the recognizable codewords indicating lost data.[0042] Data partitioning MPEG decoder 510, in re-sponse to the inputted LP partition data and the HP par-tition data decompresses and reformulates the transmit-ted video data in accordance with its standardized algo-rithm. Where, with respect to specific pels in the frame,corresponding HP data is available (by being actuallyreceived or recovered) but LP data is not available, thesubjective video quality of the reconstructed video frameis degraded. Furthermore, that spatial portion of the re-constructed video frame that follows, in a scanningsense, those pels in the frame associated with the lostLP partition data, is also degraded since the LP partitiondata in the next received packet cannot be associatedwith specific spatial points within the frame and thus withthe HP data. As noted, only picture headers are includedwithin both the HP and LP partitions. Thus, until the nextpicture header is received, all LP partition data within thevideo frame that follows a lost packet will be unable to

15 16

EP 0 996 292 B1

11

5

10

15

20

25

30

35

40

45

50

55

be combined with spatially corresponding HP data to de-code and reconstruct the video signal. Further, since thetype of frames in this embodiment which are divided intoseparate HP and LP partitions are the P frames, whichare used to predict the next frame, the loss of LP datafor reconstructing the remainder of the frame will havean effect on the quality of the next P and B frames, albeitat a substantially reduced level as compared to the effectthat a total loss of data would have caused without thepresent invention in which the more important HP parti-tion data has been protected. In the event that more thanh packets are lost in transmission, both HP and LP par-tition data is lost and not recoverable. Standard error con-cealment techniques can then be used to minimize thedegradation in video quality.[0043] A decoder network 600 which is associated withthe encoder network of FIG. 4 is illustrated in FIG. 6. Inthis embodiment of a decoder network, a de-packetiz-er/decoder 601 receives from the UDP/IP network thedata stream that was transmitted by encoder network ofFIG. 4. De-packetizer/decoder 601 functions in a manneridentical to that of de-packetizer/decoder 502 in FIG. 5and includes the same elements as those described inconnection with the description above of FIG. 5. The de-packetizer, interleaver, RS decoder, de-interleaver andprocessor will not, therefore, be discussed in detail. Theoutputs of de-packetizer/decoder 601 are the LP and HPpartition data streams, as are the outputs of de-packetiz-er/decoder 502 in FIG. 5. In the absence of any lost pack-ets in a frame, the LP and HP partitions are equivalentto the HP and LP outputs of data splitter 402 in the en-coder network of FIG. 4. If one through h packets arelost, then any lost HP data is recovered and the HP datastream is equivalent to the output of data splitter 402.The LP stream outputted by de-packetizer/decoder 601includes, however, a codeword indicating that LP datahas been lost. Where more than h packets are lost, thenboth the HP and LP data streams include error codes.Alternatively, the de-packetizer/decoder 601 could sendinformation to a data merger 602 indicating the positionsof missing HP and LP information.[0044] The LP and HP data streams outputted by de-packetizeNdecoder 601 are inputted to the data merger602, which combines these data streams into a singledata stream having a format which is decodable by astandard MPEG decoder 603. MPEG decoder 603, usingits standard algorithm, decompresses the coded videosignal to reconstruct the digital video signal, which canthen be transformed to an analog video signal and dis-played on a video terminal.[0045] As earlier noted in connection with the descrip-tion of the encoder in FIG. 4, slice headers are includedin both the LP and HP partitions outputted by data splitter402. Thus, the for decoder in FIG. 6, upon a loss of apacket and its LP data, a point of entry can be locatedfor the LP data received in subsequent packets upondetection of the next slice header in such data. Thus,unlike the embodiment of the decoder in FIG. 5 in which,

as noted, received LP data could only be reincorporatedwith HP data upon receipt of the next picture header forthe next frame, the embodiment of FIG. 6 enables theLP data received in those packets following a lost packetto be incorporated with received or recovered HP dataat a point of entry at the next slice header. Therefore,only that portion of the frame that is associated with thelost LP data until the next received slice header will bedegraded, rather than the entire remainder of the frameas in the embodiment of FIG. 5. Further, by minimizingthe visually degraded portion of the decoded frame, thedegradation of the subsequent frames that may be pre-dicted based on that frame is also minimized.[0046] The preceding merely illustrates the principlesof the invention. It will thus be appreciated that thoseskilled in the art will be able to devise various arrange-ments which, although not explicitly described or shownherein, embody the principles of the invention and areincluded within its spirit and scope. Furthermore, all ex-amples and conditional language that have been recitedhereinabove are principally intended expressly to be onlyfor pedagogical purposes to aid the reader in understand-ing the principles of the invention and the concepts con-tributed by the inventor to furthering the art, and are tobe construed as being without limitation to such specifi-cally recited examples and conditions. Moreover, allstatements that made hereinabove reciting principles,aspects, and embodiments of the invention, as well asspecific examples thereof, are intended to encompassboth structural and functional equivalents thereof. Addi-tionally, it is intended that such equivalents include bothcurrently known equivalents as well as equivalents de-veloped in the future, i.e., any elements developed thatperform the same function, regardless of structure.[0047] Thus, for example, it will be appreciated bythose skilled in the art that the block diagrams hereinrepresent conceptual views of illustrative circuitry em-bodying the principles of the invention. Similarly, it willbe appreciated that any flow charts, flow diagrams, andthe like represent various processes which may be sub-stantially represented in computer readable medium andso executed by a computer or processor, whether or notsuch computer or processor is explicitly shown.[0048] The functions of the various elements shown inthe FIGS. described above, including functional blockslabeled as "processors" may be provided through the useof dedicated hardware as well as hardware capable ofexecuting software in association with appropriate soft-ware. When provided by a processor, the functions maybe provided by a single dedicated processor, by a singleshared processor, or by a plurality of individual proces-sors, some of which may be shared. Moreover, explicituse of the term "processor" or "controller" should not beconstrued to refer exclusively to hardware capable of ex-ecuting software, and may implicitly include, without lim-itation, digital signal processor (DSP) hardware, read-only memory (ROM) for storing software, random accessmemory (RAM), and non-volatile storage. Other hard-

17 18

EP 0 996 292 B1

12

5

10

15

20

25

30

35

40

45

50

55

ware, conventional and/or custom, may also be included.Their function may be carried out through the operationof program logic, through dedicated logic, through theinteraction of program control and dedicated logic, oreven manually, the particular technique being selectableby the implementer as more specifically understood fromthe context.[0049] In the claims hereof any element expressed asa means for performing a specified function is intendedto encompass any way of performing that function includ-ing, for example, a) a combination of circuit elementswhich performs that function or b) software in any form,including, therefore, firmware, microcode or the like,combined with appropriate circuitry for executing thatsoftware to perform the function. The invention as definedby such claims resides in the fact that the functionalitiesprovided by the various recited means are combined andbrought together in the manner which the claims call for.Applicant thus regards any means which can providethose functionalities as equivalent to those shown herein.

Claims

1. A method of decoding a compression-coded videosignal that has been packetized and transmitted overa packet-based network in a sequence of packetsthat are associated with at least one frame of thecompression-coded video signal, the method com-prising the steps of:

receiving packets associated with the at leastone frame of the compression-coded video sig-nal, the received packets containing low prioritypartition information bytes and forward er-ror/erasure correction (FEC)-coded high prioritypartition data bytes associated with the at leastone frame of the compression-coded video sig-nal, the compression-coded video signal havingbeen split into low priority and high priority par-titions before being packetized and transmitted,and the received FEC-coded high priority parti-tion data bytes comprising a combination of thehigh priority partition information bytes associ-ated with the at least one frame and associatedparity bytes;determining which transmitted packets associ-ated with the at least one frame of the video se-quence have not been received;applying a FEC decoding only to the FEC-codedhigh priority partition data bytes in the receivedpackets to determine high priority partition infor-mation bytes associated with the at least oneframe, wherein the step of applying a FEC de-coding includes reconstructing high priority par-tition information bytes in those packets deter-mined not to have been received;combining the received low priority partition in-

formation bytes and the decoded and recon-structed high priority partition information bytes;and

decompression-decoding the combined re-ceived low priority partition informationbytes and the decoded and reconstructedhigh priority partition information bytes to re-produce the at least one frame of the videosignal,

wherein each received packet has an equallength and each packet contains both high pri-ority partition data bytes and low priority partitioninformation bytes, the high priority partition databytes in each received packet comprising eitherall high priority partition information bytes asso-ciated with the at least one frame or all paritybytes.

2. The method of claim 1 further comprising the stepof providing, to the step of decompression-decodingthe low priority partition information bytes and thehigh priority partition information bytes, informationindicating which low priority partition informationbytes are missing as a result of having been trans-mitted in a packet determined not to have been re-ceived.

3. The method of any of the preceding claims whereinthe packet-based network is the Internet.

4. The method of any of the preceding claims whereinthe received FEC-coded high priority partition databytes are coded with a systematic forward er-ror/erasure correction code.

5. The method of one of the preceding claims whereinan equal number of low priority partition informationbytes are in each received packet and an equalnumber of high priority partition data bytes are ineach received packet.

6. The method of claim 4 to 5 wherein for each highpriority byte position associated with the receivedpackets, high priority partition information bytes inthe same high priority byte position from each re-ceived packet containing high priority partition infor-mation bytes, one byte per packet, are associatedwith at least one parity byte in the same byte positionin one or more received packets containing paritybytes, the FEC decoding being applied to the highpriority partition information bytes and associated atleast one parity byte at each high priority byte posi-tion, one byte position at a time, to determine highpriority partition information bytes in each high pri-ority byte position in a packet that was not received.

19 20

EP 0 996 292 B1

13

5

10

15

20

25

30

35

40

45

50

55

7. The method of claim 6 wherein the FEC-coded-bytesare coded with a Reed Solomon code and the FECdecoding uses Reed Solomon decoding.

8. The method of claim 6 further comprising the stepsof:

determining the number of packets (n) in whichthe at least one frame of the compression-codedvideo signal has been packetized and transmit-ted; anddetermining of the n transmitted packets, thenumber of packets (k) transmitted containingboth high priority partition information bytes as-sociated with the at least one frame and low pri-ority partition information bytes.

9. The method of claim 8 wherein the FEC decodinguses Reed Solomon (n, k) decoding.

10. The method of claim 8 wherein the received packetsassociated with the at least one video frame of thevideo signal are protected against a packet loss overthe packet-based network of high priority partitioninformation bytes in up to (n-k) packets.

11. The method of any of the preceding claims whereinthe step of decompression decoding the video signaluses MPEG decoding.

12. A decoder for decoding a compression-coded videosignal that has been packetized and transmitted overa packet-based network in a sequence of packetsthat are associated with at least one frame of thecompression-coded video signal, the decoderCHARACTERIZED BY:

means arranged to carry out each step of amethod as claimed in any of the precedingclaims.

Patentansprüche

1. Verfahren zum Decodieren eines kompressionsco-dierten Videosignals, das über ein paketbasiertesNetz in einer Sequenz von Paketen paketiert undübermittelt wurde, die mit mindestens einem Rah-men des kompressionscodierten Videosignals asso-ziiert sind, wobei das Verfahren die folgenden Schrit-te umfasst:

den Empfang von Paketen, die mit mindestenseinem Rahmen des kompressionscodierten Vi-deosignals assoziiert sind, wobei die empfan-genen Pakete IO-Partitionsinfarmationsbytesniedriger Priorität und Partitionsinformationsby-tes hoher Priorität mit Codierung des Typs Vor-

wärtsfehler-/Erasurekorrektur (Forward Er-ror/Erasure Correction, FEC) enthalten, die mitmindestens einem Rahmen des kompressions-codierten Videosignals assoziiert sind, wobeidas kompressionscodierte Videosignal vor demPaketieren und Übermitteln in Partitionen nied-riger Priorität und hoher Priorität aufgespaltenwurde, und wobei die empfangenen, FEC-co-dierten Partitionsdatenbytes hoher Priorität eineKombination der Partitionsinformationsbyteshoher Priorität umfassen, die mit dem mindes-tens einen Rahmen und den assozüerten Pari-tätsbytes assoziiert sind;das Bestimmen, welche übermittelten, mit demmindestens einen Rahmen der Videosequenzassoziierten Pakete nicht empfangen wurden;das Anwenden einer FEC-Decodierung nur aufdie FEC-codierten Partitionsinformationsbyteshoher Priorität in den empfangenen Paketen zurBestimmung von Partitionsinformationsbyteshoher Priorität, die mit dem mindestens einenRahmen assoziiert sind, wobei der Schritt desAnwendens einer FEC-Decodierung umfasst,Partitionsinformationsbytes hoher Priorität indenjenigen Paketen zu rekonstruieren, für diebestimmt wurde, dass sie nicht empfangen wur-den;das Kombinieren der empfangenen Partitions-informationsbytes niedriger Priorität sowie derdecodierten und rekonstruierten Partitionsinfor-mationsbytes hoher Priorität; unddas Dekomprimieren/Decodieren der kombi-nierten, empfangenen Partitionsinformations-bytes niedriger Priorität und der decodierten undrekonstruierten Partitionsinformationsbytes ho-her Priorität, um mindestens einen Rahmen desVideosignals zu reproduzieren,wobei jedes empfangene Paket von gleicherLänge ist und jedes Paket sowohl Partitionsda-tenbytes hoher Priorität und Partitionsinformati-onsbytes niedriger Priorität umfasst, wobei diePartitionsdatenbytes hoher Priorität in einem je-den empfangenen Paket entweder alle Partiti-onsinformationsbytes hoher Priorität, die mitdem mindestens einen Rahmen assoziiert sind,oder alle Paritätsbytes umfassen.

2. Das Verfahren nach Anspruch 1, weiterhin denSchritt umfassend, für den Schritt des Dekomprimie-rens/Decodierens der Partitionsinformationsbytesniedriger Priorität und der Partitionsinformationsby-tes hoher Priorität Informationen zur Verfügung zustellen, die anzeigen, welche Partitionsinformations-bytes niedriger Priorität aus dem Grund fehlen, weilsie in einem Paket übermittelt wurden, für das be-stimmt wurde, dass es nicht empfangen wurde.

3. Das Verfahren nach einem beliebigen der vorste-

21 22

EP 0 996 292 B1

14

5

10

15

20

25

30

35

40

45

50

55

henden Ansprüche, wobei das paketbasierte Netzdas Internet ist.

4. Das Verfahren nach einem jeglichen der vorgenann-ten Ansprüche, wobei die empfangenen, FEC-co-dierten Partitionsinformationsbytes hoher Prioritätmit einem systematischen Vorwärtsfehler-/Erasure-korrekturcode codiert sind.

5. Das Verfahren nach einem der vorgenannten An-sprüche, wobei in jedem empfangenem Paket einegleiche Anzahl von Partitionsinformationsbytesniedriger Priorität und in jedem empfangenem Paketeine gleiche Anzahl von Partitionsdatenbytes hoherPriorität enthalten sind.

6. Das Verfahren nach Ansprüche 4 bis 5, wobei fürjede Byteposition hoher Priorität, die mit den emp-fangenen Paketen assoziiert ist, mit mindestens ei-nem Paritätsbyte in derselben Byteposition in einemoder mehreren empfangenen, Paritätsbytes enthal-tenden Paketen Partitionsinformaticrnsbytes hoherPriorität in derselben Byteposition hoher Priorität vonjedem empfangenen Paket assoziiert sind, das Par-titionsinformationsbytes hoher Priorität enthält, undzwar jeweils ein Byte pro Paket, wobei die FEC-De-codierung auf die Partitionsinformationsbytes hoherPriorität angewendet wird, und wobei mindestensein Paritätsbyte mit jeder Byteposition hoher Prioritätassoziiert ist, und zwar zum jeweiligen Zeitpunkt mitnur jeweils einer Byteposition, um Partitionsinforma-tionsbytes hoher Priorität in jeder Byteposition hoherPriorität in einem Paket zu bestimmen, das nichtempfangen wurde.

7. Das Verfahren nach Anspruch 6, wobei die FEC-codierten Bytes mit einem Reed-Solomon-Code co-diert sind und wobei die FEC-Decodierung Reed-Solomon-Decodierung verwendet.

8. Verfahren nach Anspruch 6, weiterhin die folgendenSchritte umfassend:

das Bestimmen einer Anzahl von Paketen (n),in welchen der mindestens eine Rahmen deskompressionscodierten Videosignals paketiertund übermittelt wurde; unddas Bestimmen der Anzahl übermittelter Pakete(k) aus den n übermittelten Paketen, welche so-wohl Partitionsinformationsbytes hoher Prioritätumfassen, die mit dem mindestens einen Rah-men assoziiert sind, als auch Partitionsinforma-tionsbytes niedriger Priorität.

9. Das Verfahren nach Anspruch 8, wobei die FEC-Decodierung Reed-Solomon-Decodierung (n, k)verwendet.

10. Das Verfahren nach Anspruch 8, wobei die empfan-genen Pakete, die mit dem mindestens einen Vide-orahmen des Videosignals assoziiert sind, geschütztsind gegen einen Paketverlust von Partitionsinfor-mationsbytes hoher Priorität über das paketbasierteNetzwerk in bis zu (n-k) Paketen.

11. Das Verfahren nach einem jeglichen der vorgenann-ten Ansprüche, wobei der Schritt des Dekomprimie-rens/Decodierens des Videosignals MPEG-Deco-dierung verwendet.

12. Decoder zum Decodieren eines kompressionsco-dierten Videosignals, das über ein paketbasiertesNetz in einer Sequenz von Paketen paketiert undübermittelt wurde, die mit mindestens einem Rah-men des kompressionscodierten Videosignals asso-ziiert sind, wobei der Decodierer CHARAKTERI-SIERT IST DURCH:

Mittel, die eingerichtet sind für das Durchführenaller Schritte eines Verfahrens gemäß einembeliebigen der vorgenannten Ansprüche.

Revendications

1. Procédé de décodage d’un signal vidéo codé parcompression qui a été paquetisé et transmis sur unréseau à commutation de paquets dans une séquen-ce de paquets qui sont associés à au moins unetrame du signal vidéo codé par compression, le pro-cédé comprenant les étapes suivantes :

recevoir des paquets associés à l’au moins unetrame du signal vidéo codé par compression,les paquets reçus contenant des octets d’infor-mation de partition de basse priorité et des oc-tets de données de partition de haute priorité àcodage de correction d’erreur/d’effacementsans voie de retour (FEC) associés à l’au moinsune trame du signal vidéo codé par compres-sion, le signal vidéo codé par compression ayantété divisé en partitions de basse priorité et dehaute priorité avant d’être paquetisé et transmis,et les octets de données de partition de hautepriorité à codage FEC reçus comprenant unecombinaison des octets d’information de parti-tion de haute priorité associés à l’au moins unetrame et d’octets de parité associés ;déterminer quels paquets transmis associés àl’au moins une trame de la séquence vidéo n’ontpas été reçus ;appliquer un décodage FEC uniquement auxoctets de données de partition de haute prioritéà codage FEC des paquets reçus pour détermi-ner les octets d’information de partition de hautepriorité associés à l’au moins une trame, l’étape

23 24

EP 0 996 292 B1

15

5

10

15

20

25

30

35

40

45

50

55

d’application d’un décodage FEC incluant unereconstruction des octets d’information de par-tition de haute priorité dans les paquets dont ila été déterminé qu’ils n’ont pas été reçus ;combiner les octets d’information de partition debasse priorité reçus et les octets d’informationde partition de haute priorité décodés etreconstruits ; etdécoder par décompression les octets d’infor-mation de partition de basse priorité reçus com-binés et les octets d’information de partition dehaute priorité décodés et reconstruits afin de re-produire l’au moins une trame du signal vidéo,dans lequel tous les paquets reçus sont de lon-gueur identique et tous les paquets contiennentà la fois des octets de données de partition dehaute priorité et des octets d’information de par-tition de basse priorité, les octets de donnéesde partition de haute priorité de chaque paquetreçu comprenant soit tous les octets d’informa-tion de partition de haute priorité associés à l’aumoins une trame ou aux trames, soit tous lesoctets de parité.

2. Procédé selon la revendication 1 comprenant enoutre l’étape de fourniture, à l’étape de décodagepar décompression des octets d’information de par-tition de basse priorité et des octets d’information departition de haute priorité, d’informations indiquantquels octets d’information de partition de basse prio-rité sont manquants en raison de leur transmissiondans un paquet dont il a été déterminé qu’il n’a pasété reçu.

3. Procédé selon l’une quelconque des revendicationsprécédentes dans lequel le réseau à commutationde paquets est le réseau Internet.

4. Procédé selon l’une quelconque des revendicationsprécédentes dans lequel les octets de données departition de haute priorité à codage FEC reçus sontcodés avec un code de correction d’erreur/d’efface-ment sans voie de retour systématique.

5. Procédé selon l’une quelconque des revendicationsprécédentes dans lequel un nombre égal d’octetsd’information de partition de basse priorité sont con-tenus dans chaque paquet reçu et un nombre égald’octets de données de partition de haute prioritésont contenus dans chaque paquet reçu.

6. Procédé selon les revendications 4 à 5 dans lequelpour chaque position d’octet de haute priorité asso-ciée aux paquets reçus, les octets d’information departition de haute priorité situés à la même positiond’octet de haute priorité dans chaque paquet reçucontenant des octets d’information de partition dehaute priorité, un octet par paquet, sont associés à

au moins un octet de parité situé à la même positiond’octet dans un ou plusieurs paquets reçus conte-nant des octets de parité, le décodage FEC étantappliqué aux octets d’information de partition de hau-te priorité et à l’octet ou aux octets de parité associésau niveau de chaque position d’octet de haute prio-rité, une position d’octet à la fois, pour déterminerles octets d’information de partition de haute prioritédans chaque position d’octet de haute priorité d’unpaquet qui n’a pas été reçu.

7. Procédé selon la revendication 6 dans lequel les oc-tets à codage FEC sont codés avec un code ReedSolomon et le décodage FEC utilise le décodageReed Solomon.

8. Procédé selon la revendication 6 comprenant enoutre les étapes suivantes :

déterminer le nombre de paquets (n) dans les-quels l’au moins une trame du signal vidéo codépar compression a été paquetisée et transmise ;etdéterminer parmi les n paquets transmis, lenombre de paquets (k) transmis contenant à lafois des octets d’information de partition de hau-te priorité associés à l’au moins une trame etdes octets d’information de partition de bassepriorité.

9. Procédé selon la revendication 8 dans lequel le dé-codage FEC utilise un décodage Reed Solomon (n,k).

10. Procédé selon la revendication 8 dans lequel les pa-quets reçus associés à l’au moins une trame vidéodu signal vidéo sont protégés contre la perte de pa-quet sur le réseau à commutation de paquets pourles octets d’information de partition de haute prioritédans (n-k) paquets au maximum.

11. Procédé selon l’une quelconque des revendicationsprécédentes dans lequel l’étape de décodage pardécompression du signal vidéo utilise le décodageMPEG.

12. Décodeur destiné à décoder un signal vidéo codépar compression qui a été paquetisé et transmis surun réseau à commutation de paquets dans une sé-quence de paquets qui sont associés à au moinsune trame du signal vidéo codé par compression, ledécodeur étant caractérisé par :

des moyens conçus pour réaliser chaque étapede procédé selon l’une quelconque des reven-dications précédentes.

25 26

EP 0 996 292 B1

16

EP 0 996 292 B1

17

EP 0 996 292 B1

18

EP 0 996 292 B1

19

REFERENCES CITED IN THE DESCRIPTION

This list of references cited by the applicant is for the reader’s convenience only. It does not form part of the Europeanpatent document. Even though great care has been taken in compiling the references, errors or omissions cannot beexcluded and the EPO disclaims all liability in this regard.

Patent documents cited in the description

• US 5231486 A [0002] • US 5481312 A [0002]

Non-patent literature cited in the description

• R. ARAVIND ; M. CIVANLAR ; A. REIBMAN. PacketLoss Resilience of MPEG-2 Scalable Video CodingAlgorithms. IEEE Transactions on Circuits and Sys-tems for Video Technology, October 1996, vol. 6 (5[0002]

• C. PERKINS ; O. HODSON. Options for Repair ofStreaming Media. Internet Engineering Task ForceInternet RFC 2354, June 1987 [0002]

• G. CARLE ; E. BIERSACK. Survey of Error Recov-ery Techniques for IP-Based Audio-Visual MulticastApplications. IEEE Network, November 1997 [0002]

• C. PERKINS ; O. HODSON. Options for Repair ofStreaming Media. Internet Engineering Task ForceInternet RFC 2354, June 1998 [0004]

• D. BUDGE ; R. MCKENZIE ; W. MILLS ; P. LONG.Media-independent Error Correction using RTP. In-ternet Engineering Task Force Internet Draft, May1997 [0005]

• G. CARLE ; E. BIERSACK. Survey of Error Recov-ery Techniques for IP-Based Audio-Visual MulticastApplications. IEEE Nefwork, November 1997 [0005]

• M. PODOLSKY ; C. ROMER ; S. MCCANNE. Sim-ulation of FEC-Based Error Control for Packet Audioon the Internet. INFOCOM, March 1998 [0007]

• V. HARDMAN ; M.A. SASSE ; M. HANDLEY ; A.WATSON. Reliable Audio for Use over the Internet.Proc. INET ’95, June 1995, 171-178 [0007]

• RICHARDSON et al. MPEG Coding For Error-Resil-ient Transmission. Fifth International Conference OnImage Processing And Its Applications, 1995,559-563 [0009]

• GENERIC CODING OF MOVING PICTURES ANDASSOCIATED AUDIO INFORMATION. ITU-T Rec-ommendation H.262, July 1995 [0015]