TelcoTV is always a place for us to go and catch up on what’s happening in IPTV, particularly in the US. “Catching up on IPTV” has meant looking at TV based on Internet protocols and provided over managed networks. Our visit to TelcoTV 2006 in Dallas last month provided a couple of interesting observations compared to what we saw and wrote about from TelcoTV 2005 a year ago in Switching Channels at TelcoTV ( www.broadbandhomecentral.com/report/backissues/Report0509_5.html ) (BBHR 12/15/2005).
First, it's clear that the "second wave" of IPTV is now under way. The first wave was characterized by pioneering vendors and telephone companies, standard definition television, and getting the most out of ADSL and early VDSL. The second wave features many of the world's largest telephone companies and vendors, high definition television, and early deployments of VDSL2.
Second, although the show is still largely focused on IPTV delivered over managed networks, there was increasing attention being paid to "online video" provided over the unmanaged Internet. This was a reflection of the emergence over the past year of a plethora of new services, ranging from user-generated content to delivery of commercial content over the Internet; we discussed these in VON: "V" Is For Video ( www.broadbandhomecentral.com/report/backissues/Report0608_6.html ) about the Fall VON show.
We listened to many keynote speeches and panel discussions, and talked with many vendors on the show floor. We saw people from many of the second and third tier telephone companies who have already deployed IPTV and were happy to share their pioneering experience. We also saw first tier players who are just starting to roll out their services.
We have grouped what we learned into a few topics:
Advanced Video Codecs and SoCs
Most phone companies now see the need to deliver multiple streams of HD. Unless they've deployed fiber to the home, they've had to adopt AVC and VC-1, the most advanced video codecs, to get a low enough bit rate for HD over DSL. (AVC is also called "Advanced Video Coding", "MPEG-4 Part 10," and "ITU-T H.264"; VC-1 is also called "Windows Media Video 9 (WMV-9)").
The decoders for AVC and VC-1 are implemented as highly integrated system-on-chip (SoC) chips. These chips reduce the parts count and cost of advanced set top boxes for IPTV. While the SoCs sharted shipping to box makers earlier in the year, the advanced set top boxes have only recently started shipping to customers, much later than forecast. The main delay seems to have been the complex software integration getting the new chips working in the boxes.
Video Quality Issues
Several sessions at the show focused on video quality issues. From the session presentations and our discussions with vendors, it appears that most customer complaints of poor video quality are caused by jitter and lost IP packets somewhere between the video headend and the set top box in the home. It also appears that the appropriate test equipment to isolate the problems has only now become available.
Learning from the Real World
In a session called "IPTV: Making the Internal Transformation", two senior telco executives--Mike Knoll, Chief Technology Officer of Hancock Telecom and Keith Galitz, President of Canby Telcom--talked frankly about the real-world issues of setting up and operating an IPTV system. This was a great opportunity to learn from the smaller telcos who have been the IPTV pioneers in the US.
The discussions covered a wide range of topics. We were struck by their reported difficulties of dealing with video quality. When a customer complained of poor video quality, they sometimes found it difficult to determine whether the problem was coming from the video headend, the DSL link, or the home networking technology linking the residential gateway and the set top box. It could take them repeated service calls to identify and fix the problem, and sometimes they lost the customer back to cable or satellite before they could isolate the problem.
As we talked with other people at the show, we learned that many think the home network is the primary source of packet loss, and the DSL link is next. While installing a dedicated Category 5 cable between the residential gateway and the IPTV set top box works very well, many telcos are trying to reduce the cost and time required to install dedicated wiring, and are using other technologies based on existing cabling or wireless. Jitter and lost packets in these networks show up as customer reports of poor video quality. Technical leaders at the show said the first priority was to engineer the DSL and home networks properly to minimize jitter and interference.
Troubleshooting the IPTV Network
At the show, several vendors displayed test equipment to measure packet loss and isolate the source of video quality problems. Some is designed for lab simulation, other for actual measurement.
Perhaps the most impressive diagnostic product was announced by IneoQuest Technologies shortly before the show. Their new Video Quality Management System (iVMS) is an end-to-end IPTV video quality and service assurance solution. It monitors the video quality in the headend, the core network, the edge network and the home, and integrates with the telco's existing management systems.
Tools like iVMS should provide telcos with much better information than they now have on the source of video quality problems and the best approach to resolving them.
"Loss Concealment" Algoritms
Several people told us that the software in IPTV set top boxes differs a lot in how well it copes with lost packets. They said some set top boxes include "loss concealment algorithms" which try to mask the lost packets by creating the best possible image from other video information stored in the video buffer, and by quickly resynchronizing video and audio streams to restore lip synchronization. We were told that there are "orders of magnitude" differences between how well different set top boxes cope with packet loss, and that this should be one of the major criteria telcos use in selecting between the many set top boxes on the market.
FEC and "Resilient UDP"
A common way to reduce the impact of packet loss is to include some form of forward error correction (FEC) in the packet stream. If FEC is applied in the video head end, it can substantially eliminate the impact of packet loss at any point in the network.
At the show, a company called Digital Fountain showed its FEC solution. Its DF Raptor FEC technology has been adopted by several international standards bodies for video transmission over wireless networks. To use DF Raptor in IPTV networks requires encoding equipment at the video headend, and decoding software in the IPTV set top box. Digital Fountain says their system can be implemented in today's IPTV set top boxes, and can be "tuned" to find the best balance between network bandwidth and video quality.
Since FEC requires additional bandwidth for the redundant packets, some telcos have been reluctant to use it. Microsoft's IPTV system, being deployed by many tier one telcos, uses a proprietary approach called "resilient UDP". When a set top box is missing a packet, it requests a replacement from a nearby server; both the server and the set top box maintain fairly large buffers to provide time for the server to respond to these requests.
AT&T U-verse: Deployment Finally Under Way
The biggest story at last year's TelcoTV show was the "impending launch" of AT&T's ambitious U-verse service. Announced with great fanfare at CES nearly two years ago (see CES 2005--SBC Goes Big for Video ( www.broadbandhomecentral.com/report/backissues/Report0501_6.html )), AT&T had set expectations that U-verse would launch in 2005. Instead, at the time of the TelcoTV show in early November 2006, U-verse was still only in a trial in San Antonio, with high definition service not yet available and users receiving the service for free. The trade press has been full of speculation on why the launch has been delayed for so long, with many fingers pointing to the Microsoft IPTV middleware system at the heart of U-verse.
At the show, we met with Jeff Weber, AT&T's VP, Product & Strategy. He told us that by the end of 2006, equipment will be in place to serve 2.5 million homes, up from 1.3 million at the end of the third quarter; by the end of 2008, this will expand to 19 million homes. Each home will be able to receive 25 Mbps, enough for two high-definition and two standard-definition video channels; homes closer to the central office will be able to get 40 to 60 Mbps, but AT&T is limiting the maximum to 25 Mbps for now.
Weber said AT&T was preparing to "turn up service" very quickly. U-verse will launch in San Antonio and Houston during November, and then expand to reach 15 markets by the end of 2006.
We asked Weber whether the initial applications would include the "cross service" applications he had talked about in his speech at TelcoTV last year ( www.broadbandhomecentral.com/report/backissues/Report0509_5.html#link5b ). He said AT&T's initial focus was on getting the initial service rolled out, but that the IMS systems were in place to support the cross service applications.
He also mentioned that Microsoft would soon publish the APIs for the IPTV platform in the form of an IPTV software development kit based on Microsoft's .NET framework. Since Microsoft's IPTV solution is being deployed by first-tier telcos worldwide, Weber thinks many developers will be attracted to the opportunity to develop innovative IPTV applications in a familiar client-server environment.
In a later panel session, Weber said AT&T will "create an application environment that allows easy fast turnaround of applications. We're open to third parties so the market can work the way the market ought to work. I don't know and I don't care what people will do three years from now. If we do a good job--make sure the capabilities are in place across applications and devices--we will be in great shape. We will let the market talk to us and it won't matter."
Why So Late?
Weber was not willing to discuss why U-verse was launching so much later than had been projected. Our guess from talking with many people at the show is that both Microsoft and AT&T had been rather optimistic about how long it would take to get this complex hardware and software solution working.
The first SoC-based set top boxes, a central element in the solution, were only delivered in early November, much later than expected; we hear that getting all the software working reliably has been challenging. This is not a unique AT&T or Microsoft problem--all telcos that based their deployments on the new SoC set top boxes have been equally delayed.
Alcatel's IPTV Integration Labs: U-verse In the Making
Derek Kuhn of Alcatel and some members of Alcatel's staff offered us and some of the other analysts attending Telco TV the opportunity to visit Alcatel's IPTV Integration Labs in Plano. This is the site where Alcatel assembles and tests an IPTV system identical to that being used by AT&T.
The goal of the integration lab is to put the system through complete testing, not just pairwise, but assembled end-to-end. Alcatel first tests new system elements and their integration in a lab in Kanata, Canada. Once that has been completed, the next step is testing in the Plano integration lab. The concept is that there are "network solution releases" in addition to hardware and software releases. The intent is to find and fix problems in the lab, rather than in the field.
Our first stop on the visit was to see the area where they demonstrate home networking and how AT&T delivers their TV service around the home using HPNA. The major stop on the tour was an enormous area filled with all the components that make up AT&T's IPTV system. These include a huge number of servers, miles of simulated network, set-tops, switches, encoders, video subsystems, test equipment, VDSL loop simulators, etc.
The lab was impressive for both its enormous size and scope of equipment. One less-than-ideal side effect of having all that equipment assembled together was a very high noise level. The air-conditioning required to keep everything at an appropriate temperature made conversation within the labs very difficult.
After the lab tour, we went to a conference room where we were reminded of the many successes Alcatel has had in providing elements of IPTV systems around the world, including ones at Belgacom, Swisscom, Auna, KPN, BT, Telefonica, Alliant and FastWeb. Before our bus ride back to Telco TV we were served drinks and hors d'oevres--but only after we listened to several presentations, including one on regulatory issues. I guess we earned those drinks!
Deployment Under Way Now
True to Weber's claims, the U-verse deployment is indeed under way. Shipment of the Motorola VIP1200 set top boxes started in mid-November, and HD was turned on shortly after. San Antonio and Houston were launched during November, although only in limited service areas. We expect that U-verse will become available in many more markets by the end of the year, as promised, but only in limited areas and only with a few customers.
AT&T may be moving a little too quickly. Users on U-verse blogs such as UverseUsers.com ( www.uverseusers.com ) report that HD is often not being enabled on the promised dates, even though this appears to be just "throwing a switch" by entering an HD order into the system.
Users also report that AT&T is not communicating very effectively with its early users. For example, AT&T is only providing one HD channel now, which means that users cannot watch one HD channel while recording another. This appears to have taken early subscribers by surprise.
Users have also reported problems with video quality in some homes, although this seems to be getting better over time. AT&T has preferred not to deploy FEC, but could change its mind if the problems persist.
Interview With a U-verse User
As part of trying to understand how well the U-verse rollout is proceeding and how customers view the service, we interviewed Alan Weinkrantz. Alan is a PR professional who lives in San Antonio where AT&T is trialing their service. He decided to be a U-verse trial user and share his experiences in his blog ( www.3screens.net ). Although Alan has a technical background which he uses in representing his technology company clients, he told us his goal is to write from the perspective of the end-user ("a combination Martha Stewart and Oprah Winfrey").
Because of this perspective, Alan answered any of our questions that verged on the technical by saying something like "Martha Stewart doesn't need to be able to tell you the technical specs of the oven in which she bakes her cakes". Although Alan says he is not acting an advocate for the U-verse service, he sounded much more forgiving than many of the users who post on the U-verse blog. He pointed out that he is not paying for the service and knows that it has been a trial, so is forgiving of some problems. In contrast with the views expressed by many users in the U-verse blog, Alan said that AT&T has kept him informed of the status of various things and has conditioned his expectations in a realistic way. However, people are much more likely to take the time to write and complain about things than to praise something when it is going right.
On the topic of the delays in HD and in the question of when a second HD channel will be available to users' homes, Alan wrote in his blog: "I was advised that for now, with two HD sets, we'd only be able to get one HD stream at a time and that it would be some time next year, when we would get two or more streams. I was really OK with it, for while I am personally very much into this whole IPTV experiment, as a family, we're not really glued to to the TV all the time."
When we asked Alan about the HD delays, he told us about "the ice cream letter". He said AT&T sent a letter to U-verse trial users telling them that it would be a while before they could get a second stream of HD, and included a coupon for two free gallons of ice cream. Earlier in the trial they had sent out free ice cream bowls so now the users had something to put in them. Trading HD for ice cream is certainly a unique approach to marketing!
Verizon's approach to TV is very different from AT&T's. While AT&T chose to run fiber only to the neighborhood, Verizon is running fiber all the way to the home. While AT&T is using IPTV, Verizon--at least initially--is using mature television technology similar to that long employed by cable providers. With practically unlimited bandwidth over fiber, Verizon can deliver any number of simultaneous HDTV channels and as much high-speed data as is needed to meet competitive offerings.
In a keynote address at TelcoTV, Brian Whitton, Verizon's Executive Director - Access Network Design & Integration, reviewed the status of his company's FIOS TV rollout and what he believes Verizon's users expect. By year end, Verizon will have passed 6 million households with fiber. They are increasing by 3 million homes passed per year and by 2010 will have passed 18 million homes.
To support video services, they have built 2 "super-headends" for broadcast content. They have 9 video hubs--one per region--and will build more in 2007. They now have 292 "video serving offices" and 1.8 million households which are "open" for selling video. Their delivery mechanism allows them to send linear broadcast content (analog and digital) on one wavelength and interactive services on another. Their current offering has 450 linear broadcast channels--of which 25 are HD--and 3000 video on demand titles.
Verizon'a goals include category leadership in the areas of VOD, HD, multicultural programming and multi-room DVR. Their 2007 focus will be to add more multimedia and personalization. Whitton says that Verizon will be a "top ten" video provider by the end of 2006. They aim to have 20-25% FIOS TV penetration in areas where they have been available for 5 or more years.
Whitton said Verizon's vision of the home is one in which all client devices can become "sources or sinks" of information. "There is no reason to restrict a device to its original function"--for example using a TV only for seeing video. Their goal is to provide any service on any screen. In terms of "what's next for IPTV", Verizon sees the imperative for advertisers, content providers and network service providers is to move up the value chain.
Whitton spent some time talking about how the evolution of the home and home network are critical to delivering on this vision. In their view, the lowest common denominator of a service is the home router. For that reason they provide a purpose-built home router which can also do remote management. Verizon has chosen to use MoCA for home networking because it provides the necessary bandwidth and QoS.
Whitton talked about the role of IPTV. He said there was no value in "taking TV and wrapping it in IP". Rather, he said "the real value is up the value chain. IPTV has the opportunity to completely disrupt all business models--content providers, advertisers, network service providers, and customers. The full range of opportunity lies in networking all human beings to create video content like YouTube."
Verizon has been trying hard to convince Wall Street that spending the extra capital to go with fiber all the way to the home will be the winning strategy. Whitton ended his talk by saying "in the new battleground, bandwidth is the key enabler."
Packaged Solutions for Smaller Telcos
Many of the larger US telcos have rolled out IPTV or are getting ready to do so. With small specialists providing each hardware and software component, each telco has had to select the individual components and then select another company to act as the system integrator.
At TelcoTV we saw many smaller telcos getting ready to roll out IPTV. Most of these don't have the resources to evaluate all these hardware and software options.
In this "mix and match" environment, middleware vendors often find themselves working with very complex combinatorics. Matt Cuson of Minerva Networks told us that with 100 customers they were working with six different set top boxes, three conditional access systems, and five VOD servers; while they didn't support all 90 possible combinations, they did have a lot of them. Integrating and supporting all these combinations of hardware and software requires a lot of resources from a small company like Minerva.
The middleware vendors have found that they need to limit the possibilities to make it possible to deploy IPTV more broadly. Matt told us that Minerva was now offering prepackaged solutions from a much smaller menu of options. Minerva's middleware is also included in the Nortel IPTV Ecosystem ( www.nortel.com/corporate/news/newsreleases/2005d/collateral/iptv_ecosystem_backgrounder.pdf ), which also includes specific solutions for VOD servers, content security, video encoders, web browser, set top boxes and more.
SES Americom has created another packaged solution. Americom is one of the largest providers of satellite video services. Its IP-PRIME service delivers the complete IPTV program lineup over satellite to a "micro headend" at the telco, avoiding the cost and complexity of building and maintaining a complete headend.
The IP-PRIME ecosystem includes middleware from Siemes/Myrio, set top boxes from Scientific-Atlanta and Amino, content protection from NDS, etc. Americom operates an IPTV broadcast center and satellite uplink in Vernon Valley, NJ.
Americom sells IP-PRIME directly to the larger telcos; BellSouth is using IP-PRIME in its IPTV trials. For smaller telcos, Americom has signed an agreement with the National Rural Telecommunications Cooperative (NRTC) and the National Telecommunications Cooperative Association (NTCA). Both will offer IP-PRIME to their members.
NRTC includes 1,300 rural utilities and telecommunications companies. At the TelcoTV show we met with Don Mathison, NRTC's executive director, Marketing and Programming, and Joe King, director of systems services. Don and Joe told us that NRTC is negotiating program distribution rights on behalf of its member companies and is running service trials with several rural telephone cooperatives. These are currently beta trials in "friendly" customer homes. There are 200 video channels, being expanded soon to 300; high definition is part of the plan and will be available within a year.
At the VON conference a few months ago, we heard a talk on IPTV by Trevor Bonstetter, CEO of West Kentucky Rural Telephone Cooperative which serves about 17,000 customers in the western part of Kentucky. His cooperative is one of the beta sites for NRTC and IP-PRIME, and we understood how well this solution would make sense for a small telco like his. We're hoping to visit western Kentucky some time soon to report on this service.
TV Over the Internet--The End of the "Walled Garden"?
Although TelcoTV's main focus was on television delivered over managed networks, TV delivered over a user's broadband connection was also addressed. This was confronted most directly in a panel session titled "Broadband TV: Bypass Villain or Golden Opportunity" which included Mike Hudack, CEO of blip.tv; Josh Goldman, CEO of Akimbo; and Trey Gaskins, COO of Dave.TV.
In Goldman's view, services like Akimbo are a great way to provide "long-tail" content and do not need to be a threat to service providers. He pointed to the offering of Akimbo's content by AT&T HomeZone as a place where the relationship with a service provider is complementary rather than adverserial. Goldman also observed that the "traditional" model--in which managed video services go to the TV and Internet content goes to the PC--is weakening. In his view, 2007 will be the year that broadband video delivered to the TV reaches critical mass.
Mike Hudack's perspective on broadband video comes from his experiences in working with both established content (in his case from the National Hockey League) and user-generated materials. He believes that the service provider's need to control and limit content for traditional TV programming comes from the relatively scarce bandwidth available to deliver video content. Service providers go after properties that attract large audiences; thus you get TV's "walled garden". Now that video can be delivered over any broadband service, the available bandwidth for delivery expands, so that content which attracts only small audiences can still be delivered. The new model will be "any content to any device."
The panel agreed that we are at the point where the historical links of "professional content on the TV" and "any content on the PC" are being broken. In their view, service operators have the opportunity to deliver the choice of content to the user. "If they don't, consumers will go around them the way they did with PVRs".
Goldman expressed the view that the service providers have an imperative to get this right: "There is nothing more motivating than seeing your core business rapidly going away."
Addressing this pressing issue is what the Telco TV show is all about.
( www.ineoquest.com ) ( www.digitalfountain.com ) ( www.att.com ) ( www.alcatel.com ) ( www.belgacom.com ) ( www.swisscom.com ) ( www.auna.com ) ( www.kpn.com ) ( www.bt.com ) ( www.telefonica.com ) ( www.alliant.com ) ( www.fastweb.com ) ( www.verizon.com ) ( www.minervanetworks.com ) ( www.ses-americom.com ) ( www.ip-prime.tv ) ( www.nrtc.coop ) ( www.ntca.org ) ( www.wktelecom.coop ) ( www.blip.tv ) ( www.akimbo.com ) ( www.dave.tv )