NAB 2009 was fascinating with respect to how many customers were looking for solutions and insisting that manufacturers work together to provide solutions, as opposed to just supplying products. File-based workflows were the talk of the town (well, at least the talk of NAB) and there certainly continues to be significant discussion and implementation of moving digital files. Last time, we started this series with volume one of the most frequently asked questions on digital distribution and delivery. Herewith, we follow that with the cleverly titled: Volume Two.
I am going to be shooting in the field in New York and I will need to upload content to my producer in London from my hotel room. What do I need to know?
First, are you on an expense account? If so, you may want to consider raiding the mini-bar for the booze because you have a challenge ahead of you. Plus the value of the pound to the dollar is still pretty good. Okay, perhaps that was not an appropriate answer…
First, you will be using shared bandwidth. Whatever connectivity that hotel offers will be shared by every other guest who has access and is using that network. Second, you will have to use a transport protocol that may simply not allow you to maximise whatever connectivity you have. Third, the hotel network may not allow you to use an accelerated transport protocol. So, let’s consider the following: let’s say you shot DV25, edited it on your laptop, and now need to send the five minute finished story. You only have 894 megabytes to send. Now the problems start.
The line that you are using is being shared, so you want to be able to use a transport method that maximises that line. If you use FTP, we learned in Volume 1 that you will only be able to achieve about 60 kilobits per second on average against, say, a shared 1 megabit per second network. Now, if you use WAN acceleration, even though the link is relatively small at 1 megabit, you will be able to put more packets on the network than other users. Practically speaking, this will probably allow you to achieve upwards of a 15x improvement on speed, up to 900 kilobits per second.
On the other hand, that leads us to the next question…
What if I can’t use WAN acceleration? What if the hotel won’t allow it?
This is entirely possible. Many companies will simply not allow UDP-based WAN acceleration on their networks. They simply don’t want to give that much power (read: bandwidth) to people who have access to the network. Again, the most important thing is that the file gets through to its destination. The transport protocol that you end up using must have some form of Checkpointing and auto-resumption capability. If, for example, you are half-way through that 894 megabyte upload and you lose your connection, you will be faced with having to start the upload from the beginning unless the protocol you’re using can resume from the point of interruption. But, what is really likely to happen once your file starts to leave the hotel room? Well, that leads us to the next question…
What other networks, switches, and routers will my file be going over?
Unfortunately, unless you have a guaranteed quality of service and sustained bandwidth as part of your network provider contract, or if you are using the open Internet it is quite possible that you may encounter many hops between switches and routers as you try to send files. The net result of this is that your ability to send files may be compromised in terms of the speed with which you can send them. For example, I personally know of a company that has an open Internet connection that is used to send files back and forth between Los Angeles, CA and Santa Monica, CA. After using a program such as Traceroute, they found that there were 21 hops involved in just linking up these two locations!
As a result of this, it is important to note two things. First, whatever system you employ to send and receive files, you should be aware of whether the system provides for automatic transport protocol failover. What this means is that there should be an automatic mechanism in place such that files can be sent at the fastest rate possible using UDP (user datagram profile) methods. However, if UDP is not allowed, then the transfer should automatically continue using TCP and if that fails, then by using HTTP methods.
Most people do not realize that ISPs (Internet Service Providers) will taper levels of service when extended periods of time at high transfer rates are being utilised. One of the most recent examples of occurred in 2008 and culminated in the FCC’s ruling in August 2008 that Comcast’s throttling of BitTorrent traffic by its subscribers was illegal.
Does File Compression have any effect when sending files?
The answer is that it depends on the type of file. For example, if you have a file that does not already consist of compressed media, using file compression should have some positive results. Huffman coding is one method that can be used to reduce repeating patterns within the dataset that you need to send. You can see the results of this any time you zip a file before you send it via your email account. Data compression typically works well for files such as word processing documents, spreadsheets, and certain types of image files. Documents and spreadsheets both have high instances of repeating information in the form of letters and numbers. Some images have black and white pixels that can be represented as gray, thus reducing the information contained in the file but not affecting the quality of the file.
Compression, in a general sense, comes in two forms: lossy and lossless. The names clearly indicate the effects of their usage. Lossy compression removes data from the file and that data cannot be retrieved. Depending upon the nature and amount of data that has been lost, the effects may or may not be visible. Lossless compression reduces the data that represents the file but does so in such a way that the data is retrievable when the file is uncompressed.
However, for large media files, the amount of repeating information is typically quite low and the amount of time required to analyze and process files is typically high. For example, an uncompressed SDI video frame carries a payload of 270 Mbits (megabits) per second. Many digital nonlinear editing systems will compress (in real-time) this content so that the resulting files are approximately half that payload (to 135 Mbits/sec). The resulting images can be perfectly acceptable, but the data that has been lost can never be retrieved. If we attempt to introduce lossless compression on that same one second of 270 mbit/sec content, we will most likely find that the actual compression saving is less than 1% and, in some cases the resulting file will be even greater in file size than the original. The main reason for this is that the amount of repeating information in a moving image tends to be quite low.
For file transmission systems, lossy compression is not acceptable. The file that is sent and received must have the exact amount of data and, therefore, introducing lossy compression schemes is not possible.
I already have an asset management system that I use for automation. Why do I need another workflow management system?
As asset and content management systems find greater use in the broadcast, media and entertainment industry, there has been a quite a lot of discussion regarding whether or not the automation aspects of asset management systems are different from the automation systems of other products. Where is the dividing line between the automated steps that can be outlined in an asset management system and those that are undertaken by file management or distribution management systems?
The separation that typically exists is that there are actually two categories of automation. First, there is business process automation and, second, there is media-centric process automation. Rarely are the two actually found within one system. The former, business process automation, seeks to provide an overall view of all related processes, ranging from order entry through fulfillment and finally to billing. Even when business process automation systems and asset management systems store and process the movement of files these have almost always been small size files, still images, or short video clips. Rarely are these systems optimized for the handling and movement of large, complex media objects as found in moving video. Further, business process and asset management systems do not typically have many of the media-centric transformation capabilities such as encoding, transcoding, metadata conversion, bandwidth manipulation controls and so forth.
Only after a further examination of your overall requirements will you be able to determine if you, indeed, need both business process and media-centric automation addressed in your implementations.