Development of a video transmission concept in the field of teleoperated driving

Development of a Video Transmission Concept in the Field of Teleoperated Driving Report Nr. 1322/18 Editor: Roger Claverol | 2576914 Advisor: Patrick Pintscher M. Sc., Timm Ruppert M. Sc. Roger Claverol matriculation number: 2576914 course: Electrical Engineering and Information Technology Thesis Nr. 1322/18 topic: Development of a Video Transmission Concept in the Field of Teleoperated Driving submitted: 31st August 2018 Technische Universität Darmstadt Fachgebiet Fahrzeugtechnik Prof. Dr. rer. nat. Hermann Winner Otto-Berndt-Straße 2 64287 Darmstadt Summary I Summary This project initially consist on the study and the performance of the video transmission part of a project of teleoperated driving of a tram in the city of Darmstadt. The tram where will be implemented is in a part of Darmstadt that will have a 5G infrastructures. The project includes a study of 5G, also 4G and Wi-Fi transmission, including data rates and latency, to transmit in real time video from the tram to a central station. Initially the idea is to make a test-rig that would be as similar as the prototype that could be implemented in the tram. The prototype would be some cameras in front of the tram connected with cables to a computer from where the data will be processed and send it through 4G/Wi-Fi infrastructures to a static station. The computer will compress the data, applying some codecs, to use less data size for the transmission The initial situation of the test-rig is to connect a camera to a computer, process the data that receives and send it to another computer using Ethernet cable. The problem is that the Ethernet cable must be full duplex and the computers must have a specific configuration that from the prototype would not be helpful. In addition, the computer it is used to make the tests is not compatible with the computer. Finally the solution for those problems are solved transmitting with Wi-Fi, instead of the cable, and for the transmitting video is used the web cam of the computer/ and a video downloaded from internet. For the compression part there is a study of different codecs, and the programming of a code to apply this codecs. Table of content II Table of content Summary ............................................................................................................................................... I Table of content .................................................................................................................................. II Symbol and index directory ............................................................................................................... III List of abbreviation ............................................................................................................................ IV List of figures ...................................................................................................................................... V List of tables ....................................................................................................................................... VI 1 Introduction .................................................................................................................................. 1 1.1 Motivation ............................................................................................................................. 1 1.2 Concretization of the tasks .................................................................................................... 1 1.3 Methodology ......................................................................................................................... 2 2 State of the art .............................................................................................................................. 3 2.1 Concept of Teleoperated driving........................................................................................... 3 2.2 Other existing projects .......................................................................................................... 3 3 Requirements ............................................................................................................................... 4 4 Codecs .......................................................................................................................................... 5 4.1 MPEG4.................................................................................................................................. 5 4.2 H264/AVC ............................................................................................................................ 5 4.3 HEVC/H265 .......................................................................................................................... 6 4.4 Comparison of the codecs ..................................................................................................... 7 5 Transmission ................................................................................................................................ 8 5.1 4G .......................................................................................................................................... 8 5.2 5G .......................................................................................................................................... 8 5.3 Wi-Fi ..................................................................................................................................... 9 5.4 Comparison of the Transmission standards .......................................................................... 9 6 Test - Rig .................................................................................................................................... 10 6.1 Setup.................................................................................................................................... 10 6.1.1 Methodology .................................................................................................................. 10 6.2 Evaluation of latency .......................................................................................................... 11 6.3 Evaluation of data rate ........................................................................................................ 16 7 Conclusion and outlook ............................................................................................................. 17 Annex ................................................................................................................................................. 18 Bibliography.......................................................................................................................................23 Symbol and index directory III Symbol and index directory Latin letters: symbol unit description None Greek letters: symbol unit description     Indexes: symbol description None List of abbreviation IV List of abbreviation FZD Fachgebiet Fahrzeugtechnik MAAS Machbarkeitsstudie zur Automatisiening und Assistenzsystemen der Straβenbahn 5G 5th Generation 4G 4th Generation LTE Long Term Evolution Wi-Fi Wireless Fidelity MPEG Moving Picture Experts Group AVC Advanced Video Coding HEVC High Efficiency Video Coding TU Technische Universität Mbit/s Megabit per second fps frames per second px pixels Tx Transmitter Rx Receiver RAM Random Acces Memory List of figures V List of figures Figure 4.1 H264 blocks ........................................................................................................................ 6 Figure 4.2 Macroblocks in H.264 and H.265 ....................................................................................... 7 Figure 6.1 Test-rig.............................................................................................................................. 11 Figure 6.2 lbx264 with 3 frames each packet .................................................................................... 12 Figure 6.3 lbx264 with 5 frames each packet .................................................................................... 12 Figure 6.4 VP8 with 5 frames each packet ........................................................................................ 13 Figure 6.5 VP8 with 3 frames each packet ........................................................................................ 13 Figure 6.6 VP9 with 5 frames each packet ........................................................................................ 14 Figure 6.7 VP9 with 3 frames each packet ........................................................................................ 14 Figure 6.8 Latency in Transmission ................................................................................................... 15 Figure 6.9 Data rate in transmission .................................................................................................. 16 List of tables VI List of tables Table 4.1 Advantages and disadvanteges H264 ................................................................................... 6 Table 4.2 H264 vs H265 ...................................................................................................................... 7 Table 6.1 Information of computers for test-rig ................................................................................ 10 4. Codecs 7 4.4 Comparison of the codecs MPEG-4 part 2 vs MPEG-4 part 10 Part 10 is more efficient to encode and a better quality, but to be more efficient to encode it leads to a higher power consumption. For this last problem of consuming more CPU, nowadays there are good computers that can reduce this problem. Table 4.2 H264 vs H2657 H.264 H.265 Macroblocks Fixed macroblocks of 16x16 Can use different macroblocks from 8x8 to 64x64 Bit-rate Less bit-rate Resolution Limited to 4k Until 8k Frames per second Limited to 60 fps Until 300 fps Data size 50% less than MPEG-2 50% less than h264 Others More complex codec and CPU consume When the codec get each frame, divided the image in macroblocks, in the case of H264 the macroblocks are of 16x16 pixels as is shown in the left side of Figure 4.28. However, as is shown on the right side of Figure 4.2 in the case of H265 get different sizes of macroblocks because it compares the pixels and the pixels around, and if there is a great amount of similar, for example the same color, then the codec can get a higher macroblock until 64x64. Moreover, on the other case when there are many different pixels then the codec get a small macroblocks as 8x8 pixels for this part of the image. 7 H.264 & H.265/HEVC COMPARISON. 8 HEVC. What's next in video compression. Figure 4.2 Macroblocks in H.264 and H.265 5. Transmission 8 5 Transmission When we talk about transmission and wireless communication, we talk about transmission of electrical signals that travel through the air as radio waves. This is a topic connected to the mobile communication since the finals of the 70’s with the appearance of the first generation of mobile communication. Since then until now the following generations have surpassed and by far the previous ones. The first change was with the second generation that changed to a transmission of digital voice, the possibility of sending SMS and the roaming that let an owner of a mobile phone to move around the world. The second change came with the third generation. It appeared the possibility to connect to internet, a great novelty for the mobile communication sector. After the third generation appeared the actual one 4G, and nowadays is in process 5G. Those last two generations are the important for this project.9 The transmission is the most important part on this projects because, it affects on the data rate and the latency. Is the part where it could not have problem because low data rate means that the video has a break, no streaming and nothing to show in the static station. For the transmission part there are some wireless networks that could be the most appropriate. For this case there is WLAN that has a range from 10 meters to some hundreds of meters as Wi-Fi, or WWAN that use mobile communication and with the help of some repeaters is the best option to transmit data nowadays.10 Nowadays from the mobile communication the one that is being used the most is 4G, and also a new generation that exists for a few years ago is 5G but it is still in process of implementing. 5.1 4G Fourth generation is the actual mobile generation standard. It is totally based in IP, which means that use all the protocols to send and receive data in packets11. Can offer higher download speed than his predecessor, 3G, It can offer a download speed to 1 Gbps in an static position and at maximum 100 Mbps in movement. The novelty respect the previous generation is that the possibility of the streaming videos. This technology has a bandwidth between 3 kHz and 6 GHz and depends on the continent of the world in operates in different frequency bands.12 5.2 5G The new generation that will offer better conditions than his predecessor 4G. This generation has two layers; a macrolayer that works in the microwave band and provides user plan traffic, and a micro layer that works in the mm-wave band and carries user plan traffic. This two layers are reflected in 9 Evolution of mobile communication (2018). 10 Wireless Networks. 11 NATHAN CHANDLER: How 4G Works. 12 Tabla de frecuencias 4G. 5. Transmission 9 the bandwidth that has increase respect to the previous generation, 4G, to a range between 3 kHz and 300 GHz. The microwaves has a problem that they couldn´t pass through the walls, and the trees and the clouds absorb them. The solution that is found is the implementation of microcell that they can repeated the signal. This solution entails the construction of new infrastructures that could make a new generation that will arrive everywhere.13 For the term on the bit rate and the latency, it is said that the bit rate is going to increase to 1Gbps and could go faster until 10 Gbps.14 This increase on the bit rate will affect positively on the latency because the file will be transmitted faster and the latency will be lower. A very important aspect for the live videos. 5.3 Wi-Fi Is a technology that allows the connection between electronic devices normally in places with an access point that all the devices can be connected. It is used for Local area networks as companies or particular houses. It use to operates in the frequency band of 2,4 GHz15 offering a download data rata until 300 Mbps. 5.4 Comparison of the Transmission standards The most significant difference between the transmission standards is that for the new mobile generation it is an extension of the bandwidth, because it is predicted that in a future there will be more and more appliance connected to internet and the fourth generation will be short to offer service to all. This will lead to de massaging of connected objects. Also a very significant difference is that the data rates of downloading of 5G will be 100 times higher than 4G, and this implies the latency will be very low.16 13 What Is 5G and How Will It Make My Life Better? 14 Gemalto: Introducing 5G networks. 15 WLAN Frequency Bands & Channels. 16 4G vs 5G (2017). 6. Test - Rig 10 6 Test - Rig After all the study of the requirements then it must put in practice all the collected information and as there is not the final prototype that will go in the tram, it is needed a previous test-rig to see the possible problems that could be in a general situation, and solve them before the prototype in the tram 6.1 Setup For the setup, it is necessary two computers, one to transmit the video and the other one to receive and plays the video. In the Table 6.1, there are the characteristics of the computers used. Both computer are similar but the RAM is better in the receiver, that could affect on the latency of the compression of the data. Table 6.1 Information of computers for test-rig Computer 1 (Tx) Computer 2 (Rx) - Windows 10 Education - Intel® Core™ i5-2450M CPU @ 2.50 GHz 2.50 GHz - RAM 4,00 GB - Windows 7 professional - Intel® Core™ i5-2450M CPU @ 2.50 GHz 2.50 GHz - RAM 8,00 GB It is also necessary a camera. The first option was to connect a camera that has the same resolution and the same number of fps as the possible cameras that will be installed in the tram. However, for problems of compatibility with computer 1 and the camera finally it is decided to use a video downloaded on internet or the web cam of the computer 1. Once the test-rig is with all the complements, in this case just both computers, then it is investigate about how to make all the process capturing the image, compressing, transmitting and receiving and then reproduce the video. For the process, using both computers it has been used a PyCharm program that is used to modify and run python scripts and it has been also used a program named Ffmpeg that is used for applying the codecs. 6.1.1 Methodology The all process is divided in the two computers and follow the next processes. The computer that it is transmitting has four processes to perform: Capture the video, compress, transmit and finally after the transmission, delete the file from the computer. This last process is important because in the final prototype, if the files are not erased, the computer will needed a huge capacity and consequently a bigger computer to store all the data. The computer that is receiving has three processes to perform: 6. Test - Rig 11 Receive data, decompress, plays the video and delete the file, very important process as in the Transmitter computer. On the Computer 1 the caption of the video is frame by frame, and then a video file is created attaching some frames. After the creation of the file, it is compressed applying some codecs, send it to the computer 2, delete it and then another time repeating the process. On the computer 2 is the same all the process are performed one after the other and after de deleting process again to the receiving process, that have to wait until the computer 2 receives data (Figure 2). The methodology was correct, but the problem was that the reproduction of the video was so slow and the latency was high. The solution of this problem was add sockets that makes that the processes work simultaneously. On the computer 1 the capture of the video creates files with the number of frames it is desired. When there is an existing file created, then the process of compressing begins at compress this file is created. This process only can proceed if there is any existing file created that is not already compress it. Then the process of Transmit that depends on the last process. Only when a compressed file exists and is not already send it then can send it. Finally the process of delete that depends on the last process too. All the four processes works at the same time but the significant difference with the first method is that the caption of video do not wait until the delete is done. On the computer 2, as in the computer 1, the four blocks works simultaneously and the receive process do not have to wait until the delete process is done. The receive process is keeping all the data received, before being used for the other processes. 6.2 Evaluation of latency In the evaluation of the latency, first it is decided if it is better to use the web cam of the computer 1 or a downloaded video in raw video. It was decided that was going to be the web cam because, in the process of capturing the video, the latency is lower than using the downloaded video and it is closer to the final prototype. CAPTURE VIDEO COMPRESS DELETE TRANSMIT Computer 1 RECEIVE DECOMPRESS DELETE PLAY VIDEO Computer 2 Figure 6.1 Test-rig 6. Test - Rig 12 One problem on the evaluation is that both computers are connected to internet to the same webpage, but each one has a different processor and then they had different time. The two clocks differ from a few seconds, then to study exactly the how is the latency, it is needed first to synchronize both computers. The solution was to make a TCP connection, that when a packet is send there is an answer form the other computer and on this case, the answer is to send the same packet back. In this process, it is obtained the fictional time that would be in the computer 2 when the packet is received. With this time and the real time of the computer 2 capture it at the received time, it can be determined an offset. During the whole process, there are different processes that can affect considerably in the latency time: - The compression This process is the one that depending on the computer-processing unit will do it faster or slower. The next figures are applying different codecs Both figures Figure 6.3 and Figure 6.2 are performed using lbx264 that is the library of the h264 codec, it can be observed than with more frames the codec works better. 0 50 100 150 200 250 300 350 400 919 29 39 49 59 69 79 89 Latency [ms] Number package Figure 6.3 lbx264 with 5 frames each packet 0 50 100 150 200 250 300 350 919 29 39 49 59 69 79 89 Latency [ms] Number package Figure 6.2 lbx264 with 3 frames each packet 6. Test - Rig 13 Although the processor of the computer always affect in this process, it can be observed than in Figure 6.5 and Figure 6.4 it works better with more frames per packet in the codec of VP8. This codec is a free version of the codec h264. For the next two figure, Figure 6.6 and Figure 6.7, it is applied the VP9 codec that is the sucessor of the VP8, and the same case as VP8, is the free version of the codec h265. Comparing both pictures it 0 100 200 300 400 500 600 700 800 900 919 29 39 49 59 69 79 89 Latency [ms] Number package Figure 6.5 VP8 with 3 frames each packet 0 50 100 150 200 250 300 350 400 919 29 39 49 59 69 79 89 Latency [ms] Number package Figure 6.4 VP8 with 5 frames each packet 6. Test - Rig 14 an be observed than the latency of processing with less frames per packet is always near to the average, without latency peaks. 0 100 200 300 400 500 600 700 919 29 39 49 59 69 79 89 Latency [ms] Number package Figure 6.6 VP9 with 5 frames each packet 0 50 100 150 200 250 300 350 400 450 500 919 29 39 49 59 69 79 89 Latency [ms] Number package Figure 6.7 VP9 with 3 frames each packet 6. Test - Rig 15 - The transmission In this process, it is used a simple packet of 8 bytes to transmit from one computer to the other one and the result is shown on Figure 6.8. As it is shown there is an average of the latency transmission between 51 and 52 miliseconds. There are some peeks that reach up to 63 miliseconds that could be for the small size of the transmission package Figure 6.8 Latency in Transmission Latency (s) Packet number sent 6. Test - Rig 16 6.3 Evaluation of data rate In the evaluation on the data, is more or less the same than the evaluation of the latency because the data rate will be get it from the latency and the size of the sent packet. In the shown Figure 6.9 the axis in vertical is the data rate in Bytes/ second, and to get the data rate in bps, it is only needed to multiply the obtained rate by 8. The horizontal axis is the number of the package sent. There a maximum of data rate of 500 Bytes per second, equally to 4kbps, that is not ver high, and an average of data rate of 410 Bytes per second, equally to 3,28 kbps, also not very high. This low data rate does not show the maximum data rate that could reach the connection, probably for the small size of the data sent. Figure 6.9 Data rate in transmission Data rate (B/s) Packet number sent Annex 23 Bibliography 4G vs 5G (2017) 4G vs 5G; https://www.econectia.com/blog/4g-vs-5g-diferencias, 2017, Access 22/08/2018 A Novel Study on Data Rate by the Video Transmission for Teleoperated Road Vehicles (2013) A Novel Study on Data Rate by the Video Transmission for Teleoperated Road Vehicles; https://mediatum.ub.tum.de/doc/1188692/1188692.pdf, 2013, Access 27/08/2018 An Overview of H.264 Advanced Video Coding An Overview of H.264 Advanced Video Coding; https://www.vcodex.com/an-overview-of-h264-advanced-video-coding/, Access 22/08/2018 Difference between lossy and lossless Difference between lossy and lossless; http://askdifference.blogspot.com/2016/09/difference-between-lossy-and-lossless30.html Ericsson initiates 5G motorway project with cross-industry consortium in Germany (2016) Ericsson initiates 5G motorway project with cross-industry consortium in Germany; https://www.ericsson.com/en/news/2016/11/ericsson-initiates-5g-motorway-project-with-cross-industry-consortium-in-germany, 2016 Evolution of mobile communication (2018) Evolution of mobile communication; https://www.universidadviu.com/evolucion-la-red-comunicacion-movil-del-1g-al-5g/, 2018, Access 29/08/2018 Gemalto: Introducing 5G networks Gemalto: Introducing 5G networks; https://www.gemalto.com/brochures-site/download-site/Documents/tel-5G-networks-QandA.pdf H.264 & H.265/HEVC COMPARISON H.264 & H.265/HEVC COMPARISON; https://www.pamdist.com/pages/faq-what-is-the-difference-in-h-264-h-265-hevc HEVC. What's next in video compression HEVC. What's next in video compression; http://z3technology.com/News/HEVC-What-is-next-in-video-compression/26.html, Access 30/08/2018 NATHAN CHANDLER: How 4G Works NATHAN CHANDLER: How 4G Works; https://electronics.howstuffworks.com/4g2.htm Parts of MPEG-4 Parts of MPEG-4; https://mpeg.chiariglione.org/standards/mpeg-4 Tabla de frecuencias 4G Tabla de frecuencias 4G; http://www.gigahertz.es/tabla-de-frecuencias4g.html Video Compression Standards – Pros & Cons Video Compression Standards – Pros & Cons; https://www.mistralsolutions.com/articles/video-compression-standards-pros-cons/ What Is 5G and How Will It Make My Life Better? What Is 5G and How Will It Make My Life Better?; https://gizmodo.com/what-is-5g-and-how-will-it-make-my-life-better-1760847799 Wireless Networks Wireless Networks; https://www.universidadviu.com/tipos-de-redes-inalambricas-mas-comunes/, Access 29/08/2018 WLAN Frequency Bands & Channels WLAN Frequency Bands & Channels; http://www.cablefree.net/wirelesstechnology/wireless-lan/wlan-frequency-bands-channels/ Bibliography