Studentische Arbeiten am Lehrstuhl für Medientechnik

The multidimensional sensory data is computationally expensive for localization algorithms in autonomous navigation for drones. Research shows that not all sensory data are equivalently important during the entire process of SLAM to perform a reliable output. The attention control scheme is one of the effective ways to filter out the highly valuable sensory data for such a system. The predictive attention model, for instance, can help us to improve the result of the sensor fusion algorithms by concentrating on the most valuable sensory data based on the dynamic of the vehicle motion or the semantic understanding of the environment. The aim of this work is to investigate the state-of-the-art attention control models that can be adapted for the multidimensional sensory data acquisition system and compare them from different modalities. 

- Strong background in Python and C++ programming

- Solid background in robot control theory

- Be familiar with deep learning frameworks (Tensorflow)

- Be familiar with the robot operating system (ROS)

leox.karimi@tum.de

Human Activities of Daily Living are driven by our underlying intentions. For example, the Intention of "making Pasta" spawns a sequence of activities like fetch pasta, boil it, fetch and chop vegetables for the sauce, and clean up after cooking.

We use RGBD cameras and develop Computer Vision / Machine Learning algorithms for recognizing and anticipating human intentions and activities in indoor environments.

Ambition, Motivation, interest and first experience in Computer Vision, ML/DL, Python, C++

 This work can be done in german and english.

Teleoperated driving can be a solution strategy for autonomous driving failures. In order to control the vehicle safely the operator requires low delay video information of the vehicles’ surroundings. This can be achieved by a setup of multiple cameras, which are individually encoded and streamed simultaneously over the same network to the operator. Each camera stream obtains an equal part of the current available transmission rate. To avoid congestion on the network and adapt to the individual available transmission rate, a constant bitrate control algorithm calculates the encoding parameter for each camera stream.

As the image content for each camera can be very different, the residual of the encoded perspectives can have different levels of quality. Especially frames with a higher amount of motion or lots of details require a higher bitrate for encoding compared to other frames. As the bitrate is fix this will result in worse visual quality. With the overall goal of providing equal visual quality for each camera stream, the equal distribution of the available transmission rate seems not well suited.

An overall rate allocation algorithm is required to calculate the individual transmission rate for each camera stream to achieve a minimum variance in visual quality among the different camera views. This could be done by estimating the complexity of the individual camera streams or minimize the overall rate distortion.

Tasks

• Analyze different state of the art joint rate control algorithms
• Implement the most suitable joint rate control algorithm
• Integrate implemented joint rate control algorithm into the existing TELECARLA [4] setup
• Evaluate impact on camera stream quality and transmission rate distribution

References

[1] Yu Wang, Lap-Pui Chau, und Kim-Hui Yap, „Joint Rate Allocation for Multiprogram Video Coding Using FGS“, IEEE Trans. Circuits Syst. Video Technol., Bd. 20, Nr. 6, S. 829–837, June 2010.
[2] H. Fan, L. Ding, H. Jia, und X. Xie, „A Novel Joint Rate Allocation Scheme of Multiple Streams“, IEEE Trans. Circuits Syst. Video Technol., Bd. 29, Nr. 3, S. 854–867, March 2019.
[3] W. Yao, L.-P. Chau, und S. Rahardja, „Joint Rate Allocation for Statistical Multiplexing in Video Broadcast Applications“, IEEE Trans. on Broadcast., Bd. 58, Nr. 3, S. 417–427, Sep. 2012.
[4] TELECARLA: An Open Source Extension of the CARLA Simulator for Teleoperated Driving Research Using Off-the-Shelf Components, Markus Hofbauer, Christopher B. Kuhn, Goran Petrovic, Eckehard Steinbach; IV 2020.

• Good unterstanding of video compression and rate control
• Experience with ROS and C++

This work can be done in German or English

Existing teledriving setups use multiple cameras to cover the vehicle's surrounding environment in order to provide the operator with sufficient information of the current traffic situation. However, the importance of individual camera views varies for different driving tasks. Modeling the importance of individual camera view according to the current traffic situation can be used in several applications for teledriving.

The goal of this project is the development of a camera view prioritization scheme, optimal for the current traffic situation. This should be based on the vehicle's odometry and the Region of Interests (ROIs), predicted from the individual camera views. In autonomous driving, predicting ROIs allows the car to focus on areas critical to driving. A trained multi view ROI prediction model is available [1].

Tasks

• Develop basic odometry-based prioritization model
• Extend the basic model with ROI information
• Evaluate prioritization model

References

[1] Multi-View Region of Interest Prediction for Autonomous Driving Using Semi-Supervised Labeling, Markus Hofbauer, Christopher B. Kuhn, Jiaming Meng, Goran Petrovic, Eckehard Steinbach; ITSC 2020

• Experience with ROS and Python
• Good knowledge of tensorflow

This work can be done in German or English

LIDAR is one important sensor type for autonomous vehicles' perception. Human perception is mainly based on RGB data, in case of teleoperation captured by RGB cameras and transmitted to the remote operator through a communication network. As mobile networks have a variable and limited transmission rate, sending RGB video data is not always possible. Transmitting processed, high level data of the vehicle's Environment Model (bounding boxes, lane boundaries, traffic lights, etc.) results in a very low bitrate that is required for the transmission, but can be erroneously as the data are already processed. Raw or compressed LIDAR data provide the operator with a good 3D scene representation. However, perception of objects is difficult for the human operator. Using available RGB information and adding their texture in the 3D point cloud representation could improve the operator situation awareness

The objective of this project is to extend an existing LIDAR based driving and simulation setup in the Carla Simulator[2] by adding RGB texture information into the 3D scene representation.

Tasks

• Extend existing point cloud based scene representation with RGB texture information (this could be done using CARLA's RGB and depth camera sensors)
• Evaluate drivers performance for different operator representations
• Compare the results in terms of required transmission resources

References

[1] https://en.wikipedia.org/wiki/Lidar
[2] http://carla.org

• Experience Python and Linux
• Basic knowledge in ROS and C++
• General understanding of Point Clouds

In this project of the aviation research programme V (LuFo V) of the Federal Ministry for Economic Affairs and Energy the goal is to develop an autonomous drone for airplane inspection inside a hangar. The drone is expected to be equipped with a variety of sensors, such as LiDAR, stereo cameras, IMUs, compass and optionally active RGB-D sensors. The challenge in this environment are the large metal structures from the airplane, but also from the hangar itself, having a great influence on GPS signals and compass. In this GPS-denied environment the drone will collect data from the sensors and send them to a ground station, where SLAM algorithms map the environment and localize the drone precisely. The drone will follow preset inspection points at which it is capturing inspection images from the surface of the airplane. The images are then sent to the ground station, which performs machine learning techniques using IBM Watson in order to retrieve a damage classification result. The results are collected and an overall report is generated, documenting the current condition of the airplane.

The LMT will develop the software for module communication (ROS-based), sensor data acquisition and encoding, as well as data processing in the ground station, such as SLAM and drone control. In order to test the algorithms two test platforms are used at the LMT. Our small platform is based on "DJI Flame Wheel 550" for autonomous control and stability tests and the bigger drone is based on "DJI Spreading Wings S1000+" to mount all available sensors for data acquisition tests. During the development phase this data is then analyzed offline to adjust parameters and algorithms. This leads to an improvement of localization precision and better real-time control.

More information on the project can be found on the project's website:
https://www.ei.tum.de/en/lmt/research/bmwi-ki-inspektionsdrohne/


Tasks:
Working tasks can vary from week to week. Mainly we are looking for a working student to improve our drone simulation in Gazebo. So students with knowledge in Gazebo simulation will be preferred.

Interested students should send their current grade sheet and CV to the contact below.

The student is required to have excellent knowledge in C++/Python and experience with the ROS framework. We are looking for an employment of a student assistant for a time frame of +6 months with at least 10 hours per week working time.

This work can be done in German or English

For safe remote control of a vehicle the operator needs to be fully aware of the current traffic situation. The operator gets its information for camera data that are streamed from the vehicle over a constrained network to the operator workplace. A low transmission rate will result in a low visual quality which might be an issue for the operator in controlling the vehicle. But not all parts in the image are important for the operator. [1] suggest to apply a static mask on the camera data which would reduce the overall image size and result in a better image quality for the important regions.

The objective of this this thesis is to implement this mask in a block based way as an optimal input for a video encoder. Further there should be multiple masks which are applied according to the vehicle's lateral movement. The adaptive ROI masks implementation should be integrated into the existing teleoperation setup based on TELECARLA [2].

Tasks

• Implement adaptive block-based ROI masking as C++ ROS Node
• Integrate in to TELECARLA setup
• Evaluate driving performance and image qualityof ROI and non-ROI driving for an equal target bitrate

References

[1] Furman, Vadim, Andrew Hughes Chatham, Abhijit Ogale, und Dmitri Dolgov. Image and video compression for remote vehicle assistance. United States US9767369B2, filed 3. Juni 2016, und issued 19. September 2017.

[2] TELECARLA: An Open Source Extension of the CARLA Simulator for Teleoperated Driving Research Using Off-the-Shelf Components, Markus Hofbauer, Christopher B. Kuhn, Goran Petrovic, Eckehard Steinbach; IV 2020.

• Experience with ROS and C++
• Knowledge of Linux and Python
• Basic understanding of video compression

This work can be done in German or English

LIDAR is one important sensor type for autonomous vehicles' perception. Human perception is mainly based on RGB data, in case of teleoperation captured by RGB cameras and transmitted to the remote operator through a communication network. In some situation a 3D representation of the scene might be helpful for the operator which could be achieved using LIDAR data. To avoid hight transmission rate LIDAR data need to be compressed. Existing methods for point cloud compression [1, 2, 3] don't have their focus on automotive LIDAR data.

The objective of this project is to setup existing point cloud compression implementations and compare them focusing on automotive point clouds.

Tasks

• Setup available point cloud compression implementations
  * ROS PCL [1]
  * MPEG L-PCC [2] (available at MPEG Repo)
  * MPEG G-PCC [2] (available at MPEG Repo)
  * MPEG Anchor Implementation [3]
• Evaluate implementations in terms of
  * Encoding time/complexity
  * Compression rate
  * Compression quality

• Experience with ROS and Point Clouds
• Basic knowledge of C++ and Linux