Studentische Arbeiten am Lehrstuhl für Medientechnik

In this project the task is to fit a parametric hand mesh model and a set of rigid objects to a sequence of multi-view RGB-D cameras. Existing models for hand key-point detection and 6DoF pose estimation for rigid objects models have significantly evolved in recent years. Our goal is to utilize such models to estimate the hand and object poses.

Related Work

1. https://dex-ycb.github.io/
2. https://www.tugraz.at/institute/icg/research/team-lepetit/research-projects/hand-object-3d-pose-annotation/
3. https://github.com/hassony2/obman
4. https://github.com/ylabbe/cosypose

• Knowledge in computer vision.
• Experience about segmentation models (i.e. Detectron2)
• Experience with deep learning frameworks PyTorch or TensorFlow(2.x).
• Experience with Pytorch3D is a plus.

marsil.zakour@tum.de

Neural rendering refers to the set of generative deep learning methods that enables the extraction and manipulation of scene properties such as semantic information, geometry and illumination [1]. The field being relatively new, most of the methods revolve around the idea of representing the scene properties implicitly by neural networks. Recent works utilize differential rendering to backproject color values from posed images [2]. This idea is further extended by the work titled Neural Radiance Field (NeRF) by also predicting the density [3]. Nevertheless, very low convergence rate and requirement of many structured camera viewpoints hinders the way to real-life applications.

In this Master’s level research project, the student will investigate how to utilize prior information about scenes. This information will be utilized for generalizable neural rendering methods. This will enable large-scale scene analysis from sparse camera view points.

[1] Tewari, Ayush, et al. "State of the art on neural rendering." 

[2] Niemeyer, Michael, et al. "Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision."

[3] Mildenhall, Ben, et al. "Nerf: Representing scenes as neural radiance fields for view synthesis."

• Experience in Python
• Experience in machine learning, data processing and scientific computing frameworks such as: NumPy, SciPy, Tensorflow, Pytorch, Matplotlib, Pandas
• Experience in multiview geometry

Nice to have

• Experience with neural rendering frameworks

cem.eteke@tum.de

Semantic segmentation from point clouds is a challenging task since the methods have to both extract local and global features in a scene. Induced by the pioneering works such as PointNet [1], 3D scene modeling from raw point cloud data has seen a significant growth [2]. Many of the successful models focus on increasing the capacity and the receptive capabilities of the model. The frameworks utilized in these works have either high memory or time complexities or even both. A more recent work further expands the receptive field to make the models efficient [3]. However, having a larger receptive field may result in small details being averaged out and may hinder the performance when the model is fed with out of distribution samples. 

In this Master’s level research project, the student will investigate how to account for the shortcomings of utilizing a high receptive field. The goal is to transfer the pretrained light-weight models to out of distribution data for semantic segmentation task.

References

[1] Qi, Charles R., et al. "Pointnet: Deep learning on point sets for 3d classification and segmentation." 

[2] Guo, Yulan, et al. "Deep learning for 3d point clouds: A survey." 

[3] Hu, Qingyong, et al. "Randla-net: Efficient semantic segmentation of large-scale point clouds." 

• Experience in Python
• Experience in machine learning, data processing and scientific computing frameworks such as: NumPy, SciPy, Tensorflow, Pytorch, Matplotlib, Pandas
• Experience in 3D Computer Vision

Nice to have

• Experience with the ScanNet dataset
• Experience with the Open3D framework

cem.eteke@tum.de

Under this topic, the student should build and evaluate an egocentric camera setup for human activity understanding.

- Understand the requirements for capturing human activity efficiently and effectively (camera requirements, AI compute requirements, wearability, user friendliness).

- Review existing solutions (e.g. GoPro).

- Survey camera modules and suitable embedded sensor+AI platforms (Raspberry Pi / Nvidia Jetson / Intel neural compute sticks / better alternatives).

- Design the best possible setup and evaluate it for predicting a set of daily-life human activities.

• Experience integrating sensor modules and compute platforms
• Experience/interest in Computer Vision, Machine Learning, Deep Learning
• Experience in Linux, Python, C++

https://www.ei.tum.de/lmt/team/mitarbeiter/chaudhari-rahul/

In this project, we aim to capture low-latency RGB data from a CSI camera in sync with an inertial measurement unit(IMU) and use the TPU unit for AI processing, especially for real-time segmentation applications. The first step of the project is to modify the provided embedded Linux kernel and the embedded board with an ARM Cortex processor to communicate with the TPU module. In the next step of the project, the synchronization of the inertial measurement data must be addressed with the frames of the RGB data from the camera. We use Edge TPU ML accelerator ASIC designed by Google that provides high-performance ML inferencing for TensorFlow Lite models.

Good programming skills in C++.

Strong understanding of embedded Linux programming.

Knowledge about Debian Kernel and UBoot.

Knowledge about ARM Cortex Processors.

Please send your application to:

mojtaba.karimi@tum.de

edwin.babaians@tum.de

In this thesis, we want to investigate how captions can be leveraged for weakly supervised object detection.

Weakly supervised learning is branch of machine learning which is very practically relevant, as it aims at alleviating the cost of manually labelling a dataset. 

• Knowledge of state-of-the-art object detection networks
• Python, Numpy, Pytorch
• Optional: Recurrent models, language processing

martin.piccolrovazzi@tum.de

When teaching robots remotely via teleoperation, the communication (between the human demonstrator and the remote robot learner) imposes challanges. Network delay, packet loss and data compression might cause completely or partially degraded demonstration qualities. In this thesis, we would like the make the best out of varying quality demostrations provided via teleoperation with haptic feedback. This will require the identification and classification of the demonstrations based on quality or reliability, and an effective integration of the demonstrations into the imitation learning algorithm.

Requirements:

C/C++ coding skills,

High motivation to learn and conduct research

Familiarity wih learning from demonstration ( HSMM, DMP, GMM/GMR)

basak.guelecyuez@tum.de

This work can be done in German or English

At the current stage of autonomous driving, failures in complex situations are inevitable. A learning-based method to predict such failures could prevent dangerous situations or crashes. However, collecting real-life training data of crashes caused by autonomous vehicles is not feasible. A different solution is to use data from realistic simulations of a self-driving car, such as CARLA [1].

In this project, the objective is to setup available autonomous driving models such as [2,3] and use our existing data logging pipeline to evaluate these model's failure cases. The whole process should be further improved by extending our logging pipeline.

Tasks

• Improving the existing data logging pipeline
• Setup of existing autonomous driving models
• Collection of driving data with the implemented system
• Evaluation of autonomous driving model failures and collection of failure data

References

[1] A. Dosovitskiy, „CARLA: An Open Urban Driving Simulator“, S. 16, 2017.
[2] https://github.com/erdos-project/pylot
[3] https://github.com/commaai/research

• Experience with Python (ROS and Linux)
• Knowledge about Docker would be helpful
• General knowledge about Machine Learning

Object detection can be performed with different sensor modalities, such as camera images, LIDAR point clouds or a fusion thereof. Adverse weather conditions such as rain or fog can cause different failure cases for different sensor types. In this work, an approach for finding the best suited models for different conditions is investigated. In training, each sensor configuration (camera, LIDAR or fusion) is trained separately. Then, the models are finetuned using only the training images they performed well on, leading to a set of expert models designed to work best with a subset of the training images. Finally, a selection model needs to be designed that is trained to select which expert model is most suitable for the current scene. The models can be trained and evaluated in the CARLA simulator, where different weather conditions can be easily generated.

Knowledge of Deep Learning and Linux

In this project we will use sensor data coming from a robot and we will further develop our existing system that has a server-client architecture. The client side is a web-based GUI and the server side includes various interactive segmentation algorithms. We will conduct tests using the developed interface.

- Basic knowledge of /experience with:

• Python
• ROS
• Vue.js JavaScript Framework

- Motivation to learn the necessary tools and yield a successful work.

furkan.kaynar@tum.de

Please provide your CV and Transcript of Records for application.