Studentische Arbeiten am Lehrstuhl für Medientechnik

When teaching robots remotely via teleoperation, the communication (between the human demonstrator and the remote robot learner) imposes challanges. Network delay, packet loss and data compression might cause completely or partially degraded demonstration qualities. In this thesis, we would like the make the best out of varying quality demostrations provided via teleoperation with haptic feedback. This will require the identification and classification of the demonstrations based on quality or reliability, and an effective integration of the demonstrations into the imitation learning algorithm.

Requirements:

C/C++ coding skills,

High motivation to learn and conduct research

Familiarity wih learning from demonstration ( HSMM, DMP, GMM/GMR)

basak.guelecyuez@tum.de

Tasks:

• Help students with basics of MATLAB (e.g. matrix operations, filtering, image processing, runtime errors)
• Correct some of the homework problems 
• Understand the DSP coursework material

We offer:

• Payment according to the working hours and academic qualification
• Workload is approximately 6 hours per week from April to July
• Technische Universität München especially welcomes applications from female applicants

Application:

• Please send your application with a CV and transcript per e-mail to basak.guelecyuez@tum.de 

• Former DSP students are welcome!
• Matlab experience
• Background in signal or image processing is helpful

In multimodal object detection, different sensors such as cameras or LIDAR have different strengths that are combined for optimal detection rates. However, different sensors also have different weaknesses. In this thesis, a monitoring model for each individual sensor is trained with previous performances of that sensor. For a new input, the sensor's performance is then predicted based only on the sensory input. The predicted performance score is then used in the subsequent sensor fusion to reduce the impact of challenging sensory readings, allowing the fusion architecture to dynamically adapt and rely more on the other sensors instead.

Experience with Machine Learning and Object Detection

We have already a framework of teleoperation system running in Windows, where two haptic devices are connected through UDP protocol, one as the leader device and the other is the follower.

Your tasks are:

1. move the framework to Linux system.

2. setup a ROS-based virtual teleoperation environment.

3. shift the framework to a raspberry PI.

Linux, socket programming (e.g. UDP protocol), C and C++, ROS

1) Installation of the new Force/Torque sensors

2) Installation of the new Real-sense Cameras

3) Installation of the new Grippers

4) Setup the new computer of the robot

5) Be responsible for robot demos and learn how to run the robot demos 

6) Maintain the robot software

7) Add torque sensor checking in robot middleware

8) Validate the robot functionality

9) Updating robot's URDF

10) Maintain the Lab computers

11) NTP Time Synchronization

12) Documentation

13) Maintain lab devices and robot battery

Strong robotic background

Strong C++ background

Highly Motivated to learn

Previous similar experiments

Open for the long period contract (6 Months)

edwin.babaians@tum.de

We are looking for a highly motivated working student, who would like to contribute to our current robotic simulator project. This project's aim is to bring robotics in Unity3D using a high-performance bridge in particular for haptic-enabled reinforcement learning skill learning. 

Responsibility:

1) Add torque/force sensor to robot's end-effector

2) Add joint torque sensors for robot joints

3) Benchmarking of the robotic simulator for reinforcement learning

4) Find differences in Reality and Simulator in manipulation tasks

5)  Add and maintain the docker image of the project

6) Add unit tests

7) build automation

8) documentation

9) continues testing

10) update and maintain the URDF of the LMT's Movo Platform

10) force feedback rendering 

Strong Background in C++

Strong Background in C# and Uniy3D 

Be familiar with Nvidia Physics and Nvidia Flex

Experience in software development 

Familiar with unit testing

Familiar with docker

Familiar with Robotics

edwin.babaians@tum.de

Simulation is a good way of investigating the possibility of a complex behavior implementation for service robots. Since we can not consider all the characteristics of the real work scenario there would be the sim-to-real adaptation step in order to fine-tune the current learned skill from simulations. In this project, the scenario will be the liquid pouring tasks which we already learned in a simulation environment. We will tackle the sim-to-real adaptation problem using haptic communication in order to demonstrate and skill for extra time and the robot will use the expert user knowledge in order to fine-tune the current learned skill from the simulation.

Be familiar with skill refinement techniques. 

Be familiar with haptic teleoperation

Strong background in c++

String background in python

edwin.babaians@tum.de

We want to investigate what are the simulation bottlenecks in order to learn the pouring task. How we can tackle this problem. This project is more paper reading and the field of research is skill refinement and domain adaptation. In addition, we will try to implement one of the states of the art methods of teaching by demonstration in order to adapt the simulation skill to the real-world scenario.

Creativity

Motivation

Strong C++ Background

Strong Phyton Background

edwin.babaians@tum.de

In this work, students will investigate the state-of-the-art methods in order to pour liquids with a service robot. We are eager to implement a Reinforcement Learning algorithm in order to infer liquids, estimated pouring volume, and different characteristics of the fluid dynamics. 

In this project, we will utilize the Unity3D game engine and the simulator which is already developed for Kinova's Movo platform. Using the Unity3D ML-Agent and Nvidia's Flex physics engine we will investigate the state-of-the-art methods in particular in Deep Reinforcement Learning, in order to understand liquids by force interaction. We will also implement the learned model to the real robot and investigate the sim-to-real gap in the liquid pouring task.

References:

1. Pan, Zherong, and Dinesh Manocha. "Feedback motion planning for liquid pouring using supervised learning." In yle="color: #222222; font-family: Arial, sans-serif; font-size: 13px;">2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1252-1259. IEEE, 2017.
2. Zhu, Yuke, Ziyu Wang, Josh Merel, Andrei Rusu, Tom Erez, Serkan Cabi, Saran Tunyasuvunakool et al. "Reinforcement and imitation learning for diverse visuomotor skills." arXiv preprint arXiv:1802.09564 (2018).
3. Kennedy, Monroe, Karl Schmeckpeper, Dinesh Thakur, Chenfanfu Jiang, Vijay Kumar, and Kostas Daniilidis. "Autonomous precision pouring from unknown containers." IEEE Robotics and Automation Letters 4, no. 3 (2019): 2317-2324.
4. http://akihikoy.net/info/res/lfdpour.php
5. Guevara, Tatiana López, Nicholas K. Taylor, Michael U. Gutmann, Subramanian Ramamoorthy, and Kartic Subr. "Adaptable pouring: Teaching robots not to spill using fast but approximate fluid simulation." In yle="color: #222222; font-family: Arial, sans-serif; font-size: 13px;">Proceedings of the Conference on Robot Learning (CoRL). 2017.
6. https://github.com/qiang-ma/Learning-to-Pour-with-Reinforcement-Learning

• Strong C++ Background
• Strong Phyton Background
• Be familiar with Deep Reinforcement Learning methods like SAC or PPO

edwin.babaians@tum.de

Simulation of various sensors to observe virtual 3D Human Activities -- environment sensors (pressure, motion/PIR, open-shut, etc.), wearable sensors (IMU, ego-cams).

Interest/experience in 3D Computer Graphics, Game Engines (esp. Unity3D), C#.

Generation of large-scale synthetic 3D indoor scenes by exploiting scene semantics

Interest/experience in 3D Computer Graphics, Game Engines (esp. Unity3D), C#, C++, ML/DL

Design/integration of sensors/electronics and embedded FW/SW for a novel Wearable device for Human Context Analysis.

• Strong interest and experience in electronics design and integration
• Experience with embedded FW/SW dev and embedded OS

https://www.ei.tum.de/lmt/team/mitarbeiter/chaudhari-rahul/

This work can be done in German or English

At the current stage of autonomous driving, failures in complex situations are inevitable. A learning-based method to predict such failures could prevent dangerous situations or crashes. However, collecting real-life training data of crashes caused by autonomous vehicles is not feasible. A different solution is to use data from realistic simulations of a self-driving car, such as CARLA [1].

In this project, the objective is to setup available autonomous driving models such as [2,3] and use our existing data logging pipeline to evaluate these model's failure cases. The whole process should be further improved by extending our logging pipeline.

Tasks

• Improving the existing data logging pipeline
• Setup of existing autonomous driving models
• Collection of driving data with the implemented system
• Evaluation of autonomous driving model failures and collection of failure data

References

[1] A. Dosovitskiy, „CARLA: An Open Urban Driving Simulator“, S. 16, 2017.
[2] https://github.com/erdos-project/pylot
[3] https://github.com/commaai/research

• Experience with Python (ROS and Linux)
• Knowledge about Docker would be helpful
• General knowledge about Machine Learning

Object detection can be performed with different sensor modalities, such as camera images, LIDAR point clouds or a fusion thereof. Adverse weather conditions such as rain or fog can cause different failure cases for different sensor types. In this work, an approach for finding the best suited models for different conditions is investigated. In training, each sensor configuration (camera, LIDAR or fusion) is trained separately. Then, the models are finetuned using only the training images they performed well on, leading to a set of expert models designed to work best with a subset of the training images. Finally, a selection model needs to be designed that is trained to select which expert model is most suitable for the current scene. The models can be trained and evaluated in the CARLA simulator, where different weather conditions can be easily generated.

Knowledge of Deep Learning and Linux

In this project we will use sensor data coming from a robot and we will further develop our existing system that has a server-client architecture. The client side is a web-based GUI and the server side includes various interactive segmentation algorithms. We will conduct tests using the developed interface.

- Basic knowledge of /experience with:

• Python
• ROS
• Vue.js JavaScript Framework

- Motivation to learn the necessary tools and yield a successful work.

furkan.kaynar@tum.de

Please provide your CV and Transcript of Records for application.

LIDAR sensors allow to capture a scene in 3D while being more robust than cameras to distortions like rain. They are therefore an important part of autonomous driving, where they can be used for semantic segmentation of the environment. For this, each point in the 3D point cloud is classified as belonging to a semantic class such as "car", "pedestrian" or "road". In a safety-critical application such as driving, knowing when such a classification can be trusted or not is important. To this end, failure prediction methods such as introspection [1] can be used to predict where the segmentation failed.

In this internship, a state-of-the-art neural network such as [2] will be implemented to perform semantic segementation of LIDAR point clouds. After implementing the semantic segmentation, a state-of-the-art failure prediction approach will be implemented to detect incorrect classifications. The evaluation will be done using the CARLA driving simulator [3]. A reference implementation based on camera input for both semantic segmentation and failure prediction is available for a comparison.

References:

[1] "Introspective Failure Prediction for Semantic Image Segmentation", Kuhn et al., IEEE ITSC 2020

[2] "RangeNet++: Fast and accurate LiDAR semantic segmentation", Milioto et al., IEEE IROS 2019

[3] https://carla.org/

Basic knowledge of Machine Learning, Python and Linux

Using a teleoperation system with haptic feedback, the users can thus truly immerse themselves into a distant environment, i.e., modify it, and execute tasks without physically being present but with the feeling of being there. A typical teleoperation system with haptic feedback (referred to as a teleoperation system) comprises three main parts: the human operator OP)/master system, the teleoperator (TOP)/slave system, and the communication link/network in between [43]. During teleoperation, the slave and master devices exchange multimodal sensor information over the communication link. This work aims to develop a referenceless subjective quality index for time-delayed teleoperation system. This index is able to describe the subjective quality of experience based on a series of objective metrics.

Your work:

(1) build up a teleoperation system.

(2) design subjective test and collect enough subjective evaluation data as the ground truth.

(3) collect the objective metrics of teleoperation systems and design a training scheme based on the ground truth. Machine learning approaches may be needed.

(4) Evaluate the proposed quality index under different conditions.

C++

Do you want to do your thesis with Stanford professors and work with Apple Siri’s founder Dr. Tom Gruber?

Hospital on Mobile is a Silicon Valley-based Startup in California. Our vision is to make predictive and personalized healthcare accessible to every individual. We use smartphones and embedded sensors to monitor users’ behavioral, environmental, fitness, and health data. We collaborate our research with the world’s leading universities, including Stanford, Harvard, Oxford, and MIT. Achieving such a big vision also requires high-caliber expert support such as worldwide renowned German AI engineer and Chief Scientist of Salesforce Dr. Richard Socher, as well as Siri’s former founder Chief Technology Officer Dr. Tom Gruber, which are just two of them.

We are especially specialized in vital signs monitoring, like heart rate, respiratory rate, oxygen saturation, and blood pressure monitoring through smartphone sensors. Our technology uses computer vision, deep learning and signal processing. The pain attack or cycle of a disease can be predicted and treated personalized at an early stage. Our mission is to make it accessible and affordable (even free to everyone) regardless of somebody’s wealth status. Therefore, we decided to use smartphones, compared to smartwatches or other smart wearables, as they are widely used, even the most remote place in the world. Also, this solution does not need additional devices and break the flow of people’s daily life. With the world’s top universities, we are in the process of running multiple clinical studies for neurodegenerative diseases like Parkinson and Alzheimer; mental health problems like anxiety and depression; migraine; infectious diseases including COVID.

If you want to create huge impact, to help people not only in your country but even in deep Africa, and to work with world’s leading universities like Stanford alongside top scientists, we are looking for you. Don’t be shy, shoot us an email with your CV and a short paragraph about yourself!

www.hospitalonmobile.com www.migraine.ai www.virologic.io

684 Roble Avenue, Menlo Park, CA, USA

Prerequisites:

• Experience with python
• Very good to excellent knowledge in signal processing, image-/video processing
• Good to very good knowledge in linear algebra
• Logic and Algorithms
• Very high motivation and commitment
• Motivation to learn necessary skills and yield a successful work

Optional/Preferred Prerequisites:

• Knowledge in machine learning
• Knowledge in computer vision
• Knowledge in NLP
• Probability Theory
• Good to very good knowledge in linear algebra
• Experience in C/C++
• Background in iOS/Android programmings
• Experience in visual computing and communication
• Experience in electronics, hardware, and firmware programming

What we offer:

• Cutting-edge topics; work that creates impact
• Highly motivated team with weekly group meetings
• Work with top engineers and scientists worldwide
• Remote work
• Compensation

tamaykut@hospitalonmobile.com

The goal is to build a sensor array with which we can acquire vibrotactile signal data on many points simultanesouly. Special consideration is to be given to the human hand. The goal is to build a signal database using different materials and exploration patterns. 

MATLAB, FPGA programming recommended

This topic is about 3D simulation for human activity analysis in indoor environments. The student(s) will use the Unity3D game engine to replicate human activity flows from daily life in a 3D simulator. The student(s) will extend the simulator capabilities to cover a large and complex spectrum of activity flows.

If time permits, the 3D data generated from the simulation will be processed using Machine Learning Techniques for Human intention recognition/anticipation.

This is a great opportunity to contribute to Open-source Software.

Interest/experience in 3D game engines (esp. Unity3D), C#

Dr.-Ing. Rahul Chaudhari

Human Activities of Daily Living are driven by our underlying intentions. For example, the Intention of "making Pasta" spawns a sequence of activities like fetch pasta, boil it, fetch and chop vegetables for the sauce, and clean up after cooking.

We develop Computer Vision / Machine Learning algorithms for RGBD video and other sensors to recognize and anticipate human intentions and activities in indoor environments.

We also work on collecting state-of-the-art sensor datasets and 3D simulation of activities using game engines.

Take a look at the currently running projects below to get a more detailed idea about the above topics.

Ambition, Motivation, interest and first experience in Computer Vision, ML/DL, Python, C++

This work can be done in German or English

LIDAR is one important sensor type for autonomous vehicles' perception. Human perception is mainly based on RGB data, in case of teleoperation captured by RGB cameras and transmitted to the remote operator through a communication network. In some situation a 3D representation of the scene might be helpful for the operator which could be achieved using LIDAR data. To avoid hight transmission rate LIDAR data need to be compressed. Existing methods for point cloud compression [1, 2, 3] don't have their focus on automotive LIDAR data.

The objective of this project is to setup existing point cloud compression implementations and compare them focusing on automotive point clouds.

Tasks

• Setup available point cloud compression implementations
  * ROS PCL [1]
  * MPEG L-PCC [2] (available at MPEG Repo)
  * MPEG G-PCC [2] (available at MPEG Repo)
  * MPEG Anchor Implementation [3]
• Evaluate implementations in terms of
  * Encoding time/complexity
  * Compression rate
  * Compression quality

• Experience with ROS and Point Clouds
• Basic knowledge of C++ and Linux