We constantly offer topics for student projects (engineering experience, research experience, student, IDPs) and final year projects (bachelor thesis or master thesis).
Model Predictive Controller for Franka Panda
In this project, the student will extend the currently developed motion controller for the well-known Franka Panda robot. Finally, he/she will evaluate the planner on the real robot.
Strong C++ background
Topics in game-engine based human activity simulation
Existing human activity simulators should be made operational and evaluated.
Comfortable with C++ (for Unreal Engine based activity generation and sensor data recording)
Comfortable with matlab/python for sensor data analysis.
low-latency embedded inertial-camera synchronization system
embedded linux, ARM Cortex, CSI port, IMU
In this project, we aim to capture low-latency RGB data from a CSI camera in sync with an inertial measurement unit. A CSI camera must be defined in a device tree of the Linux kernel in order to have access to the low-level functionality of the camera sensor. The first step of the project is to modify the provided embedded Linux kernel and the embedded board with an ARM Cortex processor to read the raw data of the sensors. In the next step of the project, the synchronization of the inertial measurement data must be addressed with the frames of the RGB data from the camera.
Good programming skills in C++.
Strong understanding of embedded Linux programming.
Knowledge about Debian Kernel and UBoot.
Knowledge about ARM Cortex Processors.
Please send your application to:
Semi-automatic Extraction of a Knowledge Graph of Human Activities from Data
For our research on Human Activity Understanding, we want to represent relations between different entities (objects, actions, sub-activities, affordances, etc.) and attach information/properties to nodes and edges.
Initially, we would like to be able to
- Visualize the Knowledge Graph
- Update the nodes/edges in a GUI
We would also look into existing tools that would allow building/visualization/queries on such graphs.
Some use cases for this would be:
- Works as a labeling tool to create a hierarchy of activities/sub-activities/actions to enrich the EPIC egocentic visual dataset of daily life activities (https://epic-kitchens.github.io/2021).
- Learn to automatically/semi-automatically do such labeling based on users' feedback.
Weakly Supervised Object Detection Using Captions
Object Detection, Convolutional Neural Networks, Language Models, Recurrent Models
In this thesis, we want to investigate how captions can be leveraged for weakly supervised object detection.
Weakly supervised learning is branch of machine learning which is very practically relevant, as it aims at alleviating the cost of manually labelling a dataset.
- Knowledge of state-of-the-art object detection networks
- Python, Numpy, Pytorch
- Optional: Recurrent models, language processing
Digital Twinning of a Smart Home for Semantic Communication
digital twin, smart home, sensors, Unity3D
This project is about creating a dynamic digital twin of a smart home:
- A realistic 3D representation of the apartment, the furniture, and the objects in it in should be created for our game-engine based activity simulator. A 2D CAD model of the apartment is available as a starting point. Our activity simulator is based on the Unity3D game engine.
- The smart home is equipped with a variety of sensors -- ceiling-mounted mocap cameras, magnetic contact sensors for doors and drawers, pressure sensors in the floor, werable IMUs and eye-trackers, etc. These sensors should be simulated in Unity3D.
- The digital twin should be made dynamic -- human activity and changes effected in the physical smart home by a human should be reflected in quasi real-time in the digital twin.
- How can the digital twin be used to support seamless interpersonal video conferencing under variable data rate or lossy network conditions? (investigate blending/transition between real and simulated video).
- How to minimize the sim (digital twin) - to - real (physical smart home) gap for automatic activity recognition?
This project is part of a collaboration with the TU Ostrava (Czech republic) Biomedical Engineering Research Group. The apartment mentioned abive refers to TU Ostrava's "Living Lab".
Network Aware Imitation Learning
When teaching robots remotely via teleoperation, the communication (between the human demonstrator and the remote robot learner) imposes challanges. Network delay, packet loss and data compression might cause completely or partially degraded demonstration qualities. In this thesis, we would like the make the best out of varying quality demostrations provided via teleoperation with haptic feedback. This will require the identification and classification of the demonstrations based on quality or reliability, and an effective integration of the demonstrations into the imitation learning algorithm.
C/C++ coding skills,
High motivation to learn and conduct research
Familiarity wih learning from demonstration ( HSMM, DMP, GMM/GMR)
Upgrading Movo's Platform
1) Installation of the new Force/Torque sensors
2) Installation of the new Real-sense Cameras
3) Installation of the new Grippers
4) Updating robot's URDF
Strong robotic background
Strong C++ background
Previous similar experiments
Open for the long period contract (6 Months)
Realisitc Robotic Simulator
Unity3D, ML-agents, Nvidia Physics 4
We are looking for a highly motivated working student who would like to contribute to our current robotic simulator project. This project aims to bring robotics in Unity3D using a high-performance bridge, particularly for haptic-enabled reinforcement learning skill learning.
1) Add torque/force sensor to robot's end-effector
2) Add joint torque sensors for robot joints
3) Build automation
4) Update and maintain the URDF of the LMT's Movo Platform
5) Force feedback rendering Prerequisites Strong Background in C++
Strong Background in C++
Strong Background in C# and Uniy3D
Be familiar with Nvidia Physics and Nvidia Flex (Optional)
Experience in software development
Familiar with Robotics
Open for the long period contract (6 Months)
How Well Perform Today's Autonomous Driving Models
This work can be done in German or English
At the current stage of autonomous driving, failures in complex situations are inevitable. A learning-based method to predict such failures could prevent dangerous situations or crashes. However, collecting real-life training data of crashes caused by autonomous vehicles is not feasible. A different solution is to use data from realistic simulations of a self-driving car, such as CARLA .
In this project, the objective is to setup available autonomous driving models such as [2,3] and use our existing data logging pipeline to evaluate these model's failure cases. The whole process should be further improved by extending our logging pipeline.
- Improving the existing data logging pipeline
- Setup of existing autonomous driving models
- Collection of driving data with the implemented system
- Evaluation of autonomous driving model failures and collection of failure data
 A. Dosovitskiy, „CARLA: An Open Urban Driving Simulator“, S. 16, 2017.
- Experience with Python (ROS and Linux)
- Knowledge about Docker would be helpful
- General knowledge about Machine Learning
Multimodal Object Detection with Introspective Experts
Object detection can be performed with different sensor modalities, such as camera images, LIDAR point clouds or a fusion thereof. Adverse weather conditions such as rain or fog can cause different failure cases for different sensor types. In this work, an approach for finding the best suited models for different conditions is investigated. In training, each sensor configuration (camera, LIDAR or fusion) is trained separately. Then, the models are finetuned using only the training images they performed well on, leading to a set of expert models designed to work best with a subset of the training images. Finally, a selection model needs to be designed that is trained to select which expert model is most suitable for the current scene. The models can be trained and evaluated in the CARLA simulator, where different weather conditions can be easily generated.
Knowledge of Deep Learning and Linux
Development of Interactive Teleassistance Interface for Interactive Segmentation
In this project we will use sensor data coming from a robot and we will further develop our existing system that has a server-client architecture. The client side is a web-based GUI and the server side includes various interactive segmentation algorithms. We will conduct tests using the developed interface.
- Basic knowledge of /experience with:
- Motivation to learn the necessary tools and yield a successful work.
Please provide your CV and Transcript of Records for application.
A blind/referenceless subjective quality assessment for time-delayed teleoperation
quality metrics, teleoperation, time delay, haptics
Using a teleoperation system with haptic feedback, the users can thus truly immerse themselves into a distant environment, i.e., modify it, and execute tasks without physically being present but with the feeling of being there. A typical teleoperation system with haptic feedback (referred to as a teleoperation system) comprises three main parts: the human operator OP)/master system, the teleoperator (TOP)/slave system, and the communication link/network in between . During teleoperation, the slave and master devices exchange multimodal sensor information over the communication link. This work aims to develop a referenceless subjective quality index for time-delayed teleoperation system. This index is able to describe the subjective quality of experience based on a series of objective metrics.
(1) build up a teleoperation system.
(2) design subjective test and collect enough subjective evaluation data as the ground truth.
(3) collect the objective metrics of teleoperation systems and design a training scheme based on the ground truth. Machine learning approaches may be needed.
(4) Evaluate the proposed quality index under different conditions.
Student Thesis & Internships at Hospital on Mobile (Extern)
Do you want to do your thesis with Stanford professors and work with Apple Siri’s founder Dr. Tom Gruber?
Hospital on Mobile is a Silicon Valley-based Startup in California. Our vision is to make predictive and personalized healthcare accessible to every individual. We use smartphones and embedded sensors to monitor users’ behavioral, environmental, fitness, and health data. We collaborate our research with the world’s leading universities, including Stanford, Harvard, Oxford, and MIT. Achieving such a big vision also requires high-caliber expert support such as worldwide renowned German AI engineer and Chief Scientist of Salesforce Dr. Richard Socher, as well as Siri’s former founder Chief Technology Officer Dr. Tom Gruber, which are just two of them.
We are especially specialized in vital signs monitoring, like heart rate, respiratory rate, oxygen saturation, and blood pressure monitoring through smartphone sensors. Our technology uses computer vision, deep learning and signal processing. The pain attack or cycle of a disease can be predicted and treated personalized at an early stage. Our mission is to make it accessible and affordable (even free to everyone) regardless of somebody’s wealth status. Therefore, we decided to use smartphones, compared to smartwatches or other smart wearables, as they are widely used, even the most remote place in the world. Also, this solution does not need additional devices and break the flow of people’s daily life. With the world’s top universities, we are in the process of running multiple clinical studies for neurodegenerative diseases like Parkinson and Alzheimer; mental health problems like anxiety and depression; migraine; infectious diseases including COVID.
If you want to create huge impact, to help people not only in your country but even in deep Africa, and to work with world’s leading universities like Stanford alongside top scientists, we are looking for you. Don’t be shy, shoot us an email with your CV and a short paragraph about yourself!
www.hospitalonmobile.com www.migraine.ai www.virologic.io
684 Roble Avenue, Menlo Park, CA, USA
- Experience with python
- Very good to excellent knowledge in signal processing, image-/video processing
- Good to very good knowledge in linear algebra
- Logic and Algorithms
- Very high motivation and commitment
- Motivation to learn necessary skills and yield a successful work
- Knowledge in machine learning
- Knowledge in computer vision
- Knowledge in NLP
- Probability Theory
- Good to very good knowledge in linear algebra
- Experience in C/C++
- Background in iOS/Android programmings
- Experience in visual computing and communication
- Experience in electronics, hardware, and firmware programming
What we offer:
- Cutting-edge topics; work that creates impact
- Highly motivated team with weekly group meetings
- Work with top engineers and scientists worldwide
- Remote work
Multidimensional Vibrotactile Signal Acquisition
The goal is to build a sensor array with which we can acquire vibrotactile signal data on many points simultanesouly. Special consideration is to be given to the human hand. The goal is to build a signal database using different materials and exploration patterns.
MATLAB, FPGA programming recommended