Jan Hendrik Metzen's publications

We introduce a new online skill discovery method for reinforcement learning in discrete domains. The method is based on the bottleneck principle and identifies skills using a bottom-up hierarchical clustering of the estimated transition graph. In contrast to prior clustering approaches, it can be used incrementally and thus several times during the learning process. Our empirical evaluation shows that “assuming high connectivity in the face of uncertainty” can prevent premature identification of skills. Furthermore, we show that the choice of the linkage criterion is crucial for dealing with non-random sampling policies and stochastic environments.

Machine learning is increasingly used to autonomously adapt brain-machine interfaces to user-specific brain patterns. In order to minimize the preparation time of the system, it is highly desirable to reduce the length of the calibration procedure, during which training data is acquired from the user, to a minimum. One recently proposed approach is to reuse models that have been trained in historic usage sessions of the same or other users by utilizing an ensemble-based approach. In this work, we propose two extensions of this approach which are based on the idea to combine predictions made by the historic ensemble with session-specific predictions that become available once a small amount of training data has been collected. These extensions are particularly useful for Brain Reading Interfaces (BRIs), a specific kind of brain-machine interfaces. BRIs do not require that user feedback is given and thus, additional training data may be acquired concurrently to the usage session. Accordingly, BRIs should initially perform well when only a small amount of training data acquired in a short calibration procedure is available and allow an increased performance when more training data becomes available during the usage session. An empirical offline-study in a testbed for the use of BRIs to support robotic telemanipulation shows that the proposed extensions allow to achieve this kind of behavior.

Machine learning approaches are increasingly used in brain-machine-interfaces to allow automatic adaptation to user-specific brain patterns. One of the most crucial factors for the practical success of these systems is that this adaptation can be achieved with a minimum amount of training data since training data needs to be recorded during a calibration procedure prior to the actual usage session. To this end, one promising approach is to reuse models based on data recorded in preceding sessions of the same or of other users. In this paper, we investigate under which conditions it is favorable to reuse models (more specifically spatial filters) trained on data from historic sessions compared to learning new spatial filters on the current session's calibration data. We present an empirical study in a scenario in which Brain Reading, a particular kind of brain-machine-interface, is used to support robotic telemanipulation.

Many medical applications require a registration of different images of the same organ. In many cases, such a registration is accomplished by manual placement of landmarks in the images. In this paper we propose a method which is able to find reasonable landmarks automatically. To achieve this, bifurcations of the vessel systems, which have been extracted from the images by a segmentation algorithm, are assigned by the so-called association graph method and the coordinates of these matched bifurcations can be used as landmarks for a non-rigid registration algorithm. Several constraints to be used in combination with the association graph method are proposed and evaluated on a ground truth consisting of anatomical trees from liver and lung. Furthermore, a method for preprocessing (tree pruning) as well as for postprocessing (clique augmentation) are proposed and evaluated on this ground truth. The proposed method achieves promising results for anatomical trees of liver and lung and for medical images obtained with different modalities and at different points in time.

For many complex Reinforcement Learning problems with large and continuous state spaces, neuroevolution (the evolution of artificial neural networks) has achieved promising results. This is especially true when there is noise in sensor and/or actuator signals. These results have mainly been obtained in offline learning settings, where the training and evaluation phase of the system are separated. In contrast, in online Reinforcement Learning tasks where the actual performance of the systems during its learning phase matters, the results of neuroevolution are significantly impaired by its purely exploratory nature, meaning that it does not use (i.e. exploit) its knowledge of the performance of single individuals in order to improve its performance during learning. In this paper we describe modifications which significantly improve the online performance of the neuroevolutionary method Evolutionary Acquisition of Neural Topologies (EANT) and discuss the results obtained on two benchmark problems.

Many multiagent problems comprise subtasks which can be considered as reinforcement learning (RL) problems. In addition to classical temporal difference methods, evolutionary algorithms are among the most promising approaches for such RL problems. The relative performance of these approaches in certain subdomains (e.g. multiagent learning) of the general RL problem remains an open question at this time. In addition to theoretical analysis, benchmarks are one of the most important tools for comparing different RL methods in certain problem domains. A recently proposed multiagent RL benchmark problem is the RoboCup Keepaway benchmark. This benchmark is one of the most challenging multiagent learning problems because its state-space is continuous and high dimensional, and both the sensors and the actuators are noisy. In this paper we analyze the performance of the neuroevolutionary approach called Evolutionary Acquisition of Neural Topologies (EANT) in the Keepaway benchmark, and compare the results obtained using EANT with the results of other algorithms tested on the same benchmark.

Many medical applications require a registration of different images of the same organ. In many cases, such a registration is accomplished by manually placing landmarks in the images. In this paper we propose a method which is able to find reasonable landmarks automatically. To achieve this, nodes of the vessel systems, which have been extracted from the images by a segmentation algorithm, will be assigned by the so-called association graph method and the coordinates of these matched nodes can be used as landmarks for a non-rigid registration algorithm.