Offene Themen

The main objective of this project is to provide ultra-fast networking for Java-based big-data applications, which rely on Apache Arrow, a unified columnar in-memory data format which defines a language-independent columnar memory format for flat and hierarchical data, organized for efficient analytic operations on modern hardware like CPUs and GPUs. Furthermore, Arrow is providing numerous language bindings and is used in many big-data and AI frameworks.

For networking the Apache Arrow Flight interface should be used. The default transport in Flight is gRPC which is adding significant overhead. However, there is an alternative providing fast Remote Direct Memoy Access (RDMA) based on the Unified Communication X (UCX) library [4]. UCX is an open-source library, written in C, providing hardware agnostic access to modern networks. But UCX support is only available for C/C++ Arrow bindings and not for Java.

In this project a Flight transport should be implemented providing RDMA for Arrow data for the Java language binding by using Infinileap to access UCX. Infinileap is providing an efficient objectoriented interface between Java and the native UCX ibrary by leveraging Project Panama's Foreign Func9on Interface and Foreign-Memory Access API.

• https://arrow.apache.org
• https://arrow.apache.org/blog/2019/10/13/introducing-arrow-flight/
• https://openucx.org
• https://www.cs.hhu.de/en/research-groups/opera?ng-systems/research/translate-to-english-hermes