From Xenomai

Real-time applications should be designed with real-time constraints in mind from the ground up. Unfortunately, it sometimes happens that we have to modify a non real-time application to make it, or more likely some part of it, meet deadlines. This document tries to be a list of recipes which I found useful when doing such a job using Xenomai, and in particular for the Xenomai POSIX skin.

If you would like to see this document changed in any way, please send suggestions or modifications to the xenomai-help mailing list.

Overview

On a system running Xenomai there are actually two kernels: the Xenomai real-time kernel, scheduling threads using real-time scheduling policies and the Linux kernel, scheduling its threads as usual, running when no real-time thread of higher priority is runnable by the Xenomai kernel.

To ease working with this dual-kernel system, a Xenomai application thread may run in two modes: either the primary mode, where it is scheduled by the Xenomai kernel, and benefits from hard real-time scheduling latencies, or the secondary mode, where it is an ordinary Linux thread, and as such may call any Linux services.

Such a thread may change mode dynamically, that is, when this thread calls a Xenomai real-time service while running in secondary mode, it switches to primary mode, when it calls any non real-time Xenomai service or any Linux service (including exceptions such as page faults) while running in primary mode, it switches to secondary mode.

This description of Xenomai is a bit sketchy, for more information, you should have a look at some Xenomai publications:

• Life with Adeos
• Native API Tour

Your aim, when getting an application to meet its deadlines using Xenomai, is to identify the "performance constrained" loop in your application, and arrange for the thread running this loop to never get out of primary mode. This means that you must identify the calls made by this thread to Xenomai non real-time services and to Linux services, and replace them some way or another.

To identify the call to services making your application switch to secondary mode, there are essentially two approaches:

• the static code analysis;
• the run-time checks.

The best method is probably a blend of the two: find the problematic service calls with a static code analysis, then validate your modified code with run-time checks.

Compilation for the Xenomai POSIX skin

The very first step to make your application meet deadlines using the Xenomai posix skin is to recompile it for this skin. It will not make your application a real-time application auto-magically, but it is needed if you intend to find the problematic service calls using run-time checks.

Compilation flags

To ease that task, the 'xeno-config' script, installed when compiling Xenomai user-space support, is able to give you these flags. For instance, you can add the following snippet to your makefile:

XENO_DESTDIR:=/home/gilles/cross-install
XENO_CONFIG:=$(XENO_DESTDIR)/usr/xenomai/bin/xeno-config
XENO_POSIX_CFLAGS:=$(shell DESTDIR=$(XENO_DESTDIR) $(XENO_CONFIG) --posix-cflags)
XENO_POSIX_LIBS:=$(shell DESTDIR=$(XENO_DESTDIR) $(XENO_CONFIG) --posix-ldflags)

XENO_DESTDIR is the staging directory used when installing cross-compiled binaries and libraries, in other words, the local copy of the root file-system of your embedded system. It is / when the compilation system is also the system where Xenomai runs.

/usr/xenomai/bin/xeno-config is the default path of the xeno-config script under this root file-system, but it may be different if you chose a different installation prefix using Xenomai's configure script --prefix parameter (whose default value is /usr/xenomai).

You may then add $(XENO_POSIX_CFLAGS) to the compilation flags of all your object files, and $(XENO_POSIX_LIBS) to the link flags of all your dynamic libraries (using POSIX services) and executable files.

Beware: this way of obtaining the compilation flags is recommended, if for anything because it will make using a different release of Xenomai easier: the flags may change between two different releases.

Under the hood: the --wrap flag

So, what do these flags do? The main part of the job is done in the XENO_POSIX_LIBS flags. They contain a list of --wrap directives to substitute calls to Linux POSIX services with calls to Xenomai POSIX services.

For instance, if the compiled application contains a call to pthread_create, and you pass --wrap pthread_create on the linker command line (so, if you link with gcc, you have to pass -Wl,--wrap,pthread_create on the command line), this call to pthread_create will be replaced with a call to __wrap_pthread_create.

The Xenomai POSIX skin library, called libpthread_rt.so, also passed in XENO_POSIX_LIBS does contain an implementation of __wrap_pthread_create which will be the one finally called by your program.

Among the few methods available to get POSIX applications to use Xenomai services, this one was chosen because:

• plain Linux services are still available; you may need them, and they would not have been available if, for instance macros had been used.
• there is no need to change the POSIX services names in the program; after all, the POSIX services names are standard.

Using static libraries

Due to the use of the --wrap flag, it is necessary to do the link edition of an application with static versions of the Xenomai POSIX skin library two stages. To help in this matter, recent versions of Xenomai include a script called "wrap-link.sh".

For more information try wrap-link.sh --help.

Incompatibilities

Now that you have recompiled your application to use the Xenomai POSIX skin library, it may no longer work, there are small differences which you should address before going further.

mlockall

A well behaved Xenomai application must call

mlockall(MCL_CURRENT | MCL_FUTURE);

before anything else. When you do not call mlockall, Linux uses an on-demand paging scheme, that is memory pages are only allocated upon first use. The problem is that for this to work an exception (namely a page fault) has to happen, and some Linux kernel services have to run, including the dynamic allocation of a physical page, so the exception requires a thread running in primary mode to switch to secondary mode. When using mlockall, on the other hand, any memory demanded to the kernel is immediately commited, so that page faults no longer happen. Note that this paging scheme is not related to swapping, so, even on a system without swap, mlockall must be called.

On some architectures, calling mlockall may not be enough to eliminate all the page faults, but apart from such cases, the Adeos I-pipe avoids page faults when possible.

Unfortunately, using mlockall has some effects, and you must pay attention to a detail otherwise considered unimportant in a Linux application: the threads stack size. As a matter of fact, if the Linux threading library asks for 2MiB (the default on most platforms) to the system for thread stacks, the 2MiB are immediately allocated. With numerous threads and a memory-constrained embedded system, this quickly becomes a problem.

So, in order to avoid this problem, you should always set the threads stack size to some sensible value, using the pthread_attr_setstacksize service before calling pthread_create. Please note, however, that some apparently innocuous libc services such as printf use the stack, so setting the stack size to some really low value like 4KiB will cause segmentation faults due to stack overflows in these services.

Real-time priorities

To get your application threads to be really considered as real-time threads by Xenomai scheduler, you will have to get them to use the real-time scheduling policy (called SCHED_FIFO). To do that, you either have to use the pthread_attr_setinheritsched, pthread_attr_setschedpolicy, pthread_attr_setschedparam services before the call to pthread_create, or to use the pthread_setschedparam to change an existing thread scheduling parameters.

Note, however, what the SCHED_FIFO scheduling policy means: it means that the scheduler will run a thread with this scheduling policy as long as it is runnable, and no other thread of higher priority is runnable. Concretely, if a thread using the SCHED_FIFO policy runs an infinite loop, nothing else runs, your system is locked up. Such things happen. For instance, such innocent piece of code as:

pthread_mutex_lock(&mutex);
/* (...) */
while (!cond)
     pthread_cond_wait(&cond, &mutex);
/* (...) */
pthread_mutex_unlock(&mutex);

may cause such an infinite loop if cond or mutex are not initialized, or after one of them has been destroyed.

Static mutex and condition variables initializations

The POSIX standard defines PTHREAD_COND_INITIALIZER and PTHREAD_MUTEX_INITIALIZER for static mutexes and condition variables initialization.

Unfortunately, the Xenomai POSIX skin requires a system call to initialize these objects. So, we were left we two choices when implementing these objects:

• either initialize the object upon first call to another service,
• or require the users to call initialization services.

We chose the second solution: having pthread_mutex_lock call the initialization routine would destroy the determinism expected from such a service, on the other hand having users call pthread_mutex_init by themselves force them to do it at a non critical time.

So, to get an application to use Xenomai POSIX skin mutexes and condition variables, you have to look for all the static initializer and replace them with calls to pthread_mutex_init/pthread_cond_init made at non critical times.

Use of Linux original services

It may happen that you would like to use Linux services instead of Xenomai POSIX skin overloaded services. In this case, the --wrap mechanism described in section 2.2 offers a solution: prefix the name of the service you would like to use with the __real_ prefix, such as, for instance __real_pthread_create.

If you do that, and would still want to be able to compile your application without Xenomai (it may be a good idea, as it allows, for instance, to run your application with valgrind, which you can not do with an application compiled for Xenomai), Xenomai compilation flags define a preprocessor macro (__XENO__) which allows you to know whether or not you are compiling the application for Xenomai. You can use it for instance in the following way:

/* Open a plain Linux UDP socket. */
#ifndef __XENO__
      fd = socket(PF_INET, SOCK_DGRAM, 0);
#else /* __XENO__ */
      fd = __real_socket(PF_INET, SOCK_DGRAM, 0);
#endif /* __XENO__ */

Mixing fork with Xenomai POSIX skin services

Most Xenomai services are handled on a per-process basis, which means that by default, you can not use in a process, an object (mutex or condition variable, for instance), defined in another process. Unfortunately, this means that when using fork, contrarily to what happen for a plain Linux process, the child can not use objects which have been initialized by its parent process.

There are two ways out of this issue. Either, what you really want to do is simply to make your process a daemon, you do not really want to share objects between; in this case, you should arrange for the initialization services to be called after the fork, and everything should work normally. Please note that this may not be as easy as it seems, for instance, when using C++ static objects with a non trivial constructor, the constructor gets invoked before even entering the main function. To solve this particular issue, a possible approach is to modify the object constructor to put the uninitialized objects in a list and exit immediately, and walk the list after the fork to trigger the constructor again and run the POSIX skin objects initialization services.

If, on the other hand what you want to do is to really share the POSIX skin objects between several processes, in which case, you should use the pthread_mutexattr_setpshared, pthread_condattr_setpshared or pass 1 as second argument of the sem_init services.

Chasing the unwanted mode switches

If you have followed the indications in the previous sections, you should now have an application which compiles and runs on a Xenomai-enabled system. It may still not be a real-time application, because the time-critical loop, or real-time loop may still be synchronized with some Linux activities, and as such, may not be able to meet short deadlines. There are various causes why this may happen.

The first of them is the reproducible unwanted mode switch to secondary mode. It may be due to the use of a Linux system call, or exceptions of any kind (unaligned accesses on processor where this is not supported, or FPU exceptions cause by floating point computations errors come to my mind). As we will see later, this one is easy to detect.

The second is the seldom unwanted mode switch to secondary mode. As a matter of fact, there are function calls such as malloc for instance, which do their job most of the time without issuing a system call, but which issue a system call from time to time. Due to their unfrequent nature, these ones are harder to catch, but it is still possible.

Finally comes a special kind of priority inversion. It is not an unwanted mode switch per se, but has the same effect. It happens if a thread shares data with the real-time loop thread and protects these data with a mutex, and experiences a rescheduling, or a mode switch while holding this mutex. If our critical thread, the one running the real-time loop, now wants to lock the mutex, it will have to wait for the non critical thread to synchronize with Linux, and end the critical section. For instance:

Xenomai run-time checks have recently been extended to detect such condition, but it has not always been the case, and the condition may gone unnoticed, so you will have to rely on static code analysis to avoid this issue. The rule of thumb is this: if a mutex is shared between critical and non-critical threads, enable priority inheritance for this mutex, and do not ever make a call to a secondary mode service while holding it.

Detection

Using the PTHREAD_WARNSW bit

This Xenomai feature enables run-time checks on a per-thread basis.

To enable these checks for the current thread use:

pthread_set_mode_np(0, PTHREAD_WARNSW);

As this call is specific to Xenomai (as indicated by the _np suffix), you may want to surround it with a #ifdef __XENO__.

This will detect run-time errors and cause a SIGXCPU signal to be sent to the thread, you will find an example of a way to use this signal in the file examples/native/sigxcpu.c in Xenomai sources distribution.

Using --wrap

For the second kind of unwanted mode switches (the unfrequent ones), for which the run-time checks may not be enough, there is a way to get them detected anyway.

Define for instance the following function:

void *__wrap_malloc(size_t size)
{
	getpid();
	return __real_malloc(size);
}

And link the final executable passing -Wl,--wrap,malloc on gcc command line. This way, when malloc happens to be called by a thread running in primary mode, the call to getpid() will cause a systematic switch to secondary mode.

Actually, malloc is a bad example, because recent versions of Xenomai already handle the case of malloc, but the same trick may be used with other services.

Remedies

This section gives a list of the usual causes of secondary mode switches and proposes various remedies.

Access to drivers

In this case the secondary mode switches are due to calls to open, read, write, ioctl, socket, connect, sendto, recvfrom, etc...

The cure is to rewrite drivers using a Xenomai based driver framework. The common drivers skin is RTDM, a set of Xenomai services which offer the equivalent of Linux services for writing drivers like character devices and socket protocols.

On top of RTDM, other APIs exist such as Real-time socket CAN, an API for writing drivers for the CAN protocol, Comedi/RTDM, an API for acquisition cards, RTnet, an implementation of an UDP/IP layer for real-time ethernet drivers, USB4RT, an API for USB drivers, and probably other such APIs.

Porting a Linux driver to RTDM is usually not as hard as it seems: the RTDM services resembles their Linux equivalent, so any people with Linux driver knowledge should be able to port drivers to RTDM. For more information on the RTDM framework, see: RTDM and Applications

From an application point of view, using the Xenomai POSIX skin wraped services allows for manipulation of file descriptors provided by the RTDM skin as if they were ordinary file descriptors

Logging / writing to files

This should not come as a surprise, but calls to printf, fprintf, and more generally all the stdio functions may result in the call the write system call, which means a switch to secondary mode.

The cure, on newer versions of Xenomai, is to use the rt_printf service, from the rtdk library. On older versions, the cure was to delegate the write to file to a separate thread, and communicate with that thread using for instance POSIX message queues (opened in non-blocking mode on its real-time side), or a simple FIFO.

Reading from file

Of course, reading from files also causes switches to secondary mode. However, a simple solution is available: the mmap service. Thanks to the use of mlockall, mmaping a file is equivalent to loading it entirely in memory. The call to mmap itself causes a switch to secondary mode, but in most cases, it is possible to call this service in a non critical part of the code. Note however, that doing this may consume a lot of memory if the file is large.

Timing services

The only supported clock service handled by the Xenomai POSIX skin is clock_gettime. This may be seen as a limitation, and will probably be extended in the future, but in the mean time, if you do not want to replace all calls to gettimeofday with calls to clock_gettime, you should use the --wrap technique, described previously. Note that because Xenomai and Linux clocks are not synchronized by default, it is best if an application uses only Xenomai timing services. So, for instance, it is better to also wrap calls to the seemingly un-critical calls to time.

Here are examples of wrappers for the time and gettimeofday services:

int __wrap_gettimeofday(struct timeval *tv, struct timezone *tz)
{
       struct timespec ts;
       int ret = clock_gettime(CLOCK_REALTIME, &ts);
       if (!ret) {
               tv->tv_sec = ts.tv_sec;
               tv->tv_usec = ts.tv_nsec / 1000;
       }
       return ret;
}

time_t __wrap_time(time_t *t)
{
       struct timespec ts;
       int ret = clock_gettime(CLOCK_REALTIME, &ts);
       if (ret)
		return (time_t) -1;

       if (t)
		*t = ts.tv_sec;
	return ts.tv_sec;
}

The situation of timer services is indeed a limitation of the Xenomai POSIX skin, and a bigger one. The old-style setitimer service is not supported. The new style services timer_settime, timer_gettime, timer_getoverrun are supported, but unfortunately, since they use signals, they cause switches to secondary mode. So, the only way to handle this currently is to create a thread for handling timers at application level. This thread may be based on the clock_nanosleep service, which allows passing absolute expiration dates. We currently have no example of this.

Dynamic allocations

As should now be obvious, dynamic allocation services, i.e. malloc, calloc, posix_memalign, etc... cause switches to secondary mode, so they should be avoided.

The usual way to handle allocations in a real-time application is to allocate memory at startup, and have finite limits on the memory usable by the application. This is acceptable in the case of real-time applications, because they should be simple enough, and their usage known in advance, to be able to assess their maximum memory usage.

Other methods include using stack for allocation.

One technique we found useful in a C++ program using the STL was to implement an allocator for the STL containers implementing the standard allocator interface.

I/O multiplexing with select

The select service allows to wait for events on several file descriptors. Recent versions of the Xenomai POSIX skin support the select service, however, it only works for RTDM file descriptors and POSIX message queue descriptors. You can not mix these kinds of file descriptors with plain Linux file descriptors. So, if your application runs a thread handling events on various file descriptors, and after porting this application to Xenomai, you would like to run select on Linux file descriptors as well as Xenomai file descriptors, you will have to replace this thread with two threads: one for Linux file descriptors, one for Xenomai file descriptors.

Unfortunately, you are not finished when you have done that. That is because an application using select has some kind of built-in protection again multiple accesses to data: everything done by the thread calling select is sequenced, so there is no multiple access. By splitting the thread in two threads, you break that protection.

One way to avoid this issue may be to get the threads to communicate. For instance, when one thread suppresses data, and does not want a second thread to use the data which has been freed, the first thread may notify the second.

For a non real-time thread to notify a real-time thread, a suitable notification mechanism is to use POSIX message queues. As they may be used in a select syscall, the real-time thread may wait for such notification as part of its global call to select.

For a real-time thread to notify a non real-time thread, a suitable mechanism would be to use a POSIX interface for the Xenomai real-time pipes, they would be the symetric service of the message queues: they can be signaled from real-time context, and their non real-time end is a plain Linux file descriptor which can be used with select. If need comes for such services, they could be implemented.