From Xenomai

Adapting ARM I-pipe patch to a new board.

Linux board X specific code is located in the arch/arm/mach-X and include/asm-arm/arch-X subdirectories of Linux kernel sources.

To start porting the I-pipe patch to a new board, look in these directories to see where the hardware timer and its interrupt are handled (usually arch/arm/mach-X/time.c) and where cascaded interrupt, if any, are decoded (usually arch/arm/mach-X/irq.c). In the rest of this document we will call these files time.c and irq.c, even though they may have another name for your board.

New variables and functions.

You are going to modify these two files to define the following variables and functions:

I-pipe and dynamic tick.

The dynamic tick (the old version, CONFIG_NO_IDLE_HZ) is not supported with the I-pipe patch. So, please include the following snippet in time.c:

#ifdef CONFIG_IPIPE
#ifdef CONFIG_NO_IDLE_HZ
#error "dynamic tick timer not yet supported with IPIPE"
#endif /* CONFIG_NO_IDLE_HZ */
#endif /* CONFIG_IPIPE */

Note however that the I-pipe supports the newer tickless mode (CONFIG_NO_HZ) if the underlying board supports CONFIG_GENERIC_TIME and CONFIG_GENERIC_CLOCKEVENTS, see Adapting to boards using CONFIG_GENERIC_TIME and CONFIG_GENERIC_CLOCKEVENTS for more details.

Modifying the timer management.

The timer interrupt handler should be modified in two ways:

• first, if the timer interrupt needs to be acknowledged in some way (clearing a

flag in a hardware register, reading a status in a hardware register, etc...), this acknowledgement must be put in __ipipe_mach_acktimer, because when the timer is handled by Xenomai, timer interrupts will no longer get handled by Linux timer interrupt handler.

• second, if the timer interrupt runs timer_tick() in a while loop, accounting

for lost ticks by comparing the current value of a free-running counter with a hardware match register, this comparison must be replaced with a comparison of the free-running counter with a variable, because when Xenomai controls the timer, the match register is no longer related to Linux timer tick date.

Note that this paragraph departs from what was in this wiki for versions of the I-pipe patch until version 1.7-06. This because in past versions of the I-pipe a log was maintained for each interruption, which resulted in the interrupt handler being invoked as many times as the interrupt was triggered, so we did not need to account for lost ticks. Starting with I-pipe version 1.8, the log has been "flattened" which means that it now acts as a single bit, if an interrupt is triggered several times before its handler is invoked, the handler will be invoked only once, so we need again to account for lost tick in Linux tick handler.

So the typical pxa_timer_interrupt, which looks like:

pxa_timer_interrupt()
{
        /* (...) */
        do {
                timer_tick();
                OSSR = OSSR_M0;
                next_match = (OSMR0 += LATCH);
        while ((signed long) (next_match - OSCR) <= 8);
        /* (...) */
}

becomes:

static unsigned long last_jiffy_time;

void __ipipe_mach_acktimer(void)
{
        OSSR = OSSR_M0;  /* Clear match on timer 0 */
}

pxa_timer_interrupt()
{
        int next_match;
         /* (...) */
         do {
               timer_tick();
#ifndef CONFIG_IPIPE
               OSSR = OSSR_M0;  /* Clear match on timer 0 */
#else /* CONFIG_IPIPE */
               last_jiffy_time += LATCH;
               if (__ipipe_mach_timerstolen)
                       next_match = last_jiffy_time + LATCH;
               else
#endif /* CONFIG_IPIPE */
               next_match = (OSMR0 += LATCH);
         } while( (signed long)(next_match - OSCR) <= 8 );
         /* (...) */
}

Modifying struct sys_timer::offset

Another Linux function that the I-pipe patch should modify is the offset member of the sys_timer struct. Here, there are two cases:

• either the platform has a free-running counter, then adding LATCH to a

variable at each Linux timer handler invocation, and subtracting this variable from the free-running counter is enough to implement the offset function. If last_jiffy_time is maintained as in the code above, the offset function for PXA could be, for instance:

static unsigned long pxa_gettimeoffset (void)
{
        long elapsed, usec;

        elapsed = OSCR - last_jiffy_time;

        /* Now convert them to usec */
        usec = (unsigned long)(elapsed * (tick_nsec / 1000))/LATCH;

        return usec;
}

• or the only available hardware register is a decrementer, then the

implementation is a bit more complicated, have a look at the integrator implementation (file arch/arm/mach-integrator/core.c).

Implementing a high-resolution counter.

The I-pipe patch for ARM implements an emulation of a 64 bits high-resolution counter (the one called tsc on x86, we will hence call it tsc for short), using whatever hardware is proposed by the underlying machine. A port of the I-pipe patch for ARM is expected to define:

• __ipipe_mach_get_tsc() which emulates the tsc;
• __ipipe_mach_get_tscinfo() which is used to export the tsc to user-space.

Note that the tsc emulation can not work in user-space in the SMP case, because there is no fast (meaning without a system call) way to get the current processor id.

In order to allow sharing data between kernel and user-space with no cache effect, the Linux kernel uses the vectors page, the I-pipe patch defines an area of 20 bytes in this page to share tsc emulation data between kernel-space and user-space. The symbol __ipipe_tsc_area is a pointer to the beginning of this area.

Currently, there are two kinds of hardware counters which Xenomai can access from user-space in order to emulate a tsc.

Free-running counters

We will use the PXA as an implementation example in this section.

In this case, only the first 8 bytes of __ipipe_tsc_area are used to store a previous result of the tsc. A typical implementation is as follows:

• a pointer called tsc is initiliazed to __ipipe_tsc_area if not SMP or to a

static array if SMP:

union tsc_reg {
#ifdef __BIG_ENDIAN
	struct {
		unsigned long high;
		unsigned long low;
	};
#else /* __LITTLE_ENDIAN */
	struct {
		unsigned long low;
		unsigned long high;
	};
#endif /* __LITTLE_ENDIAN */
	unsigned long long full;
};

#ifdef CONFIG_SMP
static union tsc_reg tsc[NR_CPUS];

/* (...) */
#else /* !CONFIG_SMP */
static union tsc_reg *tsc;

/* (...) */
#endif

in pxa_timer_init:

#ifdef CONFIG_IPIPE
#ifndef CONFIG_SMP
	tsc = (union tsc_reg *) __ipipe_tsc_area;
	barrier();
#endif /* CONFIG_SMP */

	pxa_timer_initialized = 1;
#endif /* CONFIG_IPIPE */

• a function is expected to update the tsc area from time to time (at least once before the underlying counter reaches the same value again, in order to avoid losing some wrap arounds). If the free-running counter wraps fast, the best place to do this is __ipipe_mach_acktimer. If the free-running counter wraps slowly, doing this from Linux timer interrupt will reduce the timer interrupt latency. The PXA implementation wraps slowly, so uses Linux timer interrupt for this:

static void ipipe_mach_update_tsc(void);

 /* (...) */

static irqreturn_t
pxa_ost0_interrupt(int irq, void *dev_id)
{
	struct clock_event_device *c = dev_id;

	/* Disarm the compare/match, signal the event. */
#ifndef CONFIG_IPIPE
	OIER &= ~OIER_E0;
	OSSR = OSSR_M0;
#else /* CONFIG_IPIPE */
	ipipe_mach_update_tsc();
#endif /* CONFIG_IPIPE */
	c->event_handler(c);

	return IRQ_HANDLED;
}

 /* (...) */

static void ipipe_mach_update_tsc(void)
{
	union tsc_reg *local_tsc;
	unsigned long stamp, flags;

	local_irq_save_hw(flags);
	local_tsc = &tsc[ipipe_processor_id()];
	stamp = OSCR;
	if (unlikely(stamp < local_tsc->low))
		/* 32 bit counter wrapped, increment high word. */
		local_tsc->high++;
	local_tsc->low = stamp;
	local_irq_restore_hw(flags);
}

• the function __ipipe_mach_get_tsc reads the tsc area and free-running counter

to get the current tsc value:

notrace unsigned long long __ipipe_mach_get_tsc(void)
{
	if (likely(pxa_timer_initialized)) {
		union tsc_reg *local_tsc, result;
		unsigned long stamp;

		local_tsc = &tsc[ipipe_processor_id()];

		__asm__ ("ldmia %1, %M0\n"
			 : "=r"(result.full): "r"(local_tsc), "m"(*local_tsc));
		barrier();
		stamp = OSCR;
		if (unlikely(stamp < result.low))
			/* 32 bit counter wrapped, increment high word. */
			result.high++;
		result.low = stamp;

		return result.full;
	}

        return 0;
}

Note that we use ldmia to read the tsc area in order to avoid having to shut interrupts off.

Also note that the function has the "notrace" attribute, this because it is called by the I-pipe tracer code, so we would enter an infinite loop if it was traced. As such, this function is only permitted to call other non traced functions, in particular, if interrupts need to be disabled, the "notrace" variants of the interrupts disabling/enabling macros should be used.

• the function __ipipe_mach_get_tscinfo is defined as follows:

#ifdef CONFIG_SMP

/* (...) */
void __ipipe_mach_get_tscinfo(struct __ipipe_tscinfo *info)
{
	info->type = IPIPE_TSC_TYPE_NONE;
}
#else
/* (...) */
void __ipipe_mach_get_tscinfo(struct __ipipe_tscinfo *info)
{
	info->type = IPIPE_TSC_TYPE_FREERUNNING;
	info->u.fr.counter = (unsigned *) 0x40A00010;
	info->u.fr.mask = 0xffffffff;
	info->u.fr.tsc = &tsc->full;
}
#endif

info->type indicates that the tsc is based on a free-running counter,

info->u.fr.counter is set to the PHYSICAL address of the free-running counter,

info->u.fr.mask is a mask indicating which bits in the free-running counter are valid.

info->u.fr.tsc is a pointer to the shared tsc area

Decrementer

See s3c24xx implementation.

Adapting to boards using CONFIG_GENERIC_TIME and CONFIG_GENERIC_CLOCKEVENTS

Most of the rest of this document applies to boards using CONFIG_GENERIC_TIME and CONFIG_GENERIC_CLOCKEVENTS as well. One difference, however, is in the handling of tick interrupts.

Whereas in previous implementations, the interrupt handler did all the work, it now calls the clockevent device handler, which ends up calling the "set_next_event" callback registered in the clock_event_device structure, to program the timer for the next timer tick, if the system is in one-shot (aka dynamic tick, or tickless) mode, or if the system is in periodic mode, but there is no hardware support for a periodic timer.

There is no need to modify this set_next_event callback, since when Xenomai starts handling the timer, it will register its own callback instead (which uses a Xenomai software timer to trigger Linux timer interrupt).

There is no need either to care about lost ticks, since the Linux kernel already takes care about it in its generic code.

Another difference is the way the __ipipe_mach_release_timer() function is implemented. The implementation should call the "set_mode" and "set_next_event" callbacks of the clock_event_device structure to restore the timer subsystem to a known state. The typical PXA implementation looks like:

void __ipipe_mach_release_timer(void)
{
        pxa_osmr0_set_mode(ckevt_pxa_osmr0.mode, &ckevt_pxa_osmr0);
        if (ckevt_pxa_osmr0.mode == CLOCK_EVT_MODE_ONESHOT)
                pxa_osmr0_set_next_event(LATCH, &ckevt_pxa_osmr0);
}

Modifying Xenomai headers to add a new board

Most of the work needed to get Xenomai to run on a new ARM board takes place in the I-pipe patch, however, some minimal modifications need to be made to Xenomai's include/asm-arm/hal.h.

Two macros need to be provided:

• RTHAL_TIMER_DEVICE, which is a string describing the timer device used by Xenomai, if the board does not support CONFIG_GENERIC_TIME, it can be anything, but will be displayed in /proc/xenomai/timer, if the board uses CONFIG_GENERIC_TIME, it must be the same as the board clock_event_device structure name field;
• RTHAL_CLOCK_DEVICE, which is a string describing the clock device used by Xenomai, it may be anything, but will be displayed in /proc/xenomai/timer, and if the board uses CONFIG_GENERIC_CLOCKSOURCES, it make sense to use the same as the board clocksource structure name field.

Tips and tricks.

• When the screen remains blank: activate Kernel low-level debugging functions in

the kernel hacking menu, then add a call to printascii(printk_buf) rigth after the call to vscnprintf(printk_buf, ...) in function vprintk, file kernel/printk.c

• If the kernel locks at "Calibrating delay loop...", it means that the timer

interrupt is not ticking and that the delay calibration routine is running an infinite loop at while (ticks == jiffies) in function calibrate_delay, file init/calibrate.c. To help debugging this situation, you can put printks in the while (ticks == jiffies) loop, printing any hardware timer registers or status register you want.

• If the kernel crashes when I-pipe tracer is enabled, and the function where it

crashes is marked with the "naked" attribute, you should also add the notrace attribute to this function, "naked" functions can not be instrumented.

• If Xenomai seems to run for a while, then locks up. Sometimes, there are minimum

delays below which the hardware timer can not be reprogrammed. For instance the PXA timer hardware can not be programmed for delays shorter than 8 ticks, hence __ipipe_mach_set_dec:

void __ipipe_mach_set_dec(unsigned long delay)
{
        if (delay > 8) {
                unsigned long flags;

                local_irq_save_hw(flags);
                OSMR0 = delay + OSCR;
                local_irq_restore_hw(flags);
        } else
                ipipe_trigger_irq(IRQ_OST0);
}