Prepping device drivers for Real-Time LinuxApr 25, 2018 — by Eric Brown
- Those electronics use Windows System drivers and installation is seamless on most systems. The driver process loads three parts. The problem comes if your system fails to install CDC-ACM. This is not unique to the Kenwood radio. Other devices suffer from this same problem since the problem lies in the system files on your machine.
- Download Retrieval for RT Systems software. This section is not for purchasing the RT Systems Programmers. If you have not yet purchased the Programmer for your radio, select the manufacturer of your radio on the left, then follow the prompts to find the RT Systems Programmer for your specific radio model.
USB cDAQ devices (except the cDAQ-9172 and cDAQ-9179) are supported on PharLap Real-Time systems with DAQmx 9.6 or later and LabVIEW Real-Time 2012 or later. The cDAQ-9179 USB 3.0 chassis is an exception. It is not supported on PharLap Real-Time systems. For information about how to use the cDAQ-9179 on a Linux RT system, refer to cDAQ-9179 USB.— 1756 views
At Embedded Linux Conference 2018, NI’s Julia Cartwright told device driver authors how to ensure their drivers will run without a hitch on Real-Time Linux.
Although Real-Time Linux (RT Linux) has been a staple at Embedded Linux Conferences for years — here’s a story on the RT presentations in 2007 — many developers have viewed the technology to be peripheral to their own embedded projects. Yet as RT, enabled via the PREEMPT_RT patch, prepares to be fully integrated into the mainline kernel, a wider circle of developers should pay attention. In particular, Linux device driver authors will need to ensure that their drivers play nice with RT-enabled kernels.
At the recent Embedded Linux Conference in Portland, National Instruments software engineer Julia Cartwright, an acting maintainer on a stable release of the RT patch, gave a well-attended presentation called “What Every Driver Developer Should Know about RT.” Cartwright started with an overview of RT, which helps provide guarantees for user task execution for embedded applications that require a high level of determinism. She then described the classes of driver-related problems that can have a detrimental impact to RT, as well as potential resolutions.
Julia Cartwright explains how RT Linux uses shims to tamp down irq_dispatch latency
(click image to enlarge)
One of the challenges of any real-time operating system is that most target applications have two types of tasks: those with real-time requirements and latency sensitivity, and those for non-time critical tasks such as disk monitoring, throughput, or I/O. “The two classes of tasks need to run together and maybe communicate with one another with mixed criticality,” explained Cartwright. “You must resolve two different degrees of time sensitivity.”
One solution is to split the tasks by using two different hardware platforms. “You could have an Arm Cortex-R, FPGA, or PLD based board for super time-critical stuff, and then a Cortex-A series board with Linux,” said Cartwright. “This offers the best isolation, but it raises the per unit costs, and it’s hard to communicate between the domains.”
Another approach is to use the virtualization approach provided by Xenomai, the other major Linux-based solution aside from PREEMPT_RT. “Xenomai follows a hypervisor, co-kernel approach using a hypervisor or AMP solution to separate an RTOS from Linux,” said Cartwright. “However, there’s still a tradeoff in limited communications between the two systems.”
RT Linux’s PREEMPT_RT, meanwhile, enables the two systems to share a kernel, scheduler, device stack, and subsystems. In addition, Linux IPC mechanisms can be used to communicate between the two. “There’s not as much isolation, but much greater communication and usability,” said Cartwright.
One challenge with RT is that “because the drivers are shared between the real-time and non real-time systems, they can misbehave,” said Cartwright. “A lot of the bugs we’re finding in RT come from device drivers.”
In RT, the time between when an event occurs – such as a timer firing an interrupt or an I/O device requesting service – and the time when the real-time task executes is called the delta. “RT systems try to characterize and bound this in some meaningful way,” explained Cartwright. “A cylic test takes a time stamp, then sleeps for a set time such as 10ms, and then takes a time stamp when the thread wakes up. The difference between the time stamps, which is the amount of time the thread slept, is called the delta.”
This delta can be broken down into two phases. The first is the irq_dispatch, which is the time it takes between the hardware firing and the dispatch occurring up until the time the thread scheduler is instructed that the thread needs to run. The second phase is scheduling latency, the time between when the scheduler has been made aware that a high priority task needs to run to the moment when the CPU is given the task to execute.
When using mainline Linux without RT extensions, irq_dispatch latency can be considerable. “Say you have one thread executing in user mode, and an external interrupt such as a network event fires that you don’t care about in your real time app,” said Cartwright. “But the CPU is going to vector off to into hard interrupt context and start executing the handler associated with that network device. If during that interrupt handler duration, a high priority event fires, it’s not able to be scheduled on the CPU until the low priority interrupt is done executing.”
Drivers Rt Systems Port Devices List
The delta between the internal event firing and the external event “is a direct contributor to irq_dispatch latency,” said Cartwright. “Without an RT patch it would be a mess to define bounds on this because the bound would be the bound of the longest running interrupt handler in the system.”
RT avoids this latency by forcing irq threads. “There’s very little code that we execute in a hard interrupt context – just little shims that wake up the threads that are going to execute your handler,” said Cartwright. “You may have a low priority task running, and perhaps also a medium priority task that the irq fires, but only a small portion of time is spent waking up the associated handlers for the threads.”
RT also provides other guarantees. For example, because interrupt handlers are now running in a thread, it can be preempted. “If a high priority, real-time critical interrupt fires, that thread can be scheduled immediately, which reduces the irq_dispatch latency,” said Cartwright.
Cartwright said that most drivers require no modification to participate in forced irq threading. In fact, “Thread irq actually exists in mainline now. You can boot a kernel and pass the thread irq parameter and it will thread all your interrupts. RT will add a forced enablement.”
Yet there are a few cases when this causes problems. “If your drivers are invoked in the process of delivering an interrupt dispatch, you can’t be threaded,” said Cartwright. “This can happen with irqchip implementations, which should not be threaded. Another issue may arise if the driver is invoked by the scheduler.”
Other glitches can emerge when “explicitly disabling interrupts using local_irq_disable or local_irq_save. Cartwright recommended against using such commands in drivers. As an alternative to local_irq_disable, Cartwright suggested using spinlocks or local locks, a new feature that will soon be proposed for mainline. In a separate presentation at ELC 2018, called Maintaining a Real Time Stable Kernel, Linux kernel developer Steven Rostedt goes into greater depth on local locks.
Cartwright finished up her discussion of irq_dispatch latency issues by discussing some rare, hardware related MMIO issues that can occur. Once when she accidentally pulled the Ethernet cable during testing, it caused buffering in the interconnect, which screwed up the interrupts and was a pain to fix. “To solve it we had to follow each write by a readback, which prevents write stacking,” she said. The ultimate solution? “Take the drugs away from the hardware people.”
There appear to be fewer driver problems related to the second latency phase – scheduling – which is the time from when you execute a real-time thread to the time when the irq thread is scheduled. One example stems from the use of preempt_disable, when prevents a higher priority thread to be scheduled upon return from interrupt because preemption has been disabled.
“The only reason for a device driver to use preempt_disable is if you need to synchronize with the act of scheduling itself, which can happen with cpufreq and cpuidle,” said Cartwright. “Use local locks instead.”
On mainline Linux, spinlock-protected critical sections are implicitly executed with preemption disabled, which can similarly lead to latency problems. “With RT we solve this by making spinlock preemptible in critical sections,” said Cartwright. “We turn them into pi-aware mutexes and disable migration. When a spinlock is held by a thread it can be preempted by a higher priority thread to bring in the outer bound.”
Most drivers require no changes in order to have their spinlock critical sections preemptible, said Cartwright. However, if a driver is involved in interrupt dispatch or scheduling, they must use raw_spin_lock(), and all critical sections must be made minimal and bounded.
More information may be found in Cartwright’s RT slide show (PDF) and YouTube video.
This article is copyright © 2018 Linux.com and was originally published here. It has been reproduced by this site with the permission of its owner. Please visit Linux.com for up-to-date news and articles about Linux and open source.
LinuxGizmos related posts:
The RT difference is in our relationship and service level with its customers. RT will adapt our software to meet your unique requirements. RT listens to our customers’ needs and turns around requests quickly.
Warehouse Management System
Fully Functional Real-Time WMS
- EDI, Purchase Orders, Bills of Ladings, Advance Shipment Notices
- Store/Put Away
- Dynamic store rules available based upon velocity, home locations, customer-specified business rules
- Cycle Counting
- Physical Inventory
- Unlimited Units of Measure
Drivers Rt Systems Port Devices Free
- Picking Orders
- Single Orders, Batch/Wave
- Reverse Delivery Stop Picking
- Route Release
- UPS/Fed Ex – Ship Order Type
- Will Calls/Pick Ups
- Pick from Methodology Examples
- Least Number of Picks
- Pick to Clear Location
- Bulk Picking
- Load Truck/Verify Shipments
- Shipping Documents and or Electronic Shipment Notifications
- Multiple Cross References – UPCs, EANs, Manufacturer Part Number
- Cross Docking
- Assembly/Kitting/Bills of Materials
- Production Recording and Tracking
- Repack and or Packaging
- Billing for 3PL’s and Public Warehouses
- Warehouse Set Up and Configuration
- Zones, Work Areas, Aisles, Locations, Pick Path(s)
- Labor Management and Tracking
- Reports – Numerous Standard Reports and Custom Reports Easily Created
- Secure – Role Based Security Set Up for Users
RTLocator is Available in the Cloud – Low Entry Cost Model & 24/7/365 Support
Delivery Management System Real-Time App
- Drivers Sign out on Mobile Device for Delivery ‘Trip’
- Drivers can Take Pictures of Items Delivered
- Drivers Capture Electronic Signatures from Customer
- DMS can Email Receipts and or Invoices to Customer
- DMS Stores Documents and Signatures for Easy Retrieval when Needed
- Customer Service can Monitor Drivers ETA’s and History for each Active Trip
- Customer Service can Schedule Pick Ups for Returns and Alert Drivers
- Customer Service can Schedule Transfers from Location to Location
- DMS Stores Data Locally in Case of Poor Cell Coverage
- Mobile Devices can be Apple IOS (iPhone) or Android IOS(Droids)
- RT Delivery App can be downloaded at the Apple Store or Google Play
DMS is Available in the Cloud – Low Entry Cost Model & 24/7/365 Support
- Flexible What If Scenarios – Infinite
- Historical Velocity (User Selected Date Ranges), Predicted Future Velocity, New Products Added
- MRP Calculations and Inquiries
- When do I run out of stock?
- Supplier Schedules
- Purchase Order Recommendation and or Creation
- Reorder Reports – Based Upon Min/Max, Lead Times, Minimum Buys
- Multiple Sequencing Methods
- Real-Time Tracking – Finished Goods Bank vs. Customer Usage
- Alerts for Errors and Exceptions
- Monitoring of Inbound Customer Orders – Ensure 100% Uptime and Connectivity
- Integrates with RTLocator WMS – Tracking of Inventory from Receipt to Shipment
- Integrates with GM Broadcast, FCA Fiat Chrysler Automotive Broadcast, Ford Broadcast, and Ford 866 Sequence Schedules
- Can be purchased as a Stand-Alone Module
- 20+ Years of Best Practices Included
- 24/7/365 Support Available