1. Introduction --------------- This is an attempt to solve the race conditions while module unloading in Linux and is a demostration of Read-Copy-Update mechanism, a special two phase mutual exclusion method where readers (threads trying to access but not modifying the data) can access the shared data without acquiring any conventional lock. The writers (threads that update the data) however, have to use a special callback scheme to update the data. Have a look at more details on Read-Copy-Update and a patch from http://lse.sourceforge.net/locking/rclock.html In this version I have tried to keep the changes simpler and lesser. The changes are done in include/linux/module.h, kernel/ksyms.c and kernel/module.c files. The solution enhances the existing module unloading logic to a two phase cleanup model. The first phase as already exists, involves indicating the new users of the module's code that module is no longer present, by marking the MOD_DELETED flag. The next step in phase 1 is to call the module's cleanup() routine. In the cleanup routine the module registers a read-copy-update callback handler to free up the module's internal data structures and the module code. In phase 2 the callback handler gets called and it invokes a function in the module code to free any dynamically allocated memory for module's internal data structures. Then it unmaps module code and freeis the callback structure. 2. How to use -------------- For using this solution the module writer has to do the following changes in the module code. - in the cleanup routine after doing the usual stuff like free_irq or unregister_xxx etc., invoke a new kernel-API kmod_def_cleanup() in kernel/module.c). This api will register a callback handler on behalf of the module. It has two parameters, first a pointer to module structure (THIS_MODULE) and second is the pointer to module_data_destructor_t (include/linux/module.h). void kmod_def_cleanup(struct module *, module_data_destructor_t *); The structure module_data_destructor has a pointer to a function that frees any internal data structures that need to be freed at the time of cleanup and a pointer to the data to be freed. - The module has to do MOD_INC_USE_COUNT before calling any blocking routine and do MOD_DEC_USE_COUNT after the blocking call returns. - The usecount should be incremented in the open function by try_inc_mod_count() and decremented in the release funsiton if the module has an open-read/write-release format. 3. How to install ------------------ The patch is based on 2.4.1 kernel and can be installed as follows: - On a clean 2.4.1 tree apply the read-copy-update patch from http://lse.sourceforge.net/locking/rclock-2.4.1-01.patch - Apply module_unloading-2.4.1-01.patch. - Enable the CONFIG_EXPERIMENTAL, CONFIG_MODULES, CONFIG_RCLOCK and CONFIG_CLEANUP_RCU during configuration. - As existing modules do not use this logic do _not_ build any thing as modules excpet the test modules. Though the existing modules can be changed easily as described above and can use this logic for cleanup. - Copy the test modules sources (mod*) and Makefile in the kernel source tree (drivers/test/), so as to use the kernel compile flags and make options. They are compiled separately from /usr/src/linux directory as make drivers/test/mod.o make drivers/test/moda.o make drivers/test/modb.o - Compile the test applications (test2app.c and test3app.c). 4. How to test -------------- For testing the module cleanup you can run the following tests preferably on an SMP box in multiple instances so as to observe any race conditions. If ran in multiple instances errors will be reported from insmod(module already exists) or from rmmod(module is not loaded) as they should. But I hope you will not see any panic. "printk" messages can be seen coming from init_module and cleanup routines as and when module is inserted or removed respectively. "lsmod" can be used to verify whether a module has been removed or not. - test1.sh -------- This test just simply inserts and removes a simple module mod.o in a loop. - test2.sh -------- This test inserts the module moda.o and runs an application program which opens the char device associated with the module and does read-write operations on the device. For running this test first make a node for the char device as # mknod moda c 209 1 # ./test2.sh - test3.sh -------- This test runs a bit complex scenario in which one module interacts with another. First it inserts moda.o and then modb.o. Module "b" uses data and code from module "a". Then, the test runs an application program which does read/write operations on both the modules. Make one more node apart for the moda as done in test2.sh. # mknod moda c 209 1 # mknod modb c 210 1 # ./test2.sh 5. License ---------- Copyright (C) 2001 IBM Corporation This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. 6. Credits ---------- 1. Paul McKenney - originator of Read-Copy-Update Mechanism 2. Andi Kleen - gave us the idea to use this mechanism for module unloading and also gave suggestions for the implementation 3. Andrew Morton - for important suggestions 4. Rusty Russell - for important suggestions 5. Keith Owens - for explaining module locking rules 6. Dipankar Sarma - for clearing doubts in using rclock interfaces 7. Contact ---------- For any problems and questions, contact Maneesh Soni (smaneesh@in.ibm.com)