Over the last two decades plus, I have used multiple queueing modules in perl. Some of them are:
- Thread::Queue
- IPC::Lite
- Hand rolled on top of BerkeleyDB::Queue or LMDB_File
- Forks::Queue - most recently
I'm sure that there were others, but the are the ones that come to mind.
I am currently using Forks::Queue with a SQLite back-end in a personal application that runs in two separate processes. The first is a server that pulls URLs from the queue and downloads them using yt-dlp. The second is a client that grabs URLs from the clipboard and places them in the queue. Both processes run on the same Debian 12 instance. The two characteristics of queueing that led me to select Forks::Queue were: 1. works across processes, 2. persistent over stop/start.
In general, Forks::Queue has worked for me. In the last month or so, I have observed an annoying behavior. Maybe it existed before and I didn't remember it or maybe it is due to a few changes that I have made to add an additional capability to the application. When I first start the server, it works fine until the client loads the first entry in the queue. The server crashes when reading the queue with "I/O Possible" display on the screen. When I restart the server, it then reads the entry and processes it without problems. Subsequent entries are also processed without problems.
Through logging, I have able to localize the failure to the dequeue-nb()
call that reads from the queue. Enables Forks::Queue debugging with the environment variable FORKS_QUEUE_DEBUG
also does not reveal anything. Neither eval nor the new feature 'try' will catch the error. Searches in Google, none of them related to perl, suggest that the problem is somewhere in the bowels of the OS's I/O routines.
For a one off personal project, I can obviously live with this; however, everytime that I encounter it is grates on me.
As such, I am requesting recommendations from your experiences for alternatives to Forks::Queue.
The requirements are:
- Supports general queueing methods (a la Thread::Queue like API)
- Works across processes
- Persist over stops and starts of processes
While not pertinent to my immediate needs, I would like for it to be fairly fast. The current application has no need for speed but uses in the future could. Additionally, it would be nice if the module handled serialization and deserialization of arrays, hashes and blessed objects but this can easily be accomplished with a wrapper function.
Thanks in advance for your help! lbe
submitted by /u/LearnedByError
[link] [comments]