My job has led me down the rabbit hole of doing some scripting work in Perl, mainly utility tools. The challenge being that these tools need to parse several thousand source files, and doing so would take quite some time.
I initially dabbled in doing very light stuff with a perl -e
one-liner from within a shell script, which meant I could use xargs. However, as my parsing needs evolved on the Perl side of things, I ended up switching to an actual Perl file, which hindered my ability to do parallel processing as our VMs did not have the Perl interpreter built with threads support. In addition, installation of any non-builtin modules such as CPAN was not possible on my target system, so I had limited possibilities, some of which I would assume to be safer and/or less quirky than this.
So then I came up with a rather ugly solution which involved invoking xargs via backticks, which then called a perl one-liner (again) for doing the more computation-heavy parts, xargs splitting the array to process into argument batches for each mini-program to process. It looked like this thus far:
my $out = `echo "$str_in" | xargs -P $num_threads -n $chunk_size perl -e ' my \@args = \@ARGV; foreach my \$arg (\@args) { for my \$idx (1 .. 100000) { my \$var = \$idx; } print "\$arg\n"; } '`;
However, this had some drawbacks:
- No editor syntax highlighting (in my case, VSCode), since the inline program is a string.
- All variables within the inline program had to be escaped so as not to be interpolated themselves, which hindered readability quite a bit.
- Every time you would want to use this technique in different parts of the code, you'd have to copy-paste the entire shell command together with the mini-program, even if that very logic was somewhere else in your code.
After some playing around, I've come to a nifty almost-metaprogramming solution, which isn't perfect still, but fits my needs decently well:
sub processing_fct { my u/args = u/ARGV; foreach my $arg (@args) { for my $idx (1 .. 100000) { my $var = $idx; } print "A very extraordinarily long string that contains $arg words and beyond\n"; } } sub parallel_invoke { use POSIX qw{ceil}; my $src_file = $0; my $fct_name = shift; my $input_arg_array = shift; my $n_threads = shift; my $str_in = join("\n", @{$input_arg_array}); my $chunk_size = ceil(@{$input_arg_array} / $n_threads); open(my $src_fh, "<", $src_file) or die("parallel_invoke(): Unable to open source file"); my $src_content = do { local $/; <$src_fh> }; my $fct_body = ($src_content =~ /sub\s+$fct_name\s*({((?:[^}{]*(?1)?)*+)})/m)[1] or die("Unable to find function $fct_name in source file"); return `echo '$str_in' | xargs -P $n_threads -n $chunk_size perl -e '$fct_body'`; } my $out = parallel_invoke("processing_fct", \@array, $num_threads);
All parallel_invoke() does is open it's own source file, finds the subroutine declaration, and then passes the function body captured by the regex (which isn't too pretty, but it was necessary to reliably match a balanced construct of nested brackets) - to the xargs perl call.
My limited benchmarking has found this to be as fast if not faster than the perl-with-threads equivalent, in addition to circumventing the performance penalty for the thread safety.
I'd be curious to hear of your opinion of such method, or if you've solved a similar issue differently.
submitted by /u/Wynaan
[link] [comments]
submitted by /u/perlancar [link] [comments] |
List of new CPAN distributions – Jun 2024
dist author abstract date Alien-RtAudio JBARRETT Install RtAudio 2024-06-23T15:44:22 Alien-SunVox JBARRETT Install The SunVox Library – Alexander Zolotov's SunVox modular synthesizer and…perlancar's blog
I need some help with the old Perl Gunnar Hjalmarsson's Ringlink program on my site. The forms work, the database gets added to and everything seems ready to go except for the email functions that depend on sendmail.
I have tried several things, installed the CPAN dependencies the program needs, tried Auron SendEmail and other programs and have thoroughly confused myself.
There's a test installation on my site, with the admin and password are both 'test'. There are copies of the CGI files and probably what needs looking at are rlmain.pm, rlconfig.pm and sender.pm
I am running Apache 2.4.54 on Windows 10 with Strawberry Perl installed. I am using the last published version Ringlink (v3.4)
I know this is an old program and the project probably not worth pursuing, but I really would like to give this a go to get it working and would be grateful for any suggestions.
submitted by /u/brisray
[link] [comments]
Auron SendEmail - Auron Software Portable E-mail Freeware
The Auron SendEmail set of portable freeware tools for Windows enable you to send E-mails through from either GUI or commandline.Auron Software
[link] [comments]
(dii) 7 great CPAN modules released last week
Updates for great CPAN modules released last week. A module is considered great if its favorites count is greater or equal than 12. App...niceperl.blogspot.com
[link] [comments]
(cdxcvii) 8 great CPAN modules released last week
Updates for great CPAN modules released last week. A module is considered great if its favorites count is greater or equal than 12. App...niceperl.blogspot.com
Hi, I recently got an offer for Senior SWE (current title at my company now) for a company that heavily utilizes Perl. I was wondering if folks from this community could offer some insight on what it's like working with Perl and also what, if any, potential long-term career implications are of becoming a Perl developer? Particularly I'm worried of pigeon-holing myself since Perl is not as heavily used in todays age and this company does not make use of modern cloud tools and deployments.
I am a Java developer (5 YOE) at a enterprise software company that is deployed in GCP. We are pretty regularly adopting new technologies so I'm gaining some valuable and relevant industry experience here but I am looking for a change and more opportunity to lead projects and mentor junior engineers.
The company seems good, great WLB, I liked the manager, and with the bonus (base is roughly the same) it would be about a ~8% TC increase plus a lot more stock (monopoly money, private RSUs).
Does anyone have experience transitioning from a Perl based company to a cloud based company with a more modern tech stack? Is this a backwards direction for me, should I continue with my Java development and instead look for opportunities that will offer more marketable skills?
Any input is appreciated, thank you for reading.
submitted by /u/Roodiestue
[link] [comments]
I've been trying to debug an issue for 3 days now, am getting nowhere, and am about to headbutt my laptop. If anyone's done any heavy lifting with Net:DBus then, for the sake of my laptop, I'd really appreciate the help!
Problem description: I have a hash table with a bunch of keys. The values relating to those keys are of different types (as in, I've cast them to dbus types). So:
my $testhash = {}; $testhash->{"xesam:albumArtist"} = [dbus_string("Tom Waits")]; $testhash->{"xesam:album"} = dbus_string("Mule Variations"); $testhash->{"xesam:trackNumber"} = dbus_int32(1); $testhash->{"xesam:artist"} = [dbus_string("Tom Waits")]; $testhash->{"xesam:title"} = dbus_string("Big in Japan"); $testhash->{"mpris:artUrl"} = dbus_string("file://mnt/storage/Music/mp3/Tom Waits/Mule Variations/folder.jpg"); $testhash->{"mpris:length"} = dbus_int64(64182857); $testhash->{"mpris:trackid"} = dbus_object_path("/0"); $testhash->{"xesam:url"} = dbus_string("file://mnt/storage/Music/mp3/Tom Waits/Mule Variations/01 - Big in Japan.mp3");
I've created a DBus service, and have successfully implemented a method that returns that hash table ($IFACE is the interface name I'm using for all of my test methods):
dbus_method("ReturnDynamicHash", [], [["dict", "string", ["variant"]]], $IFACE); sub ReturnDynamicHash { my $self = shift; print "Object: ReturnDynamicHash called.\n"; my $return = {}; my @keys = keys(%{$testhash}); my $count = scalar(@keys); if ($count) { foreach my $key (@keys) { $return->{$key} = $testhash->{$key}; } } return $return; }
As a DBus method, this works perfectly:
% dbus-send ....... .ReturnDynamicHash array [ dict entry( xesam:trackNumber variant int32 1 ) dict entry( mpris:trackid variant /0 ) dict entry( xesam:albumArtist variant array [ Tom Waits ] ) dict entry( xesam:album variant Mule Variations ) dict entry( mpris:length variant int64 64182857 ) dict entry( xesam:url variant file://mnt/storage/Music/mp3/Tom Waits/Mule Variations/01 - Big in Japan.mp3 ) dict entry( mpris:artUrl variant file://mnt/storage/Music/mp3/Tom Waits/Mule Variations/folder.jpg ) dict entry( xesam:artist variant array [ Tom Waits ] ) dict entry( xesam:title variant Big in Japan ) ]
However, the interface I'm implementing requires that a DBus Property return that hashtable, not a method:
dbus_property("StaticHashProperty", [["dict", "string", ["variant"]]], "read", $IFACE); sub StaticHashProperty { print "Object: StaticHashProperty accessed.\n"; my $return = {}; my @keys = keys(%{$testhash}); my $count = scalar(@keys); if ($count) { foreach my $key (@keys) { $return->{$key} = $testhash->{$key}; } } return $return; }
and this doesn't work.
From the dbus-send client I get
Error org.freedesktop.DBus.Error.NoReply: Remote peer disconnected
and from the Perl server stderr i get:
dbus[93409]: Array or variant type requires that type array be written, but object_path was written. The overall signature expected here was 'a{sas}' and we are on byte 4 of that signature. D-Bus not built with -rdynamic so unable to print a backtrace Aborted (core dumped)
Now, this error is coming from libdbus itself, not the Perl wrapper (though could of course still be a bug in the Perl module that's causing the error). It seems to have entirely the wrong signature ( a{sas}, not a{sv} as defined above the Property method) and therefore appears to be complaining that the type of one of the values is wrong (each time I run it I get a slightly different error; I think it's deducing the signature from the first key-value pair it pulls from the hash and assumes they should all be the same - so if the first pair it pulls has a uint64 value, then it complains that the next pair doesn't also have a uint64 value).
Since the Method works I know Net::DBus can handle these sorts of return values, but for some reason, as a property, it just isn't working. I also know that other applications do implement this interface, including this Property, successfully, so I know this isn't a limitation of DBus.
I've been looking at the code in Net::DBus that handles serialization, assuming there must be some difference between how Properties and Methods are handled, but can't see anything obvious.
Anyone? Any idea? Literally anything at all? Thank you!!!!!
submitted by /u/Flashy_Boot
[link] [comments]
The following is a quick ramble before I get into client work, but might give you an idea of how AI is being used today in companies. If you have an questions about Generative AI, let me know!
The work to make the OpenAI API (built on Nelson Ferraz's OpenAPI::Client::OpenAI module) is going well. I now have working example of transcribing audio using OpenAI's whisper-1 model, thanks to the help of Rabbi Veesh.
Using a 7.7M file which is about 16 minutes long, the API call takes about 45 seconds to run and costs $0.10 USD to transcribe. The resulting output has 2,702 words and seems accurate.
Next step is using an "instruct" model to appropriately summarize the results ("appropriate" varies wildly across use cases). Fortunately, we already have working examples of this. Instruct models tend to be more correct in their output than chat models, assuming you have a well-written prompt. Anecdotally, they may have smaller context windows because they're not about remembering a long conversation, but I can't prove that.
Think about the ROI on this. The transcription and final output will cost about 11 cents and take a couple of minutes. You'll still need someone to review it. However, think of the relatively thankless task of taking meeting minutes and producing a BLUF email for the company. Hours of expensive human time become minutes of cheap AI time. Multiply this one task by the number of times per year you have to do it. Further, consider how many other "simple tasks" can be augmented via AI and you'll see why it's becoming so powerful. A number of studies show that removing many of these simple tasks from people's plates, allowing them to focus on the "big picture," is resulting in greater morale and productivity.
When building AI apps, OpenAPI::Client::OpenAI should be thought of as a "low-level" module, similar to DBIx::Class. It should not be used directly in your code, but hidden behind an abstraction layer. Do not use it directly.
I tell my clients that their initial work with AI should be a tactical "top-down mandate/bottom-up implementation." This gives them the ability to start learning how AI can be used in different parts of their organization, given that marketing, HR, IT, and other departments all have different needs.
Part of this tactical approach is learning how to build AI data pipelines. With OpenAI publishing their OpenAPI spec, and with Perl using that, we can bring much of the power of enterprise-level AI needs to companies using Perl. It's been far too long that Perl has languished in the AI space.
Next, I need to investigate doing this with Gemini and/or Claude, but not now.
Note, if you're not familiar with the BLUF format, it's a style of writing email that is well-suited for email in a company that is sent to many people. It's "bottom-line up front" so that people can see the main point and decide if the rest of the email is relevant to them. It makes for very effiicient email communication.
submitted by /u/OvidPerl
[link] [comments]
BLUF: THE method for better and effective written communication
Communication in remote work is all about written communication. That’s the big shift that happened regarding how we communicate. The Economist and Information Week are two examples of many…Gabriela De Luca (Medium)
Mo utilities for email.
Changes for 0.02 - 2024-04-26T23:02:53+02:00
- Add tests for error parameters.
- Rewrite the tests so that the functional tests are first and then the errors.
Hi! Asking for a wisdom here...
We have a module that modifies signal handler $SIG{__DIE__} to log information and to die afterwards. Hundreds of scripts relied on this module which worked fine in perl 5.10.1.
Recently we had the opportunity to install several Perl versions but unfortunately a large number of scripts that used to work with Perl 5.10.1 now behave differently:
- Failed in 5.14.4:
/home/dev/perl-5.14.4/bin/perl -wc
test.pl
RECEIVED SIGNAL - S_IFFIFO is not a valid Fcntl macro at /home/dev/perl-5.14.4/lib/5.14.4/File/stat.pm line 41
- Worked without changes in 5.26.3:
/home/dev/perl-5.26.3/bin/perl -wc
test.pl
test.pl
syntax OK
- Worked without changes in 5.38.2:
/home/dev/perl-5.38.2/bin/perl -wc test.pl
test.pl syntax OK
Many of the scripts can only be updated to 5.14.4 due to the huge jumps between 5.10 and 3.58; But we are stuck on that failures.
Was there an internal Perl change in 5.14 which cause the failures but works on other recent versions without any update on the scripts?
Cheerio!
submitted by /u/Longjumping_Army_525
[link] [comments]
Sanity-check calling context
Changes for 0.04
- (no code changes)
- Switched to MIT license.
- Switched README from POD to Markdown.
- Removed Travis CI.
Sort lines of text by a SortKey module
Changes for 0.001 - 2024-03-07
- First release.
Sort lines of text by a Comparer module
Changes for 0.002 - 2024-03-07
- No functional changes.
- [doc] Mention some related links.
An assortment of date-/time-related CLI utilities
Changes for 0.128 - 2024-03-07
- [clis strftime, strftimeq] Use localtime() instead of gmtime(). We can still show UTC using "TZ=UTC strftime ...".
I understand that many disagree with this statement, but it really makes it easier to build distributions for people who not monks. Wish the documentation was more detailed
submitted by /u/ReplacementSlight413
[link] [comments]
Sah schemas related to BCA (Bank Central Asia) bank
Changes for 0.002 - 2024-04-03
- Rename module/dist Sah-Schema{s,Bundle}-* following rename of Sah-Schema{s,Bundle} (for visual clarity and consistency with naming of other bundles).
search nested hashref/arrayref structures using JSONPath
Changes for 1.0.5 - 2024-04-22T16:10:46-05:00
simulating paper and pencil techniques for basic arithmetic operations
Changes for 0.01 - 2024-04
- First version, with the four basic operations, plus square-root, GCD and radix conversion. And HTML rendering
Use a type to validate values in a deep comparison.
Changes for 1.0.1 - 2024-04-22
- Add Test2::Tools::Type
An open source web-based network management tool.
Changes for 2.076000 - 2024-04-22
- NEW FEATURES
- ENHANCEMENTS
- BUG FIXES
Silverpeak Orchestrator REST API client library
Changes for 0.011000 - 2024-04-22T17:58:39+02:00
- add support for version 9.3+ API endpoints
A module that performs semantic similarity in PXF/BFF data structures and beyond (JSON|YAML)
Changes for 0.07 - 2024-04-22T00:00:00Z
- Excluded keys with {} or [] values (e.g. subject.vitalStatus: {})
- Precompiled regex patterns where possible
- Schema validation errors related to weights are now directed to STDERR
- Added support for variables with non-word characters (e.g., "Survey.Timestamp.1")
submitted by /u/leejo [link] [comments] |
London Perl & Raku Workshop, 2024
London Perl & Raku Workshop. Perl & Raku Conference and training in London.act.yapc.eu
Pango style markup formatting
Changes for 0.036 - 2024-04-22
- Introducing struts. Update SYNOPOSIS example to modern PDF::API2 conventions. Fix/enhance drawing of brackgrounds. Fix problem with wrongly calculated underlining.
My environment is perl/5.18.2 on CentOs 7
I'm trying to use a SWIG generated module in perl, which has a c plus plus backend. The backend.cpp sets an environment variable, $MY_ENV_VAR =1
But when I try to access this in perl, using $ENV{MY_ENV_VAR} this is undef.
However doing something like print echo $MY_ENV_VAR
works
So the variable is set in the process, but it's not reflected automatically since nothing updates the $ENV data structure.
I'm assuming it may work using some getEnv like mechanism, but is there a way to reset/ refresh the $ENV that it rebuilds itself from the current environment?
submitted by /u/sarcasmwala
[link] [comments]
By default cpanm drops Perl modules into ~/perl5
. How do I tell cpanm to use a different location, such as ~/.local/share/perl5
instead?
submitted by /u/s-ro_mojosa
[link] [comments]
Get stock and mutual fund quotes from various exchanges
Changes for 1.61_01 - 2024-04-21T17:50:16-07:00
- YahooWeb.pm - Issue #377. Modified YahooWeb to account for changes from Yahoo.
Sah schemas related to SortKey
Changes for 0.002 - 2024-03-07
- Rename module/dist Sah-Schema{s,Bundle}-SortKey.
Perl Data Language
Changes for 2.088 - 2024-04-22
- Slatec::ch{ic,sp} work arrays now [t]
- add Slatec::bvalu
- add Func::{pchip,spline}, and a demo
- add Ufunc::diff2
- extra ) on end of Pars now an error
- PP dim sizes can be =CALC(...) instead of explicit RedoDimsCode
- PP add loop(n=start:end:inc) idiom to stop not at end and have non-1 inc
- updated README.md - thanks @falsifian
- support T_PTROBJ in typemap
an interactive plotting widget and library for PDL and Prima
Changes for 0.21 - 2024-04-22
- demo now only loads Prima when run, reducing perldl problems
demonstrate PDL::Graphics::PLplot capabilities
Changes for 0.84 - 2024-04-22
- add perldl demo
- fix plFreeGrid to use free that matches the allocator
- pl{Alloc,Free}{,2d}Grid error if not match allocating and freeing routine
- minimum PDL 2.083 due to output OtherPars
SAML bindings and protocol implementation
Changes for 0.79 - 2024-04-21T18:13:48Z
- Compatibility Deprecation Warning
- Significant Changes
- Detailed Change Log
Validate HTTP requests and responses against an OpenAPI v3.1 document
Changes for 0.062 - 2024-04-21T18:11:43Z
- BREAKING CHANGE: operation_path is dropped from the populated $options hash in validate_request, validate_response, and find_path, and replaced with path_item_uri
- "servers" entries are now allowed in openapi documents, but they are not yet considered when matching URIs to paths
Plack search application.
Changes for 0.05 - 2024-04-21T17:53:19+02:00
- Add 'tags_after' parameter.
Fast Perl SSG: now with automatic Language Translation via OCI and translate.pl. Check it out at
https://github.com/SunStarSys/orion
submitted by /u/joesuf4
[link] [comments]
GitHub - SunStarSys/orion: SunStar Systems' Orion Enterprise Wiki SSG
SunStar Systems' Orion Enterprise Wiki SSG. Contribute to SunStarSys/orion development by creating an account on GitHub.GitHub
Tags helper for table view.
Changes for 0.06 - 2024-04-21T17:26:14+02:00
- Fix of setting of table data. Copy in this time.
Tags helper for CPAN::Changes object.
Changes for 0.04 - 2024-04-21T16:06:08+02:00
- Add trivial Changes file and test on it.
- Fix output on ex1.changes test file.
parses .pm file as PAUSE does
Changes for 0.47 - 2024-04-21
- Improve class detection
Plack application for running Tags::HTML objects.
Changes for 0.17 - 2024-04-21T12:34:17+02:00
- Add test for 'css_data'.
- Rewrite examples in doc to version with indented output.
Perl extension for finding the memory usage of Perl variables
Changes for 0.83_50 - 2024-04-21
- patch from Slaven:
- CvOUTSIDE is again set on all cloned subs [CPAN #149902]