Ann: SWI-Prolog 8.1.15

[jan]

Dear SWI-Prolog user,

NOTE Google is giving false alarms on this site. See Parts of website blocked by Google Safe Browsing?

SWI-Prolog 8.1.15 is ready for download. This versions comes with with
some enhancements, bug fixes and a big change to the location of user
config and data file locations. Highlights:

• New default file_search_path/2 aliases for user_app_{config,data},
  common_app{config,data} and app_{config,data}. Moved locations for
  all hidden files normally kept in the user’s home directory. See
  https://www.swi-prolog.org/modified/config-files.html.

• Race condition between AGC and message_queue_destroy/1 that can
  cause crashes. The bug has been there a long time. Keri Harris
  found and fixed it.

• Replace unreachable volatile atoms by ‘<garbage_collected>’, improving
  the effectiveness of atom garbage collection.

• Galileo Sartor provided a specification for generating a
  snap package. Can be used to generate
  your snap. Some more work will follow to get an official snap
  version, providing up-to-date binaries that run on many Linux
  distros. Thanks!

• Improve trie_gen/2,3 when matching an instantiated value
  against a trie choice node that contains variables. Mostly
  improves performance for some incremental tabling use cases.

• Improved randset/3 and randseq/3 performance.

• More rewrites to the execution profiler. More will follow.

• XSB: Emulate standard module and support using XSBs gpp
  preprocessor.

  Enjoy — Jan

SWI-Prolog Changelog since version 8.1.14

• MODIFIED: Introduced user_app_{config,data} and common_app{config,data}
  and make app_{config,data} and alias for the user and common versions
  (in that order).

• ADDED: win_folder/2: keys common_appdata and local_appdata.

• SNAP: Add snap package

• DOC: Updated customization files and install them.

• MODIFIED: Place the personal init file in a common place on all
  operating systems. This directory is located in one of the
  directories below and names swi-prolog - Windows: CSIDL directory
  APPDATA (see win_folder/2) - Otherwise
  - $XDG_CONFIG_HOME
  - $HOME/.config

  In this directory we find:

  • init.pl (previously .swiplrc or swipl.ini (Windows) Personal
    initialization file
  • lib (previously ~/lib/prolog) Personal library
  • xpce (previously ~/.xpce) Directory holding xpce application data

  If you use multiple versions we suggest to create the new hierarchy and
  make symbolic links from the old locations. Windows users will have to
  live with copies.

• FIXED: Windows: tmp_file/2 memory leak

• PORT: Use _NSGetEnviron() instead of environ on MacOS when available.

• ADDED: PL_put_uint64(). Suggested by Mathijs van Otterdijk.

• FIXED: lock queue table when performing AGC as another thread may be
  destroying a message queue

• DOC: XSB dialect issues.

• XSB: Added library(dialect/xsb/gpp) to use the XSB preprocessor.

• XSB: Exploit peek_string/3 to realise the required two phase processing
  of XSB files if the input cannot be repositioned.

• ENHANCED: peek_string/3 now allows peeking the remainder of long
  inputs using a huge value for the peek length while allocating
  the minimum of the peek length and the input length rather than
  always allocating the peek length.

• ADDED: prolog:open_source_hook/3 to allow hooking the final open for
  the compiler.

• FIXED: statistics/2 for table_space_used multiplied the value
  incorrectly with sizeof(trie_node). New stats also include the
  trie itself and the call trie. It ignores node indexes though.

• XSB: Added emulation of Term Writing to Designated I/O Streams,
  module standard.

• ENHANCED: If a significant part of the clauses that are candidate
  for CGC have an (atom) clause reference, first call AGC.

• ENHANCED: GC: remove links to unreachable volatile atoms.
  This improves the efficiency of atom garbage collection.

• FIXED: profile/2 using cumulative(true) without GUI. Abramo Bagnara.

• ENHANCED: trie_gen/2,3 for when matching an instantiated value
  against a trie choice node that contains variables. Test by
  Edward Schwartz.

• ENHANCED: library(random): performance of randset/3 and randseq/3
  for selecting a small number of elements in a large range.

• MODIFIED: low-level profiler access predicates to communicate
  about predicates using predicate indicators rather than heads.
  This simplifies the code and makes most data structures ground.
  Note that the hook user:prolog_predicate_name/2 is now called with
  a predicate indicator rather than a head.

• FIXED: profile_data/1. Broken after adding profile_procedure_data/2.

• ADDED: profile_procedure_data/2 to quickly interrogate individual
  predicates.

Package xpce

• MODIFIED: Load PceEmacs extensions from app_config('xpce/emacs')
  instead of ~/lib/xpce/emacs

• ENHANCED: Allow edit script and PceEmacs mode to pass the column
  to edit.

• MODIFIED: Place PceEmacs socket in xpce’s data directory

• MODIFIED: Inherit new SWI-Prolog search for storing application data.

• FIXED: Details menu and single click in pce profiler view.

• COMPAT: Sync with new profile data format and reuse more code from
  the core Prolog system.

[fnogatz]

Note that the package manager snap and the visual programming language Snap! share nothing but their name – and the coincidental timing to attract the SWI-Prolog community recently

[peter.ludemann]

There seem to be some permission/ownership problems if installing from the PPA. I had to do things like sudo chown -R peter ~/.config/swi-prolog/ ~/.local/share/swi-prolog.

Also the startup message only mentions ~/.swiplrc, but it seems I need to reinstall all the packs. (I’m still working through the changes, so there might be more things that broke with the directory layout change.)

[ttmrichter]

More false alarms (I’m presuming): both Windows executables are being flagged as containing malware. :-/ I’m not sure if this is Windows Defender or Firefox screwing up, but one or the other is complaining about it. I’m trying a different downloader to narrow down which is the culprit.

[ttmrichter]

It’s Firefox, not Windows Defender. At least there’s that.

[jan]

@fnogatz started this discussion on Parts of website blocked by Google Safe Browsing?

[peter.ludemann]

To clarify – some of the directories are owned by root, which is why I had to change their ownership.

Also, it would be good if the warning listed the directories that are now obsolete, so that they can be deleted (e.g., ~/lib/swipl/pack).

[ttmrichter]

Another problem: the latest Ubuntu (“eoan” is the code word) isn’t in the PPA dists directory, making an upgrade to said latest Ubuntu drop SWI.

[jan]

but it seems I need to reinstall all the packs

That is possible, but in general this should do the job

mkdir -p ~/.local/share/swi-prolog
mv ~/lib/swipl/pack ~/.local/share/swi-prolog

If you want to be able to run the old version, link them. That should also solve problems if some pack depends in the location it is installed (should not be the case unless the pack did something special).

ln -s ~/.local/share/swi-prolog/pack ~/lib/swipl/pack

permission/ownership problems

Seems installing the PPA runs Prolog with $HOME pointing at the original user. As ~/.config/swi-prolog also contains the history, not creating it is not so easy. Seems there are several ways out:

• Aggressively trying to avoid creating these directories as long as no file needs to be placed there. That should deal with this case which runs SWI-Prolog non-interactively, but still gets us into trouble if a user runs sudo swipl
• Not create (and not read?) such files if the EUID doesn’t match the UID of the parent directory.
• Use permissions similar to /tmp on the created config dirs.
• If we run as root, create the directories/files with the original UID.

Ideas?

[jan]

eoan

Added to the script. Might do a quick re-release, depending on the config file issue.

[peter.ludemann]

• Aggressively trying to avoid creating these directories as long as no file needs to be placed there. That should deal with this case which runs SWI-Prolog non-interactively, but still gets us into trouble if a user runs sudo swipl

I think that if a user does sudo swipl, they can expect files and directories to be created with ownership root. Also, anyone who does sudo swipl is probably at least as sophisticated as me (a low bar) and can figure out how to chown things or chmod +t (the latter is probably what I should have done rather than chown).

Not create (and not read?) such files if the EUID doesn’t match the UID of the parent directory.

This probably would lead to subtle changes in behaviour and would confuse people.

• Use permissions similar to /tmp on the created config dirs.

You mean setting the sticky bit in directories?

• If we run as root, create the directories/files with the original UID.

/usr/lib/swi-prolog should be owned by root, and writeable only by root; otherwise multi-user machines would invite chaos. ~/.local/share/swi-prolog should be owned by the user and writeable by the user.

Python’s “pip” installation tool (which is similar in what it does to SWI-Prolog’s “pack”) can install either for the user (in ~/.local/lib/python3.7) or globally (in /usr/lib/python3.7). This is controlled by an option (--user). Maybe SWI-Prolog should have a similar option? (The Python community has also evolved various “virtual environment” tools to allow specifying what should be installed, similar to docker containers but without the complexity of a VM.)

[jan]

eoan

Is now present

if a user does sudo swipl , they can expect files and directories to be created with ownership root

That seems to be the case in general. Tried with eog (Eye of Gnome) by removing the config files and running using sudo.

This probably would lead to subtle changes in behaviour and would confuse people.

I don’t really see how this confuses people. Note that we are talking (only) about the files placed there that the user is not or barely aware of: command line history, positions of xpce windows, etc. I guess people consider running the program as sudo as something happening in a different environment. So, I’ve uploaded new PPAs, also for eoan, that stop creating the swi-prolog directory and these implicitly created files if the place where they are created is not owned by the same user that runs the program. I think that will reduce confusion. Surely there are a lot of people able to run sudo apt install swi-prolog but who have no clue what to do about not being the issues that arise from a not writeable config directory.

Also added check_config_files/0 and update_config_files/0 to library(check_installation).

can install either for the user (in ~/.local/lib/python3.7 ) or globally

Searching packs now follows the XDG (data) standard, i.e., for the user in $XDG_DATA_HOME or ~/.local/share and globally in $XDG_DATA_DIRS or /usr/local/share and /usr/share, all using the swi-prolog subdirectory. pack_install/2 provides a package_directory option, but should probably get something like user/global distinction. We could also opt for a command line option as

swipl --install-pack dotcloud --global

Would that make sense?

[pmoura]

Also added check_config_files/0 and update_config_files/0 to library(check_installation).

Using the current git version, I get:

?- check_config_files.
Warning: Config: (add-ons) 
Warning:   found different directory "/Users/pmoura/lib/swipl"
Warning:   new location is "/Users/pmoura/.local/share/swi-prolog/pack"
true.

But the packs are already at the new location:

$ ll /Users/pmoura/lib/swipl/
total 392
-rw-r--r--    1 pmoura  staff   1823 Jan 14  2019 LICENSE
-rw-r--r--    1 pmoura  staff   1693 Apr  7  2019 README.md
drwxr-xr-x    5 pmoura  staff    160 Oct  8 18:21 bin
drwxr-xr-x   28 pmoura  staff    896 Oct  8 14:24 boot
-rw-r--r--    1 pmoura  staff  86396 Oct  8 18:21 boot.prc
-rw-r--r--    1 pmoura  staff  81163 Jan 18  2019 boot64.prc
drwxr-xr-x    5 pmoura  staff    160 Oct  6 16:11 customize
drwxr-xr-x    5 pmoura  staff    160 Mar  9  2019 demo
drwxr-xr-x    5 pmoura  staff    160 Oct  8 18:21 doc
drwxr-xr-x    7 pmoura  staff    224 Oct  7 20:48 include
drwxr-xr-x    4 pmoura  staff    128 Sep 21 09:23 lib
drwxr-xr-x  204 pmoura  staff   6528 Oct  9 08:48 library
-rw-r--r--    1 pmoura  staff   4172 Jul 20 15:06 swipl-win.rc
-rw-r--r--    1 pmoura  staff      2 Oct  8 18:20 swipl.home
-rw-r--r--    1 pmoura  staff   4172 Jul 20 15:06 swipl.rc
drwxr-xr-x    8 pmoura  staff    256 Jan 14  2019 xpce

The /Users/pmoura/lib/swipl exists due to compiling using DCMAKE_INSTALL_PREFIX=$HOME. I don’t think the warning above should be printed in this (common?) case as the packs directory is elsewhere.

[jan]

Thanks. Should have been ~/lib/swipl/pack. Fixed. The added benefit is that packs are no longer installed in the SWI-Prolog installation if you install SWI-Prolog under $HOME. As usual, such changes are painful, but I think it will be worth it …

[ttmrichter]

As usual, such changes are painful, but I think it will be worth it …

If you didn’t want pain, you shouldn’t have started in software. Self-inflicted injury, really.

[fnogatz]

Sorry to pull this old topic back to top. But it seems to be the best place to further discuss the things regarding the changed ~/.local/share/swi-prolog/pack vs. ~/lib/swipl/pack paths

I work with swivm to automate the installation of multiple versions of SWI-Prolog, for instance for testing as part of a CI workflow. In this thread, it is stated that the ~/.local/share/swi-prolog/pack directory is created only if needed, i.e. typically on the first call of ?- pack_install(...) with no explicit package_directory(...) option given. Is there some toplevel goal that returns the path ~/.local/share/swi-prolog/pack for versions starting with this 8.1.15, and the legacy path ~/lib/swipl/pack for the older? This would be useful for a compatibility layer in swivm, so globally installed packs can be linked to or re-installed.

Also the proposed CLI option --install-pack would be quite handy. As of now, I usually have to call swipl -g 'pack_install(dcg4pt,[interactive(false)]),halt(0)' -t 'halt(1)' at some point to install dependencies, like in the Makefile of plammar. Any chance the --install-pack feature will come true at some point in the future?

[swi]

@fnogatz

Is there some toplevel goal that returns the path ~/.local/share/swi-prolog/pack for versions starting with this 8.1.15, and the legacy path ~/lib/swipl/pack for the older?

You may be able to use:

file_search_path(pack,PackDir).

[fnogatz]

Hm, then I’m probably not using it the right way:

Welcome to SWI-Prolog (threaded, 64 bits, version 8.0.3)
SWI-Prolog comes with ABSOLUTELY NO WARRANTY. This is free software.
Please run ?- license. for legal details.

For online help and background, visit http://www.swi-prolog.org
For built-in help, use ?- help(Topic). or ?- apropos(Word).

?- file_search_path(pack,PackDir).
PackDir = app_data(pack) ;
PackDir = swi(pack).

[swi]

@fnogatz

file_search_path(pack,PackDir).

what does the following return?:

file_search_path(pack,PackDir), absolute_file_name(PackDir,D).

[fnogatz]

Ah, that does the trick. ?- absolute_file_name(app_data(pack), Dir). works as expected for all versions I tested:

$ swivm ls
->       v6.6.6
         v7.2.3
        v7.7.25
         v8.0.3
        v8.1.32
         v8.2.4
        v8.3.27