After more than a year, we released a new version of LUE. A lot has happened since the previous release. Details can be found in the Releases page in the manual. The sharp reader may notice that the release of version 0.3.8 hasn't been announced in the past. We skipped it due to issues with the Conda package build process. That is all fixed now, but that does mean that the set of changes made since the previous officially released version (0.3.7) is the combination of those made for both the 0.3.8 release and the 0.3.9 release.

The detailed list of changes contains too much detail for the LUE end-user. Here is the list of changes most relevant to them:

• Added new operations, e.g.: clump, log10, negate, normal, round_down, round_off, round_up, sqr, square
• cast now supports casting from floating point element types to integral element types
• Updated the documentation.
• Added support for default partition shapes. These are used when no partition shape is passed to commands and operations that need one. For best performance a good partition shape must be determined experimentally, but to get started a default one can be used.
• Added the lue_calculate.py command for one-off computations from the command-line
• Improved the lue.pcraster Python sub-package. For most operations in PCRaster we now have LUE equivalents. Porting existing PCRaster models to use LUE has never been easier.
• Improved support for asynchronously computed scalar values. Most operations accepting a regular scalar value have been overloaded to also accept ascynhronously computed scalar values.
• Added support for NumPy-2

See the Community page for ways to get in touch with us and stay up to date about LUE.

One of the goals for developing LUE is to research scalable algorithms for spatial analysis. A LUE computation should make good use of additional hardware, to make computations finish faster, or to make larger computations feasible.

Scalability experiments involve executing a certain computation many times, while varying the amount of hardware and/or data, and recording the latencies involved. Afterwards, all recorded latencies have to be aggregated, statistics computed, and further postprocessed to end up with scalability plots showing the efficiencies.

The example below shows the results of a strong scalability experiment of a LUE version of Conway's Game of Life. The second plot shows that using 12 CPU cores instead of one, speeds up the model almost 10 times.

To make it convenient to perform scalability experiments, we have developed the lue_scalability.py command. This command removes much of the administrative and practical burdens of performing such experiments. For more information, see the manual about scalability.

Sometimes you may want to quickly compute some new result based on existing datasets, maybe just for visualization purposes, or to test an idea. Creating a new C++ program or Python script may feel like a lot of work for such a simple task. For that we have created the lue_calculate command. It allows you to execute an assignment statement which you pass on the command-line, and it will do everything needed to finish the computations as quickly as possible. By default, lue_calculate will use all CPU cores available, but this can be configured by passing additional arguments. Here are two examples:

lue_calculate 'slope.tif = slope("$MY_DATA/elevation.tif", 100.0)'
lue_calculate 'flux, state = accu_threshold("flow_direction", "material", "threshold")'

As you can see, the command supports expanding environment variables (on Windows, %VARIABLE% is also expanded), multiple return values, and default file extensions. Nested expressions are also supported. The one limitation is that only a single assignment statement can be passed to the command. If you need more, then you can still write a C++ or Python program, of course.

The lue_calculate command will be part of the upcoming 0.3.8 release.

For more details, execute lue_calculate --help and see the LUE documentation.

At June 29 we organized the first hands-on LUE short course. In about three hours the course participants learned everything that is needed to get started with using LUE to build scalable simulation models.

In turn, the following topics where handled:

• Introduction of the LUE modelling framework

• How to execute LUE models

• The hydrological case-study model

• Measuring performance and scalability

• The factors influencing performance and scalability

• Calculating the scalability of the case-study model

• Relation LUE vs PCRaster, and upcoming developments

The group of course participants consisted of people working at both research institutes and private companies. Some of them where in the room with us and some where joining online.

We think the course was a success. Everybody learnt a lot, including us.

We will be organizing more LUE courses over time. If you are interested in joining us, then do let us know (lue@computationalgeography.org), and we will notify you once we picked a date.

The Computational Geography group is organizing a workshop about parallel programming using asynchronous many-tasks (AMT) and the HPX C++ library during the EGU General Assembly 2023 in Vienna. AMT/HPX is used in the implementation of the LUE framework. During this workshop we will introduce this approach and illustrate how it is used in LUE.

This workshop is especially relevant for C++ software developers interested in developing scalable software. Knowing about AMT/HPX is not required for LUE users, but it might be interesting for them too.

For more information, see the associated short course page on the EGU23 website.

We are planning one or several online workshops, specifically about the support for large scale field-based modelling that is part of LUE. These will be informal sessions of one hour in which we will shortly introduce LUE and discuss opportunities for cooperation with the attendants.

In the next few weeks we will add information about these workshop and how to subscribe to them to this page. See the LUE homepage for possibilities to keep up to date about the LUE project, including these workshops.

Last week, our paper Scalability and composability of flow accumulation algorithms based on asynchronous many-tasks was published in the Computers & Geosciences journal. The full paper can be found online: 10.1016/j.cageo.2022.105083.

We have released version 0.3.0 of LUE. This version contains new flow accumulation algorithms that scale well over CPU cores and cluster nodes, and that compose well with other modelling operations. As the version number shows, LUE is not yet ready for production, but every new release gets us closer to that point. We will release Conda packages including the LUE framework ASAP.

List of closed issues associated with this release: https://github.com/computationalgeography/lue/milestone/2?closed=1