Autoplot in 2025 - Review of Lessons Learned

Autoplot in 2025 - Review of Lessons Learned Jeremy Faden Cottage Systems University of Iowa Introduction to Autoplot Autoplot was first published in 2007 as part of the Virtual Observaties project at NASA. It was preceded by PaPCo, an IDL application for plotting a stack of time-series plots. Research groups would publish codes which would plug-in to PaPCo, making their data available to others using the software. Das2, a Java platform for building science applications for the Radio and Plasma Waves group at the University of Iowa, was identified as a better platform for the Virtual Observatories, and Autoplot was designed and released that year. It would provide no-code solutions to data analysis, and an alternative to the expensive IDL software many groups were using. Matlab was used a bit in Space Physics as well, and we'd barely heard of NumPy use. PaPCo also lacked a standard data model, which would benefit the software. PaPCo modules would be provided the graphics context on which their time series would be drawn. The module might read data from specific CDF files, and then draw spectrograms on the graphics context using codes the group had developed. This meant that much of each module's codes were nearly copies of other modules' codes, and the scientist looking at the plot would have to understand what each graphic was communicating. For example, how is missing data displayed? Why is this spectrogram smooth while this other is blocky? We'd been experimenting with standard models for data, where once data is adapted to the model, standard codes would be used for data handling. This is how Das2 had been developed, but its model was limited to specific data types for spectrograms and line plots. Within PaPCo and Autoplot, a new model, QDataSet, was introduced which would handle all the data types seen in CDF files and more. This model has been effective and is still in use, and will be presented here. No-Code Autoplot Use Autoplot provides a lot of functionality without having to write any code. There are configurations to learn, like URIs, but GUIs provide a graphical way to build them. QDataSet Data Representation Model QDataSet is a flexible model intended to be language-agnostic. It uses little syntax and uses semantics to build structure. Inspired by CDF and NetCDF, it has a rank which indicates the number of indices, and properties which are used to document the data and connect datasets together to form more abstract data. Np[Epoch=288] LABEL="SW H Num Density" UNITS=#/cc Epoch[288] UNITS=milliseconds since 1970-01-01T00:00 DEPEND_0 Electron_DEF[Epoch=6260,Energy=29] LABEL="Electron DEF" TITLE="Election Differential Energy Flux UNITS=1/(cm^2-s-sr) VALID_MIN=0.0 VALID_MAX=1.24e10 QUBE=true Epoch[6260] UNITS=milliseconds since 1970-01-01T00:00 DEPEND_0 Energy[29] UNITS=eV SC ALE_TYPE=log LABEL="Elect ron Energy" DEPEND_1 BGSM[Time=24,[Bx,By,Bz]] UNITS=nT VALID_MIN=-65534.0 VALID_MAX=65534.0 QUBE=true Time[24] UNITS=days since 2000-01-01T00:00 DEPEND_0 Bx LABEL="Bx (GSM )" BUNDLE_1 By LABEL="By (GSM )" Bz LABEL="Bz ( GSM) " Das2's original data model was designed for the workgroup and could only represent 2-D spectograms and stacks of data for lineplots. It used Java types to identify data, and any new type would require new Java types and single inheritance. QDataSet uses duck-typing, so that if a dataset is capable of being handled in some way, then this is correct. No property is required. The number 1.23 can be represented as a QDataSet, as can a CDF file with all of its parameters each with labels and units. Units are used to interpret the numbers, providing support for timetags ("seconds since 1970-01-01T00:00Z") and nominal data (1="Chicago" 2="Paris" 3="Tokyo") Timetags are propagated automatically though operations. Unit conversions are done automatically. Autoplot uses a browser metaphor for looking at data. An Autoplot data URI is entered into the address bar and the data is displayed with reasonable default settings. A URI might be the name of a CDF file and then a parameter within the CDF file. To implement this, Autoplot sees the .cdf extension and hands the URI off to the CDF plug-in which is responsible for resolving the URI into a QDataSet. This QDataSet is then inspected for its type, which is a duck-type, meaning if it looks like a 2-D spectrogram, then it is one. (We call these dataset schemes internally.) The appropriate method for displaying the type is then used. Autoplot provides no-code methods for slicing, dicing, and manipulating the data, as well as a "Mash Up" tool for combining data together. One of my greatest joys working with this software is when the grandparents of our field tell me how happy they are to be working with the data again, instead of asking thier students to prepare figures. That said, Autoplot would be rather limited if the scientist couldn't write code, and it has always had a scripting environment built into it. If Autoplot were to match the flexibility of IDL and Matlab, it would have to allow for scripts to handle odd data formats and to control the application itself. Jython, which is Python implemented in Java, is used for this purpose, and functions are added to the language for brevity. (Instead of no-code, this could be thought of as low-code.) Our computing world is much different now than in 2007. Python now dominates the field, and the goal of providing a free alternative to IDL and Matlab is met. However Autoplot is still used widely, and I believe it has wisdom to offer. Events List is used to control time range shown. Create PNG Walk GUI runs the current plot, using a URI template or an events list. Fun PNG Walk Tool ContextFlow view shows plots in wider context. Jython Scripting in Autoplot It's often the case that software is used in new ways, beyond the scope of problems imagined when it was written. Jython scripting provides for this in Autoplot. This is constrained and maybe silly, but our goal was to provide the flexibility of IDL to Das2 graphics. This is also constrained to remove the complexity a full software environment provides. Imports are discouraged and typically avoided, and Python's mechanism where one code calls into another code is not used. (There is no search path for the scientist to manage.) This allows scripts to be served from a website and run without the burden of downloading and keeping track of the software. The software developer can fix a bug and the scientist using the software will see the fix, automatically. Over the years this model grew, for example supporting rank 0 datasets (1.23), arrays of complex numbers, and bundles containing both line plots and spectrograms. Using this short script, the student digitizes whistler events observed on the Juno mission. The student enters the script URL in the address bar and Autoplot offers to run the script. Das2's mouse event handling handles the mouse events and calls back into the script, which records the whistler in a table. The script also draws the digitized events on top of the spectrogram. https://research-git.uiowa.edu/abbith/juno/-/blob/main/u/george/2023/20230823/whistlerDigitizer.jy Note this is in a GitLab repository, which Autoplot mounts as a Das2 FileSystem object. Changes to the script are distributed automatically. GitHub is also supported in this way Mash-Up Tool provides a graphical way of combining datasets. A few of Autoplot's supported formats use scripts to load data. For example, "tfcat" events files are read in with a script maintained in France, as are RadioJove's .sps files. These scripts reside in a github repository, maintained independently by remote people using these formats. Some Lessons Learned Code is a barrier for some scientists. They may not have time, or they may have thousands of lines of code in another language that they don't want to abandon. I have many people that use Autoplot interactively, and I have others who only use code to create plots. Writing code and reading code are very different things. I can write a code for a scientist, and they are comfortable using it or even hacking it to experiment on their own. The data model is important and should be considered carefully. With QDataSet I've able to extend to add new types, but it does require many semantic checks. Adding a code which tests and generates QDataSets of every scheme has been useful. Named functions might be better than special symbols. It's easier and more legible to have "matmul" than to expect new people to know that "@" is matrix multiply. This also provides search terms to find more information. I've code for propagating units through operations, similar to how Maplesoft's Maple would, and I've seen some of the Python libraries are doing this. I can't easily make this change in Autoplot's scripting because it surely introduce errors in the hundreds of scripts in operation. Make sure the problem you are solving is the scientist's problem, not the problem you want to solve. Every feature in Autoplot was requested by a scientist to solve their particular problem, and my job is to find a way to either properly generalize a problem so that it benefits the community, or provide the solution as a script which does not affect others. The data tab provides slice operation of data from HAPI server. https://autoplot.org "Schemes" identify types of data, and a Java code provides examples and tests for each identified scheme. See https://github.com/das-developers/das2java/blob/ main/QDataSet/src/org/das2/qds/examples/Schemes.java and https://cottagesystems.com/jenkins/job/autoplot-test037/ lastSuccessfulBuild/artifact/test037_showAllSchemes.png.