Getting Started with SOMns Development

This guide gives an overview of the development infrastructure and setup used for working on SOMns. First, the basics of obtaining the code, setting up software dependencies, the development environment, and how to contribute code are discussed. Afterwards, we briefly give an overview of the repository and code layout. And finally, look at the additional infrastructure used by the project.

1. Setup

1.1 Minimal Software Requirements

SOMns is implemented in Java 8, uses Ant as a build system, git as source control system, and Python for a launcher script.

On Ubuntu, the following instructions will install the necessary dependencies:

sudo add-apt-repository ppa:webupd8team/java
sudo apt install oracle-java8-installer git ant

1.2 Getting the Code and Running Hello World

To checkout the code:

git clone https://github.com/smarr/SOMns.git

Then, SOMns can be build with Ant:

cd SOMns
ant compile  ## will also download dependencies

Afterwards, the simple Hello World program is executed with:

./som -G core-lib/Hello.som

1.3 Development Environment

To setup a complete development environment, we need also the Graal just-in-time compiler, and Node.js to work on the Kompos Web Debugger.

On Ubuntu, the necessary software can be installed with:

## First, installing Node.js and the Node Package Manager (NPM)
curl -sL https://deb.nodesource.com/setup_7.x | sudo -E bash -
sudo apt install npm nodejs

## Second, installing and compiling the Graal JIT Compiler
cd .. ## leaving the SOMns folder
git clone --recursive https://github.com/smarr/GraalBasic.git
cd GraalBasic
yes "n" | ./build.sh
cd ../SOMns
ant ## to ensure everything is compiled

At this point, it should be possible to use Graal to run SOMns by dropping the -G option from the command line:

./som core-lib/Hello.som
Eclipse Project outline
Eclipse Project outline.

1.3.1 Eclipse to Develop the Interpreter

SOMns is currently developed with Eclipse. While other Java IDEs can also be used, for brevity, we’ll focus on Eclipse only.

At the time of writing, I am using Eclipse Neon 4.6. Please also install the Eclipse Checkstyle Plugin from the Eclipse Marketplace.

For development, we also need to setup all Eclipse projects:

ant ideinit

After the Eclipse projects for the Truffle library were generated by this step, we can import the existing projects into Eclipse with File -> Import… -> Existing Projects into Workspace and pointing Eclipse to the SOMns folder.

I like to import the Truffle projects into a separate Truffle working set. This makes working in Eclipse with many projects easier.

The result should be looking roughly, but not exactly like in the screenshot on the right.

For debugging the interpreter with Eclipse, create a Eclipse run configuration for the SOMns project. The main class is som.VM. To run for instance the Mandelbrot benchmark, add the following as program arguments:

core-lib/Benchmarks/Harness.som Mandelbrot 2 0 500

In VM arguments, enable assertions with:

-ea -esa

I personally start the various SOMns programs from the command line:

./som -d -G core-lib/Benchmarks/Harness.som Mandelbrot 2 0 500

For this approach, we need a Remote Java Application debug configuration in Eclipse. After starting SOMns, it should tell you that it is waiting on port 8000, which is used as the port in the Eclipse debug configuration.

1.3.2 VS Code to Develop with the SOMns Language

To have IDE-like support for working with SOMns code, we are using VS Code, a cross-platform editor for which we have SOMns integration.

SOMns support can be installed via the VS Code marketplace.

1.4 Summary

A brief list of steps:

  1. Install software dependencies: ant, git, Java 8, Eclipse 4.6 (or later), VS Code 1.8 (or later), Node.js, NPM, Graal JIT compiler

  2. Create Truffle Eclipse projects: ant ideinit

  3. Import Eclipse projects

  4. Run ant from the command line or Eclipse

2. Repository and Code Layout

This section gives a brief overview over the most relevant elements.

2.1 Repository Root

.gitlab-ci.yml   # Configuration for the SOFT Build Server
.travis.yml      # Configuration for the Travis Build Service
CHANGELOG.md     # The changelog contains a high-level change overview
build.xml        # The Ant build description
codespeed.conf   # Benchmark configuration, based on ReBench
core-lib         # All SOMns code, including standard lib, tests, and benchmarks
som              # Launcher script
src              # Java sources
src_gen          # Java sources generated by the Truffle DSL
tests            # Java unit tests and tests for DynamicMetrics tool
tools            # Contains the Kompos Web Debugger

2.2 Code Layout: Java

som
|- compiler         # Parser, AST creation, and source representation
|- instrumentation  # AST instrumentation support, used e.g. by DynamicMetrics tool
|- interop          # Interoperability with Truffle languages, only minimally implemented
|- interpreter      # Dynamic SOMns language semantics, i.e., interpreter and AST nodes
   |- nodes                           # AST and dispatch node implementations
   |- objectstorage                   # SOMns object model implementation
   |- actors, processes, transactions # core elements of concurrency models
|- primitives       # Basic operations, exposed via vmMirror object to SOMns
|- vm               # Basic VM setup, startup, and object system initialization
|- vmobjects        # Representation for build-in object and arrays
|- VM.java          # Java entry point and bridge between interpreter and tools

tests
|- debugger.JsonTests   # Tests GSON serialization of debugger messages
|- som              # JUnit tests for interpreter, including runner for BasicInterpreterTests

tools
|- actors           # Tracing support for actor features
|- debugger         # Connection to the Kompos Web Debugger
|- dym              # DynamicMetrics tool

2.3 Code Layout: SOMns

core-lib
|- Benchmarks       # Collection of various benchmarks
|- TestSuite        # Collection of tests
   |- BasicInterpreterTests  # Minimal tests only executable as JUnit tests
   |- Minitest.som           # Newspeak's Minitest framework
   |- *Tests.som             # Various test suites
|- Actors.som                # Actor and promise classes
|- Collections.som           # Sets, Dictionaries, etc
|- Hello.som                 # Hello World program
|- Kernel.som                # Core classes: Integer, Boolean, String, Array etc
|- Mirrors.som               # Minimal and incomplete mirror API
|- Platform.som              # Application loader
|- Processes.som             # Communicating Sequential Processes classes
|- System.som                # Minimal API to access system functionality
|- Threading.som             # Threading and fork/join-related classes
|- Transactions.som          # Software transactional memory classes

2.4 Code Layout: Kompos

Kompos is a web-based debugger integrated into SOMns. It is based on Truffle’s language agnostic debugger support and extends it to provide actor-specific debugging facilities. It is implemented in TypeScript.

Note that Kompos and the VS Code debugger share the TypeScript code that is used to interface with the Java backend.

/tools/kompos/
|- index.html            # The HTML elements for the debugger
|- package.json          # NPM package definition
|- src
   |- breakpoints.ts     # Breakpoint related code
   |- controller.ts      # MVC controller to connect model and view
   |- debugger.ts        # Debugger model to manager state needed in UI
   |- history-data.ts    # Model for the actor message history
   |- main.js            # Main JS file used from HTML frontend
   |- messages.ts        # Definitions of messages exchanged with SOMns interpreter
   |- view.ts            # The HTML view code
   |- visualizations.ts  # Visualizes actor interactions as graph
   |- vm-connection.ts   # Web socket connection to the SOMns interpreter

|- tests                 # Tests debugger interaction with SOMns, including some SOMns integration tests
|- tsconfig.json         # TypeScript config
|- tslint.json           # TypeScript lint settings

3. Basic Design

This section gives a brief overview of some basic design aspects of SOMns and Kompos.

3.1 Communication with Kompos

Communication between Kompos and SOMns is implemented with WebSockets, the code can be found in tools.debugger.FrontendConnector and vm-connection.ts. Data is transferred on two separate sockets, one is for JSON and the other one is for binary data.

JSON is used for two-way communication in a message-oriented style. Message objects are serialized to JSON when sending, and received JSON data is used to create message objects. Kompos also (de)serializes communication data, messages.ts provides some interfaces that help accessing data of message objects.

On the SOMns side, message classes need to be “registered” in tools.debugger.WebDebugger.createJsonProcessor(), they extend either IncomingMessage or OutgoingMessage. When an IncomingMessage is received, its process method is called.

The binary socket is used for directly sending tracing data, which is pushed to Kompos whenever available. Kompos parses the tracing data according to the trace format and uses the data to generate the actor graph.

3.2 Trace Format

The trace data includes currently the following events:

  • Actor creation
  • Promise creation
  • Promise resolution: when a Promise is resolved with a value.
  • Promise chained: when a Promise is resolved with another Promise.
  • Mailbox (continued): set the context for following message events (receiver, base message id), occurs when a mailbox is processed.
  • Message: can only occur after a mailbox (continued) or a message event, represents messages in the mailbox.

3.3 Startup Protocol

The following diagram gives an overview of the startup protocol. For simplicity the binary WebSocket for trace data and the view object are not included.

kompos startup protocol
Simplified overview of startup protocol between SOMns interpreter and Kompos debugger.

4. Infrastructure

Give a brief overview of some of the infrastructure used in SOMns.

4.1 The som Launcher Script

To document, simplify, and standardize how SOMns is started or debugged, we use a Python script to execute the actual Java program.

The Python script manages various command-line parameters and selects the JVM to be used for execution.

Below, we see that ./som --help supports a large set of options, of which we detail only a few.

The basic options include -d to allow to attach a Java debugger, for instance from Eclipse, -dnu to print stack traces on a #doesNotUnderstand message, or -G to run without Graal-based JIT compilation. Generally, the options are designed to use upper-case shorthands when they disable a feature.

Further below, we see different categories of other options. This includes flags to investigate and understand how Graal executes the interpreter, options for various profiling tools, as well as tools for SOMns code coverage, dynamic execution metrics, or interactive debugging of SOMns code (currently called ‘web debugger’ -wd).

$ ./som --help
usage: som [-h] [-d] [-t THREADS] [-p SOM_PLATFORM] [-k SOM_KERNEL] [-dnu]
           [-i] [-if] [-io ONLY_IGV] [-l] [-ti] [-w] [-f] [-v] [-gp] [-ga]
           [-gi] [-gb] [-tp] [-td] [-wd] [-dm] [-at]
           [-atcfg ACTOR_TRACING_CFG] [-mt] [-tf TRACE_FILE] [-TF]
           [--coverage COVERAGE] [-o ONLY_COMPILE] [-A] [-B] [-C] [-G] [-X]
           [-T] [--no-graph-pe] [-vv] [--print-graal-options]
           [-D JAVA_PROPERTIES]
           ...

optional arguments:
  -h, --help                   show this help message and exit
  -d, --debug                  wait for debugger to attach
  -dnu, --stack-trace-on-dnu   print a stack trace on #doesNotUnderstand:
  -G, --interpreter            run without Graal

Investigate Execution
  -i, --igv                    dump compilation details to IGV
  -l, --low-level              enable low-level optimization output
  -v, --visual-vm              connect to VisualVM for profiling

Profile Execution
  -gp, --graal-profile         enable Graal-level profiling after warmup
  -tp, --truffle-profile       enable Graal-level profiling after warmup

Tools
  -td, --truffle-debugger      start Truffle debugger
  -wd, --web-debugger          start Web debugger
  -dm, --dynamic-metrics       capture Dynamic Metrics
  --coverage COVERAGE          determine code coverage and store in given file

4.2 Build System

As seen earlier, SOMns uses Ant as build system. The setup tries to minimize the external software dependencies. Currently, instead of using some automatic dependency management system for SOMns, we use an uberjar that combines all rarely changing Java dependencies.

The corresponding project is SOMns-deps, which is essentially a shell script creating a jar file from a set of libraries an then uploading it onto Bintray.

The Truffle library is however directly used as a git submodule dependency, because it changes frequently, and we sometimes need changes in Truffle. Currently, SOMns also relies on a personal fork of Truffle to support changes in the instrumentation and debugging support.

4.3 GitHub

SOMns relies on GitHub, its issue tracking, and pull request system for development.

Change Tracking with Pull Requests: The general approach is that all kind of changes are supposed to be documented at least with a pull request to simplify tracking.

When you are getting started with working on the SOMns interpreter internals, consider checking out the Getting Started label. These issues are more or less simple changes that with a bit of guidance should provide a good introduction to the SOMns code base, an basic understanding of how Truffle-based interpreters work, and a few SOMns specific insights.

4.4 Code Style

When working on SOMns code, please look at the code around you and stick to the style. It might be particular, but it is consistent in this code base.

To ensure basic compliance with the style, we use checkstyle. It is integrated into the build system and continuous integration system. Please use something like Eclipse Checkstyle to integrate it in your editor.

We are also using Codacy to monitor additional style issues or potential bugs. See the STM pull request for examples.

4.5 Development Support

Continuous Integration: To automatically run unit tests for the interpreter, SOMns, and the debugger, we use Travis CI (see .travis.yml) as well as a private GitLab instance to run benchmarks (see .gitlab-ci.yml).

The current build status is: Build Status

Performance Tracking: Since one goal of SOMns is to be a platform for research on concurrency with performance close to state-of-the-art JVMs, we continuously track benchmark performance, for startup as well as peak performance with Codespeed. It is run on every change to the master branch, and can be used to track and compare performance of experimental changes as well.

SOMns Codespeed: Havlak performance
SOMns Codespeed, tracking benchmark performance. Example shows *Havlak* peak performance.

The benchmark execution is configured with codespeed.conf and are executed with the ReBench tool.

SOMns Code Coverage: To track SOMns code coverage, we use Coveralls.