Nodes – Our Story

Intro

Nodes* is a JavaScript-based 2D canvas for computational thinking. It’s powered by the npm ecosystem and lives on the web. We take inspiration from popular node-based tools but strive to bring the visual interface and textual code closer together while also encouraging patterns that aid the programmer in the prototype and exploratory stage of their process.

*(not to be confused with node.js)

Nodes was created and developed by Nick Nikolov and Marcin Ignac between 2017 and 2019 inside Variable. Even as the very nature of the tool is to experiment and explore new ways to play with code and data, Nodes has been used in several production-grade projects and tested in various real-world scenarios. We’ve used it to create realtime 3D graphics installations, explore and visualise data, experiment with AI and export results in various formats like images for print, videos for social media, 3D models for mobile AR and data files for further processing.

On this page we are going to explain our motivations, the evolution and the current state of Nodes and how it changed our approach to creative coding.

Quick primer on visual programming

Here’s a very quick primer if you’re a programmer but you’ve never heard of visual or node-based programming. In some tools, the actual programming language is visual, instead of writing text you construct your program visually. Think Scratch, or a modern take like Luna (which technically has a dual nature – visual and textual).

Most of the popular visual programming tools today however use the visual interface as an abstraction one level above textual code. Usually, instead of files you have 2D representations of your program. Very often they are rectangles of some form, connected by wires of some form. Just like most programming languages have similar syntax (for loops and class constructs), visual tools generally tend to be rectangles wired up together representing data flows. This makes them very popular in areas like visual effects programming, real-time graphics, data-processing pipelines, procedural architecture and so on.

These days, neural networks is another area where computational graphs tend to be visualised with nodes and wires.

Background, motivations and inspirations

At Variable – our data visualisation and generative design studio – we are programmers first and foremost. But each of us has their own unique background: interaction and graphic design, real-time graphics, new media art, music production etc. What these fields have in common is that artists rely on professional-grade, usually very visual and sophisticated software-based tools. As our main medium of expression shifted to code, we’ve enjoyed unparalleled creative flexibility but at the same time we traded that for our ability to “see”.

Visual programming tools are as old as computer monitors. Serious web-based implementations however have been rare and impractical. Today, JavaScript and the npm ecosystem offer a wide variety of actively maintained open-source libraries for creative, visual, generative and data-driven projects. Further, modern browsers provide for an ubiquitous networked, collaborative platform that is hard to match. Instead of creating another bespoke and isolated tool, we opted to extend and enhance an already existing and productive environment.

When we set off to create a tool of our own, there were already plenty of different node-based or visual scripting/programming tools and environments – VVVV, Houdini, TouchDesigner, Cables.gl, Vizor Patches, Lichen, MaxMSP, UE4 Blueprints and Origami among the more popular ones. In the years since we started Nodes, we encountered even more, some old, some brand new like the latest Unity Visual Effects Graph. Looking at it this way it’s hard to see why the world needed yet another one of those but when dissecting most of them in-depth, one quickly sees the limitations and various trade-offs each particular tool forces you to make. None in particular offers the JavaScript and npm-first, programmer heavy approach of Nodes.

We made a provisional breakdown of the modern visual and web-based programming landscape in this Visual Tools Landscape Figma Board. You can see high res version here: visual-tools-landscape.pdf.

Features

As of June 2019 we are currently at version 0.11.0.

You use Nodes by installing the package from npm and run a local server. A cloud-based version for collaboration and client work was always one of our goals but isn’t a priority just yet. The benefit of running locally is that you maintain control over your data and don’t have to worry about assets sizes.

Node-based programming

In Nodes you write programs by connecting “blocks” of code. Each node – as we refer to them – is a self contained piece of functionality like loading a file, rendering a 3D geometry or tracking the position of the mouse. The source code can be as big or as tiny as you like. We’ve seen some of ours ranging from 5 lines of code to the thousands. Conceptual/functional separation is usually more important.

You create a node by double clicking anywhere on the Graph Editor canvas and choosing a node from a list of templates. Nodes can declare public properties called ports allowing them to receive and send data via connections. There are two types of ports:

• Trigger: expected to be updated every frame for realtime applications. We use those to define node hierarchy like render graph or UI widget tree. Because of their nature we sometimes say they are “live”.
• Param: used to send data between nodes, updated only if their value changes. Examples might be: cube size, image url, material color etc

So what’s happening here?

1. We start at Canvas node. Every frame, 60 times a second, on requestAnimationFrame Canvas node is firing its triggerOut passing reference to its 2D drawing context to child nodes as part of “props” object (think how React is passing props to child components).
2. That reference is used by Background Color to clear the background and then is passed further down to the Blend node enabling multiplicative blending.
3. We then split into two branches, each one receiving the same props via out trigger every frame:
   1. Time node updates every frame, tracks time passed and outputs that value via time out param
   2. Rectangle node draws a red rectangle with width and height equal to radius in param. We additionally connect Time.time out param to Rectangle.rotation in param to animate it.

Anatomy of a node

Each node is defined by its code. By declaring triggers and params and evaluating the node, at the same time the graph view of the node is updated and the inspector view is populated with widgets reflecting given port type.

1. Incoming trigger – connection from parent node
2. Outgoing trigger – connection to child nodes
3. Incoming parameter – with type color resulting in a color picker widget in the inspector
4. Importing a package from npm (has to be added to the project first)
5. Trigger callback – this is usually called every frame
6. Reading properties provided by the parent node
7. Reading input parameter value
8. Main node code – drawing a rectangle with a given color
9. Passing data further down to the child nodes

Graph Editing

The graph canvas is of unlimited size which allows you to organise your code spatially and navigate around by panning (space + drag) and zooming (mouse scroll) or search (/ + type, see Powerful Search.

You can select a node by clicking on it or select multiple nodes by Shift+Click or drawing selection rectangle around them. Nodes can be deleted by pressing Backspace, and duplicated using copy (Ctrl/Cmd+C) and paste (Ctrl/Cmd+V). We support copying and pasting nodes between different graphs and browser windows. As the copied node is simply a piece of JSON, you can even paste piece of graph into an e-mail or Slack message.

It’s also possible to Undo/Redo your modifications using familiar Cmd+Z, Cmd+Shift+Z (or Ctrl on Windows).

Built-in code editor

One common differentiation between visual programming tools is the level of abstraction they operate on. Some tools like Origami for example are meant to prototype interactions of user interfaces and not much more. Others – like Pure Data for instance – are meant to be very low-level and let users craft their own environments. Some – like UE4 Blueprints or the Unity Visual Effects Graph – are embedded and part of a larger system (e.g. a game engine environment).

Nodes is more directly comparable to Cable.gl, FlowHub or MaxMSP does come with some batteries included but there are less preconceptions of what you use it for. However, unlike most of them we adopt a more programmer-first approach where the nodes themselves are generally a lot more high-level and users are encouraged to peek inside, copy and modify, and write their own. While you could drop on a lower level and express language-level semantics like if-checks and for-loops, or math concepts like add or multiply on a node level we generally don’t feel that is a productive way to work and prefer to write that logic in a textual programming language. This might feel conceptually familiar if you have used NoFlo.

Whatever your preference, with the built-in code editor we are able to fluently move between such levels of abstraction and create nodes that range from rounding up a number to complicated particle system running on GPU. Edge cases can be handled by simply copying node and adding more code. Reusable components can be created by splitting big nodes into few smaller ones. As long as we keep the corresponding types of input/output ports intact, this can be done live on a running graph without needing to reload the whole application. We focused on opening, modifying and seeing changes as straightforward and as instant as possible.

This brings us to the most powerful feature of the code editor: live eval. At any point in time while editing the code, you can evaluate it by pressing shift + enter. All ports already present in the node will have their values and connections preserved. If you make a mistake or typo you can edit code and compile again or roll back to the last saved version.

Changing code on the fly while the state and relationships are both persistent and dynamic can be very freeing. If we introduce far ranging and disruptive changes all nodes with errors will be highlighted in the graph and are usually quick to find and fix. Nodes tries to preserve the last working state as much as possible giving the programmer the chance to fix all mistakes until everything runs smoothly again.

Powerful search

We believe that while an unlimited 2D canvas can be very helpful to think about the “bigger picture” and reason about the architecture of your app, zooming in and jumping around the codebase should be just as fast and fluid. To search you can simply press / and start typing. Use up/down arrows to jump between results and automatically focus on the given node. To search for code prefix your search query with ". For example "ctx.texture2D to find all nodes that call ctx.texture2D function. You can then select a node using arrows and press enter to open code editor. The editor opens at the line of code that matched your query.

Graph State Visualisation

To help you better understand the internal flow of the application, we fade out nodes with trigger ports that haven’t been updated in the last frame. Similarly, the profiling mode visualises the CPU usage of each node. By colouring nodes by the amount of time the trigger took in the previous frame, we can identify hot paths of the graph worth optimising.

We think we are just scratching the surface here and other similar dynamic visualisations of the source code and the program logic can be very helpful in large projects. Before Nodes, we would very often created custom “debug views” alongside the deliverable of your project but now it usually happens right there in the graph.

Static and dynamic node comments

Continuing that line of thinking, you can create a generic textual comment box using ctrl+left mouse click. We allow for some basic color and font size options. Dynamic node comments are created by adding node.comment = 'my comment' and are attached below the name of the node.

We use this feature to visualise global state as serialised JSON, loading status, timers, simple graphs and many more. It allows for a basic but powerful and really fast way to “annotate” your source code and present whatever is the most important top-level information in your program.

Node comments can also have images – for example looking at raw textures in big rendering graphs or even custom “drawings” using an image buffer. See more in Visual Literate Programming.

Export and publishing

From the very beginning we wanted to be able to share prototypes with clients and collaborators. That’s where export functionality comes in. In one click we bundle the JavaScript, graph metadata and all assets into an archive, ready to be hosted on a static web server or bundled as Electron app. Additionally, the inspector allows you to “publish” selected parameters that are then available (although hidden by default) during runtime for tweaking and configuration of your app.

This functionality allowed us to build tools for other designers, screen based installations and visualisations on the web or anything else that has to be tweaked on the fly.

Technology

Core

• choo – A 4kb framework for creating sturdy frontend applications
• We used Choo to implement the UI layer of Nodes – panels, the inspector, modals, menus and so on.
• browserify – to bundle and manage npm dependencies
• tachyons + custom CSS – for styling the interface
• raw HTML Canvas – to visualise the graph
• codemirror – text editor

User-space

Some of the more common modules that seem to work well in such an environment and that we keep coming back to.

• PEX is an ecosystem of open source JavaScript modules developed in-house by Variable for building modern WebGL applications. The main modules are:
  • pex-context – a low-level WebGL wrapper based on command and pipeline abstraction similar to Metal, Vulkan or WebGPU.
  • pex-renderer – a physically based renderer (PBR) and a scene graph similar to Three.js and Babylon.js.

The choice felt natural as both PEX and Nodes are developed at Variable. Having full control over the source code allowed us on improving features of PEX that were needed for Nodes and vice versa. One of the drawbacks of that is less examples and documentation compared to eg. Three.JS that increased the learning curve for new Nodes users.

• @thi.ng/umbrella for functional programming & immutable state primitives
• D3.js for svg generation and data visualisation
• stack-gl for geometry processing

Use Cases

In the last 2 years we have used Nodes for multiple types of projects and output formats. Below is a list of a few possible use cases.

Learnings and Observations

Visual Literate Programming

We noticed two things when working on bigger graphs and writing comments for our nodes.

Firstly, spatial layout of the graph allows you to organise nodes in area or sub-graphs responsible for distinct functionality of your app (e.g. data processing, geometry generation etc) or part of user flow like home screen or an interactive scene. This layout provides an excellent way to see an overview of your application and explain it to a co-worker or client before jumping into the code.

On bigger projects it also helps coming back to an old graph and refreshing your brain on how it works (where as normally you would usually have a bunch of text files and hopefully a README file). By creating empty nodes, giving them appropriate names and writing comments in the graph (not code) you can brainstorm the architecture of your app similar to UML or data-flow diagrams.

Secondly, as we mentioned a few times already, we naturally started including richer data in the node comments like text templates with live values, images and mini graphs. We use those not only for debugging but also for navigation as different parts of the graph become recognisable even on further zoom levels. This allows you to quickly read or scan the graph and get a rough idea of what it’s doing. Our vision is to extent that functionality and have true digital canvas with your graph, snippets of code, visualisations and references all in one place. Which brings us to…

Sketching In Code

Creative coding environments like Processing popularised idea of “sketching in code” – quick and dirty prototyping allowing you to explore multiple ideas quickly. This type of programming is especially suitable for visuals problems and early explorations where focusing too much on code structure and optimisation can slow us down and limit possibilities. If we compare this way of working to pen and paper sketching, the analogy breaks down quickly as we would be limited to draw only one thing per page. Having an explicit “edit, compile, run” loop (as opposed to surgical live changes) and no easy way to maintain multiple versions of the code (as opposed to quickly copy-pasting nodes or parts of the graph) usually hold you back from easily exploring multiple branches of your ideas. Small changes in the ergonomics of your tool can amount to unlocking hidden creative powers.

Nodes on the other hand allows you to not only edit code live but also run multiple branches of graph at the same time e.g. 2D visualisation in d3.js, basic raw WebGL visualisation and full-blown 3D scenes with physically-based lighting using pex-renderer. Another example would be to start with CPU implementation of particle system and later implement GPGPU version while keeping the same input data and final rendering parts of the graph, and even run both versions at the same time.

Programming in Nodes enables easier code decoupling and in combination with copy-and-paste, we quite often “transplant” parts of the graph between projects (e.g. data parsing pipeline or lighting setup).

Appendix

Short history and evolution of the interface

We started off in February 2017 inspired by seeing the user interface of Cables.gl even before they launched into beta. We knew we wanted Nodes backed up by JavaScript code that can be added, modified and compiled instantly.

In July 2017, after 5 months of work (on what was back then still a side project for our studio), we could create nodes from templates, program new ones in the browser, live recompile them with Shift + Enter and see instant changes, expose and tweak their parameters and manually add npm modules to be required in their source code.

It was so much fun to use that we immediately started using it for prototyping in our commercial projects. Despite common crashes, sporadical data losses and UI quirks, the productivity gain was tremendous. See Learnings and Observations for more info on how our workflow has changed after adopting Nodes.

As our graphs grew, we experimented with breaking away from the columns and grid-based interface so common in content creation tools. Beside the appealing aesthetics, it proved to be only minor improvement from a UX point of view as you still need your code editor, inspector and preview to be visible and accessible.

Data in our graphs flows from top to bottom; inputs are at the top of the node, outputs at the bottom and text labels in the middle. This way vertical space becomes very scarce. Borrowing some ideas from Houdini, we moved labels outside of the nodes to compress the graph vertically and remove constraints on the name length of the nodes. That layout allowed us to add node comments used to describe node behaviour but also peek at its internal state. We talk more about it in [Visual literate programming] section.

These are only few major iterations out of many many more developed both in code and manually in Figma. You can see the whole journey in this Nodes Themes Figma Board or ui-evolution.pdf

Future

We keep using nodes on a daily basis and actively work on making it better.

That said, Nodes is not yet ready for public release.

How do I get Nodes?

Subscribe to our newsletter to be first to know about public release.

Want to know more or schedule a demo? Drop us a line: hello@nodes.io

Follow @nodes_io