Computing with Lines
Maximilian Ernestus, 1st of March 2025Unhappy with your computer? Join me in blaming all those bytes for it.
Already convinced we should have geometric computing models? Feel free to jump to the meat↓ or try the demo in your browser (desktop only).
When the first finger traced sand guided by the intention to refer to something else, this was the first step towards modern information processing technology. Later came techniques like painting on a cave wall, carving into stone or wood, leaving marks in clay, writing on paper and many more.
The medium changed, but the principle stayed the same for thousands of years: a pointed tool leaves complex arrangements of traces which are charged with meaning. Those traces could be iconic (refer to something by similarity) or symbolic (refer to something by convention). In both cases, they are built up from line segments as the smallest unit.
Digital Traces
In the middle of the last century, we invented the computer which, as a medium, brings a fundamentally new set of capabilities. It can undo previous actions and remove information without leaving a trace, the information density is much higher and information can be replicated and distributed at almost no cost. On top of that, it can transform the stored information by itself much faster than any human ever could . At least for a certain class of transformations. Consequently, the computer became the dominant tool for information processing, especially since people started to carry one with them 24/7 about 15 years ago. There is no doubt, that computers transformed our lives, how we relate to the world and to each other.
The underlying smallest unit of information, however, is no longer a line segment. If we look at various models of computation and their models of memory, they all fundamentally rely on symbols in cells:
The Turing Machine has an infinite tape with symbols accessed by a wandering head, the von Neumann architecture has a flat memory where any cell can be accessed by an address, stack machines operate on the top of one or more stacks and cellular automata operate on a grid of cells, where only neighboring cells can be accessed. As these models of computation are inherently symbolic, they require agreed-upon meanings for symbols. Early defaults, like ASCII, favored English due to its US origins and 7-bit limits. UTF-8 improved inclusivity, but Unicode remains dominated by US tech giants. This is likely just one facet of the broader cultural biases embedded in technology. Both of those standarts have a heavy bias towards text instead of spatial modes of computation.
So why would this matter? To you? From a technical perspective, this seems like a small implementation detail, that should only concern the people who write assembly code or who write software that generates assembly code: The people that work on the lowest levels of our tech stack. However, this technological bias bubbles up throug all the leaky levels of abstraction and eventually affects all people that invited digital technology to the most intimate parts of their lives: if you read this, that is probably you and also all of your friends.
To understand this, lets take a look at design theory, which states that there is an ongoing dialogue between the designer and the material. Through its affordances, the material suggests or limits possible actions Pye, David. "The Nature and Art of Workmanship", 1968. . Embodied knowledge is gained through hands-on engagement with the material Sennett, Richard. "The Craftsman", 2008. . The material is even seen to have some sort of agency Bennett, Jane. "Vibrant Matter: A Political Ecology of Things", 2010. . A chair designed for production in steel looks different from a chair made from wood, which looks different from a beanbag. And a dinner with your friends will get a different quality whether it is at a glass table with plastic cups and cardboard plates or at a wooden table with glassware to drink and earthenware plates, even if the shapes of the objects are mostly the same.
I think the same applies to computing technology. The memory of the computer, the long array of bytes, make up the fundamental computing material. It influences the language design, which influences the libraries and frameworks which in turn influence the application software we write and ultimately how it feels to use our digital devices.
The disconnect many feel from their computers stems from alienation from the material they work with.
Unlike with physical tools, where material properties can be directly perceived and manipulated,
digital information must undergo layers of transformation before appearing on-screen.
Our vast memory capacities mean only aggregated views or a tiny peephole—often less than 1% of memory—is visible at any moment.
Worse, the actual ground truth of this data may not even reside on the machine itself
but be scattered across somebody else's computers
, raising the question of ownership.
Especially among developers, this alienation has led many to favor plain text formats, believing they provide a more direct connection to the medium. Code, typesetting (LaTeX, HTML+CSS, Markdown), and even graphics (ASCII art, TikZ, SVG) all reflect this tendency. Those text-based formats ultimately shift the burden of rendering onto the human mind. Instead of directly manipulating content, one must mentally simulate what the machine should be doing.
I think we need to talk about the fact, that there is only one flavor of material in digital information processing. And we need to talk about how that material feels Because the people working on the lowest level of the tech stack are mostly male and rather talk about how hard their achievements are. See A Case for Feminism in Programming Language Design by Felienne Hermans and Ari Schlesinger. . and question if it provides the right affordances for what we want to achieve with it. For example, I see many people longing for a more visual way to engage with their computer. They build exciting software for visual programming, large canvases where you can freely draw and write and organize your information and even prototypes on how you could ad dynamic behavior to your drawings. But by relying on the inherently symbolic memory model and only operating on the application level, they go against the grain of the material. Thereby, they deny themselves the opportunity to be guided by the material through dialogue. They have to design everything in their head and then force their vision upon a stubborn, inadequate material.
To contribute to this discourse, I propose a new memory model. Along with that comes a piece of rhetoric software to simulate this memory model hoping that you can get some a hands-on experience with a new computing material.
A Line Segment Computer
What if we allow ourselves the luxury to compute with geometric objects instead of the old and dusty bytes? Here, we take the path of visual computing, which is to be distinguished from the existing explorations into visual programming!
Note: what comes now is inspired by cellpond by Lu Wilson but with line segments instead of cells. It is also inspired by Viewpoint by Scott Kim but with line segments instead of pixels.
Memory Model
I propose the following memory model: a bunch of colored line segments on a canvas.
If that seems too much like a toddler Toddlers are fine. Nothing against toddlers here. playing with crayons, here is a more formal definition.
The memory of the computer is defined as the set of line segments \(L\), where each line segment \(l \in L\) is defined by a triple \((a, b, T)\) with \(a, b \in \mathbb{R}^2\) and \(T\) is a finite set of tags.
To me, this seems like a decent compromise between a point, which would be too close to a pixel and a triangle, which would be make too many assumptions about the kind of objects we want to operate on.
Tags
We need to distinguish different categories of line segments by coloring them,
placing them in different layers, effectively grouping them.
The group is determined by its set of tags \(T\).
For any practical purposes you can think of the tags a short strings.
For example, the groups in the above example could be:
{"letter", "y"},
{"freehand"},
{"cursor", "eraser"},
{"counter", "blu", "tmp"}.
Queries
The memory is not addressed with an index,
but using queries on the properties of the line segments such as their
length, orientation, position or tags.
For example, a query on the above memory could look like (T("freehand") & AX < 50) | (T("blu") & AY < 20) and result in:
We selected all line segments with a freehand tag whose \(a\) handle is left of 50 and all lines tagged tmp whose \(a\) handle is above 20.
Memory Modification
The basic operations to modify the memory are
ADD(segment)to add a new line segmentREMOVE(segment)to remove a line segmentTAG(segment, tag)to add a tag to \(T\) of a line segmentUNTAG(segment, tag)to remove a tag from \(T\) of a line segmentSET_A(segment, a)to set a new start point of a line segmentSET_B(segment, b)to set a new end point of a line segment
For convenience, there are also operations that take selections instead of segments:
REMOVE(selection)TAG(selection, tag)UNTAG(selection, tag)
Programs and Processes
A program is a pure function that maps the memory state \(L\) to a set of memory modifications. The modifications are subsequently applied to obtain the next memory state . I am aware, that I don't specify how those functions are evaluated. My focus is on the memory model right now. In my draft implementation, programs are Lua code snippets that I evaluate in a fresh execution context to avoid any possible side effects. Later revisions might also include a way to directly encode programs as line segments. In the conventional understanding of a computer process, it executes the program once on some input and then terminates. In contrast, I envision processes as the repeated cycle of evaluating the program function, applying the changes, applying the program function and so on. The process has no state in itself. To memorize something for the next iteration, the information needs to be stored in the form of line segments in the memory.
We can run multiple programs in parallel: in each iteration we evaluate their function, create the union of their change sets and apply the resulting union. Note, that the change sets can contain conflicts. When two processes change the same line, it is undefined which change takes precedence . In my current single-threaded draft implementation, there is an order to the programs. The "later" modification overrides the earlier modification and I warn about the conflict. Conflicts only arise when the same tag or the same end of the line are modified. So it is possible the e.g. one process adds a tag to a line and another process changes the endpoints. How to handle and even resolve conflicts in detail is still an open question for me.
Input and Output
User input like mouse, keyboard, sound, sensors or audio comes via line segments, that reflect the current state of the input devices. On each iteration, before programs are executed, line segments are added, removed or updated accordingly.
The mouse is represented as a line segment tagged mouse with
\(a\) set to the last mouse position and \(b\) set to the current mouse position.
The mouse keys are represented by line segments with the same geometry tagged {mouse, key, [left|center|right]}.
Tags like down, clicked and released are added depending on the button actions.
Each key on the keyboard is a line segment, tagged {keyboard, key, [letter]}.
Key line segments are tagged down when the key is held down, released when just released and pressed for each time the key was pressed.
The position of those segments is arbitrary but arranging them like on a physical keyboard seems sensible.
The sound recorded since the last frame is represented with one line per sample tagged as audio.
The lines are connected such that they form the amplitude curve of the sound bit.
This might not be the most efficient way to encode the information, but when viewing the raw memory, it provides for the most natural interpretation.
For output, lines are rendered to the screen. Other options such as audio out, controlling actuators, printing or even network streams seem sensible, but I did not work on them yet.
Try It!
I wrote a first draft implementation in C++ with SDL2, Dear ImGui, Lua, Sol and compiled it to WASM. Try it here now. Highly inefficient and buggy ...