Escher Circuits: Using Vision to Perform Computation
source link: https://esoteric.codes/blog/escher-circuits-using-vision-to-perform-computation
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
Escher Circuits: Using Vision to Perform Computation
In 2008, Mark Changizi, a Sloan-Swartz Fellow in Theoretical Neuroscience at Caltech, noted that the eye can make sense of complex relationships that often mystify the brain.
Our everyday visual perceptions rely upon unfathomably complex computations carried out by tens of billions of neurons across over half our cortex. In spite of this, it does not “feel” like work to see. Our cognitive powers are, in stark contrast, “slow and painful,” and we have great trouble with embarrassingly simple logic tasks.
Might it be possible to harness our visual computational powers for other tasks, perhaps for tasks cognition finds difficult?
In other words, could we trick the eye into performing computation?
Changizi proposed the Escher Circuit, "a special kind of image that amounts to 'visual software' our 'visual hardware' computes" merely through perception.
Escher Circuits work by creating different states corresponding to how the eye resolves ambiguous images. Its core component, the wire, holds a state of 0 when we perceive it as if we are to the right of it; otherwise, it holds a state of 1.
Although we can "flip" the wire visually back and forth in our head, it is difficult to hold both views at once (you can flip it easily by glancing at the 0 and then the wire, and then the 1 and back to the wire again).
Changizi combines visual clues that establish one reading of the wire over the other. The strongest visual clue is the input of 0 or 1, a box with no ambiguity in its visual presentation:
To perform computation, we need to implement logic gates. AND and OR gates use transparency to favor one reading or another (OR favors 1, AND favors 0).
Because transparency is a less strong visual indicator than the blocks, it takes two 0s to override the OR, etc.
As soon as both 0 and 1 blocks exist in our circuit, it becomes a paradoxical image with two distinct readings. Our eyes choose one reading over the other based on this contradictory evidence.
Below is a feedback loop that oscillates between 0 and 1. The block to the upper right of the circuit is transparent, allowing it to flip between the two states. Note the NOT used here: the bottom half of the block has a plane just below it that indicates a turn of direction, amplified by the "ambiguous prism" below; the charge then circles back around, up the left side of the circuit and into the block again.
The ambiguity Escher Circuits rely on is in our reading of depth signals. This is not the only choice, in fact it took hundreds of attempts to come up with a design where state can be followed through a circuit.
It never even occurred to me at the time that I'd end up with more Escherian stimuli. I went through so many different styles of stimuli, and within the vision science world, Escher is just one of many.
Here are some of the other considered designs:
In the upper right (which resemble Jan Robert Leegte's pastel drawings), we have squares that can be read as "in" or "out"—but which lean to one reading, as we tend to see illumination as coming from above, a reading reinforced by decades of UI design.
Most of these were rejected for the bleeding problem, a lack of wire insulation: reading one part of the circuit tends to jump across to nearby wires it should not directly affect.
This becomes a problem for Escher Circuits when they become large enough; some people will have trouble following the XOR gate below, which requires remembering the state of several different wires at once.
Note here the NOT gate with the cone, different from the oscillator example, but a change of direction that we can still take in at a glance:
Changizi posits that, for more complex circuits, smaller or more specialized visual circuit components may be needed to make them viable.
[I]t can be difficult to 'perceive one's way through' these visual circuits, although I have found considerable personal improvement with practice.
However, even if practice is needed to read larger circuits, they don't require understanding special symbols or knowledge of logic gates to follow, relying on our perception to unlock. Whether this can be expanded into more complex scenarios is still an open question.
People are notoriously poor reasoners, whether in the probabilistic domain or logical domain, something I have also personally witnessed when teaching logic and computer science... Although our reasoning and logic powers are poor, we are all walking around with computers in our heads that are far more powerful in many respects to any computing device ever built, or likely to be built in the foreseeable future.
Changizi's original paper (2010) can be read here, and a more concise write-up for Perception Journal (2008) is here.
His website has info about his current work as a theoretical cognitive scientist working on unifying theories in biology, psychology, and the structure of society.
h/t @emollick
Recommend
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK