Interactive Devices

System Configuration

          |------|
          |  mp  | ---> D_out
          |------|  
             ^
             |
             |
             | 
             v
          |------|
A_in ---> |  mc  |
          |------|

In this module, we extend our previous configuration with analog input. We use the term analog input to capture the class of circuit designs that allow us to pull data from the real (analog) world into the digital world. Accordingly, we now include Ain (analog input) in our system configuration specification.

Digital vs Analog Input

In addition to analog input, we now also include digital input in our systems. In fact, the distinction between digital and analog input is somewhat arbitrary. Digital input is voltage reduced to either high (1) or low (0) value on your computation device. In practice, a high digital signal value is an analog signal value with a value near the top of the voltage range, while a low digital is a voltage signal near the bottom of the range. Similarly, analog input is voltage reduced to the space of the number of bits your computation device support on its digital pins.

From a hardware perspective, the Raspberry Pi includes a set of GPIO pins that allow you to interface with circuits. Without an ADC (analog to digital converter, for input) or DAC (digital to analog converter, for output), you cannot use the Raspberry Pi for analog signals - all signals are digital (0 or 1). This is not to say that you cannot use analog input devices, just that they will be reduced to digital inputs. At times this will be the most appropriate approach for certain hardware constraints.

For higher fidelity readings, you will need a ADC pin. This will convert analog voltages to a range of digital values. Every ADC has a different bit resolution - for example, the ESP32 has ADCs with a bit resolution of 12 bits - allowing for readings from 0 to 4095.

Digital input pins can also be used to read complex values beyond 0 and 1, but then require the use of a communication protocol. The digital GPIO pin will still only receive 0s and 1s, but the sequence and timing of these bits can encode more complex information. More information on communication protocols is available in this text, as well as online.

Application Domains

Digital Instruments

Interactive devices can be used to assist artists in performance. As before, our system will produce some output, but now we have a physical input that is visible to the viewer. This is a key component of digital instrument design, and interactive devices more generally. The ability of the viewer to appreciate the input mechanism allows the viewer to construct a model of virtuosity of the performer. In the same why the physical challenge of playing a violin communicates expertise, by seeing how the artist interacts with the interactive device, a viewer can appreciate the expertise the artist needed to develop in order to create the performance.

This issue of physically visible input also brings us to the problematic distinction between artist and device designer/developer. Should we distinguish between these two roles? And if so, who’s responsibility/right is it to make this distinction? By designing a creative embedded system, is the device designer now, by virtue of working in an artistic realm, an artist? In the case of a team of an artist and device designer, is it possible to assign absolute roles, and is there any value to such an assignment Is the concept of an identity as an artist or embedded system engineer inherently exclusionary (the purpose is to give value to membership of an group) or inclusionary (the purpose is to build a community around those who self-identify)?

This discussion is complicated by the concept of the observer. An observer typically has limited time with the device, and approaches the device without prior instruction on its use. This can lead no interaction behavior that pushes boundaries on the intended functionality - potentially leading to novel uses, and/or system failures. A device designed for the causal observer will implement a different design strategy than a device designed for the expert user.

We can make a clear distinction between casual observers and expert users of a device, especially in the design phase. A device designed for an expert user may have more functionality that is less accessible, with the understanding that it is the expert performer’s responsibility to gain mastery of the functionality. It may also be the responsibility of the expert, like the casual observer, to push the boundaries of the device’s interaction modes.

The video above shows an example of an expert device - the Mimo glove, as performed by Imogen Heap.

Multiplayer instruments

Digital instruments can also be designed for use with multiple performers. In the video below, the Data Duo synth demonstrates how the physical design of the instrument lends itself to multiplayer input.

Enclosure Design

Building the hardware components is only half the task in an embedded systes - you must also consider how your device will be packaged. An enclosure for a device can make the difference between a “fragile experiment” and a “functional product”. An enclosure defines the physical form factor of the device. In designing a physical form factor for a device, the artist implicitly suggests an interface and mode of interaction to users.

The idea of an implicit mode of interaction has been explored in the computer graphics community. In the image below from Lau et al. Tactile Mesh Saliency in SIGGRAPH 2016, machine learning is used to predict how users will interact with a device. Follow up work from Brahmbhatt et al. ContactDB: Analyzing and Predicting Grasp Contact via Thermal Imaging in CVPR 2019 continued to explore this idea as well. This prediction can then be shown to the artists during the device design stage to more quickly iterate designs.

When time allows, iterating designs with user feedback in the design loop produces devices with more usable interfaces. In so called pilot studies of your device, it is important to observe - not guide. This will allow you to see the natural interaction modalities user tend to explore on their own. To explore further, Ask non-leading questions to users. For example, ask “what did you expect this button to do?” rather than “did you expect this button to do X?” Ask “what do you want the device could do?” rather than “would you want the device to do X?”

A device may have multiple enclosures, either connected via wires or (in Module 4) wirelessly. Additionally, when we explore networked systems in Module 6, a system may comprise multiple devices. In this case, complementary form factors may help increase usability.

In designing an enclosure, it is also important to take into account where the enclosure will be open. Depending on the hardware being used and intensity of the computation, it may be necessary to leave openings for ventilation. When a system is completely sealed, heat transfer may be a problem - potentially overheating your components under periods of prolonged use. Some openings may also be left for wired connections, for example power or 3.5mm audio cables.

Documenting enclosures

In the presentation of a device, an enclosure is one of the most immediately obvious features to the viewers. As the vast majority of your audience is likely to only observe your device through the documentation you produce, it is worth investing more time in cleaning up documentation as much as possible. In documenting the enclosure, basic best practices for photography should be observed. As an example, photographing or video recordings should be (as appropriate) against solid color backgrounds to better call attention to the device itself. When documenting the device in action, extra wires should be hidden as much as possible. More information on best practices that apply well to this domain can be found in Picturing Science and Engineering by Felice C. Frankel.

Assignment

Due Feb 15

The second assignment is to place yourself in the position of an interactive instrument device designer. You are asked to build a “performable” device. Performable is broadly interpreted here and may include devices such as digital instruments, or game consoles.

You are placed under the following design constraints:

Two primary technical challenges you must overcome are 1) aggregating the sensor readings from both the Pi and the ESP32 2) manage switching between three states with only one binary switch

Deliverables:

Submit a link to your blog post on the course blog. That post should contain:

(10 pts total - see below for breakdown) A link to your git repository with a program that runs on a Raspberry Pi as well as the program running on the ESP32. Your system must meet the following criteria:

(10 pts) As always, the standard deliverables. The post should detail the creative vision of the device you have created. You should detail the technical challenges that you faced during the implementation of the device. Include a video of your device in action. Specifically, you must make a recording of you peers performing on your device, for whatever “perform” means for your device. The video should be recording following the best practice procedure for documentation covered in class.