Projects
Doctrine of Error
These images were generated with a pretty simple program. It takes an input image and compares it against the current painted image. The program generates a new brush stroke and the new error is calculated between the input image and the painted image, if the stroke reduces the error it is kept, otherwise it is rejected. Final image here
Tank Man – Painted with the Doctrine of Error
I like how this one turned out, the source image of Tank Man was a bit noisy and I think it interacted with the algorithm in a cool way. Final image here.
I wrote this program in Racket. I’ve been meaning to learn a LISP or Scheme based language for a long time. I thought this program would be a fun way to start. The brush strokes use a technique called Globs which basically connect to circles together with splines in a way that it has first-derivative continuity with the circles. It’s a nice way to generate brush strokes. Final image here.
It was inspired by the Create Applications of Deep Learning class and Ben Garney’s blog posts about writing a proof of concept video chat system. In both cases, the technique of defining, measuring, and reducing error is the core concept that makes it all work. Final brain here.
In the future, I’m planning to apply this to video. It’ll be straightforward to take the previous frame of a video and use it as the starting point for the next frame. I think the effect could be cool. I might also port a simple version of this to a more popular language and give a DorkbotPDX workshop on it. Final clocks here.
iOSBNIZ
iOSBNIZ is a port of IBNIZ by viznut to iPhone and iPad. This allows me to program little audiovisual hacks on my iPhone with minimal typing. IBNIZ features a VM with opcodes that are one character long, ideal for not hitting too many buttons. It also has a FORTH like stack which is a fun puzzle to play with. The VM calls the program for each pixel and pushes time, x, and y onto the stack. Your program uses these values to generate the visuals (and audio). You can also write a full program and ignore this loop as well.
iOSBNIZ is available on the Apple App Store. Download and have fun!
Source code is avaiable on Github
IOSBNIZ Features
- IOSBNIZ features a keyboard which is tailored for the OPCODE structure of IBNIZ. Characters are grouped together by function.
- Holding a button down on the keyboard will display some HELP to remind you what the opcode is.
- It keeps your last program for fast restarting of hacks.
- You swipe right to display your program without code and swipe left to bring the editor back.
- Swiping left will display help.
- Swiping left again will allow you to save and load programs.
- Shake the phone to switch between swipe mode and “U” opcode mode. In U opcode mode your touch location is sent to the U opcode.
This Is How We Disappear
I’ve been meaning to document my work on this project for about year now. TBA 2014 reminded me it is time to actually do it! So here it is:
For the past year and a half I’ve been working with a dance company called BOBBEVY. I’ve been creating graphics that go along with the dance performance called “This is how we disappear”. Here’s a review at Portland Monthly.
Jesse Meija was doing music and got me involved with this project, I’m very grateful!
Version 1
Effects for the first set of performances:
- A forest of trees created from a drawings by David Stein.
- Particle effects that mimiced the dancers movement.
- Particle effects that just fly across the screen
In order to accomplish this, I wrote a piece of software that would do the animation and handle tracking the dancers. There were two versions, the first version was used to perform a few times. Notably Dance+ in 2012 and as part of Experimental Half Hour XXXVII. It consisted of the following pieces:
- Cinder, used as a framework
- Freenect, used to interface to the Microsoft Kinect.
- Control, used to control the software from an iPad
This version worked ok. Before I started using Control, I had been triggering all of the sequences with keyboard commands. It worked fine, but I had to have a cheatsheet that told me what keys did what. Also, each command just mutated the state of the program, so if you triggered things in a different order you’d end up in different states. This made some rehersals hard, because it was difficult to return the graphics to a previous state. However, with Control, it became easier to use the software. BOBBEVY performed in Milwaukee without me and was able to use the software just fine! For Dance+ the Kinect refused to work in the studio, I think because the temperature in the room was so high. So I ended up “drawing” the dancers with a multi-touch interface in Control.
For the particle effects that followed the dancers, I ended up using blob tracking and distingugishing blobs based on distance away from the Kinect. I liked the stateless design because the dancers would move in and out of view of the Kinect and I feel that keeping track of them properly would have been a nightmare. This created some surprising benefits though. The swarms move between the dancer when their relationship to the Kinect changes and it created some really nice animations. Also, this piece has a lot of tension between the dancers and the particles ended up expressing some tension when the dancers were about the same distance away from the Kinect.
Version 2
Additional effects for the second set of performances:
- Static/simple projection mapped screens (similar to my Party House project).
- Realtime projection mapped patterns on the dancers bodies
This second version of the software was used to perform at NW New Works Festival 2013 and at TBA 2013.
For the second version of the software, I used the following new pieces:
- QTimeline, this is a timeline that allows one to control tweens of variables
- QuNeo, as much as I liked Control using a touchscreen while not looking at it (I had to look at the dancers for my cues!) is not ideal. A physical controller allows you to rest you finger on the button/slider you need to push without triggering it. (The cool kids pronounce it keeen-wah).
- LabMidi, used to interface to the QuNeo
- For TBA, 3, yes 3! Projectors. Two of them were used to cover the wide background, and one was used to project onto the dancers themselves.
The timeline solved many problems for me. It took what I used to have hard code in the application (fade times, animation speed, etc) and moved it to a data format. The editing GUI was nice to have as well. A new version of Cinder that made using multiple displays easier to use was really nice to have as well. I didn’t need to mirror my desktop screen anymore, which meant I could display debug and other helpful info on my screen. The QuNeo also allowed me to directly control ramp parameters which meant I didn’t need to rely on predetermined fades as much. This also allowed me to be more engaged with the visuals which was really fun. I think the trick to this will be finding the right balance between direct control and triggers to presets. It is probably the same balance electronic musicians search for.
The newest effect for the second run was the projection mapped dancers. In order to accomplish this, I was going to have to find the dancers with the Kinect and then project onto them as close as possible. I used the vvvv patch from here as a starting point to learn how to calibrate my projector with the Kinect. In the end, I wrote my own calibration code because it fit the setup workflow a bit better.
The projection mapped dancers worked pretty well. I was really excited to see them turn into just an indistigushable mass at moments and then turn back into dancers the next. I think this is what projection mapping should do: transform objects and confuse you, then bring you back to reality. I hope to do more of this in the future!
Party House
Party House is a projection mapped dollhouse that was quickly thrown together for the DorkbotPDX Open Mic Surgery event. It was written in a few days using the Cinder library. The source code is available here: PARTY HOUSE REPO.
It was super fun to perform. It wasn’t a complex or hard thing to do, but sitting up on stage pumping my fist and hitting space bar over and over again to the beat was fun. I’d definitely like to do more of this kind of “work”.
I had originally followed this Madmapper tutorial on mapping a building with madmapper and creating a video to go along with it in After Effects. It was a great introduction. I ended up running into problems in my case because the doll house had lots of overhangs and different planes to work with. Maybe if you’re projecting on a huge building, the building dwarfs the projector by enough that you can treat it as one plane. But in this case, I had to create a bunch of planes to make everything line up nicely.
This is also the first time I’ve used After Effects at all. It was nice to just make some videos and them layer them on the fly during my set. I definitely want to spend more time learning After Effects for future projects.
I ended up writing my own software to do the mapping because I’m a hacker and just wanted to. The code isn’t that great, but I do think the EditorViewport and ControlPoint classes are nice enough to re-use in future projects. I also updated the code to use the newest branch of Cinder, so it now supports multiple windows. It was also fun to start using C++11 features.
Church of Robotron
The Church of Robotron is a future looking group that is attempting to save the last human family. We did a large pop-up church installation at Toorcamp 2012. It had the following features:
- During gameplay Jacob’s Ladder and Sparker were running
- Fog machine randomly triggered
- Lasers fired in real life when enforcer shots were fired in game
- Rotating flapper near player hands spun when humans were killed by Robotrons
- Animated gif of your face at time of death in game which was displayed on a leaderboard in the other room.
- Bright LED flash on death, this allowed us to get a decent picture from the webcam on death and added to the players disorientation on death
- Kneeler base which detected players and controlled lights.
- Readerboard which displayed top player and witty statements
- Randomly shuffling sermon videos
- Lit totem pole
- Reading room which contained stickers, chick tract, and a zine.
I hacked MAME to expose debugging events (breakpoints, memory watchpoints) to clients via sockets. I also did some reverse engineering of the ROM to find out game events (deaths, high score, etc). To do this, I built upon the work of the great Sean Riddle. I also used OpenCV to capture the players faces when they died which were posted to the high scores page. We had about 15 people working on various pieces of the project. It was great to see it all come together and it was great to see people’s reaction to it!
Source code is available here. More tech details will be written at some point. I’m planning to write a quick post about mame hacking at some point soon.
BOBBEVY, This is how we Disappear
(Seek 19 minutes in)
“This is how we Disappear” is a 25 minute dance performance created by the BOBBEVY dance company. I used a lot of random technologies: Cinder, Microsoft Kinect, libfreenect, OSC, and Control to create realtime dancer reactive visuals to go along with the performance. We performed it quite a few times and my favorite time was at Hollywood Theatre as part of Experimental Half Hour. It was great to see the graphics projected so huge!
I’ve learned quite a bit about what’s required to do visuals for a performance from this project. It’s definitely influenced everything I’ve done afterwards. I’m hoping to do more of this type of work in the future.
Some post-mortem notes:
Using the Kinect is a performance space is a pretty harrowing experience. If there’s a lot of reflection or IR noise in the room, the Kinect can just die. I think there are some tweaks you can do in libfreenect to allow one to continue to get data, but it’s likely to be too noisy to work. I ended up creating a fallback that allowed me to “draw” the dancers with my figures with a Control sketch. But that method isn’t ideal.
I initially triggered everything with keyboard keys, but it became hard to remember what did what after a while (even for a powah user like myself). Have an iPad user interface was much nicer and allowed the BOBBEVY crew to run the show themselves when they performed in Milwaukee. In theory, this would also let me trigger the cues far from the computer which would give more options for where the computer and Kinect go. In practice, the UDP packet loss when doing Adhoc networking seemed to high to rely on this. It’d be nice to have a TCP transport as an option in the future.
I also added Bonjour discovery to everything, this made it much much easier to get everything configured before a show. I recommend this to people doing multi-machine work.
DorkbotPDX Exquisite Corpse
I contributed to this project. I can’t reveal details until we reveal the whole project!
(PS: We did complete this project)
The Missing Link
I created the initial firmware for the Missing Link. It’s an OSC receiver that outputs MIDI. The purpose is to hook up OSC clients such as TouchOSC or Control to older synths. The source is available here.
Read below for my post mortem notes.
This was my first “large” microcontroller project. It consisted of the following pieces:
- WIFI module
- MIDI in/out ports
- USB in (used to upgrade firmware, provide power, and did USB midi)
- Was able to route between all of the ports. Wifi->USB Midi or normal MIDI and all permutations.
Software wise we used the following:
- Arduino libraries (ended up stripping this down alot to fit the ROM space we had)
- Two libraries by my friend Alex: a MIDI library focused on microprocessors, and a bootloader that uses sysex as a data transport.
- uIP a tiny TCP/IP stack for microcontrollers, this was fun to hack on!
- Protothreads, was a great way to organize async code. It abuses C via Duff’s device, and I LOVE IT.
By far, the hardest pieces to get working was communication channels. First, getting I2C working between the PIC and the AVR we used was hard to get reliable. It was hard to even get working initially! I learned the hard way that people interpret I2C address differently. It’s best to spam them all.
Secondly, getting the USB midi code working well was a challenge as well. But it was fun to dive in and learn about USB.
Overall, I learned quite a bit from this project. If I were to do it again, I’d probably not use the Arduino prototype code we started with and go with straight C. The current codebase is a combination of different approaches. But as with a lot of projects, we were working under time constraints.
Processing Class
I gave a class on using Processing to create simple graphics. Also quickly “covered” using the OpenCV addon to detect faces. Notes, slides, etc are available here.
It was the first time I’ve taught a class in quite a while. I got a fairly positive response (from my friends, so they have to be nice to me!). I think I did a good job setting up mini-goals that took 20-30 minutes to accomplish. The idea was to give everyone little victories quickly so they’d be motivated to stick around and conquer more.
Morse Code Lights
One of my first electronics projects: Morse code xmas lights. Also here.