Recent Forum Posts
From categories:
page 1 of 212next »

This looks quite interesting- basically a video based mocap system for hands and feet!

Would be good to use two of these to do basic (and very cheap) 3d mocap!.. also opens up the minority report type interface!

wow, thats amazing! I wonder if its one of those cases where brain function is impared leading to literally nearby brain functions growing, much like savants a la rain man. There was a case of a man from liverpool who had a burst blood vessel whilst on the toilet and after he recovered, he became obsessed with painting. His paintings were terrible, but his creative urge was unstoppable!

Re: ideas around interface by AdamVannerAdamVanner, 17 Aug 2009 15:11

I think that analogy's not bad. In addition though, considering the amazing capabilities of the brain, for example its plasticity, it may be also the case, that if one part of the engine fails, another would be able to pick up some of its function. Such as, the lights don't work and the stereo will support some of its circuitry. (sorry, I'm not very good with cars!)

I have come across research, where it seemed for example that the visual cortex lit up during certain activities, even though the person was blind. It is also known that the auditive and haptic system in the case of blindness can become more sensitive and attuned..

the question is, for us, do we need to go down to the reductionist parts, or can we use tools such as your mocap project to see what works and draw conclusions for salient design? (and hither perhaps also the theory of mind)

Re: ideas around interface by moohnamoohna, 14 Aug 2009 11:15
Re: ideas around interface by moohnamoohna, 14 Aug 2009 11:00

He did use perspective, that's the crazy thing! It was a New Scientist article I think, I have it somewhere as well. He also used colour, merely from what people have told him. So some things were learned, but even the fact that he could learn them is astounding.

Re: ideas around interface by moohnamoohna, 13 Aug 2009 18:43

wow - what kind of representation did he use? presumably perspective was out the question. Was it just outlines of forms? I can imagine how someone without sight could be an excellent sculptor, but one would guess that the process of reducing from 3D to 2D would be as difficult as us understanding what a 4D object looks like!

Re: ideas around interface by AdamVannerAdamVanner, 13 Aug 2009 17:51

Have you read about that man who painted (naive) landscapes but had never seen? I will see if I can find the article. the interesting thing about his not very good (terrible) paintings is that they do look like what we see, in the same way as childrens' pictures do.

Are you saying, by the Paterson quote, that the field is so subjective that its akin to trying to acertain the appearance and construction of a car's running engine by sound alone, without being able to open the bonnet? It seems to me that this analogy fits because if collecting even the most basic information about sense with very clearly defined problems isnt clear cut, how can we expect to understand the mechanisms which arent clear? i.e. even if we can open the bonnet and see the engine, we can see some of the structure, but the vast majority of the working parts are much more difficult to see.

Would, therefor, other methods of scientific evaluation, such as experimentation in insects and mice be more fruitful? i.e. understanding from the "circuitry" upwards, rather than the effect downwards? (or am I going MASSIVELY off point here?!)

Re: ideas around interface by AdamVannerAdamVanner, 13 Aug 2009 14:36

Oliver Sacks recounts a very similar story in "An Anthropologist on Mars" (To See or Not To See), throwing up very interesting questions to the necessity to 'force' everybody to see/be like the norm (he quotes the Gregory case extensively). It brings up the idea that 'curing' for any price may not be the desired cause of action. This blind man was actually encouraged to have the operation by his wife before their wedding, but became very upset as you say with not being at home in either world. He happily worked as a masseur for years, and then became upset at the sight of everybody's bodies, so he shut his eyes again…! His sight was not restored without problems, at at times he would choose not to see, or not be conscious of seeing.

There are differences between congenitally blind and having some sight for some time, or some residual sight, and a lot of early research failed to control for this in the early days of the first catarac operations. And I think the the question is not as simple as a translation from vision to haptic. Blindness and sight need to be defined to a degree, what does it mean exactly? And individual accounts of experience are that: individual reactions to individual 'therapeutic' situations.

Does the fact that the Alpes are visually as impressive as the haptice experience of them mean, that the sensations are equivalent? Can dimensions such as magnitude, distance, perspective, figure be directly translated? The question is what purely haptic spatiality really 'feels like'. And how do you talk about them?

"However sophisticated the questioning, qualitative inquiry is hampered by the attempt to gauge empirically the sensory contents of another person's consciousness, and this problem persists in current examinations of experiences of blindness" Paterson,M (2007) The Senses of Touch, p.43

Re: ideas around interface by moohnamoohna, 13 Aug 2009 14:07

Surely the question regarding "a congenitally blind person, after a catarac operation" has already been answered, hasnt it? or at very least there was the case of the man who lost his sight at the age of 3 (those 3 years may well have been crucial, though) and then had an implant at the age of 60. He could recognise the lathe he used but only once it was touched. Conversely, though, when skiiing, seeing the alpes for the first time, he described them as "just as impressive visually" meaning that his understanding of their topology, merely from years of travelling on roads, feeling the rock and snow, was surprisingly equally as concrete an "image" as the one image that entered his eyes, and could impress in a similar experience to that of a spectacular view.

Weirdly, and sadly, though, he was dissapointed about the appearance of his wife, and decided to return to blindness!.. I dont think for that reason, but it appears this is a common problem when the blind return to sightedness in that they are masters of neither sense, and so become very confused and depressed. This is a very interesting phenomena because it shows us that overloading senses results in a net reduction in the overall perception.

Re: ideas around interface by AdamVannerAdamVanner, 13 Aug 2009 12:15

Thanks Adam, for some excellent questions. You could be my supervisor?

I am presently wading through Merleau-Ponty as I am sure he has something to say in way of answering that (from a phenomenological standpoint) - or at least, positioning the questions somehow. But although I have now read the English AND German translation, I struggle to put it into my own words. He refers to this fixation as 'palpation with the eyes' for example. I think I may try to write something and put it up here somewhere (well, i have to get it in my head and write about it anyway). I think it is going to be something along the lines of Wanda's answer, I have a feeling.

You are touching on a number of interesting questions here, that have been debated philosophically for centuries, and continue to intrigue. I think I mentioned before that this is one reason I am excited about the construction of multisensory interfaces: the deconstruction (which is necessary for the computer processing) forces one to think about the synthesis, but it also enables us to research the whole, artificially put together, or with some 'channels' disabled.

maybe a quote from Paterson to start off with (yes, my External Examiner)

"First, the assumption of an equivalence of the senses, substituting hands for eyes, touch for sight, is fundamentally to ask whether sensory perception is straightforwardly cross-modal (or inter-modal, sensory information being transferable from touch to vision) or actually amodal (sensory information prior to its processing as specifically audio, visual, tactile, etc.)" Paterson, M. (2007) The Senses of Touch, p.38

"These observations would entail a modified answer to the Molyneux question, where what is commonly known as 'visuo-spatial working memory' relies on 'mental images' in the blind that are neither specifically visual nor spatial, […] and so memory allows the transfer of information between modalities. But at some level there is convergence between the sensory modalities, Graven argues, so that a 'cognitive vision-touch link is derived from converging subsystems'. In terms of the absolute pragmatic experience of working memory his evidence suggests that memory alone is not the intermediary of cross-modal transfer. Instead, there are underlying encoding processes at a 'lower cognitive level' than memory, encoding experiences from different sensory subsystems. In other words, while memory might allow cross-modal transfer between modalities such as vision and touch, perception is more amodal than cross-modal." Paterson, M (2007), The Sense of Touch, p.45

The Molyneux question is the question whether a congenitally blind person, after a catarac operation, would be able to tell the difference between a sphere and a cube, by vision alone. This debate was started before catarac operations were possible, but is still not clearly answered now.

PS I suppose I will open a new thread for this somewhere, under Crossmodality/Synaesthesia, or Holistic/Embodied Interaction maybe?

Re: ideas around interface by moohnamoohna, 13 Aug 2009 11:29

I think you have some lovely ways of looking at this, Wanda, and notion of an interface which is unclinical seemed to be a theme which existed during the lab as well as you thoughts here. We are (forced to be) accustomed to screen based interfaces where buttons need to be pressed in a particular, logical orders with spacial accuray. Again, the button is either pressed, or it isnt. I've always thought that some of the most interesting work is done when a tool is subverted and used in a way which was not intended, and more often than not that comes from happy accidents. Perhaps making an interface which is flawed; one which makes "mistakes", in a way which is personal may be a way of facilitating this. There, of course would be a balance: every sharpening of a pencil is slightly different, and most of us get used to cutting it slightly differently, but if the pencil breaks it frustrates us. It could be that we develop a Graphical User Interface, for example, where buttons are not "on" or "off", but "slightly on" - whats known as "fuzzy logic". Again, this would relate very clearly to gestures, where the way a gesture is peformed is integral to the nature of the gesture.
One example of combining precise results and expressive gesture might be sign language. A word is a word is a word, but in a SL, more stress and emphasis can but on words by the signer. Clearly, this in the case in speech too, but not in written text. Could this even be applied, then to something as mundane and exact as word processing? A little like the spoken word piece in the video I posted above where louder words are shown on a screen as being larger?

It's interesting how you listen to music as a whole, and you are probably right; its most likely the music washes across you ands its impossible to break it down. Some music is so penetrating, its inescapable. I think what I'm considering all the time is my project idea of interpreting sound into movement, and based on the concentration which was clear on Olu's face during the demonstration in the mocap suite, I had presumed we can only focus on one, perhaps two sounds at once. I had hoped that as we relax and subconciously begin to understand the patterns at work, the movement would happen more automatically, and with less intellectual demands; i.e. the body takes over. Like a dancer dancing to a piece of music they havent heard for the first time. This all remains to be seen, but perhaps interestingly from what you are saying, that the mere influence the sound has on the soul would be enough to spontaneously evoke particular movements within the dancers.

Of course, I was under no illusion that I/we were to be the first people to try this:

Yes, about the falling backward onto broken glass: lets not try that as an exercise at the next lab ;)

Re: ideas around interface by AdamVannerAdamVanner, 13 Aug 2009 11:10

Wow, you have a very abstract process- very interesting to see!.. once weve got a few more people's words we can put them through wordle, and perhaps after then, we could try adding technology words, even specific software that relate to our work.

Over baring means I cant spell overbearing correctly ;)

I meant to write:
obviously, the trival or unimportant words are overbearing

Re: Manifesto process by AdamVannerAdamVanner, 13 Aug 2009 10:14

Hi, I love the idea of the gestural interface, it makes me think of the impossibility of expressive mark making with any package I have tried. The idea of something, tentative, fuzzy, accidental or variable in a way that implies touch, in a visual way as well as in sound, is wonderful and part of something I was trying to say to you about drawing on computers Adam. The haptic not just in touch, but in the way we experience looking at a drawing or painting, imagining the bodily movement to make it, in particular the movements and touch of the hand/s, just as we imagine it when we hear a piano for instance, the touch of the hands, fingers on the keys. Part of some of the digital art I made was trying to get away from that untouched by human hand look. To put the homemade into it, to subvert the way it was making me work.

But is this still just trying to make it like a human, instead of being open to its own peculiar possibilities, that might be more interesting? Maybe trying to do both at the same time is the answer.

While I am talking about trying to do two or more things at once, I am not convinced about those ideas about only experiencing one thing at a time, one instrument constantly being picked out for a start: For me when listening to a lot of instruments at once the hard thing is to pick one out, unless the music is doing that on purpose. It sounds like a complex whole, but it does vary, we do concentrate on parts, and filter out for example coughs and door slams, it's not flat. And when I feel a grape or a plum with a stalk, I am sure it is the complex experience I am remembering, the contrast of smooth and full with thin and harder and sticklike. I think it's computers that only do one thing at a time - or is that what you were saying? Isn't that one of the ways our sight is always different from what a camera sees? We edit and pick, but not just one thing at a time, and it includes all those decisions assumptions and additions that our brains make that enhance what it is that our eyes see, so that we see texture and depth for example, and imagine forms that may not be there.

Not sure about pain, that might indeed be more focussed. I shall have to pay attention next time I fall over.

Here they are, it's 13!?

respond, record, compare (x 2 and/or more), repeat, repeat, focus, analyse, forget, remember, present (production values VIP), revise, archive (method), learn (welcome suggestions and challenges).

I hope these aren't too abstract? They are actually what I think I do, in my practice where it interfaces with technology, but they don't say what I currently do it with, because presumably we are hoping to do it with something better in the future?


(Adam, what does over baring mean?)

Re: Manifesto process by Wanda ZyborskaWanda Zyborska, 13 Aug 2009 09:16

Mmmm more very interesting stuff!

…so he invented mirror therapy! We have a quite well known neurosurgeon and author, Jonathon Cole who visits us from time to time with absurd numbers of ideas on using mocap for sensoury purposes, VR solutions to limb pains which he is developing with NASA to go beyond the mirror solution etc etc. His work is all about tricking the mind and nervous system, in particular those with spinal injuries. His book "Still Lives" is a very interesting read.

Do you have any information about the maximum dimentiality of salient interfaces that are possible between particular senses and scenarious? When I touch something, such as a grape on its stalk, I'm aware that its a grape, and its still on its stalk, but when I try to feel the texture, I can really only concentrate on one aspect at a time. The same I would say is true of sight and sound. Thinking of listening to music and "tuning in" to one instrument or another, but never all. This of course doesnt mean that I'm not aware of them, but is it true to say that long term memory of a sense has to be concious, and focused, and therefor serial and linear? Do we experience our senses less meaningfully and memorably when they are all stimulated in parallel? Clearly, if I fall backward and reaching my hands out, they both land on broken glass at the same time, do I feel it in both hands in parallel, immediately, or does one signal always get processed first, even if only for a split second? If I can only recognise the pain in one hand at a time, presumably, that doesnt affect my long term memory; in my mind, I remember the eventual effect on both hands equally.

Just interested whether multiple "buba"s and "kiki"s, or their equivalent in other senses, can be considered at once, or whether they require some serial core left brain or intellectual activity and therefor can only be considered one at a time.
Are structure providing expected patterns the only way around which changes in sense can be multi-dimentional and multi-modal, but equally meaningful, such as dancing in time both to a beat and to a partner?

If they must be intellectually processed, I cant see how my project could work since it would be much more complicated than patting your own head whilst rubbing your tummy, especially as that's an exercise that can only work if relaxed.

Re: ideas around interface by AdamVannerAdamVanner, 12 Aug 2009 22:15

I've found a website called which creates word maps such as above.

Here's an example using all the text I could find within the wiki, including our biogs:


obviously, the trial or unimportant words are over baring

Re: Manifesto process by AdamVannerAdamVanner, 12 Aug 2009 19:37

Good idea, I'll have a crack at it as well!


Re: Manifesto process by moohnamoohna, 12 Aug 2009 16:39

Yes, that's exactly what I mean when I refer to 'salience'. In a salient interface, the cause and effect relationship needs to be clear enough for people to be able to use the interface without lengthy explanation or huge cognitive effort, just to find the right 'button'. It seems that to use metaphors that occur naturally, is a good source of inspiration. For example, a vibration getting rougher, communicating the urgency of a text message (rough = signifying urgency). Or a busy newsgroup transmitting a 'buzzy' vibration (buzzy = signifying busy). These are in my eyes natural mappings, that have been intuitively used to design, and can be intuitively read. However, it is necessary to deconstruct these somehow and be aware that it is no coincidence that these mappings are salient.
I am rather interested in the deconstructing of such successful mappings and exploring the context behind them. That's pretty much what I'm battling with, mainly in terms of the body being the foundation of experience and providing certain schemata of internal representation, which could be crossmodal or even amodal. I.e. when the sense data comes in, what happens to it, and when does it start interacting with the sense data from the other organs?
Then, how do we research these relationships and how can we design for them? In fact, it looks like I'm trying to re-write my thesis from that perspective mainly. That's why I'd be rather interested to see what you find with your mocap sound/body synchronisation tool.

Another thing that your post made me think of is the philosophical models I've been using to classify Haptic device design. On the one hand, you have the physical-sensory category, where touch is seen merely as a sense impression, in quite a utilitarian way - sense data, which can be broken down into bits and reproduced digitally/mechanically. On the other hand, you have they psychological-humanistic category, where touch is seen as a way of establishing or maintaining a connection, the sense data is merely the basis on which the intended message will travel. I suppose you could call it more expressive and capable of evolving.
I agree with you completely that gestural interfaces lend themselves particularly well to musical expression - hence this is where a lot of experimental stuff has been happening. I like the idea of charting what other areas/applications might be able to benefit from this. Of course, in terms of my own work, this would be interpersonal communication.

In terms of "mouaba" and "takenaka" that's very interesting and just like Ramachandran's "kiki" and "buba" - here is a very interesting talk he did on synaesthesia..

Oh, but there's also a talk on your favourite site..

Re: ideas around interface by moohnamoohna, 12 Aug 2009 16:37

Another TED talk, this one from artist Golan Levin which is well worth a watch regarding interface and interaction. Some really interesting ideas:

"the mouse is the narrowest straw you could try to suck all of human expression through"

Some really interesting work on synethsesia; a subject which came up a few times during the first lab. I suppose whether an interface is to be used for public interactivity, productivity, or fun, there needs to be a synethetic element where responses are natural, transparent and predictable. The example of the "mouaba" and the "takenaka" are really interesting becuase they show that the metaphors for interaction can be quite abstract. I think I mentioned something like this in relation to sound when we were in the mocap suite; that the type of foley sounds used in martial arts films reflect a physical movement: we dont need to think too hard to correlate between audio and visual if the "shape" corresponds.
Gestural interfaces have been in existance for many years, in particular on tablets and touch phones / pdas dating back to before 2000, but no one, that I know at least, uses them. Why? perhaps its because all gesture systems are merely fluffy keyboards, meaning that the user may have to try a few times to perform what could be achieved with one keypress. They are not "on" or "off" like a mouse button or key press, and the gesture is either correctly performed or not, so they are an expressive analogue interface which is stripped down to a bare 0 or 1. Sadly then, gestures, in the eyes of developers, will be used to open web browsers; the web browser is either open, or it isn't.

Perhaps, then, the "gait" of the gesture needs to be more important than the gesture itself, allowing an infinately more expressive and detailed interface than the button or slider… but how can any application handle nuanced gestures? Why have so few developers tried? Do we need to classifiy tasks or software which is gesture suitable such as music software, and that which isnt such as word processors?

ideas around interface by AdamVannerAdamVanner, 12 Aug 2009 15:42
page 1 of 212next »
Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License