Mixed is a creative learning initiative which has an aim to teach colour theory to year one students with a particular focus on colour blind and visually impaired users. This aims to be achieved through the adoption of a pressure-based mat. Mixed allows users to draw and create artwork on the mat's canvas through utilising pressurized colour pads. Standing, jumping or, in some other way, applying pressure to a combination of colour pads is able to mix the selected colours together. The design hopes to promote experimentation and creative freedom regardless of colour perception abilities. The colour mixing process is contextualised in the Mixed's audio feedback for colour blind and visually impaired users. Mixed converts the frequency of light into sound waves which has been done with the intention of allowing users to hear colour. A demonstration of Mixed can be seen below:
| Problem Space
Team Twisted has recognised that creative learning has the capacity to facilitate fundamental factors in a child's cognitive, emotional and social development. However, there has been a noted decline in the creativity integrated into the early childhood curriculum over the past 25 years (Core Knowledge Foundation, 2015). Further, my individual design direction noted that there is an apparent accessibility issue with the current educational methods for teaching colour theory to colour blind and visually impaired students (TeachThought, 2019). With that being said, a problem space pertinent to the Australian Curriculum's lack of accessibility considerations given to colour theory learning initiatives became apparent.
| Solution Space
There has been an identified need for a creative learning initiative to be designed in response to the established problem space. Team Twisted is working under the collective domain which stands as a creative learning initiative to teach colour theory to year one students through active engagement and open-ended interaction techniques (Cheung, 2010). Additionally, the digital solution should also facilitate the cognitive benefits of experimenting with colour theory for colour blind and visually impaired users (Ward, 2006).
There is no right way to use Mixed. Mixed is a design which aims to facilitate total creative freedom through an open-ended interaction approach and innovative physical engagement principles. The multi-tiered feedback allows users, with varying accessibility requirements, to be engaged by the immersive and playful design. Users can focus on drawing complex artwork, experiment with colour mixing or even generate their own musical composition. With that being said, the intended benefits of Mixed can be loosely categorised under three main interactions: Hypothesizing, Mixing and Drawing. This design decision has been made with the intention of promoting creativity, experimentation and logical deduction abilities.
| Hypothesize
The first step – hypothesizing, ideally promotes the users to begin employing a scientific and experimentational mindset. This, in essence, means the users should be predicting the quantity of each colour that they need to mix together in order to achieve their desired output. So, for example, if the user wanted to draw in pink, they would need to hypothesize which of the colour pads they will need to select as a means of mixing their 'goal' colour. To keep this pink example running, the user would ideally hypothesize that the red and white colour pads would need to be selected from the mat.
| Mix
After hypothesizing, the user then moves onto the mixing step. The physical interaction for the system's colour mixing takes on a similar approach to the popular children's game Twister. Users will need to stand, jump or, in some other way, apply pressure to their hypothesized colour pads. Any colour pad the user has selected is stored and mixed in their drawing-colour buffer. So, if the user was to select red, they would hear the relevant audio feedback and red would be added to their drawing-colour. If the user then selected white, the drawing-colour buffer would mix red and white together and the user would then hear the audio feedback for pink. If users want to draw in or mix a new colour, they can select 'reset' and begin this process again.
| Draw
In this final stage the user is able to see if their generated colour aligns with the hypothesis they established in step one. In the ideal final form, users would draw in the mat's centre. However, this form of Mixed is utilising the touch screen of a tablet which stands as a small-scale version of the canvas. This is where the user receives visual and audio feedback regarding the success of their experimentational approach. This feedback takes the form of paint which is reflectove of the colour that the user mixed. This consequently aims to build on the user's experimentation and logical deduction capabilities.
Several components contribute to the current form and functionality of Mixed. Such components include the colour pressure-pads, the Arduino, the colour mixing processes, audio feedback and the drawing feature. Below is a table which is reflective of the pressure pads made available to the user through the mat in addition to how the various technical components display this data:
Colour Pads
Serial Output
Colour Codes
Drawing Colour
Audio Feedback
1
0.929f, 0.11f, 0.141f
Red
2
0f, 0.682f, 0.937f
Blue
3
1f, 0.949f, 0f
Yellow
4
0f, 0f, 0f
Black
5
1f, 1f, 1f
White
6
0f, 0.65f, 0.318f
Green
7
0.969f, 0.58f, 0.114f
Orange
8
0.4f, 0.176f, 0.569f
Purple
9
1f, 1f, 1f
Reset
| Colour Pressure Pads
These pressure-pads are minimalistic in form. Each of the eight colour pads (and one reset button) consist of a rubber surface, two wires for digital input and ground in addition to two metal interior plates which are separated by foam. When pressure is applied to a colour pad the foam separating the metal plates compress, this subsequently allows for a connection to be formed and the circuit to be completed. This can, in turn, be read into an Arduino. While simplistic in nature, the structure of these colour pads proves to be strong enough to support the weight of an adult and durable enough to handle the curiosities of a year one student.
| Arduino
The Arduino acts as an intermediary between the physical colour pad controller and the colour mixing code. When a colour pad is pushed, the Arduino reads in LOW from the completed circuit. This LOW, depending on the digital input pin which is being read, is then converted into a serial output. As there are nine pressure pads (inclusive of the reset button) the serial outputs of the Arduino range from 1 through to 9.
| Colour Mixing
The colour mixing takes place in Unity's C# code. Each serial number, which is read in from the Arduino, is associated with a given colour. These colours are then deconstructed and categorised into their respective RGB (red, green, blue) colour codes. In doing this, a user is able to select a colour pad from the physical controller and this selected colour is then added to their drawing-colour buffer. These colours' respective RGB values are averaged if two or more colour pads are selected by the user. This means that, when red which has a code of (0.9f, 0.1f, 0.1f) and yellow which has a code of (1f, 0.95f, 0f) are both selected by the user, the colour mixing code is able to average their respective RGB values to result in (0.95f, 0.97f, 0.05f) or a shade of orange.
| Audio Feedback
The accessibility requirements of Mixed are underpinned by audio feedback. The audio feedback aims to stand as the sound equivalent of the visual colour mixing process. With that being said, colour perceiving users can see that mixing red and yellow results in orange while visually impaired users can hear the audio equivalents of these colours being mixed together. The audio feedback for Mixed has been theoretically justified through sound and light wave frequencies. All light sources and, by extension, colour sources have a related frequency. This frequency can then be translated into an audio track which subsequently forms an objective link between colour and sound. Higher frequencies indicate lighter tones (yellows and reds) while lower frequencies denote deeper colours (blues and purples). In the current version of Mixed there are fifteen colours made available through audio feedback. Each of these fifteen frequencies have a colour range threshold which have been integrated into the C# code. A visual depiction of this can be seen below. Hence, when users select a colour pad, they will head the audio equivalent of that sound in the given threshold. Further, if two colours (such as red and yellow) are mixed together by the user, Mixed will provide the audio equivalent of orange in feedback – the feedback is reflective of the colour in the user's drawing-colour buffer. Mixed's audio feedback aims to contextualise the colour mixing process for users who can not actively perceive it.
| Drawing
The final component of Mixed is the drawing functionality. The drawing effect has been achieved through C# code in Unity which utilises an asset called FreeDraw. However, this asset has been reimagined to cater to the desired functionality of Mixed. The user's drawing colour is reflective of the user's physical interaction with the colour pads and 'paint colour' functions have been implemented to achieve this. Conditional statements have also been integrated into Mixed with the intention of generating colour range thresholds for a given audio frequency. The audio feedback repeats the generated sound from the mixing step when the user begins to draw in the mat's centre. This aims to strengthen the user's cognitive ties between their perception of the mixed colour and the associated sound frequency.
Mixed has made a conscious effort to conform to the iterative design process. The process began with team Twisted selecting the theme we wished to work under and in the case of Twisted, it was Creative Learning. The team then drew inspiration from a proposed concept in World Cafe – Chroma, and the popular children's game Twister. With this inspiration in mind, the team began conducting preliminary research into an area of interest – colour theory, in addition to the definition of creative learning and common educational techniques. This first round of research was able to identify a relevant problem space which the team was willing to work towards.
| Team Domain and Target Users
This problem space definition and team domain has changed over time with respect given to key insights obtained from additional research and user evaluations. However, at the beginning of the design process, this problem space highlighted a decline in creativity facilitated in early childhood education (Core Knowledge Foundation, 2015). One of the first steps – a cultural probe – identified that creativity and experimentation were amongst the most popular educational methods for early primary school students. The team's research also drew attention to the benefits of injecting creative learning approaches into the learning plans of Pre-Foundation to Year Two students (Cheung, 2010). These benefits revolved around an increase in innovative capabilities, problem-solving and collaboration. Next, the team needed an academic premise to underpin the creative learning initiative.
Analysis and evaluation of colour theory highlighted an apparent potential to facilitate experimentation, logical deduction and, by extension, creative learning (Hasbro, 2020). With a problem space now established, the team began proposing potential designs which addressed these findings while adhering to the DECO3850 brief. The team sketched, ideated and critiqued potential concepts before settling on a form relatively similar to the pitched concept – Twisted.
| Initial Concept Ideation
The team pitched Twisted, which was a creative learning initiative to teach colour theory to primary school students. The team aimed to achieve this through utilising a pressure-based mat and a large screen display. Users would collaborate on the mat through applying pressure to colour pads. When done correctly, this would mix the selected colours and camouflage an alien into its respective background. This design aimed to promote active collaboration and problem solving while teaching the fundamentals of colour theory. The team then received feedback from the pitch which highlighted the strengths and weaknesses of the proposed concept. The team then collated all feedback and identified recurring themes which subsequently drew attention to usability issues and non-conformances to the project brief. These themes related predominantly to a lack in accessibility considerations for colour blind/visually impaired users, an apparent closed and restricted interaction approach in addition to a dated screen-based interaction paradigm.
| Shift to Individual
The team was then informed that the structure of the course would shift to a team supported individual project. The team subsequently developed a collective domain that each member would work under. As it currently stands, the team domain relates to a creative learning initiative to teach colour theory to Year 1 students through active engagement and open-ended interaction techniques. I then began ideating for an individual design direction that adhered to this collective team domain. In this stage, I also took into consideration the recurring themes that the team identified from critique analysis. Thus, any concept proposed had a self-imposed requirement to consider accessibility, open-ended interactions and a novel mode while effectively conveying colour theory to the users.
Research suggested that audio feedback would be the most effective means of contextualizing colour theory for users who could not actively perceive colours. Several approaches including music underpinned by emotion, sounds that could be found in nature and meditative music were tested with expert users. These approaches were met with varying degrees of success but overall findings indicated that these subjective links between colour and sound were ineffective. It was suggested that Synaesthesia (a theoretical concept that posits that music notes in the chromatic scale can be linked to colour) should be used to underpin the audio feedback aspect of Mixed. Thus, I conducted some initial research into the theory and noted recurring trends between sound and colour. The recurring patterns were then translated into the audio feedback that Mixed provided when the user interacts with the pressurised colour pads.
| First Major Prototype
A prototype was developed with respect given to these findings. A screen-based interface which can be drawn on represents the mat's canvas. Further, a functional small scale version of the mat and non-functional large-scale form indicator prototypes were developed in order to provide a compartmentalised understanding of the ideal final form, functionality and features of Mixed. Users were able to engage with a functional hand-sized edition of the conceptual design to gauge the physical interactions, cognitive benefits and colour mixing objectives that Mixed facilitates.
Feedback for this prototype noted a positive response to the colour mixing and drawing approach while also highlighting the weaknesses and subjectivity of using Synaesthesia as the sole form of audio feedback. Evaluations posited that translating light frequencies to audio frequencies could stand as a more objective link from colour to sound. A majority of expert users were able to correctly determine colours based on their audio equivalents when provided with a reference note/colour. However, this was largely due to expert users' capacity to understand the underlying theory of converting light waves to sound frequencies. Due to the current social distancing regulations, it has been difficult to obtain evaluations from five to six year old target users and hence difficult to gauge the effectiveness of this new audio feedback approach.
| Final Major Prototype
Below is demonstration of Mixed being used by a colour blind user. The user mixes a colour, by using the colour pads, receives the relevant audio feedback and then draws on the canvas. The user then repeats this process until they have finished creating their complex artwork:
The major differences between the last prototype and the current form of Mixed is the scale of the project and the audio feedback component. Mixed, as it stands, has been scaled to be ten times larger than the previous prototype. This allows for a more engaging and playful user experience to be facilitated when interacting with the mat. The audio feedback that has been integrated into the system's working functionality is now underpinned by the conversion of light waves to sound waves. With respect given to research and user evaluations, this form of feedback currently facilitates the strongest link between colour and sound for colour blind and visually impaired users. It is hoped that in employing these techniques Mixed will allow all users, regardless of colour perception abilities, to actively interact and benefit from colour mixing.
Mixed, as it currently stands, is not yet in its ideal final form. Gaps between the 'actual' product and 'ideal' intended concept relate to the effectiveness of the accessibility considerations in addition to the physical form. Ideally, additional forms of haptic and light feedback would be provided for users to better cater to accessibility requirements. Revision of the audio feedback would also be necessary in order to accurately reflect the colour spectrum for users who cannot actively perceive it. Additionally, having a large flexi-screen or projection alternative to the small drawing tablet would add to a more immersive user experience.
| Relevance to Team Domain
The intended user experience has been developed with respect given to the previously established team domain. This domain relates to a creative learning initiative with an aim to teach colour theory to year one students through active engagement and open-ended interaction techniques. Mixed relates to this domain through fostering creative freedom and experimentation through the interactive pathways made available through the mat. A focus on teaching colour theory also links Mixed to other concepts in this domain. Mixed aims to contextualise colour theory for colour blind and colour perceiving students through audio feedback reflective of the colour mixing process. Thus, it can be seen that Mixed reflects team Twisted's collective domain.
| Relevance to Studio Domain
Further, it is believed that Mixed is able to effectively uphold the studio theme of:
“designing for playful and open-ended interaction approaches”
As previously mentioned, there is no right way to interact with Mixed. Users are encouraged to experiment, mix and create using the large interactive mat. It is hoped that, through this interactive experience, users will be motivated to take control of their own learning. This approach aims to facilitate the supplementary benefits of employing a deductive reasoning mindset, increasing cognitive load and, of course, obtaining an understanding colour theory.
| Project Objectives and Evaluation
I believe Mixed, as it stands, has been able to communicate its intended interactive experience to the users. This has been determined as a result of comparing Mixed against a set of predetermined objectives. With that being said 100% of users were able to recognise the necessary steps required to mix colours together. Additionally, all users were able to follow two interactive pathways to achieve the same goal (ie generate orange two different ways). Further, all users were also able to demonstrate the ability to generate complex multi-coloured artworks. This is indicative of a strong competence in drawing and mixing capabilities when using Mixed. 60% of users were able to correctly determine the colour equivalents of the audio feedback when multiple colours were mixed together while over 80% of users were able to correctly identify a single colour from a small selection of audio tracks after being provided with a reference note. These are promising results for the audio feedback approach, however, further revision would need to be conducted in order to have sound accurately reflect the colour mixing process.
| Exhibition Reflection
The exhibition went really well for team Twisted. Our channel on Discord had a steady and constant stream of visitors, peers and teaching staff who were keen to see our team's individual projects in action. Team Twisted had a very cohesive presenting style where we would pitch, demonstrate and then introduce the next team member who would then repeat this process. This cohesion was even commended by visitors. The team also received positive feedback regarding our design approaches and complementary projects which were clearly reflective of one unified domain. All team members were able to succinctly and accurately answer any questions the visitors asked as well as effectively exhibiting the work completed to date.
No real problems occurred during aside from a small mishap with my red colour pad which triggered unexpectedly late into the exhibit. I realised that this was because the single core wire which was leading into the Arduino had been knocked and was touching another exposed wire. I fixed this and it worked without error for the rest of the night. In future development, Mixed would utilise a printed circuit board (PCB) in order to add to the design's durability and robustness. Another small issue was that some visitors were unfamiliar with how to use Discord which occasionally made it difficult to demonstrate our projects.
Overall, I believe that the exhibit went seamlessly for me and my team. It was a learning curve but very rewarding in the long run. DECO3850 has showcased some exceptional designs and it was brilliant to see these in action at the online exhibition. As for my own concept, I believed that it has taken colour theory, creative learning and accessibility requirements and Mixed them all together.
A blog was maintained over the course of the semester to keep track of my thoughts as I progressed through the iterative design process for Mixed. All relevant reflections, inspiration, design decisions and justifications can be found in this living document.
This report outlines the design process for the first major prototype of Mixed. An overview of the concept, stakeholders, related works, project objectives and a demonstration video can be found in this document.
These pressure-pads are minimalistic in form. Each of the eight colour pads (and one reset button) consist of a rubber surface, two wires for digital input and ground in addition to two metal interior plates which are separated by foam. When pressure is applied to a colour pad the foam separating the metal plates compress, this subsequently allows for a connection to be formed and the circuit to be completed. This can, in turn, be read into an Arduino. While simplistic in nature, the structure of these colour pads proves to be strong enough to support the weight of an adult and durable enough to handle the curiosities of a year one student.
The final component of Mixed is the drawing functionality. The drawing effect has been achieved through C# code in Unity which utilises an asset called FreeDraw. However, this asset has been reimagined to cater to the desired functionality of Mixed. The user's drawing colour is reflective of the user's physical interaction with the colour pads and 'paint colour' functions have been implemented to achieve this. Conditional statements have also been integrated into Mixed with the intention of generating colour range thresholds for a given audio frequency. The audio feedback repeats the generated sound from the mixing step when the user begins to draw in the mat's centre. This aims to strengthen the user's cognitive ties between their perception of the mixed colour and the associated sound frequency.
The team pitched Twisted, which was a creative learning initiative to teach colour theory to primary school students. The team aimed to achieve this through utilising a pressure-based mat and a large screen display. Users would collaborate on the mat through applying pressure to colour pads. When done correctly, this would mix the selected colours and camouflage an alien into its respective background. This design aimed to promote active collaboration and problem solving while teaching the fundamentals of colour theory. The team then received feedback from the pitch which highlighted the strengths and weaknesses of the proposed concept. The team then collated all feedback and identified recurring themes which subsequently drew attention to usability issues and non-conformances to the project brief. These themes related predominantly to a lack in accessibility considerations for colour blind/visually impaired users, an apparent closed and restricted interaction approach in addition to a dated screen-based interaction paradigm.
