Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Hybrid deep-learning model to recognise emotional responses of users towards architectural design alternatives

Hybrid deep-learning model to recognise emotional responses of users towards architectural design... JOURNAL OF ASIAN ARCHITECTURE AND BUILDING ENGINEERING 2019, VOL. 18, NO. 5, 381–391 https://doi.org/10.1080/13467581.2019.1660663 ARCHITECTURAL PLANNING AND DESIGN Hybrid deep-learning model to recognise emotional responses of users towards architectural design alternatives Sunwoo Chang and Hanjong Jun School of Architecture, Hanyang University, Seoul, South Korea ABSTRACT ARTICLE HISTORY Received 27 March 2019 In architectural planning and initial designing process, it is critical for architects to recognise Accepted 19 August 2019 users’ emotional responses toward design alternatives. Since Building Information Modelling and related technologies focuses on physical elements of the building, a model which KEYWORDS suggests decision-makers’ subjective affection is strongly required. In this regard, this paper Generative adversarial proposes an electroencephalography (EEG)-based hybrid deep-learning model to recognise networks; deep-learning the emotional responses of users towards given architectural design. The hybrid model classification; affection consists of generative adversarial networks (GANs) for EEG data augmentation and an EEG- recognition; electroencephalography; based deep-learning classification model for EEG classification. In the field of architecture, TensorFlow a previous study has developed an EEG-based deep-learning classification model that can recognise the emotional responses of subjects towards design alternatives. This approach seems to suggest a possible method of evaluating design alternatives in a quantitative manner. However, because of the limitations of EEG data, it is difficult to train the model, which leads to the limited utilisation of the model. In this regard, this study constructs GANs, which consists of a generator and discriminator, for EEG data augmentation. The proposed hybrid model may provide a method of developing supportive and evaluative environments in planning, design, and post-occupancy evaluation for decision-makers. 1. Introduction and to quantitatively track and review the responses of users towards revised design alternatives (Kiviniemi 1.1 Research background and aim 2005). Therefore, design processes are often ineffi- Architectural planning and initial designing involve cient and the design quality is not guaranteed. communication among various decision-making par- Building information modelling (BIM) technology is ticipants including clients, designers, engineers, used to structure physical architectural elements and potential users, and local community members. In their relationships in a form of a database throughout the architectural planning step, information that can the lifecycle of a building based on a digitalised build- be a basis for future decision making is generated. In ing information management platform and to main- the initial designing step, various design alternatives tain the accumulated building information. BIM are suggested by the designer based on the informa- technology has been utilised in various fields such as tion generated in the planning step. These design building design, construction, and maintenance. It alternatives are then reviewed and revised by deci- plays an important role in supporting the architectural sion-making participants. The designing step is of design step, providing various types of building infor- considerable importance because as it proceeds, mation that can be utilised in the decision-making design alternatives are specified and used as the fun- process. However, BIM technology does not reflect damental basis for future procedures (Sebastian the emotional evaluation of decision-making partici- 2007). pants because it focuses solely on information regard- However, in architectural planning and initial ing the physical attributes of buildings. This is a major designing, communication problems have constantly limitation of BIM technology because in the planning arisen between the client and designer (Siva and and designing steps, the emotional evaluation of deci- London 2012). There are inevitable limitations in the sion-making participants with regard to the design client gaining an accurate understanding of the sug- alternatives suggested by the designer can be an gestions of the designer and in the process of com- important basis for decision making. Against this municating requirements and aspects to be revised background, research has been conducted actively in (Shen et al. 2013). In addition, it is difficult for the the fields of brain science and neuroscience on the designer to grasp the client’s requirements accurately emotional responses of users towards certain CONTACT Hanjong Jun hanjong@hanyang.ac.kr School of Architecture, Hanyang University, 407, ADCC, Science and Technology Building, 222, Wangsimni-ro, Seongdong-gu, Republic of Korea © 2019 The Author(s). Published by Informa UK Limited, trading as Taylor & Francis Group on behalf of the Architectural Institute of Japan, Architectural Institute of Korea and Architectural Society of China. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. 382 S. CHANG AND H. JUN conditions by utilising electroencephalography model for the augmentation of the brain wave data (Ekman and Friesen 1971). In the fields of architecture acquired under experimental settings and describes and design as well, quantitative research has been the details of an EEG-based deep-learning model conducted actively on the emotions of users by utilis- training and evaluation using both existing data ing EEG. Brain waves are electric signals generated and augmented data. In the process of establishing during different brain activities, and EEG is the GANs model and EEG-based deep-learning clas- a technology to generate numerical data by attaching sification model, the TensorFlow platform developed electrodes to the scalps of subjects for brain wave by Google is utilised, which is a core open-source augmentation (Schomer and Silva 2010). machine learning library (TensorFlow, 2018). The pro- Recently, in the field of architecture, some posed hybrid model in this research is built espe- research studies have presented design alternatives cially for EEG data augmentation and classification to subjects in the form of images while recording using the TensorFlow libraries. The brain wave data- their EEG and analysing the EEG data using deep- sets are recorded using the “EMOTIV EPOC+ 14 learning classification models (Chang, Dong, and Channel Mobile EEG” device from Emotiv and the Jun 2018). The models proposed in these studies EmotivPro software (Emotiv 2018). show the basic possibility of estimating the emo- tional responses of decision-making participants towards certain spaces. However, there are limita- 2. Theoretical investigation tions in training the model because the brain wave 2.1 Utilisation of brain waves in field of data collected from experiments are insufficient. In architecture this reagard, as an alternative, a generative adversar- ial networks (GANs) may be utilised to generate vir- Brain waves are electric signals generated during tual data using only a small volume of input data. brain cell and neurons activities. The numerical data Goodfellow et al. (2014) proposed a GANs consist- of these waves are recorded using EEG, for which ing of a generator and discriminator and generated electrodes are attached to the scalps of subjects virtual data through the augmentation of a small (Schomer and Silva 2010). Brain waves have drawn volume of input data. A GANs can be utilised when attention because it is possible to usually detect the available volume of data is not sufficient for the them in any person and to utilise their quantitative training and evaluation of a deep-learning model. The numerical data, which indicate the responses of the GANs concept is drawing attention in many fields, brain towards external stimuli. In the field of architec- particularly in fields where data acquisition is challen- ture, the quantitative data are utilised for research on ging, such as facial recognition (Yang, Zhang, and Yin the concentration and productivity of space occu- 2018) and voice recognition (Han et al. 2018). pants, dependence of sleep patterns on environmen- Accordingly, this paper proposes an EEG-based tal changes, biological reactions of occupants or hybrid deep-learning model that can classify the emo- subjects in satisfaction analysis, and so on. Lan and tional responses of potential users towards architec- Lian (2009), Lee, Choi, and Chun (2012), and Zhang tural design alternatives. This model is a combination et al. (2017) utilised brain waves to observe and verify of a GANs and EEG-based deep-learning classification changes in the productivity of occupants depending model. The GANs augments brain wave data acquired on the indoor temperature. Pan, Lian, and Lan (2011), in previous studies, and the augmented brain wave Lan et al. (2013), and Lan, Lian, and Lin (2016) utilised data are provided tothe classification model to train it. EEG for obtaining quantitative data, using which they analysed the dependence of the sleep patterns and satisfaction of subjects on the spatial environment. All 1.2 Study method such studies utilise brain waves for obtaining quanti- tative data, which are used to examine the physiolo- The rest of the paper is structured as follows. Section gical changes in occupants depending on the physical 2 presents a theoretical investigation on previous architectural conditions. studies that utilise brain waves in the field of archi- Many previous studies relevant to architectural tecture and establishes the theoretical background design have used EEG to grasp the emotional for GANs models and deep-learning classification responses of users towards changes in the architec- models. Section 3 describes the development of tural space. One such research included a preference a GANs model for brain wave data augmentation survey and brain wave experiments among youths, and an EEG-based deep-learning classification who were given indoor images with emotional voca- model for the data classification based on the find- bularies of community facilities in an apartment com- ings of the previous studies; hence, a hybrid deep- plex (Hwang, Kim, and Kim 2013). Hwang et al. (2014) learning model based on EEG is developed by com- used brain wave data to examine facility preferences bining these two deep-learning models. Section 4 among youths. Ryu and Lee (2015) examined the presents the implementation of the proposed GANs JOURNAL OF ASIAN ARCHITECTURE AND BUILDING ENGINEERING 383 correlations between colour changes in an indoor method algorithm processes data based on the fea- residential space and brain waves. Such studies are ture values of input data, calculates differences(loss) important in that they utilise the quantitative data between model predictions and actual data labels obtained from brain to grasp the emotional responses based on the user-given results, and proceeds to of subjects in architectural design perspectives and to reduce the loss from such differences (Hinton and observe relevant changes. However, these studies Salakhutdinov 2006). When this process occurs while have limitations in terms of establishing specific mod- classification values of actual data (labels) are pro- els for EEG data analysis. In contrast, Chang, Dong, vided, it is called supervised learning. When any clas- and Jun (2018) recorded the brain waves of subjects sification values of actual data are provided, it is called while showing them images of a small residential unsupervised learning. However, in the case of super- space and asking them to select the most-preferred vised learning, model training is limited if any label and least-preferred images. This study also proposed values of actual data are provided or if the quantity of an EEG-based deep-learning classification model data is small. In contrast, the GANs model is different through which the recorded brain wave data could in that this algorithm is based on unsupervised learn- be analysed. This model is important in that it ana- ing and thus can generate virtual data based on input lyses the subjective emotional responses of users data even if the quantity of the input data is small. towards architectural spaces in a limited experimental This data augmentation supports the process of train- situation, but the quantity of brain wave data ing the supervised deep-learning algorithm. The GANs obtained from experiments is limited. Therefore, it is model consists of a generator and discriminator. The difficult to apply this model universally, and further generator generates virtual data using latent data and learning and evaluation of the model is required. In delivers the generated data to the discriminator. The addition, the process of obtaining brain wave data is discriminator processes the data from the generator complicated and requires EEG equipment, making it and the original data simultaneously, distinguishing difficult to apply this model to various other research the original data from the virtual data. As the training areas. Thus, the present study aims to establish is repeated, the generator and discriminator continue a generative adversarial networks model through to adjust the weight and bias values that indicate the which a small quantity of brain wave data acquired extent of connection among the nodes in each hid- through experiments from previous studies could be den layer. The discriminator continues learning with augmented and converted into virtual data and to regard to distinguishing the original data from the train the EEG-based deep-learning classification virtual data, whereas the generator proceeds to gen- model using the acquired data. Figure 1 shows erate virtual data that are similar to the original data. a diagram that illustrates how the proposed hybrid Because the generator and discriminator continue deep-learning model could be applied to an architec- learning in a hostile and competing environment, tural design process. the algorithm is adversarial, competitive and generative. The GANs model is utilised to increase the size of 2.2 Generative adversarial networks data whose actual classification values are hardly secured as well as to generate certain data types Goodfellow et al. (2014) proposed the concept of such as image, video, and voice. Caramihale, a GANs, which is a deep-learning algorithm based Popescu, and Ichim (2018) and Luo and Lu (2018) on unsupervised learning. Supervised deep-learning Figure 1. Application scenario. 384 S. CHANG AND H. JUN augmented key facial expression data and brain wave (w ), and then, the result works as an input for the data by utilising the GANs. Their finding indicates that activation function. The activation function then cal- as augmented data are utilised for training an emo- culates the input and transmits it to the following tional lassification model, the model’s accuracy is node as an output value. enhanced. If the classification values of the actual data are Figure 2 shows the structure of the GANs consist- provided to the model in the process of entering ing of the discriminator and generator. The discrimi- data to the deep-learning model and training it, it is nator receives virtual data generated by the generator called supervised learning. If such values are not pro- in addition to actual data, whereas the generator vided, it is called unsupervised learning. If the model receives random data. Input datasets pass through is trained through supervised learning, the training the neural networks of both the generator and dis- may proceed in a manner that reduces the difference criminator. The discriminator receives actual data and between the values predicted by the model and the data generated by the generator simultaneously. It classification values of the actual data. However, in distinguishes the actual data from the fake data and this case, the learning may be limited if the quantity calculates the loss. The generator and discriminator of data is small or if no classification values are pro- continue learning in a manner that reduces this loss. vided. In contrast, the unsupervised learning method has limitations in that it may produce different values depending on the model’s structure or parameters 2.3 Deep-learning classification although it is possible to train the model even with- out the classification values of the data (Jain, Duin, Hinton and Salakhutdinov (2006) proposed a concept and Mao 2000). of deep learning that combines multiple artificial Accordingly, this study aims to develop an EEG- neural networks. The deep-learning model is mainly based hybrid deep-learning model that can recognise divided into three types of layers: the input layer, the emotional responses of users in the initial design- hidden layers, and output layer. The hidden layers ing step. It consists of a GANs for brain wave data consist of multiple nodes and the weight and bias augmentation and an EEG-based deep-learning classi- values that indicate the extent of connection between fication model for brain wave data classification. Brain the nodes. These values pass through the activation wave datasets of subjects from previous studies are function and then through the following layer. As augmented through the GANs, and then, the aug- data enter into the deep-learning model, computa- mented datasets are utilised to train and evaluate tional operations are performed on them while they the classification model (Chang, Dong, and Jun 2018). pass through each layer. The output layer presents the predicted values after the model classifies the data. In the case of the supervised learning classifica- 3. Proposed hybrid deep-learning model tion model, both the labels and feature values of the data are put into the model. The difference between In this work, two distinct deep-learning-based models the classification values of the actual data and the are suggested. The first is a generative adversarial values predicted by the model is calculated which is networks (GANs) model for EEG data augmentation, so called loss. and the second is an EEG-based deep-learning model Figure 3 shows the structure of the deep neural for data classification. The hybrid deep-learning networks. An input (x ) is multiplied by a weight value model consists of a combination of these two models. Figure 2. Generative adversarial networks (GANs). JOURNAL OF ASIAN ARCHITECTURE AND BUILDING ENGINEERING 385 Figure 3. Deep neural networks (DNNs) structure. Both these models were built in this study using previous studies. In the process of establishing this Google TensorFlow, which is a core open-source model, the open-source library Google TensorFlow library for machine learning and deep learning. was used. Brain wave data collected from experiments work The EEG data that are to be used by the estab- as input values for GANs and then augmented in lished model have 14 channels, and datasets are nor- a competitive environment between the generator malised for each channel. This step is necessary to and discriminator. The augmented data work as prevent data distortion in the event that the value input values for deep-learning classification model of a certain channel is larger than that of another along with raw brain wave data collected through channel. experiments, and then, the model proceeds with The generator and discriminator of the established training and evaluation based on the data. Figure 4 GANs consist of three hidden layers, which consist of 50, shows how the GANs used for brain wave data aug- 100, and 50 nodes, respectively. The number of nodes in mentation is combined with the EEG-based deep- each layer was determined after evaluations with multi- learning classification model used for EEG data ple combinations of hyper parameters that showed the classification. highest training speeds. The generator receives random values between 0 and 1 and generates virtual EEG data. The discriminator receives EEG collected through 3.1 Generative adversarial networks for brain experiments and virtual data and distinguishes the ori- wave data augmentation ginal data from the virtual data. Once the training is completed, the EEG data generated by the generator This section aims to establish a GANs model to aug- are presented to users and saved as csv format. Figure 5 ment a small quantity of brain wave data collected in 386 S. CHANG AND H. JUN Figure 4. Hybrid deep-learning model - GANs + EEG-based Deep-Learning Classification Model. Figure 5. GANs generator and discriminator. shows the structure of the generator and discriminator the brain wave data as “positive” or “negative” of the established GANs. toward architectural design alternatives. The classi- fication model utilises TensorFlow as well as appli- cation programming libraries related to supervised 3.2 EEG-based deep-learning classification model learning. for brain wave data classification The model consists of three layers: the input layer, hidden layers, and output layer. And the hidden layers The purpose of the EEG-based deep-learning classi- consist of three layers which consist of 100,200 and fication model is to classify the emotional states of JOURNAL OF ASIAN ARCHITECTURE AND BUILDING ENGINEERING 387 100 nodes respectively. A number of nodes in the 4. Implementation of hidden layers were selected after preliminary tests electroencephalography-based hybrid with various combinations of hyper-parameters and deep-learning model were optimised for training speed and accuracy. 4.1 Brain wave dataset The classification model proceeds with learning according to the following steps: (1) data input and Chang, Dong, and Jun (2018) performed an experiment normalisation, (2) distinction of training data from in which they recorded EEG data of subjects. The evaluation data(train-test split), (3) training, and (4) experiment was performed in an experiment room of evaluation and accuracy measurement. (1) In the Hanyang University with the help of 18 individuals (12 data input and normalisation step, the brain wave women and 6 men) who were healthy physically and data of 14 channels are entered according to each mentally. The experiment was performed in the follow- channel’s feature data values and are normalised. ing order: (1) presentation of space images to subjects The actual classification values of the corresponding for selection, (2) machine calibration and meditation, brain wave data are also given to the model. (2) In (3) brain wave measurement, and (4) questionnaire the data splitting step, the entire input data are completion. (1) In the step of presenting space images shuffled and divided: 70% for training and 30% for to subjects for selection, eight space images were pre- testing. The division between the training and test sented and subjects selected images that they most datasets was determined after conducting multiple preferred and the ones they least preferred. (2) In the optimisations of the model and was intended to machine calibration and meditation step, subjects were prevent over-fitting or under-fitting problems. (3) In asked to wear the “EMOTIV EPOC+ 14 Channel Mobile the training step, the classification model presents EEG” device from Emotiv for machine calibration using the predicted values based on the feature values of the EmotivPro software. (3) In the brain wave measure- the input data, and the weight and bias values are ment step, the brain waves of subjects were recorded adjusted in a manner that reduces the difference for 20 s. (4) In the questionnaire completion step, between the predicted output values and actual subjects filled out the Positive and Negative Affect classification values. (4) In the evaluation and accu- Schedule (PANAS) questionnaire. The PANAS question- racy measurement step, the model accuracy is mea- naire included 20 questions: 10 regarding “positive sured and given to users. Figure 6 shows the emotion” and 10 regarding “negative emotion.” For structure of the model. the answer to each question, “very much” was given Figure 6. EEG-based Deep-Learning Classification Model. 388 S. CHANG AND H. JUN 5 points and “notatall” was given 1 point. In this for the first 5 s and last 5 s were excluded to remove manner, the scores of the “positive emotion” and noise, and the remaining brain wave data for 10 s were “negative emotion” questions were calculated used. The number of rows in the “positive emotion” (Watson, Clark, and Tellegen 1988). Chang, Dong, and and “negative emotion” datasets was 10,240 Jun (2018) excluded the emotion scores that were (8 × 128 × 10) and 21,760 (17 × 128 × 10), respectively. lower than the average scores of “positive emotion” Figure 7(a) shows the result of the brain wave and “negative emotion” presented in the research of experiment performed on one subject, Figure 7(b) Watson, Clark, and Tellegen (1988). Consequently, the shows the brain wave recording software called brain wave data of 8 individuals for the “positive emo- EmotivPro(Emotiv 2018), and Figure 7(c) shows the tion” questions and the data of 17 individuals for the electrode position of the “EMOTIV EPOC+ 14 “negative emotion” questions were included in the Channel Mobile EEG” device, which is the EEG equip- dataset. From the brain wave data for 20 s, the data ment used in this experiment (Emotiv 2018). Figure 7. (a) Subject recording EEG, (b) EmotivPro software interface with recording EEG (Emotiv 2018), (c) “EMOTIV EPOC+ 14 channel mobile EEG” device technical specifications (Emotiv 2018). Figure 8. EEG-based Deep-learning classification model training dataset: (a) Features: 14 channels of recorded EEG data, (b) Affection: Re-classified PANAS Questionnaire Results, (c) Labels: “Negative” = 1 and “Positive” = 2, (d) Subject Initials, (e) Frequency (Hz). JOURNAL OF ASIAN ARCHITECTURE AND BUILDING ENGINEERING 389 Figure 9. Suggested Model Use Research Scenario. Figure 8 shows the structure of the EEG dataset. simultaneously. The discriminator distinguished the Figure 8(a) shows the raw brain wave data of 14 chan- actual data from the fake data. In this phase, loss values nels, which include feature values. Figure 8(b) shows the of discriminator and generator are calculated. If gener- re-classified affection results of the PANAS survey. ated data becomes similar to actual data during training Figure 8(c) shows numerically casted values from the process, generator loss decreases. (4) In the virtual data survey results. Actual classification values were entered output step, virtual brain wave data generated by the into the model as follows: 1 for “negative” items and 2 generator were saved in csv format. In total, 30,000 rows for “positive” items. Figure 8(d) shows the initials of each were generated by the generator for the “positive emo- subject, and Figure 8(e) shows the frequency. tion EEG dataset” and “negative emotion EEG dataset” items respectively. Table 1 indicates that as the training was repeated, the 4.2 Brain wave dataset augmentation using virtual data in blue gradually became similar to the actual generative adversarial networks data in orange. Among the 14 channels of the “positive emotion” brain wave data, the two channels AF3 and T7 The learning process of the GANs includes the following were visualised on the x axis and y axis, respectively. When four steps: (1) data input, (2) virtual data generation by the training was repeated 1,000 times, the generator the generator, (3) discriminator training, and (4) virtual generated data containing negative numbers with no data output. (1) In the data input step, the “positive actual data. When the training was repeated 8,000 emotion EEG dataset” and “negative emotion EEG data- times, the overlap of the virtual data with the actual set” recorded in the previous study were entered into the data increased. As the training was repeated, the over- model. (2) In the step involving virtual data generation by lapped parts of the actual data and virtual data on the the generator, random values between 0 and 1 were coordinate axes increased to the point that the discrimi- entered into the generator and the generator generated nator could no longer distinguish the actual data from the virtual brain wave data. (3) In the discriminator training virtual data. Accordingly, the generator’s loss decreased step, the actual EEG data and virtual EEG data generated gradually. by the generator were entered into the discriminator Table 1. Comparison between real data and augmented data through iterations (Positive affection). Iterations: 1,000 Iterations: 8,000 Iterations: 18,000 Iterations: 22,000 390 S. CHANG AND H. JUN reflect real business processes. This scenario is sug- 4.3 DEEG-based deep-learning classification gested in Figure 9. model training using virtual brain wave data The virtual EEG data generated using the GANs model were utilised in the process of training the EEG-based 5. Conclusion deep-learning classification model. The virtual brain The objective of this study was to establish an EEG- wave dataset generated based on the “positive emo- based hybrid deep-learning model through which the tion EEG dataset” was added to this dataset, and then, emotional responses of users towards a architectural the “negative emotion EEG dataset” was added in the space could be evaluated. To this end, brain wave same manner. Consequently, the “positive emotion data collected in previous studies were augmented brain wave dataset” and “negative emotion brain using the GANs model, and an EEG-based deep- wave dataset” were combined with 40,240 and learning classification model was trained and evalu- 51,760 rows, respectively. ated by using both actual and virtual data. The find- The expanded “positive emotion EEG dataset” and ings of this study are as follows. “negative emotion EEG dataset” were then entered First, the GANs showed a major possibility of aug- into the classification model for training. While the menting atypical EEG data. It is expected that this previous classification model utilised 32,000 datasets networks can be used in the field of architecture to for training, in the present model, 92,000 datasets augment and utilise datasets whose scale and avail- were utilised in training as virtual brain wave data ability are limited, such as simulation data, environ- were added. The final accuracy was 0.984, which was mental sensor data, and observation data. Second, as better than the existing accuracy (0.979). If more data the quantity of data increases, the accuracy of training are added in the future using the GANs model, the and evaluating the deep-learning model will improve accuracy and universality may improve further. continually. The model has demonstrated that it can handle big data pertaining to architecture, traffic, and other environments that can be utilised in the field of 4.4 Implications and model utilisation scenarios architecture. Third, by using the brain wave data and The proposed EEG-based hybrid deep-learning EEG-based hybrid deep-learning model, the study has model enables quantitative identification of user demonstrated the possibility of helping the designer emotions towards architectural space images. At in the architectural design step to grasp the emo- the design stage, the architects consider the emo- tional responses of future users or clients towards tional qualitative reviews of the decision makers, proposed design alternatives. In this study, it was including owners and preliminary users, as well as possible to utilise brain wave data as indicators of the quantitative reviews, including legal and loca- the emotional responses or preferences of subjects tion-based reviews, on the design alternatives to be only in limited environments. important. In this regard, the proposed model can In future studies, the following research activities be effectively used. However, at the current stage of should be considered. The proposed hybrid deep- research, because it is difficult to manipulate EEG learning model should be applied to actual design pro- recording machines and because the experimental cesses. The hybrid deep-learning model should be eval- procedures are complicated, applying the proposed uated and verified. Experiments should be performed for model to actual design processes will require addi- recording additional brain wave data to facilitate the tional research in the fields of brain science and universal application of the model. The EEG-based hybrid architecture in the future. deep-learning model should be combined with an eye At present, we can present research scenarios that tracking system for the evaluation of the emotional quantitatively measure specific subjects’ responses responses of users towards actual spatial environments. towards specific architectural spaces and build data- bases by obtaining measurement data and incorpor- ating them into design knowledge using the Acknowledgments proposed model. Previous studies on design knowl- We would like to express our sincere gratitude to GARAM edge databases mainly used surveys and in-depth Architects and Associates Research and Development interviews to measure the emotional responses of Center for offering supportive environment to finish this subjects to a specific type of architecture. However, research. if the model proposed in this study is adopted, archi- tects can quantitatively measure the emotional Disclosure statement responses of users under certain conditions and sui- tably incorporate them into the knowledge base to No potential conflict of interest was reported by the authors. JOURNAL OF ASIAN ARCHITECTURE AND BUILDING ENGINEERING 391 Funding Sleeping Period Delays the Onset of Sleep in Summer.” Building and environment 103: 36–43. doi:10.1016/j. This research was supported by a grant (19AUDP-B127891-03) buildenv.2016.03.030. from the Architecture & Urban Development Research Program Lan, L., L. Pan, Z. Lian, H. Huang, and Y. Lin. 2013. funded by the Ministry of Land, Infrastructure and Transport of “Experimental Study on Thermal Comfort of Sleeping the Korean government. People at Different Air Temperatures.” Building and envir- onment 73: 24–31. doi:10.1016/j.buildenv.2013.11.024. Lee, H. J., Y. L. Choi, and C. Y. Chun. 2012. “Effect of Indoor Air Temperature on the Occupants’ Attention Ability References Based on Electroencephalogram Analysis.” Journal of the Architectural Institute Of Korea Planning & Design 28 (3): Caramihale, T., D. Popescu, and L. Ichim. 2018. “Emotion 217–225. Classification Using a TensorFlow Generative Adversarial Luo, Y., and B. Lu. 2018. “EEG Data Augmentation for Networks Implementation.” Symmetry 10 (9): 414. Emotion Recognition Using a Conditional Wasserstein doi:10.3390/sym10090414. GAN.” In 40th Annual International Conference of the Chang, S. W., W. H. Dong, and H. J. Jun. 2018. “EEG-Based IEEE Engineering in Medicine and Biology Society Deep Neural Networks Classification Model for (EMBC), Honolulu, Hawaii, 2535–2538. Recognizing Emotion of Users in Early Phase of Design”. Pan, L., Z. Lian, and L. Lan. 2011. “Investigation of Gender Journal of the Architectural Institute of Korea Planning & Differences in Sleeping Comfort at Different Environme- Design 34(12): 85-94. ntal Temperatures.” Indoor and Built Environment 21 (6): Ekman, P., and W. V. Friesen. 1971. “Constants across Cultures 811–820. doi:10.1177/1420326X11425967. in the Face and Emotion.” Journal of personality and social Ryu, J. S., and J. S. Lee. 2015. “Correlation Analysis of psychology 17 (2): 124–129. doi:10.1037/h0030377. Emotional Adjectives and EEG to Apply Color to the Emotiv. 2018. “EMOTIV EPOC+, EPOC+ Headset Details.” Indoor Living Space.” Journal of Korea Society of Color Accessed 27 September 2018. https://emotiv.gitbooks. Studies 29 (3): 25–35. doi:10.17289/jkscs.29.3.201508.25. io/epoc-user-manual/content/epoc+_headset_details/ Schomer, D., and F. Silva. 2010. Niedermeyer’s Electro- Goodfellow, I., J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde- encephalography: Basic Principles, Clinical Applications, Farley, S. Ozair, A. Courville, and Y. Bengio. 2014. and Related Fields. 6th ed. Lippincott Williams & Wilkins. “Generative Adversarial Nets.” In Proceedings of the Philadelphia, Pennsylvania, United States. 27th International Conference on Neural Information Sebastian, R. 2007. “Managing Collaborative Design.” Processing Systems, Montreal, Canada, 2 vols, 2672–2680. Doctoral Thesis, Delft University of Technology. Han, J., Z. Zhang, Z. Ren, F. Ringeval, and B. Schuller. 2018. Shen, W., X. Zhang, G. Q. Shen, and T. Fernando. 2013. “The “Towards Conditional Adversarial Training for Predicting User Pre-Occupancy Evaluation Method in Designer– Emotions from Speech.” In IEEE International Conference Client Communication in Early Design Stage: A Case on Acoustics, Speech and Signal Processing (ICASSP), Study.” Automation in Construction 32: 112–124. Calgary, Alberta, Canada, 6822–6826. doi:10.1016/j.autcon.2013.01.014. Hinton, G. E., and R. R. Salakhutdinov. 2006. “Reducing the Siva, J., and K. London. 2012. “Client Learning for Successful Dimensionality of Data with Neural Networks.” Science Architect-Client Relationship.” Engineering, Construction 313 (5786): 504–507. doi:10.1126/science.1127647. and Architectural Management 19 (3): 253–268. Hwang, Y. S., J. Y. Kim, A. L. Chang, E. Y. Lim, and H. W. Jung. doi:10.1108/09699981211219599. 2014. “Planning of Apartment Community Facilities TensorFlow. 2018. Accessed 12 November 2018. https:// According to EEG Analysis by School Age of Youth www.tensorflow.org/ Emotional Words.” Korean Institute of Interior Design Watson, D., L. A. Clark, and A. Tellegen. 1988. “Development Journal 23 (4): 181–189. doi:10.14774/JKIID.2014.23.4.181. and validation of brief measures of positive and negative Hwang, Y. S., S. Y. Kim, and J. Y. Kim. 2013. “An Analysis of affect: The PANAS scales.” Journal of personality and social Youth EEG Based on the Emotional Color Scheme Images psychology 54 (6): 1063–1070. doi:10.1037/0022- by Different Space of Community Facilities.” Korean 3514.54.6.1063. Institute of Interior Design Journal 22 (5): 171–178. Yang, H., Z. Zhang, and L. Yin. 2018. “Identity-Adaptive Facial doi:10.14774/JKIID.2013.22.5.171. Expression Recognition through Expression Regeneration Jain, A. K., R. P. W. Duin, and J. Mao. 2000. “Statistical Pattern Using Conditional Generative Adversarial Networks.” In Recognition: A Review.” IEEE transactions on pattern analysis 13th IEEE International Conference on Automatic Face & and machine intelligence 22 (1): 4–37. doi:10.1109/34.824819. Gesture Recognition, Xi’an, China, 294–301. Kiviniemi, A. 2005. “Requirements Management Interface to Building Product Models.” Thesis, Stanford University. Zhang, F., S. Haddad, B. Nakisa, M. N. Rastgoo, C. Candido, Lan, L., and Z. Lian. 2009. “Use of Neurobehavioral Tests to D. Tjondronegoro, and R. D. Dear. 2017. “The Effects of Higher Temperature Setpoints during Summer on Office Evaluate the Effects of Indoor Environment Quality on Workers’ Cognitive Load and Thermal Comfort.” Building Productivity.” Building and environment 44 (11): and environment 123: 176–188. doi:10.1016/j.buil- 2208–2217. doi:10.1016/j.buildenv.2009.02.001. denv.2017.06.048. Lan, L., Z. W. Lian, and Y. B. Lin. 2016. “Comfortably Cool Bedroom Environment during the Initial Phase of the http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Journal of Asian Architecture and Building Engineering Taylor & Francis

Hybrid deep-learning model to recognise emotional responses of users towards architectural design alternatives

Hybrid deep-learning model to recognise emotional responses of users towards architectural design alternatives

Abstract

In architectural planning and initial designing process, it is critical for architects to recognise users’ emotional responses toward design alternatives. Since Building Information Modelling and related technologies focuses on physical elements of the building, a model which suggests decision-makers’ subjective affection is strongly required. In this regard, this paper proposes an electroencephalography (EEG)-based hybrid deep-learning model to recognise the emotional responses...
Loading next page...
 
/lp/taylor-francis/hybrid-deep-learning-model-to-recognise-emotional-responses-of-users-YnusWhCRrJ
Publisher
Taylor & Francis
Copyright
© 2019 The Author(s). Published by Informa UK Limited, trading as Taylor & Francis Group on behalf of the Architectural Institute of Japan, Architectural Institute of Korea and Architectural Society of China.
ISSN
1347-2852
eISSN
1346-7581
DOI
10.1080/13467581.2019.1660663
Publisher site
See Article on Publisher Site

Abstract

JOURNAL OF ASIAN ARCHITECTURE AND BUILDING ENGINEERING 2019, VOL. 18, NO. 5, 381–391 https://doi.org/10.1080/13467581.2019.1660663 ARCHITECTURAL PLANNING AND DESIGN Hybrid deep-learning model to recognise emotional responses of users towards architectural design alternatives Sunwoo Chang and Hanjong Jun School of Architecture, Hanyang University, Seoul, South Korea ABSTRACT ARTICLE HISTORY Received 27 March 2019 In architectural planning and initial designing process, it is critical for architects to recognise Accepted 19 August 2019 users’ emotional responses toward design alternatives. Since Building Information Modelling and related technologies focuses on physical elements of the building, a model which KEYWORDS suggests decision-makers’ subjective affection is strongly required. In this regard, this paper Generative adversarial proposes an electroencephalography (EEG)-based hybrid deep-learning model to recognise networks; deep-learning the emotional responses of users towards given architectural design. The hybrid model classification; affection consists of generative adversarial networks (GANs) for EEG data augmentation and an EEG- recognition; electroencephalography; based deep-learning classification model for EEG classification. In the field of architecture, TensorFlow a previous study has developed an EEG-based deep-learning classification model that can recognise the emotional responses of subjects towards design alternatives. This approach seems to suggest a possible method of evaluating design alternatives in a quantitative manner. However, because of the limitations of EEG data, it is difficult to train the model, which leads to the limited utilisation of the model. In this regard, this study constructs GANs, which consists of a generator and discriminator, for EEG data augmentation. The proposed hybrid model may provide a method of developing supportive and evaluative environments in planning, design, and post-occupancy evaluation for decision-makers. 1. Introduction and to quantitatively track and review the responses of users towards revised design alternatives (Kiviniemi 1.1 Research background and aim 2005). Therefore, design processes are often ineffi- Architectural planning and initial designing involve cient and the design quality is not guaranteed. communication among various decision-making par- Building information modelling (BIM) technology is ticipants including clients, designers, engineers, used to structure physical architectural elements and potential users, and local community members. In their relationships in a form of a database throughout the architectural planning step, information that can the lifecycle of a building based on a digitalised build- be a basis for future decision making is generated. In ing information management platform and to main- the initial designing step, various design alternatives tain the accumulated building information. BIM are suggested by the designer based on the informa- technology has been utilised in various fields such as tion generated in the planning step. These design building design, construction, and maintenance. It alternatives are then reviewed and revised by deci- plays an important role in supporting the architectural sion-making participants. The designing step is of design step, providing various types of building infor- considerable importance because as it proceeds, mation that can be utilised in the decision-making design alternatives are specified and used as the fun- process. However, BIM technology does not reflect damental basis for future procedures (Sebastian the emotional evaluation of decision-making partici- 2007). pants because it focuses solely on information regard- However, in architectural planning and initial ing the physical attributes of buildings. This is a major designing, communication problems have constantly limitation of BIM technology because in the planning arisen between the client and designer (Siva and and designing steps, the emotional evaluation of deci- London 2012). There are inevitable limitations in the sion-making participants with regard to the design client gaining an accurate understanding of the sug- alternatives suggested by the designer can be an gestions of the designer and in the process of com- important basis for decision making. Against this municating requirements and aspects to be revised background, research has been conducted actively in (Shen et al. 2013). In addition, it is difficult for the the fields of brain science and neuroscience on the designer to grasp the client’s requirements accurately emotional responses of users towards certain CONTACT Hanjong Jun hanjong@hanyang.ac.kr School of Architecture, Hanyang University, 407, ADCC, Science and Technology Building, 222, Wangsimni-ro, Seongdong-gu, Republic of Korea © 2019 The Author(s). Published by Informa UK Limited, trading as Taylor & Francis Group on behalf of the Architectural Institute of Japan, Architectural Institute of Korea and Architectural Society of China. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. 382 S. CHANG AND H. JUN conditions by utilising electroencephalography model for the augmentation of the brain wave data (Ekman and Friesen 1971). In the fields of architecture acquired under experimental settings and describes and design as well, quantitative research has been the details of an EEG-based deep-learning model conducted actively on the emotions of users by utilis- training and evaluation using both existing data ing EEG. Brain waves are electric signals generated and augmented data. In the process of establishing during different brain activities, and EEG is the GANs model and EEG-based deep-learning clas- a technology to generate numerical data by attaching sification model, the TensorFlow platform developed electrodes to the scalps of subjects for brain wave by Google is utilised, which is a core open-source augmentation (Schomer and Silva 2010). machine learning library (TensorFlow, 2018). The pro- Recently, in the field of architecture, some posed hybrid model in this research is built espe- research studies have presented design alternatives cially for EEG data augmentation and classification to subjects in the form of images while recording using the TensorFlow libraries. The brain wave data- their EEG and analysing the EEG data using deep- sets are recorded using the “EMOTIV EPOC+ 14 learning classification models (Chang, Dong, and Channel Mobile EEG” device from Emotiv and the Jun 2018). The models proposed in these studies EmotivPro software (Emotiv 2018). show the basic possibility of estimating the emo- tional responses of decision-making participants towards certain spaces. However, there are limita- 2. Theoretical investigation tions in training the model because the brain wave 2.1 Utilisation of brain waves in field of data collected from experiments are insufficient. In architecture this reagard, as an alternative, a generative adversar- ial networks (GANs) may be utilised to generate vir- Brain waves are electric signals generated during tual data using only a small volume of input data. brain cell and neurons activities. The numerical data Goodfellow et al. (2014) proposed a GANs consist- of these waves are recorded using EEG, for which ing of a generator and discriminator and generated electrodes are attached to the scalps of subjects virtual data through the augmentation of a small (Schomer and Silva 2010). Brain waves have drawn volume of input data. A GANs can be utilised when attention because it is possible to usually detect the available volume of data is not sufficient for the them in any person and to utilise their quantitative training and evaluation of a deep-learning model. The numerical data, which indicate the responses of the GANs concept is drawing attention in many fields, brain towards external stimuli. In the field of architec- particularly in fields where data acquisition is challen- ture, the quantitative data are utilised for research on ging, such as facial recognition (Yang, Zhang, and Yin the concentration and productivity of space occu- 2018) and voice recognition (Han et al. 2018). pants, dependence of sleep patterns on environmen- Accordingly, this paper proposes an EEG-based tal changes, biological reactions of occupants or hybrid deep-learning model that can classify the emo- subjects in satisfaction analysis, and so on. Lan and tional responses of potential users towards architec- Lian (2009), Lee, Choi, and Chun (2012), and Zhang tural design alternatives. This model is a combination et al. (2017) utilised brain waves to observe and verify of a GANs and EEG-based deep-learning classification changes in the productivity of occupants depending model. The GANs augments brain wave data acquired on the indoor temperature. Pan, Lian, and Lan (2011), in previous studies, and the augmented brain wave Lan et al. (2013), and Lan, Lian, and Lin (2016) utilised data are provided tothe classification model to train it. EEG for obtaining quantitative data, using which they analysed the dependence of the sleep patterns and satisfaction of subjects on the spatial environment. All 1.2 Study method such studies utilise brain waves for obtaining quanti- tative data, which are used to examine the physiolo- The rest of the paper is structured as follows. Section gical changes in occupants depending on the physical 2 presents a theoretical investigation on previous architectural conditions. studies that utilise brain waves in the field of archi- Many previous studies relevant to architectural tecture and establishes the theoretical background design have used EEG to grasp the emotional for GANs models and deep-learning classification responses of users towards changes in the architec- models. Section 3 describes the development of tural space. One such research included a preference a GANs model for brain wave data augmentation survey and brain wave experiments among youths, and an EEG-based deep-learning classification who were given indoor images with emotional voca- model for the data classification based on the find- bularies of community facilities in an apartment com- ings of the previous studies; hence, a hybrid deep- plex (Hwang, Kim, and Kim 2013). Hwang et al. (2014) learning model based on EEG is developed by com- used brain wave data to examine facility preferences bining these two deep-learning models. Section 4 among youths. Ryu and Lee (2015) examined the presents the implementation of the proposed GANs JOURNAL OF ASIAN ARCHITECTURE AND BUILDING ENGINEERING 383 correlations between colour changes in an indoor method algorithm processes data based on the fea- residential space and brain waves. Such studies are ture values of input data, calculates differences(loss) important in that they utilise the quantitative data between model predictions and actual data labels obtained from brain to grasp the emotional responses based on the user-given results, and proceeds to of subjects in architectural design perspectives and to reduce the loss from such differences (Hinton and observe relevant changes. However, these studies Salakhutdinov 2006). When this process occurs while have limitations in terms of establishing specific mod- classification values of actual data (labels) are pro- els for EEG data analysis. In contrast, Chang, Dong, vided, it is called supervised learning. When any clas- and Jun (2018) recorded the brain waves of subjects sification values of actual data are provided, it is called while showing them images of a small residential unsupervised learning. However, in the case of super- space and asking them to select the most-preferred vised learning, model training is limited if any label and least-preferred images. This study also proposed values of actual data are provided or if the quantity of an EEG-based deep-learning classification model data is small. In contrast, the GANs model is different through which the recorded brain wave data could in that this algorithm is based on unsupervised learn- be analysed. This model is important in that it ana- ing and thus can generate virtual data based on input lyses the subjective emotional responses of users data even if the quantity of the input data is small. towards architectural spaces in a limited experimental This data augmentation supports the process of train- situation, but the quantity of brain wave data ing the supervised deep-learning algorithm. The GANs obtained from experiments is limited. Therefore, it is model consists of a generator and discriminator. The difficult to apply this model universally, and further generator generates virtual data using latent data and learning and evaluation of the model is required. In delivers the generated data to the discriminator. The addition, the process of obtaining brain wave data is discriminator processes the data from the generator complicated and requires EEG equipment, making it and the original data simultaneously, distinguishing difficult to apply this model to various other research the original data from the virtual data. As the training areas. Thus, the present study aims to establish is repeated, the generator and discriminator continue a generative adversarial networks model through to adjust the weight and bias values that indicate the which a small quantity of brain wave data acquired extent of connection among the nodes in each hid- through experiments from previous studies could be den layer. The discriminator continues learning with augmented and converted into virtual data and to regard to distinguishing the original data from the train the EEG-based deep-learning classification virtual data, whereas the generator proceeds to gen- model using the acquired data. Figure 1 shows erate virtual data that are similar to the original data. a diagram that illustrates how the proposed hybrid Because the generator and discriminator continue deep-learning model could be applied to an architec- learning in a hostile and competing environment, tural design process. the algorithm is adversarial, competitive and generative. The GANs model is utilised to increase the size of 2.2 Generative adversarial networks data whose actual classification values are hardly secured as well as to generate certain data types Goodfellow et al. (2014) proposed the concept of such as image, video, and voice. Caramihale, a GANs, which is a deep-learning algorithm based Popescu, and Ichim (2018) and Luo and Lu (2018) on unsupervised learning. Supervised deep-learning Figure 1. Application scenario. 384 S. CHANG AND H. JUN augmented key facial expression data and brain wave (w ), and then, the result works as an input for the data by utilising the GANs. Their finding indicates that activation function. The activation function then cal- as augmented data are utilised for training an emo- culates the input and transmits it to the following tional lassification model, the model’s accuracy is node as an output value. enhanced. If the classification values of the actual data are Figure 2 shows the structure of the GANs consist- provided to the model in the process of entering ing of the discriminator and generator. The discrimi- data to the deep-learning model and training it, it is nator receives virtual data generated by the generator called supervised learning. If such values are not pro- in addition to actual data, whereas the generator vided, it is called unsupervised learning. If the model receives random data. Input datasets pass through is trained through supervised learning, the training the neural networks of both the generator and dis- may proceed in a manner that reduces the difference criminator. The discriminator receives actual data and between the values predicted by the model and the data generated by the generator simultaneously. It classification values of the actual data. However, in distinguishes the actual data from the fake data and this case, the learning may be limited if the quantity calculates the loss. The generator and discriminator of data is small or if no classification values are pro- continue learning in a manner that reduces this loss. vided. In contrast, the unsupervised learning method has limitations in that it may produce different values depending on the model’s structure or parameters 2.3 Deep-learning classification although it is possible to train the model even with- out the classification values of the data (Jain, Duin, Hinton and Salakhutdinov (2006) proposed a concept and Mao 2000). of deep learning that combines multiple artificial Accordingly, this study aims to develop an EEG- neural networks. The deep-learning model is mainly based hybrid deep-learning model that can recognise divided into three types of layers: the input layer, the emotional responses of users in the initial design- hidden layers, and output layer. The hidden layers ing step. It consists of a GANs for brain wave data consist of multiple nodes and the weight and bias augmentation and an EEG-based deep-learning classi- values that indicate the extent of connection between fication model for brain wave data classification. Brain the nodes. These values pass through the activation wave datasets of subjects from previous studies are function and then through the following layer. As augmented through the GANs, and then, the aug- data enter into the deep-learning model, computa- mented datasets are utilised to train and evaluate tional operations are performed on them while they the classification model (Chang, Dong, and Jun 2018). pass through each layer. The output layer presents the predicted values after the model classifies the data. In the case of the supervised learning classifica- 3. Proposed hybrid deep-learning model tion model, both the labels and feature values of the data are put into the model. The difference between In this work, two distinct deep-learning-based models the classification values of the actual data and the are suggested. The first is a generative adversarial values predicted by the model is calculated which is networks (GANs) model for EEG data augmentation, so called loss. and the second is an EEG-based deep-learning model Figure 3 shows the structure of the deep neural for data classification. The hybrid deep-learning networks. An input (x ) is multiplied by a weight value model consists of a combination of these two models. Figure 2. Generative adversarial networks (GANs). JOURNAL OF ASIAN ARCHITECTURE AND BUILDING ENGINEERING 385 Figure 3. Deep neural networks (DNNs) structure. Both these models were built in this study using previous studies. In the process of establishing this Google TensorFlow, which is a core open-source model, the open-source library Google TensorFlow library for machine learning and deep learning. was used. Brain wave data collected from experiments work The EEG data that are to be used by the estab- as input values for GANs and then augmented in lished model have 14 channels, and datasets are nor- a competitive environment between the generator malised for each channel. This step is necessary to and discriminator. The augmented data work as prevent data distortion in the event that the value input values for deep-learning classification model of a certain channel is larger than that of another along with raw brain wave data collected through channel. experiments, and then, the model proceeds with The generator and discriminator of the established training and evaluation based on the data. Figure 4 GANs consist of three hidden layers, which consist of 50, shows how the GANs used for brain wave data aug- 100, and 50 nodes, respectively. The number of nodes in mentation is combined with the EEG-based deep- each layer was determined after evaluations with multi- learning classification model used for EEG data ple combinations of hyper parameters that showed the classification. highest training speeds. The generator receives random values between 0 and 1 and generates virtual EEG data. The discriminator receives EEG collected through 3.1 Generative adversarial networks for brain experiments and virtual data and distinguishes the ori- wave data augmentation ginal data from the virtual data. Once the training is completed, the EEG data generated by the generator This section aims to establish a GANs model to aug- are presented to users and saved as csv format. Figure 5 ment a small quantity of brain wave data collected in 386 S. CHANG AND H. JUN Figure 4. Hybrid deep-learning model - GANs + EEG-based Deep-Learning Classification Model. Figure 5. GANs generator and discriminator. shows the structure of the generator and discriminator the brain wave data as “positive” or “negative” of the established GANs. toward architectural design alternatives. The classi- fication model utilises TensorFlow as well as appli- cation programming libraries related to supervised 3.2 EEG-based deep-learning classification model learning. for brain wave data classification The model consists of three layers: the input layer, hidden layers, and output layer. And the hidden layers The purpose of the EEG-based deep-learning classi- consist of three layers which consist of 100,200 and fication model is to classify the emotional states of JOURNAL OF ASIAN ARCHITECTURE AND BUILDING ENGINEERING 387 100 nodes respectively. A number of nodes in the 4. Implementation of hidden layers were selected after preliminary tests electroencephalography-based hybrid with various combinations of hyper-parameters and deep-learning model were optimised for training speed and accuracy. 4.1 Brain wave dataset The classification model proceeds with learning according to the following steps: (1) data input and Chang, Dong, and Jun (2018) performed an experiment normalisation, (2) distinction of training data from in which they recorded EEG data of subjects. The evaluation data(train-test split), (3) training, and (4) experiment was performed in an experiment room of evaluation and accuracy measurement. (1) In the Hanyang University with the help of 18 individuals (12 data input and normalisation step, the brain wave women and 6 men) who were healthy physically and data of 14 channels are entered according to each mentally. The experiment was performed in the follow- channel’s feature data values and are normalised. ing order: (1) presentation of space images to subjects The actual classification values of the corresponding for selection, (2) machine calibration and meditation, brain wave data are also given to the model. (2) In (3) brain wave measurement, and (4) questionnaire the data splitting step, the entire input data are completion. (1) In the step of presenting space images shuffled and divided: 70% for training and 30% for to subjects for selection, eight space images were pre- testing. The division between the training and test sented and subjects selected images that they most datasets was determined after conducting multiple preferred and the ones they least preferred. (2) In the optimisations of the model and was intended to machine calibration and meditation step, subjects were prevent over-fitting or under-fitting problems. (3) In asked to wear the “EMOTIV EPOC+ 14 Channel Mobile the training step, the classification model presents EEG” device from Emotiv for machine calibration using the predicted values based on the feature values of the EmotivPro software. (3) In the brain wave measure- the input data, and the weight and bias values are ment step, the brain waves of subjects were recorded adjusted in a manner that reduces the difference for 20 s. (4) In the questionnaire completion step, between the predicted output values and actual subjects filled out the Positive and Negative Affect classification values. (4) In the evaluation and accu- Schedule (PANAS) questionnaire. The PANAS question- racy measurement step, the model accuracy is mea- naire included 20 questions: 10 regarding “positive sured and given to users. Figure 6 shows the emotion” and 10 regarding “negative emotion.” For structure of the model. the answer to each question, “very much” was given Figure 6. EEG-based Deep-Learning Classification Model. 388 S. CHANG AND H. JUN 5 points and “notatall” was given 1 point. In this for the first 5 s and last 5 s were excluded to remove manner, the scores of the “positive emotion” and noise, and the remaining brain wave data for 10 s were “negative emotion” questions were calculated used. The number of rows in the “positive emotion” (Watson, Clark, and Tellegen 1988). Chang, Dong, and and “negative emotion” datasets was 10,240 Jun (2018) excluded the emotion scores that were (8 × 128 × 10) and 21,760 (17 × 128 × 10), respectively. lower than the average scores of “positive emotion” Figure 7(a) shows the result of the brain wave and “negative emotion” presented in the research of experiment performed on one subject, Figure 7(b) Watson, Clark, and Tellegen (1988). Consequently, the shows the brain wave recording software called brain wave data of 8 individuals for the “positive emo- EmotivPro(Emotiv 2018), and Figure 7(c) shows the tion” questions and the data of 17 individuals for the electrode position of the “EMOTIV EPOC+ 14 “negative emotion” questions were included in the Channel Mobile EEG” device, which is the EEG equip- dataset. From the brain wave data for 20 s, the data ment used in this experiment (Emotiv 2018). Figure 7. (a) Subject recording EEG, (b) EmotivPro software interface with recording EEG (Emotiv 2018), (c) “EMOTIV EPOC+ 14 channel mobile EEG” device technical specifications (Emotiv 2018). Figure 8. EEG-based Deep-learning classification model training dataset: (a) Features: 14 channels of recorded EEG data, (b) Affection: Re-classified PANAS Questionnaire Results, (c) Labels: “Negative” = 1 and “Positive” = 2, (d) Subject Initials, (e) Frequency (Hz). JOURNAL OF ASIAN ARCHITECTURE AND BUILDING ENGINEERING 389 Figure 9. Suggested Model Use Research Scenario. Figure 8 shows the structure of the EEG dataset. simultaneously. The discriminator distinguished the Figure 8(a) shows the raw brain wave data of 14 chan- actual data from the fake data. In this phase, loss values nels, which include feature values. Figure 8(b) shows the of discriminator and generator are calculated. If gener- re-classified affection results of the PANAS survey. ated data becomes similar to actual data during training Figure 8(c) shows numerically casted values from the process, generator loss decreases. (4) In the virtual data survey results. Actual classification values were entered output step, virtual brain wave data generated by the into the model as follows: 1 for “negative” items and 2 generator were saved in csv format. In total, 30,000 rows for “positive” items. Figure 8(d) shows the initials of each were generated by the generator for the “positive emo- subject, and Figure 8(e) shows the frequency. tion EEG dataset” and “negative emotion EEG dataset” items respectively. Table 1 indicates that as the training was repeated, the 4.2 Brain wave dataset augmentation using virtual data in blue gradually became similar to the actual generative adversarial networks data in orange. Among the 14 channels of the “positive emotion” brain wave data, the two channels AF3 and T7 The learning process of the GANs includes the following were visualised on the x axis and y axis, respectively. When four steps: (1) data input, (2) virtual data generation by the training was repeated 1,000 times, the generator the generator, (3) discriminator training, and (4) virtual generated data containing negative numbers with no data output. (1) In the data input step, the “positive actual data. When the training was repeated 8,000 emotion EEG dataset” and “negative emotion EEG data- times, the overlap of the virtual data with the actual set” recorded in the previous study were entered into the data increased. As the training was repeated, the over- model. (2) In the step involving virtual data generation by lapped parts of the actual data and virtual data on the the generator, random values between 0 and 1 were coordinate axes increased to the point that the discrimi- entered into the generator and the generator generated nator could no longer distinguish the actual data from the virtual brain wave data. (3) In the discriminator training virtual data. Accordingly, the generator’s loss decreased step, the actual EEG data and virtual EEG data generated gradually. by the generator were entered into the discriminator Table 1. Comparison between real data and augmented data through iterations (Positive affection). Iterations: 1,000 Iterations: 8,000 Iterations: 18,000 Iterations: 22,000 390 S. CHANG AND H. JUN reflect real business processes. This scenario is sug- 4.3 DEEG-based deep-learning classification gested in Figure 9. model training using virtual brain wave data The virtual EEG data generated using the GANs model were utilised in the process of training the EEG-based 5. Conclusion deep-learning classification model. The virtual brain The objective of this study was to establish an EEG- wave dataset generated based on the “positive emo- based hybrid deep-learning model through which the tion EEG dataset” was added to this dataset, and then, emotional responses of users towards a architectural the “negative emotion EEG dataset” was added in the space could be evaluated. To this end, brain wave same manner. Consequently, the “positive emotion data collected in previous studies were augmented brain wave dataset” and “negative emotion brain using the GANs model, and an EEG-based deep- wave dataset” were combined with 40,240 and learning classification model was trained and evalu- 51,760 rows, respectively. ated by using both actual and virtual data. The find- The expanded “positive emotion EEG dataset” and ings of this study are as follows. “negative emotion EEG dataset” were then entered First, the GANs showed a major possibility of aug- into the classification model for training. While the menting atypical EEG data. It is expected that this previous classification model utilised 32,000 datasets networks can be used in the field of architecture to for training, in the present model, 92,000 datasets augment and utilise datasets whose scale and avail- were utilised in training as virtual brain wave data ability are limited, such as simulation data, environ- were added. The final accuracy was 0.984, which was mental sensor data, and observation data. Second, as better than the existing accuracy (0.979). If more data the quantity of data increases, the accuracy of training are added in the future using the GANs model, the and evaluating the deep-learning model will improve accuracy and universality may improve further. continually. The model has demonstrated that it can handle big data pertaining to architecture, traffic, and other environments that can be utilised in the field of 4.4 Implications and model utilisation scenarios architecture. Third, by using the brain wave data and The proposed EEG-based hybrid deep-learning EEG-based hybrid deep-learning model, the study has model enables quantitative identification of user demonstrated the possibility of helping the designer emotions towards architectural space images. At in the architectural design step to grasp the emo- the design stage, the architects consider the emo- tional responses of future users or clients towards tional qualitative reviews of the decision makers, proposed design alternatives. In this study, it was including owners and preliminary users, as well as possible to utilise brain wave data as indicators of the quantitative reviews, including legal and loca- the emotional responses or preferences of subjects tion-based reviews, on the design alternatives to be only in limited environments. important. In this regard, the proposed model can In future studies, the following research activities be effectively used. However, at the current stage of should be considered. The proposed hybrid deep- research, because it is difficult to manipulate EEG learning model should be applied to actual design pro- recording machines and because the experimental cesses. The hybrid deep-learning model should be eval- procedures are complicated, applying the proposed uated and verified. Experiments should be performed for model to actual design processes will require addi- recording additional brain wave data to facilitate the tional research in the fields of brain science and universal application of the model. The EEG-based hybrid architecture in the future. deep-learning model should be combined with an eye At present, we can present research scenarios that tracking system for the evaluation of the emotional quantitatively measure specific subjects’ responses responses of users towards actual spatial environments. towards specific architectural spaces and build data- bases by obtaining measurement data and incorpor- ating them into design knowledge using the Acknowledgments proposed model. Previous studies on design knowl- We would like to express our sincere gratitude to GARAM edge databases mainly used surveys and in-depth Architects and Associates Research and Development interviews to measure the emotional responses of Center for offering supportive environment to finish this subjects to a specific type of architecture. However, research. if the model proposed in this study is adopted, archi- tects can quantitatively measure the emotional Disclosure statement responses of users under certain conditions and sui- tably incorporate them into the knowledge base to No potential conflict of interest was reported by the authors. JOURNAL OF ASIAN ARCHITECTURE AND BUILDING ENGINEERING 391 Funding Sleeping Period Delays the Onset of Sleep in Summer.” Building and environment 103: 36–43. doi:10.1016/j. This research was supported by a grant (19AUDP-B127891-03) buildenv.2016.03.030. from the Architecture & Urban Development Research Program Lan, L., L. Pan, Z. Lian, H. Huang, and Y. Lin. 2013. funded by the Ministry of Land, Infrastructure and Transport of “Experimental Study on Thermal Comfort of Sleeping the Korean government. People at Different Air Temperatures.” Building and envir- onment 73: 24–31. doi:10.1016/j.buildenv.2013.11.024. Lee, H. J., Y. L. Choi, and C. Y. Chun. 2012. “Effect of Indoor Air Temperature on the Occupants’ Attention Ability References Based on Electroencephalogram Analysis.” Journal of the Architectural Institute Of Korea Planning & Design 28 (3): Caramihale, T., D. Popescu, and L. Ichim. 2018. “Emotion 217–225. Classification Using a TensorFlow Generative Adversarial Luo, Y., and B. Lu. 2018. “EEG Data Augmentation for Networks Implementation.” Symmetry 10 (9): 414. Emotion Recognition Using a Conditional Wasserstein doi:10.3390/sym10090414. GAN.” In 40th Annual International Conference of the Chang, S. W., W. H. Dong, and H. J. Jun. 2018. “EEG-Based IEEE Engineering in Medicine and Biology Society Deep Neural Networks Classification Model for (EMBC), Honolulu, Hawaii, 2535–2538. Recognizing Emotion of Users in Early Phase of Design”. Pan, L., Z. Lian, and L. Lan. 2011. “Investigation of Gender Journal of the Architectural Institute of Korea Planning & Differences in Sleeping Comfort at Different Environme- Design 34(12): 85-94. ntal Temperatures.” Indoor and Built Environment 21 (6): Ekman, P., and W. V. Friesen. 1971. “Constants across Cultures 811–820. doi:10.1177/1420326X11425967. in the Face and Emotion.” Journal of personality and social Ryu, J. S., and J. S. Lee. 2015. “Correlation Analysis of psychology 17 (2): 124–129. doi:10.1037/h0030377. Emotional Adjectives and EEG to Apply Color to the Emotiv. 2018. “EMOTIV EPOC+, EPOC+ Headset Details.” Indoor Living Space.” Journal of Korea Society of Color Accessed 27 September 2018. https://emotiv.gitbooks. Studies 29 (3): 25–35. doi:10.17289/jkscs.29.3.201508.25. io/epoc-user-manual/content/epoc+_headset_details/ Schomer, D., and F. Silva. 2010. Niedermeyer’s Electro- Goodfellow, I., J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde- encephalography: Basic Principles, Clinical Applications, Farley, S. Ozair, A. Courville, and Y. Bengio. 2014. and Related Fields. 6th ed. Lippincott Williams & Wilkins. “Generative Adversarial Nets.” In Proceedings of the Philadelphia, Pennsylvania, United States. 27th International Conference on Neural Information Sebastian, R. 2007. “Managing Collaborative Design.” Processing Systems, Montreal, Canada, 2 vols, 2672–2680. Doctoral Thesis, Delft University of Technology. Han, J., Z. Zhang, Z. Ren, F. Ringeval, and B. Schuller. 2018. Shen, W., X. Zhang, G. Q. Shen, and T. Fernando. 2013. “The “Towards Conditional Adversarial Training for Predicting User Pre-Occupancy Evaluation Method in Designer– Emotions from Speech.” In IEEE International Conference Client Communication in Early Design Stage: A Case on Acoustics, Speech and Signal Processing (ICASSP), Study.” Automation in Construction 32: 112–124. Calgary, Alberta, Canada, 6822–6826. doi:10.1016/j.autcon.2013.01.014. Hinton, G. E., and R. R. Salakhutdinov. 2006. “Reducing the Siva, J., and K. London. 2012. “Client Learning for Successful Dimensionality of Data with Neural Networks.” Science Architect-Client Relationship.” Engineering, Construction 313 (5786): 504–507. doi:10.1126/science.1127647. and Architectural Management 19 (3): 253–268. Hwang, Y. S., J. Y. Kim, A. L. Chang, E. Y. Lim, and H. W. Jung. doi:10.1108/09699981211219599. 2014. “Planning of Apartment Community Facilities TensorFlow. 2018. Accessed 12 November 2018. https:// According to EEG Analysis by School Age of Youth www.tensorflow.org/ Emotional Words.” Korean Institute of Interior Design Watson, D., L. A. Clark, and A. Tellegen. 1988. “Development Journal 23 (4): 181–189. doi:10.14774/JKIID.2014.23.4.181. and validation of brief measures of positive and negative Hwang, Y. S., S. Y. Kim, and J. Y. Kim. 2013. “An Analysis of affect: The PANAS scales.” Journal of personality and social Youth EEG Based on the Emotional Color Scheme Images psychology 54 (6): 1063–1070. doi:10.1037/0022- by Different Space of Community Facilities.” Korean 3514.54.6.1063. Institute of Interior Design Journal 22 (5): 171–178. Yang, H., Z. Zhang, and L. Yin. 2018. “Identity-Adaptive Facial doi:10.14774/JKIID.2013.22.5.171. Expression Recognition through Expression Regeneration Jain, A. K., R. P. W. Duin, and J. Mao. 2000. “Statistical Pattern Using Conditional Generative Adversarial Networks.” In Recognition: A Review.” IEEE transactions on pattern analysis 13th IEEE International Conference on Automatic Face & and machine intelligence 22 (1): 4–37. doi:10.1109/34.824819. Gesture Recognition, Xi’an, China, 294–301. Kiviniemi, A. 2005. “Requirements Management Interface to Building Product Models.” Thesis, Stanford University. Zhang, F., S. Haddad, B. Nakisa, M. N. Rastgoo, C. Candido, Lan, L., and Z. Lian. 2009. “Use of Neurobehavioral Tests to D. Tjondronegoro, and R. D. Dear. 2017. “The Effects of Higher Temperature Setpoints during Summer on Office Evaluate the Effects of Indoor Environment Quality on Workers’ Cognitive Load and Thermal Comfort.” Building Productivity.” Building and environment 44 (11): and environment 123: 176–188. doi:10.1016/j.buil- 2208–2217. doi:10.1016/j.buildenv.2009.02.001. denv.2017.06.048. Lan, L., Z. W. Lian, and Y. B. Lin. 2016. “Comfortably Cool Bedroom Environment during the Initial Phase of the

Journal

Journal of Asian Architecture and Building EngineeringTaylor & Francis

Published: Sep 3, 2019

Keywords: Generative adversarial networks; deep-learning classification; affection recognition; electroencephalography; TensorFlow

References