Fullscreen HTML5Med Res (??MB)HTML4

Big Brother may be is watching you! Facial Recognition Have you ever had that feeling that “someone is watching you”? You’re not being paranoid . . . because the chances are that someone, somewhere is doing exactly that – from social media apps to government/ law enforcement surveillance systems and possibly even by criminal enterprises. And while serious privacy concerns have been raised, facial recognition is also a useful tool for fighting crime and terrorism. AS the name suggests, facial recognition is where a computer or hardware device determines the unique characteristics of a person’s face, based on still images or video, to identify them. In today’s world of widespread terrorism, identity theft, criminal activity and online socialising, its use is becoming widespread. In many cases, people’s photos are available on the internet – whether they want it or not. Some people may not even be aware of it. These photos can be fed into facial recognition software and used to identify and even track individuals, whether by government organisations or third parties. Apart from law enforcement and social media applications, modern smartphones such as the iPhone X, Galaxy Note 9 and LG G7 can use facial recognition to automatically unlock the device for its owner and prevent use by others. Commercial organisations such as casinos also use facial recognition to enforce bans against specific individuals and for other reasons, which will be discussed later. Facial recognition comes under the heading of biometric systems, just like fingerprint or iris recognition. by Dr David But unlike most other biometric sys14 Silicon Chip tems, it can be performed without the knowledge or even cooperation of the subject and is amenable to mass surveillance due to the huge number of cameras already installed around the world. Use in Australia Controversially, it is likely that the Australian Government will soon have in place a national facial biometric matching capability with images of a substantial number of Australians. These will be “harvested” from passports, driver’s licenses, citizenship documents and visa applications as well as presumably any number of other sources of opportunity. Various states and government agencies already have their own systems in operation but the proposed system will integrate these and other systems on a national basis. See siliconchip.com.au/link/aan8 and siliconchip.com.au/ link/aan9 for more details. The Government system will be known as “The Capability” – not a sinister name at all! See: http://siliconchip. com.au/link/aana Facial recognition systems require large amounts of computer power to be Maddison used in real time; hence, it is only with Australia’s electronics magazine siliconchip.com.au Fig.1: a RAND tablet from the 1960s which allowed the human operator to input facial landmarks into a computer database. The digitising surface was approximately 25cm x 25cm with one million possible locations. One operator could process about 40 photos per hour. Little was published on this work, due to it having been funded by the US Government (DARPA). It also had other uses, as seen in this photo. (siliconchip.com.au/link/aanf) the development of sufficiently fast and cheap computers in the last 15-20 years or so that these systems have become practical and commonplace. It has also been necessary to develop appropriate computer algorithms to perform the task of facial recognition. This is an ongoing task. Facial recognition involves one of the most challenging problems in computing and artificial intelligence, which is visual pattern recognition. This is something that humans do easily and intuitively – it is built into our brains from birth. We can easily recognise a familiar face, even with only a partial, non-frontal view under poor lighting conditions for a very brief moment. But that is a difficult task for a machine. Early photograph-based systems Facial recognition of sorts has origins back to the time when cameras became widely available, around 1839. Prisoners in Belgium were photographed as early as 1843 and in some parts of England, prisoners were photographed from 1848, so that they could be more easily found if they escaped. The Pinkerton National Detective Agency, a private detective agency established in the USA in 1850 (and still siliconchip.com.au in existence), also photographed people it apprehended. At the time, the alternative to a photograph (which too had its critics in Victorian society) was to brand certain convicted criminals who had committed serious offences. Otherwise, it was tough to identify known criminals. Before photographs, this was usually done by written descriptions or direct testimony of victims or police. For a discussion on photographing prisoners, see siliconchip.com.au/link/aanb Alphonse Bertillon was a French police officer and early biometrics researcher who invented a system of physical measurements to enable police to identify a criminal objectively. He also developed the “mug shot”, the technique for which was standardised in 1888. Bertillon noted the difficulty in searching a collection Google reverse image search If you have a link to an image online or a saved copy of that image, Google can often find other exact or similar copies of that picture online and also possibly identify the people in the image. Go to https://images.google.com/, click on the camera icon and use “&imgtype=face” in the query. Australia’s electronics magazine April 2019  15 Fig.2: a collection of faces known as the AT&T “Database of Faces”, which is a standard set used for testing and research by people working in the facial recognition field. It consists of 10 pictures of each of 40 people. of photos with no other criteria applied. He said that it was hard to identify an individual “if you have no other means but your eyes to search for the photograph among the thousands in an ordinary collection”. Early computerised systems Modern computerised facial recognition systems have their origins in the 1960s, with the first work carried out by Woodrow W. Bledsoe with Helen Chan and Charles Bisson during 1964-1966 at Panoramic Research in Palo Alto, California. In this work, an early digitising device known as a RAND tablet (Fig.1) was used by a human operator to mark the location and size of various facial landmarks of a person on photographs. This included the eyes, nose, mouth and hairline. These locations were then compared with the locations stored in a database and the closest match was used to identify the person. This early system was limited by the lack of computer power and memory storage of that time but was an important first step to prove the viability of the technology. Following Bledsoe, in the 1970s, Goldstein, Harmon and Lesk used 21 subjective facial markers such as hair colour False facial identifications and overall accuracy Facial recognition software is not perfect, far from it, and a bad identification can ruin someone’s life, as explained in the article at: siliconchip.com.au/link/aane Accurate facial recognition is very much dependent on the quality of the original picture(s) stored in the database, including conditions such as lighting, orientation toward the camera, facial expression etc. There’s also the question of just how useful it is, even when it works. Critics have made the argument that in places like the United Kingdom, where there is widespread surveillance and facial recognition technology in use, no (or few) criminals or terrorists have been apprehended specifically due to these systems. 16 Silicon Chip Fig.3: the set of eigenfaces computed from the AT&T Database of Faces shown above. In this case, principal component analysis mapping has been computed and the first 24 principal components (eigenfaces) are shown. These eigenfaces can be added together in various proportions to recreate all the original faces with little loss of accuracy. Australia’s electronics magazine siliconchip.com.au Fig.6: elastic bunch graph mapping showing a subject in three different poses. Fig.4: reconstructing a photo of one person by combining Eigenfaces computed from the AT&T Database of Faces using the OpenCV software. and lip thickness to achieve greater recognition accuracy. But the facial features still had to be manually entered into the computer. In 1987, mathematicians L. Sirovich and M. Kirby developed an approach to efficiently represent facial images using principal component analysis (PCA). This was used as the basis of facial recognition by computer scientists Matthew Turk and Alex Pentland in 1991. PCA is a statistical technique whereby a large number of possibly correlated variables are reduced to a smaller number of non-correlated variables. While the resulting set of variables is significantly smaller than the starting set, it still contains most of the same information. In other words, it is a method of “lossy” data compression, or dimensionality reduction as it is also known. The first principal component accounts for most of the Fig.5: a selection of Fisherfaces from Yale Face Database A, computed by OpenCV. siliconchip.com.au variability in the data set, the second accounts for most of the remaining variability and so on. Principal component analysis as applied to human faces results in a set of images known as eigenfaces (Fig.3). In practice, relatively few principal components can account for most of the variability of human faces (Fig.4). This technique dramatically simplifies data processing as much less data needs to be stored and compared. Sirovich and Kirby determined that a large collection of facial images could be simply represented by a small set of “standard” faces (eigenfaces) to which are applied weighting factors to approximately represent all members of the collection. Eigenfaces might also be thought of as “standardised face ingredients” and any human face can be considered a combination of various proportions of these standard faces, eg, an individual might comprise 10% of eigenface #1, 16% of eigenface #2 etc. Relatively few eigenfaces are needed to represent all human faces, as long as the appropriate mix of each is applied. For example, combinations of 43 eigenfaces can be used to represent 95% of all human faces. Turk and Pentland essentially applied the inverse of Sirovich’s and Kirby’s work (a way to represent known faces) to identify unknown faces. Their technique took unknown faces and determined what weighting factors needed to be applied to generate the features of a known individual in a database (eigendecomposition). The closer the weighting factors were between the known Fig.7: a faceprint of a test subject for Aurora 3D facial recognition software. Australia’s electronics magazine April 2019  17 Fig.8: an idealised 3D facial recognition model as seen from various angles. With a 3D model, a face can be recognised from many different angles, not just from straight ahead or with a slight deviation from straight. face in the database and those calculated from the unknown, the more likelihood there was of a match between the unknown and known face. The computer code to calculate eigenfaces is relatively simple to implement in software such as Matlab, as shown in the following example, which uses the facial database “yalefaces”. There is also a video explaining the technique of principal component analysis and eigenfaces titled “Lecture: PCA for Face Recognition” at siliconchip.com.au/link/aaoe clear all; close all; load yalefaces [h,w,n] = size(yalefaces); d = h*w; % vectorize images x = reshape(yalefaces,[d n]); x = double(x); % subtract mean mean_matrix = mean(x,2); x = bsxfun(<at>minus, x, mean_matrix); % calculate covariance s = cov(x’); % obtain eigenvalue & eigenvector [V,D] = eig(s); eigval = diag(D); % sort eigenvalues in descending order eigval = eigval(end:-1:1); V = fliplr(V); % show mean and 1st through 15th principal eigenvectors figure,subplot(4,4,1) imagesc(reshape(mean_matrix, [h,w])) colormap gray for i = 1:15 subplot(4,4,i+1) imagesc(reshape(V(:,i),h,w)) end More advanced facial recognition From 1993 to the early 2000s, the US Defense Advanced Research Projects Agency (DARPA) and the National In18 Silicon Chip Fig.9: Apple’s iPhone X uses its TrueDepth front-facing 3D camera to illuminate a face with a pattern of 30,000 infrared dots which are then converted to a 3D facial model. The system is highly accurate and in tests could not be fooled by identical twins; it would only unlock the phone for the twin to whom it was authorised. stitute of Standards and Technology (NIST) developed a facial database called FERET that eventually consisted of 2413 24-bit colour images of 856 different people. Its purpose was to establish a large database of images that could be used for testing facial recognition systems. Controversially, in 2002, the US Government used facial recognition technology at that year’s Super Bowl (American Football grand final). Several petty criminals were detected but the test was seen as a failure, as the technology of that time did not work well in crowds. This also led to concerns over the civil liberties implications of such technology. Facebook started using facial recognition technology in 2010 to identify users who appeared in photos posted to the site by other users. Google Photos and Apple Photos have now deployed similar technology. Facial recognition is now also used in airports and border crossings around the world, and by law enforcement agencies. Steps for facial recognition For software systems to recognise a face, five main steps must occur. These are: 1. Detection of a human face in a still or video image (which may have a cluttered background). 2. Alignment and normalisation of the face to a standardised position with even illumination. 3. Representation of the normalised image with an appropriate mathematical pattern. 4. Feature extraction to determine those characteristics that are unique to the face and at variance to an “average” face. 5. Searching a database of known faces for a match using these characteristics or variances. Common problems in facial recognition are: 1. A differing facial expression, pose or angle to that in the database. 2. Differing or uneven illumination. 3. Ageing of the subject or changes to hairstyle, hair colour etc. 4. Low size or poor quality of the image. 5. Additions or deletions of items such as facial hair, Australia’s electronics magazine siliconchip.com.au Fig.10: how OpenBR works. It is an open-source biometric software framework for facial recognition (http:// openbiometrics.org/) OpenBR can use a variety of different facial recognition algorithms such as PCA (principal component analysis), LBP (local binary patterns), SVM (support vector machines), LDA (linear discriminant analysis), HOG (histogram of oriented gradients) and more. glasses, scarves or other objects disguising part of the face or surrounds. Fig.11: OpenCV is an open-source software library for computer vision which includes the ability to perform facial recognition (https://opencv.org/). In this example, facial landmark detection is being used with two different techniques. On the left, it is using Dlib and on the right, CLMframework. The blue lines represent the direction of gaze of the face, which it also detects. See the video titled “Facial Landmark Detection” at siliconchip.com.au/link/aaod for more information. Statistical facial recognition techniques Geometric techniques The principal component analysis and eigenfaces technique developed by Turk, Pentland, Sirovich and Kirby mentioned above is still in use today in facial recognition systems. But many other techniques have now also been developed. The eigenface approach has an accuracy of about 90% with frontal face images, assuming good lighting and an appropriate pose, but is very sensitive to those factors. There are two main approaches to facial recognition. These are so-called template-based methods and geometric-feature based methods. Template-based methods utilise the whole face and extract features from the full face image, which are then matched to an existing face in a database using a pattern classifier algorithm. Geometric-feature methods locate specific landmarks on the face such as the location of the eyes, nose, chin etc and determine the geometric relationship between them, or alternatively and more recently, match a three-dimensional image of a face to a stored representation. Template matching techniques require an image or a set of images of a person’s face. The facial features are then extracted via a mathematical process and a unique “template” for that face is produced. With the eigenfaces described above, this can result in as little as 2-3kB of data per face. This allows vast numbers of templates to be searched in short amounts of time to find matching faces, at rates of perhaps 100,000 faces per second. So searching a database of all Australian residents for a match could take less than 300 seconds with a modest computer system. Template-based methods can be divided into the following categories: statistical, neural network, hybrid methods (which incorporate both) and other methods. Statistical methods are the most common. Of those statistical methods, PCA and Linear Discriminant Analysis are very popular. Other statistical tools include Independent Component Analysis , Support Vector Machines and kernel methods for PCA and LDA. Of the geometric methods, elastic bunch graph matching is a common method. PCA was also developed into Local Feature Analysis. Fisherfaces (Fig.5) are used with the LDA statistical technique and they are similar to the eigenfaces used with PCA. LDA is less sensitive to lighting variation and facial expressions than PCA and is said to be more accurate overall, but it is computationally more intensive (ie, searching a similarly sized database takes longer). EGBM works similarly to the processes that occur in the human brain when recognising a face. To create a facial model for the database, facial landmarks are determined and nodes are created at these points and joined to one another. The result is a graph, akin to a spider’s web, over the face. Landmarks might include points such as the centre of the eyes, tip of the nose, chin etc. This process is usually carried out with images of multiple different poses. To work well, this method requires facial landmarks to be accurately located, a process that can be assisted by the use of PCA and LDA methods. When it is required to identify an unknown face, the database is searched for the most similar geometric model. Three dimensional (3D) facial recognition is another example of a geometric facial recognition method (see Fig.6). This method records a three-dimensional scan of a subject’s face (known as a “faceprint”; see Fig.7) and uses that to make an identification. It has the advantage that, because it is comparing 3D shapes instead of 2D images, there are no problems that arise from uneven lighting, differing facial orientation, facial expression, makeup etc. 3D images of a face can also siliconchip.com.au Using DNA evidence to reconstruct an unknown face In theory, it is possible to use traces of a criminal suspect’s DNA to reconstruct an image of their face. Already it is possible to determine eye, skin and hair colour from DNA but in the future, DNA phenotyping is said to be able to predict the appearance of a face. A website at which users can predict eye, skin and hair colour from a DNA sequence is at: https://hirisplex.erasmusmc.nl/ Australia’s electronics magazine April 2019  19 Fig.12: the output from Human’s software, showing specific identified individuals and their real-time emotional states, including a ranking for such parameters as angry, happy, afraid, disgust, consent (?), neutral, surprise (!) and sad. be used to generate a 2D image in a specific orientation, to match with photographs in the database that were taken with a similar orientation (see Fig.8). Three-dimensional facial recognition has a high level of accuracy, equivalent to fingerprint identification, but one drawback is that it’s much more difficult to acquire data for the 3D facial database as people are likely to have an aversion to having their face “scanned”, compared to having a simple photograph taken. Nevertheless, the technique is making inroads and is used in the new Apple iPhone X (Fig.9). See the video titled “Using An Infrared Camera To Show How Face ID Works” at siliconchip.com.au/link/aaob Skin texture analysis is a supplemental process to facial recognition. A picture is taken of a section of skin and any distinguishing lines, skin pores and texture analysed and reduced to a mathematical identifier. An example of measurements taken might be the size, shape and distance between pores and/or lines. This technique can improve the accuracy of face recognition alone and can help distinguish between identical twins. Fig.13: the use of facial recognition in China is extensive and advanced. This image comes from Chinese company Megvii (https://megvii.com/) who combine artificial intelligence with their facial recognition technology. This shows Face++ which can detect faces within images; mark 106 facial landmarks; determine face-related attributes including age, gender, emotion, head pose, eye status, ethnicity, face image quality and blurriness; compare two facial images and provide a confidence score as to whether they are the same face or not; and search a database for a match. There are diverse uses for face recognition, both now and in the future. Among these (in no particular order) are: • access control to facilities, computers or mobile devices • for blind people to recognise friends and family • for finding relevant photos on social media platforms • border security • police use • intelligence agency use • military use (eg, identifying terrorists) • identification of unknown people in historical photographs • finding pictures of known people in collections of photographs blers that make too many winning bets so they can also be excluded from the premises in future. A more recent development of facial recognition in casinos is to use software that can determine a gambler’s emotional state, including feelings of anxiety and depression, by analysing subliminal, involuntary facial expressions. These may only last for milliseconds and usually are not noticed by other people (Fig.12). This software is provided by artificial intelligence startup Human (https://wearehuman.io/). In casinos, it is said to be used to identify problem gamblers as a matter of social responsibility. The CEO of Human, Yi Xu said: “The ongoing scanning of people’s emotions and characteristics in casinos and other gambling environments has provided our clients with the ability to flag any extreme highs and lows in players’ emotions, for example, if a player is gambling irresponsibly or while distressed”. Human’s software also has another interesting application. It can be used by poker players to improve their “poker faces” by helping them to train to eliminate any nonverbal cues they may inadvertently give to other players. Beyond the casino, Human’s software can also detect whether someone is lying, disagreeing, nervous or passionate. Applications include identifying the best candidates for a job, minimising human bias, understanding customer feelings and predicting human behaviour by understanding their feelings. Use by casinos Use by the government of China Uses for facial recognition Casinos were early adopters of facial recognition technology for a variety of reasons, including the ability to exclude banned individuals from their establishments, including known “card counters”. Card counting is a gambling technique banned by casinos worldwide as it improves the chances of the gambler to win against the house. Another use is to identify gam20 Silicon Chip China’s government makes widespread use of surveillance, with street cameras spread throughout their cities (Fig.13). The national surveillance system is known as “Xue Liang”, or in English, “Sharp Eyes”. This network is used for crime prevention but could also be used to track political activists or even to enforce their idea of “social credit”, where people who behave in ways Australia’s electronics magazine siliconchip.com.au Fig.14: an image processed (right) with D-ID’s software to protect the biometric data of the individual that is in the original image (left). Facial recognition systems cannot recognise the individual in the processed image, even though it looks almost the same to a human. that are undesirable but not necessarily criminal can be punished in other ways, such as having restricted travel or being prevented from buying certain products. Facial recognition and tracking is combined with all records pertaining to a person such as a criminal record (if any), medical records, travel bookings, online purchases, social media comments, friends on social media or elsewhere with the view of tracking where an individual is, who they are associating with, what they are up to, where they are heading, etc. Apart from Xue Liang’s use of physical records, it combines artificial intelligence, data mining and deep learning technologies to further enhance the system’s effectiveness. In addition to government surveillance cameras, the system also integrates private security cameras from places such as apartment blocks and shopping malls. For more information, see this video from the Washington Post titled “How China is building an all-seeing surveillance state” at siliconchip.com.au/link/aaoa Facial recognition at concerts In April 2018, there was a concert of 60,000 people in China. A wanted person was identified among the vast crowd by facial recognition technology and arrested by authorities for “economic crimes”. The suspect was apparently extremely surprised that he could be identified and pulled out of a crowd of so many people. Letting a neural network decide what features are important in a face A YouTube user by the name of “CodeParade” took 1700 faces and used a neural network program of his own design to encode information from those faces. Without human decision making, the program automatically decided what facial features were most important and assigned them a level of importance. A number of adjustable slider bars were generated which were ranked by the program in order of importance, and these could be adjusted to discover what facial features they corresponded to. It was not always obvious what facial feature(s) the neural network had selected. When the sliders were adjusted, the faces sometimes changed in unusual ways and the changes were dependent upon the position of the other sliders. See the video titled “Video Computer Generates Human Faces” at siliconchip.com.au/link/aaoc siliconchip.com.au Fig.15: PrivacyFilter is another system to modify images, preventing them from being used for face recognition. It was developed by Joey Bose, an engineering student at the University of Toronto. This system has now been developed into a commercial product, “faceshield” (https:// faceshield.ai/) In May 2018, Taylor Swift used facial recognition at one of her concerts to identify any of hundreds of stalkers she claims to have. See the Rolling Stone article at: http:// siliconchip.com.au/link/aand Thwarting facial recognition Many people who have nothing to hide still have concerns about being photographed or recorded without their knowledge. Their biometric data could be added to a database, which may cause problems for them in future, or their presence at certain locations could be logged to some central “Big Brother” database and used to track their movements. One particular concern is the unauthorised use of their image in identity theft, or to gain access to restricted areas or devices such as smartphones protected with facial ID security. As a result, Israeli company D-ID (www.deidentification. co) has developed a method to process pictures and videos to render them unidentifiable by facial recognition systems. Pictures to be protected might be staff pictures on company websites, for example. The images are subtly altered in a way which is barely or not discernible to a person but prohibits identification by a machine (Fig.14). There are also legal ramifications of this because according to the European Union’s General Data Protection Regulation (GDPR), currently in force, face images are regarded as “sensitive personal information” and organisations are required to protect this data or face penalties (no pun intended!). Another approach to thwarting unwanted facial recognition involves the use of 3D printed eyeglass frames and this was the subject of an academic paper; see siliconchip. com.au/link/aanc Unlike the 3D printed glasses that were the subject of this paper, regular glasses can be ignored by more advanced face recognition systems. The Japanese Government’s National Institute of Informatics (NII) developed “privacy visors” in 2015 to thwart unwanted facial recognition. Many other methods have been developed to thwart unwanted facial recognition such as unusual facial makeup or clothing with printed faces etc, but one would hardly go unnoticed! SC Australia’s electronics magazine April 2019  21