Building a mass college surveillance system using google's facenet + faiss + postgres.
Consider this a part 3 of the series , where I scraped a (1) college's vulnerable library database and (2) made a face mash/hot||not clone. For the third part, I am going to deploy a SOTA facial recognition model for all the 8000+ student faces I scraped from the library's site.
If you, like me, haven't been following the state of the art in facial recognition, would't be aware that how GOOD these models have gotten over the years.
Experiments show that human beings have 97.53% accuracy on facial recognition tasks whereas some models like googles facenet have already reached and passed that accuracy level. (facet has 99.65% accuracy on Labeled Faces in the Wild benchmark)
serengil has made an awesome wrapper for all the popular soda models like facenet, egg-face etc. called deepFace (not to be confused with facebook's deepface model).
First order of business is getting all the images from our database with all the enums.
We do so by using this simple command >
Next we'll need to convert these images to vector embeddings generated by the facenet mode. We'll use deepFace deepface.represent() method to do the same. It can take in base64 image as its input, which is perfect for our use case.
Then we package this data into a 1d vector and push into a representation list. faiss library we'll use in the future will required a 2d vector. So we'll use a simple for loop to get a vector of (8000,512) dims (facenet512 produces a array of 512 chunks).
Then we encode everything into a numpy array and save it on the disk. This way we won't have to generate the embeddings again when we deploy it to production later.
Next we pass this numpy array to the faiss library to index. Without faiss we would have to calculate the cosine distance between each vector when we search for a similarity match for a face. Faiss is pretty neat overall and allow us to run a nearest neighbour search that runs in millisecond even on gigantic multi-million vector datasets. check it out here.
To search for a match in an image we first convert the image to its vector representation then make a 3d np array and use index.search method to find the k most similar faces in the database.
Faiss return the index of the vector, we can pass this index to our representation vector to get the enum of the face.
And voila! we have made ourselves a super scalable surveillance system. Now to actually deploy this we'll first have to go over some trivial hurdles.
Here's the quick and dirty jupyter notebook showcasing what I described above.
In [ ]:
mName = "Facenet512"
In [ ]:
#Connect to DB
import psycopg2
try:
conn = psycopg2.connect("dbname='nalanda' user='postgres' host='localhost' password='password'")
print("connected")
except:
print("I am unable to connect to the database")
In [ ]:
cur = conn.cursor()
cur.execute("SELECT enum,profile from raw limit 500")
rows = cur.fetchall()
len(rows)
In [ ]:
from deepface import DeepFace
import numpy as np
representation = []
failed_enums = []
for row in rows:
try:
vector = DeepFace.represent(row[1],model_name=mName)
except:
print("failed to process enum",row[0])
failed_enums.append(row[0])
continue
pair = []
pair.append(row[0])
print(row[0])
pair.append(vector)
representation.append(pair)
embeddings = []
for i in range(0, len(representation)):
embedding = representation[i][1]
embeddings.append(embedding)
embeddings = np.array(embeddings, dtype='f')
np.save("embeddings",embeddings)
In [ ]:
import faiss # make faiss available
index = faiss.IndexFlatL2(embeddings.shape[1]) # build the index
In [ ]:
index.add(embeddings)
print(index.ntotal)
In [ ]:
testVector = DeepFace.represent(img_path = "test.jpg",model_name=mName)
target_representation = np.array(testVector, dtype='f')
target_representation = np.expand_dims(target_representation, axis=0)
In [ ]:
k = 20
distances, neighbors = index.search(target_representation, k)
representation[neighbors[0][0]][0]
# distances, neighbors
Connecting DB and Making it deployable to ☁️
In [ ]:
#Connect to DB
import psycopg2
try:
conn = psycopg2.connect("dbname='nalanda' user='postgres' host='localhost' password='password'")
print("connected")
except:
print("I am unable to connect to the database")
cur = conn.cursor()
In [ ]:
import numpy as np
embeddings = np.load("all_embeddings.npy",allow_pickle=True)
In [ ]:
import faiss # make faiss available
index = faiss.IndexFlatL2(embeddings.shape[1]) # build the index
In [ ]:
index.add(embeddings)
print(index.ntotal)
In [ ]:
from deepface import DeepFace
mName = "Facenet512"
testVector = DeepFace.represent(img_path = "d.jpeg",model_name=mName)
target_representation = np.array(testVector, dtype='f')
target_representation = np.expand_dims(target_representation, axis=0)
In [ ]:
k = 5
distances, neighbors = index.search(target_representation, k)
ids = (int(neighbors[0][0]),int(neighbors[0][1]),int(neighbors[0][2]),int(neighbors[0][3]),int(neighbors[0][4]))
In [ ]:
cur.execute("select student.fname, student.lname from student join address on student.enum=address.enum join contact on student.enum=contact.enum join raw on student.enum=raw.enum join facenet on student.enum=facenet.enum where facenet.index=%s or facenet.index= %s or facenet.index= %s or facenet.index= %s or facenet.index= %s;",(ids))
cur.fetchall()
In [ ]:
cur.execute("select * from student join address on student.enum=address.enum join contact on student.enum=contact.enum join raw on student.enum=raw.enum join facenet on student.enum=facenet.enum where facenet.index=%s;",(int(neighbors[0][0]),))
cur.fetchall()
I created a new table with all the enums of the people whose images were available in the database with the index of those enums in the faiss database. Then it was just the game of converting input images to vectors and running a faiss search to get the nearest neighbor and find the enum of that index number in the DB and joinig it with the rest of tables to get the complete info.
Here's the modified notebook which does the same while making embeddings.
In [ ]:
mName = "Facenet512"
In [ ]:
#Connect to DB
import psycopg2
try:
conn = psycopg2.connect("dbname='nalanda' user='postgres' host='localhost' password='password'")
print("connected")
except:
print("I am unable to connect to the database")
In [ ]:
cur = conn.cursor()
cur.execute("SELECT enum,profile from raw where LENGTH(profile) > 100 order by enum limit 10")
rows = cur.fetchall()
len(rows)
In [ ]:
from deepface import DeepFace
import numpy as np
index = 0 #representation length
representation = []
failed_enums = []
for row in rows:
try:
vector = DeepFace.represent(row[1],model_name=mName)
cur.execute("INSERT INTO facenet (enum, index) VALUES (%s, %s)",(row[0],index))
conn.commit()
except:
print("failed to process enum",row[0])
failed_enums.append(row[0])
continue
pair = []
pair.append(row[0])
print(row[0])
pair.append(vector)
representation.append(pair)
index+=1
embeddings = []
for i in range(0, len(representation)):
embedding = representation[i][1]
embeddings.append(embedding)
embeddings = np.array(embeddings, dtype='f')
np.save("embeddings",embeddings)
In [ ]:
import faiss # make faiss available
index = faiss.IndexFlatL2(embeddings.shape[1]) # build the index
In [ ]:
import numpy as np
embeddings = np.load("all_embeddings.npy",allow_pickle=True).shape
In [ ]:
index.add(embeddings)
print(index.ntotal)
In [ ]:
testVector = DeepFace.represent(img_path = "test.jpg",model_name=mName)
target_representation = np.array(testVector, dtype='f')
target_representation = np.expand_dims(target_representation, axis=0)
In [ ]:
k = 20
distances, neighbors = index.search(target_representation, k)
representation[neighbors[0][0]][0]
# distances, neighbors
In [ ]:
In part 2 I'll explain how I made a custom gradio frontend for it and pushed it to the cloud for ez inference.