Making Writing Fun with Data Science
What is Story Squad?
Story Squad is a non-profit children’s application that promotes creative writing by making the writing and drawing process competitive. Story Squad uses Optical Character Recognition and Neural Networks to ensure fair competition. The problem that needed to be solved was improving the matching up of children based on writing and drawing level to allow for a fair competition. The application was using Google Vision API to transcribe the handwritten submissions into text data, and a CNN for scoring the drawings.
What needed to be done:
• Implement Tesseract OCR Model for text data
• Generate synthetic text training data
• Scrape the web for children's drawings to improve the illustration scoring model
• Create endpoints for all of the data to be passed to the backend
Implementation
Tesseract OCR
The issue with the current OCR model, Google Vision API, is that it can become very costly quickly. For this reason, we were tasked with implementing our own OCR model. This posed two challenges, creating the model, and generating handwriting data for the model to train on. The OCR model was created by Victoria Debebe, and the code to generate handwriting data was created by Elizabeth Edwards-Appell.
Drawing Data
The neural network to score children's illustrations were already implemented by a previous developer. The current problem we faced was a lack of quality data. We had around 1300 pictures of drawings done by people between the ages of 5 and 25. We needed drawings from children between 6 and 13. Jeannine Proctor was able to implement a selenium web scraper to obtain appropriate data.
Connecting the models to the backend
I created endpoints for the FastAPI for the new models that were implemented by my colleagues. Prior to creating these new endpoints, there were two endpoints that dealt with the submissions. The first used the Google Vision API to transcribe the handwritten data and passed the string data through a function called “squad_score” to calculate a metric for complexity scoring. The other endpoint was to moderate the images using the Google Vision safe search. Both of these endpoints have been left in until the code is ready to completely stop using GV.
The endpoints I implemented utilize the models that my colleagues created to score both the illustrations and story submissions. To do this I saved the model as a .h5 file and loaded it in from the path.
The endpoint takes a submission ID as an argument and it passes the image URL through the loaded model. The return value is a JSON dictionary that includes the complexity score and the submission ID.
The Future of Story Squad
Currently, story squad is still using the Google Vision API for moderation and text transcription. The endpoints for those are still implemented. The model for scoring the illustrations is well-tuned and the best way to improve accuracy is to acquire more training data.
Once both of the models have been polished and are ready to use the Google Vision API endpoints can be removed. If the stakeholders want to continue to moderate the submissions an alternative option will need to be implemented. Once the illustration scoring is ready to go the clustering endpoint will need to be adjusted to return either both of the complexity scores or an average of the two. Another feature that needs to be implemented is a bot. Since this game is played between four people, if the total number of people playing is not divisible by 4 bots will need to be added in that submit old submissions that are stored in a relational database. Currently, if there are not enough people in a squad (a group of four) a submission from another squad will be substituted to fill the gap.
