Holy smokes Hilary, you actually built the darn thing .
A lot of people have been toying with this idea in their heads for a while but I don’t know of anyone that actually put the effort in .
How far have you pushed it so far in terms of stacked genes? Are we talking single genes or can it decipher 2/3/4 genes in one animal?
You would be extremely surprised. Dustin Sandlin recently worked on a simple device that attaches to preexisting CCTV cameras and can detect a firearm (and differentiate between a gun/phone/food/ stapler…) present in a room, from any angle whatsoever.
The program had a rediculous amount of photos of guns from films, TV, games, CCTV footage though (20,000+) but it worked. This is doable with today’s technology.
So far it’s only doing single genes, I haven’t figured out how to do stacked genes/combos yet. I feel like I’d have to train those in separately, for example if it can detect leopard or banana on it’s own, it won’t be able to call a banana leopard without being trained on banana leopards.
Also, if anyone wants to post a picture of one of their snakes with one of the above listed genes without saying what it is, I’ll test it out and see if it can guess
This is freaking awesome lol. I just wonder how many genes you can stack in before it’s not effective. Also can you use it to tell of lesser and butter are the same thing lol.
That’s what I’m wondering too, because at least in my eyes, some morphs seem almost indistinguishable. Also since I am using all pictures from morph market, some have really weird lighting that make them look brighter than they are, which skews the results a bit.
I have just been grouping lesser and butter as one, but maybe I should separate them?
Update: I added Enchi and pastel and it’s still currently running at 95% accuracy. I’m actually pretty impressed with how well it’s differentiating some of these
However I’m pretty sure I’m going to be hitting a wall with computing power before too long, so if anyone with a pretty solid gaming setup or a super computer wants to take over, I’ll export the project to you
That’s really cool. It will be interesting to see how it would cope with the subtle morphs like yellow-belly, redstripe or blackhead. Have you noticed a difference in accuracy based on the age of a snake? E.g. how some breeders will wait until a hatchling has had a few sheds and gained some size before definitively making an id.
I don’t think it would have a hard time with yellowbelly. It’s actually a fairly easy gene to identify. I think the hardest part would be between morphs that look similar. And multiple gene animals.
Yellowbelly has definitely been one of the genes I’m scared to take on, but I’ll give it a tray next.
Surprisingly, I have not. I was worried this would be an issue as most hatchlings seem to fade somewhat as they age, however I trained it using both hatchlings and adults for each morph, so it seems to work pretty consistently across ages.
Currently it’s running at 95% accuracy guessing normal, pastel, butter/lesser, leopard, mojave, clown, enchi and black pastel, which is pretty good.
My main concern is that it treats each label as an either/or and doesn’t allow one image to have two labels. For example I have it trained to identify leopard, and to identify pastel, however if I upload a pastel leopard, it is going to have to choose between labels and will either classify it as one or the other. I would have to train it specifically to identify pastel leopards as their own label. Which makes it significantly less helpful, so I have to find a way around this issue.
However, I was able to input a picture of an albino black pastel and even though I don’t have it trained to detect albino, it did classify the picture as black pastel, which gives me hope.
Given how much space and power this thing is taking, I’m going to talk to some more knowledgable people about this tomorrow and how to possibly upload it to a server and then also make it accessible to others to play with. Because currently I’m in way deeper than my skill level
You’d have to teach the program how the genes relate and work and manipulate together, here are my 3 suggestions:
A: Teach the program how they relate and how every single gene works and reacts with every single other gene and you end up with millions of possibilities which would probably be best to have breeders upload pictures of every single hatchling with all of the visible genetics to a website and a program would read that and automatically edit the master program adding on more and more genetics and complexity to the program.
B: Have the program look at certain key areas on the snake, example(not realistic), mojave causes a certain head pattern in the heterozygous form that is visible with any combination, the program would know that and automatically look at the snake and divide it into sections. Head, spine, sides, spine right behind the head. From there it would have a database of different possibilities like the one you have now does except with different areas of the body and only looks at those areas, except it looks at all of the areas separately at once to combine the findings into supers, standards, and hets.
C. You manually enter in pictures with the full name, including all visual hets, this would take hours and hours just for a few names let alone thousands which is why if you went this route you’d need to have multiple people working on the project.
These are my ideas, you could mix them or try some different ones and see which one works best.
@erie-herps In this approach, you don’t usually “teach” deep learning models anything directly. You just show them many instances and they learn for themselves. That’s a big part of the beauty of deep learning. It moves away from the manual creation of features and heuristics as we done in the 2000’s.
@chesterhf may I ask what type of model you trained? I assume you’re using some kind of transfer learning, perhaps fine-tuning over a model that was pretrained on ImageNet?
This project on my longer list to play with as well. I knew that impressive results could probably be obtained with just a few hours of playing around. Transfer learning does quite well with a small amount of data, which is good since it might be difficult to gather a large amount of data given that sites like MorphMarket usually have TOS that prohibit scraping. I am also curious how it performs on the harder cases versus experts.
Unfortunately this isn’t feasible as this system looks at images as a whole and forms algorithms based on patterns and similarities. Short of providing accurately labeled images for it to train on, I can’t really teach it anything.
Currently I don’t have hets as their own categories, because even thought they can influence colors and patterning, that would create more problems then it would solve. Given that I need 50+ pictures of each morph to train on, including hets as their own category would take away from the images I need to train the basic morphs and I wouldn’t be able to use possible hets at all. Under the current system, a single gene Mojave, Mojave het pied and a Mojave 66% pos het lavender would all just be labeled as “Mojave”.
Honestly something like this would be ideal if we wanted to forge ahead with creating a morph ID program. I think “crowdsourcing” some of it will be essential in order to get enough images of all the morphs and because of the vast amount of time this is going to consume.
I’m using Microsoft Lobe, which was just released on Oct. 26th and is an image classification machine learning software. It’s essentially a blank slate that chooses the best algorithm/architecture for your model once you start training. Currently I’m using ~50 images per morph to train on and then 10-20 more for practice. Most of the images are from MorphMarket ads and I’ve tried to be vigilant about only using pictures that I was confident were correctly labeled and appeared to be a valid example of what the morph generally looks like.
What do you mean by scraping in this context. Usually scraping is with bots that extract the html and data from the main database along with the pictures which extracts personal information of users and would allow the scraper to copy the website under a different similar domain and make it look like the original and edit the entire website for data harvesting. If you meant scraping as that then I don’t know what harm could be done by @chesterhf using lots of images for the software. If you meant scraping as another term then I don’t know if that would be in violation or not but under this definition I think it would be okay.
I’m not doing all of that, just literally going ad by ad and downloading images and labeling them by morph, so no information is downloaded or attached to them. Since it didn’t fit the definition of scraping I figured it would be fine to use the images for training purposes
That’s why if that definition is accurate to the intent of the rule then you should be fine since you’re extracting pure pictures, not information and definitely no ill-intent.
This is a really cool project. I have zero experience with machine learning but have often wondered if it could be applied.
Random thoughts from someone that knows nothing.
It would be interesting if you could tier the morphs in a sense for the program. If there was a way to designate W/T Normal as the base then incomplete dominate morphs, both heterozygous and homozygous, then hets visual and recessive, allowing for the computer to determine over time how they affect the look of a snake. In other words, defining a punnet square and genetic theory into a program. Over time, it could be combined with a genetic calculator type model and predict changes.
I haven’t done too much with it recently, I think the last morphs I added were Pinstripe and Piebald, and then ran into the immediate problem that even though a “high white”, “medium white” and “low white” piebald all have the same morph label, they look pretty different. It really confused the program and threw things off a bit and I wasn’t sure how to proceed with morphs have have super variable phenotypes (like Pied) or morphs that changed color/pattern as they aged (like Banana). Also as it got more complex I wasn’t sure how much computing power would be required to continue because my laptop only has 16GB of RAM, and I really don’t want to bog it down too much because I need it for work.
Then I realized I had spent entirely too much time focusing on it and really needed to finish my thesis so I can graduate.
So it has a lot of potential, is surprisingly accurate, but I’ve put it to the side for now. If anyone wants to continue with it or collaborate on something I’m happy to to export the project, because it would make a really cool app or addition to Morph Market
.
.
