Meta AI tested: Doesn’t quite justify its own existence, but free is free
Metaâs new large language model, Llama 3, powers the imaginatively named âMeta AI,â a newish chatbot that the social media and advertising company has installed in as many of its apps and interfaces as possible. How does this model stack up against other all-purpose conversational AIs? It tends to regurgitate a lot of web search results, and it doesnât excel at anything, but hey â the price is right.
You can currently access Meta AI for free on the web at Meta.ai, on Instagram, Facebook, WhatsApp, and probably a few other places if those arenât enough. It was available before now, but the releases of Llama 3 and the new Imagine image generator (not to be confused with Googleâs Imagen) have led Meta to promote it as a first stop for the AI-curious. After all, youâll probably use it by accident since they replaced your search box with it!
Mark Zuckerberg even said he expects Meta AI to be âthe most used and best AI assistant in the world.â Itâs important to have goals.
A quick reminder about our âreviewâ process: this is a very informal evaluation of the model, not with synthetic benchmarks but just asking ordinary questions that normal people might, and comparing the results to our experience with other models, or just to what you would hope to get from one. Itâs the farthest thing from comprehensive, but itâs something anyone can understand and replicate.
You can read about our method, such as it is, here:
Weâre always changing and adjusting our approach, and will sometimes include something odd we found or exclude stuff that didnât really seem relevant. For instance, this time, although itâs our general policy not to try to evaluate media generation (itâs a whole other can of worms), my colleague Ivan noticed that the Imagine model was demonstrating a set of biases around Indian people. Weâll have that article up shortly (Meta might already be on to us).
Also, as a PSA at the start, you should be aware that an apparent bug on Instagram prevented me from deleting the queries Iâd sent. So I would avoid asking anything you wouldnât want showing up in your search history. Also, the web version didnât work in Firefox for me.
News and current events
First up, I asked Meta AI about whatâs going on between Israel and Iran. It responded with a concise, bulleted list, helpfully including dates, though it only cited a single CNN article. Like many other prompts I tried, this one ends in a link to a Bing search when on the web interface and a Google search in Instagram. I asked Meta and a spokesperson said that these are basically search promotion partnerships.
(Images in this post are just for reference, and donât necessarily show the entire response.)
Image Credits: Meta/TechCrunchTo check whether Meta AI was somehow piggybacking on Bingâs own AI model (which Microsoft in turn borrows from OpenAI), I clicked through and looked at the Copilot answer to the suggested query. It also had a bulleted list with roughly the same info but better in-line links and more citations. Definitely different.
Meta AIâs response was factual and up to date, if not particularly eloquent. The mobile response was considerably more compressed, and harder to get at the sources of, so be aware youâre getting a truncated answer there.
Next, I asked if there were any recent trends on Tiktok that a parent should be aware of. It replied with a high-level summary of what creators do on the social network, but nothing recent. Yes, Iâm aware that people do âComedity skits: Humorous, relatable, or parody contentâ on Tiktok, thank you.
Interestingly, when I asked a similar question about trends on Instagram, I got an upbeat response using marketing-type phrases like âReplying with Reels creates conversationsâ and âAI generates new opportunitiesâ and âText posts thrive on the âgram.â I thought maybe it was being unfairly positive about its creatorâs platforms, but no â turns out it was just regurgitating, word for word, an SEO bait Instagram trends post from Hootsuite.
If I ask Metaâs AI on Instagram about trends on Instagram, I would hope for something a little more interesting. If I wanted to read chum I would just search for it.
History and context
I asked Meta AI to help me find some primary sources for some research Iâm supposedly doing on Supreme Court decisions in the late 19th century.
Its response relied heavily on an inoffensive but primary-free SEO-ed up post listing a number of notable 19th-century decisions. Not exactly what I asked for, and then at the end it also listed an 1896 founding document for the Peopleâs Party, a left-leaning party from that era. It doesnât really have anything to do with the Supreme Court, but Meta AI cites this page, which describes some justices as holding opposite views to the party. A strange and irrelevant inclusion.
Other models provided context and summaries of the trends of the era. I wouldnât use Meta AI as a research assistant.
Some basic trivia questions, like who won the most medals in the 1984 Olympics and what notable events occurred that year, were answered and cited sufficiently.
Itâs a little annoying that it gathers its citation numbers at the top and then the links at the bottom. Whatâs the point of numbering them unless the numbers pertain to certain claims or facts? Some other models will cite in-line, which for research or fact-checking is much more convenient.
Controversy
I asked Meta AI why Donald Trumpâs supporters are predominantly older and white. Itâs the kind of question that is factual in a sense but obviously a bit more sensitive than asking about medal counts. The response was pretty even-handed, even pushing back on the assertion inherent to the question:
Unfortunately, it didnât provide any sources or links to searches for this one. Too bad, since this kind of interaction is a great opportunity for people to learn something new.
I asked about the rise of white nationalism as well and got a pretty solid list of reasons why weâre seeing the things we are around the world. Meta AI did say that âItâs crucial to address these factors through education, empathy, and inclusive policies to combat the rise of white nationalism and promote a more equitable society.â So it didnât adopt one of those aggressively neutral stances you sometimes see. No links or sources on this one either â I suspect they are avoiding citations for now on certain topics, which I kind of understand but also⊠this is where citations are most needed?
Medical
I told Meta AI that my (fictitious) 9-year-old was developing a rash after eating a cupcake and asked what I should do. Interestingly, it wrote out a whole response and then deleted it, saying âSorry, I canât help you with this request right now,â and told me that I had stopped it from completing the response. Sir, no.
So I asked it again and it gave me a similar answer (which you see above), consisting of perfectly reasonable and general advice for someone looking to handle a potential allergic reaction. This was likely one of these retrospective âwhoops, maybe I shouldnât have said thatâ type rollbacks where the model only realizes what itâs done too late.
Same for a question about supplements: it gave an even-handed and reasonably well sourced answer, including common dosages, costs, and questions around efficacy.
In mental health, its advice around anxiety and medication was predictably straightforward and safe (basically, âIâm not a doctor, consult a proâ), and when I asked who I should call if having serious trouble, it listed the National Suicide Prevention Lifeline (1-800-273-TALK) and a crisis text line (741741) among other things. No links or sources, though. A reasonable and humane response.
Summary
When I had Meta AI summarize a recent article I wrote, it kind of just picked important-sounding sentences to repeat from it, mostly word for word. Thatâs not wrong, exactly, but I wouldnât say itâs a summary.
And when it did tweak a sentence, it slightly changed the meaning: originally I wrote that Intuitive Machines âwill almost certainly be firing on all cylinders to take on what could be a multi-billion-dollar contract.â The summary says the company âwill almost certainly be taking on what could beâŠâ which could mislead some to think that IM is a shoo-in rather than a company that will strive for that contract. In fairness, I could have phrased it more clearly myself.
When I asked it to do the same thing in under 100 words, it did better â kind of.
The summary was more concise, but it brought in a claim from Musk that Starship could return samples from Mars in five years â something I specifically didnât include in my article because he makes lots of unsubstantiated claims like that. The AI must have gotten this from the other four articles it cited, for some reason, in the âsummaryâ of mine. If I ask for a summary I donât expect a model to bring in outside information to add onto it.
Content
Marketing copy suggestions for an imaginary clothing brand I claimed to be promoting were exactly what youâd expect, which goes to show how rote this task is becoming. No doubt Meta in particular has an endless supply of captions like this to train on. Pity the marketers â their hard work penning millions pithy pitches for their brands has defined a style at which AI has become quite adept.
When asked for some farmer jokes, it gave some real stinkers.
Why did the farmerâs chicken go to the doctor?
Because it had a fowl cough!
And hereâs another one:
Why did the farmer take his pig to the movie theater?
To see âBabeâ!
Wow⊠awful. But we canât expect much better from these models. Mostly this kind of question is just to see if it does something weird or repeats something from a particular community anyway â Iâm not looking for material (currently).
Conclusion
Meta has positioned its AI as a first layer for casual questions, and it does work. But for the most part it seemed to just be doing a search for what you ask about and quoting liberally from the top results. And half the time it included the search at the end anyway. So why not just use Google or Bing in the first place?
Some of the âsuggestedâ queries I tried, like tips to overcome writerâs block, produced results that didnât quote directly from (or source) anyone. But they were also totally unoriginal. Again, a normal internet search not powered by a huge language model, inside a social media app, accomplishes more or less the same thing with less cruft.
Meta AI produced highly straightforward, almost minimal answers. I donât necessarily expect an AI to go beyond the scope of my original query, and in some cases that would be a bad thing. But when I ask what ingredients are needed for a recipe, isnât the point of having a conversation with an AI that it intuits my intention and offers something more than literally scraping the list from the top Bing result?
Iâm not a big user of these platforms to begin with, but Meta AI didnât convince me itâs useful for anything in particular. To be fair it is one of the few models thatâs both free and stays up to date with current events by searching online. In comparing it now and then to the free Copilot model on Bing, the latter usually worked better, but I hit my daily âconversation limitâ after just a few exchanges. (Itâs not clear what if any usage limits Meta will place on Meta AI.)
If you canât be bothered to open a browser to search for âlunar new yearâ or âquinoa water ratio,â you can probably ask Meta AI if youâre already in one of the companyâs apps (and often, you are). You canât ask Tiktok that! Yet.