Large Language Models Passing Subliminal Messages
I’ve previously written about great research by the team at Anthropic that shows both the strengths and weaknesses of LLMs in their current form. In one of their latest research…
Leaders at enterprises around the world are pushing their teams not only to use AI, but to show results. The imperative is very clear, either become more efficient and innovate using Generative AI, or risk being left in the dust by your competitors. But very few organizations have yet to realize significant results from AI, so what separates the leaders from the laggards?
1. RIGHT USE CASE > A USE CASE > NO USE CASE
There are amazing use cases happening in so many industries and domains. The application of LLMs and Generative AI is truly breathtaking. That said, the number of times LLMs are being used to do things for which there are not only better analytic methods, but for which LLMs are horribly bad is equally breathtaking.
Two years ago, researchers measured subjects’ brainwaves while they were dreaming. They recorded what these subjects described in their dreams, and produced images depicting what they said. Today, researchers can recreate with accuracy subjects’ dreams as they sleep.
If you heard this story, you might think it would be fairly straightforward to have a tax expert simply load your company’s financial documents and request the full tax filings be produced and filed. If it can interpret my dreams, surely it can follow the tax code, right?
Picking the right use cases to get started is incredibly important, and this should be done by people who understand the fundamentals of how LLMs work and what they are good (and not so good) at. While getting started and working with LLMs to try to get some use cases going is good, getting started on the right use cases is much, much better.
2. RESULT ≠ CORRECT ANSWER
Another one of my favorite use cases illustrates a second key lesson for successful AI implementations. When Open AI’s initial LLMs launched, researchers wanted to push the limits and see what these models could do. In one use case, a researcher took the foundational model that had been trained on about 40 different languages and asked the model to translate a 41st language, one that it had not seen before.
Some were surprised when the translation was pretty good, as the model had never seen that specific language, though others suggested that if a person spoke 5 Romance languages and you showed that person a 6th, of course they would be able to translate it fairly well.
So the researchers then made the challenge even more difficult; they provided the model with a conversation between two whales. The translation said that one whale told the other whale that they should get married and have a family. The other whale said “I do” and suggested they live together forever.
Did the LLM get the translation correct? How would you verify the answer?
When the two whales then fought each other and swam to opposite corners of the ocean, one of the researchers exclaimed, “See, they really did get married,” while the other researcher said it was clear the translation was rubbish.
Perhaps, the exact exchange between the two whales was embellished a bit, but this is real research that is happening, and the need to validate the results is critically important. There are actual PhD researchers in the ocean following whales and watching their behavior and comparing it to the LLM translations. Over time, they are tuning and validating the models to successfully make these translations accurate.
If we can validate this use case, you can validate yours. What are you doing to validate not only the initial answer from the models, but the ongoing results, as LLMs likely will not consistently give the same answers over time, whether due to models maturity, temperature parameters within the model, incoming data variation, or a myriad of other factors.
3. EDUCATION ≠ TRAINING
When companies roll out new technologies, it is very normal to offer training. Let’s say your organization wanted to drive better interactive visualizations, and you procured a tool such as Tableau, PowerBI, Qlik, or other software. You then offered a couple of hours of training to your teams. Very quickly you likely had hundreds of dashboards.
While one could say the training worked because your employees clearly know how to build lots of dashboards now, your frustration is that the dashboards are pretty bad. They aren’t used consistently and rarely have measurable ROI.
Instead, what if we were to offer one hour of education, teaching people the best practices of visualization? How do you display data, use KPIs, and make the visualization actionable? This session wouldn’t be about any specific piece of software, but instead cover the higher-level concept of producing visualizations that drive results.
You would likely maximize your investment in visualization tools if instead of two hours of training, people received one hour of training and one hour of education. AI technologies are quite similar, and having your teams receive some foundational education is critically important to accelerate your pace of innovation.
4. SHALLOW END > DEEP END
The challenge to successfully implement AI use cases can be very high if you pick the wrong problem to solve. Imagine you pick a use case where you are going to take customer information, including PII, as well as internal documents, and create a chatbot to directly interact and answer customer questions.
The number of risks and failure points that could occur in this use case is high. How do you prevent one customer from seeing another customer’s PII. What happens if the chatbot gives a bad answer, or insults the customer? What if the internal documents contain sensitive information, how do you ensure that information is not fed into the chatbot? Is your internal data ready for this use case? Do you have the appropriate methods to continuously validate and monitor the solution?
Let’s compare this to a different use case. Your procurement department leverages an LLM to shop for the best available pricing for all of your indirect purchasing items. Where can you buy a laptop computer at a lower price? You could then use this information in an automation, or perhaps an even easier use case, the strategic buyer leverages the data to negotiate a lower rate with the current vendor. In this case, you are taking non-sensitive information (the catalogue description and model number of an item you are procuring) and you are taking publicly available data on the web and simply structuring it to make it easy to consume and use. This is likely similar to using the Google Shopping tab to benchmark pricing.
Teams should start at the shallow end of the pool with these easier use cases. They can then build their muscles around how to execute AI use cases successfully. As your organization’s strength grows, you can then accelerate to the deeper end of the pool.
5. EVERYONE > FEW
Finally, and maybe most importantly, you need to bring everyone along on the journey. AI technology isn’t simply the domain of your data science team. Just as we all know how to use analytic based mapping technologies and can barely remember paper maps, and we all leverage Google to search for information, so too will LLMs and Generative AI become a tool used in almost every knowledge worker’s job.
The best ideas for how to leverage AI to move the business forward will likely come from the business experts, and the better they understand the potential of AI, the more ideas that will emerge. This isn’t to say data scientists will not have a role to play, as they may help implement some solutions and will work with the business to help educate the masses. In the end, companies become analytically mature when everyone is participating on the journey.
References
If you would like to learn more about the use of LLMs to translate animal language, follow this link: https://bioneers.org/can-ai-decode-whale-sounds-project-ceti-is-here-to-find-out-ztvz2504/?utm_medium=email&utm_source=newsletter&utm_campaign=content&utm_content=article
If you would like to learn more about the use of LLMs to translate dreams, follow this link: https://www.kdnuggets.com/2023/03/reading-minds-ai-researchers-translate-brain-waves-images.html
