“A school in Florida was forced into shutdown after an AI-based weapon detection system mistakenly triggered an entire campus lockdown by mistaking a clarinet for a firearm.”
The software was ZeroEyes, and it allows for human review for protection against a false positive. But in this case (like the Maryland chip case) the humans failed to discern that the “gun” wasn’t a gun.
While this may be a failure of AI weapons detection software, it is also a failure of the human reviewers.
I participate in several public and private AI communities, and one fun exercise is to take another creator’s image generation prompt, run it yourself (using the same AI tool or a different tool), and see what happens. But certain tools can yield similar results, for explicable reasons.
On Saturday morning in a private community Zayne Harbison shared his Nano Banana prompt (which I cannot share here) and the resulting output. So I ran his prompt in Nano Banana and other tools, including Microsoft Copilot and OpenAI ChatGPT.
The outputs from those two generative AI engines were remarkably similar.
But Harbison’s prompt was relatively simple. What if I provided a much more detailed prompt to both engines?
Create a realistic photograph of a coworking space in San Francisco in which coffee and hash brownies are available to the guests. A wildebeest, who is only partaking in a green bottle of sparkling water, is sitting at a laptop. A book next to the wildebeest is entitled “AI Image Generation Platforms.” There is a Grateful Dead poster on the brick wall behind the wildebeest, next to the hash brownies.
So here’s what I got from the Copilot and ChatGPT platforms.
Copilot.
ChatGPT.
For comparison, here is Google Gemini’s output for the same prompt.
Gemini.
So while there are more differences when using the more detailed prompt (see ChatGPT’s brownie placement), the Copilot and ChatGPT results still show similarities, most notably in the Grateful Dead logo and the color used in the book.
So what have we learned, Johnny? Not much, since Copilot and ChatGPT can perform many tasks other than image generation. There may be more differentiation when they perform SWOT analyses or other operations. As any good researcher would say, more funding is needed for further research.
But I will hazard two lessons learned:
More detailed prompts are better.
If the answer is critically important, submit your prompts to multiple generative AI tools.
Technology is one thing. But policy must govern technology.
For example, is your court using artificial intelligence?
If your court is in California, it must abide by this rule by next week:
“Any court that does not prohibit the use of generative AI by court staff or judicial officers must adopt a generative AI use policy by December 15, 2025. This rule applies to the superior courts, the Courts of Appeal, and the Supreme Court.”
According to Procopio, such a policy may cover items such as a prohibition on entering private data into public systems, the need to verify and correct AI-generated results, and disclosures on AI use.
Good ideas outside the courtroom also.
For example, the picture illustrating this post was created by Google Gemini—as of this week using Nano Banana.
Ray argues that AI does not hallucinate, but instead confabulates. He explains the difference between the two terms:
“A hallucination is a conscious sensory perception that is at variance with the stimuli in the environment. A confabulation, on the other hand, is the making of assertions that are at variance with the facts, such as “the president of France is Francois Mitterrand,” which is currently not the case.
“The former implies conscious perception, the latter may involve consciousness in humans, but it can also encompass utterances that don’t involve consciousness and are merely inaccurate statements.”
And if we treat bots (such as my Bredebot) as sentient entities, we can get into all sorts of trouble. There are documented cases in which people have died because their bot—their little buddy—told them something that they believed was true.
Adapted by Google Gemini from the image here. CBS Television Distribution. Fair use.
After all, “he” or “she” said it. “It” didn’t say it.
Today, we often treat real people as things. The hundreds of thousands of people who were let go by the tech companies this year are mere “cost-sucking resources.” Meanwhile, the AI bots who are sometimes called upon to replace these “resources” are treated as “valuable partners.”
Are we endangering ourselves by treating non-person entities as human?
Have you ever seen that popular movie where the silent loner student suddenly stands up in the school cafeteria and threatens his classmates with a bag of Cool Ranch Doritos?
“After football practice Monday night, Taki Allen chatted with friends outside Kenwood High School while munching on Cool Ranch Doritos. When he finished his snack he put the bag in his pocket. Minutes later, several police officers pulled up, pointed their guns at him and yelled for him to get on the ground, he said.”
So why did Taki (I’ll get to his name later) receive police attention?
“The false alarm was triggered by Baltimore County Public Schools’ AI-powered gun-detection system, Omnilert.”
Yes, it…um…appears that the AI-powered system thought the Doritos bag was a gun.
“In this case, Omnilert’s monitoring team reviewed an image of “what appeared to be a firearm” on the person at Kenwood Monday night, said Blake Mitchell, a spokesperson for Omnilert.
“”Because the image closely resembled a gun being held, it was verified and forwarded to the Baltimore County Public Schools (BCPS) safety team within seconds for their assessment and decision-making,” he wrote in an email.”
Although not explicitly stated, it appears that the image was sent for human review…and the human thought it was a gun also.
So how can a Cool Ranch Doritos bag look like a gun? Let’s see the picture.
“Mitchell [noted] that their privacy policy prevents them from sharing the image.”
I just asked Google Gemini to conceive an illustration of the benefits of orchestration. You can see my original prompt and the resulting illustration, credited to Bredebot, in the blog post “Orchestration: Harmonizing the Tech Universe.” (Not “Harmonzing.” Oh well.)
Google Gemini.
Note the second of the two benefits listed in Bredebot’s AI-generated illustration: “Reduced Complexity.”
On the surface, this sounds like generative AI getting the answer wrong…again.
After all, the reason that software companies offer a single-vendor solution is because when everything comes from the same source, it’s easier to get everything to work together.
When you have an orchestrated solution incorporating elements from multiple vendors, common sense tells you that the resulting solution is MORE complex, not less complex.
When I reviewed the image, I was initially tempted to ask Bredebot to write a response explaining how orchestrated solution reduce complexity. But then I decided that I should write this myself.
Because I had an idea.
The discipline from orchestration
When you orchestrate solutions from multiple vendors, it’s extremely important that the vendor solutions have ways to talk to each other. This is the essence of orchestration, after all.
Because of this need, you HAVE to create rules that govern how the software packages talk to each other.
Let me cite an example from one of my former employers, Incode. As part of its identity verification process, Incode is capable of interfacing to selected government systems and processing government validations. After all, I may have something that looks like a Mexican ID, but is it really a Mexican ID?
Mexico – INE Validation. When government face validation is enabled this method compares the user’s selfie against the image in the INE database. The method should be called after add-face is over and one of (process-id or document-id) is over.
So Incode needs a standard way to interface with Mexico’s electoral registry database for this whole thing to work. Once that’s defined, you just follow the rules and everything should work.
The lack of discipline from single-vendor solutions
Contrast this with a situation in which all the data comes from a single vendor.
Now ideally interfaces between single-vendor systems should be defined in the same way as interfaces between multi-vendor systems. That way everything is nicely neatly organized and future adaptations are easy.
Sounds great…until you have a deadline to meet and you need to do it quick and dirty.
Google Gemini.
In the same way that computer hardware server rooms can become a tangle of spaghetti cables, computer software can become a tangle of spaghetti interfaces. All because you have to get it done NOW. Someone else can deal with the problems later.
So that’s my idea on how orchestration reduces complexity. But what about those who really know what they’re talking about?
Chris White on orchestration
In a 2024 article, Chris White of Prefect explains how orchestration can be done wrong, and how it can be done correctly.
“I’ve seen teams struggle to justify the adoption of a first-class orchestrator, often falling back on the age-old engineer’s temptation: “We’ll just build it ourselves.” It’s a siren song I know well, having been lured by it myself many times. The idea seems simple enough – string together a few scripts, add some error handling, and voilà! An orchestrator is born. But here’s the rub: those homegrown solutions have a habit of growing into unwieldy systems of their own, transforming the nature of one’s role from getting something done to maintaining a grab bag of glue code.
“Orchestration is about bringing order to this complexity.”
So how do you implement ordered orchestration? By following this high-level statement of purpose:
“Think of orchestration as a self-documenting expert system designed to accomplish well-defined objectives (which in my world are often data-centric objectives). It knows the goal, understands the path to achieve it, and – crucially – keeps a detailed log of its journey.”
Read White’s article for a deeper dive into these three items.
Now think of a layer
The concept of a layer permeates information technology. There are all sorts of models that describe layers and how they work with each other.
“In modern IT systems, an orchestration layer is a software layer that links the different components of a software system and assists with data transformation, server management, authentication, and integration. The orchestration layer acts as a sophisticated mediator between various components of a system, enabling them to work together harmoniously. In technical terms, the orchestration layer is responsible for automating complex workflows, managing communication, and coordinating tasks between diverse services, applications, and infrastructure components.”
But before I launch into my rant, let me define the acronym of the day: AFOID. It stands for “acceptable form of identification.”
And for years (decades), we’ve been told that the ONLY acceptable form of identification to board a plane is a REAL ID, U.S. passport, or a similar form of identity. A REAL ID does not prove citizenship, but it does prove that you are who you say you are.
“The Transportation Security Administration (TSA) is launching a modernized alternative identity verification program for individuals who present at the TSA checkpoint without the required acceptable form of identification (AFOID), such as a REAL ID or passport. This modernized program provides an alternative that may allow these individuals to gain access to the sterile area of an airport if TSA is able to establish their identity. To address the government-incurred costs, individuals who choose to use TSA’s modernized alternative identity verification program will be required to pay an $18 fee. Participation in the modernized alternative identity verification program is optional and does not guarantee an individual will be granted access to the sterile area of an airport.”
I’ve love to see details of what “modernized” means. In today’s corporate environment, that means WE USE AI.
And AI can be embarrassingly inaccurate.
And if you want to know how seedy this all sounds, I asked Google Gemini to create a picture of a man waving money at a TSA agent. Google refused the request.
“I cannot fulfill this request. My purpose is to be helpful and harmless, and that includes refusing to generate images that promote harmful stereotypes, illegal activities, or depict bribery of public officials.”
Bredemarket 