Stat of the Week. A majority of the 300 chief marketing officers (CMOs) in BCG’s 2026 global survey say they have set up dedicated teams for agentic engine optimization (AEO) and generative engine optimization (GEO). BCG rates the discipline alongside search and paid media optimization. (BCG, June 2026.)
The marketing teams have reacted to this change more quickly than normal; within two years, being cited and referenced by an AI assistant has gone from being something new and unusual to becoming a specific role.
These new roles are based on the idea that the work is editorial: writing better answers, organizing pages, and earning citations. But for many organizations, the main obstacle comes before that, and it’s not a marketing issue. If a machine can’t access your page, it can’t quote it. Getting cited depends on three things, in order: permission, then clarity, then terms. The new teams usually handle the second, share the third, and rarely control the first.
Most websites still block AI crawlers
Speaking at AdExchanger’s Programmatic AI event in Las Vegas in May, Nikhil Kolar, who runs publisher product at Microsoft AI, told the room that four out of five websites currently block traffic from AI bots and crawlers. When that happens, he said, “the content or products that you have are not legible to agents,” so “you get no discovery, no recommendations that you’re a part of and no demand” (AdExchanger, May 2026).
If a crawler hits a challenge page, it can’t read anything, and if it can’t read, it can’t cite your content.
Blocking crawlers can be a smart move, but not in all cases. Jonathan Roberts, chief innovation officer at People Inc., said at the same event that his company allows 38 crawlers but blocks 30,000 to 35,000 each day. He explained that starting with everything blocked gives publishers leverage, though most sites don’t have that kind of influence. His advice is to allow access mainly for retail and merchant sites that want AI assistants to recommend their products. Publishers, whose content is their product, have different priorities.
For most brands, the choice is clear. If your business relies on being recommended rather than just being read, you should keep access open, except in specific cases you can identify.
IT may have already made this decision for you
A home-services franchise network spent six months working on the wrong part of the problem. The marketing team formed a small group and rewrote about 400 local service pages. They replaced brochure-style copy with clear answers to customer questions, listed price ranges and technician credentials, and wrote out service areas in text rather than hiding them in a map widget. The updates were solid, but the number of citations stayed the same.
No one checked if the pages were actually accessible. Eighteen months earlier, after automated scraping increased hosting costs, the web infrastructure team set a rule at the content delivery network to challenge every automated client except a few search engines. The rule worked as intended, and at the time, it didn’t affect anything. The issue only came up when someone in the working group tried to access a service page like an assistant would and got a challenge screen.
It took two weeks to fix the problem and create a shared list of allowed clients, along with a process for quarterly reviews. After that, the rewritten pages started earning citations. The content was always good—it just hadn’t been accessible.
This kind of problem happens when a technical rule remains in place long after its purpose has been fulfilled. No one made a mistake, and the conflict isn’t obvious, because files like robots.txt and bot-management rules are reviewed by infrastructure teams, not marketing.
AI crawlers have different expectations from content
Even if your site is accessible, you’re now dealing with a new kind of reader. Chiefmartec’s 2026 martech landscape census highlights this shift. For twenty years, websites were built for people and search engines. Now, there’s a third audience: machines working for people. These machines “may not want to browse your site” but want to “extract, compare, summarize, verify, and act” (chiefmartec, May 2026).
Spending is following this trend. While the total number of products stayed about the same this year, content management and web experience platforms grew by 21.4%, from 504 to 612 products. Ecommerce platforms grew by 19.9%, from 547 to 656. Chiefmartec points to AEO and GEO as areas where growth is picking up.
This third type of reader needs less exciting things than people do. Mostly, it wants information placed where a parser can find it. It looks for claims it can attribute to a source, specs it can compare directly, prices and availability in text instead of images, and pages it can load without running your JavaScript. Convincing the human visitor is a different task, and one that’s already well understood.
AI crawler access is starting to affect marketing effectiveness
Most marketing teams haven’t faced this third issue yet, and it’s showing up as a business deal, not a technical setting. Microsoft’s Publisher Content Marketplace, launched in February 2026, helps publishers and AI developers set up licensing agreements. By May, eight publishers had joined. Kolar points out an important distinction: using published content to train a model builds its base knowledge, whereas using it to ground a model is a separate matter. In grounding, the model pulls from up-to-date, trusted sources through model context protocol (MCP) connections when it answers questions.
Right now, this is a publisher-focused market, and the idea that it will expand to brands is just an inference here, not a prediction from others. It’s important to understand the difference: for publishers, content is the product, so giving it away is a clear loss. For brands, content is meant to help sales, so sharing it freely is usually the goal. The only exception is information a competitor would pay for, not what a customer needs. This should be written as a policy with specific exceptions, not just a simple on-off switch.
Check the access before you rewrite the pages
You don’t need an extra budget for any of this, and the first steps take only an afternoon.
This week, do two checks at the same time. Ask the main AI assistants the questions your customers ask and see if your business shows up and what gets credited to you. Then, have someone fetch your ten most important pages as a crawler would and see what appears. These two checks will show you where the problem is. If you’re missing from the answers and blocked at the fetch stage, you need to fix access. If the pages load fine but you’re still missing, you need to fix your content.
This quarter, document your access policy: list which automated clients are allowed, why they’re allowed, what they can take, and who approves changes. Also, make sure the person in charge of web infrastructure is involved when marketing sets citation goals, since they’re often making these decisions alone without realizing it.
This is a real challenge, and building new teams is the right response. These teams will prove their value as soon as the pages they’ve updated become accessible, which for many organizations just means changing permissions, not starting a big project. Does anyone in your marketing team know which automated clients your site allowed in yesterday, and who made that decision?
Sources
BCG, “Moving the Agentic Marketing Transformation from Illusion to Reality (BCG CMO Survey 2026),” June 15, 2026 (n=300 CMOs globally across business-to-consumer and business-to-business sectors, supplemented by structured interviews with 50 of them.
AdExchanger, “Microsoft To Publishers: Don’t Block The AI Bots,” May 26, 2026, by Anthony Vargas, reporting from AdExchanger’s Programmatic AI event in Las Vegas. The “four out of five websites” figure is a count stated on stage by Nikhil Kolar, VP of publisher product at Microsoft AI, with no sample, method, or measurement date given; it is an attributed claim rather than research, and AdExchanger’s own article notes the commercial interest (”There is a clearly self-serving angle for Microsoft”). The 38-allowed / 30,000-to-35,000-blocked figures are Jonathan Roberts of People Inc., describing his own company. The roster of eight is Microsoft’s publisher roster, a number Roberts stated rather than a figure about People Inc. The Publisher Content Marketplace is dated “February” in the article with no year stated; February 2026 is inferred from the article’s own May 26, 2026 date.
chiefmartec, “2026 Marketing Technology Landscape Supergraphic: Peak Martech Achieved! (Maybe),” May 5, 2026; annual census of the martech landscape. Category figures are product counts, not revenue or adoption: CMS & web experience management 504 to 612 (+21.4%), ecommerce platforms & carts 547 to 656 (+19.9%).