NEXT JS site metatags not being scraped

Question

NEXT JS site metatags not being scraped

129 Views Asked by Anja Vojta At 20 October 2024 at 05:40

I've built a NEXT JS site and i'm trying to insert metatags that will make my posts on e.g. LinkedIn & FB look nice & customized. Here's the code I've used and inserted into my index.js site:

export default function Home() {
  return (
    <>
      <Layout>
        <Head>
          <meta property="og:type" content="website" />
          <meta property="og:title" content="Business- und Karrierecoaching" />
          <meta
            property="og:description"
            content="Beratung und Begleitung in jeder Phase Ihres Berufslebens!"
          />
          <meta
            property="og:image"
            content="https://anjavojta.com/wp-content/uploads/2023/10/title_photo_blue.png"
          />
          <meta property="og:url" content="https://marliestheresbrunner.at" />
        </Head>
        <Hero />
      </Layout>
    </>
  );
}

Here is my folder structure:

Folder Structure

Still, the tags are not being scraped and if I use https://www.linkedin.com/post-inspector/ it's even telling me that "no og:image is found".

The output being shown (for instance on a LinkedIn post) is always code from my Hero component which is weird. I don't have any metatags inserted in my Hero component. Here is the code to my Hero component (which is inside the folder "components" - see folder structure in the image above):

export default function Hero() {
  return (
    <>
      <Head>
        <title>Business- und Karrierecoaching</title>
      </Head>
      <div css={pageContainer}>
        <div css={heroContainer}>
          <div css={heroHeadingContainer}>
            Business- <br />
            und Karrierecoaching
          </div>
          <div css={heroSubheadingContainerItalic}>
            <i>Einstieg – Aufstieg – Orientierung – Veränderung – Neustart</i>
          </div>
          <div css={heroSubheadingContainer}>
            Beratung und Begleitung in jeder Phase <br />
            Ihres Berufslebens!
          </div>

          <Link href="/offer/" css={buttonStylesBlue}>
            Mein Angebot
          </Link>
          <Link href="/contact" css={buttonStylesLight}>
            Infogespräch
          </Link>

          <div css={signatureStyles}>
            <Image
              src={'/images/signature.png'}
              alt="Unterschrift von Marlies Theres Brunner"
              width={397}
              height={120}
            />
          </div>
        </div>
      </div>
    </>
  );
}

Thank you so much for the help!

Anja

Here's what I've tried so far (and didn't work either):

put the metatags into my _app.js & Hero component Head instead
changed the image url

Original Q&A

There are 2 best solutions below

**Beast80K** · Answer 1

Problem :

the tags are not being scraped and if I use https://www.linkedin.com/post-inspector/ it's even telling me that "no og:image is found".

Possible Cause :

Head Tag placement

The output being shown (for instance on a LinkedIn post) is always code from my Hero component which is weird. Hero Component has title which will get applied to the page in which it is imported (if that page has no title)

Solution :

Place that Head Tag in _app.js, which will make them available on homepage, as well as pages which don't have any Head tag .

You can also place Head tag in _document.js but you can't keep title tag in it (Read the 1st link given below). This metadata is available(appended to) on all pages.

If you have metadata common then place it in _document.js & metadata that changes(title) keep it in its own page /component.

Here's a small example code I made : (copy-paste & run in it)

Outcome : (i made a build & tested for presence of metadata, they were present)

it demonstrates behavior & placement of Head tag & how it affects the page (title & metadata)

Explanation :

The code below has a Hero component with Head tag.
Then a page nomd which has no Head tag (or metadata of any kind, just content)
Then a page md which has its own Head tag
Then a hero page which imports Hero component, it gets metadata from component
In _app.js, I have made links to above pages. Open console, go in Element tab & see metadata.

NextJS Version : 14.0.1

Folder Structure :

projectName
├── .gitignore
├── components
│   └── Hero.js
├── jsconfig.json
├── next.config.js
├── package-lock.json
├── package.json
├── pages
│   ├── api
│   ├── hero
│   │   └── index.js
│   ├── index.js
│   ├── md
│   │   └── index.js
│   ├── nomd
│   │   └── index.js
│   ├── _app.js
│   └── _document.js
├── postcss.config.js
├── public
│   ├── favicon.ico
│   ├── next.svg
│   └── vercel.svg
├── README.md
├── styles
│   └── globals.css
└── tailwind.config.js

_document.js : projectName\pages\_document.js

import { Html, Head, Main, NextScript } from 'next/document'

export default function Document() {
  return (
    <Html lang="en">
      <Head >
        <meta name="keywords" content="MY, WEBSITE, KEYWORDS" />
      </Head>
      <body>
        <Main />
        <NextScript />
      </body>
    </Html>
  )
}

_app.js projectName\pages\_app.js :

import Head from 'next/head'

export default function App({ Component, pageProps }) {
  return (
    <>
      <Head>
        <title>Homepage (title from _app.js)</title>
        <meta
                property="og:description"
                content="ADD YOUR METADATA TAGS, BELOW !!"
            />
      </Head>
      <Component {...pageProps} />
    </>
  )
}

index.js projectName\pages\index.js (i have added links, to easily navigate between pages to see title & metadata change):

import { Inter } from 'next/font/google'
import Link from 'next/link'

const inter = Inter({ subsets: ['latin'] })

export default function Home() {
  return (
    <main
      className={`flex min-h-screen flex-col items-center justify-between p-24 ${inter.className}`}
    >
      <h1>This the Homepage !</h1>

      <Link href='/hero'>Page with Hero Component</Link>

      <br />

      <Link href='/md'>Page with Head tag in it.</Link>
      <br />


      <Link href='/nomd'>Page with no Head tag at all, nor in component.</Link>
    </main>
  )
}

index.js (hero page) projectName\pages\hero\index.js :

import Hero from '@/components/Hero'
import React from 'react'

const index = () => {
    return (
        <div>
            <h1> HERO PAGE </h1>
            <h3>This page has no head tag, Hero component imported which has head tag.</h3>

            <Hero />
        </div>
    )
}

export default index

Hero Component projectName\components\Hero.js:

import Head from 'next/head'
import React from 'react'

const Hero = () => {
    return (
        <>
            <Head>
                <title>Title by Hero Component</title>
            </Head>

            <h3>Hero Component !</h3>
        </>
    )
}

export default Hero

md page projectName\pages\md\index.js :

import Head from "next/head"
    const MDPage = () => {
        return (
            <div>
                <Head>
                     <title>MD PAGE</title>
    
                    <meta
                        property="og:description"
                        content="ADD YOUR METADATA TAGS, BELOW !!"
                    />
                    
                </Head>
    
                <h1>Metadata Page</h1>
                <h3>This page has head tag, which contains meta-data.</h3>
    
            </div>
        )
    }
    
    export default MDPage

nomd page projectName\pages\nomd\index.js:

import React from 'react'

const NoMDPage = () => {
    return (
        <div>
            <h1>NoMDPage</h1>
            <h3>This page has no Head Tags.</h3>

            <p>This page will get Title & Head from _app.js </p>
        </div>
    )
}

export default NoMDPage

Output :

Keep Dev Tool's Elements Tab open,
On going to home page (http://localhost:3000/), Title = Homepage (title from _app.js) & metadata in Head tag from _app.js
On going to hero page (http://localhost:3000/hero), Title = Title by Hero Compononent & metadata in Head tag from Hero Compononent
On going to md page (http://localhost:3000/md), Title = MD PAGE & metadata from its own Head tag
On going to nomd page (http://localhost:3000/nomd), Title = Homepage (title from _app.js) & metadata in Head` tag from _app.js
<meta name="keywords" content="MY, WEBSITE, KEYWORDS"> is present on all pages, which i placed in _document.js

Please Read :

<title> should not be used in _document.js <Head> https://nextjs.org/docs/messages/no-document-title
Metadata : https://nextjs.org/docs/pages/building-your-application/optimizing#metadata
Head: https://nextjs.org/docs/pages/api-reference/components/head

If you still have any doubts, leave a comment (i will update the answer)

**Arpi Antonyan** · Answer 2

Had the same problem and after struggling for over a week, here's what I learned.

Make sure you don't set ssr:false in your _app.js. This was my main issue since my _app.js file was rendering on the client side causing issues with the meta tags.
Keep in mind that in order to check the presence of meta tags you'll need to open the Page source / devtools 'Sources' section. You'll see the HTML code Facebook crawler sees so if there are no tags, the issue is in your code.
When I had this issue, the Facebook debugger was showing the same error message as in your case. This is because (I am not sure if I recall this right) the crawler looks for the image tag first and when failed, immediately shows an error.
After adding meta tags for the title and description, the debugger was showing a badge from my website. I suppose the incorrect data just indicates the absence or issue in meta tags.

If none of these work, please let me know!

NEXT JS site metatags not being scraped

There are 2 best solutions below

Related Questions in WEB-SCRAPING

Related Questions in NEXT.JS

Related Questions in COMPONENTS

Related Questions in META-TAGS

Related Questions in SHARE-OPEN-GRAPH

Trending Questions

Popular # Hahtags

Popular Questions