Adding shortcodes to the Marksmith editor

Occasionally, when creating content using an editor, be it Markdown or WYSIWYG, we need specific parts that exceed standard formatting options.

Whether it's highlighting important information, adding visually enriched snippets or embedding third-party content, the basic editor features often fall short.

This is where adding a short code or callout feature is useful.

In this article, we will learn how to add shortcode support to the Marksmith editor by building a blog with enriched content abilities.

Let's start by understanding what are shortcodes:

What are shortcodes?

Shortcodes, a term popularized by the WordPress CMS, are a way to add enriched content using a text editor, without having to do any custom coding or even knowing about the underlying implementation.

They are also known as callouts, custom content blocks, components, among other terms.

Some use cases for them include:

• Enriched existing HTML tags: like introducing a testimonial using a blockquote tag that also includes a user picture and name, or a button that might trigger a newsletter subscription dialog.
• Third party content embeds: think adding a YouTube video or a podcast player that plays a specific episode, etc. Some editors have integrations with services like Embedly, but we can avoid using those with shortcodes.
• Complex widgets: this includes things like image galleries, interactive components, calculators, real-time data components like a sport results widget, etc.

The applications are numerous, and they actually depend on the business requirements.

What we will build

To demonstrate how the Marksmith editor works, we will build a simple blog with a Post model that has title, excerpt, body and cover attributes.

We will use a Rails scaffold to generate the views so we don't waste much time on that part which, in this case, is not that important.

Then, we will add support for shortcodes using two formats:

• Braces syntax: the shortcodes will use a syntax like {{shortcode}} to display the shortcodes.
• VitePress custom container's syntax: the shortcodes will use a syntax like ::: info to start the block and ::: to mark the end of the block.

We will also have self-closing shortcodes {{newsletter_signup/}} and the support to add attributes to the shortcodes in case our templates need them.

The final result should look something like this:

So, without further ado, let's start by setting our application up:

Application setup

Let's start by creating the application:

$ rails new marksmith_codes --css=tailwind --javascript=esbuild

Next, we will install ActiveStorage:

$ bin/rails active_storage:install

Then, we will add a Post scaffold:

$ bin/rails generate scaffold Post title excerpt body:text

We run the migrations to create the actual tables:

$ bin/rails db:migrate

Now, let's add and configure Marksmith:

Marksmith setup

The first step is to add the gem to our Gemfile and install it:

$ bundle add marksmith && bundle install

Next, we have to add the @avo-hq/marksmith NPM package:

$ yarn add @avo-hq/marksmith

Or, if you're using importmap:

$ bin/importmap pin @avo-hq/marksmith

Next, we will import and register the required controllers. Note that the ListContinuationController is optional, but I think it's usually the expected behavior, so we will add it:

// app/javascript/controllers/index.js
import { MarksmithController, ListContinuationController } from "@avo-hq/marksmith"

application.register("marksmith", MarksmithController)
application.register("list-continuation", ListContinuationController)

Then, we add the Marksmith styles to our layout:

<%# app/views/layouts/application.html.erb %>
<%= stylesheet_link_tag "marksmith" %>

To finish everything, we replace the text_area in the _form partial with the marksmith method of the custom Marksmith FormBuilder extension:

<div class="space-y-2">
  <%= f.label :body, class: "block text-sm font-medium text-gray-700" %>
  <%= f.marksmith :body, value: @post.body, rows: 12 %>
</div>

Finally, and after some styling tweaks using Tailwind, the Post form should look something like this:

Now, we can create a post and render the HTML using the <%== marksmithed @post.body %> helper, and it should look something like this:

{{info}}
There are two ways to render the Marksmith editor: one is using the <%= marksmith_tag :body %> helper, and the other is using the form-specific helper <%= form.marksmith :body %>.
{{/info}}

Adding the shortcodes feature

The core of adding shortcodes or callouts is parsing the content and searching for a given pattern that matches what we expect is an actual shortcode and replacing that with a template.

For example, if we find an {{example}}Content{{/example}} shortcode inside a Markdown field, we could replace that with the actual template HTML:

<div><p>Hello from the example shortcode template</p></div>

This use case seems straight-forward enough: we just use a Regex to identify the shortcode and replace it with the actual HTML, but we actually need a way to make this dynamic and extensible.

If you've been paying attention, we used the <%== marksmithed @post.body %> helper to render our posts content and that's ok because the gem is actually using Redcarpet under the hood to render the HTML from our markdown content.

But, if we want to customize how our Markdown is rendered, we will need to create a custom renderer and for that, we need to use the redcarpet gem.

The first step is to add the gem and install it:

$ bundle add redcarpet && bundle install

Now, we can start actually building the feature by adding a custom Markdown renderer:

The custom renderer

The first step is to actually create a custom renderer under the lib folder that will parse the shortcodes to render the actual HTML.

By default, the library works by taking a Markdown parser that receives a renderer that, by-default, can output HTML or XHTML.

What we will do is create a CustomRenderer which inherits from Redcarpet::Render::HTML and receives Markdown text, parses it using the configuration we pass through the extensions method and returns HTML using the config we pass to it with the renderer_options:

class CustomRenderer < Redcarpet::Render::HTML
  def initialize
    super(renderer_options)
    @markdown = Redcarpet::Markdown.new(self, extensions)
  end

  def render(text)
    @markdown.render(text)
  end

  private

  def extensions
    {
      lax_spacing: true,
      fenced_code_blocks: true,
      tables: true,
      space_after_headers: true,
      autolink: true,
      highlight: true,
      quote: true,
      strikethrough: true,
    }
  end

  def renderer_options
    {
      hard_wrap: true,
      link_attributes: { target: "_blank" }
    }
  end
end

If you'd like to find out about what every config option is doing, feel free to check Redcarpet's documentation.

Now, if we render the content using this custom renderer:

class Post < ApplicationRecord
  ## Rest of the code

  def formatted_body
    CustomRenderer.new.render(body)
  end
end

<%= sanitize @post.formatted_body %>

We would get something very similar to what we saw above using the marksmithed helper.

Now that we're at a similar starting point, let's start by adding our first shortcode:

Adding self-closing shortcodes

These are shortcodes that don't need to “receive” content. They will appear whenever we enclose the shortcode name in double curly braces with a closing slash: {{newsletter/}}

To add them, we will use the preprocess callback that Redcarpet provides, which receives the full Markdown document.

We just need to define the method and do our processing in there:

class CustomRenderer < Redcarpet::Render::HTML
  def initialize
    super(renderer_options)
    @markdown = Redcarpet::Markdown.new(self, extensions)
  end

  def preprocess(document)
    ## Here 
  end

  def render(text)
    @markdown.render(text)
  end

  ## Rest of the code
end

Rendering will be a two-step process: matching the shortcode using a regular expression and replacing it with HTML generated in a partial.

So, to handle self-closing shortcodes we need the following:

def preprocess(document)
  document.gsub!(/\{\{(\w+)\s*\/\}\}/) do |match|
    name = $1
    render_shortcode(name)
  end
  document
end

What the Regex does is:

1. Matches two opening curly braces: {{.
2. Captures one or more word characters (letters, numbers, underscores) and puts that into the first capture group. That's where we get the name from.
3. Matches zero or more white spaces, so {{shortcode/}} and {{shortcode/}}both work.
4. Matches a / followed by two closing curly braces: }}.

Then, we define the render_shortcode method:

def render_shortcode(name)
  result = ApplicationController.render(
    partial: "shortcodes/#{name}",
    layout: false,
  )
  result
  rescue => e
    Rails.logger.error("Error rendering shortcode #{name}: #{e.message}")
    "{{#{name} /}}"
end

It tries to render a partial that has the shortcode's name passing layout: false so we get the partial itself.

If that fails for some reason, we log the error to the console and return the shortcode as it was defined originally.

You might want to return nil or an empty string to avoid rendering textual shortcodes. I actually added like that because I need to cite the shortcodes, but you might have different needs.

Now, we will try adding a banner shortcode to the content and add the partial inside app/views/shortcodes to see if everything is working:

Unfortunately, we're getting this, which doesn't seem to be what we want.

This error occurs because, since version 6.1, Rails includes the action_view.annotate_rendered_view_with_filenames config option that produces HTML comments around each partial to help us debug our applications.

The latest version of Rails, which is version 8, happens to have it by default, so we have to disable it around the shortcode rendering process to make it work correctly.

Before rendering the shortcode we store the config in a variable, we disable the feature, and then we return to the previous configuration:

# lib/custom_renderer.rb
def render_shortcode(name)
  original_setting = ActionView::Base.annotate_rendered_view_with_filenames
  ActionView::Base.annotate_rendered_view_with_filenames = false
  result = ApplicationController.render(
    partial: "shortcodes/#{name}",
    layout: false,
  )
  ActionView::Base.annotate_rendered_view_with_filenames = original_setting
  result
  rescue => e
    Rails.logger.error("Error rendering shortcode #{name}: #{e.message}")
    "{{#{name} /}}"
end

Now, if we try again, we should get the banner like we wanted it to:

Everything works, that's good news! But…

Currently, the feature is only useful to render static partials, which may be insufficient for most use cases. Let's learn how to add parameters:

Self-closing shortcodes with parameters

These are vital when rendering things that might have external IDs: embeds, audio or video players, etc.

We will pass the parameters within the curly braces using a key-value pair for each parameter: {{you_tube id="123456"/}}.

To achieve this, we need to modify the regex in the preprocess method to capture the params and then parse them using a method to return a hash:

def preprocess(document)
  document.gsub!(/\{\{(\w+)(.*?)\/\}\}/) do |match|
    name = $1
    params = parse_params($2)
    render_shortcode(name, params)
  end
end

This new regex adds the (.*?) which captures any character zero or more times into our second capture group that's assigned to the $2 variable.

Using that regex, we would get the following results evaluating {{alert type="warning"}}

# $1 => 'alert'
# $2 => "type='warning'"

Next, we add the parse_params function:

def parse_params(param_string)
  params = {}
  param_string.scan(/(\w+)="([^"]*)"/).each do |key, value|
    params[key.to_sym] = value
  end
  params
end

We're using the Ruby scan method, which can receive a string or a regex to find matches for the patterns it receives in a given string and produce an array of matches.

The regex /(\w+)="([^"]*)"/ has two capture groups: the first is /(\w+) which matches one or more word characters. The second one is ([^"]*) which matches any character that's not a quote occurring zero or more times.

The second capture group is preceded by a literal =" and followed by a literal quote ".

The method now produces the desired params hash, which we are now passing to the render_shortcode:

def render_shortcode(name, params)
  original_setting = ActionView::Base.annotate_rendered_view_with_filenames
  ActionView::Base.annotate_rendered_view_with_filenames = false

  result = ApplicationController.render(
    partial: "shortcodes/#{name}",
    locals: {params: params },
    layout: false,
  )
  ActionView::Base.annotate_rendered_view_with_filenames = original_setting
  result
end

After this, we can create a partial that requires a parameter and use that in the content to see that everything is working fine.

So a YouTube embed sounds like a good example for this:

<%# app/views/shortcodes/_you_tube.html.erb %>
<div class="youtube-embed aspect-video">
  <iframe width="100%" height="100%" src="https://www.youtube.com/embed/<%= params[:id] %>" frameborder="0" allowfullscreen></iframe>
</div>

Now, after we use the shortcode in the content, we should get:

We've made quite some progress. But we're missing the ability to add long form content to the shortcode to display things like callouts, testimonials, rich quotes, etc.

Let's build that now:

Block shortcodes

The process of adding block shortcodes is very similar, except we will add another regex to capture the content inside the curly braces.

The syntax for this type of shortcodes is {{shortcode key="value"}}The content goes here{{/shortcode}}.

So, we now have to add another regex to the preprocess method. This regex has 3 capture groups: the shortcode name, the params and the content.

def preprocess(document)
  ## Matches self closing shortcodes with params:
  ## <div class="youtube-embed aspect-video">
  <iframe width="100%" height="100%" src="https://www.youtube.com/embed/1234567890" frameborder="0" allowfullscreen></iframe>
</div>
  document.gsub!(/\{\{(\w+)(.*?)\/\}\}/) do |match|
    name = $1
    params = parse_params($2)
    render_shortcode(name, params, nil)
  end

  ## Matches shortcodes with content:
  ## {{callout type="success"}}Alert content{{/callout}}
  document.gsub!(/\{\{(\w+)(.*?)\}\}(.+?)\{\{\/\1\}\}/m) do |match|
    name = $1
    params = parse_params($2)
    content = $3
    render_shortcode(name, params, content)
  end
end

The first two capture groups are the same we used for the past example. The third one is using (.+?) which matches one or more characters that are within the opening and closing tags which is essentially the content.

The other change is in \{\{\/\1\}\} which states that the closing braces should include the name of the shortcode to match correctly.

Lastly, the /m at the end modifies the behavior of the . in the regex and makes it capture a multi-line content, which is what we will usually pass to the shortcodes.

Now, we need to add the content argument to the render_shortcode method, and we parse it by calling the @markdown.render method on it and then passing the result to the partial.

def render_shortcode(name, params, content)
  original_setting = ActionView::Base.annotate_rendered_view_with_filenames
  ActionView::Base.annotate_rendered_view_with_filenames = false

  parsed_content = content&.present? ? @markdown.render(content) : nil

  result = ApplicationController.render(
    partial: "shortcodes/#{name}",
    locals: {params: params, content: parsed_content },
    layout: false,
  )
  ActionView::Base.annotate_rendered_view_with_filenames = original_setting
  result
  rescue => e
  Rails.logger.error("Error rendering shortcode")
end

After this, we will add a notice partial:

<%# app/views/shortcodes/_notice.html.erb %>
<div class="relative bg-blue-100 p-5 my-5 border-l-4 rounded-r-lg border-blue-700">
  <div class="w-10 h-10 rounded-full flex items-center justify-center text-blue-700 bg-white absolute -left-6 -top-4">
    <%= inline_svg_tag "info-circle.svg", class: "w-7 h-7 fill-blue-800" %>
  </div>
  <div class="text-blue-950 shortcode-content">
    <%= content.html_safe %>
  </div>
</div>

And now, when we add it to the post content, and we should get the following result:

We could also pass a parameter {{notice title="Remember this"}}Hello from the info shortcode{{/notice}}, include the parameter in the partial:

<%# app/views/shortcodes/_notice.html.erb %>
<div class="relative bg-blue-100 p-5 my-5 border-l-4 rounded-r-lg border-blue-700">
  <div class="w-10 h-10 rounded-full flex items-center justify-center text-blue-700 bg-white absolute -left-6 -top-4">
    <%= inline_svg_tag "info-circle.svg", class: "w-7 h-7 fill-blue-800" %>
  </div>
  <div class="text-blue-950 shortcode-content">
    <% if params[:title].present? %>
      <h3 class="text-base font-bold text-blue-800 mb-0"><%= params[:title] %></h3>
    <% end %>
    <%= content.html_safe %>
  </div>
</div>

And we get the following result:

Currently, the feature is working as we expect, and we should be able to use it to add as many partials as we want, and it should cover almost any scenario.

Of course, your way of implementing the feature can vastly differ from what we showed here. But, all in all, the principles are the same, and you should be able to translate them to fit your needs.

Let's demonstrate that by adding a VitePress styled shortcodes or callouts and make them work

VitePress styled shortcodes

VitePress includes a series of so called custom-containers, which are basically a type of Markdown shortcode just like the ones we added before.

They come in 5 flavors: info, tip, warning, danger and details.

The first 4 are just regular alerts or callouts and the last one uses the details HTML element to show a disclosure widget that expands when clicked and contracts when clicked again, just like an “accordion” component.

The way VitePress defines this custom-containers is the following:

::: info
This is the content that's passed to the interior of the callout
:::

::: details
Are you aware that 80% of 100 is 80?
:::

So, following what we already did, adding them is a matter of adding a Regex that can capture what we require out of these shortcodes.

Let's add it to the preprocess method:

document.gsub!(/^:::[ ]+(\w+)\s+(.*?)\s+^:::/m) do |match|
  name = $1
  content = $2.strip
  render_shortcode(name, {}, content)
end

This regular expression is evaluated after the first two, which means that it only gets read if the shortcode doesn't match the curly brace style.

It has two capture groups to pick the shortcode name and the content.

Now, if we pass the :::info or the :::warning code block, we will get the corresponding partials rendered:

Lastly, we add the details partial to add that feature too:

<%# app/views/shortcodes/_details.html.erb %>
<details class="my-5 p-5 border border-gray-300 rounded-lg">
  <summary>Details</summary>
  <%= content.html_safe %>
</details>

Now, if we add the shortcode to our content, the result should look like this:

If you want specific callouts to render different partials, you can always namespace the partials to handle them differently. You will just have to create different partials under different namespaces like vite_press/info and so on.

Now, to finish the article, we will add GitHub flavored callouts by adding another regex:

GitHub-flavored alerts

The last part of our shortcode feature involves adding the GitHub-flavored alerts to render as shortcodes or callouts.

The basic format of a GitHub alert is:

> [!CAUTION]
> Take into consideration that this is the content for my NOTE alert.

They also come in 5 flavors: NOTE, TIP, IMPORTANT, WARNING and CAUTION.

This means that we can have a name spaced partial for every one of them, so we can pass the flavor to the render_shortcode method.

To simplify things, we will just render the partial that corresponds to the callout name. But you can namespace the rendering to better handle this feature.

The first step is to add the right regex to the preprocess method:

## Matches the GitHub flavored alert shortcodes (must come first):
## > [!NOTE]
## > Content
document.gsub!(/>\s*\[!(\w+)\]\s*\n((?:>.*\n?)*)/m) do |match|
  name = $1.downcase
  content = $2.gsub(/^>\s*/, '').strip
  render_shortcode(name, {}, content)
end

This regex has two capture groups: the shortcode name and the content.

It basically captures a line that starts with > has a whitespace followed by one or more words which constitute the shortcode name, followed by a space and a newline that contains any multi-line content and ends with a newline character.

If we want to imitate the GitHub alerts, we would need to add a partial for each one of them and namespace them if they collide with the shortcodes we already have.

After adding a GitHub callout to the content, we get something like this:

TL;DR

The Marksmith editor is a simple, yet powerful, Markdown editor built for Ruby on Rails applications.

It comes with many features that allow us to concentrate on the content instead of thinking about the formatting.

We can add a shortcode/callout feature to this editor using a custom redcarpet renderer, regular expressions and run-off-the-mill Rails partials.

There are many things we have to consider the content before adding the shortcode feature but, all in all, it's about refining the Regex to make it do what we want.

In this article, we added custom, VitePress and GitHub flavored shortcodes to demonstrate how we can add this feature independent of the format we choose to produce the shortcode templates.

I hope you got to better understand the Marksmith editor and how we can integrate a shortcode feature to it.

Have a good one and happy coding.