AprilTag with Python

In this tutorial, you will learn how to perform AprilTag detection with Python and the OpenCV library.

AprilTags are a type of fiducial marker. Fiducials, or more simply “markers,” are reference objects that are placed in the field of view of the camera when an image or video frame is captured.

The computer vision software running behind the scenes then takes the input image, detects the fiducial marker, and performs some operation based on the type of marker and where the marker is located in the input image.

AprilTags are a specific type of fiducial marker, consisting of a black square with a white foreground that has been generated in a particular pattern (as seen in the figure at the top of this tutorial).

The black border surrounding the marker makes it easier for computer vision and image processing algorithms to detect the AprilTags in a variety of scenarios, including variations in rotation, scale, lighting conditions, etc.

You can conceptually think of an AprilTag as similar to a QR code — a 2D binary pattern that can be detected using computer vision algorithms. However, an AprilTag only holds 4-12 bits of data, multiple orders of magnitude less than a QR code (a typical QR code can hold up to 3KB of data).

So, why bother using AprilTags at all? Why not simply use QR codes if AprilTags hold such little data?

The fact that AprilTags store less data is actually a feature and not a bug/limitation. To paraphrase the official AprilTag documentation, since AprilTag payloads are so small, they can be more easily detected, more robustly identified, and less difficult to detect at longer ranges.

Basically, if you want to store data in a 2D barcode, use QR codes. But if you need to use markers that can be more easily detected in your computer vision pipeline, use AprilTags.

Fiducial markers such as AprilTags are an integral part of many computer vision systems, including but not limited to:

• Camera calibration
• Object size estimation
• Measuring the distance between the camera and an object
• 3D positioning
• Object orientation
• Robotics (i.e., autonomously navigating to a specific marker)
• etc.

One of the primary benefits of AprilTags is that they can be created using basic software and a printer. Just generate the AprilTag on your system, print it out, and include it in your image processing pipeline — Python libraries exist to automatically detect the AprilTags for you!

In the rest of this tutorial, I will show you how to detect AprilTags using Python and OpenCV.

To learn how to detect AprilTags with OpenCV and Python, just keep reading.

AprilTag with Python

In the first part of this tutorial, we will discuss what AprilTags and fiducial markers are. We’ll then install apriltag, the Python package we’ll be using to detect AprilTags in input images.

Next, we’ll review our project directory structure and then implement our Python script used to detect and identify AprilTags.

We’ll wrap up the tutorial by reviewing our results, including a discussion on some of the limitations (and frustrations) associated with AprilTags specifically.

What are AprilTags and fiducial markers?

AprilTags are a type of fiducial marker. Fiducials are special markers we place in the view of the camera such that they are easily identifiable.

For example, all of the following tutorials used fiducial markers to measure either the size of an object in an image or the distance between specific objects:

• Find distance from camera to object/marker using Python and OpenCV
• Measuring size of objects in an image with OpenCV
• Measuring distance between objects in an image with OpenCV

Successfully implementing these projects was only possible because a marker/reference object was placed in view of the camera. Once I detected the object, I could derive the width and height of other objects because I already know the size of the reference object.

AprilTags are a special type of fiducial marker. These markers have the following properties:

1. They are a square with binary values.
2. The background is “black.”
3. The foreground is a generated pattern displayed in “white.”
4. There is a black border surrounding the pattern, thereby making it easier to detect.
5. They can be generated in nearly any size.
6. Once generated, they can be printed out and added to your application.

Once detected in a computer vision pipeline, AprilTags can be used for:

• Camera calibration
• 3D applications
• SLAM
• Robotics
• Autonomous navigation
• Object size measurement
• Distance measurement
• Object orientation
• … and more!

A great example of using fiducials could be in a large fulfillment warehouse (i.e., Amazon) where you’re using autonomous forklifts.

You could place AprilTags on the floor to define “lanes” for the forklifts to drive on. Specific markers could be placed on large shelves such that the forklift knows which crate to pull down.

And markers could even be used for “emergency shutdowns” where if that “911” marker is detected, the forklift automatically stops, halts operations, and shuts down.

There are an incredible number of use cases for AprilTags and the closely related ArUco tags. I’ll be covering the basics of how to detect AprilTags in this tutorial. Future tutorials on the PyImageSearch blog will then build off this one and show you how to implement real-world applications using them.

Installing the “apriltag” Python package on your system

In order to detect AprilTags in our images, we first need to install a Python package to facilitate AprilTag detection.

The library we’ll be using is apriltag, which, lucky for us, is pip-installable.

To start, make sure you follow my pip install opencv guide to install OpenCV on your system.

If you are using a Python virtual environment (which I recommend, since it is a Python best practice), make sure you use the workon command to access your Python environment and then install apriltag into that environment:

$ workon your_env_name
$ pip install apriltag

From there, validate that you can import both cv2 (your OpenCV bindings) and apriltag (your AprilTag detector library) into your Python shell:

$ python
>>> import cv2
>>> import apriltag
>>> 

Congrats on installing both OpenCV and AprilTag on your system!

Having problems configuring your development environment?

All that said, are you:

• Short on time?
• Learning on your employer’s administratively locked system?
• Wanting to skip the hassle of fighting with the command line, package managers, and virtual environments?
• Ready to run the code right now on your Windows, macOS, or Linux system?

Then join PyImageSearch Plus today! Gain access to PyImageSearch tutorial Jupyter Notebooks that run on Google Colab’s ecosystem right in your browser! No installation required.

And best of all, these notebooks will run on Windows, macOS, and Linux!

Project structure

Before we implement our Python script to detect AprilTags in images, let’s first review our project directory structure:

$ tree . --dirsfirst
.
├── images
│   ├── example_01.png
│   └── example_02.png
└── detect_apriltag.py

1 directory, 3 files

Implementing AprilTag detection with Python

# import the necessary packages
import apriltag
import argparse
import cv2

# construct the argument parser and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-i", "--image", required=True,
	help="path to input image containing AprilTag")
args = vars(ap.parse_args())

Next, let’s load our input image and preprocess it:

# load the input image and convert it to grayscale
print("[INFO] loading image...")
image = cv2.imread(args["image"])
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

Line 14 loads our input image from disk using the supplied --image path. We then convert the image to grayscale, the only preprocessing step required for AprilTag detection.

Speaking of AprilTag detection, let’s go ahead and perform the detection step now:

# define the AprilTags detector options and then detect the AprilTags
# in the input image
print("[INFO] detecting AprilTags...")
options = apriltag.DetectorOptions(families="tag36h11")
detector = apriltag.Detector(options)
results = detector.detect(gray)
print("[INFO] {} total AprilTags detected".format(len(results)))

In order to detect AprilTags in an image, we first need to specify options, and more specifically, the AprilTag family:

A family in AprilTags defines the set of tags the AprilTag detector will assume in the input image. The standard/default AprilTag family is “Tag36h11”; however, there are a total of six families in AprilTags:

1. Tag36h11
2. TagStandard41h12
3. TagStandard52h13
4. TagCircle21h7
5. TagCircle49h12
6. TagCustom48h12

You can read more about the AprilTag families on the official AprilTag website, but for the most part, you typically use “Tag36h11”.

Line 20 initializes our options with the default AprilTag family of tag36h11.

The final step here is to loop over the AprilTags and display the results:

# loop over the AprilTag detection results
for r in results:
	# extract the bounding box (x, y)-coordinates for the AprilTag
	# and convert each of the (x, y)-coordinate pairs to integers
	(ptA, ptB, ptC, ptD) = r.corners
	ptB = (int(ptB[0]), int(ptB[1]))
	ptC = (int(ptC[0]), int(ptC[1]))
	ptD = (int(ptD[0]), int(ptD[1]))
	ptA = (int(ptA[0]), int(ptA[1]))

	# draw the bounding box of the AprilTag detection
	cv2.line(image, ptA, ptB, (0, 255, 0), 2)
	cv2.line(image, ptB, ptC, (0, 255, 0), 2)
	cv2.line(image, ptC, ptD, (0, 255, 0), 2)
	cv2.line(image, ptD, ptA, (0, 255, 0), 2)

	# draw the center (x, y)-coordinates of the AprilTag
	(cX, cY) = (int(r.center[0]), int(r.center[1]))
	cv2.circle(image, (cX, cY), 5, (0, 0, 255), -1)

	# draw the tag family on the image
	tagFamily = r.tag_family.decode("utf-8")
	cv2.putText(image, tagFamily, (ptA[0], ptA[1] - 15),
		cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0), 2)
	print("[INFO] tag family: {}".format(tagFamily))

# show the output image after AprilTag detection
cv2.imshow("Image", image)
cv2.waitKey(0)

We start looping over our AprilTag detections on Line 26.

Each AprilTag is specified by a set of corners. Lines 29-33 extract the four corners of the AprilTag square, while Lines 36-39 draw the AprilTag bounding box on the image.

We also compute the center (x, y)-coordinates of the AprilTag bounding box and then draw a circle representing the center of the AprilTag (Lines 42 and 43).

The last annotation we’ll perform is grabbing the detected tagFamily from the result object and then drawing it on the output image as well.

Finally, we wrap up our Python by displaying the results of our AprilTag detection.

AprilTag Python detection results

Let’s put our Python AprilTag detector to the test! Make sure you use the “Downloads” section of this tutorial to download the source code and example image.

From there, open up a terminal, and execute the following command:

$ python detect_apriltag.py --image images/example_01.png 
[INFO] loading image...
[INFO] detecting AprilTags...
[INFO] 1 total AprilTags detected
[INFO] tag family: tag36h11

Despite the fact that the AprilTag has been rotated, we were still able to detect it in the input image, thereby demonstrating that AprilTags have a certain level of robustness that makes them easier to detect.

Let’s try another image, this one with multiple AprilTags:

$ python detect_apriltag.py --image images/example_02.png 
[INFO] loading image...
[INFO] detecting AprilTags...
[INFO] 5 total AprilTags detected
[INFO] tag family: tag36h11
[INFO] tag family: tag36h11
[INFO] tag family: tag36h11
[INFO] tag family: tag36h11
[INFO] tag family: tag36h11

Here we have a fleet of autonomous vehicles, each with an AprilTag placed on it. We are able to detect all AprilTags in the input image, except for the ones that are partially obscured by other robots (which makes sense — the entire AprilTag has to be in view for us to detect it; occlusion creates a big problem for many fiducial markers).

Be sure to use this code as a starting point for when you need to detect AprilTags in your own input images!

Limitations and frustrations

You may have noticed that I did not cover how to manually generate your own AprilTag images. That’s for two reasons:

1. All possible AprilTags across all AprilTag families can be downloaded from the official AprilRobotics repo.
2. Additionally, the AprilTags repo contains Java source code that you can use to generate your own tags.
3. And if you really want to dive down the rabbit hole, the TagSLAM library contains a special Python script that can be used to generate tags — you can read more about this script here.

All that said, I find generating AprilTags to be a pain in the ass. Instead, I prefer to use ArUco tags, which OpenCV can both detect and generate using it’s cv2.aruco submodule.

I’ll be showing you how to use the cv2.aruco module to detect both AprilTags and ArUco tags in a tutorial in late-2020/early-2021. Be sure to stay tuned for that tutorial!

Credits

In this tutorial, we used example images of AprilTags from other websites. I would like to take a second and credit the official AprilTag website as well as Bernd Pfrommer from the TagSLAM documentation for the examples of AprilTags.

What’s next?

Now that you have completed our first AprilTag tutorial, you have laid a solid foundation for the implementation and use of ‘markers’ in computer vision.

For point-of-reference (pun intended), this was the first of many blog-post tutorials that will take you through real-world application of AprilTags and ArUco tags. Mastery of fiducial tagging will be essential to your AI’s ability to detect, communicate, and develop spatial awareness.

NEXT – if you are interested in building out your computer vision arsenal even further, and learning other advanced computer vision techniques such as:

• Automatic License/Number Plate Recognition (ANPR)
• Face recognition
• Training your own custom object detector
• Deep learning and Convolutional Neural Networks
• Content-based Image Retrieval (CBIR)
• … and much more!

Then, PyImageSearch Gurus is the way to go!

Our community and lessons comprise the most comprehensive computer vision education online today. Based on my personal vault of code and my years of knowledge and experience, you will learn concepts and code through practical application and hands-on experience.

Inside PyImageSearch Gurus, you’ll find:

• An actionable, real-world course on Computer Vision, Deep Learning, and OpenCV. Each lesson in PyImageSearch Gurus is taught in the same hands-on, easy-to-understand PyImageSearch style that you know and love.
• The most comprehensive computer vision education online today. The PyImageSearch Gurus course covers 13 modules broken out into 168 lessons, with over 2,161 pages of content. You won’t find a more detailed computer vision course anywhere else online; I guarantee it.
• A community of like-minded developers, researchers, and students just like you, who are eager to learn computer vision, level-up their skills, and collaborate on projects. I participate in the forums nearly every day. These forums are a great way to get expert advice, both from me as well as the more advanced students.

Take a look at these previous students’ success stories — each of these students invested in themselves and have achieved success. You can too in a short time after you take the plunge by enrolling today.

If you’re on the fence, grab the course syllabus and 10 free sample lessons.

If you are ready to take the first step in achieving a new level of computer vision skill, sign-up for PyImageSearch Gurus. We will be here to guide you through to the finish line!

Summary

In this tutorial, you learned about AprilTags, a set of fiducial markers that are often used for robotics, calibration, and 3D computer vision projects.

We use AprilTags (as well as the closely related ArUco tags) in these situations because they tend to be very easy to detect in real time. Libraries exist to detect AprilTags and ArUco tags in nearly any programming language used to perform computer vision, including Python, Java, C++, etc.

In our case, we used the april-tag Python package. This package is pip-installable and allows us to pass in images loaded by OpenCV, making it quite effective and efficient in many Python-based computer vision pipelines.

Later this year/in early 2021, I’ll be showing you real-world projects of using AprilTags and ArUco tags, but I wanted to introduce them now so you have a chance to familiarize yourself with them.

To download the source code to this post (and be notified when future tutorials are published here on PyImageSearch), simply enter your email address in the form below!

The post AprilTag with Python appeared first on PyImageSearch.